[The following notes were generated by Mark Slivkoff.]

 IAMSE Webinar Series, Fall 2018

Speaker: Stanley J. Hamstra, PhD
VP, Milestones Research and Evaluation
Accreditation Council for Graduate Medical Education (ACGME)
Title: Realizing the Promise of Big Data: Learning Analytics in Competency-Based Medical Education
Series: Evolution and Revolution in Medical Education: Technology in the 21st Century

• • Hamstra outlined the following for his webinar:
    • A Review of Milestones
    • Learning Analytics
    • Future Directions
  • Overall, the use of milestones is important because their use helps health professionals remain accountable for the education of our students

   

  A Review of Milestones

  • A special report on milestones was published by Nasca et al. in The New England Journal of Medicine (366:1051-6; 2012).
    • A main point expressed by its authors was that there had been a significant amount of variability in the quality of resident education (graduate medical education, GME)
    • The ACGME, founded in 1981, responded appropriately and the quality of GME has increased over the past 30 years
  • A competency-based approach was necessary to combat the variability characteristic of the traditional model (curriculum with educational objectives and assessments).
  • In a competency-based education model, the health needs and systems are analyzed, then the competencies are built.
  • Competence is multi-dimensional and allows faculty to ask residents:
    • What do they know? (Medical Knowledge)
    • What can they do? (PatientCare)
    • How do they conduct themselves? (Interpersonal and Communication Skills, Professionalism)
    • Are they critical and reflective? (Practice-based Learning and Improvement, Systems-based Practice)
  • Milestones were modeled on the five stages outlined in the Dreyfus Developmental Model of Learning (Novice, Advanced beginner, Competent, Proficient, and Expert).
  • Some of the key points regarding milestones:
    • Articulate shared understanding of expectations
    • Describe trajectory from a beginner in the specialty to an exceptional resident or practitioner
    • Set aspirational goals of excellence
    • Organized under six domains of clinical competency
    • Used as one indicator of a resident’s educational progress
  • Various studies have been done on the effectiveness of milestones. One article considered in the webinar was Conforti et al. (J Surg Educ. 2018;75(1):147-55), who focused on the benefits to the residency Program Director:
    • Changes in the remediation process can be implemented
      • Catching struggling residents earlier
      • Targeted improvements for individual learners
      • Identifying gaps in otherwise high performers
    • Structuring of learning goals
    • Making defensible decisions
      • Milestones provide “built-in”documentation which potential helps mitigate resident’s fighting their evaluation
    • There are numerous sub-competencies (~22) spread across over 130,000 residents, thus crunching the data is a significant task since the number of data points approaches about 3.2 million!

   

  Learning Analytics

  • Learning analytics was broken down into three pieces:
    • Concepts
    • Examples
    • Implementation
  • The purpose of the introduction of milestones revolves around entrustability: can we develop a system to ensure residents and fellows are ready for unsupervised practice by graduation?
  • S. Department of Education describes learning analytics as such: the interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues”
  • Hamstra described a Generic Milestones Template
    • Five levels
    • Emphasis placed on Level 4, What does a graduating resident look like? This level serves as the main target for graduation for most specialties, however, is NOT a requirement for graduation
  • The Data: Cross-sectional analysis at the specialty level…
    • Hamstra noted that his favorite data set, his favorite graph, is the “Proportion of Residents Attaining Level 4 or Higher: PC Sub-Competencies (June 2015) – Neurological Surgery”
      • Data makes sense, according the feedback from residency directors
      • In order of attainment, with PC08 at about 95% and PC03 at about 72%:
        • PC08: Traumatic Brain Injury
        • PC02: Critical Care
        • PC01: Brain Tumor
        • PC06: Spinal Neurosurgery
        • PC05: Pediatric Neurological Surgery
        • PC07: Vascular Neurosurgery
        • PC04: Pain and Peripheral Nerves
        • PC03: Surgical Treatment of Epilepsy and Movement Disorders
      • Conclusion: Pain and Peripheral Nerves, Epilepsy and Movement Disorders are not getting covered at specific sites
      • Change in curriculum design and assessment can be addressed
    • The Data: Longitudinal analysis at the individual level…
      • Various graphs were presented which showed milestone “trajectories” (entrance to graduation) within various specialties including:
        • Surgery, pathology (MK01 competency)
        • Wound management
      • Hamstra discussed the Odds Ratio (OR) for residents not attaining Level 4 under threshold. Students at level 1.5 or above during their first assessment (year 1) have a much greater chance at attaining Level 4 than those who score lower.
    • QUALITATIVE RESEARCH: How do Raters Make Decisions?
      • The phenomenon of “straight-lining” has been extensively looked at as well. Straight-lining is when an evaluator, such as a busy physician, gives the same score for each milestone. This, of course, is not good to do.
      • Hamstra’s group is working on how best to mitigate this phenomenon by combing through lots of specialty and sub-specialty data

  Future Directions

  • Version 2.0 of milestones are currently underway.
  • Overall, the new version (new tables) contain milestones which are more refined

  Questions asked after seminar:

  (Note that some questions and/or answers have been reworded for clarity.)

  How do these measures play in to the overall evaluation process? Some teachers do not want to personally evaluate or judge residents, and residents may take their evaluations personally.
  A couple ways to mitigate the personal nature of evaluations. First, evaluations can be done by groups rather than by individual faculty. Second, as evaluator you can have a conversation with the resident in the beginning, letting them know that they will start at level 1.

  What about the variability between residencies?
  We’re looking at the data to help us address a few questions. Do milestone ratings as a whole differ between large and small residencies? Does the size of the program matter? What is the low hanging fruit, that is, what explains the differences?

  What is a good way to ask for feedback on milestones?
  The “O” score assessment which can be applied to other specialties and skills. We’ve created a form called the OCAT (Ottawa Clinical Assessment Tool). Overall, when building forms for evaluation, you need to keep them simple.

  Do you think that the milestones should reflect the Dreyfus model?
  Dreyfus models seems to be the best fit for designing and building milestones.

  If a student finishes the milestones early, do they finish the program early?
  This is a good question and is a key debate right now. Again, milestones are used to supplement evaluations of program directors. But the gist of competency education is that if you are comfortable in graduating someone early, then go for it. Jury is still out, but in theory we’d strive for this.

  Do you have qualitative data on which students make it to Level 4 earlier?
  We do not, but we also want to ask why didn’t those residents make it to Level 4.

  How can a student reach a level of 2.5 when 2.5 is not defined?
  Half-levels are defined. I didn’t talk about this but there are specific instructions on giving a 2.5. The scale is actually a 9-point scale (1 to 5, in 0.5 increments).

  What software do you use to analyze your data?
  We use SAS, but programmers use other programs as well.

  ---

  Dr. Hamstra can be reached at shamstra@acgme.org

[The following notes were generated by Mark Slivkoff.]

 IAMSE Webinar Series, Fall 2018

Speaker: Douglas Danforth, PhD
The Ohio State University College of Medicine, Columbus, Ohio
Title: Virtual Reality and Augmented Reality in Medical Education
Series: Evolution and Revolution in Medical Education: Technology in the 21st Century

• The objectives of Dr. Danforth’s discussion included:
  • Define and describe virtual reality (VR), augmented reality (AR), and mixed reality (MR)
  • Go through a history of VR, AR, and MR in medical education
  • Describe current technologies
  • Opportunities and examples
  • Describe challenges and future directions
• VR is the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors.
  • Advantage is that you can create simulations that are impossible in the real world
  • VR can be created via software, or by using special video cameras with 360 degree angles
• In contrast to VR, AR is a technology that superimposes a computer-generated image on a user’s view of the real world
  • Advantage is that it augments experiences in the physical world
• MR, also referred to as hybrid reality, is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.
• The history of VR and AR dates back decades:
  • Morton Hellig created the Sensorama in 1957
  • Before this there was the ViewMaster in 1939
  • Flight simulator came out in 1966
  • The term VR was popularized by a VPL scientist, Jaron Lanier, in 1987
• In medical education, VR and AR has their roots at the University of Chicago with CAVE, Cave Automatic Virtual Environment
  • Use of rear projection screens that surround the user
  • User wears 3D glasses
  • University of Toledo and The Ohio State University both have CAVE
  • CAVE environments are expensive
• Second Life (software), also used in medical education, was started in the early 2000s.
  • Used by many academic institutions
  • Danforth used the software to design a testis through which you could fly and learn about all the relevant anatomy and physiology
  • 2007 was the heyday of Second Life’s use, with over 1000 users/month added
  • Not used any more at many institutions
• Current VR technologies include:
  • Google Cardboard
    • Advantages: inexpensive ($10), uses your smartphone, and it’s entry level
    • Disadvantages: low resolution, you can’t walk around in the environment, you can only turn your head with limited to no interaction
    • It’s a great way to expose folks to VR, especially the rollercoaster ride
    • Use with YouTube’s Virtual Reality channel
  • Samsung Gear VR and Google Daydream
    • Advantages: inexpensive, use your smartphone, some interaction
    • Disadvantages: low resolution, stationary VR
  • Oculus Go (and the Quest which just got released)
    • Advantages: relatively inexpensive ($200), built-in display and sound, allows for interaction
    • Allows for 6 degrees of freedom (up, down, right, left, forward, backward)
  • HTC Vive, PlayStation VR, Oculus Rift and Samsung Odyssey
    • Advantages: high resolution, smooth video, allows interaction, mobile VR
    • Disadvantages: expensive ($400 to $800), requires a powerful computer that you have to be tethered to, can be challenging to set up
    • Danforth uses an HTC Vive
  • Current AR technologies: Google Glass, Microsoft HoloLens
    • Advantages: not isolated from surroundings, good for training, access to real time schematics
    • Disadvantages: expensive ($3000), requires a powerful computer (but not too expensive these days…$1500)
    • Microsoft’s Holo Anatomy put to use
  • Opportunities and Examples
    • Anatomy education is the most obvious discipline to target
    • Surgery applications
    • Microsoft HoloLens, HoloAnatomy: https://www.youtube.com/watch?rel=0&start=35&v=SKpKlh1-en0
      • Case Western is extensively using HoloLens
    • Voxel Bay at Nationwide Children’s Hospital: https://www.youtube.com/watch?rel=0&showinfo=0&start=35&v=uVRilk_6UWI
      • The Ohio State U. collaborates with them
      • Used by pain researchers to distract kids from pain
    • Mass Casualty Training at the Ohio State University
      • University used to use an analog subway set which bordered on immersive but “lame”: https://www.youtube.com/watch?rel=0&v=JNdKo1uSRto
      • They have now recreated the set virtually
      • Proof-of-Concept Disaster Sim – Subway bombing (featuring Dr. Danforth): https://www.youtube.com/watch?v=nvuxMnSHHtg
    • Conan Visits YouTube’s VR Lab (hilarious, if you enjoy Conan’s humor): https://www.youtube.com/c126d7c3-4fa3-4c75-a3dc-d35ea8ad155a
    • Visual standardized Patients: https://youtu.be/mvXIruMt9Ek
      • High fidelity, simulated “real” standardized patient
      • Conversational, can understand and respond to student questions
      • Easy to use, require little or no training
      • Soon students will wear VR goggles
    • Challenges of VR and AR
      • Fatigue, disorientation and vertigo
      • headsets are heavy, and there’s usually a lag when watching
      • Difficult to scale, to do multiplayer VR
      • Movement in virtual space
      • Lack of haptic tools
      • Limited interoperability, but software is allowing for some cross-platform compatibility
    • Future of VR and AR
      • Portability
      • Untethered systems
      • Smaller more comfortable headsets
      • Fidelity
      • Higher resolution displays/increased framerates
      • Haptic feedback
      • Multiplayer
      • Team based simulations
      • Interoperability
      • Build once – deploy everywhere
    • Upcoming in Content and Applications
      • Surgical simulation
      • Patient specific simulations
      • Remote surgery
      • Virtual Patients
      • Practice history taking, physical exam skills, differential diagnoses
      • Automated assessment
      • Team training
      • Emergency medicine, surgical stimulation

Questions asked after seminar:
(Note that some questions and/or answers have been reworded for clarity)

Do you suggest any starter tools to get your feet wet in the technology?
Recommend that you collaborate (with gaming folks, the software folks). It’s simple to get started with 360 degree camera. Higher end creation requires software such as Second Life. And you’ll need a programmer who knows Unity or Unreal Engine (software).

How do everyday operating systems (Oss) figure into all this?
Apple has some VR labs but they don’t compare to what I’ve discussed. Some material can be ported to operating systems such as iOS.

Where is the market going?
Single software use for all purposes. Hardware includes the newly released Oculus Quest.

How are your students at The Ohio State University Evaluated?
Pre and Post Tests are administered. Working on building the mass casualty training system similar to games in that there are various difficulty levels. Must pass one level before moving to the next.

Have you received any pushback from certain populations (e.g. students who get sick)?
There has been very little pushback but we have to have alternatives. Enhances in technology should mitigate the sickness factor.

Has VR or AR been mapped to high fidelity mannequins?
Not yet, but someday it is bound to happen. Companies are trying to merge the two. Seeing different things inside the same mannequin is an example.

What about procedural skills such as suturing or lobotomy?
This is much further down the road. The main problem is that there are no commercially available gloves yet that allow for haptic feedback.

You showed a couple VR anatomy simulations. What were they?
3D Orgnan VR Anatomy and Microsoft HoloLens Anatomy.

You showed the Google Cardboard. What’s the entry point for medical education?
Invest in Oculus Rift or HTC Vibe, plus the computer. But Google Cardboard is a great starting point. We’ve recently attended a conference and had a bunch of headsets at our poster.

Related links supplied by audience:
https://link.springer.com/article/10.1007%2Fs10916-016-0459-8