[The following notes were generated by Mark Slivkoff.]

 IAMSE Webinar Series, Fall 2018

Speaker: Stanley J. Hamstra, PhD
VP, Milestones Research and Evaluation
Accreditation Council for Graduate Medical Education (ACGME)
Title: Realizing the Promise of Big Data: Learning Analytics in Competency-Based Medical Education
Series: Evolution and Revolution in Medical Education: Technology in the 21st Century

• • Hamstra outlined the following for his webinar:
    • A Review of Milestones
    • Learning Analytics
    • Future Directions
  • Overall, the use of milestones is important because their use helps health professionals remain accountable for the education of our students

   

  A Review of Milestones

  • A special report on milestones was published by Nasca et al. in The New England Journal of Medicine (366:1051-6; 2012).
    • A main point expressed by its authors was that there had been a significant amount of variability in the quality of resident education (graduate medical education, GME)
    • The ACGME, founded in 1981, responded appropriately and the quality of GME has increased over the past 30 years
  • A competency-based approach was necessary to combat the variability characteristic of the traditional model (curriculum with educational objectives and assessments).
  • In a competency-based education model, the health needs and systems are analyzed, then the competencies are built.
  • Competence is multi-dimensional and allows faculty to ask residents:
    • What do they know? (Medical Knowledge)
    • What can they do? (PatientCare)
    • How do they conduct themselves? (Interpersonal and Communication Skills, Professionalism)
    • Are they critical and reflective? (Practice-based Learning and Improvement, Systems-based Practice)
  • Milestones were modeled on the five stages outlined in the Dreyfus Developmental Model of Learning (Novice, Advanced beginner, Competent, Proficient, and Expert).
  • Some of the key points regarding milestones:
    • Articulate shared understanding of expectations
    • Describe trajectory from a beginner in the specialty to an exceptional resident or practitioner
    • Set aspirational goals of excellence
    • Organized under six domains of clinical competency
    • Used as one indicator of a resident’s educational progress
  • Various studies have been done on the effectiveness of milestones. One article considered in the webinar was Conforti et al. (J Surg Educ. 2018;75(1):147-55), who focused on the benefits to the residency Program Director:
    • Changes in the remediation process can be implemented
      • Catching struggling residents earlier
      • Targeted improvements for individual learners
      • Identifying gaps in otherwise high performers
    • Structuring of learning goals
    • Making defensible decisions
      • Milestones provide “built-in”documentation which potential helps mitigate resident’s fighting their evaluation
    • There are numerous sub-competencies (~22) spread across over 130,000 residents, thus crunching the data is a significant task since the number of data points approaches about 3.2 million!

   

  Learning Analytics

  • Learning analytics was broken down into three pieces:
    • Concepts
    • Examples
    • Implementation
  • The purpose of the introduction of milestones revolves around entrustability: can we develop a system to ensure residents and fellows are ready for unsupervised practice by graduation?
  • S. Department of Education describes learning analytics as such: the interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues”
  • Hamstra described a Generic Milestones Template
    • Five levels
    • Emphasis placed on Level 4, What does a graduating resident look like? This level serves as the main target for graduation for most specialties, however, is NOT a requirement for graduation
  • The Data: Cross-sectional analysis at the specialty level…
    • Hamstra noted that his favorite data set, his favorite graph, is the “Proportion of Residents Attaining Level 4 or Higher: PC Sub-Competencies (June 2015) – Neurological Surgery”
      • Data makes sense, according the feedback from residency directors
      • In order of attainment, with PC08 at about 95% and PC03 at about 72%:
        • PC08: Traumatic Brain Injury
        • PC02: Critical Care
        • PC01: Brain Tumor
        • PC06: Spinal Neurosurgery
        • PC05: Pediatric Neurological Surgery
        • PC07: Vascular Neurosurgery
        • PC04: Pain and Peripheral Nerves
        • PC03: Surgical Treatment of Epilepsy and Movement Disorders
      • Conclusion: Pain and Peripheral Nerves, Epilepsy and Movement Disorders are not getting covered at specific sites
      • Change in curriculum design and assessment can be addressed
    • The Data: Longitudinal analysis at the individual level…
      • Various graphs were presented which showed milestone “trajectories” (entrance to graduation) within various specialties including:
        • Surgery, pathology (MK01 competency)
        • Wound management
      • Hamstra discussed the Odds Ratio (OR) for residents not attaining Level 4 under threshold. Students at level 1.5 or above during their first assessment (year 1) have a much greater chance at attaining Level 4 than those who score lower.
    • QUALITATIVE RESEARCH: How do Raters Make Decisions?
      • The phenomenon of “straight-lining” has been extensively looked at as well. Straight-lining is when an evaluator, such as a busy physician, gives the same score for each milestone. This, of course, is not good to do.
      • Hamstra’s group is working on how best to mitigate this phenomenon by combing through lots of specialty and sub-specialty data

  Future Directions

  • Version 2.0 of milestones are currently underway.
  • Overall, the new version (new tables) contain milestones which are more refined

  Questions asked after seminar:

  (Note that some questions and/or answers have been reworded for clarity.)

  How do these measures play in to the overall evaluation process? Some teachers do not want to personally evaluate or judge residents, and residents may take their evaluations personally.
  A couple ways to mitigate the personal nature of evaluations. First, evaluations can be done by groups rather than by individual faculty. Second, as evaluator you can have a conversation with the resident in the beginning, letting them know that they will start at level 1.

  What about the variability between residencies?
  We’re looking at the data to help us address a few questions. Do milestone ratings as a whole differ between large and small residencies? Does the size of the program matter? What is the low hanging fruit, that is, what explains the differences?

  What is a good way to ask for feedback on milestones?
  The “O” score assessment which can be applied to other specialties and skills. We’ve created a form called the OCAT (Ottawa Clinical Assessment Tool). Overall, when building forms for evaluation, you need to keep them simple.

  Do you think that the milestones should reflect the Dreyfus model?
  Dreyfus models seems to be the best fit for designing and building milestones.

  If a student finishes the milestones early, do they finish the program early?
  This is a good question and is a key debate right now. Again, milestones are used to supplement evaluations of program directors. But the gist of competency education is that if you are comfortable in graduating someone early, then go for it. Jury is still out, but in theory we’d strive for this.

  Do you have qualitative data on which students make it to Level 4 earlier?
  We do not, but we also want to ask why didn’t those residents make it to Level 4.

  How can a student reach a level of 2.5 when 2.5 is not defined?
  Half-levels are defined. I didn’t talk about this but there are specific instructions on giving a 2.5. The scale is actually a 9-point scale (1 to 5, in 0.5 increments).

  What software do you use to analyze your data?
  We use SAS, but programmers use other programs as well.

  ---

  Dr. Hamstra can be reached at shamstra@acgme.org

[The following notes were generated by Mark Slivkoff.]

 IAMSE Webinar Series, Fall 2018

Speaker: Douglas Danforth, PhD
The Ohio State University College of Medicine, Columbus, Ohio
Title: Virtual Reality and Augmented Reality in Medical Education
Series: Evolution and Revolution in Medical Education: Technology in the 21st Century

• The objectives of Dr. Danforth’s discussion included:
  • Define and describe virtual reality (VR), augmented reality (AR), and mixed reality (MR)
  • Go through a history of VR, AR, and MR in medical education
  • Describe current technologies
  • Opportunities and examples
  • Describe challenges and future directions
• VR is the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors.
  • Advantage is that you can create simulations that are impossible in the real world
  • VR can be created via software, or by using special video cameras with 360 degree angles
• In contrast to VR, AR is a technology that superimposes a computer-generated image on a user’s view of the real world
  • Advantage is that it augments experiences in the physical world
• MR, also referred to as hybrid reality, is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.
• The history of VR and AR dates back decades:
  • Morton Hellig created the Sensorama in 1957
  • Before this there was the ViewMaster in 1939
  • Flight simulator came out in 1966
  • The term VR was popularized by a VPL scientist, Jaron Lanier, in 1987
• In medical education, VR and AR has their roots at the University of Chicago with CAVE, Cave Automatic Virtual Environment
  • Use of rear projection screens that surround the user
  • User wears 3D glasses
  • University of Toledo and The Ohio State University both have CAVE
  • CAVE environments are expensive
• Second Life (software), also used in medical education, was started in the early 2000s.
  • Used by many academic institutions
  • Danforth used the software to design a testis through which you could fly and learn about all the relevant anatomy and physiology
  • 2007 was the heyday of Second Life’s use, with over 1000 users/month added
  • Not used any more at many institutions
• Current VR technologies include:
  • Google Cardboard
    • Advantages: inexpensive ($10), uses your smartphone, and it’s entry level
    • Disadvantages: low resolution, you can’t walk around in the environment, you can only turn your head with limited to no interaction
    • It’s a great way to expose folks to VR, especially the rollercoaster ride
    • Use with YouTube’s Virtual Reality channel
  • Samsung Gear VR and Google Daydream
    • Advantages: inexpensive, use your smartphone, some interaction
    • Disadvantages: low resolution, stationary VR
  • Oculus Go (and the Quest which just got released)
    • Advantages: relatively inexpensive ($200), built-in display and sound, allows for interaction
    • Allows for 6 degrees of freedom (up, down, right, left, forward, backward)
  • HTC Vive, PlayStation VR, Oculus Rift and Samsung Odyssey
    • Advantages: high resolution, smooth video, allows interaction, mobile VR
    • Disadvantages: expensive ($400 to $800), requires a powerful computer that you have to be tethered to, can be challenging to set up
    • Danforth uses an HTC Vive
  • Current AR technologies: Google Glass, Microsoft HoloLens
    • Advantages: not isolated from surroundings, good for training, access to real time schematics
    • Disadvantages: expensive ($3000), requires a powerful computer (but not too expensive these days…$1500)
    • Microsoft’s Holo Anatomy put to use
  • Opportunities and Examples
    • Anatomy education is the most obvious discipline to target
    • Surgery applications
    • Microsoft HoloLens, HoloAnatomy: https://www.youtube.com/watch?rel=0&start=35&v=SKpKlh1-en0
      • Case Western is extensively using HoloLens
    • Voxel Bay at Nationwide Children’s Hospital: https://www.youtube.com/watch?rel=0&showinfo=0&start=35&v=uVRilk_6UWI
      • The Ohio State U. collaborates with them
      • Used by pain researchers to distract kids from pain
    • Mass Casualty Training at the Ohio State University
      • University used to use an analog subway set which bordered on immersive but “lame”: https://www.youtube.com/watch?rel=0&v=JNdKo1uSRto
      • They have now recreated the set virtually
      • Proof-of-Concept Disaster Sim – Subway bombing (featuring Dr. Danforth): https://www.youtube.com/watch?v=nvuxMnSHHtg
    • Conan Visits YouTube’s VR Lab (hilarious, if you enjoy Conan’s humor): https://www.youtube.com/c126d7c3-4fa3-4c75-a3dc-d35ea8ad155a
    • Visual standardized Patients: https://youtu.be/mvXIruMt9Ek
      • High fidelity, simulated “real” standardized patient
      • Conversational, can understand and respond to student questions
      • Easy to use, require little or no training
      • Soon students will wear VR goggles
    • Challenges of VR and AR
      • Fatigue, disorientation and vertigo
      • headsets are heavy, and there’s usually a lag when watching
      • Difficult to scale, to do multiplayer VR
      • Movement in virtual space
      • Lack of haptic tools
      • Limited interoperability, but software is allowing for some cross-platform compatibility
    • Future of VR and AR
      • Portability
      • Untethered systems
      • Smaller more comfortable headsets
      • Fidelity
      • Higher resolution displays/increased framerates
      • Haptic feedback
      • Multiplayer
      • Team based simulations
      • Interoperability
      • Build once – deploy everywhere
    • Upcoming in Content and Applications
      • Surgical simulation
      • Patient specific simulations
      • Remote surgery
      • Virtual Patients
      • Practice history taking, physical exam skills, differential diagnoses
      • Automated assessment
      • Team training
      • Emergency medicine, surgical stimulation

Questions asked after seminar:
(Note that some questions and/or answers have been reworded for clarity)

Do you suggest any starter tools to get your feet wet in the technology?
Recommend that you collaborate (with gaming folks, the software folks). It’s simple to get started with 360 degree camera. Higher end creation requires software such as Second Life. And you’ll need a programmer who knows Unity or Unreal Engine (software).

How do everyday operating systems (Oss) figure into all this?
Apple has some VR labs but they don’t compare to what I’ve discussed. Some material can be ported to operating systems such as iOS.

Where is the market going?
Single software use for all purposes. Hardware includes the newly released Oculus Quest.

How are your students at The Ohio State University Evaluated?
Pre and Post Tests are administered. Working on building the mass casualty training system similar to games in that there are various difficulty levels. Must pass one level before moving to the next.

Have you received any pushback from certain populations (e.g. students who get sick)?
There has been very little pushback but we have to have alternatives. Enhances in technology should mitigate the sickness factor.

Has VR or AR been mapped to high fidelity mannequins?
Not yet, but someday it is bound to happen. Companies are trying to merge the two. Seeing different things inside the same mannequin is an example.

What about procedural skills such as suturing or lobotomy?
This is much further down the road. The main problem is that there are no commercially available gloves yet that allow for haptic feedback.

You showed a couple VR anatomy simulations. What were they?
3D Orgnan VR Anatomy and Microsoft HoloLens Anatomy.

You showed the Google Cardboard. What’s the entry point for medical education?
Invest in Oculus Rift or HTC Vibe, plus the computer. But Google Cardboard is a great starting point. We’ve recently attended a conference and had a bunch of headsets at our poster.

Related links supplied by audience:
https://link.springer.com/article/10.1007%2Fs10916-016-0459-8

[The following notes were generated by Mark Slivkoff.]

 IAMSE Webinar Series, Fall 2018

Speakers: Kirsten Brown, PhD, MA and Katherine Chretien, MD – The George Washington University, Washington, DC
Title: Using Social Media as an Educational Tool
Series: Evolution and Revolution in Medical Education: Technology in the 21st Century

• Brown and Chretien set out to cover the following agenda:
  • State of social media use
  • Why use social media
  • Overview of social media in medical education
  • Buff and Blue Instagram case study
  • Reflective writing class blog case study
• Approximately 69% of the American public uses some aspect of social networking sites (Pew Research Center, 2017)
• Although young adults were among the earliest of the social media adopters, older adult usage has also surged (Pew Research Center, 2017)
• Approximately 55% of accredited medical schools also have current student groups on Facebook (Kind, Genrich, Sodhi, & Chretien, 2010)
• Most used platform is YouTube, followed by Facebook, Snapchat, and Instagram
• The latter three platforms are far more popular among the younger crowd (18-24 year olds)
• Why use social media in medical education?
  • Enhance social learning, connection, community
  • Harness power of social media as interactive, communication tools
  • Optimize its benefits for education (e.g., Bialy and Jalai, 2015; Hennessey et al., 2016; Jaffar, 2013; Pickering and Bickerdike, 2016)
• Social media platforms offer an untapped potential for medical education
• The most relevant learning theory (Ileris, 2007) resides with Social Connections:
  • Social Learning Theory (Bandura, 1977, 1978)
  • Communities of practice (Wenger, 2000)
  • Connectivism (Siemens, 2005)
• Recent reviews by Cheston et al. (2013) and Sterling et al. (2017) offer much insight into the use of social media in medical education
• If you’re interested in using social media, where do you start? You need to ask yourself some questions:
  • What are my educational goals?
  • What social media platform do I want to use?
  • How will this add value to the traditional approach?
  • What outcomes will I measure?
• When considering the platform, you must address:
  • Its strengths and features
  • Is the site going to be public or private?
  • Who will be the moderator?
  • Who commonly uses the platform?
  • Do you have experience with the platform?
• Brown, an anatomist, uses Twitter to supplement her coursework. Her site is “Buff and Blue Anatomy”
• She asked herself the same questions mentioned above before beginning her journey:
  • The goal was to the bridge the gap in formal instructional time
  • The platform was of choice was Instagram
  • The value to be added was novel, instant feedback, formative assessment
• The details of her work, and the students’ perception of it, were covered in detail and will be described in her ongoing study. She needs one more batch of data after this Fall of 2018 before completing her manuscript. She did report that:
  • Instagram was a useful supplement to traditional anatomical instruction
  • Content and timing of the posts are important for student engagement
  • Most popular posts were those of encouragement and candid photos of faculty (and not of quizzes)
• Brown described her main lessons learned to date:
  • Respondents overwhelmingly found that the @BuffBluetAnat Instagram was a useful supplement to traditional anatomical instruction.
  • Those that viewed and interacted with the account more frequently rated it significantly more useful than those who viewed and interacted it with rarely
  • Instagram enables interactions and connections between users that can foster learning by tapping into informal educational opportunities and maintaining student motivation
• Chretien also described her use of a WordPress blog a few years ago.
• She also addressed the same questions before she started:
  • Her goals were centered on reflection, student professional development, and discussion
  • The social medial platform of choice was a blog
  • The value added includes asynchronous learning, issues can be discussed as they happen, active dialogue, and anonymity
  • The desired outcomes included satisfaction, engagement, and reflectivity
• Chretien discussed the instructions which she gave to her clinical students which provided a sufficient synopsis of her project:
  • “Requirement: A minimum of two reflective posts per 4- week rotation, with the first post within the first 2 weeks to ensure classmates have a chance to read and respond to your writing. There is no length or subject requirement, but the posts must be reflective, that is, not just telling a story but reflecting on how this experience affects you or changes the way you think about something. Commenting on other classmates’ posts is encouraged. The facilitator will read every post and give feedback in the form of comments. Participation is required but not graded.”
• Her evaluation of the experiences were detailed in a paper (Chretien et al. 2008). Some of the major points:
  • 91 students participated, 177 posts. 1/3 left comments
  • 53% chose anonymous name
  • Most students enjoyed writing posts, reading posts, and found instructor’s comments helpful.
  • Post themes: being humanistic, professional behavior, understanding caregiving relationships, being a student, clinical learning, dealing with death and dying.
  • 8/177 posts not reflective
• And some of her described lessons learned:
  • Participation was variable
  • Students wrote about sensitive issues
  • Hidden curriculum elements revealed, discussed
  • Burden of logistics
  • Not true anonymity in many cases
  • Faculty comments could spur deeper reflection
  • No professionalism or patient privacy issues
• To wrap up, Drs. Brown and Chretien reminded us of the important considerations when it comes to the use of social media in medical education:
  • Public versus private
  • Outcome measures
  • Patient privacy
  • Professionalism
  • Expertise in the platform
  • Learner buy-in
• Encouraged all of us to innovate!

Questions asked after seminar:
(Note that some questions and/or answers have been reworded for clarity)

Can a learning management system (LMS) such as Blackboard provide the same experience as social media platforms?
No. Students already use Blackboard for email, and we wanted to use other software that was different. The built in blog in Blackboard is not unique as well. Furthermore, programs like Instagram are driven by apps which makes use easier.

Are quizzes and examinations still administered via traditional means?
Yes, they are. All quizzes in Instagram are purely supplemental.

Do you recycle the content, or new you create new material each year?
Brand new sites/content are posted yearly. Old posts may show up but they are captured as images and placed appropriately.

Have you run into professionalism issues?
No. There have been some minor issues in Blackboard and Wikis, but overall there have been no major issues.

Do you get any pushback from students regarding the WordPress blog? Do they ask questions such as “how does this help me get into residency”?
There has been very little pushback. If anything, students appreciate not having to express themselves face-to-face. Blogging is a better alternative.

Can you share your list of references?
Yes! We will send out after this presentation.

How much time do you spend on your social media “work”?
About 1 hour per day but of course this varies depending on the week. Overall, it takes a considerable amount of time.

 How do you get academic credit doing this work, outside of publications?
The work is related to the scholarship of teaching and learning, with an emphasis on innovating educational technology.