``` toc style: number min_depth: 2 max_depth: 6 ``` # Creation of the BARS for AIM ## Delphi approach - 4 rounds - 39 experts - anonymously created a BARS ## Delphi article - Insider/Outsider approach - this is what a well described methods is like # A rubric article: Rough draft - It looks like the authors JWR, DS, JP, MKF - Who else is in on this? ## What I can do to move this forward? - [x] [[Problem Gap Hook]] ## F/u Questions - [x] what is my role reading the "rough draft"? # AIM for Mastery Project Proposal Communication education in the health professions has scaling and outcomes problems. Health Professions education communication educational processes result in uneven outcomes with high levels of facilitator input. Mastery Learning (ML) is an educational process intended to reduce the variability in learning outcomes. Peer- and expert-facilitated learning is a learning process intended to make educational interventions more reach more participants with less expert facilitator input. By combining peer- and expert-facilitated learning with mastery learning for acquisition of skill in the advocacy-inquiry molecule (AIM), we will describe a communication learning process that results in better learner outcomes with less expert input. ML describes a learning process designed around outcomes instead of time. The learning is assessment heavy, and the assessments use a standard around which the participant is judged. Previous work by Dr. Clement Buleon with Dr. Jenny Rudolph resulted in a rubric that can be used by new learners to observe attempts at AIM allowing them to give feedback to their peers. The next step in the development of this ML for AIM (AIM for Mastery) is to create a pilot learning process that can be trialed with various groups to determine both the effectiveness and the acceptability of this approach to learners. The design described below is a pilot intended to evaluate the outcome of peer-facilitated feedback and the variability in a small group in terms of expected time involved, as well as to determine if our current tools are usable by the expert in evaluating the outcome and if experts agree on the outcome. ### Proposed educational intervention: --------------------- | Outcome | Evidence | Process | | | ----------------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------- | --- | | Novice debriefers will perform the steps of the AIM (PAAIL) | Expert facilitators will use the AIM rubric to rate attempts | *See below for details*: online synchronous peer-facilitated learning | | | | | | | ## Online facilitated synchronous learning: ### Pre-Work: - Step 1: Learners see an example of an AIM (trigger video) - Step 2: Learners receive and review the AIM rubric and PAAIL guide - Step 3: The expert performs a AIM while talking through their approach (could be another trigger video) - Step 4: The learners watch a trigger video on giving feedback using a rubric - this is key step that needs to be developed - expectations around giving feedback as a novice ### In person: - Step 4: Peer Assessment with feedback for PAAI - Learners work in groups of two in breakout room, they are given a picture of a simulation moment to briefly review and discuss what might be going on - Learner 1 performs an attempt at AIM using the PAAI tool - with preview, advocacy1, advocacy2, Inquiry - Learner 2 observes and gives feedback based on the AIM Rubric - Learner 1 performs same AIM after receiving the feedback from Learner 2 and negotiating how Learner 1 will attempt to improve *here I need help on describing this process* - After 2 attempts Learner 1 and Learner 2 swap roles for 2 attempts - Think-pair-share style check-in - The group should return to the large classroom, what have people noticed, what have people learned? - Go back to the groups and practice until they believe either of the learners has mastered in their minds. - Step 5: Expert validation - video each learner attempt AIM after this process - 2-3 experts will watch video and decide based on rubric if mastery was reached #### I think this is a separate project but would appreciate others input here: - Step 6: incorporate listening - This is the "Find the frame" part that needs to be developed further ### Pilot process to do: - [ ] develop trigger video - AIM - [ ] develop trigger video - AIM with talking - [ ] develop feedback video - [ ] create simulation photos 1, 2, 3 - [ ] Find 20 novice debriefers - [ ] pilot process with novice debriefers - [ ] semi-structured interview to get feedback from novice debriefers on learning process - [ ] Analyze time spent for learners and expert facilitators - [ ] Variability in learner time to reach mastery - [ ] agreement between experts on mastery outcome We hope to forward the state of the research on ML for communication skills in HPEd. ### Proposed Research plan Comparative to the HSE course, student time, expert time, learner outcomes - Park et al Rater Reliability paper (Advances in Health Sciences Education): provides a template for response process and internal structure validity evidence - Lockspeiser et al Validity paper (Academic Medicine): provides a template for all five sources of validity evidence: content, response process, internal structure, relations to other variables, and consequences - I think our paper will have some combination of elements from above. - As discussed our paper will incorporate the following sources of validity evidence: - Content: Delphi and instrument development - Response process: (1) rater accuracy, (2) rater consistency, (3) qualitative comments from prior (and current) development phases, and (4) impact of rater training - Internal structure: instrument statistics (internal-consistency reliability), discrimination, measurement error - Consequences: Improvement in learner/participant skills by serving as assessors – will decide whether to include on first paper, depending on data/results Please let me know the final number of assessors participating. I will use this to design the rater-assignment matrix for our next meeting. # Growing Super Learners: Self- and Peer- Guided Mastery Learning with Expert Calibration _Clément Buleon, Demian Szyld, Robert Simon, Jenny Rudolph, 2019_ ## Overview - This is a review of the entire concept, brief, conceptual, and well framed. - Is this unique ML added to self- and peer-guided? # _WHY HAVE WE NEGLECTED VICARIOUS EXPERIENTIAL LEARNING?_ ## Review 2021_08_18 ## Hoover, J. D., & Giambatista, R. C. (2009). Why Have We Neglected Vicarious Experiential Learning? _Developments in Business and Experiential Learning: Proceedings of the Annual ABSEL Conference_, _36_, 33–37. # Review - This piece integrates vicarious experiential learning concepts and compares to direct experiential learning. VEL has benefits over DEL in attention, motivation, and consequences, reducing the cognitive load on the learner and allowing them to develop a scaffold. ## Bottom Line - VEL combined with DEL may work in tandem to allow more rapid skill acquisition. ## Overview - Experiential learning references almost always discuss Kolb cycle of experiential learning model and almost no work has been done on vicarious experiential learning - Vicarious learning involves observing others for scaffolding and imitation - Learners may benefit from vicarious experiential learning (VEL) when the costs to direct experiential learning (DEL) are high - modeling through VEL helps promote [[self-efficacy]] ## Definition "Vicarious experiential learning pedagogy may be viewed as a methodology of education whereby structure and individual or group experiences are contrived to develop learning and perceptual capacities, to develop and reinforce cognitions, to impact on emotions and attitudes, and, importantly, to function in developing capacities to behave consistently with the insights of these processes and experiences by designing learning systems conducive to observation of behaviors and by conscious processes of providing positive models for imitation." ## Resources -Bandura, A. (1965). Vicarious processes: A case of no-trial learning. In _Advances in experimental social psychology_ (Vol. 2, pp. 1-55). Academic Press. ## People - Albert Bandura starting in 1965 ## Meaning Making ![[Screen Shot 2021-08-18 at 5.49.25 AM.png]] - VEL reduces the cognitive load of the learner but requires enough activation for learning to occur. VEL helps the learner scaffold skills prior to attempts, but requires attention to the necessary tasks they are observing. VEL does not come with the negative stimulation of being incorrect, so learning may not be as sticky. # Vicarious learning during simulations: Is it more effective than hands-on training?: Vicarious learning during the use of simulations ## Review 2021_08_18 ## Stegmann, K., Pilz, F., Siebeck, M., & Fischer, F. (2012). Vicarious learning during simulations: Is it more effective than hands-on training?: Vicarious learning during the use of simulations. _Medical Education_, _46_(10), 1001–1008. [https://doi.org/10.1111/j.1365-2923.2012.04344.x](https://doi.org/10.1111/j.1365-2923.2012.04344.x) # Review - Experiment! 2x2 VEL then DEL, DEL then VEL with controls for with or without feedback scripts and with or without observation scripts ## Meaning Making - VEL outperformed DEL when learners were provided observation scripts [[🐓 Idea Farm/🌿 Ideas/Collaboration Scripts]] but not when learners were provided feedback scripts - No specific order was necessary - VEL Learners required attention direction through observation scripts - the VEL learners knew more, but action was not tested (did they DO better is unknown) # _Eyes On, Hands On: Vicarious Observational Learning as an Enhancement of Direct Experience_ ## Review 2021_08_18 ## Hoover, J. D., Giambatista, R. C., & Belkin, L. Y. (2012). Eyes On, Hands On: Vicarious Observational Learning as an Enhancement of Direct Experience. _Academy of Management Learning & Education_, _11_(4), 591–608. [https://doi.org/10.5465/amle.2010.0102](https://doi.org/10.5465/amle.2010.0102) # Review: - complex experimental research in education showing that the first attempt by participants observing a complex task (negotiation) was similar to the second attempt by the person participating in the direct educational learning experience, and the second attempt by the VOL was similar to the third attempt DEL. ## Bottom Line - Experimental evidence for nearly the benefit through VOL that one achieves through DEL! - [[Screen Shot 2021-08-18 at 7.57.06 AM.png]] - " VOL processes can function quite effectively as a complement to DEL, especially in the creative design and application of VOL and DEL hybrids" ## Meaning Making - The learning activity was the STARSHIP leadership communication game exercise - It is unclear to me the external validity of this activity to the concepts we are focusing on in our learning - the VOL to DEL process is more efficient and resulted in higher learning outcomes - It is unclear to me how the VOL group observed, was there a [[🐓 Idea Farm/🌿 Ideas/Collaboration Scripts]]? # # _Vicarious Reinforcement Learning Signals When Instructing Others_ ## Review 2021_08_18 ## Apps, M. A. J., Lesage, E., & Ramnani, N. (2015). Vicarious Reinforcement Learning Signals When Instructing Others. _Journal of Neuroscience_, _35_(7), 2904–2913. [https://doi.org/10.1523/JNEUROSCI.3669-14.2015](https://doi.org/10.1523/JNEUROSCI.3669-14.2015) # Review - fMRI based study demonstrating the regions of the brain involved in [[Vicarious Learning]], but really a vicarious teaching study ## Meaning Making - theoretical to brain activity science, not empirical learning theory science Reference to [AIMBARS Detailed Protocol 1.3.2022.docx](hook://file/8dfdYHDCb?p=U2V0bmlrL0lSQiByZXNvdXJjZXM=&n=AIMBARS%20Detailed%20Protocol%201%2E3%2E2022%2Edocx) [[I saw I think I wonder]] - Project Proposal [[2022-01-27 AIMBARS IRB Action Items]] We have two strands: [[2022-03 AIM Audio Clip Ratings]] [[Assessment Strand]] and [[Learning Strand]] 2/12: - We wanted to create a simplified positive only anchored scale and piloted with a group of 3 interprofessional health professions educators. We got an ICC of >>> and found that we did not have enough agreement. We redesigned the rubric to have both positive and negative anchors to reflect the top elements identified in the Delphi. 3/11 Did we have the expert inversion effect at play? 3/6 4/1 [[Manuscript Draft for AIRS article Setnik, Park, Rudolph]] Pre-work: Delphi -> Created AIRS Beta tested with novices -> feedback 1) Created MVP with google slides, only top 4 positive anchors for simplicity 2) trialed with expert debriefers 3) Asynchronous rater training 4) assessed IRC -> too low. Attempted to identify barriers to agreement, unable to come to consensus 5) hypothesized that excluding the remainder of the anchors for experts unintentionally made the rating more difficult and vague 6) Rebuild MVP using genial.ly -> more information dense and able to create new workflows that more closely mirrored the rating 1) AIRS is Subjective-Developmental 1) BOT - Behavioral Observational Training - where the learners need to pull the element from the bigger AIM, later in the training 2) SLT will be incorporated in terms of positive feedback and comments about "we hope you feel ..." 3) FOR - Frame of Reference Training 1) The section of genial.ly where the learners see multiple examples of each behavior within the rating scheme (highly effective, somewhat effective, ineffective) - Slides 12, 13, 14 within preview 4) SLT - increase rater confidence within the program (slide 5, others with shine) -> verbiage about how "we hope they have gained confidence, if they have gotten 4/5 correct ..." 7) waited 2 months 8) Re-performed asynchronous rater training 9) re-performed