#note/idea | #on/research | #on/validity | #on/writing ### The Hypothesis of Determining Validation Arguments "Does the evidence support the intended use?" ### Build the Argument 1. **Interpretation (Assumption) Use Argument** (Upfront) 2. **Validity Argument (Final Argument)** → Final reasoning ### Readiness for: | Readiness for... | Evidence of Readiness | Activity that Demonstrates | |-----------------------------------------------------|--------------------------------------------------|-------------------------------------------------| | Using AIR to provide formative feedback to a debriefing peer | Formative | AIR training module completion | | Using DASH to determine readiness for solo debriefing of an onboarding peer | Summative | DASH rater training workshop with clean real cases, WBA comparison in the user context | ### Prioritize Most Worrisome Inferences/Assumptions: - Using empirical online into social dynamics - Audio-only → using empirical live, in-person = **Extrapolation** **For Article Next Steps:** - Develop a plan that will refute or confirm. --- # Best Practices in Assessment: Part 1 - Define Goal ### Checklists vs. Global Ratings: - **Balance** between checklists and global rating rubrics, they have their tradeoffs ### Steps: 1. Make validity argument influence **use** → upfront 2. Define roles/key deficiencies → prioritize gaps 3. Refine to collect what's easy — collect what's important - May need to experiment here 4. PDSA to make sure it is practical or won't be questioned 5. Collect data 6. Determine validity argument (end) ![[CleanShot 2024-12-06 at 17.52.29.jpg]] ### Metaphors: - **Bookends:** Interpretation — Use and Validity Argument - The center: You may need to add or remove a book here or there to end with a sustainable, meaningful, useful program. ### Manufacturing Metaphor: **C-I-P-P** 1. **Raw Materials** → Unrefined inputs 2. **Process** → Refinement/testing 3. **Product** > If our product is a ready program for that domain, there are just a series of things that **must** happen to get to that. What we care about: how we emphasize/triangulate/take tradeoffs. Series of choices = cost. ## Source: Cook, D. A., & Hatala, R. (2016). Validation of educational assessments: A primer for simulation and beyond. _Advances in Simulation_, _1_(1), 31. [https://doi.org/10.1186/s41077-016-0033-y](https://doi.org/10.1186/s41077-016-0033-y)