topic: [[Researching]] created: 2025-03-10 *How do experts come to agree? If you are building a rubric, a guideline, or a competency framework, you'll have to get to agreement. Here's how to do it right!* ## What is it? Consensus, or coming to agreement, is a process, and there are a variety of processes that can be used. By following the best practices, currently in development and called the ACCORD guidelines (🤣 I do love me some dad-jokes in the scientific literature 🤓), a writer can add to the literature through use of an explicit evidence-based process. Other processes, such as literature review, focus groups, and others can be used to create a sense of agreement, but these are not as powerful as the consensus methods below. ![[CleanShot 2025-03-10 at 20.35.00.jpg]] This graphic represents the frequency of different techniques for developing competency frameworks. (Batt et al., 2019) ## Specific processes to follow include: - [[Delphi]] - [[Modified Delphi]] - [[Nominal Group Technique]] - [[RAM - RAND ULCA Appropriateness Method]] - Consensus Development Conference Other, non-consensus approaches exist such as: - World Café Method ## Why does it matter? In any scientific project, goals should be [[trust]] and reproducibility, and [[trust comes from transparency]]. >“Consensus methods are defined as a systematic means for measuring and developing consensus.” (Humphrey-Murto et al., 2017, p. 14) The ACCORD guidelines - “ACcurate COnsensus Reporting Document (ACCORD) project, which aims to develop a reporting guideline for methods used to reach consensus.” (Van Zuuren et al., 2022, p. 2) - are still being developed, and are part of the [EQUATOR](https://www.equator-network.org) network. ### What should I consider? To select the correct approach for your situation consider: - Time - Cost - Availability of people, synchronous or asynchronous - Importance of discussion, anonymous or face-to-face - What is already known, and to what extent - Presence of pre-existing groups, meetings, or conferences that you can leverage For me, I'm looking to a balance between the anonymity and asynchronicity of Delphi and the emergent magic of a facilitated conversation in one of the synchronous group approaches. In this case, having the "what are the steps" part be in the form of a Delphi, and the "when should you do it" be in the form of a RAND-UCLA might make a nice hybrid, and would have a clear reason behind choosing those methods to hybridize. 🤔 Darn, turns out [[There are no solutions. There are only trade-offs.]] ## Problems with consensus The constraints of these systems have led to widespread modifications, bathing the entire process in a confusion of labels (ie [[Modified Delphi]] can mean almost anything), issues with defining consensus, starting information, and other issues. >“In the field of medical education, a recent scoping review of consensus methods demonstrated similar findings (HumphreyMurto et al. 2014). This research clearly suggests that it is important to move away from the use of labels and move toward a comprehensive description of the steps taken in a specific study.” (Humphrey-Murto et al., 2017, p. 16) >“The reporting quality varied, with 70.0% (180/257) of articles reporting a literature review, 27.2% (70/257) reporting what background information was provided to participants, 66.1% (170/257) describing the number of participants, 40.1% (103/257) reporting if private decisions were collected, 37.7% (97/257) reporting if formal feedback of group ratings was shared, and 43.2% (111/257) defining consensus a priori.” (Humphrey-Murto et al., 2017, p. 1491) Each project must balance the support and the constraint of these labels, and would do better to explicitly name the specific steps they used and explain the reasons, rather trying to hammer the square of their study through the round hole of one of these frameworks. Alignment between the goals, the process, and the results is a key outcome to seek. >“While consensus is a worthwhile strategy that aligns with previous guidance, the rationale for a given approach over another, the sequence, or application was often unclear, and this poses a challenge when we attempt to examine alignment.” (Batt et al., 2019, p. 15) Finally, a false-god risk exists, that consensus is interpreted as correct, and that despite limited validity evidence, certain competency frameworks as felt to be "true." [[Validity]] is an argument that is made, not a state of being, and some arguments are stronger for some contexts. >“In developing competency frameworks limited in conceptual, theoretical or “use” alignment, we risk the perpetuation of frameworks that adopt a form of unintended or unwarranted legitimacy. This may subsequently result in the creation of what we could term a ‘false-god’ framework, which refers to an object of afforded high value that is illegitimate or inaccurate in its professed authority or capability” (Batt et al., 2019, p. 17) ## So what to do? _Whatever method you choose—Delphi, Nominal Group, or RAND-UCLA—don’t just slap a label on it. Define your process, align it with your goals and your constraints, and be transparent. That’s how arguments for the validity of your consensus report can be strengthened. ## Why even develop a consensus-based rubric? For my purposes, I want to have a [[Mastery Learning]] curriculum. After creating the rubric, we now need to decide on the standards for passing. There exist multiple approaches to standard setting in Mastery Learning: - [[]] tags: #moc/publish | #on/research | #on/consensus | #on/guidelines | #on/validity ##### Sources: Batt, A. M., Tavares, W., & Williams, B. (2019). _The development of competency frameworks in healthcare professions: A scoping review_. Medical Education. [https://doi.org/10.1101/19003475](https://doi.org/10.1101/19003475) Humphrey-Murto, S., Varpio, L., Gonsalves, C., & Wood, T. J. (2017). Using consensus group methods such as Delphi and Nominal Group in medical education research. _Medical Teacher_, _39_(1), 14–19. [https://doi.org/10.1080/0142159X.2017.1245856](https://doi.org/10.1080/0142159X.2017.1245856) Humphrey-Murto, S., Varpio, L., Wood, T. J., Gonsalves, C., Ufholz, L.-A., Mascioli, K., Wang, C., & Foth, T. (2017). The Use of the Delphi and Other Consensus Group Methods in Medical Education Research: A Review. _Academic Medicine_, _92_(10), 1491–1498. [https://doi.org/10.1097/ACM.0000000000001812](https://doi.org/10.1097/ACM.0000000000001812) Kea, B., & Sun, B. C. (2015). Consensus development for healthcare professionals. _Internal and Emergency Medicine_, _10_(3), 373–383. [https://doi.org/10.1007/s11739-014-1156-6](https://doi.org/10.1007/s11739-014-1156-6) Van Zuuren, E. J., Logullo, P., Price, A., Fedorowicz, Z., Hughes, E. L., & Gattrell, W. T. (2022). Existing guidance on reporting of consensus methodology: A systematic review to inform ACCORD guideline development. _BMJ Open_, _12_(9), e065154. [https://doi.org/10.1136/bmjopen-2022-065154](https://doi.org/10.1136/bmjopen-2022-065154)