STAKEHOLDER DEFINED RESEARCH DESIGNING STUDIES THAT REFLECT THE PERSPECTIVES OF PATIENTS, CLINICIANS AND PAYERS EPG Grand Rounds, University of Maryland February 6, 2014 Patricia Deverka, MD, MS Center for Medical Technology Policy The Center for Medical Technology Policy (CMTP) is an independent ,non-profit 501(c) (3) organization that seeks to advance health care innovation and effectiveness by improving the quality, relevance, and efficiency of health care research. CMTP works on methods, infrastructure and policy to support the conduct of comparative effectiveness research that generates information to

assist patients, clinicians, and payers in making informed clinical and health policy decisions. OVERVIEW Rationale for engaging stakeholders Definitions and approach Case examples Generative discussion THE EVIDENCE PARADOX

How to reconcile: ~18,000 RCTs are published each year* A growing number of non-experimental studies Many systematic reviews, health technology assessments, clinical guidelines conclude that the available evidence is limited or studies are poor quality Up to 60% of clinical recommendations made by ACC or AHA based on expert opinion and/or low quality studies Systematic review of off-label uses of 19 FDA-approved oncology drugs (428 pages, several thousand trials) Because of the paucity of high quality evidence, the data available though voluminous may have little meaning or value for informing clinical practice * Chalkidou, Tunis, Whicher, et al. The role for pragmatic, randomized controlled trials (pRCTs) in comparative effectiveness research. Clin icalTrials .Published online before print July 2, 2012, doi: 10.1177/1740774512450097

REASONS EVIDENCE PRODUCED BY CURRENT CLINICAL RESEARCH ENTERPRISE NOT TRANSLATED INTO PRACTICE Differences between settings where research is conducted and where medicine is practiced Patient population Interventions, including usual care Providers, referral patterns access to care Failure to (be able to) report how treatment effects vary in individual patients and subgroups Underrepresentation of children, women, elderly, ethnic & racial

minorities, patients with comorbidities Research priorities, study questions, endpoints, etc. defined by researchers and funders, not end users THE CER HYPOTHESIS Gaps in evidence will be reduced with increased guidance from payers, patients and clinicians in study design A functional definition of CER would be research designed in light of meaningful engagement of these decision makers DEFINITIONS OF CER AND PCOR

Definition of CER Definition of PCOR The generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels. Patient-Centered Outcomes Research (PCOR) helps people and their caregivers communicate and make informed health care decisions, allowing their voices

to be heard in assessing the value of health care options. Source: Institute of Medicine PCOR has the following characteristics: Actively engages patients and key stakeholders throughout the research process. Compares important clinical management options. Evaluates the outcomes that are most important to patients. Addresses implementation of the research finings in clinical care environments. Source: PCORI WHAT IS UNIQUE ABOUT CER?

Many CER studies will require an understanding of the trade-offs between internal validity and increased generalizability, relevance, feasibility and timeliness The right balance is not solely a scientific issue, its also a social judgment about an acceptable level of uncertainty, involving multiple stakeholders Process to achieve this with stakeholder input is evolving THEORY OF STAKEHOLDER ENGAGEMENT Evidence suggests that engaging stakeholders in research increases: Relevance and Content Relevance and Content

Knowledge Exchange Application of Evidence Dissemination Avenues BARRIERS TO INVOLVING STAKEHOLDERS IN CER Confusing terminology, lack of standard definitions Timing; restrictions on availability of stakeholders Training needs for all stakeholders to maximize participation Concerns that process will add time and costs to project plans Lack of shared conceptualization of what it means to successfully or effectively involve stakeholders in research Limited data regarding impact; systematic evaluation rare Sources: Guise, O'Haire, McPheeters, et al. A practice-based tool for engaging stakeholders in future research: a synthesis

of current practices. J Clin Epidemiol. 2013 Jun;66(6):666-74. doi: 10.1016/j.jclinepi.2012.12.010. Epub 2013 Mar 13. and CMTP experience ADDRESSING THE BARRIERS Literature review Biomedical Social science Practical experience based on projects involving stakeholders Drafted definitions and conceptual model Review and revision by an expert panel Patient and Consumer Advisory Council NICE Patient and Public Involvement Program and Citizen Council in the UK

Applied it to a complex multi-stakeholder project Diverse Roots of Public Participation Activities CBPR* Reducing health disparities Social change & action Public policy Environmental planning

Nuclear power Biotechnology Health care Health technology assessment FDA Health research (UK) *Community-based participatory research TYPOLOGY OF STAKEHOLDER ENGAGEMENT STAKEHOLDER ENGAGEMENT LIGHT

STAKEHOLDER ENGAGEMENT Source: Nass, Levine, and Yancy. Methods for Involving Patients in Topic Generation for Patient-Centered Comparative Effectiveness Research An International Perspective STAKEHOLDERS Individuals, organizations, or communities that have a direct interest in the process and outcomes of a project, organization, or policy. Deverka, Lavallee, Desai, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Compar Effect Res 2012; 2:181-94

. TARGET STAKEHOLDER GROUPS FOR CER STAKEHOLDER ENGAGEMENT A process of actively soliciting the knowledge, experience, judgment and values of individuals selected to represent a broad range of direct interests in a particular issue, for the dual purposes of: 1) Creating a shared understanding; 2) Making relevant, transparent, and effective decisions. CONCEPTUAL MODEL FOR STAKEHOLDER ENGAGEMENT IN CER

Types of evidence Analytic-Deliberative Model Inputs Values Research Professional Experience Patient and consumer knowledge and experience Methods of combining evidence Methods Quantitative Questionnaires

Delphi method Multi-Criteria Mapping Value of Information modeling Qualitative Facilitated workshops/meetings Stakeholder decision analysis Decisions Outputs Process CER

Topic generation Research priorities Study designs Evidentiary thresholds for clinical and health policy decision making Implementation strategies Outcomes Meta-criteria, Trust, Respect, Accountability, Legitimacy, Fairness, Competence Change in Knowledge/attitudes Change in CER project decisions (e.g. choice of interventions, study design, funding priorities) More useful evidence for clinical and health policy decision making More efficient use of healthcare resources

Improved health outcomes. DECIDING WHICH STAKEHOLDER GROUPS TO INVOLVE IN A PROJECT What topic(s) does the research address? What health care decision is the research meant to inform? Who are the decision makers responsible for these decisions? Who are the individuals and groups that are affected by these decisions? Concannon TW, Meissner P, Grunbaum JA, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. JGIM 2012;27(8):985-91.

WHEN TO INVOLVE PATIENTS AND OTHER STAKEHOLDERS IN RESEARCH? Topic identification and refinement Priority-setting Writing proposals (including deciding research methods) Reviewing research conduct

Interpretation of findings Dissemination of information Implementation Evaluation Curtis, Slaughter-Mason, Thielke, et al. PCORI Expert Interviews Project: Final Report. Portland, OR: Center for Evidence-based Policy. Oregon Health & Sciences University. STAKEHOLDER INVOLVEMENT IN GOVERNANCE

Decisionmaking Data access Data security Data privacy Conflicts of interest Consent Communication Coordination with other committees

Infrastructure* Assess needs Recommend data elements Identify applications Monitor Research Education Adapted from: Rein A, Holve E, Hamilton Lopez M, and Winkler J. A framework for patient and consumer engagement in evidence generation, EDM Forum,Academy-Health, September 2012.

Generate & prioritize research questions Design studies Data collection, processing &analysis Results interpretation Translation Dissemination Evaluation Continuous learning healthcare system *Refers to the creation/modification of the Patient

Engagement Platform needed to support both clinical management and the conduct of PCOR PATient-centered Involvement in Evaluating the effectiveNess of TreatmentS PATIENTS Aims of PATIENTS 1) Foster sustainable partnerships with local, regional, and national communities of diverse patients and healthcare systems 2) Conduct and expand PCOR in partnership with patients and healthcare delivery systems 3) Advance dissemination and implementation strategies for PCOR findings

23 Vision The PATIENTS Program vision is that its projects will: Further the process of UM institutional transformation for MPowering the State in the area of health Eliminate health disparities within Baltimore, throughout Maryland, and across the nation Align with the spirit of the NIH roadmap for transformative and interdisciplinary research 24 University of Maryland has broad participation UMB professional schools

Pharmacy Medicine Nursing Social Work Dentistry Law UM College Park 25 PATIENTS Partners 26 Innovation in the PATIENTS Program

Conducting PCOR with continuous patient and stakeholder engagement Translating research into practice Continuous development through formative and impact evaluation Bidirectional learning Sustainability 27 The Ten-Step Process for Conducting CER Public Announcements 1. Topic Solicitation

2. Prioritization 3. Framing the Question Delphi Process Patient Forums Based on: Mullins CD, Abdulhalim AM, Lavallee DC. Continuous Patient Engagement in Comparative Effectiveness Research. JAMA 2012; 307(15): 1587-8. 28 The Ten-Step Process for Conducting CER

4. Selection of Comparators and Outcomes In-person Meetings Telephone Calls 5. Creation of Conceptual Framework 6. Analysis Plan Focus Group Inter-views

Electronic Social Media 7. Data Collection Teleconference s Based on: Mullins CD, Abdulhalim AM, Lavallee DC. Continuous Patient Engagement in Comparative Effectiveness Research. JAMA 2012; 307(15): 1587-8. 29 The Ten-Step Process for Conducting CER

8. Reviewing & Interpreting Results Teach-Back Method 9. Translation 10. Dissemination Media Critique Documents (e.g. Patient Guides) Based on: Mullins CD, Abdulhalim AM, Lavallee DC. Continuous Patient Engagement in Comparative Effectiveness Research. JAMA 2012; 307(15): 1587-8. 30

Education and Training: Engaging Partners Bidirectional Learning Community Partners Stakeholder Engagement Cultural Competence Using Online Platforms UM Faculty and Staff Research Methods Manuscript Writing Grant Writing Examples: PatientsLikeMe: Social Media

BSBHS/Riverside Heath System: Implementation 31 EXAMPLES: STUDY DESIGNS REFLECTING VIEWS OF DECISION MAKERS TWO EXAMPLES Genomic Testing in Cancer (CANCERGEN) Evidence Guidance Documents Molecular Dx in Cancer THE PROMISE: TRANSFORM CANCER CARE

Molecular diagnostic (MDx) tests have the potential to transform oncology practice by helping physicians classify and manage various cancers Diagnose and stage cancers Help guide therapy selection and dosing Assess treatment response Aid in detection of residual or recurrent disease 34 CANCERGEN STRUCTURE Thariani R, Wong W, Carlson JJ, et al.

Prioritization in Comparative Effectiveness Research: The CANCERGEN Experience in Cancer Genomics. Medical Care 2012; 50(5):388-393 Figure 1: CANCERGEN project milestones including landscape analysis, stakeholder assessment and final selection. RANKING OF TESTS No Order 1. ERCC1 Expression Testing for Platinum-Based Adjuvant Therapy in NSCLC Final Order

2. EGFR Mutation Testing for Erlotinib Maintenance Therapy in Advanced NSCLC 1. ERCC1 3. EGFR gene copy number (FISH) testing and first-line cetuximab therapy in stage IV or recurrent NSCLC 2. BCTM 4. BRAF mutation testing in Colorectal Cancer 3. EGFR mutation 5. Genetic Expression Profile (GEP) in Multiple Myeloma (MM) to Identify Patients with Poor Prognosis

6. Breast CA Tumor Markers for Detection of Recurrence After Primary Breast Cancer Therapy Center for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN) LESSONS LEARNED Full participation of all stakeholders on highly technical topics is possible with adequate preparation Multi-modal approaches are necessary Engagement method should be matched to particular study question Stakeholders were open to novel methods (e.g., VOI), but more work needs to be done to ensure full benefits of the approach are fully realized Possible to engage most stakeholders for a multi-year


DECISION MAKERS KEY QUESTIONS FOR CANCER MDX TESTS 1. Does the MDx test provide correct information? (analytic validity) 2. How well does the test result correlate with clinical outcome? (clinical validity) 3. Does use of the MDx test lead to improved patient outcomes as compared with the alternative? (clinical utility) 4. Does use of the MDx test lead to greater value as compared with the alternative? (cost-effectiveness) PROBLEM Analytic validity and clinical validity now available for an

increasing number of MDx tests Clinical utility largely unknown for most MDx tests Uncertain clinical utility has consequences for patients and health care system Decreases quality through inconsistent or unnecessary use of tests Wastes health care resources WHATS NEEDED Structured data about MDx test use Stakeholder-driven process

Clear evidentiary standards for clinical utility Willingness to consider range of methods, outcome measures that are relevant to real-world clinical decisions THE RESPONSE: EFFECTIVENESS GUIDANCE DOCUMENTS Provide specific recommendations on the design of studies intended to inform decisions by patients, clinicians and payers Developed for specific clinical conditions and categories of technologies Based on a structured, transparent, multistakeholder process led by CMTP Aim to balance internal validity, relevance, timeliness and feasibility

provide decision-makers with a reasonable level of confidence that the intervention improves net health outcomes Analogous and complementary to FDA guidance Targeted to researchers working in industry or academic settings 44 TECHNICAL WORKING GROUP

Academic researchers 2 Industry 3 Payers 2 Research funders 1 Policy makers 1 Patient advocate 1 RECOMMENDATIONS: 10 COVERING CLINICAL VALIDITY AND CLINICAL UTILITY MDx test development follows phases similar to the phases of drug

development Recommendations have been organized around these phases Biomarker discovery (Phase 0) and the assessment of population impacts (Phase 5) go beyond the scope of this EGD 46 HOW EGDS MIGHT BE USED By test developers and researchers in designing studies

By payers in evaluating evidence submitted for coverage and reimbursement By guidelines developers in judging quality of evidence and strength of recommendations By research funding organizations in evaluating grant proposals By patient advocacy and other groups generating guidance for patients SUMMARY Stakeholder engagement is essential for fulfilling the objectives of CER Research funding requests and support need to account for resources

required to meaningfully implement engagement activities The terminology and methods are being developed and tested worked for CER Methods need to be tailored to the particular phase of research, but stakeholders should be involved throughout the process Careful attention to communication at multiple levels is critical to ensuring true collaboration and a respectful, accountable process There is a growing body of examples of rigorous methods of SE being applied to CER Evaluation is critical for measuring impact and process improvement

Extra Slides CMTP: PRINCIPLES FOR INVOLVING PATIENTS IN COMPARATIVE EFFECTIVENESS RESEARCH Each CER-related project includes patient representatives.** Examples of such projects include identifying research topics, setting priorities, developing questions to be studied, designing study protocols and establishing methodological standards. Project leaders recruit a diverse group of patients for whom the project topic is relevant. Project leaders, patients and other stakeholders make their mutual expectations for patient involvement known to each other. The project team, other stakeholders and patients disclose potential conflicts of interest.

Project budget includes appropriate remuneration for patients and support for their participation, including training, stipends, travel and lodging, and other resources critical for their full involvement in the project. **Includes patients, family caregivers and consumers representatives of all types CMTP: PRINCIPLES FOR INVOLVING PATIENTS IN COMPARATIVE EFFECTIVENESS RESEARCH..CONTD The project team and other stakeholders recognize and respect the different skills, knowledge and experience of patients. Patients recognize and respect those of the other participants. The project team communicates regularly with patients throughout the life of the project to ensure the quality and sustainability of the involvement process. The project team obtains periodic assessments from patients and other stakeholders to evaluate the effectiveness of the engagement process and

inform the design of future research involving patients. Project reports and publications describe in the methods sections how patients were involved in research. The project team and other stakeholders work with patients to present study findings in a way that can be easily understood by patients.

Recently Viewed Presentations

  • Rebuilding Lynmouth - Think Geography

    Rebuilding Lynmouth - Think Geography

    They can use the following strategies: 'Hard Engineering strategies' - these require major alterations and changes to the river and try to stop the river from flooding 'Soft Engineering strategies' - this is where limited alterations take place and flooding...


    Pour les S.A.R.L.: l'obligation de passer par un notaire pour la création a été supprimée et le capital minimum exigé est passé de 1 000 000 F à 100 000 F.
  • Introduction to Discounting

    Introduction to Discounting

    Strengthening and Lifting Covers Formalization where C is a set of management units (nodes) that form a connected sub-graph of the underlying adjacency graph, and for which holds, but for any The minimal cover constraint has the general form of:...
  • Teaching The Holocaust: Exploring Pedagogical Dilemmas Avi Patt,

    Teaching The Holocaust: Exploring Pedagogical Dilemmas Avi Patt,

    In what ways does the film reinforce or challenge traditional historical narratives? Does the film present the historical experiences of marginalized groups that might otherwise receive little attention? What political or social values are reflected by the film and which...
  • Reducing Equations to a Linear Form - Exeter

    Reducing Equations to a Linear Form - Exeter

    Reducing Equations to a Linear Form Andrew Robertson Today Reducing data relationships to straight lines WHY ? One good reason is to be able to make forecasts from data It is much easier to work and predict future results when...
  • Food and the Planet Tllberg Forum 2007 Hal

    Food and the Planet Tllberg Forum 2007 Hal

    How do we make public food service a driver of sustainability and health? Is there a place for family farms? How do we turn things around? together The U theory posits a new understanding of the world that no longer...
  • World War II - Miami Beach Senior High School

    World War II - Miami Beach Senior High School

    Hitler does not want a two-front war like World War I. Sends ministers to Russia to negotiate deal with Stalin. Stalin sees chance to turn capitalists against each other. August 23, 1939: Nazi-Soviet Non-Aggression Pact. After Hitler invades Poland, Britain...
  • What is a Postdisciplinary Engineer? Jim Brenner Florida

    What is a Postdisciplinary Engineer? Jim Brenner Florida

    Has internal support of Bill Allen (Cybersecurity), Phil Bernhard (new dept. head for Computer Sci. & Electrical & Computer Engineering), Dan Kirk (Assoc. Dean), and David Beavers (Makerspace; likely to supervise GSA's who would teach it) ChE. advisory board &...