Criteria 11, 12 and 13 David Pieper ([email protected])

Criteria 11, 12 and 13 David Pieper (dpieper@med.wayne.edu)

Criteria 11, 12 and 13 David Pieper ([email protected]) with help from Brooke Taylor and Rebecca Daniel ACCME Accreditation Requirements and Descriptions Criterion 11: The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program's activities/educational interventions. Criterion 12 The provider gathers data or information and conducts a program based analysis on the degree to which the CME mission of the provider has been met through the conduct of CME activities/educational interventions. Criterion 13 The provider identifies, plans and implements the needed or desired changes in the overall program (eg, planners, teachers, infrastructure, methods, resources, facilities, interventions) that are required to improve on ability to meet the CME mission. Criterion 11 The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program's activities/educational interventions. Criterion 11: Example of Citation The evidence presented in the self-study report and in the activities reviewed demonstrated that the provider evaluated learner satisfaction and whether learning objectives were met. The provider did not present data related to, or an analysis of, changes in physician competence, performance, and/or patient outcomes Majority of WSU Activities Assess Competence Example of Evaluation of Competence Based on what you learned in this session, will you make changes in your practice? Yes or No

If yes, please describe the changes you intend to make What barriers to change do you anticipate? What strategies or mechanisms will you apply to overcome the barriers? Example of Responses WSU/DMC Antithrombosis Conference BASED ON WHAT YOU LEARNED TODAY, WILL YOU MAKE CHANGES IN YOUR PRACTICE? YES: 101 NO (will keep the same): 21 NO (Retired): 2 If Yes, please describe the changes you intend to make: will encourage our cardiologists to prescribe anticoagulation meds in stable non stented CAD patients no ASA without assessment of patient will look at CHADS2 VASC2 score along with CHADS establish TSOAC monitoring procedure duration of therapy of anticoagulants in first event DVT/PE => more than 6 months is beneficial making recommendations for candidates for anticoagulation therapy I intend to implant formed INR goals on referring physicians to eliminate unwarranted goals CHADS2VASC score initiated evaluate bridging protocol for warfarin interruptions for patients with old (> 612 months) PE/DVT and factor check Leiden (didnt realize this doesnt increase risk of recurrent VTE) more confident to defend the role of warfarin over NOAC. I have more understanding of that data recommend wider use of NOACs for VTE Example of Responses WSU/DMC Antithrombosis Conference What barriers to change do you anticipate? insurance prior authorizations and cost of new meds none DMC will implement changes none time away from clinic to discuss insurance coverage, or lack of patient preference to DOT and MD preference of DOT physician resistant/threat to autonomy physician pushback physician resistance to new agents

education; physician buyin time cost of new anticoagulants (No GEQ) time and resources billing for service Example of Responses WSU/DMC Antithrombosis Conference What strategies or mechanisms will you apply to overcome these barriers? follow policies DMC speak to leadership assistance programs (?) describe evidence support longer treatment of AC in first event of DVT/PE change referrals, enforce new rules persistence delegate to resident assignments refer programs to help patients pay for meds patient assistance programs use of EMR, resource allocation/shift, costavoidance justification, costbenefit justification present primary literature discussed/presented today change note template in chart small sections at a time (of learning information) back up change in studies from program Case Conferences / Tumor Boards Example from WSU/DMC Tumor Board Nodule Size Case #1 Nodular density 18mm RML & scarring; subcm LUL subpleural nodule Status or Comments No Patient

Identifiers Recommendations the board's opinion these findings are: Consistant with New PET/CT done 1/30 for In malignancy. At this time it is recommended that patient is review. referred to CT surgery. Case #2 Mediastinal In the board's opinion these findings are: Inconsistant with lymphadenopathy Hx NSCLC & melanoma. PET/ malignancy. with subcarinal 15 At this time it is recommended that the patient continue CT done 2/4 for review. mm & R hilar 19 surveillance per mm oncology. Case #3 In the board's opinion these findings are: highly suspicious 10mm spiculated PET/CT done 2/3 to review. for malignancy. nodule PFT's scheduled for Pt at NH. The board agrees with primary pulmonary team plan to refer patient to thoracic surgery and obtain PFTs. Case #4 RUL mass the board's opinion these findings are: Consistant with

24x21mm & nodule Hx of Renal Cell CA. New CT In malignancy. 17x13mm; LUL from 1/27 to review. The board recommends: the patient should be referred to CT nodule 10x7mm; surgery. RLL infiltrate Example from DMC VTE Prophylaxis Performance Improvement Project Measure Name DMC CY2013 Baseline Target Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14 Jul-14 Aug-14 Sep-14 Oct-14 Nov-14

Dec-14 CY YTD 98.86% 87 88 100.00% 69 69 100.00% 5 5 100.00% 4 4 100.00% 68 68 100.00% 56 56 95.45% 42 44 97.62% 82 84 98.80% 100.00% 63 63 100.00% 48 48 100.00% 1

1 75.00% 3 4 97.92% 47 48 100.00% 33 33 96.88% 31 32 100.00% 60 60 98.96% 100.00% 70 70 100.00% 53 53 100.00% 5 5 100.00% 4 4 93.88% 46 49 100.00% 44 44 100.00% 33 33

100.00% 64 64 99.07% 100.00% 74 74 100.00% 55 55 100.00% 5 5 100.00% 7 7 100.00% 51 51 100.00% 48 48 100.00% 30 30 100.00% 68 68 100.00% 97.35% 954 980 99.74% 754 756 95.71% 67

70 86.21% 50 58 98.07% 712 726 98.33% 588 598 94.71% 448 473 98.81% 910 921 97.84% Tenet Clinical Operations Dashboard Stroke VTE Prophylaxis (STK1) 96.9% Antithrombotics at DC (STK2) 99.2% Thrombolytic Tx (STK4) 88.2% Antithrombotics at by Day 2 (STK5) 97.8% DC on Statin (STK6) 95.8%

Stroke Education (STK8) 90.9% Rehab Assessment (STK10) 98.1% Composite Pending 100% 93.8% 98.95% Anticoag Tx for Afib/Aflutte r (STK3) 89.66% 96.25% 92.94% 96.10% 97.00% 98.80% 100.00% 100.00% 78 77 79 74 97 82 82 91 87 80 85 77 100 83 82 91 100.00% 98.39% 98.51% 100.00% 100.00% 100.00% 100.00% 100.00%

66 61 66 54 76 64 66 76 66 62 67 54 76 64 66 76 88.89% 88.89% 100.00% 66.67% 100.00% 100.00% 100.00% 100.00% 8 8 7 2 6 8 10 2 9 9 7 3 6 8 10 2 83.33% 100.00% 75.00% 62.50% 100.00% 80.00% 100.00% 100.00% 5 1 6 5 4

4 5 2 6 1 8 8 4 5 5 2 96.72% 98.33% 96.88% 100.00% 97.37% 100.00% 100.00% 96.05% 59 59 62 47 74 62 64 73 61 60 64 47 76 62 64 76 96.61% 94.23% 98.04% 100.00% 96.55% 97.96% 98.11% 100.00% 57 49 50 48 56 48 52 47 59 52

51 48 58 49 53 47 86.11% 83.33% 90.48% 95.74% 93.88% 100.00% 95.65% 100.00% 31 30 38 45 46 29 44 49 36 36 42 47 49 29 46 49 98.78% 100.00% 97.50% 94.81% 97.80% 100.00% 100.00% 100.00% 81 76 78 73 89 76 76 87 82 76 80 77 91 76 76

87 94.83% 96.01% 95.54% 96.40% 97.39% 99.20% 99.25% 99.30% Pediatric Outpatient Process Measure Other Evaluation Methods Survey Monkey 3 Month follow-up Audience Response Simulation Mock scenarios Hands-on performance C11 Example from ACCME Focus Group Reports (for sessions held March 2015June 2015) Facilitator: Dr. Dre 7 participants from conference sessions (including 3 attendings, 2 nurse leaders, 1 resident, and 1 genetics counselor) All representatives shared new strategies for managing early diagnosis and consultation regarding surgery Identified systembased obstacles for implementing some strategies (e.g. scheduling delays for genetic counselors, poor communication between clinical and social work staff). Shared that it would be helpful to share this feedback with the group as a whole. (for sessions held July 2015September 2015) Facilitator: Ms. Black 8 participants (2 surgeons, 1 medical records, chief quality officer, 1 attending, 2 nurse practitioners, and 1 medical scribe) Sessions on medical informatics were interesting, but didnt provide actionable next steps. Participants didnt know how to apply the information. Suggestions to include a tool (maybe a checklist) for future sessions (end of year) Facilitator: Dr. Long 9 participants (2 surgical staff, 1 pathology attending, 1 path student, 1 case manager, quality manager, 1 pharmacist, and 2 oncology residents) The coffee is terrible. Can we go back to having Starbucks?! Participants said it was helpful to have the team participate this past quarter had great discussions how to implement changes to our shared processes. Participants appreciated seeing their data Great to see how survivor data is improving. and suggest that we should bring more data into conferences to provide feedback on how were improving. Answer Beaumont Health Criterion 11 Evaluate Changes in Competence Immediate post-conference survey:

Based on information received at this CME activity, what primary action will you take to change your practice? Seek additional information Implement new information/skill(s) into my practice Do nothing, current practice reflects activity recommendations Do nothing, system barriers prevent me from changing my practice Other, specify Give an example of at least 1 action you plan to take to change your practice: Beaumont Health Example of Responses Beaumont Health Criterion 11 Evaluate Changes in Performance 6 Week follow-up survey: Beaumont Health Example of Reponses Beaumont Health Criterion 11 Evaluate Changes in Performance Have you noticed an improvement or change in your patient outcomes since implementing this change in practice? Yes please describe No Have not tracked Beaumont Health Example of Reponses Beamont Health Examples for C11 Competence what will you do? Performance what did you do? Beaumont Health Criterion 11: Activity Analysis of Data

60% 50% 40% 30% 60% 62% 59% 20% 36% 27% 26% 10% 5% 0% Immediate Post-Conference Evaluation 60-Day Post-Conference Follow-Up Evaluation Beaumont Health Criterion 11: Activity Analysis of DataOf the learners who successfully implemented a change in practice, 50% indicated they had noticed an improvement in their patient outcomes. Patients happier Shortened wait times Better outcomes with bladder control Improved headache control in selected patients

C11 Example from SJMAA Dermatology RSS assessment Areas of Yes CME activity met all objectives 100 Material was relevant to the learners practice 100 Learner intends on utilizing content of activity to make a practice change 98 Presentation was free of commercial bias 100 Learner feels this was a valuable activity and would attend again 100 Ways in which this activity changed competence, performance or patient outcomes Main discussion points during the presentation Comments or suggestions for this educational activity

No 2 Increased knowledge base to take care of cong nevi, eruptive KA, Improved ability to utilize broad approach and perspective in management, awareness of morphine metabolism to enable better dosing, improve outcomes in NICU, facilitate early diagnosis of histiocytosis, helped improve my decision making with melanoma management. Topical dermatology, excellent review Made me smart! Evaluation Challenges Low percent of evaluations returned Meaningless responses (All 4 syndrome) Information not summarized or utilized Speakers need individual ratings for P&T Criterion 12 Based on data and information gathered, provide your program based analysis on the degree to which the expected results component of your CME mission has been met through the conduct of your CME activities/educational interventions. WSU Reached Target Audience (Attendance Data) 35000 30000 25000 20000 15000 10000 5000 0

2010 2011 Blue bars = Physicians Open bars= NonPhysicians 2012 2013 2014 WSU: Produced the types of activities stated in the mission: DMC: Expected Results Measure Name DMC CY2013 Baseline Target Jan-14 Feb-14 Mar-14 Apr-14 May-14 Jun-14 Jul-14 Aug-14

Sep-14 Oct-14 Nov-14 Dec-14 CY YTD 98.80% 82 83 100.00% 64 64 100.00% 8 8 80.00% 4 5 100.00% 62 62 97.96% 48 49 100.00% 29 29 100.00% 76 76 99.20% 100.00% 82 82

100.00% 66 66 100.00% 10 10 100.00% 5 5 100.00% 64 64 98.11% 52 53 95.65% 44 46 100.00% 76 76 99.25% 100.00% 91 91 100.00% 76 76 100.00% 2 2 100.00% 2 2 96.05% 73 76 100.00%

47 47 100.00% 49 49 100.00% 87 87 99.30% 98.86% 87 88 100.00% 69 69 100.00% 5 5 100.00% 4 4 100.00% 68 68 100.00% 56 56 95.45% 42 44 97.62% 82 84 98.80% 100.00% 63 63

100.00% 48 48 100.00% 1 1 75.00% 3 4 97.92% 47 48 100.00% 33 33 96.88% 31 32 100.00% 60 60 98.96% 100.00% 70 70 100.00% 53 53 100.00% 5 5 100.00% 4 4 93.88% 46 49 100.00%

44 44 100.00% 33 33 100.00% 64 64 99.07% 100.00% 74 74 100.00% 55 55 100.00% 5 5 100.00% 7 7 100.00% 51 51 100.00% 48 48 100.00% 30 30 100.00% 68 68 100.00% 97.35% 954 980

99.74% 754 756 95.71% 67 70 86.21% 50 58 98.07% 712 726 98.33% 588 598 94.71% 448 473 98.81% 910 921 97.84% 100.00% 33 33 100.00% 33 33 100.00% 32 32 100.00% 100.00% 46 46 100.00% 46

46 89.13% 41 46 96.38% 100.00% 83 83 100.00% 83 83 100.00% 83 83 100.00% 100.00% 138 138 100.00% 138 138 96.32% 131 136 98.79% 100.00% 71 71 100.00% 71 71 98.59% 70 71 99.53%

100.00% 46 46 100.00% 46 46 100.00% 46 46 100.00% 100.00% 36 36 100.00% 36 36 100.00% 36 36 100.00% 100.00% 639 639 100.00% 639 639 98.11% 624 636 99.37% Tenet Clinical Operations Dashboard Antithrombotics at DC (STK2) 99.2%

Anticoag Tx for Afib/Aflutter (STK3) 93.8% Thrombolytic Tx (STK4) 88.2% Antithrombotics at by Day 2 (STK5) 97.8% DC on Statin (STK6) 95.8% Stroke Education (STK8) 90.9% Rehab Assessment (STK10) 98.1% Composite Pending 100% 96.9% 96.25% 77 80 98.39% 61 62 88.89%

8 9 100.00% 1 1 98.33% 59 60 94.23% 49 52 83.33% 30 36 100.00% 76 76 96.01% 100% VTE Prophylaxis (STK1) 98.95% Stroke 89.66% 78 87 100.00% 66 66 88.89% 8 9 83.33% 5 6

96.72% 59 61 96.61% 57 59 86.11% 31 36 98.78% 81 82 94.83% 100.00% 14 14 100.00% 14 14 100.00% 14 14 100.00% 100.00% 24 24 100.00% 24 24 100.00% 24 24 100.00% 92.94% 79 85 98.51%

66 67 100.00% 7 7 75.00% 6 8 96.88% 62 64 98.04% 50 51 90.48% 38 42 97.50% 78 80 95.54% 96.10% 74 77 100.00% 54 54 66.67% 2 3 62.50% 5 8 100.00% 47 47 100.00% 48

48 95.74% 45 47 94.81% 73 77 96.40% 97.00% 97 100 100.00% 76 76 100.00% 6 6 100.00% 4 4 97.37% 74 76 96.55% 56 58 93.88% 46 49 97.80% 89 91 97.39% CAC-Relievers for Inpt Asthma 100%

CAC-Corticosteriods for Inpt Asthma 100% Home Management Plan of Care (HMPC) Document Given to Pt./Caregiver- (CAC 3) Pending Composite N/A 98.95% Pediatric Outpatient Process Measure 100.00% 25 25 100.00% 25 25 96.00% 24 25 98.67% 100.00% 52 52 100.00% 52 52 100.00% 52 52 100.00%

100.00% 71 71 100.00% 71 71 100.00% 71 71 100.00% (WSU) Survey Responses: What do you like about the CME program at WSU/DMC? Accessibility Tracking my CME Readily available Variety of activities Easy, convenient Variety of speakers at Grand Rounds. Easily available, Practical. Close by, incorporated in clinical schedule

Well organized Ability to present and discuss new information It is collegial Very active Available and high quality Cell phone sign in Broad coverage Survey Responses: What would you like to see improved with regard to the CME program at WSU/DMC? Website listing upcoming CME activities, including those coming in months ahead, so that I can plan my Schedule a few months in advance More use of teleconferencing to allow physicians that aren't in proximity to participate More timely documentation of participation Improved access to my updated CME hours Online / SMART phone effort Online versions for credit if unable to attend in person Interdepartmental CME courses Easier access, web based learning without blackboard hassels WSU Survey Responses: What are the areas of your practice that are most in need of improvement? Continuity for current molecular diagnostics Continued medical knowledge improvement Work life balance Safety Community Resources in Wayne County (Free / Low Cost Clinics; etc.) Use of rating scales to measure change Patient scheduling Practice organization Automatic access to current practice guidelines and updates preparing Beaumont Health Criterion 12: Program-based Analysis Types of Activities what formats did you certify? Hours of Instruction did you offer more than just AMA credit?

Target Audience who attended your activities? Internal staff? External? Learners who were your learners? Physicians? Others? Content Areas did you offer content in all areas you hoped? Are you satisfied with the results? How did this year compare to last year? Beaumont Health Criterion 12 Seminars Regularly Scheduled Series Web-based Enduring Web-based Live Learning from Teaching Total Activities 2010 2011 2012 2013 2014 2015 52 93 263 151 122 138 96 104 125 132 147 175 3 20 41 86 87 94 0 4 2 1 4 8 0 0 0 0 1 1

151 221 431 370 361 416 Number of Activities 400 350 300 250 200 150 100 50 0 Courses RSS Internet (enduring) 2011 2012 2013 Internet (Live) Learning from Teaching 2014 2015 Total Beaumont Health Criterion 12: Program-based Analysis Expected Outcomes Learners are expected to gain competence in their approach to clinical problem solving, apply newly acquired strategies in their practice, and change their performance in order to provide evidence-based and patient-centered care. 100%

93% 90% 80% 70% 60% 50% 40% 54% 97% 73% 70% 63% 30% 23% 20% 25% 10% 0% 7% Competence Educational Planning National Average Performance Patient Outcomes Immediate Post-Conference Evaluation (WSU)

C13 The provider identifies, plans and implements the needed or desired changes in the overall program (eg, planners, teachers, infrastructure, methods, resources, facilities, interventions) that are required to improve on ability to meet the CME mission Areas for Improvement Plan(s) for Implementation Plans for measuring impact of improvements on WSUs ability to meet its CME mission Steps Taken and Changes Implemented Need to integrate CME with quality improvement and GME

Joined AAMC AE4Q. Implemented large QI/PICME projects with QI and GME. Developed new template for web based CME using CME Tracker Improved patient care CMS Core measures for VTE prophylaxis showed marked improvement New web template established and a variety of activities now available on web for CME Need to increase number of RSS with QI related topic Plan more RSS based on QI related topics A number of CME activities are now available on web using the new web portal infrastructure (CME Tracker) Increase number of RSS based on QI related topics Need to improve efficiency of attendance data entry for RSS Need to give WSU Faculty ability to receive credit for Learning from Teaching Implemented Cell Phone call in attendance system

Organized Grand Rounds for Internal Medicine and Surgery on QI topics Gradually increasing number of About 30% of RSS now on the RSS on the new call in attendance call in system system Policies and forms developed and Need to revise the existing implemented policy and forms to make the expectations easier to understand System is working but is Working hard to make sure cumbersome and requires a lot of speakers do not use trade diligence names in their presentations Would move WSU SOM more into demonstrable patient care quality improvement We have participated in several quality improvement projects that were accredited for CME and were also awarded MOC Part IV credits Need for online CME activities Need to improve compliance of presentations with regard to trade names Developed policy and forms for Learning from Teaching

Continue to review presentations and work with faculty to get them to understand policy Need to integrate MOC Part Considering ABMS IV projects into QI type CME Portfolio sponsorship programs ACCME Example C13 In its self-study report, County Medical Hospital has described areas where it identified, planned and implemented needed or desired changes to its CME program. These included, incorporating an audience response system, getting a better database to store and analyze participant data and records,and making changes in the evaluation forms to collect more information from learners about future topics. Answer Has the provider described improvements to the overall program that will help their ability to meet their CME mission per the expectation of Criterion 13? ACCME Example C13 The Accredited Provider conducts an overall program review and finds that theyve met their mission. During the conduct of that overall program review, however, they have identified organizational challenges with regard to managing the planning and evaluation of CME activities. They describe implementation of a new CME tracking database as a solution to overcome these obstacles. Has the provider described improvements to the overall program that will help their ability to meet the CME mission to the expectation of Criterion 13? Answer ACCME Example C12 and C13 The accredited providers self-study narrative includes a description that it achieved some of the expected results (i.e., changing performance) in its mission statement, but has fallen short of goals to change patient outcomes. As a result, the provider

described implementing strategic planning with its Executive Committee to identify ways to better align education with quality and safety for better patient outcomes. The provider reported at the interview that early results are showing progress in addressing some patient care issues like falls and poor HCAHPS (patient satisfaction) scores. Please indicate your findings for the providers performance for Criterion 12 and Criterion 13: A. B. C. D. Compliance with both C12 and C13 Compliance with C12, but not C13 Compliance with C13, but not C12 Noncompliance for both C12 and C13 Answer

Recently Viewed Presentations

  • 771 Fall04 Natural Language Processing - Computer Science & E

    771 Fall04 Natural Language Processing - Computer Science & E

    Helvetica Arial Wingdings Times New Roman Century Gothic Courier New Times white212 1_white212 Microsoft Equation 3.0 Lecture 21 Computational Lexical Semantics Overview Model Building in NLTK - Chapter 10continued PowerPoint Presentation 10.4 The Semantics of English Sentences Representing the λ-Calculus...
  • Imparfait/passé composé

    Imparfait/passé composé

    Diaporama utilisation combinée de l'imparfait et du passé composé. Français langue seconde, adolescents, jeunes adultes, adultes.
  • Chapter 3 Data and Signals 3.1 Copyright  The

    Chapter 3 Data and Signals 3.1 Copyright The

    Solution HDTV uses digital signals to broadcast high quality video signals. The HDTV screen is normally a ratio of 16 : 9. There are 1920 by 1080 pixels per screen, and the screen is renewed 30 times per second. Twenty-four...
  • The Age of Exploration - Mr. Baker's History Class

    The Age of Exploration - Mr. Baker's History Class

    Daily Warm Up. Use your text to define or explain the following terms: Line of Demarcation p. 42. Treaty of Tordesillas p. 41. Conquistador (glossary)
  • Project Management Community Meeting

    Project Management Community Meeting

    HHS sponsored program. Upcoming classes. IT PM Training (Continued) April 14-15 - FPM 218 Business, Cost and Financial Management (Level II) April 19-20 - FPM 118 Business, Cost and Financial Management (Level I) ... May 18 - FPM 433: EPLC...
  • Towards A Strategic Framework on Climate Change for the World ...

    Towards A Strategic Framework on Climate Change for the World ...

    CIFs will: Utilize skills and capabilities of MDBs to deliver financing at significant scale to unleash the potential of the public and private sectors to address climate change. Complement other multilateral financial mechanisms, such as GEF and Adaptation Fund.
  • Welcome School of Business Information Technology Services *Chris

    Welcome School of Business Information Technology Services *Chris

    Return to the Gmail settings page and refresh your browser. Select . Forward a copy of incoming mail to. Choose what you want to happen with the Gmail copy of your emails. We recommend . Keep Gmail's copy in the...
  • From Exceptional Utility to Strategic Pricing

    From Exceptional Utility to Strategic Pricing

    Buyer Utility. Does your offering unlock exceptional utility? Is there a compelling reason for the mass of people to buy it? Price: set the right strategic price. Is the offering priced to attract the mass of target buyers so that...