Enhancing Assessment and Feedback Principles and Practice David Nicol Professor of Higher Education Centre for Academic Practice and Learning Enhancement (CAPLE] Director of REAP and PEER projects (www.reap.ac.uk) University of Strathclyde, Scotland ATN Australia: 20th October 2011 National Student Survey (UK) Assessment & feedback (2008) N o
Survey Statement Englan d Scotlan Norther d n Ireland 5. The criteria used in marking have been clear in advance
69 69 69 6. Assessment arrangements and marking have been fair 74 74
73 7. Feedback on my work has been prompt 56 51 53 8.
I received detailed comments on my work 61 52 52 9. Feedback on my work has helped clarify things I did not understand
56 51 50 82 86 83 22 Overall, I am satisfied with the
Plan Background Re-engineering Assessment Practices (REAP) project [1m] Concepts and example of practice An institutional viewpoint PEER project Background Departments and faculties: supporting educational improvement projects. REAP project: big implementation across 3 universities Policy/strategy: led development of policy in assessment and feedback [based on REAP] Students: Feedback as dialogue campaign
PEER project JISC funded [50k] HE Sector: Project facilitator for QAA Scotland on A&F Research/publications: assessment, learning, change See www.reap.ac.uk REAP : Re-engineering Assessment Practices Scottish Funding Council for Universities (1m project) 3 Universities - Strathclyde, Glasgow & Glasgow Caledonian Large 1st year classes (160-600 students) A range of disciplines (19 modules ~6000 students) Many technologies: online tests, simulations, discussion boards, e-portfolios, e-voting, peer/feedback software, VLE, online-offline Learning quality and teaching efficiencies
Assessment for learner self-regulation www.reap.ac.uk Background (1) Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students learning, Learning and Teaching in Higher Education, 1, 331. See: Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtl Gibbs and Simpson (2004) Assessment tasks [Conditions 1-4]
1. Capture sufficient study time (in and out of class) 2. Are spread out evenly across timeline of study 3. Lead to productive activity (deep vs surface) 4. Communicate clear and high expectations i.e concern here is with time on task how much work students do - their active engagement in study Background (2) Literature Review Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice.
Studies in Higher Education, 34 (1), 199-218 Background Student Enhanced Learning through Effective Feedback [SENLEF] project funded by HE Academy REAP project: www.reap.ac.uk Rethinking assessment and feedback 1. Consider self and peers as much as the teacher as sources of assessment and feedback Tap into different qualities than teacher can provide Saves time Provides considerable learning benefits (lifelong learning) 2. Focus on every step of the cycle:
Understanding the task criteria Applying what was learned in action 3. Not just written feedback: Also verbal, computer, vicarious, formal and informal Seven principles of good feedback Good feedback: 1. Clarifies what good performance is (goals, criteria, standards). 2. Facilitates the development of reflection and selfassessment in learning 3. Delivers high quality information to students: that enables them to self-correct 4. Encourages student-teacher and peer dialogue around learning
5. Encourages positive motivational beliefs & self esteem 6. Provides opportunities to act on feedback 7. Provides information to teachers that can be used to help shape their teaching (making learning visible) Source: Nicol and Macfarlane-Dick (2006) Principle 1: Clarify what good performance is (the context of dialogue) EMPOWERMENT/ SELF-REGULATION Students create criteria Students add own criteria Students identify criteria
from samples of work Exemplars of different performance levels provided Students rephrase criteria in own words Provide document with criteria ENGAGEMENT Two meta principles 1. Meta-PRINCIPLE 1: time and effort on task (structured engagement) i.e. steers on how much work to do and when Gibbs and Simpson 4 conditions
2. Meta-PRINCIPLE 2: developing learner selfregulation (empowerment/self-regulation) i.e steers to encourage ownership of learning the seven principles discussed above. Key task for teacher is to balance 1 and 2 Example: Psychology Psychology 560 first year students 6 topic areas (e.g. personality, classical conditioning), 48 lectures, 4 tutorials, 12 practicals Assessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in
experiment (5%), essay exam (66%) Problems identified No practice in writing skills but required in the exam More detail provided in lectures than mentioned in exams (not enough independent reading) No feedback except on Multiple Choice Questions (percent correct) Didnt want to increase staff workload Psychology Redesign Discussion board in Learning Management System
Students in 85 discussion groups of 7-8, same groups throughout year Also open discussion board for class Friday lectures cancelled discover material themselves Series of online tasks Structure of group tasks 6 cycles of 3 weeks (one cycle x major course topic) First week: light written task (e.g. define terms) = 7 short answers (all answer) Second week = guided reading Week three: deep written task: students collaborate in writing a 700-800 word essay on
the same topic. Within each week: The Monday lecture introducing material Immediately after lecture, task posted online for delivery the following Monday Model answers (selected from students) posted for previous weeks task The teaching role Participation in the discussions was compulsory but not marked (in subsequent years there was 2% mark for participation) Course leader provided general feedback to the whole class often motivational He encouraged students to give each other
feedback The group discussions were not moderated but monitored for participation An example of deep task The Task 800 word essay: Assess the strengths and weaknesses of Freuds and Eysencks theories of personality. Are the theories incompatible? readings suggested questions provided which all students should try Relation to the Gibbs & Simpsons four assessment conditions 1. Tasks require significant study out of class
(condition 1) 2. Tasks are distributed across topics and weeks (condition 2) 3. They move students progressively to deeper levels of understanding (condition 3) 4. There are explicit goals and progressive increase in challenge (condition 4) Relation to 7 feedback principles 1. Standard format and model answers provide 2. 3. 4. 5.
6. 7. progressive clarification of expectations (principle 1) Students encouraged to self-assess against model answer (principle 2) Course leader provides motivational and meta-level feedback and selects model answers (principle 3) Online peer discussion aimed at reaching consensus is core feature of design about response (principle 4) Focus on learning not just marks, sense of control/challenge enhanced motivation (principle5) Repeated cycle of topics and tasks provide opportunities to act on feedback (principle 6) VLE captures all interactions allowing course leader to
monitor progress and adapt teaching (principle 7) Benefits Students worked exceptionally hard Written responses of exceedingly high standard Students took responsibility for learning High levels of motivation: atmosphere in class improved Online interactions showed powerful scaffolding and community building Feedback with 560 students through peer and self-feedback (model answers) Easy for tutors to monitor participation Improved mean exam performance (up from 51-59%, p<0.01) weaker students benefit most
Online postings/interaction 24,362 messages posted by groups for essay tasks Average number of postings per student 44.3 1067 postings to general open discussion forum Students set up online study groups for other subjects Structured tasks online triggered important socio-cognitive processes Has it worked? I read more about Psychology and read it earlier in each semester than I would have done without the online projects
Why use principles 1. 2. 3. 4. 5. 6. 7. Provides a framework for operationalizing the big idea the development of learner self-regulation Helps translate the research into accessible guidelines for teaching practice Ensures change is educationally driven Enables connections to be made across innovations
in different disciplines Provides a common language to talk about innovation and for dissemination Can add to the evaluation process measures Helps identify where technology can leverage benefits Guidelines for Implementation A single principle or many? 2. Tight-loose maintain fidelity to the principles (tight) but encourage disciplines to develop their own techniques of implementation (loose) 3. Balance teacher feedback with peer and selfgenerated feedback 4. The more actively engaged students are, the better the course design
1. Developments since REAP 1. Principles of Assessment and Feedback approved by University Senate and embedded in policy (2008) 2. Use of principles to inform curriculum renewal and Quality Assurance processes 3. Feedback as Dialogue campaign to gain commitment of students 4. PEER Project (Peer Evaluation in Education Review) Peer Review in Education
Evaluation [PEER] The aims of the PEER project are to: Review evidence base for peer review Develop educational designs for peer review (and self-review) Identify software support for peer review Pilot implementations of peer review with large student numbers Produce guidelines for higher education why do it, how to do it, pitfalls and solutions and software possibilities. see http://www.reap.ac.uk/peer.aspx Peer feedback: augmenting teacher feedback
Increasing quantity and variety of feedback No extra workload on teacher with software (e.g. PeerMark, Aropa) More timely e.g. collaborative projects Simulates professional life reconciling different feedback perspectives The argument Not enough attention has been focused on the potential of peer feedback not just as a way of increasing the quantity and quality of the feedback students receive, but as a way of giving students practice in constructing feedback.
The focus Scenarios where students make evaluative judgements about the work of peers and provide a feedback commentary, usually written Not talking about scenarios involving ...informal feedback in collaborative tasks .....students evaluating each others contribution to group working .....students grading/marking each others work, although some rating might be part of peer design
Benefits of feedback construction (1) Constructivist rather than telling paradigm 1. High-level cognitive activity: students cannot easily be 2. 3. 4. 5. passive Students actively exercise assessment criteria from many perspectives Writing commentaries develops deep disciplinary expertise
See many approaches and learn that quality can be produced in different ways Shifts responsibility to student puts them in role of assessor exercising critical judgement Benefits of feedback construction (2) 6. Learn to assess own work as exactly the same skills are involved Develops capacity to make evaluative judgements - a fundamental requirement for life beyond university. Also, this capacity underpins all graduate attribute development (Nicol, 2010) Nicol, D (2011) Developing students ability to construct feedback,
Published by QAA for Higher Education, UK Example 1:Peer feedback Laboratory work [Gibbs 2011 www.testa.ac.uk/resources] Reason poor quality of lab reports in science Strategy students organised in groups and produce poster to represent their lab report Peer process hang poster in class and all students individually walk round analyse posters and write feedback on them (e.g post-its): questions, suggestions, inaccuracies etc Result significant improvements in lab work and reporting, positive competition in class, students did not want to look bad.
Example 2: Engineering Design Peer Project case study DM 100 Design 1: first-year class Dr Avril Thomson, Course Leader, Design Manufacturing and Engineering Management (DMEM), University of Strathclyde [email protected] Caroline Breslin, Learning Technology Adviser, University of Srathclyde [email protected] Example 2: Design 1 82 first-year students Design a product theme eating and resting in the city Research in groups (in city, in library etc.)
Individually produce a Product Design Specification (PDS) detailed requirements for and constraints on design (rationale, performance,standards, manufacturing etc) Given a PDS exemplar from another domain to show whats required (stainless steel hot water cylinder) Online learning environment: Moodle and PeerMark part of Turnitin suite Product Design Specification Typical Headings
Rationale Performance Environment Packaging Maintenance Weight and size Disposal Shipping Processes Time Scale Patents Product costs Aesthetics Ergonomics
Materials Testing Quality/reliability Competition Marketing Life in service DM 100: Design 1 Peer review task Individually, each student peer-reviewed and provided feedback on the draft PDS of two other students Criteria: completeness, convincingness of rationale, specificity of values (performance) and one main suggestion for improvements with reasons Students used experience, giving and receiving feedback, to
update PDS which comprises part of a Folio Students self-review Folio and meet/discuss with tutor Peer review not assessed directly but 10% marks for professionalism which included participation in peer review. Peer review rubric: DM 100 Do you feel the PDS is complete in the range of headings covered? If no, can you suggest any headings that would contribute towards the completeness of the PDS and explain why they are important? Is the PDS specific enough? Does it specify appropriate target values or ranges of values? Please suggest aspects that would benefit from further detail and explain.
To what extent do you think the rationale is convincing for the PDS? Can you make any suggestions as to how it might be more convincing? Please explain. Can you identify one main improvement that could be made to the PDS? Provide reason(s) for your answer. Evaluation 1. Online survey completed by 64 students 2. Course work marks compared to previous years 3. Focus group interviews 4. Peer review comments recorded online
Results 1 Which aspects of the peer review did you learn from? Giving feedback Receiving feedback Giving and receiving feedback Neither giving or receiving 10.9% 26.6%
54.7% 7.8% Results 2 Did you modify your initial submission as a result of the peer review activity? Yes, as a result of the peer review given 23.4% Yes, as a result of the peer review received 25.0%
Yes as a result of the peer review given AND received 28.1% No 21.9% N/A 1.6% Results: student comments
If yes, please give specific examples of modifications (n=41) I added a couple of paragraphs and improved existing paragraphs, this added two full A4 pages to my work I included specific materials as changed the formatting of the document so it looked more professional I provided more specific numeric values and expanded my rationale after seeing someone elses PDS and after receiving feedback Results: RECEIVING feedback Please give examples of what you learned from RECEIVING peer reviews from other students (n=54)
Parts that I had previously missed were brought to eye such as market competition (noticing) Receiving peer reviews gave me insight into what others thought of my work and gave me a direction to improve (reader response) Where the PDS was confusing to understand (reader response) I found out how good mine was (motivational) The person who peer reviewed my PDS gave me positive feedback which helped me a lot (motivational) Not much, they werent very good Results: PROVIDING feedback Please give examples of what you learned from PROVIDING peer reviews of others work (n=47) When giving advice to people on theirs, it gave me greater perception when reviewing my own work by listening to my
own advice for example [transfer] I had a chance to see other peoples work and aspects of their work that I felt were lacking in my work, this helped me to improve my work [transfer] I was given a greater understanding of the level of the work the course may be demanding [standards] Allowed me to see from an assessors perspective Thinking from a critical point of view [critical judgement] How to look at work critically that isnt your own Results: How you carried out peer review Could you make any comments about how you carried out the peer review? How did you evaluate the quality of the work to provide a response to the peer review
questions? (n=37) I compared it to mine and the ideal PDS and said how I would improve it Partly by comparing my work to theirs I tried to think about what I wrote and whether this PDS was better or worse Looked through the headings first and noted any that should have been included then read more in detail about each individually and compared it with my own PDS Focus groups How did you go about reviewing? I read it through and compared it with what I had done to see if they had put something I had not done and then I added it in if they hadnt. The four questions were useful as
they provided a framework for the review. If we hadnt had the questions it would have been difficult. I did the reviews separately and then answered one then the other. The first was a better standard than the other so I used the ideas from the better one to comment on the weaker one. I also read the guidelines in class when I did the peer review. There were ideas from the good one that I hadnt even thought of in mine Focus groups What do you think is best for learning giving or receiving feedback? For me it would probably be to give feedback because I think seeing what other people have done is more helpful than getting other peoples comments on what you have
already done. By looking at other peoples work you can see for yourself what you have forgotten or not even thought about. When people give feedback on yours they generally just talk about what is there. They dont say, well I did this on mine and you could put that in yours. Results Would you choose to participate in a peer review exercise in the future?
Yes No Maybe Dont know 76.6% 3.1% 18.8% 1.6% Some tentative conclusions Giving and receiving feedback are qualitatively different Feedback receipt helps bring deficiencies in own work to students attention, can be motivational and gives a
experience of different audiences (readers) Feedback production appears to be better for developing judgement critical thinking, using criteria and standards, being more objective about own work Fortunately receiving and giving usually occur in the same domain of assignment production hence double duty. More research required
Chapter 3 Projectile Motion. Part 1 - Math Rev; intro to Trig and Vectors. ... Head to tail method (aka tip to tail) Draw first vector. Start next vector where the last one ended (so its tail is connected to...
In C++, cin is an object of a data type (class) named istream, and coutis an object of a class ostream. Header files iostream and fstream contain definitions of stream classes . A class generally contains . private. data and...
Identify all items that contain ICT features or components, and for each identify applicable exceptions and standards. 1. Request detailed accessibility conformance information from potential offerors. 2. Consider including provisions and clauses to manage agency risk after award. 3. How...
7 Small: Albion College, Aquinas College, Calvin College, Hope College, Kalamazoo College, Northwestern MI College, Southwestern MI College ... Majority of schools promoted Flu Challenge through email blasts to students, social media, and on-campus events. Majority of schools did NOT...
10°N O O 0° 10°N 15°S 0° O O 15°S O O West Hemisphere Centered at: 30°N, 30°W East Hemisphere Centered at: 30°N, 210°W EOS CHASMA Who: R. Kuzmin, R. Greeley, D. Nelson, J. Farmer, C. Klein What: Potential sites...
Background. HUGO is formally closing as a project, yet there is a need to make improvements on what exists today. The Optimization project is the term used to describe the projects that will be implemented over the next 12-18 months...
Many people who write in the Bible use other parts of the Bible as examples. Job also had his "friends." With friends like these, one hardly needs enemies! An impatient Christian is a powerful weapon in Satan's hands.