Why Choose WebCT - Curtin University

Enhancing Assessment and Feedback Principles and Practice David Nicol Professor of Higher Education Centre for Academic Practice and Learning Enhancement (CAPLE] Director of REAP and PEER projects (www.reap.ac.uk) University of Strathclyde, Scotland ATN Australia: 20th October 2011 National Student Survey (UK) Assessment & feedback (2008) N o

Survey Statement Englan d Scotlan Norther d n Ireland 5. The criteria used in marking have been clear in advance

69 69 69 6. Assessment arrangements and marking have been fair 74 74

73 7. Feedback on my work has been prompt 56 51 53 8.

I received detailed comments on my work 61 52 52 9. Feedback on my work has helped clarify things I did not understand

56 51 50 82 86 83 22 Overall, I am satisfied with the

Plan Background Re-engineering Assessment Practices (REAP) project [1m] Concepts and example of practice An institutional viewpoint PEER project Background Departments and faculties: supporting educational improvement projects. REAP project: big implementation across 3 universities Policy/strategy: led development of policy in assessment and feedback [based on REAP] Students: Feedback as dialogue campaign

PEER project JISC funded [50k] HE Sector: Project facilitator for QAA Scotland on A&F Research/publications: assessment, learning, change See www.reap.ac.uk REAP : Re-engineering Assessment Practices Scottish Funding Council for Universities (1m project) 3 Universities - Strathclyde, Glasgow & Glasgow Caledonian Large 1st year classes (160-600 students) A range of disciplines (19 modules ~6000 students) Many technologies: online tests, simulations, discussion boards, e-portfolios, e-voting, peer/feedback software, VLE, online-offline Learning quality and teaching efficiencies

Assessment for learner self-regulation www.reap.ac.uk Background (1) Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students learning, Learning and Teaching in Higher Education, 1, 331. See: Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtl Gibbs and Simpson (2004) Assessment tasks [Conditions 1-4]

1. Capture sufficient study time (in and out of class) 2. Are spread out evenly across timeline of study 3. Lead to productive activity (deep vs surface) 4. Communicate clear and high expectations i.e concern here is with time on task how much work students do - their active engagement in study Background (2) Literature Review Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice.

Studies in Higher Education, 34 (1), 199-218 Background Student Enhanced Learning through Effective Feedback [SENLEF] project funded by HE Academy REAP project: www.reap.ac.uk Rethinking assessment and feedback 1. Consider self and peers as much as the teacher as sources of assessment and feedback Tap into different qualities than teacher can provide Saves time Provides considerable learning benefits (lifelong learning) 2. Focus on every step of the cycle:

Understanding the task criteria Applying what was learned in action 3. Not just written feedback: Also verbal, computer, vicarious, formal and informal Seven principles of good feedback Good feedback: 1. Clarifies what good performance is (goals, criteria, standards). 2. Facilitates the development of reflection and selfassessment in learning 3. Delivers high quality information to students: that enables them to self-correct 4. Encourages student-teacher and peer dialogue around learning

5. Encourages positive motivational beliefs & self esteem 6. Provides opportunities to act on feedback 7. Provides information to teachers that can be used to help shape their teaching (making learning visible) Source: Nicol and Macfarlane-Dick (2006) Principle 1: Clarify what good performance is (the context of dialogue) EMPOWERMENT/ SELF-REGULATION Students create criteria Students add own criteria Students identify criteria

from samples of work Exemplars of different performance levels provided Students rephrase criteria in own words Provide document with criteria ENGAGEMENT Two meta principles 1. Meta-PRINCIPLE 1: time and effort on task (structured engagement) i.e. steers on how much work to do and when Gibbs and Simpson 4 conditions

2. Meta-PRINCIPLE 2: developing learner selfregulation (empowerment/self-regulation) i.e steers to encourage ownership of learning the seven principles discussed above. Key task for teacher is to balance 1 and 2 Example: Psychology Psychology 560 first year students 6 topic areas (e.g. personality, classical conditioning), 48 lectures, 4 tutorials, 12 practicals Assessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in

experiment (5%), essay exam (66%) Problems identified No practice in writing skills but required in the exam More detail provided in lectures than mentioned in exams (not enough independent reading) No feedback except on Multiple Choice Questions (percent correct) Didnt want to increase staff workload Psychology Redesign Discussion board in Learning Management System

Students in 85 discussion groups of 7-8, same groups throughout year Also open discussion board for class Friday lectures cancelled discover material themselves Series of online tasks Structure of group tasks 6 cycles of 3 weeks (one cycle x major course topic) First week: light written task (e.g. define terms) = 7 short answers (all answer) Second week = guided reading Week three: deep written task: students collaborate in writing a 700-800 word essay on

the same topic. Within each week: The Monday lecture introducing material Immediately after lecture, task posted online for delivery the following Monday Model answers (selected from students) posted for previous weeks task The teaching role Participation in the discussions was compulsory but not marked (in subsequent years there was 2% mark for participation) Course leader provided general feedback to the whole class often motivational He encouraged students to give each other

feedback The group discussions were not moderated but monitored for participation An example of deep task The Task 800 word essay: Assess the strengths and weaknesses of Freuds and Eysencks theories of personality. Are the theories incompatible? readings suggested questions provided which all students should try Relation to the Gibbs & Simpsons four assessment conditions 1. Tasks require significant study out of class

(condition 1) 2. Tasks are distributed across topics and weeks (condition 2) 3. They move students progressively to deeper levels of understanding (condition 3) 4. There are explicit goals and progressive increase in challenge (condition 4) Relation to 7 feedback principles 1. Standard format and model answers provide 2. 3. 4. 5.

6. 7. progressive clarification of expectations (principle 1) Students encouraged to self-assess against model answer (principle 2) Course leader provides motivational and meta-level feedback and selects model answers (principle 3) Online peer discussion aimed at reaching consensus is core feature of design about response (principle 4) Focus on learning not just marks, sense of control/challenge enhanced motivation (principle5) Repeated cycle of topics and tasks provide opportunities to act on feedback (principle 6) VLE captures all interactions allowing course leader to

monitor progress and adapt teaching (principle 7) Benefits Students worked exceptionally hard Written responses of exceedingly high standard Students took responsibility for learning High levels of motivation: atmosphere in class improved Online interactions showed powerful scaffolding and community building Feedback with 560 students through peer and self-feedback (model answers) Easy for tutors to monitor participation Improved mean exam performance (up from 51-59%, p<0.01) weaker students benefit most

Online postings/interaction 24,362 messages posted by groups for essay tasks Average number of postings per student 44.3 1067 postings to general open discussion forum Students set up online study groups for other subjects Structured tasks online triggered important socio-cognitive processes Has it worked? I read more about Psychology and read it earlier in each semester than I would have done without the online projects

60 Numbers 50 40 30 20 10 0 1 2 3 agreed-disagreed

4 5 numbers I found that reading other people's contributions helped me understand Psychology 80 70 60 50 40 30

20 10 0 Series1 1 2 3 agreed-disagreed 4

5 I learned more in Psychology because of online projects than I did in my other subjects 60 numbers 50 40 30 Series1 20

10 0 1 2 3 agreed-disagreed 4 5 Questions and Discussion

Why use principles 1. 2. 3. 4. 5. 6. 7. Provides a framework for operationalizing the big idea the development of learner self-regulation Helps translate the research into accessible guidelines for teaching practice Ensures change is educationally driven Enables connections to be made across innovations

in different disciplines Provides a common language to talk about innovation and for dissemination Can add to the evaluation process measures Helps identify where technology can leverage benefits Guidelines for Implementation A single principle or many? 2. Tight-loose maintain fidelity to the principles (tight) but encourage disciplines to develop their own techniques of implementation (loose) 3. Balance teacher feedback with peer and selfgenerated feedback 4. The more actively engaged students are, the better the course design

1. Developments since REAP 1. Principles of Assessment and Feedback approved by University Senate and embedded in policy (2008) 2. Use of principles to inform curriculum renewal and Quality Assurance processes 3. Feedback as Dialogue campaign to gain commitment of students 4. PEER Project (Peer Evaluation in Education Review) Peer Review in Education

Evaluation [PEER] The aims of the PEER project are to: Review evidence base for peer review Develop educational designs for peer review (and self-review) Identify software support for peer review Pilot implementations of peer review with large student numbers Produce guidelines for higher education why do it, how to do it, pitfalls and solutions and software possibilities. see http://www.reap.ac.uk/peer.aspx Peer feedback: augmenting teacher feedback

Increasing quantity and variety of feedback No extra workload on teacher with software (e.g. PeerMark, Aropa) More timely e.g. collaborative projects Simulates professional life reconciling different feedback perspectives The argument Not enough attention has been focused on the potential of peer feedback not just as a way of increasing the quantity and quality of the feedback students receive, but as a way of giving students practice in constructing feedback.

The focus Scenarios where students make evaluative judgements about the work of peers and provide a feedback commentary, usually written Not talking about scenarios involving ...informal feedback in collaborative tasks .....students evaluating each others contribution to group working .....students grading/marking each others work, although some rating might be part of peer design

Benefits of feedback construction (1) Constructivist rather than telling paradigm 1. High-level cognitive activity: students cannot easily be 2. 3. 4. 5. passive Students actively exercise assessment criteria from many perspectives Writing commentaries develops deep disciplinary expertise

See many approaches and learn that quality can be produced in different ways Shifts responsibility to student puts them in role of assessor exercising critical judgement Benefits of feedback construction (2) 6. Learn to assess own work as exactly the same skills are involved Develops capacity to make evaluative judgements - a fundamental requirement for life beyond university. Also, this capacity underpins all graduate attribute development (Nicol, 2010) Nicol, D (2011) Developing students ability to construct feedback,

Published by QAA for Higher Education, UK Example 1:Peer feedback Laboratory work [Gibbs 2011 www.testa.ac.uk/resources] Reason poor quality of lab reports in science Strategy students organised in groups and produce poster to represent their lab report Peer process hang poster in class and all students individually walk round analyse posters and write feedback on them (e.g post-its): questions, suggestions, inaccuracies etc Result significant improvements in lab work and reporting, positive competition in class, students did not want to look bad.

Example 2: Engineering Design Peer Project case study DM 100 Design 1: first-year class Dr Avril Thomson, Course Leader, Design Manufacturing and Engineering Management (DMEM), University of Strathclyde [email protected] Caroline Breslin, Learning Technology Adviser, University of Srathclyde [email protected] Example 2: Design 1 82 first-year students Design a product theme eating and resting in the city Research in groups (in city, in library etc.)

Individually produce a Product Design Specification (PDS) detailed requirements for and constraints on design (rationale, performance,standards, manufacturing etc) Given a PDS exemplar from another domain to show whats required (stainless steel hot water cylinder) Online learning environment: Moodle and PeerMark part of Turnitin suite Product Design Specification Typical Headings

Rationale Performance Environment Packaging Maintenance Weight and size Disposal Shipping Processes Time Scale Patents Product costs Aesthetics Ergonomics

Materials Testing Quality/reliability Competition Marketing Life in service DM 100: Design 1 Peer review task Individually, each student peer-reviewed and provided feedback on the draft PDS of two other students Criteria: completeness, convincingness of rationale, specificity of values (performance) and one main suggestion for improvements with reasons Students used experience, giving and receiving feedback, to

update PDS which comprises part of a Folio Students self-review Folio and meet/discuss with tutor Peer review not assessed directly but 10% marks for professionalism which included participation in peer review. Peer review rubric: DM 100 Do you feel the PDS is complete in the range of headings covered? If no, can you suggest any headings that would contribute towards the completeness of the PDS and explain why they are important? Is the PDS specific enough? Does it specify appropriate target values or ranges of values? Please suggest aspects that would benefit from further detail and explain.

To what extent do you think the rationale is convincing for the PDS? Can you make any suggestions as to how it might be more convincing? Please explain. Can you identify one main improvement that could be made to the PDS? Provide reason(s) for your answer. Evaluation 1. Online survey completed by 64 students 2. Course work marks compared to previous years 3. Focus group interviews 4. Peer review comments recorded online

Results 1 Which aspects of the peer review did you learn from? Giving feedback Receiving feedback Giving and receiving feedback Neither giving or receiving 10.9% 26.6%

54.7% 7.8% Results 2 Did you modify your initial submission as a result of the peer review activity? Yes, as a result of the peer review given 23.4% Yes, as a result of the peer review received 25.0%

Yes as a result of the peer review given AND received 28.1% No 21.9% N/A 1.6% Results: student comments

If yes, please give specific examples of modifications (n=41) I added a couple of paragraphs and improved existing paragraphs, this added two full A4 pages to my work I included specific materials as changed the formatting of the document so it looked more professional I provided more specific numeric values and expanded my rationale after seeing someone elses PDS and after receiving feedback Results: RECEIVING feedback Please give examples of what you learned from RECEIVING peer reviews from other students (n=54)

Parts that I had previously missed were brought to eye such as market competition (noticing) Receiving peer reviews gave me insight into what others thought of my work and gave me a direction to improve (reader response) Where the PDS was confusing to understand (reader response) I found out how good mine was (motivational) The person who peer reviewed my PDS gave me positive feedback which helped me a lot (motivational) Not much, they werent very good Results: PROVIDING feedback Please give examples of what you learned from PROVIDING peer reviews of others work (n=47) When giving advice to people on theirs, it gave me greater perception when reviewing my own work by listening to my

own advice for example [transfer] I had a chance to see other peoples work and aspects of their work that I felt were lacking in my work, this helped me to improve my work [transfer] I was given a greater understanding of the level of the work the course may be demanding [standards] Allowed me to see from an assessors perspective Thinking from a critical point of view [critical judgement] How to look at work critically that isnt your own Results: How you carried out peer review Could you make any comments about how you carried out the peer review? How did you evaluate the quality of the work to provide a response to the peer review

questions? (n=37) I compared it to mine and the ideal PDS and said how I would improve it Partly by comparing my work to theirs I tried to think about what I wrote and whether this PDS was better or worse Looked through the headings first and noted any that should have been included then read more in detail about each individually and compared it with my own PDS Focus groups How did you go about reviewing? I read it through and compared it with what I had done to see if they had put something I had not done and then I added it in if they hadnt. The four questions were useful as

they provided a framework for the review. If we hadnt had the questions it would have been difficult. I did the reviews separately and then answered one then the other. The first was a better standard than the other so I used the ideas from the better one to comment on the weaker one. I also read the guidelines in class when I did the peer review. There were ideas from the good one that I hadnt even thought of in mine Focus groups What do you think is best for learning giving or receiving feedback? For me it would probably be to give feedback because I think seeing what other people have done is more helpful than getting other peoples comments on what you have

already done. By looking at other peoples work you can see for yourself what you have forgotten or not even thought about. When people give feedback on yours they generally just talk about what is there. They dont say, well I did this on mine and you could put that in yours. Results Would you choose to participate in a peer review exercise in the future?

Yes No Maybe Dont know 76.6% 3.1% 18.8% 1.6% Some tentative conclusions Giving and receiving feedback are qualitatively different Feedback receipt helps bring deficiencies in own work to students attention, can be motivational and gives a

experience of different audiences (readers) Feedback production appears to be better for developing judgement critical thinking, using criteria and standards, being more objective about own work Fortunately receiving and giving usually occur in the same domain of assignment production hence double duty. More research required

Recently Viewed Presentations

  • Tasks of BLO Related to Part, Section and

    Tasks of BLO Related to Part, Section and

    superimposed on part boundary map as Fig-5. then habitation is marked by hand and section are created by grouping the electors based on house number. if house number is not there then a notional house number is given. after creation...
  • Chinese Dynasties Review

    Chinese Dynasties Review

    New group, compradors, emerge - wealthy Chinese merchants who specialized in imports/exports that tied China to global mkt. Dynastic cycle problems return: breakdown of bureaucracy due to cheating on exams, buying gov't positions (esp. by wealthy compradors), tax $ pocketed,...
  • Portrait Series (midterm)

    Portrait Series (midterm)

    Create drawing on 6x9 inch paper. Examples. Student Examples. Portrait 2. Draw the same portrait image with the same composition. Shade the portrait in using your choice of scumbling, stippling, hatching, or cross-hatching. Shading must be completed in pen/ink. Demonstrate...
  • Advanced Biology 9/5/12

    Advanced Biology 9/5/12

    Which organisms have prokaryotic cells? Endosymbiotic Theory The Endosymbiotic Theory Review: What is a theory? What is the difference between prokaryotic and eukaryotic cells? The endosymbiotic theory is the idea that a long time ago, prokaryotic cells engulfed other prokaryotic...
  • Engine Oil - Santa Rosa Junior College

    Engine Oil - Santa Rosa Junior College

    Engine Oil Jason Dearden Steven Lemos Engineering 45 Santa Rosa Junior College Spring 2009 What Is Engine Grade Oil A Brief History of Motor Oil 1846 Crude oil refining process discovered Single Grade oil Multi Grade oil Synthetic oil first...
  • Embryology Review - University of Arizona

    Embryology Review - University of Arizona

    *bilaminar disk of 2 germ layers: epiblast and hypoblast *2 cavities: amniotic cavity and yolk sac *2 placental components: syncytiotrophoblasts, cytotrophoblasts. This is when the mother will miss her period. Wk 3 gastrulation: Rule of 3's. 3 germ layers: ectoderm,...
  • PowerPoint Presentation

    PowerPoint Presentation

    14-1 What Are the Earth's Major Geological Processes and Hazards? Concept 14-1A: Gigantic plates in the earth's crust move very slowly atop the planet's mantle, and wind and water move the matter from place to place across the earth's surface.
  • Introducing Atoms - Sierra Vista Chemistry

    Introducing Atoms - Sierra Vista Chemistry

    There are about 100 different elements. Each element is made up of very tiny particles called atoms. However, each element is made up of just one particular type of atom, which is different to the atoms in any other element....