Members of The Careers Group alongside the Ƶ’s Centre for Online and Distance Education were delighted to be awarded one of 16 funded projects from the QAA.
Blog
Redesigning Assessment at RIDE 2021
The fifteenth in the CODE’s acclaimed series of Research in Distance Education (RIDE) conferences was the first to be held wholly online. It took place over three half-days in June 2021, with the overall theme of Learning through Disruption. Online assessment is a particularly tricky issue, and much has been written about it since the first lockdown sent almost all HE activities into the virtual world. It is not surprising, then, that the RIDE committee chose to devote the whole of the second day at the 2021 conference – Thursday 17th June – to this topic under the strapline Redesigning Assessment.
Moving assessment online at scale
Dr Linda Amrane-Cooper, Prof Stylianos Hatzipanagos, Prof Alan Tait
The day began with a keynote panel of CDE stalwarts: Linda Amrane-Cooper, head of the CDE, and the two CDE research leads (and RIDE conference chairs), Stylianos Hatzipanagos and Alan Tait. The team presented an evaluation of how the Ƶ had adapted exams for tens of thousands of students in many countries for this new digital world. It was appropriate, therefore, that the session was chaired by the Director of Operations for the University’s International Programmes, Craig O’Callaghan.
Stylianos began by explaining that they would focus on three important stakeholders in the exam process: programme directors, examiners and the students themselves. He pointed out that, although anecdotally students seem to prefer online exams once they have experienced them, this nonetheless raises important issues of equitable access, pedagogic validity and staff competence. It seems that a slightly larger proportion of students gained ‘good’ degrees in 2020 compared to previous, more conventional years, and there are still questions about how candidates’ identity can be confirmed online. Coronavirus has been described as representing a ‘perfect storm’ for assessment offences but, on balance, digital assessment is viewed as having had more positive effects than negative ones.
The Ƶ’s International Programmes have provided an excellent case study of how these general principles are borne out in practice at scale. In the summer of 2020, a total of 38,000 Ƶ University students in 23 time zones took over 100,000 individual online exams. The survey presented here covered areas of student behaviour; student sentiment (using student interviews and an extensive survey); outcomes in terms of marks and awards, and operational issues.
The first finding presented was a basic but reassuring one: 93% of the exams scheduled actually took place, which was slightly up on previous years. Furthermore, 79% of students reported that their exams had enabled them to demonstrate their learning. The dissatisfied minority were, generally, those who had had difficulties accessing and using the technology. A wide variety of exam formats were used, with some students given a few hours and others, days to complete ‘open note’ versions of traditional exams, and the proportion of referrals for potential offences was acceptably low.
The team also reported on the views of examiners and course directors. Just over half of all examiners surveyed reported that students had achieved higher standards than in previous years, and most of the others thought they had stayed the same. Staff workload was reported as a serious issue, and some staff felt that they lacked support and training in dealing with new procedures. It was clear that, although a promising start has been made, further assessment redesign will be needed.
Alan ended the presentation by looking to the future of assessment, as everyone accepts that ‘there will be no return to 2019 in 2022’ – or, almost certainly, at any time after that. The genie of online assessment is out of the bottle, and students are largely satisfied, but problems with access and ensuring academic integrity remain. With many problems surrounding online proctoring, the focus is likely to turn to designing ‘unplagiarisable’ assessments: probably, many different types.
Monica Ward of Dublin City University responded to the presentation by commending its scale, surveying a large number of students and covering a wide range of themes. Despite the overall success of the 2020 exam season, the study raised specific questions around academic integrity and access to technology. Disabled students are likely to benefit from being able to use their own laptops with their own assistive software, and redesigning assessments to test skills rather than knowledge should improve them significantly, but these benefits will only be accessed with increased support and training for staff and students alike.
Large scale redesign of assessment in the UK and Spain
Moving towards inclusive assessment
Assessment innovation in a time of disruption
If nothing else, the disruption caused by the pandemic has driven an enormous wave of innovation across the higher education sector, and this is perhaps particularly true of the way our students are assessed. One of the parallel sessions on RIDE’s ‘assessment day’ looked specifically at this type of innovation. Firstly, Elisabeth Hill of Goldsmith’s, Ƶ looked at assessment in a discipline that could be one of the hardest to move online: the performing arts. Then, CODE fellows Alan Tait and Gwyneth Hughes followed on from the opening keynote session by addressing the question of whether covid has, after all, triggered the radical rethink of assessment that is being widely discussed.
Disrupting the assessment of performance in the creative disciplines: lessons from Covid
Elisabeth Hill, Goldsmiths Ƶ
Elisabeth focused in on assessment in courses on theatre and performance, which is one of many creative subjects taught at Goldsmith’s. This department offers a radical BA programme in drama and theatre arts, in which third-year undergraduates form small theatre companies to create a critically informed, collaborative piece of performance art that is presented live as part of a festival at the end of the year. Before covid, most of these performances took place in the college’s George Wood theatre. Occasionally, presentations would take place outside on location.
The first lockdown in March 2020 sent students who were already working on their group presentations home, and for a few months all collaborative work took place from individual bedrooms and studies, not all in Ƶ or even the UK. Later in the year, the same students had to move their work in and out of lockdown. There were, essentially, three types of performance:
- Work developed online performed in small groups to tiny audiences in the George Wood theatre, recorded and shared online
- Completely digital performances, recorded with students in different locations (often their homes)
- Installations recorded live in studios
These students faced much harder technical challenges than previous cohorts but overcame them very well and the overall quality was similar to that in previous years. Elisabeth commented that (like others) Goldsmith’s would not be returning to 2019, but future performances will probably be mixed mode rather than fully digital. All the student performances can be viewed at https://www.tapoutgoldsmiths.com/.
Has the pandemic triggered a radical rethink of assessment?
Gwyneth Hughes UCL, Alan Tait Centre for Online and Distance Education
Gwyneth and Alan then turned from this fascinating but specific example to general principles of assessment, by reporting on a project they had led exploring how the purpose of assessment has changed. They conceived this project, which covered a wide range of Ƶ colleges and disciplines, in 2019 but found it could readily, and usefully, be adapted to the circumstances of the pandemic.
The Ƶ’s exams system was set up in 1858, and over more than a century and a half it has assessed millions of students in centres across the world. Until very recently, the vast majority of these were primarily assessed by ‘classic’ end of year exams. In 2017, Gwyneth and some CODE colleagues launched an assessment toolkit, encouraging staff to move away from these and use assessment to support learning. A few years of slow, steady progress were interrupted by 2020’s shift to digital, when 160 years’ worth of assessment strategies were swept away almost overnight.
The exam, which had previously been seen as a ‘gold standard’ of assessment, is a good way of testing students’ attainment but often provides little support for learning and feedback is limited. The switch to open book exams in 2020, however, showed that these provide students with far more opportunities to think and to learn.
Gwyneth and Alan interviewed programme directors from a range of undergraduate and postgraduate programmes about their experience of online assessment, and came up with three suggested ‘rethinks’ for designing assessment for a digital age:
- Strengthen formative assessments, which provide more opportunities for learning, and design these assessments alongside or even before the course materials so students can take opportunities to improve throughout a module or from module to module across a programme.
- Abandon or limit the use of plagiarism detection software in assessment and move to ‘designing plagiarism out’ with skills-based assessments. Instead, allow or ask students to use this software to test their own work.
- Invest as much effort in developing assessments as you do in developing course materials, particularly for distance learning.
Rethinking teaching and learning
Thursday’s final RIDE session bridged the gap between the theme of assessment and the next day’s of ‘rethinking teaching and learning’. It was chaired by Linda Amrane-Cooper, head of the CODE, and featured presentations by Betty Vandenbosch, Chief Content Officer at Coursera in the US, and Sam Brenton, Director of Innovation and Educational Development at the Ƶ. Each, in a different way, tackled the question n online pedagogy as we – hopefully – move out of the crisis towards that widely discussed ‘new normal’.
Preparing for the new normal: The promise of online learning
Betty Vandenbosch, Chief Content Officer at Coursera
Coursera is one of the world’s major commercial providers of massive open online courses (MOOCs). It was thus well placed to help with the sudden adoption of online learning in 2020. Overall, about 1.6 billion learners (school children, students and adult learners) have been affected by COVID-related institutional closures. Betty used UK data to show that a majority of university students want to continue learning online after the pandemic, but few lecturers feel fully prepared to teach in this way.
It is now clear that the educational landscape has changed forever. Blended learning, affordability, employability and supporting teaching staff are the new buzzwords. Betty showed in case studies from India and Hungary that students value flexibility and diversity in their learning.
However, the workforce is facing disruption from many factors other than covid. It has been estimated that millions of jobs will be displaced by 2025, and other workers will have to upskill to stay in theirs. The lost jobs will be displaced by others that require advanced digital skills, and most of these can be learned online. Digital courses are significantly cheaper than traditional ones and are taken up by a different type of student. The average degree student on the Coursera platform is 35 years old; these are busy professionals who appreciate the opportunity to study anywhere, at any time and to ‘stack’ together modules into tailor-made programmes. This model is particularly helpful to those who want to taste programmes before signing up or to accredit prior learning. Millions of people worldwide could benefit from this flexible, affordable and employer-led model of learning, but it will be really important to support and train staff to deliver it in a rich and engaging way. Betty ended her presentation with the thought that access to online education creates more equal opportunities, and that this creates a more just world.
Questioning the consensus in online pedagogy for higher education
Sam Brenton, Ƶ
Sam ended the day with an eloquent and impassioned call – without slides – for us to think again about the pedagogy that advocates of online education have adopted over the past few decades. He started with a few reflections from when he started working in higher education in the late 90s, and of the beginning of the web and social media. Then, every development and every technology was being flagged as ‘going to end long-form teaching for once and all’. He remembered arguments between discipline-based academics and learning technologists, and a young German lecturer expecting to be told that his knowledge-focused approach to teaching was simply unacceptable. This is the perennial conflict between the ‘sage on the stage’ and the ‘guide on the side’ approaches to education; it is almost taken for granted that those who teach at a distance will be enlightened, progressive guides.
But even now, when the world has moved online out of necessity, he argued that long-form teaching, and the much-maligned and – according to the Vice-Chancellor of the University of Leeds – ‘pedagogically unsound’ 45-minute lecture, can still have value. In the online world, even, long podcasts are widely used by students who, according to theorists, can concentrate for no more than six minutes. This talk, he said, was not so much a defence of traditional teaching methods as a plea for a flexible approach. There are a wide range of potentially valuable pedagogical approaches, and the most appropriate ones may be discipline-specific; law students, for example, might benefit from learning methods that echo courtroom practice. He ended with a call: “Let’s invent [pedagogy] together: and let’s do so with open minds.”
This thought was an excellent one to bear in mind going into the final day of sessions, with its strapline of ‘Reinventing teaching and learning’.
