29 June 2021
Reflecting on holding high-stakes OSCE style assessments in a COVID-19 environment
Objective structured clinical examinations (OSCEs) are high-stakes, stressful assessments for both examiners and students under any circumstances. During the COVID-19 global pandemic, a UK wide lockdown resulted in a temporary postponement of all veterinary nursing practical assessments, until appropriate procedures and government sanctions permitted OSCEs to be carried out in a safe environment. This is a reflective account of holding OSCE style assessments using a social distancing protocol, highlighting the positive impact these measures had on student assessment experience.
Being a member of the BVNA family means you have a free Veterinary Nursing Journal in the post every month, providing expert written content, and online access to all past issues with topic search facilities for easy research. Join us today to access all of our VNJ content!
The UK entered COVID-19 lockdown on Monday March 23, 2020, with all but essential travel and work put on hold to limit the spread of the highly infectious respiratory disease. A two metre “social distance” was required between people not living in the same household, except for essential workers where appropriate respiratory and droplet screening personal protective equipment was required. The RCVS requested that veterinary nursing (VN) educators postpone all practical assessments for a period of three months on the 27th March (RCVS, 2020), leaving many final year VN students facing uncertainty and potential delays to completing their qualification.
At Nottingham Trent University (NTU), the final year VN practical assessments were due to take place during the week commencing March 23, 2020, so had to be postponed until government guidance permitted returning to work for non-essential workers. As this end point assessment is a requirement for registration as a VN, this postponement effectively prevented any final year VN students from graduating.
On June 1, 2020, primary schools in England were scheduled to re-open to children, alongside outdoor sports facilities (The Health Protection, (Coronavirus, Restrictions) (England) (Amendment) (No. 3) Regulations, 2020). The “stay alert” advice also permitted travel to work at businesses not subject to restrictions where working from home was not possible (Cabinet Office, 2020). As practical skills assessments require students to complete around 10–12 practical tasks, typically with individual examiners overseeing each individual task (RCVS, 2018), conducting OSCE like assessments under social distancing rules appeared challenging.
Proposed social distancing measures for practical assessment
Following reports of a successful OSCE being run for medical students in Singapore under COVID-19 restrictions (Boursicot et al., 2020), and following the easing of the UK lockdown to permit non-essential workers to return to the workplace, considerable time was spent exploring options to facilitate running VN practical assessments for final year students.
Figure 1. Fence numbers indicated the one-way system to students, with sufficient space around each station to allow safe movement of staff and students throughout the assessment, and both station examiner and senior examiner to be present for internal verification.
At NTU a proposal was put forward to hold our OSCE like assessments – time-constrained scenario-based practical examinations (TSPEs) – in the Mary King Indoor Equine Arena in mid June. The principle difference between TPSEs and OSCEs, is the use of a scenario to link the practical skills being assessed, and a single time limit to complete the scenario rather than individual tasks (Hall et al., 2019). Crucially, students complete the assessment one by one, completing each station in a fixed order allowing a one-way system to be established throughout the assessment (Figure 1). The indoor arena at NTU measures 40 × 60 m, with Yorkshire boarding providing constant fresh air ventilation, a roof to protect the arena from all but the worst weather, and both power and lighting to create a safe working environment. The use of one additional classroom was proposed to house the radiography TSPE station, thus ensuring all Day One Skills were included in the assessment blueprint (RCVS, 2016). Each practical station therefore had a footprint of at least 3x3m, with stations in the arena distanced approximately six metres apart (see Figure 2). The space available in the indoor arena ensured two metres social distancing measures could be maintained by all persons present for the assessment at all times.
Figure 2. White barriers were used to demark each station outline.
Figure 3. Groundsheets protected the arena surface from contamination. The edges of the tarpaulin where buried in sand to reduce trip hazards, the sheets themselves protecting the surface from contamination by sharps or small items.
Tables, chairs, and other equipment were transported from the VN Centre to the arena by means of a trailer, with both VN and Equine Technical staff setting up the practical stations. The arena has a Prowax sand riding surface and is used primarily for equestrian events and training. It was therefore essential that no contamination of the surface occurred during the TSPEs. All practical stations involving the use of sharps (needles or glass slides) had a protective sheet (tarpaulins or storm sheets) placed on the ground to ensure any dropped equipment could be safely retrieved. To prevent the risk of tripping, the edges of the sheets were covered in sand to ensure no edges protruded (see Figure 3).
Traditionally, practical examiners stand close to the student performing the clinical skills, in order to observe their completion of the task. As examiners were required to stand at least two metres away from the student at all times, video cameras were positioned to record each practical task, providing a means of reviewing student performance should any queries arise (Figure 4). Where examiners were required to check a calculation, or an aspect of the station before the student left, students were asked to step away from the station temporarily to allow the examiner to approach. Face coverings (typically home-made cloth masks) were worn by most of the examiners, however in line with government guidance this was not a mandatory requirement as staff and examiners were not talking (unless necessary) or within two metres of one another at any time.
Figure 4. Fence markers were used to signpost each station, with hand sanitiser available between each task. The examiner chair (purple) and table were positioned at least 2 m away from the station, with tripods closer to the task videoing all students.
To minimise the potential for viral spread between examiners and students each practical station was disinfected with Virusolve® (Amity International) between every student, including all metal and plastic equipment. Recognising that disinfecting soft toys and fabric would be impossible (Boursicot et al., 2020), the use of such materials was limited as far as reasonably possible – for example, closed gloving and gowning was swapped for open gloving to eliminate the need for examiners to stand within two metres of students and remove the potential for transfer on the gown. Both examiners and students were required to perform hand sanitising using Virusan® gel (Amity International) between every station/student to minimise the risk of station contamination (Boursicot et al., 2020). Students were permitted to pause their timers to perform this task, with hand cleaning stations clearly signposted throughout the assessment (Figure 4).
As the university campus is home to many animals, staff were working on site throughout lockdown. It was therefore essential that the safety of all staff on campus was ensured through careful stewarding of the assessment attendees. Staff and students were asked to avoid public transport wherever possible, and if unavoidable were asked to bring a clean pair of scrubs to change into on arrival. Students were scheduled to begin their assessment every 20 minutes and asked to remain in their cars until called through by a steward. Both staff and students were asked to bring all refreshments with them, and take all waste home with them, staff were asked to take their lunch breaks either outside (maintaining two metres distance from other examiners) or in their cars. To ensure single occupancy in the narrow locker/toilet building, a cone was positioned outside the building, with all attendees instructed to use their feet to push the cone into the centre of the path to indicate the building was in use.
As suggested by Boursicot et al. (2020), the examiner briefing and standardisation training was conducted via video call, using Microsoft Teams (Microsoft Office 365). Individual station briefings were video recorded to allow examiners to familiarise themselves with their station prior to arrival. Student assessment papers and marking sheets were paper based. The team are hoping to move to electronic marking using iPads in the future, but it was impossible to instigate this in time for the assessment. The senior examiner and the assessment assistant were responsible for collating all paperwork, which was immediately scanned to create a digital copy for moderation purposes. After handling any paper hand washing was required.
The full assessment protocol and risk assessment details can be found in the supplementary material.
Following completion of the TSPEs, students were asked to provide anonymous feedback to enable the VN team to plan for future practical assessments due to take place in July and September. The feedback was overwhelmingly positive, highlighting some aspects of the TSPEs where modifications could be made to improve student experience. The feedback survey was completed by 26/37 students, of which 14 (53.8%) had been working in veterinary practice throughout lockdown, 5 (19.2%) had worked for some of the period, and 7 (26.9%) had not worked in practice at all since lockdown started. Traditionally, students are able to use the practical equipment and resources in the VN Centre the week prior to the TSPEs to practice their clinical skills on the equipment they will encounter in the actual summative assessment. This was not possible under social distancing rules, therefore some students had no opportunity to practice their clinical skills for three months prior to sitting the assessment.
Around three quarters of the students responding to the survey felt lack of access to the VN Centre negatively impacted their performance (see Figure 5). Student access to preparation support and practice opportunities was identified as a key concern to holding practical assessments during lockdown and can not be dismissed as these high-stakes assessments impact their ability to register as a VN. In lieu of physical practice opportunities, students were directed to a “virtual VN Centre” on the NTU virtual learning environment, providing access to previously recorded practical demonstration videos. These videos were previously developed to support students preparing for practical assessments (Hall & Simpson, 2018), and were the only preparation support the teaching team could offer students preparing for TSPEs under lockdown. Feedback highlighted the limitations of not being able to physically practice for what is ultimately a practical assessment, and highlighted the inequalities facing students who had not been working in practice during lockdown:
Figure 5. Student perceptions of the impact lack of practical opportunities had on their examination performance.
“Some students have been in practice this whole time, whilst others haven’t so much as picked up a needle and syringe in almost four months. I felt severely disadvantaged and out of practice.
I feel it is extremely unfair that some students have been privileged with near-unlimited practical experience before the exam, whilst others haven’t even seen the equipment with their own eyes in months.”
The move to the equine arena resulted in mostly positive feedback to the revised TSPE conditions (see Figure 6). When asked how they found the altered layout, 18 (69.2%) students indicated that they preferred the layout to the normal assessment environment however, 3 (11.5%) students reported that the conditions made it difficult to concentrate. Free text responses to this question generated mixed responses:
Figure 6. Student responses to the multi-select question regarding perceptions of the examination layout using the equine arena.
“Because it was so spread out it felt like there was less pressure and I felt a lot calmer.
Being in the arena instead of classroom did make it slightly harder to concentrate.”
As most examiners opted to wear face coverings, students were asked about the impact this had on their assessment experience. Only 3 (11.5%) students reported that the masks were off putting, whilst 7 (26.9%) indicated that they felt reassured and preferred having the examiners further away from the station and 9 (34.6%) reported that they did not notice the masks and distancing measures. No students reported that examiners had broken the social distancing requirements during their assessment. Responding to the question “How did you feel about the social distancing measures in place,” 25 (96.1%) felt they were robust and appropriate, with the remaining student indicating that they were “over the top.”
General feedback from the students suggested that the spacious layout and distance from the examiners had a positive effect on their TSPE experience, but the changes to the assessment set-up was unfamiliar and thus challenging:
“I think the WHO hand washes in between each station gave you chance to catch your breath and it didn’t feel rushed. I have never felt more calm in my life doing an exam!
Although it was quite strange, I much preferred this experience. I felt more in control and relaxed, having a little break between stations was really helpful and effective. I was much more comfortable during this exam as we weren’t tightly packed and on top of each other, sometimes the normal exam can be quite distracting when I hear other students explaining what they’re doing at their stations but this way, that was eliminated.
The exam itself was set out fine, and I’m grateful that we’ve had the opportunity to sit it this summer. The whole thing was very well organised and the social distancing layout was perfect.
The exam conditions made me feel more anxious than usual which I feel did impact on my performance but I am happy that we were able to have the opportunity to sit the exam at this time instead of waiting even longer.”
Examiners were also asked to provide anonymous feedback on their TSPE experience. Most practical examiners – 7/12 (58.3%) – had not worked in veterinary practice during lockdown, only 3/12 (25.0%) had practiced throughout with the remaining working in practice for some of the lockdown period. All 12 examiners reported that the briefing instructions were very clear, and felt the social distancing measures were robust and appropriate.
When asked how the exam layout impacted their experience assessing students, the examiners’ responses mirrored those of the students (Figure 7). The majority – 9/12 (75.5%) – of examiners indicated that they preferred the layout to the normal assessment conditions, whilst 2/12 (16.7%) found the outdoor conditions distracting. When asked how the requirement to remain two metres away from students affected assessing students, 7/12 (58.3%) examiners felt it was no different to the normal assessment experience, 2/12 (16.7%) preferred being further away, whilst 2/12 (16.7%) found it challenging being so far away from the students.
Figure 7. Examiner responses to the multi-select question regarding perceptions of the examination layout using the equine arena.
Free text responses relating to the revised examination protocol were largely positive:
“The exam conditions felt very relaxed. The students appeared to remain calm and collected. I feel the distance between examiner and student had a part to play in this, as the student did not feel pressured by an examiner being stood next to them.
I thought that although in a different environment it flowed very well and seemed to be a very quiet calm atmosphere.
There were some very minor distractions, but I feel that this relaxed people and the layout was good as you could physically see how the candidates were progressing through the stations and you could plan your time better as you knew how long it would be before they arrived at your station.
I felt safe and still had ability to assess the student appropriately.”
No staff or students reported developing COVID-19 symptoms within 14 days of completing the assessment. The protocol was therefore re-used for the subsequent assessment opportunity in July for students unable to attend the first sitting, or unsuccessful in their attempt. The assessment results were excellent, suggesting that whilst the lack of physical practice opportunities clearly disadvantaged some students, for the majority it did not impede their ability to demonstrate competence.
The positive feedback highlighted the need to review our standard delivery of the TSPE assessments. Whilst having an overall time limit empowers the students to work at their own pace and control how long they spend on individual tasks depending on their confidence, leaving the timer running between tasks adds pressure. The team plan to incorporate pauses between tasks in future, potentially still including a WHO handwash if social distancing measures remain in place, but mainly to allow students an opportunity to reflect, collect and refocus before moving on to the next task.
The comments relating to space between tasks, reduced pressure from having the examiner further away and reduced awareness of other students in the assessment have also prompted reflections on the layout of the TSPEs (Figure 8). Whilst holding assessments in the equine arena may not be logistically possible in the future (it is normally in near-constant use for competitions or training), the team plan to review room usage to try and spread stations out, allowing more space and crucially providing additional distance between the student and the examiner.
Figure 8. Students performed the practical tasks at a 2 m minimum distance from the examiners, with increased space around each station.
During the COVID-19 outbreak veterinary practices remained open to provide essential care to patients throughout the UK, veterinary nurses continued to practise alongside many student nurses. Understandably, education and assessments had to be modified to limit the potential for infection spread and resources had to be prioritised for providing both human and animal medical care. However, it is vital that the next generation of veterinary professionals are trained, assessed and given the opportunity to qualify. By adapting the TSPE protocol to ensure social distancing measures were maintained, students were able to demonstrate their clinical competence whilst also demonstrating their ability to practice safely during a global pandemic. The changes to the assessment protocol highlighted aspects of the TSPEs that could be permanently altered to improve the assessment experience for both staff and students. The next generation of veterinary nurses have overcome unprecedented challenges to complete their professional education and achieve their ambitions, and we are thrilled to be able to welcome them into the profession.
We would like to thank everyone who enabled these assessments to take place, especially: Anna Gregory, Equine Technical Team Leader, and Amy Hazlehurst, Senior Equine Technician, for overseeing the logistical challenges of setting up and stewarding the exams. Both the NTU and external practical examiners without whom the assessment would not have been possible. Kathy Kissick, our external examiner for supporting the team through the re-design and quality assurance of the assessment. All the students who embraced the challenge of completing their TSPEs during a global pandemic, adapting to challenging assessment conditions and ensuring the safety of their examiners by diligently following the protocols in place.
Notes on contributors
Alison Simpson RVN LIQA FHEA
Alison qualified as a veterinary nurse in 1994, working in small animal practice before moving into veterinary nursing education. Alison has also worked for the National Blood Service and examined for the RCVS. Alison now course leads the FdSc in Veterinary Nursing at Nottingham Trent University and is Senior Examiner for the practical skills assessments, responsible for training and standardising the teaching team.
Matthew Hall FdSc RVN CertVN ECC AFHEA
Matthew qualified as a veterinary nurse in 2017 and completed the VetsNow ECC certificate in March 2019. Matthew has previously worked in a large referral hospital in various roles, before moving into veterinary nursing education in March 2020. Matthew is now the veterinary nursing Centre Coordinator at Nottingham Trent University.
Emily J. Hall MA VetMB PGCAP MRSB MRCVS SFHEA
Emily qualified as a veterinary surgeon in 2007 and has worked in small animal first opinion practice ever since. Emily now course leads the BSc (top-up) in Veterinary Nursing Science and teaches veterinary nursing students on both the FdSc and BSc courses at Nottingham Trent University. Emily’s research interests include temperature monitoring and heat-related illness in companion animals.