Exclusive: Registrars flagged exam concerns with college months ago

The RACP stuck with its plan, citing greater efficacy and security

EXCLUSIVE


Registrars warned the Royal Australasian College of Physicians of their concerns about the switch to  computerised examinations months before this week’s inaugural attempt was spectacularly felled by an IT glitch.

Letter

The college had no option but to pull the plug midway through Monday’s five-hour Basic Training Written Divisional Exam, leaving many of the 1200 candidates baffled and some in tears.

The college has arranged for a re-sit on 2 March at no extra charge, and will investigate what went wrong.

Yet it has now emerged that the NSW and ACT Trainees’ Committee wrote to the college in October, expressing registrars’ concerns about the switch from pen and paper.

In a four-page letter, exclusively obtained by Australian Doctor, the committee wrote:

“Candidates would like to be reassured of the process for when a computer error occurs (eg power failure or program malfunction etc.) How will their results be stored, timing impacted and is there an alternative back up if the computer system fails? … Plans for this process need to be put in place prior to the examination so the process is transparent.”

The letter goes on to warn of the fears among registrars about the shift to the online test, which determines whether they progress to advanced training:

“These examinations are the culmination of 10–20 years of education for the majority of physician trainees and the uncertainty of the new testing format is contributing to significant anxiety amongst candidates, some of who are considering delaying training and examinations in the hope that paper based testing returns … The fact that we are trialing a new system in such an important examination is of concern."

In a reply sent six weeks later, the college stressed that because of the growing number of candidates, holding “continuing paper-based examinations on the same day of the year was becoming unsustainable”.

It added: “Computer-based testing will improve the efficacy and security of the exam delivery.”

In response to questions about the credentials of Pearson VUE — the company contracted by the college to hold the test — the college said it had been selected because it had “23 years’ international best-practice experience delivering computer-based testing”.

The college added:

“The Pearson VUE exam delivery system saves the exam with every key stroke. If there is a technical issue, the exam will be saved and the exam timer will stop. If a trainee experiences technical difficulty or disruption during their examination they must notify a test administrator so the appropriate mitigating action can be taken.”

At the start of this year, the college also posted a Youtube video, telling trainees in a webinar that it had developed “a number of contingency plans” in the case of computer failure.

“That is something that we have considered and if there is any issue rest assured that it will be handled swiftly," it said.

The college appears to have deleted the video of the webinar this morning and denies it ignored warnings.


Analysis: The RACP's exam fail raises fresh questions 


“There were significant checks and balances in place with our provider, Pearson Vue, and testing included delivery in a production environment,” it said in a statement.

“We have had trainees participate in sample testing since August 2017.”

The college also said it had developed a paper-based test for the day in the event of a problem, but did not explain why it was not deployed.

“The paper-based test was prepared as a contingency depending on the nature of the disruption. The questions are as rigorous as the questions that were scheduled for the computer-based test and have been validated.”

Candidates re-sit the exam on 2 March.

“We are using a trusted provider who has previously hosted the exam and RACP staff will be on site,” the college added.

An initial report into the causes of the glitch is expected to be handed down before the end of the week.


Read the full letter:

letter 2

letter 3

 

letter 4

Related:

Registrars in tears as tech meltdown forces abandonment of five-hour exam
 

Latest

Trending