Background
Five years ago, CXC seriously considered changing the way candidate’s scripts were marked. Since 1979 CXC scripts have been marked residentially by question and by table – no examiner marks an entire script – a security measure which has worked well for all the years. Residential marking involves significant costs - air travel, hotel accommodation, catering and per diem costs – in addition to the honoraria for marking. By 2008 CXC was spending in excess of BD$18 million annually on script marking activities. It was agreed that an alternative should be considered.
Justification
It was recognised that e-marking would assist the organization to reduce some costs by eliminating air travel, hotel accommodation and catering – costs that have kept rising exponentially. The Business plan approved by AFC in 2011 showed that when fully implemented at the CSEC level the cost saving would be in the region of BD$3 000 000 and that when CAPE was included the savings would be greater particularly since with fewer markers travelling these costs (travel and hotel accommodation) which had escalated significantly in the last 5 years would be reduced.
Onscreen marking would allow some of the challenges associated with table marking to be addressed at an earlier stage. In the present system, errors made by markers would mean that several folders incorrectly assessed would have to be retrieved manually – a process that may take a day or two. In the new paradigm, when markers make errors, the seeding process will identify them quickly and allow the Examining Committee to suspend the marker and take immediate corrective action.
Reducing costs is not the only reason for moving to electronic marking. Our present process of finding suitable marking accommodation, collecting the examination scripts, having them marked securely in a residential setting has stood the test of time, but must be moved to another level. Onscreen marking reduces the challenges which have been experienced in the last several years in obtaining comfortable marking accommodation and generally in managing the logistics associated with getting markers to the four centres. CXC’s resources will no longer need to be tied up in these areas, but would be transferred to ensuring the accuracy of the script marking.
Software & Preparation
The international examining boards have used onscreen marking for several years. From 2009-2011 CXC reviewed software provided by five suppliers and evaluated different solutions in an effort to determine the best fit for the type of examinations it offered and to determine whether the selected interface could capture and report the profile data – a critical requirement. AFC approved the recommendation to join forces with RM Education Solutions, a UK based company and to use their product RM Assessor (Scoris) as the e-marking solution.
The answer booklets used by candidates at the January 2012 examinations were amended in four subject areas after the examinations and re-administered to the candidates and their performance in these restructured booklets evaluated against their performance at the original sitting. The performance on those redesigned booklets was shown to be similar to the performance on the regular AB1 booklet used at the January sitting. Teachers were also invited to mark electronically the scanned images on those redesigned scripts from their homes and to share their experience with CXC staff.
In 2013 at the July script marking, 2500 Physics and 2500 Integrated Science paper 2 scripts were e-marked in parallel with the usual manual table marking. For each subject the two sets of scores assigned to the candidates were correlated to determine the extent of agreement between them. Of the 12 questions that were marked under the two conditions, correlations between the two sets of scores ranged from 0.85 to 0.98 which suggests a substantial relationship at the individual question level for the two methods.
Scanning and indexing of the scripts is 60 per cent of the e-marking operation and these services are at present, not offered in the Caribbean. The pilot in 2013 allowed for (i) an evaluation of the scanning capability offered by the US company identified to work with CXC and which has considerable experience in the scanning required for e-marking solutions and (ii) for a closer review of the standardizing, seeding and e-marking process with teams of markers of similar composition to those marking manually. At its meeting in October 2013, SUBSEC approved the proposal for eleven (11) papers in seven subject areas to be marked electronically.
Training and Orientation
This activity was undertaken at four levels:
- The in-house stakeholders (staff) participated in an extensive one week session in November 2013. This training was designed to make key staff familiar with the new process;
- Core staff (EAS and EDPD) together with several ISD officers were again involved in a one week orientation called Train the Trainers in March 2014 in preparation for the orientation activities with the markers during the Easter holiday;
- Examining Committees involved in the January marking activities were given the opportunity to participate in a short orientation session during the marking; another online session was also arranged for the Committees to resolve some of the issues they had particularly with respect to the standardizing activities. All the Committees were invited to participate in a face to face session in June with the presence of the software developers RM and to participate in a four day session to be taken through the software solution and to participate in standardising and the preparation and selection of the seeds. This is a critical component of the standardizing process and the sessions had to be carefully structured;
- the marking teams were invited to several orientation sessions some conducted by staff who visited the different territories, as well as online. Over 500 teachers participated in the orientation sessions April – May 2014.
Achievements
At the script marking of the 2014 May-June CSEC entry, 112,000 scripts were marked electronically, 2,290,898 images were captured at the US scanning bureau in Indianapolis and 661 markers were involved. The standardising process began on 18 June and the ended when the Chief Examiners had completed their work – these dates varied by subject. The onscreen marking ended on 18 July. In many instances the markers began very slowly as they became more familiar with the software. However, once they were comfortable they recognised that the software was user-friendly and they proceeded with the process.
|
