Establishing Cut Scores

Establishing Cut Scores

CASE STUDY 13.1: SETTING A CUTOFF SCORE ON A COMPREHENSIVE EXAMINATION
Alexius, a fifth-year doctoral student, had agreed to sit on the committee that was restructuring the doctoral comprehensive exams for his department. It seemedevery year students complained about the long essays they had to write and professors complained about having to read and grade the essays with little guidance. Therefore, a committee of mostly full professors in the department was formed to explore the possibility of having a new two-part multiple-choice comprehensive exam. The first part would be a 250-question multiple-choice exam covering several general areas (e.g., history and systems, statistics, and research methods). The second part of the test would also have 250 multiple-choice questions but in the student’s area of concentration (e.g., social, cognitive, clinical, or I/O psychology). Thus, the test would consist of 500 multiple-choice questions in all. While a common standard of 80% correct or 90% correct could be used to set the pass point, the committee did not feel that was wise, as they knew the questions on the test would change each year. In addition, while all students in a given year took the same general portion of the test, students in different concentrations took different area-specific tests. Thus, it was felt a new pass point should be established each year for each test segment. Several of the counseling professors on the committee sat on the state licensing board for counseling psychology, and they used a similar procedure to set the pass point for the professional licensing exam for counseling psychologists intheir state for the written multiple-choice portion of the exam.
Because the number of students who took the comprehensive exams in a given year was relatively small (i.e., usually less than 20 students), an empirical strategy for setting the cutoff score did not seem feasible. However, the committee was uncomfortable with using a purely judgmental procedure for setting the cutoff score. In addition, multiple cutoff scores had to be set, one for the general portion of the test and a separate cutoff score for each specific test. The committee chair, who also happened to be the department chairperson, asked Alexius to provide the committee with a proposal of how best to set the cutoff scores for each portion of the test. Alexius felt a bit overwhelmed. Here were all these professors in the department, many who had been there 30 years or more, and they were asking him for recommendations on how to set cutoff scores for the tests. Yes, he had just taken his comprehensive exams the year before, but that was under the old system when you had to write six or eight long essays, not this multiple-choice format. In some ways, he thought this was probably better than having to write all those questions. Therefore, Alexius went back to his notesfrom his applied psychological measurement class and started a literature search on the “best practices” for setting cutoff scores in such situations.
Questions to Ponder
1. If you were Alexius, where would you start your search for “best practices” for setting cutoff scores on a graduate comprehensive examination?
2. While a purely empirical method for setting the cutoff scores seems unrealistic given the small sample sizes, what things could Alexius do to make his judgmental procedures more empirical?
3. Who are the likely SMEs for setting the cutoff scores for the general test? The area-specific tests?
4. Is there a problem with the same individuals writing the questions and also helping to set the cutoff scores on the test they created?
5. Would information from past “pass rates” be of any use to the committee given that the format is being changed?

CASE STUDY 13.2: SETTING A CUTOFF SCORE ON A COLLEGE ENTRANCE EXAM
Lui-Ping (most people just called her Jasmin), a recent master’s graduate, had decided to return home to Malaysia after graduation. After a short job search, she obtained a job with the ministry of education. One of Jasmin’s first assignments was to help set the cutoff score for the national entrance exam for the three most sought-after public universities in Kuala Lumpur (the capital city of Malaysia). The three universities received tens of thousands of applications every year. It was no wonder; anyone who was admitted received free tuition. In addition, the top employers from across Malaysia (all of Southeast Asia, in fact) seemed to focus much of their recruitment effort for new employees at these three top public universities. Therefore, if a student were able to get into one of these three universities, he or she would be “set for life.”
The ministry of education, however, had just recently reformatted the entrance exam to cover several new topics. As a result, a new cutoff score had to be “recommended” to the universities. Ultimately, the universities were free to choose their own cutoff scores, but they relied heavily on the expertise provided by the ministry of education, as that is where much of their funding came from. Therefore, Jasmin was asked to help determine the appropriate cutoff score for the three universities. This could be difficult, she thought, as the three universities seemed to be so different. The first was a technical university, focusing on engineering and the physical sciences. The second university was a more traditional liberal arts university, with a wide breadth of offerings and a much smaller student body. The third university had a strong focus on the professional degrees, with emphases in business, social work, medicine, law, and education. These all seemed so different. How could she set a single cutoff score for all three universities? It was time to sit down with her new boss and get more clarification on what she should do next.
Questions to Ponder
1. Should Jasmin recommend the same cutoff score for each university or should different cutoff scores be recommended?
2. Instead of having one overall cutoff score, might it be better to have separate cutoff scores for different portions of the exam?
3. Should Jasmin take into consideration the other criteria used by each of the universities to select its students? If so, how?
4. Given the large sample of data Jasmin will have to work with, how might she incorporate some empirical data into the cutoff score decision?
5. Who should be the SMEs for Jasmin in helping her to set the cutoff score(s)?

Kenneth S. Shultz;David J. Whitney. Measurement Theory in Action: Case Studies and Exercises (Kindle Locations 2591-2627). Kindle Edition.

ORDER THIS ESSAY HERE NOW AND GET A DISCOUNT !!!

You can place an order similar to this with us. You are assured of an authentic custom paper delivered within the given deadline besides our 24/7 customer support all through.