Elements of Test Construction, Administration and Analysis
In addition to the preparation of a test blueprint and the skillful construction of test items that correspond to it, the final appearance of the test and the way in which it is administered can affect the validity of the test results. A haphazard arrangement of test items, directions that are confusing and typographical and other errors on the test may contribute to measurement errors.
By following certain design rules, teachers can avoid such errors when assembling a test. Administering a test is usually the simplest phase of the testing process. There are some common problems associated with test administration, however, that may also affect the reliability of the resulting test scores and consequently the validity of inferences made about those scores. Careful planning can help the teacher avoid or minimize such difficulties.
Test Design Rules
Allow Enough Time
Assembling the test is not simply a clerical or technical task; the teacher should make all decisions about the arrangement of test elements and the final appearance of the test even if someone else types or reproduces the test. The teacher must allow enough time for this phase to avoid errors that could affect the students’ test scores.
Arrange Test Items in a Logical Sequence
Various methods for arranging items on the test have been recommended, including by order of difficulty and according to the sequence in which the content was taught. However, if the test contains items of two or more formats, the teacher should first group items of the same format together.
Because each item format requires different tasks of the student, this type of arrangement makes it easier for students to maintain the mental set required to answer each type of item, and prevents errors caused by frequent changing of tasks.
Keeping items of the same format together also requires fewer sets of directions and facilitates scoring if a scannable answer sheet is not used ( Kubiszyn & Borich , 2003; Miller, Linn, & Gronlund , 2009). Miller et al. (2009) recommended arranging sections of item types in the following order, from simplest to most complex:
- True–false items,
- Matching exercises,
- Short-answer or completion items,
- Multiple-choice items,
- Context-dependent or interpretive exercises,
- Restricted-response essay items, and
- Extended-response essay items (p. 341).
Constructing a test with all of the above-listed item types is not recommended, even for a test with a large sample of items administered to a highly skilled group of learners. The longer the test, the more item formats can be included, but complex formats require more reading and processing time for the student, so they should be combined with only one or two other types.
Next, within each item format, items may be arranged according to the order in which the content was taught, which may assist students in recalling information more easily. Finally, combining the item format and content-sequence grouping, teachers should arrange items in order of increasing difficulty. Even well-prepared students are likely to be somewhat anxious at the beginning of a test, and encountering difficult items may increase their anxiety and interfere with their optimal performance.
Beginning with easier items may build the students’ confidence and allow them to answer these items quickly and reserve more time for difficult items. By having confidence in their ability to answer the beginning items correctly, students may have less anxiety about the remainder of the test ( Kubiszyn & Borich , 2003; Miller et al., 2009).
Write Directions
The teacher cannot assume that the students know the basis on which they are to select or provide answers or how and where to record their answers to test items. Depending on the level of students and their familiarity with the type of items and assessment procedures, it is not reasonable to expect that the assessment will be self-explanatory.
This is especially true with students for whom English is a second language or for those whose primary and secondary education occurred in countries where objectively scored item formats are less common. The test should begin with a set of clear general directions. These general directions should include instructions on:
■ how and where to record responses,
■ what type of writing implement to use,
■ whether or not students may write on the test booklet,
■ the amount of time allowed,
■ the number of pages and items on the exam,
■ the types and point values of items,
■ whether students may ask questions during the test, and
■ what to do after finishing the exam ( Kubiszyn & Borich , 2003; Miller et al., 2009; Nitko & Brookhart , 2007).
Students may need to know some of these instructions while they are preparing for the test; for instance, whether their answers to items requiring them to supply the names of medications must be spelled accurately to be scored as correct. Each section consisting of a particular item format should begin with specific instructions.
For multiple choice items, the student needs to know whether to select the correct or best response. Directions for completion and essay items should state whether spelling, grammar, punctuation, and organization will be considered in scoring, and the length of the desired response. For computation items, directions should specify the degree of precision required, the unit of measure, whether to show the calculation work, and what method of computation to use if there is more than one option (Miller et al., 2009).
Matching exercise directions should clearly specify the basis on which the match is to be made ( Kubiszyn & Borich , 2003). An example is: “For each definition in Column A, select the proper term in Column B. Use each letter in Column B only once or not at all.”
Use a Cover Page
The general test directions may be printed on a cover page (Exhibit 9.1). A cover page also serves to keep the test items hidden from view during the distribution of the exam so that the first students to receive the test will not have more time to complete it than students who receive their copies later. If the directions on the cover page indicate the number of pages and items, the students can quickly check their test booklets for completeness and correct sequence of pages.
The teacher can then replace defective test booklets before students begin answering items ( Gaberson , 1996). When a separate answer sheet is used, the cover page may be numbered to help maintain test security; students are directed to record this number in a particular place on the answer sheet. With this system, the teacher can track any missing test booklets after the test is done.
Additionally, if the teacher asks students to record responses to short answer or essay items directly on the test booklet, those answers can be scored anonymously; the score from the answer sheet then can be added to the score from the supply-type items for a total test score that is associated with each student’s name.
Avoid Crowding
Test items are difficult to read when they are crowded together on the page; learning-disabled students and those for whom English is a second language may find crowding particularly trying. Techniques that allow students to read efficiently and to prevent errors in recording their responses include leaving sufficient white space within and between items and indenting certain elements.
Teachers should allow enough blank space between and around items so that each item is distinct from the others. If not, the students might inadvertently read a line from a preceding or following item and think it belongs to the item they are answering. Tightly packing words on a page may minimize the amount of paper used for testing, but facilitating maximum student performance on a test is worth a small additional expense for a few more sheets of paper ( Kubiszyn & Borich , 2003; Miller et al., 2009 ).
Optimum spacing varies for each item format. The response options for a multiple-choice item should not be printed in tandem fashion, as the following example illustrates:
1. Which method of anesthesia involves injection of an agent into a nerve bundle that supplies the operative site?
A. General
B.Local
C. Regional
D. Spinal
E. Topical
The options are much easier to read if listed in a single column below the stem (Miller et al., 2009), as in this example:
1. Which method of anesthesia involves injection of an agent into a nerve bundle that supplies the operational site?
A. General
B. Local
C. Regional
D. Spinal
E. Topical
Notice in this example that the second line of the stem is indented to the same position as the first line and that the responses are slightly indented. This spacing makes the item number and its content easier to read.
Keep Related Material
Together the stem of a multiple-choice item and all related answers should appear on the same page. Both columns of a matching exercise should also be printed side by side and on one page, including the related directions; using short lists of premises and responses makes this arrangement easier.
With context-dependent and interpretive exercises, the introductory material and all related items should be contained on the same page, if possible. This facilitates reading the material and related questions (Miller et al., 2009).
Facilitate Scoring
If the test will be scored by hand, the layout of the test or the answer sheet should facilitate easy scoring. A separate answer sheet can be constructed to permit rapid scoring by comparing student responses to an answer key. If the students record their answers directly on the test booklet, the test items should be arranged with scoring in mind.
For example, a series of true–false items should be organized with columns of Ts and Fs , preferably at the left margin (Miller et al., 2009) so that students only need to circle their responses, as in the following example: TF 1. A stethoscope is required to perform auscultation. TF 2. Physical exam techniques should be performed in the order of least to most intrusive.
TF 3. When using percussion, it is easier to detect a change from dullness to resonance. Circling a letter rather than writing or printing it will prevent misinterpretation of the students’ handwriting. With completion items, printing blank spaces for the answers in tandem, as in the following example, makes scoring difficult:
1. List 3 responsibilities of the circulating nurse during induction of general anesthesia. . Instead, the blanks should be arranged in a column along one side of the page, preferably on the left, as in this example:
1. 1–3. List 3 responsibilities of the
2. Circulating nurse during induction of
3. General anesthesia.
Arrange the correct answers in a random pattern
Many teachers have a tendency to favor certain response positions for the correct or keyed answer to objective test items, for example, to assign the correct response to the A or D position of a multiple-choice item. Some teachers arrange test items so that the correct answers form a pattern that makes scoring easy (eg, TFTF, or ABCD).
Test wise students may use such test characteristics to gain an unfair advantage ( Haladyna , 2004). Response positions should be used with approximately equal frequency; there are several ways to accomplish this. Many item analysis software programs calculate the number of times the keyed response occurs in each position, or the teacher can tally the number of Ts and Fs , or As, Bs , Cs, and Ds on the answer key by hand.
For true–false items, if either true or false statements are found to predominate, some items may be rewritten to make the distribution more equal (although it is recommended by some experts to include more false than true items). Haladyna (2004) recommended that the position of the correct answer in multiple-choice items be randomly assigned.
This avoids what Attali and Bar-Hillel (2003) referred to as edge aversion. Edge aversion theory suggests that the correct answer is seldom placed in the first or last option position, giving students a clue to select instead one of the middle options. By randomly assigning the correct answer, the position of the correct answer is used about the same number of times and avoids the “effects of edge aversion” ( Haladyna , 2004, p. 113).
Arrange Options in Logical or Numerical Order
The response alternatives for multiple-choice and matching items should be arranged according to a logical or meaningful order, such as alphabetical or chronological order, or in order of size or degree. This type of arrangement reduces reading time and helps students who know the correct answer to search though the options to find it.
This strategy also tends to randomly distribute the correct answer position, especially on longer tests. When the options are numerical, they should always be in ascending or descending numerical order. This principle can be seen in the example shown in Exhibit 9.2.
Number the Items Consecutively Throughout the Test
Although test items should be grouped according to format, they should be numbered consecutively throughout the test. That is, the teacher should not start each new item format section with item number 1 but continue numbering items in continuous sequence. This numbering system helps students to find items they may have skipped and to avoid making errors when recording their answers, especially when using a separate answer sheet.
Proofread
The goal throughout the preparation and use of assessments is to obtain valid evidence that students have met learning goals. Although validity is a major focus of the planning for a test (eg, through use of a test blueprint), careful assembly and administration of the test will assure that it will function as intended (Miller et al., 2009).
The test items and directions should be free of spelling, punctuation, grammatical, and typing errors. Such defects are a source of measurement errors and can cause confusion and distraction, particularly among students who are anxious ( Haladyna , 2004).
Typographical and similar errors are a problem for any student but more so for non-native English speakers or those who have learning disabilities. Often the test designer does not recognize his or her own errors; another teacher who knows the content may be asked to proofread a copy of the test before it is duplicated.
The spell-check or grammar-check features of a word processing program may not recognize punctuation errors or words that are spelled correctly but used in the wrong context, and they may not always detect structural errors such as giving two test items the same number or two responses the same letter ( Gaberson , 1996).
Prepare an Answer Key
Whether the test will be machine-scored or hand-scored, the teacher should prepare and verify an answer key in advance to facilitate efficient scoring and to provide a final check on the accuracy of the test items. Scannable answer sheets also can be used for hand scoring; an answer key can be produced by punching holes to indicate the correct answers.
The teacher also should prepare ideal responses to essay items, identify intended responses to completion items, and make decisions regarding the point values of required answer elements if the analytical scoring method is used.
Reproducing The Test
Assure Legibility
Legibility is an important consideration when printing and duplicating the test; poor-quality copies may interfere with optimal student performance. A font that includes only upper-case letters is difficult to read; upper- and lower-case lettering is recommended.
The master or original copy should be letter-quality, produced with a laser or other high-quality printer so that it can be clearly reproduced. For best results, the test should be photocopied or printed on a machine that has sufficient toner to produce crisp, dark print without any stray lines or artifacts.
Print on One Side of the Page
The test should be reproduced on only one side of each sheet of paper. Printing on both sides of each page could cause students to skip items unintentionally or make errors when recording their scores on a separate answer sheet. It also creates distractions from excessive page-turning during the test. If the test is to be hand-scored and students record their answers on the test rather than on a separate answer sheet, printing only on one side makes it easier to score.
Duplicate Enough Copies
The teacher should duplicate more test copies than the number of students to allow for extra copies for proctors or to replace defective copies that may have been inadvertently distributed to students. Displaying test items on a screen from an overhead projector or computer projector, or writing them on the chalkboard or interactive whiteboard, may save costs or the teacher’s preparation time, but these procedures may cause problems for students with learning or visual disabilities.
When students do not have their own copies of a test for whatever reason, they cannot control the pace at which they answer items or return to a previous item. Dictating test items is not recommended except when the objective is to test knowledge of correct spelling; in addition to creating problems for students with hearing impairments, this method wastes time that students could otherwise spend in thinking about and responding to the items. In addition, there is no record of how the items were worded, which could present a problem if a student later questions how an answer was scored.
Maintain Test Security
Teachers have a serious responsibility to maintain the security of tests by protecting them from unauthorized access. Carelessness on the part of the teacher can enable dishonest students to gain access to test materials and use them to obtain higher scores than they deserve. This contributes to measurement errors, and it is unfair to honest students who are well-prepared for the test.
It is up to the teacher to make arrangements to secure the test while it is being prepared, duplicated, stored, administered, and scored. Test materials should be stored locked in areas accessible only to authorized personnel. Computer files that contain test items should be protected with passwords, encryption, or similar security devices. Only regular employees should handle test materials; student employees should not be asked to type, print, or duplicate tests.
While test items are being typed, they should be protected from the view of others by turning the monitor off if an unauthorized individual enters the area. Printed drafts of tests should be destroyed by shredding pages rather than discarding them in trash or recycling receptacles. One suggestion for preventing cheating during test administration to large groups is to prepare alternative forms of the test.
This can be done by presenting the same questions but in a different order on each form. For calculation items the teacher can modify values within the same question on different forms; in that way the responses will not be identical. The same method can be used with online tests. Faculty members can prepare alternative forms of the test for students to complete online.
Software is also available that allows for random sequencing of items on an online exam. The problem with this technique is that a random sequence may not be consistent with principles for ordering items on a test. It also may result in alternative forms of a test that are not equivalent. Similarly, the order of responses to multiple-choice and matching items might be scrambled to produce an alternative form of the test.
However, the psychometric properties of alternative forms produced in these ways might be sufficiently different as to result in different scores, especially when the positions of items with unequal difficulty are switched. If there is little or no evidence for the true equivalence of these alternative forms, it is best not to use this approach.
Test Administration Environmental Conditions
Face-to-Face and Online
The environmental conditions of test administration can be a source of measurement error if they interfere with the students’ performance. If possible, the teacher should select a room that limits potential distractions during the test. For example, if windows must be open for ventilation during warm weather, the students may be distracted by lawn mowing or construction noise; requesting a room on another side of the building may prevent the problem.
Placing a sign such as “Testing— Quiet Please” on the door of the classroom may reduce noise in the hallway. For online courses, it is critical to determine prior to the test administration that students have the computer capabilities and Internet access to take the exam for the time period allotted.
Students with dial-up modems may experience “timing out,” which means being disconnected from the Internet by their Internet Service Providers after a set period of time or what appears to be inactivity on the part of the user. When that occurs, the students cannot transmit their completed exams, and course management systems may not permit them to access another copy.
Distributing the Test Materials
Careful organization allows the teacher to distribute test materials and give instructions to the students efficiently . With large groups of students, several proctors may be needed to assist with this process. If a separate answer sheet is used, it usually can be distributed first, followed by the test booklets.
During distribution of the test booklets, the teacher should instruct students not to turn over the cover page and begin the test until told to do so. At this point, the students should check their test booklets for completeness, and the proctors should replace defective booklets. The teacher then should read the general directions aloud while the students read along.
Hearing the directions may help nonnative English speakers, students with learning disabilities, and students whose anxiety may interfere with their comprehension of the written instructions. Once the teacher answers any questions about the test procedures, the students can begin the test.
Answering Questions During the Test
Some students may find it necessary to ask questions of the teacher during a test, but responding to these questions is always somewhat disturbing to other students. Also, by responding to student questions during a test, a proctor may inadvertently give hints to the correct answer, which would put that student at an advantage while not making the same information available to other students.
Teachers should decide in advance of the test date whether or not to allow questions during a test. If not, it is helpful to instruct students to record any questions on a separate piece of paper that will be collected with the other test materials (identified with student names).
Then if a student identifies a flaw in a test item, the teacher can take the necessary action after the test is completed rather than interrupt the test to announce corrections. If the teacher decides to accept student questions during the test, distraction can be kept to a minimum by telling students to raise their hands if they have questions rather than leaving their seats to approach the teacher; a proctor then goes to each student’s seat.
Proctors should answer questions as quietly and briefly as possible. In answering questions, proctors certainly should address errors in the test copy and ambiguity in directions but should avoid giving clues to the correct answers. When writing articles, teachers should work to eliminate cultural bias and terms that would be unfamiliar to students for whom English is not their native language.
Preventing Cheating
Cheating is widely believed to be common on college campuses in the United States. A recent study found that a “steadily growing number of students cheat or plagiarize in college” ( Hinman , 2004, p. A19). Hinman suggested that when teachers know their students, interact with them about their learning, and give meaningful assignments, they create an environment in which cheating is less likely to occur.
Cheating is defined as any activity whose purpose is to gain a higher score on a test or other academic assignment than a student is likely to earn on the basis of achievement. Cheating on a test includes but is not limited to the following forms:
With adequate test security and good proctoring during the test, the teacher can prevent these opportunities for cheating. Students who do act honestly resent those who cheat, especially if dishonest students are rewarded with high test scores. Honest students also resent faculty members who do not recognize and deal effectively with cheating ( Gaberson , 1997).
Although a number of methods for preventing cheating during a test have been proposed, one effective method is careful proctoring. There should be enough proctors to supervise students adequately during exams; for most groups of students, at least two proctors are suggested so that one is available to leave the room with a student in case of emergency without leaving the remaining students unsupervised (Gaberson , 1996).
When proctoring a test, it is important to be serious about the task and devote full attention to it rather than bringing papers to grade and other materials to work on.
A particularly troublesome situation for teachers is how to deal with a student’s behavior that suggests cheating during a test. Prior to administering the test, the teacher must know the policies of the nursing program and college or university regarding cheating on an examination or another assessment.
If a teacher is certain that a student is cheating, the teacher should quietly collect the test and answer sheet and ask the student to leave the room. However, if it is possible that the teacher’s interpretation of the behavior is incorrect, it may be best not to confront the student at that time.
In addition to preventing a potentially innocent student from completing the test, confiscating test materials and ordering a student to leave will create a distraction to other students that may affect the accuracy of all the students’ test scores. A better response is to continue to observe the student, making eye contact if possible to make the student aware of the teacher’s attention.
If the student was attempting to cheat, this approach usually effectively stops the behavior. If the behavior continues, the teacher should attempt to verify this observation with another proctor, and if both agree, the student may be asked to leave the room (Gaberson , 1997). The appropriate penalty for cheating on a test is a score of zero for that test. The teacher should not just deduct points from the test score or lower the grade in some other way.
By deducting points, it appears as though the student took the exam and achieved a low score on it when that was not the case. If the teacher learns that a copy of a test is circulating in advance of the scheduled date of administration, the teacher should attempt to obtain verifiable evidence that some students have seen it. In this case, the teacher needs to prepare another test or develop a new way of assessing student learning. .
Online Testing
As more courses and programs are offered through distance education, teachers are faced with how to prevent cheating on an assessment when they cannot directly observe the students. There are different approaches that can be used, ranging from administering the tests in a traditional, face-to-face session to using computer-adaptive tests.
Examinations in an online course can be given on campus as done with face-to-face courses or in an on-campus computer-testing facility. Students may be required to take tests within a designated time period, for example, during one specified week, on campus. Some institutions with large online programs have testing facilities throughout the state where nursing students can go to take their tests in a traditional, proctored way.
Another option is to make arrangements for students to take the exam at a library, school, or other facility close to the student’s home, where the exam can be proctored. In these situations, whether on- or off-campus, students must provide proof of identity. One issue with this approach, though, is that it may conflict with the reasons that the student enrolled in an online course, for example, living in a rural area and scheduling conflicts, among others.
With online course management systems, teachers can administer a proctored examination online. The test can be set up to require the proctor to input his or her ID and a password known only to the proctor to start the exam ( McNett , 2002).
The students can only access the test with their IDs at that same time. With these course management systems, the teacher can also limit access to a test to a specific day and time, and can restrict the time allowed to complete the exam, similar to traditional test administration. By offering short tests more frequently and limiting the time for their completion, teachers provide fewer opportunities for students to look up the answers during a test.
Tests that are shorter also help students who use dial-up connections, which may “time out” before the allowable testing time expires. Another method of discouraging cheating is to post a copy of an honor code policy on a web page that precedes the test. After reading the honor code, students can be asked to affirm that they will not use course notes, textbooks, and other resources to complete the test and will not consult with others during the test.
Software is available that allows the teacher to develop a database of questions. Software programs then select items, according to principles set by the faculty member, to include in the test (McNett , 2002). Such software can be used to randomly order items in a test, developing alternative forms of the exam that can be randomly assigned to the students.
An important issue with this approach, as mentioned earlier, is that the psychometric properties of the alternative forms might not be the same. Without knowing if the forms are truly equal, it is best not to use this approach. Software also is available to develop computer-adaptive tests like the NCLEX examinations; the student’s answers determine the subsequent questions.
Collecting Test Materials
For traditional on-site tests, when students are finished with the test and are preparing to leave the room, the resulting confusion and noise can disturb students who are still working. The teacher should plan for efficient collection of test materials to minimize such distractions and to maintain test security. It is important to be certain that no test materials leave the room with the students.
Therefore, teachers should take care to verify that the students turn in their test booklets, answer sheets, scratch paper, and any other test materials. With a large group of students, one proctor may be assigned the task of collecting test materials from each student; This proctor should check the test booklet and answer sheet to assure that the directions for marking answers were followed, that the student’s name (or number) is recorded as directed, and that the student has not omitted any items.
Any such errors can then be corrected before the student leaves the room, and test security will not be compromised. If students are still working near the end of the allotted testing time, the remaining amount of time should be announced, and they should be encouraged to finish as quickly as possible. When the time is up, all students must stop, and the teacher or proctor must collect the rest of the tests.
Students who have not finished the test at that point cannot have additional time unless they have legitimate learning disabilities. In those cases, the testing time may be extended if the student’s learning disability has been confirmed according to college or university policies.
This decision should be made in advance of the test and the necessary arrangements made. Extended testing time is not an appropriate remedy for every learning disability, however. It should be provided only when specifically prescribed based on a psych educational evaluation of a student’s abilities and needs.
Conclusion
The final appearance of a test and the way in which it is administered can affect the validity of the test results. Poor arrangement of test items, confusing or missing directions, typographical errors, and careless administration may contribute to measurement errors. Careful planning can help the teacher to avoid or minimize these difficulties.
Rules for good test design include allowing sufficient time, arranging test items in a logical sequence, writing general and item-format directions, using a cover page, spacing test elements to avoid crowding, keeping related material together, arranging the correct answers in a random or logical pattern, numbering items consecutively throughout the test, proofreading the test, and preparing an accurate answer key.
In preparing to reproduce the test, the teacher should assure legibility, print the test on one side of each page, prepare enough copies for all students and proctors, and maintain the security of test materials. Although administering a test usually is the simplest phase of the testing process, there are some common problems that may affect the reliability of the resulting scores.
Teachers should arrange for favorable environmental conditions, distribute the test materials and give directions efficiently, make appropriate plans for proctoring and answering questions during the test, and collect test materials efficiently.
Strategies were described for administering tests in an online environment, including approaches to prevent cheating. Teachers have an important responsibility to prevent cheating before, during, and after a test, and should respond to verified evidence of cheating with appropriate sanctions.