Writing Test Questions

 

Key Points:

  • Adhere to evidence-based principles of item writing
    • Focused on one topic
    • Positively worded
    • Options are short and of similar length
    • All options are plausible
    • No cues to the answer are present
    • Wording is clear
  • Link test questions to learning objectives
  • Have a rationale for the content being tested
  • Ask yourself whether the test measures what it is supposed to
  • Validity refers to scores being meaningful and defensible.
  • Reliability refers to reproducibility.

 

Objectives:

  • Write test questions that follow evidence-based principles of item writing
  • Avoid common test question flaws: unfocused stem, negative stem, window dressing (extraneous information), inappropriate content, non-functional flaws, cueing effects, partial K-type (choosing several of the given options), all/none of the above
  • Be familiar with the term construct-irrelevant variance (CIV)

 

Recommended Resources: 

Brief Review

Watch

Writing Multiple Choice Test Questions. Nancy Sulla. 13 minutes.
http://youtu.be/irqXYvVnzbk

Multiple Choice Question Writing Workshop. Devshikha Bose. 20 minutes.
https://youtu.be/eVEbaZzzYmY

Read

How to Write a Better Question. A Guide to Construction of NBME-type Questions for the UASOM Pre-Clinical Curriculum. University of Alabama College of Medicine.
http://www.uab.edu/medicine/home/images/education/exam-items/UME_Test_Item_Guidelines.pdf

Question Writing Guide. American Board of Dermatology.
http://www.abderm.org/home/image.ques.guide.pdf

Guidelines for Constructing Multiple-Choice Items. American College of Surgeons.
https://www.facs.org/~/media/files/about%20acs/chapters/guidelines%20test%20item%20writing.ashx

McCoubrie P. Improving the fairness of multiple-choice questions: a literature review. Med Teach. 2004 Dec;26(8):709-12. Review. PubMed PMID: 15763874.
http://www.ncbi.nlm.nih.gov/pubmed/15763874

Boland RJ, Lester NA, Williams E. Writing multiple-choice questions. Acad Psychiatry. 2010 Jul-Aug;34(4):310-6. doi: 10.1176/appi.ap.34.4.310. Review. PubMed PMID: 20576995.
http://www.ncbi.nlm.nih.gov/pubmed/20576995

 Do

♥Writing Multiple Choice Questions: An Introductory Tutorial. NBME. Self-paced interactive tutorial that provides a foundation for writing quality multiple-choice questions. This tutorial is designed to introduce some of the main principles of item writing to individuals who write questions for examinations in the field of medicine and across the range of health professions. The tutorial is organized into sections so that you can complete only a portion of it and easily return later; the entire tutorial takes about 45 minutes to complete.
www.nbme.org/IWTutorial

Resource

Tao. Open source exam software. No usage fees but must be installed by IT, similar to Moodle.
http://www.taotesting.com

 

In-Depth Review

Watch

Creating tests that measure critical thinking in health science professions. ExamSoft. 50 minutes.
https://youtu.be/T_Rohf5PF2g

Best practices in writing test items. Assessment Systems. 27 minutes.
https://youtu.be/AUB0d31-j2M

Do This, Not That: A Survey of Multiple-Choice Question Types (Most of Which Should Be Avoided). ExamSoft. 43 minutes.
https://youtu.be/Ztgkzbxwek0

Teaching and Assessing Critical Thinking Using Assessments with Embedded Outcomes. ExamSoft. 48 minutes.
https://youtu.be/ZY2s2C7QzKQ

Embracing Multimedia in Exam Question Construction and Analysis. ExamSoft. 50 minutes.
https://youtu.be/5PhFkR8fX1U

Writing Great Clicker Questions: Faculty Workshop. Stephanie Chasteen. 48 minutes.
https://youtu.be/2PpKi0nBrVM

Read

♥NMBE Item Writing Manual. Constructing Written Test Questions for the Basic and Clinical Sciences.
http://www.nbme.org/publications/item-writing-manual-download.html

Writing Exam Questions Resources. University of Washington Center for Instructional Development and Research.
http://depts.washington.edu/cidrweb/OLD/resources/exams.html

Do

Item-Writing Seminar for Practitioners and Educators in the Health Professions. Objectives of the annual ½-day seminar include refining your item-writing techniques and enhancing the quality of items in your examinations and programs. The agenda includes a review of item-writing guidelines; examples of "good" and "bad" test items; a dynamic item-writing session; and the opportunity to interact with participants from a range of health professions. NBME Headquarters; 3750 Market Street; Philadelphia PA 19104. $375
http://www.nbme.org/health-profession-services/Resources/Form.aspx

♥NBME U is an online collection of lessons for individuals interested in learning a range of topics relevant to high quality assessment. The lessons are designed to provide brief overviews of assessment topics that are of interest to those in the health professions’ education community at no cost to the learner. http://www.nbme.org/about/nbmeu/

NBME Faculty Development Workshops are offered to groups of medical faculty, educators of healthcare disciplines, and bodies that assess or certify the competence of healthcare providers. Workshops are available globally on a year-round basis and can take place at the school. Workshops include Mastering the MCQ: Guidelines for Writing Multiple-Choice Questions, Critical Themes in Assessment, Case Development for Standardized Patient Based Examinations, Measurement Skills in Clinical Skills Assessment.
http://www.nbme.org/schools/faculty.html