Improving assessment

With Diagnostic Classification Models

W. Jake Thompson, Ph.D.

Welcome!

Who am I?

W. Jake Thompson, Ph.D.

Assistant Director of Psychometrics
- ATLAS | University of Kansas
Research: Applications of diagnostic psychometric models
- Lead psychometrician and Co-PI for the Dynamic Learnings Maps assessments
- PI for an IES-funded project to develop software for diagnostic models

@wjakethompson
@wjakethompson.com

Materials

All materials are available on the workshop website:

https://learn.r-dcm.org

Installation

Required
- R (≥ 4.5.0)
- rstan (≥ 2.32.7)
- measr (≥ 2.0.1)
Recommended
- RStudio (≥ 2026.01.1-403)
- cmdstanr (≥ 0.9.0)
- CmdStan (≥ 2.38.0)

Copy and run

install.packages(c("measr", "tidyverse", "here", "fs", "usethis"))

# Optional
install.packages(
  c("rstan", "StanHeaders", "cmdstanr"),
  repos = c("https://stan-dev.r-universe.dev", getOption("repos"))
)

## check toolchain
cmdstanr::check_cmdstan_toolchain()
cmdstanr::install_cmdstan(cores = 2)

For help installing RStan, CmdStanR, or configuring the toolchain, see the prework page (https://learn.r-dcm.org/materials/prework).

Diagnostic assessments

Traditional assessments and psychometric models measure an overall skill or ability
Assume a continuous latent trait

A normal distribution with images of Taylor Swift from each era overlayed.

The output is a weak ordering due to error in estimates
- Confident Taylor Swift (debut) is the worst
- Not confident on ordering toward the middle of the distribution

A normal distribution with images of Taylor Swift from each era overlayed.

Limited in the types of questions that can be answered.
- Why is Taylor Swift (debut) so low?
- What aspects do each era demonstrate proficiency or competency of?
- How much skill is “enough” to be competent?

A normal distribution with images of Taylor Swift from each era overlayed.

Diagnostic measurement

Designed to be multidimensional
No continuum of student achievement
Categorical constructs
- Usually binary (e.g., master/nonmaster, proficient/not proficient)
Several different names in the literature
- Diagnostic classification models (DCMs)
- Cognitive diagnostic models (CDMs)
- Skills assessment models
- Latent response models
- Restricted latent class models

Diagnostic music assessment

Rather than measuring overall musical knowledge, we can break music down into set of skills or attributes
- Songwriting
- Production
- Vocals

Three circles representing the 3 attributes. The bottom half of each circle is shaded dark, and the top half is light, to indicate there are two categories for each attribute.

Attributes are categorical, often dichotomous (e.g., proficient vs. non-proficient)

Diagnostic classification models

DCMs place individuals into groups according to proficiency of multiple attributes
Students are probabilistically placed into classes
- Classes are represented by skill profiles
Feedback on specific skills as defined by the cognitive theory and test design
No scale, no overall “ability”

	songwriting	production	vocals

Benefits of DCMs

Fine-grained, multidimensional results. Answer more questions:
- Why is Taylor Swift (debut) so low?
  - Subpar songwriting, production, and vocals
- What aspects are albums competent/proficient in?
  - DCMs provide classifications directly
High reliability with fewer items
- Less information need to classify than to place precisely along a scale

	songwriting	production	vocals

Fine-grained feedback

Distinguish between respondents who may have similar scale scores

	songwriting	production	vocals