Using AI in Psychometrics
AI psychometrics is the application of artificial intelligence to the design, development, administration, scoring, interpretation and validation of psychological assessments.
The work collected here explores how AI can contribute across the psychometric process: clarifying constructs, generating and reviewing items, proposing provisional estimates of item difficulty and discrimination, assembling adaptive tests, interpreting results and helping to design subsequent human validation studies.
It builds upon a long tradition of psychological test development, including my work with the WISC, WPPSI, British Ability Scales and Orpheus Business Personality Inventory. The statistical foundations remain those of psychometrics: reliability, validity, Item Response Theory, standardisation and evidence from human respondents.
Artificial Intelligence does not remove the need for these methods. It changes what may be possible before, during and between their conventional stages. Recent generative and agentic AI systems can now do more than produce isolated test questions. They can contribute differentiated forms of reasoning, examine the assumptions behind an item, identify possible ambiguity or unintended solution paths, propose provisional psychometric models and translate those ideas into working computer-adaptive assessments.
The central question is therefore not whether AI can replace the psychometrician. It is how human expertise, artificial intelligence and empirical evidence can be combined to develop assessments more rapidly, transparently and intelligently. The projects below trace this development from the conceptual architecture of differentiated AI collaboration to working demonstrations of Item Response Theory and computer-adaptive testing.
AI-Assisted Test Development
This working chapter examines how generative AI may contribute to psychological test development, from construct definition and item writing to review, scoring, calibration and validation.
AI-Supported Statistical Inference
Exploring how model-informed priors, curriculum evidence and targeted human data can support earlier psychometric calibration while preserving validity, fairness and empirical accountability.
Persona Pod Framework
The Persona Pod Framework is a psychologically informed architecture for differentiated AI collaboration. One generative model should not be expected to contribute to all aspects
AI supported Computer Adaptive Testing
This section brings together my work on Item Response Theory (IRT) and Computer Adaptive Testing (CAT) in the context of AI psychometrics. It includes a working CAT demonstration, a chapter on AI applications in IRT, and a discussion of why the persona pod framework is especially relevant to the development of an adaptive matrices test. Together, these materials show how AI may contribute not only to item calibration and test design, but also to the more flexible and intelligent delivery of assessment.
CAT Demo
This interactive demonstration shows how a computer adaptive test selects progressively informative matrix items, estimates ability in real time, and explains the resulting score clearly online.
IRT and Computer Adaptive Testing
Item Response Theory provides a formal relationship between a person’s position on an underlying ability or trait and the probability of responding correctly to a particular item.
Adaptive Matrix Reasoning
Using the Persona Pod Framework, different AI perspectives contribute to item generation, difficulty estimation, ambiguity detection, solution-path analysis and a coherent adaptive item sequence.