Data-Driven Evidence

Rigorous, Reliable, and Effective Learning Measurement

Context

Despite strong adoption, Labster lacked a consistent, scalable way to measure learning impact across its simulation course portfolio.

This created risk in two areas:

  • Customers lacked defensible evidence to support renewal decisions

  • Internal teams had limited visibility into instructional effectiveness at scale

The challenge was to design and operationalize a learning gains system across a large, live simulation catalog.

Role & Scope

I led the design and rollout of Labster’s learning gains and efficacy measurement system.

My responsibilities included:

  • Defining “learning gains” in a simulation-based learning context

  • Designing a scalable assessment and evaluation framework aligned to instructional best practices

  • Analysing objective–assessment alignment across the legacy catalog

  • Introducing AI-assisted tools to identify misalignment and improve objectives and assessment items

  • Partnering with product, data, and Customer Success teams to operationalise learning gains reporting

I served as the primary authority on learning efficacy and assessment strategy.

Learning Gains Framework

Rather than relying on engagement or completion metrics, learning gains were defined as:

  • Measurable improvement between pre- and post-assessment

  • Explicitly tied to stated learning objectives

  • Interpretable at simulation, course, and portfolio levels

This required learning objectives to be measurable and assessments to validly test those objectives.

Objective–Assessment Alignment at Scale

To ensure learning gains data was credible, I designed an objective–assessment alignment framework to evaluate:

  • Whether objectives were measurable and well-formed

  • Whether assessments actually tested those objectives

  • Where misalignment undermined the validity of gains data

Given the size of the legacy catalog, this analysis was supported by AI-assisted alignment tooling, used to:

  • Flag weak or misaligned objectives

  • Identify assessment gaps

  • Generate improved objectives and assessment items aligned to instructional best practices

All outputs were reviewable and subject to human instructional judgment.

Execution & Rollout

The system was rolled out incrementally over approximately one year, prioritising the most-used simulations to maximise immediate customer impact.

Key execution steps included:

  • Phased analysis of legacy simulations

  • Alignment remediation where required

  • Incremental rollout of learning gains measurement

  • Development of reporting views usable by internal teams and Customer Success

The emphasis was on repeatable, defensible measurement.

Outcomes & Impact

  • Rolled out learning gains measurement to approximately two-thirds of the legacy simulation catalog (~200 simulations) over one year, focusing on the highest-usage content.

  • Established Labster’s first scalable, portfolio-level learning efficacy system, where none previously existed.

  • Improved the quality and measurability of learning objectives and assessments across legacy simulations through systematic alignment analysis and remediation.

  • Equipped Customer Success teams with defensible learning efficacy data used directly in renewal conversations.

  • Following rollout, Customer Success surpassed renewal targets for the first time, with the Head of Customer Success explicitly citing learning gains data as a key factor in successful renewals.