You’re Producing More—but Are You Actually Getting Better?

The diagnostic starts with unsupported performance

The strongest signal of skill growth is improved performance on a relevant task with less or appropriately different support, especially after a delay or change of context. More pages, research notes, or promotional assets show production, not necessarily learning. Output may rise because templates improved, tasks became easier, an editor intervened, or time increased. Those are valuable system gains, but they should not be mislabeled as a change in the learner’s capability.

Create two records: one for operating output and one for learning evidence. The learning record should show the target behavior, task conditions, assistance, observable errors, correction, and transfer. How People Learn II stresses the roles of prior knowledge, context, feedback, and metacognition, which is why a single volume trend cannot isolate learning. Diagnosis requires competing explanations and a next observation that distinguishes them.

Evidence: National Academies of Sciences, Engineering, and Medicine

Leading signs show whether a learning mechanism exists

Before retained performance can be observed, look for leading signs: a behavioral target is written, the first attempt is preserved, feedback points to evidence, the learner selects a correction, and a second attempt occurs. The learner’s prediction should also become better calibrated over time. These signals show that a practice loop is operating, but they do not prove that performance will transfer.

Warning signs include targets that change every session, feedback delivered after the relevant work is finished, revisions completed entirely by the reviewer, and scores without anchored examples. A busy practice log can conceal these failures. CMU’s active-learning guidance supports making thinking visible through explanation, peer review, and self-assessment; use those techniques as observations of reasoning, not as attendance markers.

• Target behavior remains stable long enough to retest.
• Feedback cites a trace and arrives before the next attempt.
• Learner makes the correction rather than receiving a replacement.
• Prediction and actual difficulty are compared.

Evidence: National Academies of Sciences, Engineering, and Medicine; Carnegie Mellon University Eberly Center

Lagging evidence must survive a changed context

Compare a delayed task with the baseline using the same behavioral anchors. Change the topic, audience, source set, or channel so memorized wording is less useful. Record whether the behavior appears, how consistently, and with what prompts. If correctness holds only on the rehearsed example, the likely explanation is task familiarity. If it appears on a new task but disappears under ordinary time pressure, fluency or workflow integration may be the remaining issue.

CAST’s guidelines are important when interpreting performance across contexts. A format change can create an access barrier unrelated to the target capability. Preserve needed assistive options and separate the mode of expression from the criterion whenever possible. Improved access may itself raise performance; that should be credited as better system design rather than forced into an individual-skill story.

Evidence: National Academies of Sciences, Engineering, and Medicine; CAST

Use a five-cause evidence tree

Branch one is task drift: later work was easier or scored differently. Branch two is support substitution: a person, template, or tool supplied the critical reasoning. Branch three is familiarity: the learner remembers the example but cannot generalize. Branch four is delayed feedback: the advice cannot influence the next relevant attempt. Branch five is structural constraint: workload, missing information, or unclear authority prevents use of a skill the learner possesses.

For each branch, write an expected trace and a contradiction. Compare matched tasks to test drift; remove one nonessential prompt to inspect support; use a new domain to test familiarity; timestamp feedback and attempts; or observe the workflow to locate a structural constraint. Do not remove accommodations or create avoidable risk in pursuit of a “clean” test. Choose the least intrusive observation that can separate the two leading explanations.

Evidence: National Academies of Sciences, Engineering, and Medicine; CAST; Scrum Guides

Illustrative diagnosis: the faster research brief

Imagine a writer whose weekly briefs double while source-selection errors remain hidden. One explanation is improved skill; another is a new template that supplies most of the structure; a third is that the briefs cover familiar topics. A useful observation is a novel brief with the template available but source-role decisions left blank. If the writer explains and selects appropriate evidence independently, the template and skill may be complementary. If not, volume did not reveal the gap.

This scenario is constructed and does not report TenMultigure testing or prove a causal method. A single novel task can expose a failure mode but cannot establish durable ability. Repeat comparable observations, preserve contradictory cases, and examine whether the performance benefits live work. The Scrum Guide’s inspection-and-adaptation pattern helps frame the review decision, not the validity of the learning measure.

Evidence: Carnegie Mellon University Eberly Center; Scrum Guides

Choose the next observation, not the preferred repair

If task drift is plausible, create matched examples before assigning more training. If support substitution dominates, decide whether independent performance is actually necessary; a reliable tool-supported workflow may be the better outcome. If familiarity is the issue, vary surface features and delay the cue. If feedback is late, shorten the loop. If a structural constraint prevents use, repair the environment instead of blaming motivation.

Record the diagnosis in a practice–feedback–transfer log with confidence and a revisit date. Success means one explanation is better supported and the next decision is clearer, including a decision to retain useful support. The log does not certify expertise or replace qualified assessment. Review by 2027-02-10 or sooner when the task, rubric, tool, role, or accessibility context changes.

Evidence: National Academies of Sciences, Engineering, and Medicine; CAST