The Test That Fails the Candidate
- Paul Hobin
- Nov 7, 2025
- 6 min read
Updated: Dec 7, 2025
After being involuntarily retired at 60 because I acquired gray hair, I’m a little testy about corporate America – and I no longer have to play nice in the sandbox. If my writing comes across as sour grapes, that’s because it is. I’ve earned the right; that doesn’t invalidate the message.
I was given an Excel test during an interview, an analysis of a small dataset. I had 20 minutes and I knew I failed.
My failure (or the success of another applicant) was irrelevant to the employer’s needs and the test told them nothing useful. In fact, it may have resulted in a choice worse than picking a name out of a hat.
Follow me through a small thought experiment. We’ll divide Excel analysis into categories, for example pivot tables, plain numeric formulas, complex numeric formulas, array formulas. Someone who knows every feature of Excel could probably list a dozen. We’ll rank the skills from most commonly used, number 1, to least commonly used, number 10.
Let’s say the test I took required skills that correspond to categories 1, 2 and 7 on this imaginary scale. Only candidates with those skills would be able to complete the test within 20 minutes and pass. What do the results tell us?
Nothing. Because the test rests on multiple false assumptions:
Having working knowledge of one specific higher-level skill in Excel (skill 7) indicates broad Excel competence.
That skill predicts the ability to acquire others.
Dictating how employees must solve a problem produces the best solution.
The skill required by the test is useful in similar real world situations.
Recency of usage indicates quality of knowledge.
Assumptions 1-3 are, in my view, self-evidently wrong. Assumption 4 is addressed below. Assumption 5 required more work.
The only Excel skill I know that could achieve the test objective in 20 minutes is pivot tables. I had used pivot tables but had forgotten the mechanics. (In the real world they often fail because data is rarely clean enough, and they’re not part of my regular toolset.) Forgetting seems like a reasonable basis for disqualifying a candidate – but is it?
Forgetting is universally experienced and extensively studied; we can stipulate it as fact without citing authority. Everyone forgets; recent and non-recent users of a skill will both suffer skill loss over any length of time starting with one day. I was interested in why we forget and whether there is evidence relevant to assumption 5: the interview test measured recency, not competence. Put another way: is the test meaningfully different from asking, “When did you last use pivot tables frequently?” If the answer to both is a time interval, the test and the question are equivalent – and equally irrelevant.
My sources are meta-analyses; no primary research was performed. My quotes and summations may therefore be attributable back to the authors of the original primary research.
Arthur¹ concluded that “the single most important determinant of both skill and knowledge retention appears to be the amount or degree of overlearning” (p 59). “Overlearning” is repetition of a skill beyond the point needed to demonstrate proficiency.
Related to overlearning, using pivot tables is an example of an “open-loop” skill. “Closed-loop” is a skill like assembling an IKEA bookcase. It has a defined start, (opening the box), and a defined end, (completing the last item in the assembly instructions). All other skills are open-loop (Arthur, pp 60-61). Using pivot tables in an office environment is open-loop: the user may work on a pivot table intermittently for hours or days, interrupted by other tasks. They may update its structure next month. Being a pivot table user has no discrete endpoint and is therefore open-loop. Open-loop skills are more likely to be overlearned than closed-loop and these two factors reinforce each other, increasing skill retention for regular users.
Cognitive skills like pivot table use degrade faster than physical skills (Arthur, p80), reinforcing the argument that the test measures recency of use. Recent and non-recent users are both effected by cognitive skill decline, but the recent, frequent user’s retention is aided by overlearning coupled with open-loop. The non-recent user doesn’t receive that benefit. Regardless of past and potential future skills levels for each, the recent user’s score is predictably higher. The outcome of the test is the same as the question “When did you last use pivot tables frequently?” As is the result: equal irrelevance.
I believe in making an argument, not inventing one. I must therefore acknowledge that while the study of learning is mature, the study of forgetting that learning is not, and the methodologies have weaknesses. Tatel was unconvinced about overlearning, saying, “Arthur et al. (1998) found little evidence for an effect of overlearning on skill retention, although the majority of the studies they analyzed (87%) did not contain necessary information to determine the degree of overlearning” (p 711). In other words, the Arthur finding is suggestive, but the research has not achieved a level of sophistication warranting firm conclusions.² The two analyses also produced different conclusions on open- vs closed-loop skill retention. Arthur found less retention for open-loop (pp 80, 85-86), Tatel found more (p 711).
Combining the findings, a cognitive, open-loop, overlearned skill like use of pivot tables is weighted in favor of the regular user. That’s not a controversial statement; it “makes sense,” but the meta-analyses reveal two additional, more important truths: 1-The test measures recency. 2-More fundamentally it demonstrates the nature of pivot table use with respect to knowledge retention. In other words, it measures the degree to which the candidate’s prior job environment aligned with the skill retention variables of pivot table use.³ That’s all – it didn’t measure the candidate’s capability.
I’ve used Excel for 30 years. I’ve employed its most abstruse cell locator functions like INDEX and INDIRECT, used array formulas which few Excel users know exist,⁴ live-linked workbooks and database tables to auto-update workbooks without user intervention, and I wrote a 5,000 line VBA add-in that replaced one full-time equivalent of labor. I knew where to find Microsoft’s Office and VBA documentation before AI search made everything easy, and I could find the specifications for any function, formula or VBA element in one minute. (Real documentation, not bad advice from mostly-wrong user forums.)
That’s capability. That’s what (and who) the test failed because it didn’t evaluate capability.
The test was created as an easily measurable pass/fail check box because measurements that matter are difficult and take time. We’ve fallen for the Indeed.com exhortations that “Running a business means I can’t waste time, especially when I’m hiring.”⁵ (When your company grows to 100 employees and they do almost all the work, you don’t want to have “wasted time” choosing them, right?)
How fast can the person selected by the skill test learn when requirements change in six months – which they will? Will they welcome new knowledge, or do they resist change? Can they teach others and strengthen the team? Do they have the expansive, interactive point of view that knows what’s happening across the department and its interactions with other corporate functions, identifying ways their expertise can be put to use and create more value?
If interviews don’t target those qualities the questions aren’t answered and the hiring process is unlikely to select a candidate having them. My interview test measured nothing of value because the test design assumed a single solution – and then measured factors influencing the solution, not performance of the solution by the candidate.
The test was invalid and its candidate selection was invalid. The employer was relying on luck, not evidence.
¹ Research assistance was provided by ChatGPT, which identified the sources:
Arthur Jr et al., Factors That Influence Skill Decay and Retention (1998)
Tatel et al., Procedural skill retention and decay: A meta-analytic review (2023)
² Skill decay research is complex, with many interacting factors. The field has not settled on a set of methodologies to isolate the factors, test each individually, and document a common, consistent set of parameters. This is not to say the experimental results are unreliable – any result an author presents with a full account of the experimental method with its strengths and weaknesses is reliable in the context of those limitations. The problem is the weaknesses are sufficient to create gaps in what can be stated confidently. There is much room for new knowledge to upend what we currently believe, but until that happens the argument of this essay rests on the best experimental evidence available.
³ Statements 1 and 2 are true and show that the test results are unrelated to any measure of applicant quality other than how recently they used pivot tables. (Which I argue is not a measure of quality at all.) But in the real world, while remaining true, they are moot. The relationship between the earlier learning and the later performance is confounded not just by factors which the meta-analyses discuss, but other, unknown factors introduced by the candidate’s experiences in the interim, which no experimenter is monitoring and accounting for.
⁴ Surveys show most users know basic formulas and a few common functions. Neither the author nor ChatGPT were able to find material that broke out numbers on usage or awareness of array formulas. That few Excel users know they exist is inferred from low user adoption of advanced features overall.
Comments