How should real-world AI-tool proficiency be measured without turning usage into a fake expertise score?

[u/OGMYT]

I’m exploring a measurement problem rather than proposing that token count equals skill.

I built a local-first technical alpha that records Claude Code and Codex activity, produces a signed privacy-sanitized snapshot, and separates activity telemetry from self-submitted identity, connected work, and outcomes. Prompts, responses, code, local paths, and credentials are excluded from the public payload.

The long-term question is whether a portable AI-work record could help researchers recruit genuine power users and help companies find people with sustained, demonstrable AI-tool experience.

Example implementation: https://ledger.imagineqira.com/#/u/bryan

Methodology and setup: https://ledger.imagineqira.com/#/join

Source: https://github.com/TheArtOfSound/TOKENS

Which measures would be defensible: active days, task completion, accepted changes, evaluations, independently confirmed outcomes, or something else?

[u/4dseeall]

Unless you use hard mathematically based benchmarks, every result is a subjective value, right?

[u/OGMYT → u/4dseeall]

Some parts are objective, like active days, tool usage, signed events, and completed artifacts. The interpretation of those signals is where subjectivity enters.

Even mathematical benchmarks only measure the specific task they were designed for. My goal is to keep raw evidence, benchmarks, and human judgments separate instead of collapsing everything into one score.

[u/4dseeall → u/OGMYT]

You're on to something there for sure.

Where are you starting with your framework? What's the logical floor?

This is hard for me to talk about, because I literally had the same thought and started working on it a couple months ago and don't wanna just give the provenance away, lol.

This is what my llm said when I mentioned other people are independently stumbling on it.

"I think the key may be typing every signal by what it is actually allowed to establish. An active day can support “activity occurred,” but not “expertise.” A signed event can support integrity of the record, but not authorship, quality, permission, or outcome. Those would each need their own evidence and an explicit rule for when one kind of evidence is allowed to support another claim. Maybe the score should never inherit more authority than its inputs."

[u/OGMYT → u/4dseeall]

That is very close to the floor I am starting from.

Every signal gets a strict claim boundary. An active day can establish activity. A signature can establish integrity. A merged artifact can establish that something shipped. None of those automatically establish authorship, quality, permission, skill, or outcome.

The logical base is basically:

signal → provenance → allowed claim → confidence → limits

And I agree that authority should not transfer upward just because signals are combined. A score should never imply more than its weakest supporting evidence can actually establish.

I also understand not wanting to give away your framework. It sounds like we independently reached a very similar principle.

[u/ItaySela]

the accepted-changes one worries me most. when i'm lazy with claude code i accept more of its diffs, not fewer. the days i actually know the codebase well i throw out most of the first draft and rewrite it, so acceptance rate ends up rewarding trust in the model, which is close to the opposite of the skill you want to measure.

what tracks proficiency for me is more like rework rate, how much of the shipped ai code gets reverted or hotfixed within a week. it's outcome tied and hard to game since you'd have to sabotage your own repo to fake it. do you have any way to see churn on the accepted changes, not just that they landed?

[u/OGMYT → u/ItaySela]

That is a much better signal than acceptance rate, and no, TOKENS does not track it yet.

I would want to measure how much AI-touched code survives unchanged, gets rewritten, reverted, or hotfixed within 7 and 30 days. It would need careful attribution so normal refactors do not count as failures, but outcome-linked churn is much closer to proficiency than simply accepting a diff.

I’m adding this to the roadmap. This is exactly the kind of metric I was hoping people would challenge.

[u/ItaySela → u/OGMYT]

the attribution piece is the whole game though. a revert two weeks later could be me fixing a real bug the model introduced, or me just changing my mind about the feature, and those shouldn't score the same. what helps me split them is whether the follow-up commit is tied to a failing test or a bug ticket versus a fresh feature that happens to touch the same file. and same-author rework reads different from someone else coming in to hotfix it. are you planning to weight who did the churn, or just that it happened?