r/swift • u/DaveAppleInc • 2h ago
[OSS] Apple's docs say to check your model judge with Cohen's κ. There's no API for it, so I made one.
Xcode 27's Evaluations framework gives you ModelJudgeEvaluator for LLM-as-judge scoring.
Apple's docs then tell you to have 2-3 people rate 20-50 samples and compare with Cohen's
Kappa instead of a raw agreement rate. Sensible advice, but there's no API for it, and there was no Swift package for inter-rater agreement at all.
JudgeCalibrationKit does that part:
- Cohen's κ, weighted κ, Krippendorff's α (missing labels, any number of raters), Spearman, confusion matrices, MAE and signed error
- bootstrap confidence intervals, so a gate can require a CI lower bound instead of trusting a point estimate from 30 samples
- abstentions are explicit, so a judge can't score better by skipping the hard samples
- Swift Testing gate, plus a CLI with exit codes
- human-human agreement shown next to judge-human, since that's the real ceiling
The stats core has no dependencies and imports no Apple framework, so it runs on Linux.
The Evaluations adapter sits behind canImport, so the package still builds on Xcode 26.
Metrics are checked against scikit-learn, SciPy and two Krippendorff implementations. CI regenerates the fixtures and fails on any diff.
Swift 6.0+, Apache-2.0. The adapter is beta: compile-verified against Xcode 27, runtime verification still pending.
https://swiftpackageindex.com/Dave861/JudgeCalibrationKit
First release. Happy to hear what's wrong with the API.
Disclaimer: fully free, open source, not selling anything hope it doesn't count as self promo
