can we trust decision models for evals? @HamelHusain and I tested top decision models like Jev, d1-ultra, clef, pplx-decider, openai decisions, etc. on our dataset we explored failure detection, probability calibration, threshold optimization, and the practical tradeoffs of
Can decision models be trusted for evals? Benchmarks put to test
A researcher and Hamel Husain tested top decision models like Jev, d1-ultra, clef, and pplx-decider on their dataset, examining failure detection, probability calibration, and threshold tradeoffs.