Anthropic grants METR employee-level access for independent evals
Anthropic is giving METR, a nonprofit frontier-AI evaluator, employee-like access so it can publish risk assessments, alongside partnerships with SecureBio and Accenture.
Evaluation
All AI Pulse updates tagged "Evaluation".
3 signalsCan decision models be trusted for evals? Benchmarks put to test
A researcher and Hamel Husain tested top decision models like Jev, d1-ultra, clef, and pplx-decider on their dataset, examining failure detection, probability calibration, and threshold tradeoffs.
LangSmith Evals: Agent Quality, Cost, Throughput
Clay evaluates every agent on quality, throughput, and cost. LangChain's Jeff Barg explains how LangSmith evals measure quality and benchmark it against cost and throughput.