It has been a really interesting experience to build an eval suite for System One Models like Jev. My first two attempts didn't work out. But I learned a lot in the process, and since Jev is so fast, and cheap, it's a lot easier to run quick tests than with an LLM. I'm on attempt number three with this eval suite, and thought I'd share more about it. The eval suite is called @VulcanBench Verdict, and here's the high-level on how it works: ❤️ 32 likes on X

@morganlinton

Go SDK and CLI for TypeSafe Jev: typed decisions (yes/no, choice, score) from a model - mattn/go-jev

Tests, calibration audits and failure-mode studies of Jev (TypeSafe System One): jaggedness, consistency, injection, abstention. - Yifan-Lan/awesome-jev-robustness