Jev Does Not Play Dice: 83% probability, 19% accuracy on a hidden fair die roll

2 comments

• kantahayashi 2 hours ago
  Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.

  Code and data are here: https://github.com/KantaHayashiAI/jev-does-not-play-dice

  Happy to answer questions!

  [-]
  • magicalhippo 1 hour ago
    Maybe it watched Rosencrantz & Guildenstern Are Dead[1] once too many.

    [1]: https://www.youtube.com/watch?v=C_TfdNAXOwE

    [-]
    • kantahayashi 1 hour ago
      I found that in the play it's 92 heads in a row [1], and in my coin test Jev returned around 92%. Maybe Jev was actually trained on it.

      [1]: https://en.wikipedia.org/wiki/Rosencrantz_and_Guildenstern_A...
  • __m 2 hours ago
    why would you use ai for that though? how does that translate to other tasks?
    [-]
    • kantahayashi 2 hours ago
      I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.
• verdverm 1 hour ago
  saw this yesterday (when someone else posted your post), thanks for sharing such easy to understand cases which show Jev does not produce real world likelihoods for the choices

  related, https://www.alexmolas.com/2026/09/23/jev-cant-be-calibrated....

  [-]
  • kantahayashi 1 hour ago
    Thanks! I'm really glad that you read my article.