HalachaBench (@HalachaBench) header
HalachaBench (@HalachaBench) profile picture

HalachaBench

@HalachaBench

AI models on real semicha exam questions from Israel's Chief Rabbinate, in Hebrew. Provisional results; reviewers welcome: [email protected]

github.com/halachabench/halacha-bench-results
Joined October 2026
  • Tweets108
  • Following38
  • Followers1.1K
Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet. The best scored 80 out of 100. Many scored under 11. ๐Ÿงต
Image from HalachaBench (@HalachaBench)
88
82
74
1,141
195,962
Which semicha paper is hardest for AI? Top 3 models: Chuppah v'Kiddushin. The next 5: Niddah. 7 of 8 did best on Mitzvot HaTeluyot BaAretz. Only 12 questions per paper, and some grading sheets may be stricter, so treat it as a hint. Purely an intellectual exercise.
Chart: mean score out of 100 per exam paper, in this order: Chuppah v'Kiddushin, Niddah, Mikvaot, Tefillah u'Brachot, Mitzvot HaTeluyot BaAretz. Claude Fable 5.1 73, 77, 82, 82, 85; GPT-6 Astra 62, 65, 77, 79, 84; GPT-6.1 Sol 60, 67, 78, 76, 83; Claude Opus 5.5 63, 60, 70, 75, 78; Gemini 3.8 Flash 60, 55, 66, 66, 74; GPT-6.1 Terra 24, 21, 34, 48, 45; GPT-6 Luna 22, 17, 40, 39, 44; Claude Sonnet 5.5 26, 16, 31, 23, 48. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

ALT Chart: mean score out of 100 per exam paper, in this order: Chuppah v'Kiddushin, Niddah, Mikvaot, Tefillah u'Brachot, Mitzvot HaTeluyot BaAretz. Claude Fable 5.1 73, 77, 82, 82, 85; GPT-6 Astra 62, 65, 77, 79, 84; GPT-6.1 Sol 60, 67, 78, 76, 83; Claude Opus 5.5 63, 60, 70, 75, 78; Gemini 3.8 Flash 60, 55, 66, 66, 74; GPT-6.1 Terra 24, 21, 34, 48, 45; GPT-6 Luna 22, 17, 40, 39, 44; Claude Sonnet 5.5 26, 16, 31, 23, 48. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

151
Two older OpenAI models, as @GregHBurnham of Epoch AI suggested. GPT-5.6 Sol: 77.3, 3rd of 24, ahead of GPT-6.1 Sol (72.8). GPT-6 Sol: 48.7. Same harness, same effort setting, same questions. The newer model scored 29 points lower. @OpenAI @EpochAIResearch
Bar chart of HalachaBench scores for 24 AI models, 0 to 100 with 95% intervals. Top five: Claude Fable 5.1 79.7, Gemini 4 Argon 78.2, GPT-5.6 Sol 77.3, GPT-6 Astra 73.3, GPT-6.1 Sol 72.8. GPT-6 Sol is 8th at 48.7.

ALT Bar chart of HalachaBench scores for 24 AI models, 0 to 100 with 95% intervals. Top five: Claude Fable 5.1 79.7, Gemini 4 Argon 78.2, GPT-5.6 Sol 77.3, GPT-6 Astra 73.3, GPT-6.1 Sol 72.8. GPT-6 Sol is 8th at 48.7.

4
1
13
761
Why? GPT-6 Sol didn't make more mistakes (it got 2% of checks wrong, 5.6 got 2.5%). It just said less: ~200 words vs ~310, less thinking, and it covered 48% of the sources and rulings a full answer needs vs 77%. 5.6 did better on 57 of 59 questions.
1
101
It also gave a flat ruling where the answer depends on a fact the question leaves open 33 times, vs 9 for 5.6. Newer isn't always better. On a semicha exam, the model that writes out the conditions and the sources wins. github.com/halachabench/halaโ€ฆ
GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

HalachaBench early results: AI models on real semicha exam questions (provisional) - HalachaBench/halacha-bench-results

github.com
65
Update: 5 models added. Gemini 4 Argon: 78.2, 2nd of 22, statistically tied with Claude Fable 5.1. Grok 4.7: 29.9, 10th. Grok 4.3: 10.8. Claude Opus 4.8: 48.0. Claude Sonnet 4.6: 29.6. The checks behind the Grok result: xitter.gq/HalachaBench/status/21โ€ฆ
HalachaBench (@HalachaBench) profile pictureHalachaBench
@HalachaBench23h
New on HalachaBench: Gemini 4 Argon and Grok. Gemini 4 Argon: 78.2 out of 100, 2nd of 22, statistically tied with Claude Fable 5.1 (79.7). Grok 4.7: 29.9, 10th. Grok 4.3: 10.8. (Poll answer: under 65.) @GoogleDeepMind @xai
Image from HalachaBench (@HalachaBench)
266
New on HalachaBench: Gemini 4 Argon and Grok. Gemini 4 Argon: 78.2 out of 100, 2nd of 22, statistically tied with Claude Fable 5.1 (79.7). Grok 4.7: 29.9, 10th. Grok 4.3: 10.8. (Poll answer: under 65.) @GoogleDeepMind @xai
Image from HalachaBench (@HalachaBench)
5
3
1
61
2,183
more replies
Grok 4.7 thinks a lot but writes short answers (~210 words vs ~660 for Argon), and the grading sheets give points for each source and step. Details and all checks: github.com/halachabench/halaโ€ฆ
GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

HalachaBench early results: AI models on real semicha exam questions (provisional) - HalachaBench/halacha-bench-results

github.com
1
199
Also added: two earlier Claude models. Claude Opus 4.8: 48.0 (7th). Claude Sonnet 4.6: 29.6 (11th), level with Claude Sonnet 5.5 (28.7). @AnthropicAI
64
Quick guess: which semicha paper did AI find hardest? Answer tomorrow, with the chart.
12%Chuppah v'Kiddushin
39%Niddah
26%Mikvaot
22%Tefillah u'Brachot
76 votes โ€ข Final results
7
592
Grok is next on HalachaBench. The top score so far is 80 out of 100 (Claude Fable 5.1). Where will Grok land?
7%New #1 (above 80)
40%65 to 80
29%Under 65
24%Just show me
161 votes โ€ข Final results
6
1,211
AI takes rabbinical ordination examsโ€ฆ ๐Ÿ‘€
HalachaBench (@HalachaBench) profile pictureHalachaBench
@HalachaBenchNov 1
Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet. The best scored 80 out of 100. Many scored under 11. ๐Ÿงต
Image from HalachaBench (@HalachaBench)
32
25
4
649
73,969
Thank you to @SefariaProject: before the grading sheets were sealed, an AI checked their cited sources against Sefaria's library, about 2,000 lookups. Sources Sefaria doesn't carry are marked unverified.
1
46
5,743
Ask an AI the same semicha question three times and you can get three different answers. Keep only each model's weakest try on every question and scores drop 7 to 11 points. The order doesn't change. You'd want the same answer every time you ask.
Chart: score out of 100 for each AI, average of its three tries vs. its worst try on each question. Claude Fable 5.1 79.7 vs 70.4; GPT-6 Astra 73.3 vs 66.2; GPT-6.1 Sol 72.8 vs 63.9; Claude Opus 5.5 69.1 vs 60.0; Gemini 3.8 Flash 64.4 vs 54.3; GPT-6.1 Terra 34.6 vs 25.6; GPT-6 Luna 32.4 vs 21.8; Claude Sonnet 5.5 28.7 vs 18.3. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

ALT Chart: score out of 100 for each AI, average of its three tries vs. its worst try on each question. Claude Fable 5.1 79.7 vs 70.4; GPT-6 Astra 73.3 vs 66.2; GPT-6.1 Sol 72.8 vs 63.9; Claude Opus 5.5 69.1 vs 60.0; Gemini 3.8 Flash 64.4 vs 54.3; GPT-6.1 Terra 34.6 vs 25.6; GPT-6 Luna 32.4 vs 21.8; Claude Sonnet 5.5 28.7 vs 18.3. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

6
1
2
40
5,759
ื”ื›ื™ืจื• ืืช HalachaBench: ื ืชื ื• ืœืžื•ื“ืœื™ื ืฉืœ ื‘ื™ื ื” ืžืœืื›ื•ืชื™ืช (ืฉืœ ื™ื•ืฆืจื™ ChatGPT, Claude, Gemini ื•ืขื•ื“) ืฉืืœื•ืช ืืžื™ืชื™ื•ืช ืžืžื‘ื—ื ื™ ื”ื”ืกืžื›ื” ืฉืœ ื”ืจื‘ื ื•ืช ื”ืจืืฉื™ืช. ื”ื ืขื ื• ื‘ืขื‘ืจื™ืช, ื‘ืœื™ ืกืคืจื™ื ื•ื‘ืœื™ ืื™ื ื˜ืจื ื˜. ื”ืžื•ื“ืœ ื”ื˜ื•ื‘ ื‘ื™ื•ืชืจ ืงื™ื‘ืœ 80 ืžืชื•ืš 100. ืจื‘ื™ื ืงื™ื‘ืœื• ืคื—ื•ืช ืž-11. ๐Ÿงต
Image from HalachaBench (@HalachaBench)
4
2
23
1,877
more replies
ื—ืฉื•ื‘ ืœื”ื“ื’ื™ืฉ: ื–ื”ื• ืชืจื’ื™ืœ ืื™ื ื˜ืœืงื˜ื•ืืœื™ ื‘ืœื‘ื“. ืื™ืฉ ืื™ื ื• ื—ื•ืฉื‘ ืฉื‘ื™ื ื” ืžืœืื›ื•ืชื™ืช ื™ื›ื•ืœื” ืœืคืกื•ืง ื”ืœื›ื”. ื”ืชื•ืฆืื•ืช ืจืืฉื•ื ื™ื•ืช ื•ื–ืžื ื™ื•ืช: ืžื•ื“ืœ ืื—ื“ ื›ืชื‘ ืืช ื”ืžื—ื•ื•ื ื™ื ื•ืžื•ื“ืœ ืื—ืจ ื‘ื“ืง ืืช ื”ืชืฉื•ื‘ื•ืช, ื•ืฉื ื™ื”ื ื’ื ื ื‘ื—ื ื• ื‘ืขืฆืžื. ืืฃ ืจื‘ ืขื“ื™ื™ืŸ ืœื ืขื‘ืจ ืขืœ ื”ืžื—ื•ื•ื ื™ื ืื• ืขืœ ื”ืฆื™ื•ื ื™ื.
1
1
162
ื›ืœ ืžื™ ืฉืจื•ืฆื” ืœืขื‘ื•ืจ ืขืœ ื”ืžื—ื•ื•ื ื™ื ืื• ืœื‘ื“ื•ืง ืชืฉื•ื‘ื•ืช: ื ืฉืžื— ืžืื•ื“ ืœืขื–ืจืชื›ื. ืจื•ืฆื™ื ืœืขื–ื•ืจ ื‘ื”ืจื—ื‘ืช ื”ืžื™ื–ื? ื›ืชื‘ื• ืœื ื•: [email protected] ื”ืชื•ืฆืื•ืช ื”ืžืœืื•ืช ื•ืคื™ืจื•ื˜ ื”ืฉื™ื˜ื” (ื‘ืื ื’ืœื™ืช, ืขื ืกื™ื›ื•ื ื‘ืขื‘ืจื™ืช): github.com/halachabench/halaโ€ฆ
GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

HalachaBench early results: AI models on real semicha exam questions (provisional) - HalachaBench/halacha-bench-results

github.com
1
150
Load more