Self-consistency replaces greedy decoding in chain-of-thought prompting: sample several reasoning paths from the same model and keep the answer most of them reach. The idea is that a hard problem has several ways to the right answer and fewer ways to the same wrong one.
Findings
- Gains over plain chain-of-thought: GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2%, StrategyQA +6.4%, ARC-challenge +3.9%.
Limits
- Abstract-level reading only.
- All samples come from one model, so they share its blind spots; a confidently wrong model can agree with itself.
Relevance
Agreement among samples as a correctness signal also helps LLM judges (majority of five runs, in the LLM-as-a-Judge survey). GoGet goes a step further and asks a different model. Open question for llm-evaluation: how much independence is enough before agreement means something?
Links
- arXiv
- Subfield: Robot learning