{"data":{"question":"Does chain-of-thought prompting improve accuracy on multi-step arithmetic in large language models, and does the effect persist at larger scale?","domain":"cs_ai_engineering","llm":"ok","conclusion":[{"claimId":"s.c0","type":"FACT","text":"Few-shot chain-of-thought prompting substantially improves arithmetic reasoning accuracy in large models such as the 540B PaLM. The original authors frame this ability as emerging only in sufficiently large models.","confidence":0.9,"status":"supported"},{"claimId":"s.c1","type":"FACT","text":"Zero-shot CoT ('Let's think step by step') produced very large gains on MultiArith for GPT-3, from 17.7% to 78.7%.","confidence":0.9,"status":"supported"},{"claimId":"s.c2","type":"FACT","text":"The CoT gain does not straightforwardly persist in instruction-finetuned models. On ChatGPT, explicit CoT prompting was reported as no longer effective for arithmetic reasoning, because the model already generates CoT without being instructed to.","confidence":0.8,"status":"supported"},{"claimId":"s.c3","type":"INFERENCE","text":"Calibrated answer, part 1: CoT clearly improves multi-step arithmetic accuracy for large base (non-instruction-tuned) models.","confidence":0.7,"status":"open"},{"claimId":"s.c4","type":"INFERENCE","text":"Calibrated answer, part 2: in modern instruction-tuned and reasoning-trained models, the marginal gain from an explicit CoT prompt shrinks toward zero. The reason is that step-by-step reasoning is already the default behavior, not that intermediate computation stopped helping.","confidence":0.7,"status":"open"},{"claimId":"s.c5","type":"INFERENCE","text":"The critiques of the persistence question succeed. 'Does the effect persist at scale' is underdetermined unless one separates scale from instruction or RL tuning. A clean no-CoT baseline is often unavailable for reasoning models.","confidence":0.65,"status":"open"},{"claimId":"s.c6","type":"FACT","text":"Reported CoT effect sizes depend on the decoding strategy. Self-consistency sampling improves over greedy-decoded CoT, so cross-paper comparisons that mix decoding strategies are confounded.","confidence":0.8,"status":"supported"}],"confidence":0.779,"cruxes":[{"claimId":"h1.c0","text":"Multi-step arithmetic needs serial computation whose depth exceeds what a single forward pass can reliably do; writing intermediate results into the context adds effective serial compute, so CoT should help as the number of steps or digits grows.","status":"open"},{"claimId":"h1.o0","text":"The thesis bundles several claims, and the kill criteria in c5 do not test all of them. Gains 'emerging at scale' and 'shrinking in absolute terms' pull in opposite directions, so a single model-family sweep cannot confirm both. The test also cannot separate reasoning-trained models from base models unless the family includes both. The thesis is therefore only partly falsifiable by the proposed experiment.","status":"open"},{"claimId":"s.c7","text":"The claim that CoT gains are negative or near zero for models under roughly 10B parameters reflects the early-2022 model families. It may be partly an artifact of exact-match metrics and sparse size ladders. Math-fine-tuned small models may benefit, so this claim is weakened.","status":"open"},{"claimId":"s.c9","text":"GSM8K and MultiArith are mixed language-plus-arithmetic benchmarks with contamination risk for newer models. Gains on them therefore do not isolate arithmetic, and the 'persists at scale' claim is weakly evidenced for pure multi-digit arithmetic.","status":"open"},{"claimId":"s.c10","text":"On long multi-digit arithmetic, CoT likely still helps frontier models, but accuracy degrades with length. Residual errors may come from state tracking or operand copying, not only calculation slips. The claim that residual errors are mainly calculation slips (h1.c4) is unsupported by the retrieved records.","status":"open"}],"falsificationProbe":{"claimId":"s.c11","text":"Decisive probe: take one open model family with base and instruct checkpoints at 4 or more sizes (e.g., 1B–70B). Test on contamination-free synthetic arithmetic with controlled step count (2–10) and digit length (2–20), using matched greedy decoding. Measure per-digit accuracy and exact match for a direct answer that is forced to be short (output length capped) versus CoT. If the leading hypothesis holds, the CoT minus direct gap is positive for base models and grows with step count and size. For instruct models the gap is near zero on short problems but positive on the hardest buckets. Refutation: the gap is about zero on the hardest bucket at the largest size for base models, or per-digit accuracy scales smoothly with no CoT-specific interaction with size."},"hypotheses":[{"index":1,"lens":"mechanism-first: derive the answer from structure and first principles","status":"ok","thesisClaimId":"h1.thesis","claimIds":["h1.c0","h1.c1","h1.c2","h1.c3","h1.c4","h1.c5"],"critique":{"status":"ok","verdict":"weakened","objectionIds":["h1.o0","h1.o1","h1.o2","h1.o3","h1.o4"]}},{"index":2,"lens":"outside-view: reference classes, base rates and prior art","status":"ok","thesisClaimId":"h2.thesis","claimIds":["h2.c0","h2.c1","h2.c2","h2.c3","h2.c4","h2.c5"],"critique":{"status":"ok","verdict":"weakened","objectionIds":["h2.o0","h2.o1","h2.o2","h2.o3","h2.o4"]}}],"ledger":[{"id":"h1.thesis","type":"CONJECTURE","text":"Chain-of-thought prompting improves accuracy on multi-step arithmetic in sufficiently large LLMs by externalizing intermediate state, with gains emerging at scale and shrinking in absolute terms as base models and reasoning-trained models saturate, though the benefit does not vanish for problems with many steps or large operands.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h1.c0","type":"INFERENCE","text":"Multi-step arithmetic needs serial computation whose depth exceeds what a single forward pass can reliably do; writing intermediate results into the context adds effective serial compute, so CoT should help as the number of steps or digits grows.","evidenceDigests":[],"confidence":0.7,"status":"open","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h1.c1","type":"CONJECTURE","text":"Published results show CoT gains are small or negative for small models (under roughly 10B parameters) and become large only above a scale threshold, because small models produce fluent but arithmetically wrong intermediate steps.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h1.c2","type":"CONJECTURE","text":"At frontier scale and with RL-trained reasoning models, direct-answer accuracy on short arithmetic is already high, so the marginal gain from explicit CoT prompting shrinks on easy problems while staying positive on long or large-number problems.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h1.c3","type":"ASSUMPTION","text":"Benchmarks such as GSM8K, MultiArith, and synthetic n-digit addition/multiplication tasks are valid proxies for multi-step arithmetic, and reported gains are not mainly artifacts of prompt format, exemplar choice, or contamination.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h1.c4","type":"INFERENCE","text":"Residual CoT errors at scale should come mainly from calculation slips, such as carry or multi-digit multiplication errors, and not from planning failures, so tool use (calculator or code) should outperform CoT alone on arithmetic specifically.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h1.c5","type":"PROPOSAL","text":"Cheapest decisive test: run a model family across at least 4 sizes on synthetic arithmetic with controlled step counts and digit lengths, comparing direct, zero-shot CoT, and few-shot CoT. Kill the thesis if CoT gives no significant gain at the largest scale on the hardest bucket, or if gains do not rise from small to mid scale.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h2.thesis","type":"CONJECTURE","text":"Chain-of-thought prompting reliably improves accuracy on multi-step arithmetic (e.g., GSM8K-style word problems) in sufficiently large LLMs, with gains emerging mainly above a scale threshold; the benefit persists at frontier scale but shrinks in absolute terms as baseline accuracy saturates and as models are trained to reason internally or via RL, and for pure multi-digit calculation it is limited by per-step errors.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h2.c0","type":"INFERENCE","text":"Reference class: prior prompting-method papers (Wei et al. 2022 and follow-ups such as self-consistency and zero-shot CoT) report large gains on GSM8K, MultiArith and similar benchmarks for models of roughly 100B+ parameters, so the base rate of a positive effect at large scale is high.","evidenceDigests":[],"confidence":0.7,"status":"open","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h2.c1","type":"CONJECTURE","text":"CoT gains are small or negative for small models (below roughly 10B parameters), because they produce fluent but incorrect reasoning chains; this is an emergent-ability pattern whose threshold depends on model family and training data.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h2.c2","type":"CONJECTURE","text":"In recent instruction-tuned and reasoning-trained models, the marginal gain from explicit CoT prompts on arithmetic is much smaller, because these models already produce step-by-step reasoning by default and baseline accuracy approaches ceiling.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h2.c3","type":"ASSUMPTION","text":"Benchmark contamination, prompt format differences, and the choice between greedy decoding and self-consistency may inflate or confound reported CoT gains, so effect sizes are not directly comparable across papers.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h2.c4","type":"CONJECTURE","text":"For long multi-digit arithmetic (many-digit addition or multiplication), CoT helps but accuracy still degrades with length because of compounding per-step errors, so scale does not fully remove the failure mode.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h2.c5","type":"PROPOSAL","text":"Cheapest decisive test: run direct-answer versus CoT prompts across a model-size ladder within one family on GSM8K and a synthetic n-digit arithmetic set with matched decoding, plotting accuracy gain against scale. Kill the thesis if CoT gain is not positive at the largest sizes, or if there is no scale dependence.","evidenceDigests":[],"confidence":0.5,"status":"contested","stage":"hypothesize","origin":"model","evidence":[]},{"id":"h1.o0","type":"INFERENCE","text":"The thesis bundles several claims, and the kill criteria in c5 do not test all of them. Gains 'emerging at scale' and 'shrinking in absolute terms' pull in opposite directions, so a single model-family sweep cannot confirm both. The test also cannot separate reasoning-trained models from base models unless the family includes both. The thesis is therefore only partly falsifiable by the proposed experiment.","evidenceDigests":[],"confidence":0.6,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h1.o1","type":"CONJECTURE","text":"The 'emergence above a scale threshold' story is probably confounded. Apparent emergence can come from the accuracy metric, from pretraining data and code mix differing across model families, or from instruction tuning. Smooth scaling under continuous metrics (partial credit, per-digit accuracy) would show no sharp threshold. The '~10B' figure is also family-specific and not a stable constant, so c1 as stated may fail even if CoT helps.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h1.o2","type":"INFERENCE","text":"c4 is not supported by c0 and may be wrong. Residual errors at scale on long problems may come from long-context drift, mis-copying operands, losing track of state, or choosing the wrong decomposition, none of which are calculation slips. Tool use beating CoT is plausible for large operands, but the claim that planning failures are not the main error source is an untested empirical assertion. Word problems like GSM8K also have substantial semantic and planning errors, so 'arithmetic specifically' needs a clean synthetic setting.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h1.o3","type":"ASSUMPTION","text":"c3 is shaky. GSM8K and MultiArith are word problems that mix language comprehension with arithmetic, and GSM8K contamination is a documented concern for newer models. Gains on these benchmarks therefore do not isolate arithmetic. Synthetic digit tasks have a different distribution, and CoT format there (scratchpad vs. free-form) strongly affects results. A model may need a specific algorithmic scratchpad format, so 'CoT prompting' in general may fail on large-operand multiplication while a trained scratchpad succeeds.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h1.o4","type":"INFERENCE","text":"c2 conflates two things. For RL-trained reasoning models, CoT is built in and cannot be cleanly switched off. A 'direct' baseline is often not available, or it elicits reasoning anyway. Prompting such models to think step by step may add nothing or even hurt through redundancy and overthinking. The claim that the marginal gain 'stays positive' on hard problems is therefore hard to measure and may be a category error. Reports of zero or negative CoT prompting effects on strong reasoning models are plausible and would contradict it.","evidenceDigests":[],"confidence":0.55,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h2.o0","type":"INFERENCE","text":"The thesis bundles a scale-threshold claim with a persistence claim, so the c5 kill criterion is too weak. 'No scale dependence' could be satisfied by any monotone trend, and 'not positive at largest sizes' ignores that recent instruction-tuned models may show roughly zero or negative CoT deltas. Under the thesis's own caveats (saturation, internal reasoning), a null result at the frontier would be predicted rather than falsifying, which makes the thesis hard to refute.","evidenceDigests":[],"confidence":0.6,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h2.o1","type":"CONJECTURE","text":"The emergence-threshold framing is contested. Apparent thresholds may be artifacts of discontinuous metrics such as exact match, and of the sparse set of model sizes tested. Instruction tuning and data quality have produced small models (about 7B, with math-specific fine-tuning) that gain substantially from CoT. This undermines both the 'mainly above a scale threshold' claim and the specific 10B cutoff in c1.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h2.o2","type":"INFERENCE","text":"Using the original Wei et al. results as the reference class for c0 is a selection-biased base rate. Those results come from the authors' own prompts, a handful of hand-written exemplars, and a few model families (PaLM, GPT-3, LaMDA). Positive-result and publication bias inflate the base rate, and later work reports task-dependent CoT effects, with limited gains on non-math or symbolic tasks. The base rate is therefore not independent evidence of reliability.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h2.o3","type":"CONJECTURE","text":"c2 treats modern models as already doing CoT by default, so a 'direct answer' baseline may not be measurable. A direct-answer prompt often still elicits reasoning, so measured deltas conflate prompt effects with the model's native behavior. In reasoning-trained models, forcing no-CoT or adding CoT instructions can even hurt. The claim is therefore less a 'shrinking gain' than an ill-defined contrast, and the 'ceiling' explanation is not separated from the 'internalized reasoning' explanation.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"h2.o4","type":"ASSUMPTION","text":"c4 and c5 ignore the tokenization and tool-use confounders on synthetic n-digit arithmetic. Degradation with digit length depends heavily on number tokenization, the format of the intermediate steps (scratchpad versus free-form), and the training distribution, not just on compounding per-step errors. A within-family size ladder will not separate these causes, and scratchpad-trained models may show little degradation, so the claim that scale does not remove the failure mode may not generalize.","evidenceDigests":[],"confidence":0.45,"status":"open","stage":"falsify","origin":"model","evidence":[]},{"id":"s.c0","type":"FACT","text":"Few-shot chain-of-thought prompting substantially improves arithmetic reasoning accuracy in large models such as the 540B PaLM. The original authors frame this ability as emerging only in sufficiently large models.","evidenceDigests":["8f426a5a9251c96c052adb014445217ac5e068e22fe2acb89f6ebae5288ec7dd"],"confidence":0.9,"status":"supported","stage":"synthesize","origin":"model","role":"conclusion","evidence":[{"recordId":"arxiv:2201.11903v6","sha256":"8f426a5a9251c96c052adb014445217ac5e068e22fe2acb89f6ebae5288ec7dd","supports":true,"quote":"such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting"}]},{"id":"s.c1","type":"FACT","text":"Zero-shot CoT ('Let's think step by step') produced very large gains on MultiArith for GPT-3, from 17.7% to 78.7%.","evidenceDigests":["61b4418697b7203a7dccd740e247ecdf051f4d6f120cd6158aa4b3691f9183e3","c2179ce66606307fcbf49b8420a9424378aa414a58eb8399e1c367799220e69f"],"confidence":0.9,"status":"supported","stage":"synthesize","origin":"model","role":"conclusion","evidence":[{"recordId":"arxiv:2304.03262v2","sha256":"61b4418697b7203a7dccd740e247ecdf051f4d6f120cd6158aa4b3691f9183e3","supports":true,"quote":"GPT-3's accuracy can be improved from 17.7\\% to 78.7\\%"},{"recordId":"arxiv:2205.11916v4","sha256":"c2179ce66606307fcbf49b8420a9424378aa414a58eb8399e1c367799220e69f","supports":true,"quote":"LLMs are decent zero-shot reasoners by simply adding \"Let's think step by step\" before each answer"}]},{"id":"s.c2","type":"FACT","text":"The CoT gain does not straightforwardly persist in instruction-finetuned models. On ChatGPT, explicit CoT prompting was reported as no longer effective for arithmetic reasoning, because the model already generates CoT without being instructed to.","evidenceDigests":["61b4418697b7203a7dccd740e247ecdf051f4d6f120cd6158aa4b3691f9183e3"],"confidence":0.8,"status":"supported","stage":"synthesize","origin":"model","role":"conclusion","evidence":[{"recordId":"arxiv:2304.03262v2","sha256":"61b4418697b7203a7dccd740e247ecdf051f4d6f120cd6158aa4b3691f9183e3","supports":true,"quote":"on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning"}]},{"id":"s.c3","type":"INFERENCE","text":"Calibrated answer, part 1: CoT clearly improves multi-step arithmetic accuracy for large base (non-instruction-tuned) models.","evidenceDigests":[],"confidence":0.7,"status":"open","stage":"synthesize","origin":"model","role":"conclusion","evidence":[]},{"id":"s.c4","type":"INFERENCE","text":"Calibrated answer, part 2: in modern instruction-tuned and reasoning-trained models, the marginal gain from an explicit CoT prompt shrinks toward zero. The reason is that step-by-step reasoning is already the default behavior, not that intermediate computation stopped helping.","evidenceDigests":[],"confidence":0.7,"status":"open","stage":"synthesize","origin":"model","role":"conclusion","evidence":[]},{"id":"s.c5","type":"INFERENCE","text":"The critiques of the persistence question succeed. 'Does the effect persist at scale' is underdetermined unless one separates scale from instruction or RL tuning. A clean no-CoT baseline is often unavailable for reasoning models.","evidenceDigests":[],"confidence":0.65,"status":"open","stage":"synthesize","origin":"model","role":"conclusion","evidence":[]},{"id":"s.c6","type":"FACT","text":"Reported CoT effect sizes depend on the decoding strategy. Self-consistency sampling improves over greedy-decoded CoT, so cross-paper comparisons that mix decoding strategies are confounded.","evidenceDigests":["0cc58e07b07301a35a3a53b41de78c9426f705d21ca2fef6e55dcbbb85422800"],"confidence":0.8,"status":"supported","stage":"synthesize","origin":"model","role":"conclusion","evidence":[{"recordId":"arxiv:2203.11171v4","sha256":"0cc58e07b07301a35a3a53b41de78c9426f705d21ca2fef6e55dcbbb85422800","supports":true,"quote":"a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting"}]},{"id":"s.c7","type":"CONJECTURE","text":"The claim that CoT gains are negative or near zero for models under roughly 10B parameters reflects the early-2022 model families. It may be partly an artifact of exact-match metrics and sparse size ladders. Math-fine-tuned small models may benefit, so this claim is weakened.","evidenceDigests":[],"confidence":0.45,"status":"open","stage":"synthesize","origin":"model","role":"crux","evidence":[]},{"id":"s.c8","type":"INFERENCE","text":"It is unresolved whether CoT tokens causally drive the final answer or only correlate with it. This matters because the 'externalized serial compute' mechanism predicts persistent gains on long problems only if the chain is actually used. The causal-abstraction work on arithmetic word problems addresses this question, but the retrieved abstract truncates before its findings.","evidenceDigests":["a8dd832ed4ce0ef078332e479455c69840dee03bd61ff9cf3b11c05c42f9658a"],"confidence":0.5,"status":"supported","stage":"synthesize","origin":"model","role":"crux","evidence":[{"recordId":"openalex:W4389518346","sha256":"a8dd832ed4ce0ef078332e479455c69840dee03bd61ff9cf3b11c05c42f9658a","supports":true,"quote":"if LLMs use their CoTs to reason to their final answers"}]},{"id":"s.c9","type":"ASSUMPTION","text":"GSM8K and MultiArith are mixed language-plus-arithmetic benchmarks with contamination risk for newer models. Gains on them therefore do not isolate arithmetic, and the 'persists at scale' claim is weakly evidenced for pure multi-digit arithmetic.","evidenceDigests":[],"confidence":0.5,"status":"open","stage":"synthesize","origin":"model","role":"crux","evidence":[]},{"id":"s.c10","type":"CONJECTURE","text":"On long multi-digit arithmetic, CoT likely still helps frontier models, but accuracy degrades with length. Residual errors may come from state tracking or operand copying, not only calculation slips. The claim that residual errors are mainly calculation slips (h1.c4) is unsupported by the retrieved records.","evidenceDigests":[],"confidence":0.45,"status":"open","stage":"synthesize","origin":"model","role":"crux","evidence":[]},{"id":"s.c11","type":"PROPOSAL","text":"Decisive probe: take one open model family with base and instruct checkpoints at 4 or more sizes (e.g., 1B–70B). Test on contamination-free synthetic arithmetic with controlled step count (2–10) and digit length (2–20), using matched greedy decoding. Measure per-digit accuracy and exact match for a direct answer that is forced to be short (output length capped) versus CoT. If the leading hypothesis holds, the CoT minus direct gap is positive for base models and grows with step count and size. For instruct models the gap is near zero on short problems but positive on the hardest buckets. Refutation: the gap is about zero on the hardest bucket at the largest size for base models, or per-digit accuracy scales smoothly with no CoT-specific interaction with size.","evidenceDigests":[],"confidence":0.6,"status":"open","stage":"synthesize","origin":"model","role":"probe","evidence":[]}],"edges":[{"from":"h1.o0","to":"h1.c5","relation":"contradicts"},{"from":"h1.o1","to":"h1.c1","relation":"contradicts"},{"from":"h1.o2","to":"h1.c4","relation":"contradicts"},{"from":"h1.o3","to":"h1.c3","relation":"contradicts"},{"from":"h1.o4","to":"h1.c2","relation":"contradicts"},{"from":"h2.o0","to":"h2.c5","relation":"contradicts"},{"from":"h2.o1","to":"h2.c1","relation":"contradicts"},{"from":"h2.o2","to":"h2.c0","relation":"refines"},{"from":"h2.o3","to":"h2.c2","relation":"contradicts"},{"from":"h2.o4","to":"h2.c4","relation":"refines"},{"from":"s.c0","to":"h2.c0","relation":"supports"},{"from":"s.c0","to":"h1.thesis","relation":"supports"},{"from":"s.c1","to":"h2.c0","relation":"supports"},{"from":"s.c2","to":"h2.c2","relation":"supports"},{"from":"s.c2","to":"h1.c2","relation":"refines"},{"from":"s.c2","to":"h2.o3","relation":"supports"},{"from":"s.c3","to":"h1.thesis","relation":"refines"},{"from":"s.c3","to":"h2.thesis","relation":"refines"},{"from":"s.c4","to":"h1.thesis","relation":"refines"},{"from":"s.c4","to":"h2.thesis","relation":"refines"},{"from":"s.c5","to":"h1.o4","relation":"supports"},{"from":"s.c5","to":"h2.o3","relation":"supports"},{"from":"s.c5","to":"h1.o0","relation":"supports"},{"from":"s.c5","to":"h2.o0","relation":"supports"},{"from":"s.c6","to":"h2.c3","relation":"supports"},{"from":"s.c7","to":"h1.c1","relation":"contradicts"},{"from":"s.c7","to":"h2.c1","relation":"contradicts"},{"from":"s.c7","to":"h1.o1","relation":"supports"},{"from":"s.c7","to":"h2.o1","relation":"supports"},{"from":"s.c8","to":"h1.c0","relation":"depends_on"},{"from":"s.c9","to":"h1.o3","relation":"supports"},{"from":"s.c9","to":"h1.c3","relation":"contradicts"},{"from":"s.c9","to":"h2.o4","relation":"supports"},{"from":"s.c10","to":"h2.c4","relation":"supports"},{"from":"s.c10","to":"h1.c4","relation":"contradicts"},{"from":"s.c10","to":"h1.o2","relation":"supports"},{"from":"s.c11","to":"h1.c5","relation":"refines"},{"from":"s.c11","to":"h2.c5","relation":"refines"},{"from":"s.c11","to":"h1.o0","relation":"depends_on"}],"evidence":{"records":[{"id":"arxiv:2304.03262v2","sha256":"61b4418697b7203a7dccd740e247ecdf051f4d6f120cd6158aa4b3691f9183e3","source":"arxiv","title":"When do you need Chain-of-Thought Prompting for ChatGPT?","url":"http://arxiv.org/abs/2304.03262v2","publicationDate":"2023-04-06"},{"id":"arxiv:2410.08130v2","sha256":"c1a5ecbf90099df482fd8f94be09af08907bd7e93f7a8d9988d8fab7019cddd9","source":"arxiv","title":"Think Beyond Size: Adaptive Prompting for More Effective Reasoning","url":"http://arxiv.org/abs/2410.08130v2","publicationDate":"2024-10-10"},{"id":"arxiv:2205.11916v4","sha256":"c2179ce66606307fcbf49b8420a9424378aa414a58eb8399e1c367799220e69f","source":"arxiv","title":"Large Language Models are Zero-Shot Reasoners","url":"http://arxiv.org/abs/2205.11916v4","publicationDate":"2022-05-24"},{"id":"openalex:W7133224126","sha256":"7dda42a8f9dd94833310b6e81f00e15af3a40791bc818e33394382ee93a85587","source":"openalex","title":"Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models","url":"https://doi.org/10.52202/068431-1800","doi":"10.52202/068431-1800","publicationDate":"2022-01-01"},{"id":"openalex:W4389518346","sha256":"a8dd832ed4ce0ef078332e479455c69840dee03bd61ff9cf3b11c05c42f9658a","source":"openalex","title":"Causal Abstraction for Chain-of-Thought Reasoning in Arithmetic Word Problems","url":"https://doi.org/10.18653/v1/2023.blackboxnlp-1.12","doi":"10.18653/v1/2023.blackboxnlp-1.12","publicationDate":"2023-01-01"},{"id":"openalex:W4389523853","sha256":"4a6e589df56f690f979044334b24c1aae1e4c911008309446c172272a37b8da3","source":"openalex","title":"Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages","url":"https://doi.org/10.18653/v1/2023.emnlp-main.163","doi":"10.18653/v1/2023.emnlp-main.163","publicationDate":"2023-01-01"},{"id":"openalex:W4362707120","sha256":"dd942415ce6d024e44a6fe48fe2fde951e34e00d59201348c609b97ba8bbb1df","source":"openalex","title":"When do you need Chain-of-Thought Prompting for ChatGPT?","url":"http://arxiv.org/abs/2304.03262","doi":"10.48550/arxiv.2304.03262","publicationDate":"2023-04-06"},{"id":"openalex:W4304194220","sha256":"62e30d419248ff0e06572fc7b6dd0ae40338588769e62d52ed41f5d937854988","source":"openalex","title":"Automatic Chain of Thought Prompting in Large Language Models","url":"http://arxiv.org/abs/2210.03493","doi":"10.48550/arxiv.2210.03493","publicationDate":"2022-10-07"},{"id":"arxiv:2201.11903v6","sha256":"8f426a5a9251c96c052adb014445217ac5e068e22fe2acb89f6ebae5288ec7dd","source":"arxiv","title":"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models","url":"http://arxiv.org/abs/2201.11903v6","publicationDate":"2022-01-28"},{"id":"arxiv:2406.12288v3","sha256":"e1b447fb5c62b12555f5138eda328326e4887637fe3dd33d05d02e7c7e6c13ec","source":"arxiv","title":"An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs","url":"http://arxiv.org/abs/2406.12288v3","publicationDate":"2024-06-18"},{"id":"arxiv:2203.11171v4","sha256":"0cc58e07b07301a35a3a53b41de78c9426f705d21ca2fef6e55dcbbb85422800","source":"arxiv","title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models","url":"http://arxiv.org/abs/2203.11171v4","publicationDate":"2022-03-21"},{"id":"arxiv:2306.06427v3","sha256":"95dbb613fb0cd3cf94b1453cab3f4a563f1b1a00181cadce4f3545bc74635366","source":"arxiv","title":"Boosting Language Models Reasoning with Chain-of-Knowledge Prompting","url":"http://arxiv.org/abs/2306.06427v3","publicationDate":"2023-06-10"},{"id":"arxiv:2302.12822v3","sha256":"3fbcb4acbe5d4a22f9e66f1b85e07cff319d894530f821b7efb6a3a4174ff44c","source":"arxiv","title":"Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data","url":"http://arxiv.org/abs/2302.12822v3","publicationDate":"2023-02-24"},{"id":"openalex:W4221161695","sha256":"bc64a1a9feafcbf33be15f44e91e1d7a5287e63e43fae4ad607464db15cbcbf2","source":"openalex","title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models","url":"http://arxiv.org/abs/2203.11171","doi":"10.48550/arxiv.2203.11171","publicationDate":"2022-03-21"},{"id":"openalex:W4402672041","sha256":"cce25eeb0d94e3111d91056398432bcc886661344eafc5f35c6c36abac280338","source":"openalex","title":"Boosting Language Models Reasoning with Chain-of-Knowledge Prompting","url":"https://doi.org/10.18653/v1/2024.acl-long.271","doi":"10.18653/v1/2024.acl-long.271","publicationDate":"2024-01-01"},{"id":"openalex:W4389518664","sha256":"dc076f0195ab4df05d60f47de8ed32b549844e4ea420970473a02cef22c44c48","source":"openalex","title":"Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data","url":"https://doi.org/10.18653/v1/2023.findings-emnlp.811","doi":"10.18653/v1/2023.findings-emnlp.811","publicationDate":"2023-01-01"},{"id":"openalex:W4402671820","sha256":"abc4c9e6a84f51ce2d06816e382e9a775e3d7dd9cd06b7d9c68249b431f936d9","source":"openalex","title":"An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs","url":"https://doi.org/10.18653/v1/2024.acl-long.387","doi":"10.18653/v1/2024.acl-long.387","publicationDate":"2024-01-01"},{"id":"openalex:W7163685896","sha256":"0ae7fe3cb1d1651003ee4846768bb6f5b81a1f33c43998552110005ce94f82d1","source":"openalex","title":"Scaling Laws of Chain-of-Thought Reasoning in Large Language Models","url":"https://doi.org/10.5281/zenodo.20562346","doi":"10.5281/zenodo.20562346","publicationDate":"2026-06-05"},{"id":"openalex:W7125138108","sha256":"52ccde039cdf73c970f9fab3b925747e6924f917ebacc28ba624ce6a4ad10007","source":"openalex","title":"MS-CoTF: Multi-scale chain-of-thought fusion for interpretable biological reasoning with large language models","url":"https://doi.org/10.1016/j.compbiomed.2026.111467","doi":"10.1016/j.compbiomed.2026.111467","publicationDate":"2026-01-21"},{"id":"openalex:W7118950228","sha256":"e907369cdbabb27b275f40e95514d2f8390b68b3b4b2a7c9c6fcd54ad8df207c","source":"openalex","title":"Emergent Abilities in Large Language Models: In-Context Learning, Chain-of-Thought Reasoning, and Scaling Laws","url":"https://doi.org/10.36227/techrxiv.176784493.33333564/v1","doi":"10.36227/techrxiv.176784493.33333564/v1","publicationDate":"2026-01-08"},{"id":"openalex:W7202084756","sha256":"97249fd60435c5b01e6de658daee926674c50dc41e080dffcc102b73b71e5dba","source":"openalex","title":"A Statistically Validated Large Scale Mathematical Chain of Thought Dataset for Symbolic Reasoning and Large Language Model Benchmarking","url":"https://doi.org/10.6025/jdp/2026/16/3/161-184","doi":"10.6025/jdp/2026/16/3/161-184","publicationDate":"2026-08-10"},{"id":"openalex:W4402671980","sha256":"8158b307cc89e41b2f4db9c612c74367d2b71cde741cc2f9203c0d23abdb3d42","source":"openalex","title":"Active Prompting with Chain-of-Thought for Large Language Models","url":"https://doi.org/10.18653/v1/2024.acl-long.73","doi":"10.18653/v1/2024.acl-long.73","publicationDate":"2024-01-01"},{"id":"arxiv:2506.01341v2","sha256":"d12c028976439f8ddc255c71754fadbb4bd2895be2779607c13c6d8983729ff1","source":"arxiv","title":"TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models","url":"http://arxiv.org/abs/2506.01341v2","doi":"10.18653/v1/2025.findings-emnlp.1084","publicationDate":"2025-06-02"},{"id":"arxiv:2310.02804v1","sha256":"eac63278be2071452fab754f1d22f475af74d466f29b674fcf111bb10e68b856","source":"arxiv","title":"DOMINO: A Dual-System for Multi-step Visual Language Reasoning","url":"http://arxiv.org/abs/2310.02804v1","publicationDate":"2023-10-04"}],"lookups":[{"source":"arxiv","query":"chain-of-thought prompting improve accuracy multi-step arithmetic","ok":true,"count":3,"responseSha256":"81e7a3ddb975cc5a5a599347d03044316316c1c88109a68383b89ff395d0b1b4","error":null},{"source":"openalex","query":"chain-of-thought prompting improve accuracy multi-step arithmetic","ok":true,"count":5,"responseSha256":"d0ab913673af69fbb3b5645c3cea502bc4d854731ccafeb1062155e829ff2a81","error":null},{"source":"arxiv","query":"chain-of-thought prompting arithmetic reasoning","ok":true,"count":5,"responseSha256":"e0f634f82e27cab8c838f0fb7931632b3e804446722930bc6b304840dc41850e","error":null},{"source":"openalex","query":"chain-of-thought prompting arithmetic reasoning","ok":true,"count":5,"responseSha256":"c98648a8fc08dded2b42655e30359e416dd5b96a388dc8b678256fb4ccd25305","error":null},{"source":"arxiv","query":"chain of thought scaling large language models","ok":true,"count":0,"responseSha256":"4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945","error":null},{"source":"openalex","query":"chain of thought scaling large language models","ok":true,"count":5,"responseSha256":"025db5c1e7331ca85124668e6b602b1e75a7106799b219a04c62b5b2735f02a6","error":null},{"source":"arxiv","query":"multi-step arithmetic language model accuracy","ok":true,"count":5,"responseSha256":"ead422303bd0079afb83df9f87d866f1c79967603972373657e1edd935fa54a2","error":null},{"source":"openalex","query":"multi-step arithmetic language model accuracy","ok":true,"count":5,"responseSha256":"b77714492c48f5e94859f0634c5dda2a822d54c2885f14c6dc0cdd109a2242a7","error":null}],"retrievedAt":"2026-11-04T20:44:49.486Z"},"transcript":{"algorithm":"stage = {index, inputSha256, outputSha256, stage}; stageSha256 = sha256Hex(canonicalJson(stage)); chainSha256[0] = sha256Hex(utf8(canonicalJson(stage[0]))); chainSha256[i] = sha256Hex(bytes(chainSha256[i-1]) || utf8(canonicalJson(stage[i]))); inputSha256/outputSha256 = sha256Hex(canonicalJson(input|output)); canonicalJson is RFC 8785 JCS; bytes() is the 32-byte digest. input.upstream maps a stage name to that stage's outputSha256.","stages":[{"index":0,"stage":"classify","input":{"context":"Interested in published empirical results, not opinion.","question":"Does chain-of-thought prompting improve accuracy on multi-step arithmetic in large language models, and does the effect persist at larger scale?"},"output":{"domain":"cs_ai_engineering","queries":["chain-of-thought prompting arithmetic reasoning","chain of thought scaling large language models","multi-step arithmetic language model accuracy"],"status":"ok"},"inputSha256":"44f515892f4ea66eb8cc22a2628c0ebf11f6ca750998c70bb4d3cb32edc9151d","outputSha256":"105fd622acb28af5ee652fa9e8abd153a1b922631e9bc07934f687163a5841fa","stageSha256":"5e36a178d1ad55cf776cafb7b1dc7e765268aa576dc8d882e89b2421051dbd78","chainSha256":"5e36a178d1ad55cf776cafb7b1dc7e765268aa576dc8d882e89b2421051dbd78"},{"index":1,"stage":"hypothesize","input":{"context":"Interested in published empirical results, not opinion.","domain":"cs_ai_engineering","n":2,"question":"Does chain-of-thought prompting improve accuracy on multi-step arithmetic in large language models, and does the effect persist at larger scale?","upstream":{"classify":"105fd622acb28af5ee652fa9e8abd153a1b922631e9bc07934f687163a5841fa"}},"output":{"items":[{"index":1,"lens":"mechanism-first: derive the answer from structure and first principles","raw":{"claims":[{"confidence":0.8,"text":"Multi-step arithmetic needs serial computation whose depth exceeds what a single forward pass can reliably do; writing intermediate results into the context adds effective serial compute, so CoT should help as the number of steps or digits grows.","type":"INFERENCE"},{"confidence":0.7,"text":"Published results show CoT gains are small or negative for small models (under roughly 10B parameters) and become large only above a scale threshold, because small models produce fluent but arithmetically wrong intermediate steps.","type":"CONJECTURE"},{"confidence":0.65,"text":"At frontier scale and with RL-trained reasoning models, direct-answer accuracy on short arithmetic is already high, so the marginal gain from explicit CoT prompting shrinks on easy problems while staying positive on long or large-number problems.","type":"CONJECTURE"},{"confidence":0.6,"text":"Benchmarks such as GSM8K, MultiArith, and synthetic n-digit addition/multiplication tasks are valid proxies for multi-step arithmetic, and reported gains are not mainly artifacts of prompt format, exemplar choice, or contamination.","type":"ASSUMPTION"},{"confidence":0.6,"text":"Residual CoT errors at scale should come mainly from calculation slips, such as carry or multi-digit multiplication errors, and not from planning failures, so tool use (calculator or code) should outperform CoT alone on arithmetic specifically.","type":"INFERENCE"},{"confidence":0.8,"text":"Cheapest decisive test: run a model family across at least 4 sizes on synthetic arithmetic with controlled step counts and digit lengths, comparing direct, zero-shot CoT, and few-shot CoT. Kill the thesis if CoT gives no significant gain at the largest scale on the hardest bucket, or if gains do not rise from small to mid scale.","type":"PROPOSAL"}],"confidence":0.8,"queries":["chain-of-thought prompting arithmetic reasoning scale","emergent abilities chain of thought model size","zero-shot chain of thought multi-digit arithmetic"],"thesis":"Chain-of-thought prompting improves accuracy on multi-step arithmetic in sufficiently large LLMs by externalizing intermediate state, with gains emerging at scale and shrinking in absolute terms as base models and reasoning-trained models saturate, though the benefit does not vanish for problems with many steps or large operands."},"status":"ok"},{"index":2,"lens":"outside-view: reference classes, base rates and prior art","raw":{"claims":[{"confidence":0.85,"text":"Reference class: prior prompting-method papers (Wei et al. 2022 and follow-ups such as self-consistency and zero-shot CoT) report large gains on GSM8K, MultiArith and similar benchmarks for models of roughly 100B+ parameters, so the base rate of a positive effect at large scale is high.","type":"INFERENCE"},{"confidence":0.7,"text":"CoT gains are small or negative for small models (below roughly 10B parameters), because they produce fluent but incorrect reasoning chains; this is an emergent-ability pattern whose threshold depends on model family and training data.","type":"CONJECTURE"},{"confidence":0.65,"text":"In recent instruction-tuned and reasoning-trained models, the marginal gain from explicit CoT prompts on arithmetic is much smaller, because these models already produce step-by-step reasoning by default and baseline accuracy approaches ceiling.","type":"CONJECTURE"},{"confidence":0.7,"text":"Benchmark contamination, prompt format differences, and the choice between greedy decoding and self-consistency may inflate or confound reported CoT gains, so effect sizes are not directly comparable across papers.","type":"ASSUMPTION"},{"confidence":0.6,"text":"For long multi-digit arithmetic (many-digit addition or multiplication), CoT helps but accuracy still degrades with length because of compounding per-step errors, so scale does not fully remove the failure mode.","type":"CONJECTURE"},{"confidence":0.8,"text":"Cheapest decisive test: run direct-answer versus CoT prompts across a model-size ladder within one family on GSM8K and a synthetic n-digit arithmetic set with matched decoding, plotting accuracy gain against scale. Kill the thesis if CoT gain is not positive at the largest sizes, or if there is no scale dependence.","type":"PROPOSAL"}],"confidence":0.8,"queries":["chain-of-thought prompting arithmetic reasoning scale","emergent abilities chain-of-thought model size","when does chain-of-thought help arithmetic"],"thesis":"Chain-of-thought prompting reliably improves accuracy on multi-step arithmetic (e.g., GSM8K-style word problems) in sufficiently large LLMs, with gains emerging mainly above a scale threshold; the benefit persists at frontier scale but shrinks in absolute terms as baseline accuracy saturates and as models are trained to reason internally or via RL, and for pure multi-digit calculation it is limited by per-step errors."},"status":"ok"}]},"inputSha256":"c1b6d22af9ec0345c5da5c349c3bebdc974eedf48feccc8d6ab7bb1b088affa2","outputSha256":"c001d9a29641665478ca1964fa4ea39c642b5114b172c71837b675b09c9a38b2","stageSha256":"845b9239ca8d88a49da2978ab2ea8cbdb1648abf0eeacc717dd42cfb066cc211","chainSha256":"f52ed1d573f5eb0b87fd617894e918949f9f8345bd8f61e74e1400fc8a0bbcf5"},{"index":2,"stage":"falsify","input":{"context":"Interested in published empirical results, not opinion.","question":"Does chain-of-thought prompting improve accuracy on multi-step arithmetic in large language models, and does the effect persist at larger scale?","upstream":{"classify":"105fd622acb28af5ee652fa9e8abd153a1b922631e9bc07934f687163a5841fa","hypothesize":"c001d9a29641665478ca1964fa4ea39c642b5114b172c71837b675b09c9a38b2"}},"output":{"items":[{"index":1,"raw":{"objections":[{"confidence":0.6,"severity":"major","target":"c5","text":"The thesis bundles several claims, and the kill criteria in c5 do not test all of them. Gains 'emerging at scale' and 'shrinking in absolute terms' pull in opposite directions, so a single model-family sweep cannot confirm both. The test also cannot separate reasoning-trained models from base models unless the family includes both. The thesis is therefore only partly falsifiable by the proposed experiment.","type":"INFERENCE"},{"confidence":0.55,"severity":"major","target":"c1","text":"The 'emergence above a scale threshold' story is probably confounded. Apparent emergence can come from the accuracy metric, from pretraining data and code mix differing across model families, or from instruction tuning. Smooth scaling under continuous metrics (partial credit, per-digit accuracy) would show no sharp threshold. The '~10B' figure is also family-specific and not a stable constant, so c1 as stated may fail even if CoT helps.","type":"CONJECTURE"},{"confidence":0.5,"severity":"major","target":"c4","text":"c4 is not supported by c0 and may be wrong. Residual errors at scale on long problems may come from long-context drift, mis-copying operands, losing track of state, or choosing the wrong decomposition, none of which are calculation slips. Tool use beating CoT is plausible for large operands, but the claim that planning failures are not the main error source is an untested empirical assertion. Word problems like GSM8K also have substantial semantic and planning errors, so 'arithmetic specifically' needs a clean synthetic setting.","type":"INFERENCE"},{"confidence":0.6,"severity":"major","target":"c3","text":"c3 is shaky. GSM8K and MultiArith are word problems that mix language comprehension with arithmetic, and GSM8K contamination is a documented concern for newer models. Gains on these benchmarks therefore do not isolate arithmetic. Synthetic digit tasks have a different distribution, and CoT format there (scratchpad vs. free-form) strongly affects results. A model may need a specific algorithmic scratchpad format, so 'CoT prompting' in general may fail on large-operand multiplication while a trained scratchpad succeeds.","type":"ASSUMPTION"},{"confidence":0.55,"severity":"major","target":"c2","text":"c2 conflates two things. For RL-trained reasoning models, CoT is built in and cannot be cleanly switched off. A 'direct' baseline is often not available, or it elicits reasoning anyway. Prompting such models to think step by step may add nothing or even hurt through redundancy and overthinking. The claim that the marginal gain 'stays positive' on hard problems is therefore hard to measure and may be a category error. Reports of zero or negative CoT prompting effects on strong reasoning models are plausible and would contradict it.","type":"INFERENCE"}],"verdict":"weakened"},"status":"ok"},{"index":2,"raw":{"objections":[{"confidence":0.6,"severity":"major","target":"c5","text":"The thesis bundles a scale-threshold claim with a persistence claim, so the c5 kill criterion is too weak. 'No scale dependence' could be satisfied by any monotone trend, and 'not positive at largest sizes' ignores that recent instruction-tuned models may show roughly zero or negative CoT deltas. Under the thesis's own caveats (saturation, internal reasoning), a null result at the frontier would be predicted rather than falsifying, which makes the thesis hard to refute.","type":"INFERENCE"},{"confidence":0.55,"severity":"major","target":"c1","text":"The emergence-threshold framing is contested. Apparent thresholds may be artifacts of discontinuous metrics such as exact match, and of the sparse set of model sizes tested. Instruction tuning and data quality have produced small models (about 7B, with math-specific fine-tuning) that gain substantially from CoT. This undermines both the 'mainly above a scale threshold' claim and the specific 10B cutoff in c1.","type":"CONJECTURE"},{"confidence":0.5,"severity":"minor","target":"c0","text":"Using the original Wei et al. results as the reference class for c0 is a selection-biased base rate. Those results come from the authors' own prompts, a handful of hand-written exemplars, and a few model families (PaLM, GPT-3, LaMDA). Positive-result and publication bias inflate the base rate, and later work reports task-dependent CoT effects, with limited gains on non-math or symbolic tasks. The base rate is therefore not independent evidence of reliability.","type":"INFERENCE"},{"confidence":0.5,"severity":"major","target":"c2","text":"c2 treats modern models as already doing CoT by default, so a 'direct answer' baseline may not be measurable. A direct-answer prompt often still elicits reasoning, so measured deltas conflate prompt effects with the model's native behavior. In reasoning-trained models, forcing no-CoT or adding CoT instructions can even hurt. The claim is therefore less a 'shrinking gain' than an ill-defined contrast, and the 'ceiling' explanation is not separated from the 'internalized reasoning' explanation.","type":"CONJECTURE"},{"confidence":0.45,"severity":"minor","target":"c4","text":"c4 and c5 ignore the tokenization and tool-use confounders on synthetic n-digit arithmetic. Degradation with digit length depends heavily on number tokenization, the format of the intermediate steps (scratchpad versus free-form), and the training distribution, not just on compounding per-step errors. A within-family size ladder will not separate these causes, and scratchpad-trained models may show little degradation, so the claim that scale does not remove the failure mode may not generalize.","type":"ASSUMPTION"}],"verdict":"weakened"},"status":"ok"}]},"inputSha256":"d90dfab4c4164e36db99b43d760264e6b9ade9aecb90da9b1042fd4453da214c","outputSha256":"d2d393b70f6541205c3c33428e5c051b47a27b5fe68b9278433f2827c2f236e7","stageSha256":"77b990adc000c4dac735282aa71ecbfac9dab6892a4a06b5fcc4b366ece06b2f","chainSha256":"19d008198fb451324c1c63af73439a845f7cd81af39d00bee8228520f93c7d49"},{"index":3,"stage":"verify","input":{"queries":["chain-of-thought prompting improve accuracy multi-step arithmetic","chain-of-thought prompting arithmetic reasoning","chain of thought scaling large language models","multi-step arithmetic language model accuracy"],"question":"Does chain-of-thought prompting improve accuracy on multi-step arithmetic in large language models, and does the effect persist at larger scale?","upstream":{"falsify":"d2d393b70f6541205c3c33428e5c051b47a27b5fe68b9278433f2827c2f236e7"}},"output":{"lookups":[{"count":3,"error":null,"ok":true,"query":"chain-of-thought prompting improve accuracy multi-step arithmetic","responseSha256":"81e7a3ddb975cc5a5a599347d03044316316c1c88109a68383b89ff395d0b1b4","source":"arxiv"},{"count":5,"error":null,"ok":true,"query":"chain-of-thought prompting improve accuracy multi-step arithmetic","responseSha256":"d0ab913673af69fbb3b5645c3cea502bc4d854731ccafeb1062155e829ff2a81","source":"openalex"},{"count":5,"error":null,"ok":true,"query":"chain-of-thought prompting arithmetic reasoning","responseSha256":"e0f634f82e27cab8c838f0fb7931632b3e804446722930bc6b304840dc41850e","source":"arxiv"},{"count":5,"error":null,"ok":true,"query":"chain-of-thought prompting arithmetic reasoning","responseSha256":"c98648a8fc08dded2b42655e30359e416dd5b96a388dc8b678256fb4ccd25305","source":"openalex"},{"count":0,"error":null,"ok":true,"query":"chain of thought scaling large language models","responseSha256":"4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945","source":"arxiv"},{"count":5,"error":null,"ok":true,"query":"chain of thought scaling large language models","responseSha256":"025db5c1e7331ca85124668e6b602b1e75a7106799b219a04c62b5b2735f02a6","source":"openalex"},{"count":5,"error":null,"ok":true,"query":"multi-step arithmetic language model accuracy","responseSha256":"ead422303bd0079afb83df9f87d866f1c79967603972373657e1edd935fa54a2","source":"arxiv"},{"count":5,"error":null,"ok":true,"query":"multi-step arithmetic language model accuracy","responseSha256":"b77714492c48f5e94859f0634c5dda2a822d54c2885f14c6dc0cdd109a2242a7","source":"openalex"}],"queries":["chain-of-thought prompting improve accuracy multi-step arithmetic","chain-of-thought prompting arithmetic reasoning","chain of thought scaling large language models","multi-step arithmetic language model accuracy"],"works":[{"abstract":"Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step reasoning from Large Language Models~(LLMs). For example, by simply adding CoT instruction ``Let's think step-by-step'' to each input query of MultiArith dataset, GPT-3's accuracy can be improved from 17.7\\% to 78.7\\%. However, it is not clear whether CoT is still effective on more recent instruction finetuned (IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning while still keeping effective on other reasoning tasks. Moreover, on the former tasks, ChatGPT usually achieves the best performance and can generate CoT even without being instructed to do so. Hence, it is plausible that ChatGPT has already been trained on these tasks with CoT and thus memorized the instruction so it implicitly follows such an instruction when applied to the same queries, even without CoT. Our analysis reflects a potential risk of overfitting/bias toward instructions introduced in IFT, which becomes more common in training LLMs. In addition, it indicates possible leakage of the pretraining recipe, e.g., one can verify whether a dataset and instruction were used in training ChatGPT. Our experiments report new baseline results of ChatGPT on a variety of reasoning tasks and shed novel insights into LLM's profiling, instruction memorization, and pretraining dataset leakage.","authors":["Jiuhai Chen","Lichang Chen","Heng Huang","Tianyi Zhou"],"id":"2304.03262v2","institutions":[],"language":"en","publicationDate":"2023-04-06","source":"arxiv","title":"When do you need Chain-of-Thought Prompting for ChatGPT?","url":"http://arxiv.org/abs/2304.03262v2","venue":"arXiv"},{"abstract":"Pretrained large language models (LLMs) are increasingly utilized across a wide range of natural language processing (NLP) tasks due to their impressive capabilities as few-shot learners. Recent techniques, such as chain-of-thought (CoT) prompting, have significantly advanced multi-step reasoning by introducing step-by-step decomposition, achieving state-of-the-art results on complex reasoning benchmarks. However, these approaches often rely on static prompting templates that do not adapt to task complexity or errors during the reasoning process. In this work, we introduce Adaptive Prompting, a dynamic and iterative framework designed to enhance reasoning by incorporating real-time adjustments to prompt structures and validation mechanisms.Experimental results demonstrate that Adaptive Prompting significantly improves performance on diverse reasoning benchmarks, including arithmetic reasoning (GSM8K, MultiArith), logical reasoning and commonsense tasks, achieving substantial accuracy gains compared to static prompting baselines. By integrating guided prompts, intermediate validation, and self-corrective steps, our approach enables smaller models to achieve competitive performance with larger counterparts, such as GPT-4, while maintaining computational efficiency. The framework achieves this without requiring fine-tuning or task-specific training data, highlighting the untapped potential of iterative reasoning methods.","authors":["Kamesh R"],"id":"2410.08130v2","institutions":[],"language":"en","publicationDate":"2024-10-10","source":"arxiv","title":"Think Beyond Size: Adaptive Prompting for More Effective Reasoning","url":"http://arxiv.org/abs/2410.08130v2","venue":"arXiv"},{"abstract":"Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding \"Let's think step by step\" before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with large InstructGPT model (text-davinci-002), as well as similar magnitudes of improvements with another off-the-shelf large model, 540B parameter PaLM. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted by simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.","authors":["Takeshi Kojima","Shixiang Shane Gu","Machel Reid","Yutaka Matsuo","Yusuke Iwasawa"],"id":"2205.11916v4","institutions":[],"language":"en","publicationDate":"2022-05-24","source":"arxiv","title":"Large Language Models are Zero-Shot Reasoners","url":"http://arxiv.org/abs/2205.11916v4","venue":"arXiv"},{"abstract":"We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.","authors":["Jason Wei","Xuezhi Wang","Dale Schuurmans","Maarten Bosma","Brian Ichter","Fei Xia","Ed Chi","Quoc Viet Le","Denny Zhou"],"citedByCount":9611,"doi":"10.52202/068431-1800","id":"W7133224126","institutions":[],"language":"en","publicationDate":"2022-01-01","source":"openalex","title":"Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models","url":"https://doi.org/10.52202/068431-1800","venue":"neural information processing systems"},{"abstract":"Recent work suggests that large language models (LLMs) achieve higher accuracy on multistep reasoning tasks when prompted to generate intermediate reasoning steps, or a chain of thought (CoT), before their final answer.However, it is unclear how exactly CoTs improve LLMs' accuracy, and in particular, if LLMs use their CoTs to reason to their final answers.This paper tries to answer this question with respect to arithmetic word problems, by (i) evaluating the correctness of LLMs' CoTs, and (ii) using causal abstraction to assess if the intermediate tokens produced as part of a CoT causally impact LLMs' final answers, in line with the reasoning described by the CoT.We find that for CoT-prompted LLMs, correct answers to arithmetic problems are highly correlated with correct CoTs, and that when LLMs produce correct CoTs, they realize to a fairly large extent the causal models suggested by their CoTs.Higher degrees of realization also seem associated with better overall accuracy on the arithmetic problems.These findings suggest that some CoT-prompted LLMs may do better on multi-step arithmetic reasoning at least partly because they use their CoTs to reason to their final answers.However, for some LLMs, other internal processes may also be involved.","authors":["Juanhe Tan"],"citedByCount":2,"doi":"10.18653/v1/2023.blackboxnlp-1.12","id":"W4389518346","institutions":[{"country":"US","name":"Stanford University"}],"language":"en","publicationDate":"2023-01-01","source":"openalex","title":"Causal Abstraction for Chain-of-Thought Reasoning in Arithmetic Word Problems","url":"https://doi.org/10.18653/v1/2023.blackboxnlp-1.12","venue":"BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP)"},{"abstract":"Chain-of-thought (CoT) is capable of eliciting models to explicitly generate reasoning paths, thus promoting reasoning accuracy and attracting increasing attention.Specifically, zeroshot CoT achieves remarkable improvements in a wide range of reasoning tasks by simply instructing the LLM with the prompt \"Let's think step by step!\".Despite the success of zero-shot CoT, the existing zero-shot prompting techniques remain limited to a single language, making it challenging to generalize to other languages and hindering global development.In this work, we introduce cross-lingual prompting (CLP), aiming to improve zero-shot CoT reasoning across languages.Specifically, CLP consists of two main components: (1) cross-lingual alignment prompting and (2) taskspecific solver prompting.The cross-lingual alignment prompting is responsible for aligning representations across different languages, whereas the task-specific solver prompting is used to generate the final chain of thoughts and results for the reasoning task.In addition, we further introduce cross-lingual self-consistent prompting (CLSP) to ensemble different reasoning paths across languages.Our experimental evaluations on several benchmarks demonstrate that CLP and CLSP significantly outperform the existing prompting methods and achieve state-of-the-art performance.We hope this work will inspire further breakthroughs in cross-lingual CoT.Answer: 48.After understanding, you should act as an expert in arithmetic reasoning in German… Verstanden! Hier ist die Lösung für die gegebene Aufgabe: 1. Da Jiesha 20 Jahre alt ist, ist Qioung 20-5... 2. Da Mary zwei Jahre jünger als Qioung ist... Antwort: 48.Sorry, I am an AI language model and I can only communicate in English.May I assist you with anything else?Let's understand the task in German step-by-step!German Prompting Please act as an expert in multi-lingual understanding in Chinese.Request:玛丽比琼小两岁，琼比杰莎大五岁。如果杰莎 20 岁，这 三个女孩的年龄总和是多少？ Let's understand the task in English step-by-step!After understanding, you should act as an expert in arithmetic reasoning in English.Let's resolve the task you understand above step-by-step!Finally, you should format your answer as 'Answer: [num]'.Sure! Let's solve the task step-by-step: 1.According to the first sentence, Mary is two years younger than Joan.Let's suppose Joan's age is J, then Mary's age is J -2. 2. According to the second sentence, Joan is five years older than Jessie.If Jessie's age is x, then Joan's age is x + 5... Answer: 68.","authors":["Libo Qin","Qiguang Chen","Fuxuan Wei","Shijue Huang","Wanxiang Che"],"citedByCount":50,"doi":"10.18653/v1/2023.emnlp-main.163","id":"W4389523853","institutions":[{"country":"CN","name":"Central South University"},{"country":"CN","name":"Harbin Institute of Technology"}],"language":"en","publicationDate":"2023-01-01","source":"openalex","title":"Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages","url":"https://doi.org/10.18653/v1/2023.emnlp-main.163","venue":"Conference on Empirical Methods in Natural Language Processing (EMNLP)"},{"abstract":"Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step reasoning from Large Language Models~(LLMs). For example, by simply adding CoT instruction ``Let's think step-by-step'' to each input query of MultiArith dataset, GPT-3's accuracy can be improved from 17.7\\% to 78.7\\%. However, it is not clear whether CoT is still effective on more recent instruction finetuned (IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning while still keeping effective on other reasoning tasks. Moreover, on the former tasks, ChatGPT usually achieves the best performance and can generate CoT even without being instructed to do so. Hence, it is plausible that ChatGPT has already been trained on these tasks with CoT and thus memorized the instruction so it implicitly follows such an instruction when applied to the same queries, even without CoT. Our analysis reflects a potential risk of overfitting/bias toward instructions introduced in IFT, which becomes more common in training LLMs. In addition, it indicates possible leakage of the pretraining recipe, e.g., one can verify whether a dataset and instruction were used in training ChatGPT. Our experiments report new baseline results of ChatGPT on a variety of reasoning tasks and shed novel insights into LLM's profiling, instruction memorization, and pretraining dataset leakage.","authors":["Jiuhai Chen","Lichang Chen","Heng Huang","Tianyi Zhou"],"citedByCount":46,"doi":"10.48550/arxiv.2304.03262","id":"W4362707120","institutions":[{"country":"US","name":"University of Maryland, College Park"}],"language":"en","publicationDate":"2023-04-06","source":"openalex","title":"When do you need Chain-of-Thought Prompting for ChatGPT?","url":"http://arxiv.org/abs/2304.03262","venue":"arXiv (Cornell University)"},{"abstract":"Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like \"Let's think step by step\" to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstrations one by one. We show that such manual efforts may be eliminated by leveraging LLMs with the \"Let's think step by step\" prompt to generate reasoning chains for demonstrations one by one, i.e., let's think not just step by step, but also one by one. However, these generated chains often come with mistakes. To mitigate the effect of such mistakes, we find that diversity matters for automatically constructing demonstrations. We propose an automatic CoT prompting method: Auto-CoT. It samples questions with diversity and generates reasoning chains to construct demonstrations. On ten public benchmark reasoning tasks with GPT-3, Auto-CoT consistently matches or exceeds the performance of the CoT paradigm that requires manual designs of demonstrations. Code is available at https://github.com/amazon-research/auto-cot","authors":["Zhuosheng Zhang","Aston Zhang","Mu Li","Alex J. Smola"],"citedByCount":552,"doi":"10.48550/arxiv.2210.03493","id":"W4304194220","institutions":[],"language":"en","publicationDate":"2022-10-07","source":"openalex","title":"Automatic Chain of Thought Prompting in Large Language Models","url":"http://arxiv.org/abs/2210.03493","venue":"arXiv (Cornell University)"},{"abstract":"We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.","authors":["Jason Wei","Xuezhi Wang","Dale Schuurmans","Maarten Bosma","Brian Ichter","Fei Xia","Ed Chi","Quoc Le","Denny Zhou"],"id":"2201.11903v6","institutions":[],"language":"en","publicationDate":"2022-01-28","source":"arxiv","title":"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models","url":"http://arxiv.org/abs/2201.11903v6","venue":"arXiv"},{"abstract":"Large language models (LLMs) have shown strong arithmetic reasoning capabilities when prompted with Chain-of-Thought (CoT) prompts. However, we have only a limited understanding of how they are processed by LLMs. To demystify it, prior work has primarily focused on ablating different components in the CoT prompt and empirically observing their resulting LLM performance change. Yet, the reason why these components are important to LLM reasoning is not explored. To fill this gap, in this work, we investigate ``neuron activation'' as a lens to provide a unified explanation to observations made by prior work. Specifically, we look into neurons within the feed-forward layers of LLMs that may have activated their arithmetic reasoning capabilities, using Llama2 as an example. To facilitate this investigation, we also propose an approach based on GPT-4 to automatically identify neurons that imply arithmetic reasoning. Our analyses revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding.","authors":["Daking Rai","Ziyu Yao"],"id":"2406.12288v3","institutions":[],"language":"en","publicationDate":"2024-06-18","source":"arxiv","title":"An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs","url":"http://arxiv.org/abs/2406.12288v3","venue":"arXiv"},{"abstract":"Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to its unique correct answer. Our extensive empirical evaluation shows that self-consistency boosts the performance of chain-of-thought prompting with a striking margin on a range of popular arithmetic and commonsense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%).","authors":["Xuezhi Wang","Jason Wei","Dale Schuurmans","Quoc Le","Ed Chi","Sharan Narang","Aakanksha Chowdhery","Denny Zhou"],"id":"2203.11171v4","institutions":[],"language":"en","publicationDate":"2022-03-21","source":"arxiv","title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models","url":"http://arxiv.org/abs/2203.11171v4","venue":"arXiv"},{"abstract":"Recently, Chain-of-Thought (CoT) prompting has delivered success on complex reasoning tasks, which aims at designing a simple prompt like ``Let's think step by step'' or multiple in-context exemplars with well-designed rationales to elicit Large Language Models (LLMs) to generate intermediate reasoning steps. However, the generated rationales often come with mistakes, making unfactual and unfaithful reasoning chains. To mitigate this brittleness, we propose a novel Chain-of-Knowledge (CoK) prompting, where we aim at eliciting LLMs to generate explicit pieces of knowledge evidence in the form of structure triple. This is inspired by our human behaviors, i.e., we can draw a mind map or knowledge map as the reasoning evidence in the brain before answering a complex question. Benefiting from CoK, we additionally introduce a F^2-Verification method to estimate the reliability of the reasoning chains in terms of factuality and faithfulness. For the unreliable response, the wrong evidence can be indicated to prompt the LLM to rethink. Extensive experiments demonstrate that our method can further improve the performance of commonsense, factual, symbolic, and arithmetic reasoning tasks.","authors":["Jianing Wang","Qiushi Sun","Xiang Li","Ming Gao"],"id":"2306.06427v3","institutions":[],"language":"en","publicationDate":"2023-06-10","source":"arxiv","title":"Boosting Language Models Reasoning with Chain-of-Knowledge Prompting","url":"http://arxiv.org/abs/2306.06427v3","venue":"arXiv"},{"abstract":"Chain-of-thought (CoT) advances the reasoning abilities of large language models (LLMs) and achieves superior performance in complex reasoning tasks. However, most CoT studies rely on carefully designed human-annotated rational chains to prompt LLMs, posing challenges for real-world applications where labeled data is available without rational chains. This paper proposes a new strategy, Automate-CoT (Automatic Prompt Augmentation and Selection with Chain-of-Thought), that can bypass human engineering of CoT by automatically augmenting rational chains from a small labeled dataset, and then pruning low-quality chains to construct a candidate pool of machine-generated rationale chains based on the labels. Finally, it selects the optimal combination of several rationale chains from the pool for CoT prompting by employing a variance-reduced policy gradient strategy to estimate the significance of each example. Automate-CoT enables a quick adaptation of the CoT technique to different tasks. Experimental results demonstrate the effectiveness of our method, where competitive results are achieved on arithmetic reasoning (+2.7%), commonsense reasoning (+3.4%), symbolic reasoning (+3.2%), and non-reasoning tasks (+2.5%). The code is available at https://github.com/SHUMKASHUN/Automate-CoT.","authors":["KaShun Shum","Shizhe Diao","Tong Zhang"],"id":"2302.12822v3","institutions":[],"language":"en","publicationDate":"2023-02-24","source":"arxiv","title":"Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data","url":"http://arxiv.org/abs/2302.12822v3","venue":"arXiv"},{"abstract":"Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to its unique correct answer. Our extensive empirical evaluation shows that self-consistency boosts the performance of chain-of-thought prompting with a striking margin on a range of popular arithmetic and commonsense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%).","authors":["Xuezhi Wang","Wei, Jason","Dale Schuurmans","Quoc Viet Le","Ed H.","Narang, Sharan","Aakanksha Chowdhery","Zhou, Denny"],"citedByCount":2236,"doi":"10.48550/arxiv.2203.11171","id":"W4221161695","institutions":[{"country":"US","name":"Google (United States)"}],"language":"en","publicationDate":"2022-03-21","source":"openalex","title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models","url":"http://arxiv.org/abs/2203.11171","venue":"arXiv (Cornell University)"},{"abstract":"Recently, Chain-of-Thought (CoT) prompting has delivered success on complex reasoning tasks, which aims at designing a simple prompt like \"Let's think step by step\" or multiple incontext exemplars with well-designed rationales to elicit Large Language Models (LLMs) to generate intermediate reasoning steps.However, the generated rationales often come with hallucinations, making unfactual and unfaithful reasoning chains.To mitigate this brittleness, we propose a novel Chain-of-Knowledge (CoK) prompting, where we aim at eliciting LLMs to generate explicit pieces of knowledge evidence in the form of structure triple.This is inspired by our human behaviors, i.e., we can draw a mind map or knowledge map as the reasoning evidence in the brain before answering a complex question.Benefiting from CoK, we additionally introduce a F 2 -Verification method to estimate the reliability of the reasoning chains in terms of factuality and faithfulness.For the unreliable response, the wrong evidence can be indicated to prompt the LLM to rethink.Extensive experiments demonstrate that our method can further improve the performance of commonsense, factual, symbolic, and arithmetic reasoning tasks 1 .","authors":["Jianing Wang","Qiushi Sun","Xiang Li","Ming Gao"],"citedByCount":70,"doi":"10.18653/v1/2024.acl-long.271","id":"W4402672041","institutions":[{"country":"CN","name":"East China Normal University"},{"country":"HK","name":"University of Hong Kong"}],"language":"en","publicationDate":"2024-01-01","source":"openalex","title":"Boosting Language Models Reasoning with Chain-of-Knowledge Prompting","url":"https://doi.org/10.18653/v1/2024.acl-long.271","venue":"Proceedings of the conference - Association for Computational Linguistics. Meeting"},{"abstract":"Chain-of-thought (CoT) advances the reasoning abilities of large language models (LLMs) and achieves superior performance in complex reasoning tasks.However, most CoT studies rely on carefully designed human-annotated rational chains to prompt LLMs, posing challenges for real-world applications where labeled data is available without rational chains.This paper proposes a new strategy, Automate-CoT (Automatic Prompt Augmentation and Selection with Chain-of-Thought), that can bypass human engineering of CoT by automatically augmenting rational chains from a small labeled dataset, and then pruning low-quality chains to construct a candidate pool of machinegenerated rationale chains based on the labels.Finally, it selects the optimal combination of several rationale chains from the pool for CoT prompting by employing a variance-reduced policy gradient strategy to estimate the significance of each example.Automate-CoT enables a quick adaptation of the CoT technique to different tasks.Experimental results demonstrate the effectiveness of our method, where competitive results are achieved on arithmetic reasoning (+2.7%), commonsense reasoning (+3.4%), symbolic reasoning (+3.2%), and nonreasoning tasks (+2.5%). 1 * Equal Contribution.","authors":["Kashun Shum","Shizhe Diao","Tong Zhang"],"citedByCount":65,"doi":"10.18653/v1/2023.findings-emnlp.811","id":"W4389518664","institutions":[{"country":"HK","name":"Hong Kong University of Science and Technology"},{"country":"US","name":"Association for Computational Linguistics"}],"language":"en","publicationDate":"2023-01-01","source":"openalex","title":"Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data","url":"https://doi.org/10.18653/v1/2023.findings-emnlp.811","venue":"Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)"},{"abstract":"Large language models (LLMs) have shown strong arithmetic reasoning capabilities when prompted with Chain-of-Thought (CoT) prompts.However, we have only a limited understanding of how they are processed by LLMs.To demystify it, prior work has primarily focused on ablating different components in the CoT prompt and empirically observing their resulting LLM performance change (Madaan and Yazdanbakhsh, 2022;Wang et al., 2023;Ye et al., 2023).Yet, the reason why these components are important to LLM reasoning is not explored.To fill this gap, in this work, we investigate \"neuron activation\" as a lens to provide a unified explanation to observations made by prior work.Specifically, we look into neurons within the feed-forward layers of LLMs that may have activated their arithmetic reasoning capabilities, using Llama2 (Touvron et al., 2023) as an example.To facilitate this investigation, we also propose an approach based on GPT-4 to automatically identify neurons that imply arithmetic reasoning.Our analyses revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding. 1","authors":["Daking Rai","Ziyu Yao"],"citedByCount":10,"doi":"10.18653/v1/2024.acl-long.387","id":"W4402671820","institutions":[{"country":"US","name":"George Mason University"}],"language":"en","publicationDate":"2024-01-01","source":"openalex","title":"An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs","url":"https://doi.org/10.18653/v1/2024.acl-long.387","venue":"Proceedings of the conference - Association for Computational Linguistics. Meeting"},{"abstract":"This report synthesises findings from 18 peer-reviewed papers addressing the following research question: What are the scaling laws for chain-of-thought reasoning in large language models v6. 7 claims were extracted from source literature; 7 were independently verified against retrieved documents. An automated multi-reviewer quality assessment produced a score of 7.6/10. This report is a machine-generated literature synthesis and does not constitute original research. Research goal: What are the scaling laws for chain-of-thought reasoning in large language models v6 Autonomous literature synthesis. Automated review score: 7.6/10. Full text and citation available at Assignee Research.","authors":["Assignee Research"],"citedByCount":0,"doi":"10.5281/zenodo.20562346","id":"W7163685896","institutions":[],"language":"en","publicationDate":"2026-06-05","source":"openalex","title":"Scaling Laws of Chain-of-Thought Reasoning in Large Language Models","url":"https://doi.org/10.5281/zenodo.20562346","venue":"Zenodo (CERN European Organization for Nuclear Research)"},{"authors":["Zeyuan Song","Xiao-Cong Zhen"],"citedByCount":6,"doi":"10.1016/j.compbiomed.2026.111467","id":"W7125138108","institutions":[{"country":"US","name":"Oklahoma State University"}],"language":"en","publicationDate":"2026-01-21","source":"openalex","title":"MS-CoTF: Multi-scale chain-of-thought fusion for interpretable biological reasoning with large language models","url":"https://doi.org/10.1016/j.compbiomed.2026.111467","venue":"Computers in Biology and Medicine"},{"abstract":"This survey paper provides a comprehensive overview of emergent abilities in large language models, a phenomenon where certain capabilities appear unexpectedly at scale, without being present in smaller models. The paper explores the concept of emergent abilities, their implications, and the challenges associated with understanding them, covering key themes such as the evolution of large language model architectures, technical foundations, training methods, and applications. A historical overview of language model development highlights the significant improvements in processing and generating humanlike language, while the examination of Transformer architectures and beyond reveals the importance of self-attention mechanisms in enabling parallelization and efficient processing of sequential data. The survey also delves into training methods and objectives, including pre-training, prompting techniques, and contrastive learning, as well as applications in natural language understanding, generation, and multimodal processing. The analysis of continual learning, shortcut learning, and advanced topics such as multimodal models and domain adaptation provides a deeper understanding of the complexities and challenges associated with large language models. A comparative analysis of state-of-the-art models reveals a complex landscape of strengths, weaknesses, and future prospects, while highlighting the need for more rigorous scientific methodology in research. This paper contributes to the field by providing a comprehensive summary of recent advances, open challenges, and future directions in large language model research, with a focus on ethics, explainability, and real-world applications. The survey highlights the potential of large language models to impact various fields, including engineering design, and emphasizes the need for further research into the emergent abilities of these models. Overall, this paper aims to inform and guide researchers and practitioners in the development and application of large language models, ultimately contributing to the advancement of natural language processing and related fields.","authors":["Yu Ding","Xudong Han","Junjie Yang","Tianyang Wang","Ziqian Bi","Xinyuan Song","Junfeng Hao","Junhao Song","Enze Ge","Benji Peng","Z K Liu","Chia Xin Liang","Yichao Zhang","Ming Liu","Jiawei Xu","Binhua Huang","Yang Mo","Zhenyu Yu","Jing Qiao","Danyang Zhang","Yue Ma"],"citedByCount":0,"doi":"10.36227/techrxiv.176784493.33333564/v1","id":"W7118950228","institutions":[{"country":"CA","name":"Dalhousie University"},{"country":"GB","name":"University of Sussex"},{"country":"CN","name":"Xiamen University"},{"country":"GB","name":"University of Liverpool"},{"country":"CN","name":"Beijing University of Technology"},{"country":"US","name":"Emory University"},{"country":"CN","name":"Guangdong Medical College"},{"country":"GB","name":"Imperial College London"},{"country":"IT","name":"University of Bologna"},{"country":"US","name":"Georgia Institute of Technology"},{"country":"US","name":"University of Illinois Urbana-Champaign"},{"country":"TW","name":"National Chung Hsing University"},{"country":"US","name":"The University of Texas at Dallas"},{"country":"US","name":"Purdue University West Lafayette"},{"country":"IE","name":"University College Dublin"},{"country":"MY","name":"University of Malaya"},{"country":"US","name":"University of California System"},{"country":"CN","name":"East China Normal University"}],"language":"en","publicationDate":"2026-01-08","source":"openalex","title":"Emergent Abilities in Large Language Models: In-Context Learning, Chain-of-Thought Reasoning, and Scaling Laws","url":"https://doi.org/10.36227/techrxiv.176784493.33333564/v1"},{"abstract":"Mathematical reasoning poses a significant challenge for Large Language Models (LLMs) due to the necessity for structured symbolic manipulation, multi step deduction, and logically consistent inference.While Chainof Thought (CoT) prompting has substantially advanced reasoning capabilities, existing datasets often lack transparent, verifiable reasoning trajectories and rigorous statistical validation.This study presents a statistically validated, large scale mathematical CoT dataset comprising approximately 100,000 samples spanning six core mathematical domains (Algebra, Geometry, Trigonometry, Matrices, Derivatives, and Integrals) and three predefined difficulty levels.The dataset preserves complete intermediate reasoning traces and final solutions in standardized LaTeX notation.We conducted comprehensive structural, descriptive, and inferential statistical analyses including Chi-square tests, Spearman's rank correlation, two-way ANOVA, and Tukey's HSD post hoc comparisons to validate the dataset's internal consistency.Results demonstrate a perfectly balanced distribution across topics and difficulties, with the mathematical topic emerging as the dominant determinant of reasoning complexity (partial n² = 0.995).Furthermore, the corpus exhibits rich symbolic diversity, authentic mathematical syntax, and structured command cooccurrence networks.This statistically validated corpus provides a reliable and robust benchmark for supervised fine tuning, curriculum learning, process supervision, explainable AI, and evaluating nextgeneration mathematical LLMs.","authors":["Hathairat Ketmaneechairat"],"citedByCount":0,"doi":"10.6025/jdp/2026/16/3/161-184","id":"W7202084756","institutions":[{"country":"TH","name":"King Mongkut's University of Technology North Bangkok"}],"language":"en","publicationDate":"2026-08-10","source":"openalex","title":"A Statistically Validated Large Scale Mathematical Chain of Thought Dataset for Symbolic Reasoning and Large Language Model Benchmarking","url":"https://doi.org/10.6025/jdp/2026/16/3/161-184","venue":"Journal of Data Processing"},{"abstract":"The increasing scale of large language models (LLMs) brings emergent abilities to various complex tasks requiring reasoning, such as arithmetic and commonsense reasoning.It is known that the effective design of taskspecific prompts is critical for LLMs' ability to produce high-quality answers.In particular, an effective approach for complex questionand-answering tasks is example-based prompting with chain-of-thought (CoT) reasoning, which significantly improves the performance of LLMs.However, current CoT methods rely on a fixed set of human-annotated exemplars, which are not necessarily the most effective examples for different tasks.This paper proposes a new method, Active-Prompt, to adapt LLMs to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning).For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful to annotate from a pool of task-specific queries.By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty so as to select the most uncertain questions for annotation.Experimental results demonstrate the superiority of our proposed method, achieving superior performance on eight complex reasoning tasks.Further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracy-uncertainty relationships demonstrate the effectiveness of our method.1 (1) Uncertainty Estimation","authors":["Shizhe Diao","Pengcheng Wang","Yong Lin","Rui Pan","Xiang Liu","Tong Zhang"],"citedByCount":178,"doi":"10.18653/v1/2024.acl-long.73","id":"W4402671980","institutions":[{"country":"US","name":"University of Illinois Urbana-Champaign"},{"country":"CA","name":"University of Toronto"},{"country":"HK","name":"Hong Kong University of Science and Technology"}],"language":"en","publicationDate":"2024-01-01","source":"openalex","title":"Active Prompting with Chain-of-Thought for Large Language Models","url":"https://doi.org/10.18653/v1/2024.acl-long.73","venue":"Proceedings of the conference - Association for Computational Linguistics. Meeting"},{"abstract":"Despite impressive advances in large language models (LLMs), existing benchmarks often focus on single-turn or single-step tasks, failing to capture the kind of iterative reasoning required in real-world settings. To address this limitation, we introduce TurnBench, a novel benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task inspired by the \"Turing Machine Board Game.\" In each episode, a model must uncover hidden logical or arithmetic rules by making sequential guesses, receiving structured feedback, and integrating clues across multiple rounds. This dynamic setup requires models to reason over time, adapt based on past information, and maintain consistency across steps-capabilities underexplored in current benchmarks. TurnBench includes two modes: Classic, which tests standard reasoning, and Nightmare, which introduces increased complexity and requires robust inferential chains. To support fine-grained analysis, we provide ground-truth annotations for intermediate reasoning steps. Our evaluation of state-of-the-art LLMs reveals significant gaps: the best model achieves 84% accuracy in Classic mode, but performance drops to 18% in Nightmare mode. In contrast, human participants achieve 100% in both, underscoring the challenge TurnBench poses to current models. By incorporating feedback loops and hiding task rules, TurnBench reduces contamination risks and provides a rigorous testbed for diagnosing and advancing multi-step, multi-turn reasoning in LLMs.","authors":["Yiran Zhang","Mo Wang","Xiaoyang Li","Kaixuan Ren","Chencheng Zhu","Usman Naseem"],"doi":"10.18653/v1/2025.findings-emnlp.1084","id":"2506.01341v2","institutions":[],"language":"en","publicationDate":"2025-06-02","source":"arxiv","title":"TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models","url":"http://arxiv.org/abs/2506.01341v2","venue":"Findings of the ACL: EMNLP 2025, pp. 19892-19924, 2025"},{"abstract":"Visual language reasoning requires a system to extract text or numbers from information-dense images like charts or plots and perform logical or arithmetic reasoning to arrive at an answer. To tackle this task, existing work relies on either (1) an end-to-end vision-language model trained on a large amount of data, or (2) a two-stage pipeline where a captioning model converts the image into text that is further read by another large language model to deduce the answer. However, the former approach forces the model to answer a complex question with one single step, and the latter approach is prone to inaccurate or distracting information in the converted text that can confuse the language model. In this work, we propose a dual-system for multi-step multimodal reasoning, which consists of a \"System-1\" step for visual information extraction and a \"System-2\" step for deliberate reasoning. Given an input, System-2 breaks down the question into atomic sub-steps, each guiding System-1 to extract the information required for reasoning from the image. Experiments on chart and plot datasets show that our method with a pre-trained System-2 module performs competitively compared to prior work on in- and out-of-distribution data. By fine-tuning the System-2 module (LLaMA-2 70B) on only a small amount of data on multi-step reasoning, the accuracy of our method is further improved and surpasses the best fully-supervised end-to-end approach by 5.7% and a pipeline approach with FlanPaLM (540B) by 7.5% on a challenging dataset with human-authored questions.","authors":["Peifang Wang","Olga Golovneva","Armen Aghajanyan","Xiang Ren","Muhao Chen","Asli Celikyilmaz","Maryam Fazel-Zarandi"],"id":"2310.02804v1","institutions":[],"language":"en","publicationDate":"2023-10-04","source":"arxiv","title":"DOMINO: A Dual-System for Multi-step Visual Language Reasoning","url":"http://arxiv.org/abs/2310.02804v1","venue":"arXiv"}]},"inputSha256":"e2cb84a1e5ed0f29bfb29270f1618cbd55ee81572db922ac1a8f4a72b051fd7b","outputSha256":"1852a89d9e9f730a312f18fa31bf3c88096beff001ae4c44b4206677ec7b2521","stageSha256":"ccec14057c6834c65b18c3fa765043bce2ecf6893f9dec6ad72c6d52116466a7","chainSha256":"52ea0300057baf1942536729a05fbec4f1b9241e6ff3286a98edf751b4cb6d3c"},{"index":4,"stage":"synthesize","input":{"context":"Interested in published empirical results, not opinion.","question":"Does chain-of-thought prompting improve accuracy on multi-step arithmetic in large language models, and does the effect persist at larger scale?","upstream":{"classify":"105fd622acb28af5ee652fa9e8abd153a1b922631e9bc07934f687163a5841fa","falsify":"d2d393b70f6541205c3c33428e5c051b47a27b5fe68b9278433f2827c2f236e7","hypothesize":"c001d9a29641665478ca1964fa4ea39c642b5114b172c71837b675b09c9a38b2","verify":"1852a89d9e9f730a312f18fa31bf3c88096beff001ae4c44b4206677ec7b2521"}},"output":{"raw":{"claims":[{"citations":[{"quote":"such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting","recordId":"arxiv:2201.11903v6","supports":true},{"quote":"chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks","recordId":"arxiv:2201.11903v6","supports":true}],"confidence":0.9,"relations":[{"claimId":"h2.c0","relation":"supports"},{"claimId":"h1.thesis","relation":"supports"}],"role":"conclusion","text":"Few-shot chain-of-thought prompting substantially improves arithmetic reasoning accuracy in large models such as the 540B PaLM. The original authors frame this ability as emerging only in sufficiently large models.","type":"FACT"},{"citations":[{"quote":"GPT-3's accuracy can be improved from 17.7\\% to 78.7\\%","recordId":"arxiv:2304.03262v2","supports":true},{"quote":"LLMs are decent zero-shot reasoners by simply adding \"Let's think step by step\" before each answer","recordId":"arxiv:2205.11916v4","supports":true}],"confidence":0.9,"relations":[{"claimId":"h2.c0","relation":"supports"}],"role":"conclusion","text":"Zero-shot CoT ('Let's think step by step') produced very large gains on MultiArith for GPT-3, from 17.7% to 78.7%.","type":"FACT"},{"citations":[{"quote":"on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning","recordId":"arxiv:2304.03262v2","supports":true},{"quote":"can generate CoT even without being instructed t","recordId":"arxiv:2304.03262v2","supports":true}],"confidence":0.8,"relations":[{"claimId":"h2.c2","relation":"supports"},{"claimId":"h1.c2","relation":"refines"},{"claimId":"h2.o3","relation":"supports"}],"role":"conclusion","text":"The CoT gain does not straightforwardly persist in instruction-finetuned models. On ChatGPT, explicit CoT prompting was reported as no longer effective for arithmetic reasoning, because the model already generates CoT without being instructed to.","type":"FACT"},{"citations":[],"confidence":0.75,"relations":[{"claimId":"h1.thesis","relation":"refines"},{"claimId":"h2.thesis","relation":"refines"}],"role":"conclusion","text":"Calibrated answer, part 1: CoT clearly improves multi-step arithmetic accuracy for large base (non-instruction-tuned) models.","type":"INFERENCE"},{"citations":[],"confidence":0.75,"relations":[{"claimId":"h1.thesis","relation":"refines"},{"claimId":"h2.thesis","relation":"refines"}],"role":"conclusion","text":"Calibrated answer, part 2: in modern instruction-tuned and reasoning-trained models, the marginal gain from an explicit CoT prompt shrinks toward zero. The reason is that step-by-step reasoning is already the default behavior, not that intermediate computation stopped helping.","type":"INFERENCE"},{"citations":[],"confidence":0.65,"relations":[{"claimId":"h1.o4","relation":"supports"},{"claimId":"h2.o3","relation":"supports"},{"claimId":"h1.o0","relation":"supports"},{"claimId":"h2.o0","relation":"supports"}],"role":"conclusion","text":"The critiques of the persistence question succeed. 'Does the effect persist at scale' is underdetermined unless one separates scale from instruction or RL tuning. A clean no-CoT baseline is often unavailable for reasoning models.","type":"INFERENCE"},{"citations":[{"quote":"a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting","recordId":"arxiv:2203.11171v4","supports":true}],"confidence":0.8,"relations":[{"claimId":"h2.c3","relation":"supports"}],"role":"conclusion","text":"Reported CoT effect sizes depend on the decoding strategy. Self-consistency sampling improves over greedy-decoded CoT, so cross-paper comparisons that mix decoding strategies are confounded.","type":"FACT"},{"citations":[],"confidence":0.45,"relations":[{"claimId":"h1.c1","relation":"contradicts"},{"claimId":"h2.c1","relation":"contradicts"},{"claimId":"h1.o1","relation":"supports"},{"claimId":"h2.o1","relation":"supports"}],"role":"crux","text":"The claim that CoT gains are negative or near zero for models under roughly 10B parameters reflects the early-2022 model families. It may be partly an artifact of exact-match metrics and sparse size ladders. Math-fine-tuned small models may benefit, so this claim is weakened.","type":"CONJECTURE"},{"citations":[{"quote":"if LLMs use their CoTs to reason to their final answers","recordId":"openalex:W4389518346","supports":true}],"confidence":0.5,"relations":[{"claimId":"h1.c0","relation":"depends_on"}],"role":"crux","text":"It is unresolved whether CoT tokens causally drive the final answer or only correlate with it. This matters because the 'externalized serial compute' mechanism predicts persistent gains on long problems only if the chain is actually used. The causal-abstraction work on arithmetic word problems addresses this question, but the retrieved abstract truncates before its findings.","type":"INFERENCE"},{"citations":[],"confidence":0.6,"relations":[{"claimId":"h1.o3","relation":"supports"},{"claimId":"h1.c3","relation":"contradicts"},{"claimId":"h2.o4","relation":"supports"}],"role":"crux","text":"GSM8K and MultiArith are mixed language-plus-arithmetic benchmarks with contamination risk for newer models. Gains on them therefore do not isolate arithmetic, and the 'persists at scale' claim is weakly evidenced for pure multi-digit arithmetic.","type":"ASSUMPTION"},{"citations":[],"confidence":0.45,"relations":[{"claimId":"h2.c4","relation":"supports"},{"claimId":"h1.c4","relation":"contradicts"},{"claimId":"h1.o2","relation":"supports"}],"role":"crux","text":"On long multi-digit arithmetic, CoT likely still helps frontier models, but accuracy degrades with length. Residual errors may come from state tracking or operand copying, not only calculation slips. The claim that residual errors are mainly calculation slips (h1.c4) is unsupported by the retrieved records.","type":"CONJECTURE"},{"citations":[],"confidence":0.6,"relations":[{"claimId":"h1.c5","relation":"refines"},{"claimId":"h2.c5","relation":"refines"},{"claimId":"h1.o0","relation":"depends_on"}],"role":"probe","text":"Decisive probe: take one open model family with base and instruct checkpoints at 4 or more sizes (e.g., 1B–70B). Test on contamination-free synthetic arithmetic with controlled step count (2–10) and digit length (2–20), using matched greedy decoding. Measure per-digit accuracy and exact match for a direct answer that is forced to be short (output length capped) versus CoT. If the leading hypothesis holds, the CoT minus direct gap is positive for base models and grows with step count and size. For instruct models the gap is near zero on short problems but positive on the hardest buckets. Refutation: the gap is about zero on the hardest bucket at the largest size for base models, or per-digit accuracy scales smoothly with no CoT-specific interaction with size.","type":"PROPOSAL"}]},"status":"ok"},"inputSha256":"b1841907cfcb15efb76fc09bd28f0ce74fe540789ebbe14ccabe774412b207df","outputSha256":"624b794c008fdedfd37e1362e16996d3812b67931e79e61bed46294722d8b1e3","stageSha256":"48ab3730ac2dacd7fecec13da2bbc76a020a05f5d7d979634b5f7ee4bf7a060f","chainSha256":"8f776edcf68ad90ba15ea18feff8aef310630b8a21469e215143ee4403143a57"}],"chainHead":"8f776edcf68ad90ba15ea18feff8aef310630b8a21469e215143ee4403143a57"},"deterministic":{"domainContract":"Define metrics and constraints; generate orthogonal mechanisms; state the delta over prior art; separate novelty, feasibility and evidence; name the cheapest decisive prototype and kill criteria.","queries":["chain-of-thought prompting improve accuracy multi-step arithmetic","chain-of-thought prompting arithmetic reasoning","chain of thought scaling large language models","multi-step arithmetic language model accuracy"],"recordCount":24,"lookupFailures":0,"adjustments":[],"ledgerStats":{"total":36,"byType":{"FACT":4,"ASSUMPTION":5,"INFERENCE":12,"CONJECTURE":12,"PROPOSAL":3},"byStatus":{"open":23,"supported":5,"contested":8,"falsified":0}},"verificationCoverage":0.139,"ledgerSha256":"2353962c010a567f4461f6922e8aed4baafe1c270437e0f0c4b8a1059a47ec0a","transcriptCheck":{"ok":true,"failures":[],"recomputedHead":"8f776edcf68ad90ba15ea18feff8aef310630b8a21469e215143ee4403143a57"}},"warnings":[],"analyzedAt":"2026-11-04T20:44:49.486Z"},"_provenance":{"version":"writerslogic-provenance/1","issuer":"did:web:agents.writerslogic.com","verify":{"endpoint":"POST /api/v1/verify-provenance","did":"did:web:agents.writerslogic.com"},"manifest":{"format":"application/json","instance_id":"xmp:iid:825bfaef-a14b-441e-9069-e0e03f442048","claim_generator_info":[{"name":"WritersLogic x402 Agent Hub","version":"2.0.0"}],"title":"Crosstalk epistemic swarm & falsification engine","assertions":[{"label":"c2pa.actions.v2","data":{"actions":[{"action":"c2pa.created","when":"2026-11-04T20:45:22.447Z","softwareAgent":{"name":"writerslogic-agent-hub/crosstalk-deliberate","version":"2.0.0"},"digitalSourceType":"http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia","parameters":{"endpoint":"POST /api/v1/crosstalk-deliberate","inputDigest":{"alg":"sha256","hash":"5f8cd64a267df89b4e18e69cb10b6084bc8c3c385d2f160f1ced0e8659cb0017","canonicalization":"JCS"}}}]}},{"label":"c2pa.hash.data","data":{"alg":"sha256","hash":"126d7339f6a4b259820f3b1d96018e1179655fddaf4864f02e3b0a7ed243d40d","name":"jcs-canonical-json(data)"}},{"label":"c2pa.ingredient.v3","data":{"title":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain-of-thought+AND+all%3Aprompting+AND+all%3Aimprove+AND+all%3Aaccuracy+AND+all%3Amulti-step+AND+all%3Aarithmetic&sortBy=relevance","format":"application/atom+xml","relationship":"inputTo","dc":{"source":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain-of-thought+AND+all%3Aprompting+AND+all%3Aimprove+AND+all%3Aaccuracy+AND+all%3Amulti-step+AND+all%3Aarithmetic&sortBy=relevance","date":"2026-11-04T20:44:47.850Z"},"hash":{"alg":"sha256","hash":"766f0f38261e3685701e7325032059b0bfaf793d11c7ee8a3148047c62b4e12d","basis":"body"}}},{"label":"c2pa.ingredient.v3","data":{"title":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain-of-thought+AND+all%3Aprompting+AND+all%3Aarithmetic+AND+all%3Areasoning&sortBy=relevance","format":"application/atom+xml","relationship":"inputTo","dc":{"source":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain-of-thought+AND+all%3Aprompting+AND+all%3Aarithmetic+AND+all%3Areasoning&sortBy=relevance","date":"2026-11-04T20:44:47.928Z"},"hash":{"alg":"sha256","hash":"3181d39580c1be2d562dd8772a7c0dad03e3c90078edd1f5ca0cad99459ada48","basis":"body"}}},{"label":"c2pa.ingredient.v3","data":{"title":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain+AND+all%3Aof+AND+all%3Athought+AND+all%3Ascaling+AND+all%3Alarge+AND+all%3Alanguage+AND+all%3Amodels&sortBy=relevance","format":"application/atom+xml","relationship":"inputTo","dc":{"source":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain+AND+all%3Aof+AND+all%3Athought+AND+all%3Ascaling+AND+all%3Alarge+AND+all%3Alanguage+AND+all%3Amodels&sortBy=relevance","date":"2026-11-04T20:44:48.022Z"},"hash":{"alg":"sha256","hash":"955c11dcf426e94b9d9535e835c4bf07c5ee1c52f365200a79772dc34eca01df","basis":"body"}}},{"label":"c2pa.ingredient.v3","data":{"title":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain-of-thought+prompting+improve+accuracy+multi-step+arithmetic&api_key=REDACTED","format":"application/json","relationship":"inputTo","dc":{"source":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain-of-thought+prompting+improve+accuracy+multi-step+arithmetic&api_key=REDACTED","date":"2026-11-04T20:44:48.026Z"},"hash":{"alg":"sha256","hash":"0eacc989ea1f4d861ac523fe064da846f4aadef2d93c0156133f1a50b44201f8","basis":"body"}}},{"label":"c2pa.ingredient.v3","data":{"title":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain-of-thought+prompting+arithmetic+reasoning&api_key=REDACTED","format":"application/json","relationship":"inputTo","dc":{"source":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain-of-thought+prompting+arithmetic+reasoning&api_key=REDACTED","date":"2026-11-04T20:44:48.397Z"},"hash":{"alg":"sha256","hash":"7257e58c0f087bc40e2942df7066cd1db435f1e8205e07ea35100f3a06863feb","basis":"body"}}},{"label":"c2pa.ingredient.v3","data":{"title":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Amulti-step+AND+all%3Aarithmetic+AND+all%3Alanguage+AND+all%3Amodel+AND+all%3Aaccuracy&sortBy=relevance","format":"application/atom+xml","relationship":"inputTo","dc":{"source":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Amulti-step+AND+all%3Aarithmetic+AND+all%3Alanguage+AND+all%3Amodel+AND+all%3Aaccuracy&sortBy=relevance","date":"2026-11-04T20:44:48.511Z"},"hash":{"alg":"sha256","hash":"6cf24fd7dc0056d290c4ffe5be9cb551ddda4e8e8b2bdd637f4613496faff10b","basis":"body"}}},{"label":"c2pa.ingredient.v3","data":{"title":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain+of+thought+scaling+large+language+models&api_key=REDACTED","format":"application/json","relationship":"inputTo","dc":{"source":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain+of+thought+scaling+large+language+models&api_key=REDACTED","date":"2026-11-04T20:44:49.104Z"},"hash":{"alg":"sha256","hash":"95f5da903c351b0811cc3a80be9e6678b1ec0b13dc208b89df30c14a69e44124","basis":"body"}}},{"label":"c2pa.ingredient.v3","data":{"title":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=multi-step+arithmetic+language+model+accuracy&api_key=REDACTED","format":"application/json","relationship":"inputTo","dc":{"source":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=multi-step+arithmetic+language+model+accuracy&api_key=REDACTED","date":"2026-11-04T20:44:49.486Z"},"hash":{"alg":"sha256","hash":"f054d820756ee88f6db7506dc6ecb3a0818087a4858af7245a39713c3db5f8f4","basis":"body"}}}]},"credential":{"@context":["https://www.w3.org/ns/credentials/v2"],"id":"urn:uuid:c8d23df6-80d5-4fe3-b455-8b4811db190f","type":["VerifiableCredential","AgentOutputCredential"],"issuer":"did:web:agents.writerslogic.com","validFrom":"2026-11-04T20:45:22.447Z","credentialSubject":{"id":"urn:sha256:126d7339f6a4b259820f3b1d96018e1179655fddaf4864f02e3b0a7ed243d40d","agent":"crosstalk-deliberate","endpoint":"POST /api/v1/crosstalk-deliberate","outputDigest":{"alg":"sha256","value":"126d7339f6a4b259820f3b1d96018e1179655fddaf4864f02e3b0a7ed243d40d","canonicalization":"JCS"},"inputDigest":{"alg":"sha256","value":"5f8cd64a267df89b4e18e69cb10b6084bc8c3c385d2f160f1ced0e8659cb0017","canonicalization":"JCS"},"manifestDigest":{"alg":"sha256","value":"d1af4c06de0ec7954047e35632b66261155140fe3fe3cc52131eaac034b4e9b6","canonicalization":"JCS"},"ingredients":[{"url":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain-of-thought+AND+all%3Aprompting+AND+all%3Aimprove+AND+all%3Aaccuracy+AND+all%3Amulti-step+AND+all%3Aarithmetic&sortBy=relevance","sha256":"766f0f38261e3685701e7325032059b0bfaf793d11c7ee8a3148047c62b4e12d"},{"url":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain-of-thought+AND+all%3Aprompting+AND+all%3Aarithmetic+AND+all%3Areasoning&sortBy=relevance","sha256":"3181d39580c1be2d562dd8772a7c0dad03e3c90078edd1f5ca0cad99459ada48"},{"url":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Achain+AND+all%3Aof+AND+all%3Athought+AND+all%3Ascaling+AND+all%3Alarge+AND+all%3Alanguage+AND+all%3Amodels&sortBy=relevance","sha256":"955c11dcf426e94b9d9535e835c4bf07c5ee1c52f365200a79772dc34eca01df"},{"url":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain-of-thought+prompting+improve+accuracy+multi-step+arithmetic&api_key=REDACTED","sha256":"0eacc989ea1f4d861ac523fe064da846f4aadef2d93c0156133f1a50b44201f8"},{"url":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain-of-thought+prompting+arithmetic+reasoning&api_key=REDACTED","sha256":"7257e58c0f087bc40e2942df7066cd1db435f1e8205e07ea35100f3a06863feb"},{"url":"https://export.arxiv.org/api/query?start=0&max_results=5&search_query=all%3Amulti-step+AND+all%3Aarithmetic+AND+all%3Alanguage+AND+all%3Amodel+AND+all%3Aaccuracy&sortBy=relevance","sha256":"6cf24fd7dc0056d290c4ffe5be9cb551ddda4e8e8b2bdd637f4613496faff10b"},{"url":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=chain+of+thought+scaling+large+language+models&api_key=REDACTED","sha256":"95f5da903c351b0811cc3a80be9e6678b1ec0b13dc208b89df30c14a69e44124"},{"url":"https://api.openalex.org/works?per_page=5&select=id%2Cdoi%2Cdisplay_name%2Cpublication_date%2Clanguage%2Cabstract_inverted_index%2Ccited_by_count%2Cauthorships%2Cprimary_location&search=multi-step+arithmetic+language+model+accuracy&api_key=REDACTED","sha256":"f054d820756ee88f6db7506dc6ecb3a0818087a4858af7245a39713c3db5f8f4"}],"payment":{"scheme":"api-key","network":"eip155:8453","payer":"sample"}},"proof":{"type":"DataIntegrityProof","cryptosuite":"eddsa-jcs-2022","created":"2026-11-04T20:45:22.447Z","verificationMethod":"did:web:agents.writerslogic.com#key-1","proofPurpose":"assertionMethod","proofValue":"z3jLj2Xv7NFfdgjBjMnZFnwyB5ZkRyyWx6tx9ssiPLBEzbnKnSS5BoPaUrxH95EsDy1QBY63tXRJvRskuqAPZqNMH"}},"ethSignature":{"type":"EIP-191","signer":"did:pkh:eip155:8453:0x3DBd42560cA7a7E448A2c7DfC9E785526189411f","address":"0x3DBd42560cA7a7E448A2c7DfC9E785526189411f","message":"writerslogic-provenance:529be98409b8eceda723a588063923d9b5797fec0c8bba79faf3f04474ad8d18","signature":"0x54a039938341ef555844841052d137f417fa4ba8f0d54b28ebf3f2be2f7ad7863b5e03820cb3b0f8e745ad8013a6b5718e303215a37df293e1b9ebee527b252d1c"}}}