Quantifying non-deterministic drift in large language models

Claire Nicholson

doi:10.48550/arXiv.2601.19934

Research at a glance

Models evaluated: 2
Prompt categories: 5
Prompting modes: 3

Figures reported by the author in the published version. They are not independent validation or guarantees of future performance.

In brief

How repeatable are LLM responses when no stabilising prompt technique is applied? This study compares repeated outputs from two models across prompt categories, prompting modes and temperatures. It reports variation even at temperature 0.0, providing a baseline for later work on behavioural control. Its measures concern wording and output variation, rather than factual accuracy or the success of a mitigation technique.

Abstract

Large language models (LLMs) are employed for tasks from summarisation to decision support. Identical prompts do not always produce identical outputs, even when temperature settings are fixed. Here we conduct repeated-run experiments and report systematic empirical measurements that quantify baseline behavioural drift, defined as the variability observed when the same prompt is issued multiple times without stabilising instructions. We analyse how these baseline measurements relate to recent drift research. Two publicly accessible models, gpt-4o-mini and llama3.1-8b were tested on five categories of prompts under exact repeats, perturbed inputs and reuse modes at temperatures of 0.0 and 0.7. We measure unique output fractions, lexical similarity and word count variation, and compare behaviour across models. Recent literature on concept drift, behavioural drift and infrastructure-induced nondeterminism is reviewed to contextualise the results (Atil et al., 2025; Khatchadourian & Franco, 2025). We discuss the limits of lexical metrics and highlight emerging semantic metrics, and conclude with implications for baseline measurement and for future mitigation studies (Chen et al., 2024; Maxwell & Berenzweig, 2025; Khatchadourian & Franco, 2025).

Original manuscript abstract reproduced verbatim from version 1 on arXiv. The complete manuscript abstract is reproduced from arXiv’s HTML version and checked against the PDF. The arXiv catalogue summary uses different wording.

Research question and objectives

How much baseline behavioural drift occurs when ordinary prompts are repeated without operator-level stabilisation?

Measure variability across two model deployments, five prompt categories, three prompting modes and two temperatures, using lexical and word-count measures.

Methodology

Study design: Operator-free measurement study comparing gpt-4o-mini via API with llama3.1-8b running locally. No stabilisation or mitigation method was applied (§3.1).
Prompts and repeats: Five categories: underspecified summaries, underspecified advice, light constraints, style prompts and hard constraints. The paper reports 100 prompts in the gapfill dataset and 25 in the small battery, with 30 and 20 repeats respectively per model–prompt combination (§3.1–3.1.2).
Conditions: Exact repeats, small lexical perturbations and reuse of previous answers, at temperatures 0.0 and 0.7. API-error and refusal runs were excluded (§3.1).
Implementation: The paper reports API experiments in May–June 2025, a 512-token output limit and top-p of 1.0. The local condition used a single NVIDIA A100, bfloat16 precision, batch size 1 and fixed seeds. Configuration details are reported by the author rather than independently validated here (§3.1.1).
Measures: Unique output fraction, average pairwise Jaccard similarity and word-count statistics. Lexical differences do not by themselves establish a change in meaning or task performance (§3.2).
Reproducibility: Version 1 describes a planned Zenodo release of scripts, prompts, data and analysis code. It does not supply a repository DOI for those materials (§3.3). The manuscript source is available from arXiv; this is distinct from experimental code.

Results and key findings

Table 1: reported mean unique fractions and Jaccard similarities (version 1, PDF page 7)
Model	Temperature	Mode	Mean unique fraction	Mean Jaccard similarity
gpt-4o-mini	0.0	Exact repeats	0.240	0.893
gpt-4o-mini	0.0	Perturbed inputs	0.572	0.632
gpt-4o-mini	0.0	Reuse	0.200	0.971
gpt-4o-mini	0.7	Exact repeats	0.987	0.518
gpt-4o-mini	0.7	Perturbed inputs	1.000	0.440
gpt-4o-mini	0.7	Reuse	1.000	0.706
llama3.1-8b	0.0	Exact repeats	0.093	0.966
llama3.1-8b	0.0	Perturbed inputs	0.274	0.789
llama3.1-8b	0.0	Reuse	0.100	0.910
llama3.1-8b	0.7	Exact repeats	0.987	0.471
llama3.1-8b	0.7	Perturbed inputs	1.000	0.403
llama3.1-8b	0.7	Reuse	0.973	0.632

See the PDF for the original figures and full analyses: Figure 1: unique output fractions (PDF page 8), Figure 2: Jaccard similarity (PDF page 9), Figure 3: distribution by prompt (PDF page 10).

Interpretation and implications

Under exact repeats at temperature 0.0, the reported mean unique fractions are 0.240 and 0.093. These are proportions of distinct outputs within repeated-run sets; they should not be presented as error rates or as a general probability that the next answer will change.

The table reports much higher unique-output fractions at temperature 0.7. Perturbation and reuse have different effects across conditions; reuse does not uniformly reduce all measures of variability.

This is a baseline measurement study, not a test of operator prompting. It provides context for intervention studies, while differences between the two deployments cannot isolate the causal contributions of model identity, scale and serving infrastructure.

Limitations and future research

The work is a preprint; arXiv availability does not establish peer review.
Only two model deployments and five prompt categories were evaluated. The findings are not universal measurements for all LLMs or tasks (§5.6).
Unique-output and lexical-overlap measures can register harmless paraphrases and do not establish factual correctness, semantic equivalence or task success (§5.2).
The reported experiments do not establish long-horizon or multi-agent performance (§5.6).
Version 1 promises a later data and code release but does not identify its DOI. This page does not advertise unverified experimental downloads.
Some narrative descriptions and Table 1 are not fully consistent. For example, the prose says Jaccard similarity falls below 0.52 for every temperature-0.7 mode, while Table 1 reports reuse values of 0.706 and 0.632. The values reproduced above follow the published table without silently correcting the paper.

Paper and research materials

arXiv archival recordMetadata and original deposited file
Full-text PDF · Version 1Searchable text · 0.37 MB · Local copy of the archived file
Permanent DOI link10.48550/arXiv.2601.19934
Citation and bibliographyCopy a citation or download BibTeX
Read the original paper in HTMLarXiv experimental HTML rendering
Manuscript sourcearXiv TeX source; not the experimental analysis code

Version 1 §3.3 describes a planned Zenodo release of experimental scripts, prompts and data, but supplies no DOI for it. The arXiv manuscript source is available; no separate experimental dataset or code download is claimed here.

Local PDF copied unchanged from the version 1 arXiv PDF; its SHA-256 checksum is recorded with the source metadata. Author: Claire Nicholson. Reproduced under Creative Commons Attribution 4.0. The PDF and abstract are reproduced unchanged; the explanatory text on this page is an editorial summary.

Citation and publication information

Cite this version

Nicholson, C. (2026). Quantifying non-deterministic drift in large language models (Version 1) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2601.19934

BibTeX

@misc{nicholson2026nondeterministic,
  author = {Nicholson, Claire},
  title = {Quantifying non-deterministic drift in large language models},
  year = {2026},
  month = {jan},
  version = {1},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2601.19934},
  url = {https://doi.org/10.48550/arXiv.2601.19934},
  eprint = {2601.19934},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  note = {Preprint; not peer-reviewed}
}

Download .bib →

Version history

Version 1 · 12 January 2026 · Version described on this page
10.48550/arXiv.2601.19934

Version identifiers and dates come from arXiv. This page does not infer revision details where the repository does not supply them. Published versions remain available through the archival links.

Licence and disclosures

CC BY 4.0, as listed in the archival record.

The author is affiliated with HelixScribe.AI. Appendix B states that generative AI was not used to draft or edit the manuscript or decide the experimental design or analysis; LLM use was limited to routine tasks such as debugging code and formatting scripts. This page reports that disclosure without inferring independent validation or external peer review.

References

The complete bibliography is in the published PDF. This page summarises that paper and does not present its references as separately reviewed evidence.