entroly

One measured job of the Context OS

Entroly selected 1.02 of 16 candidate passages on average while keeping the answer-bearing passage in 298 of 300 held-out retrieval questions

Entroly is a local Context OS spanning selection, compression, memory, recovery, verification, provider controls, receipts, security, and guarded learning. This benchmark isolates one mechanism in that larger system: selecting the answer-bearing evidence before a model request.

From 16 candidate passages, Entroly selected 1.02 on average while keeping the answer-bearing passage in 298 / 300 held-out questions. It does not claim that this test proves better generated answers or lower production bills.

The mechanism is deliberately secondary to the value. BM25 found the passage in 297 / 300 questions and the local transformer found it in 293 / 300. They disagreed six times, so the guard retained both candidates only on those cases. That point difference is useful engineering evidence, not a statistically conclusive improvement.

What was frozen

Held-out result

Selector Top-1 answer-passage recall Top-2 recall MRR
Deterministic BM25 99.0% 99.7% 0.9937
Local transformer 97.7% 99.7% 0.9870
Dual-channel disagreement guard 99.3% n/a n/a

The transformer was uniquely correct once; BM25 was uniquely correct five times. Their paired two-sided exact McNemar value was p=0.21875, so the benchmark does not support a statistically significant primary-selector claim. The guard selected both passages for the six disagreements and reached 298 / 300 answer-passage recall, equivalent to 15.69x passage-count compression in this one-of-16 protocol.

This experiment measures retrieval of the paragraph containing a known answer. It does not measure generated-answer quality, token compression, production latency, billing savings, or universal neural-versus-lexical superiority.

Audit every trial in under a minute

From a source checkout:

python -m benchmarks.neural_evidence_frontier verify \
  benchmarks/results/neural_evidence_frontier.json

The verifier recomputes correctness, recall, MRR, disagreement counts, the exact McNemar value, the guard result, and the headline gate from all 600 raw trial rows. It fails if the committed aggregates or statistical conclusion do not match those rows. This audits the committed experiment; it does not rerun the transformer.

Rerun the encoder experiment

Materialize the exact model revision and SQuAD v2 dataset in the local Hugging Face cache first. The runner is deliberately offline-only and refuses an implicit download. Then run:

python -m pip install -e ".[benchmark]"
python -m benchmarks.neural_evidence_frontier run \
  --model-path /path/to/c9745ed1d9f207416be6d2e6f8de32d1f16199bf \
  --model-repository sentence-transformers/all-MiniLM-L6-v2 \
  --model-revision c9745ed1d9f207416be6d2e6f8de32d1f16199bf \
  --implementation-commit "$(git rev-parse HEAD)" \
  --trials 600 --distractors 15 --seed 73021 \
  --calibration-fraction 0.5 \
  --max-override-error-upper 0.10 --minimum-overrides 40 \
  --output .tmp/neural_evidence_frontier.json \
  --markdown .tmp/neural_evidence_frontier.md \
  --svg .tmp/neural_evidence_frontier.svg

Compare the dataset fingerprint, model-tree fingerprint, trial rows, and metrics with the committed artifact. Platform-specific floating-point behavior should be investigated rather than silently normalized away.

Try to falsify it

The most valuable response is not a star or a repost. It is a corpus where the result changes. Run lexical, neural, and disagreement-gated selection on your own evidence workload and submit a Benchmark or Evidence Report. Include the corpus or immutable hashes, selector versions, budget, trial-level outcomes, and failure cases. Entroly will publish credible counterevidence next to favorable evidence.

This site is open source. Improve this page.