entroly

Competitive evidence matrix

This protocol measures Entroly against the current released External Baseline A package without collapsing unlike properties into a promotional score. Quality, recovery, latency, provider behavior, reliability, security, packaging, cost, and operator experience are reported separately. A win in one dimension cannot hide a loss in another.

The current additive index is benchmarks/competitive_evidence_protocol_v2.json. The original competitive_evidence_protocol.json remains immutable because recovery artifacts embed it. Later dimensions use their own immutable protocol files rather than rewriting prior evidence. Thresholds and holdout parameters are frozen before results are inspected. The current 1.0.60 durability revalidation is frozen separately in recovery_resilience_protocol_v3.json. It preserves the immutable v2 protocol, its tie result, and the original 1.0.59 protocol and artifact.

Claim rules

• Preserve every failed, timed-out, missing, and incorrect row.
• Publish development losses as engineering evidence, not as product wins.
• Never replace achieved measurements with requested targets.
• Never claim universal superiority; claims name the system versions, workload, boundary, metric, and caveat.
• Correctness and data integrity cannot be traded for compression or latency.
• Provider cost claims require provider-observed usage and immutable pricing or invoice provenance.

Evaluation dimensions

Dimension	Required evidence	Current state
Active-context quality	matched caps, exact outputs, paired statistics	implemented
Recovery resilience	restart replay, concurrent writers, exact bytes	implemented
End-to-end model recovery	model-triggered retrieval and final answer	implemented
Compression latency	warm/cold p50 and p95 by content type	implemented
Provider conformance	Anthropic, OpenAI Chat/Responses, Gemini shapes	planned
Interruption recovery	crash, retry, replay, idempotency	planned
Security	secret logs, marker injection, path and tenant isolation	planned
Packaging and first run	clean install, doctor, wrap, rollback	planned
Operator UX	actionable errors, dashboard truth, recovery steps	planned
Provider-observed cost	paired usage and pricing provenance	planned

Recovery-resilience suite

The first suite stresses the local source-of-truth store behind reversible compression. Independent processes write unique omitted evidence concurrently, exit, and a fresh process attempts to recover every payload by its emitted reference.

The development matrix uses four writers with eight entries each. It is allowed to reveal defects and guide implementation. The frozen holdout uses six writers with eleven entries each and is run only after the repair.

A participant passes only when:

1. every worker exits successfully;
2. every intended write returns a reference;
3. a new process recovers every reference after the writers exit;
4. every recovered payload matches its expected SHA-256; and
5. no wrong payload is ever returned.

Store and retrieval latency are descriptive. A faster system that loses or misroutes one payload fails.

python -m benchmarks.recovery_resilience run `
  --phase development `
  --External Baseline A-python C:\path\to\External Baseline A-0.31.0\Scripts\python.exe `
  --output benchmarks/results/recovery_resilience_development.json

python -m benchmarks.recovery_resilience verify `
  benchmarks/results/recovery_resilience_holdout.json

The JSON artifact includes the frozen protocol hash, package and implementation identities, complete entry matrix, worker errors and exit codes, exact recovered hashes, latency samples, state-file sizes, aggregate gates, and a canonical payload hash.

Recorded recovery results

The development run found a real Entroly data-loss bug. With four pre-opened writers, only 8 of 32 intended entries survived restart byte-exactly; External Baseline A recovered 32 of 32. Three Entroly workers failed. This unfavorable result is retained in recovery_resilience_development_before.json.

Entroly then added cross-process read/merge/write serialization, unique durable temporary files, file and directory synchronization, stale-reader refresh, and process-level regression tests. The frozen holdout was not changed.

The v2 fresh-seed Windows/Python 3.10 revalidation tied at 66/66 byte-exact recoveries. After complete-line preservation and relevance-before-fit changed the Entroly implementation hash, v3 froze a new seed; on that run Entroly recovered 66/66 while published External Baseline A 0.31.0 recovered 55/66 — one of six writers exited with SQLite OperationalError: database is locked. A later docstring-only source cleanup changed the Entroly implementation hash again, so v4 froze a new seed and re-ran. On v4 both systems wrote and recovered 66/66 payloads byte-exactly, with zero worker errors and no incorrect payloads: the v3 competitor failure was a transient store lock that a clean re-run did not reproduce. The current result is parity — both satisfy the frozen integrity gate — and neither the tie nor the transient failure generalizes to universal recovery superiority.

Holdout metric (v4)	Entroly 1.0.65 source	External Baseline A 0.31.0
Successful writes	66 / 66	66 / 66
Byte-exact restart recovery	66 / 66	66 / 66
Worker errors	0	0
Incorrect payloads	0	0
Successful store-call p50 / p95	34.848 / 657.689 ms	1.524 / 24.545 ms
Retrieval p50 / p95	0.059 / 0.075 ms	0.441 / 0.698 ms
Live state files	95,438 bytes	1,626,736 bytes

These latency and size observations describe this workload and platform; they are not standalone product-superiority claims. External Baseline A’s successful calls were substantially faster for writes. Entroly was faster for retrieval and used less live state. External Baseline A used SQLite WAL with synchronous=NORMAL; Entroly fsynced its state file on every commit and its parent directory where supported. The suite did not simulate machine power loss, so the store-call latency is not a matched power-loss-durability comparison. The complete samples, environment identities, hashes, and errors are in the current v4 revalidation. The immutable prior v3 run and prior v2 tie remain available. The original post-repair holdout artifact remains unchanged. The post-repair development iterations remain available as first repair and Windows lock optimization evidence.

Recorded compression-latency result

The quality-gated Windows/Python 3.10 holdout measured Entroly 1.0.59 source as 2.94x faster than External Baseline A 0.31.0 for warm public compressor calls (95% stratified bootstrap CI 2.74x to 3.13x) and 2.39x faster for import plus the first call in a fresh process (1.89x to 2.70x). Both participants completed all fixtures, retained every preregistered evidence needle, remained deterministic, and never inflated tokens. See the protocol, samples, and limits.

Recorded model-triggered recovery result

On the frozen 24-case synthetic query-shift holdout, raw context and Entroly both scored 24/24 final exact answers; External Baseline A scored 18/24. All six discordant pairs favored Entroly (two-sided exact McNemar p = 0.03125). Entroly’s mean effective-context ratio was 28.88%, including source-exact recovery evidence on all 24 triggered retries, versus 42.97% for External Baseline A.

This is a scoped result for a local qwen2.5:1.5b workflow guard, not a frontier-model or universal-agent claim. The full protocol evolution—including timeouts, an over-budget variant, an incomplete-JSON failure, a stale metadata label, and a rejected low-token projection—is retained in the model-triggered recovery report.

This site is open source. Improve this page.