How this works, and what we do not know.

Dynamic keeps the legacy 16k resident ceiling. For long cold or resumed histories, start a matched comparison with adaptive lexical. Adaptive HDC adds reranking; it is not a proven upgrade. Both adaptive profiles use a 32,768-token resident ceiling and a 4,096 estimated-token incidental recent tail, not a 4k final prompt.

Recognized required records, the current objective and tool exchange take priority. Missing or oversized recognized evidence is refused in live mode. Syntax checks do not prove semantic completeness. Use your real task-quality gate.

After installation, requests pass through the sidecar but original prompts are forwarded unchanged. Assembly and local tokenization measure what the selected policy would keep. Shadow adds work, even though it does not change model input.

Every bundle starts in shadow. A saved live target in this portal never switches your running host. Short smoke requests are not representative calibration.

On your Linux GPU host: agent → CPU-side memory proxy on 127.0.0.1:8088 → your vLLM. Model weights stay unchanged. A separate process, with no credentials passed in its environment, serves the reporting dashboard on 127.0.0.1:8089. No runtime telemetry goes to Engrammatic.

The proxy publishes a fixed-size aggregate snapshot every two seconds from a background thread. No reporting I/O is on the generation request path; it does not scan or lock session memory. Reporter failure leaves serving running and makes the dashboard unavailable. Resource overhead still needs to be measured on your host.

The bundled launcher uses one Uvicorn worker. The limiter is process-local; multiple sidecars pointed at the same vLLM each have their own limit. Keep one admission domain or provide a shared external gate.

On the GPU host, open http://127.0.0.1:8089. From your laptop run ssh -N -L 8089:127.0.0.1:8089 USER@GPU_HOST, then open that same local URL. For another GPU host choose another local tunnel port. Never publish port 8089.

It shows sampled request counts, upstream reachability, calibration, profile and estimated prompt reduction. Counts are requests, not agent sessions. No current snapshot after ten seconds means reporting unavailable, not a healthy model.

Run representative shadow traffic until local metrics or the dashboard gives recommended_live_token_estimate_ratio_floor. Set it as ENGR_TOKEN_ESTIMATE_RATIO_FLOOR. The recommendation uses the largest observed exact-to-estimated ratio plus a 5% test buffer, rounded upward to three decimals. It is an efficiency prior, not the safety proof: every live candidate is exact-tokenized again.

Confirm your loaded vLLM uses its generic chat renderer, then set ENGR_TOKEN_PREFLIGHT_RENDERER=vllm-generic-chat. Live startup refuses a missing or invalid floor. If the buffered floor cannot fit 0.5–3.0, remain in shadow and investigate rather than guessing.

Set a positive max_tokens or max_completion_tokens on every request. The per-request resident cap, not a prompt target, covers exact prompt plus bounded output for every n/best_of candidate. The proxy reassembles to leave headroom; if it cannot fit or exact preflight is unavailable, it makes no generation call. Missing or mismatched generation usage opens the live circuit and fails /healthz until you investigate compatibility and restart.

GPT-OSS/Harmony, Mistral-specialized rendering, multimodal or documents inputs, and tools with explicit tool_choice=none are not covered by this live integration.

Start at one complete response at a time, including the stream. Test 1, 2, 4, 8, then 16 only after each previous gate passes. The default queue holds up to 32 requests for 300 seconds; overflow/timeouts return 503 with Retry-After. The sidecar cannot see free GPU memory, cannot make one oversized request fit, or govern bypass traffic. Rewriting can lose shared-prefix cache reuse and increase transient prefill pressure.

A modelled range from no improvement to a separate illustrative 2× capacity hypothesis, limited by the selected resident ceiling’s KV-memory model. This is not a measured guarantee, safety bound or confidence interval. Modelled resident sessions per replica are not simultaneous model requests or completed tasks per hour.

Published parameter counts, assumed weight precision and KV bytes determine the presets. Validate all of them for your actual server. A model sharded over several GPUs has one replica capacity; do not assign that capacity to every card.

A runnable source bundle: CPU-side proxy, local dashboard, Dockerfile, compose, your selected profile, quickstart and README. It is not in a public registry yet. The build needs its base-image registry and Python package index unless cached; this tarball is not an offline OCI image. The runtime makes no callback to us.

Only text-only /v1/chat/completions on a compatible self-hosted vLLM generic renderer. Other JSON routes, including the Responses API, are best-effort passthrough without memory treatment or a compatibility guarantee. Validate your exact client.

For a cold or resumed history the sidecar's shorter prompt means less prefill, so time to first token usually falls. For a warm continuation the native server may already reuse its cached prefix, and a rewritten prompt loses that reuse, so the same request can be slower. There is no automatic controller that protects warm native prefixes: if warm continuations dominate your traffic, route them to the model directly from the client and send only cold or resumed sessions through the sidecar, then measure both. It can destroy useful prefix-cache reuse. Selection and exact tokenization also take CPU and time. Measure cold/resumed histories separately from sustained warm turns, including p95 time to first token, end-to-end completion and accepted tasks per GPU-hour.

Freeze representative held-out tasks, correctness tests and latency limits before tuning. Compare native serving, your existing compaction, adaptive lexical and adaptive HDC on the same hardware/model/flags and output allowance. Counterbalance order and repeat cold and warm multi-turn sessions. A successful HTTP response is not task acceptance.

License only when the chosen profile passes quality, safety, latency and total-cost gates, including a rollback drill. The conditional 2 → 16 goal is a joint optimization target; stop at the highest validated ceiling, even if it is below sixteen.

Selection uses CPU and host RAM, not a separate GPU. Budget and measure them. Dynamic mode retains counters and a content-hash encode cache, not a prompt checkpoint. Adaptive/epochal retain one bounded in-memory checkpoint per session: 512 KiB per session and 256 MiB total by default. Reuse checks the full-history digest and policy.

Session TTL is lazy eviction on new-session creation, not guaranteed timed deletion. Restart discards process state. Keep full histories in your client, never reuse a session identifier across conversations, and use separate processes for separate trust domains.

No. Fewer required cards can release capacity or avoid a future purchase. Cash savings require reducing cancellable billed hours or an actual deferred purchase. Owned, prepaid and committed GPUs may cost exactly the same after a pilot. Include annual licence, CPU/RAM, storage, operations and failures in cost per accepted task.

One auto-renewing annual licence for each GPU card in every model replica whose traffic is served through the sidecar, charged in advance at the published card-class rate. Evaluation is free. You declare the actual deployment count; the portal does not meter your hosts.

Once a licence is active, changing the declared quantity updates the Stripe subscription first and creates its normal prorated adjustment; a change before any licence exists changes only the quote. An invoiced licence carries the same conditions. Cancel in Stripe before renewal to prevent the next charge. Contact us to change card class. No usage component or overage. The minimum while subscribed is every GPU card in one served replica: one card for a single-card replica, all twelve for a twelve-card sharded replica ($24,000 a year for one such H100 replica). Declare only the cards whose traffic goes through the sidecar; four of eight cards migrated means four.

Platform bundling, OEM/resale rights and support/SLA terms need a separate agreement.

Everything in the bundle, in shadow and in live mode, on as many hosts as you like, with no time limit; the bundle does not expire and phones nothing home. Support during evaluation is by email, best effort.

Two self-serve paid terms, the same product and the same support boundary in both. A 90-day evaluation licence priced at a quarter of the card's annual rate, charged in advance, that does not auto-renew: $500 for an H100 80GB card, $312 for an A100 80GB, and so on down the same rate card the annual licence uses. If you take a 12-month licence within 30 days of the pilot ending, the whole of what you paid is credited against that invoice, so a pilot that converts costs no more than going annual on day one. Or the 12-month licence ($2,000 per H100 card), charged in advance, auto-renewing until cancelled in Stripe before renewal. Neither has a mid-term cash refund; reducing the count creates a prorated credit against the next invoice. For a term that is neither of those, ask for an invoice quote before you start. Prices are in USD; applicable taxes are added by Stripe at checkout or on the invoice.

The declared count is on trust: we cannot see your fleet. Update it when your deployment changes; there is no audit or true-up at renewal, only the count you last declared.

By the X-Engrammatic-Session request header when your client sends one (up to 128 characters; one identifier per conversation, never reused). Without it, the sidecar derives a key from the system prompt plus the first user turn and returns it in the response header as auto-…. Two conversations that share both would share a checkpoint, so send the header from any client that opens many conversations with the same opening.

Every response carries X-Engrammatic-Route (passthrough, compact, reuse, refresh or abstain), X-Engrammatic-Recalled (the indices of the history messages quoted as evidence, counting from zero as your client sent them), X-Engrammatic-Tail-Turns (recent messages kept verbatim) and X-Engrammatic-Evidence-Passages; the bundle README lists every header with its unit. The installed dashboard shows the same fields for the last twenty requests under Recent requests. No page, header or snapshot ever shows the text itself.

The selector could not assemble a working set it can justify for that request: either the objective names a revision record that is missing or too large to fit (explicit_entity_without_revision_record, required_evidence_does_not_fit), or nothing in the history overlapped the current objective and latest tool observation at all (no_retrieval_evidence). In shadow this is only reported: the request is forwarded unchanged and the dashboard counts it as would refuse. In live the sidecar refuses with HTTP 422 before any generation call rather than sending the model a prompt it cannot justify.

Remedy: route that request natively from the client (the original full history is still there), and give the objective the concrete names it needs. Histories with no natural-language overlap, such as synthetic text or pure JSON transcripts, abstain far more often than real agent sessions; measure the rate in shadow on your own traffic (the dashboard's selection counts) before going live, and treat a high rate as a compatibility finding, not a quality one.

Yes, for some request shapes, and shadow now tells you before you switch. Live exact-counts every prompt through vLLM's own tokenizer, so it will only send a request whose rendering it can reproduce. Three shapes it will not vouch for: a non-empty tools list sent with tool_choice: "none" (vLLM may drop the tools at generation, so the count would be of a different prompt), documents, and a few model-specific fields such as reasoning_effort on models that ignore it. Live answers those with HTTP 422 token_preflight_unsupported_request before any generation call; the message names the remedy.

Shadow forwards every one of them unchanged and answers 200, which is what makes this worth saying: a pilot can look perfectly healthy and then lose that share of its traffic at the switch. Shadow counts them as shadow_live_incompatible_requests in /metrics and raises a configuration_warnings line on /healthz naming the share and the code. Check that it is zero before you go live. If it is not, the fix is in the client: send the tools with auto or a named choice, or omit the tools list, rather than none.

No, and the dashboard's headline can flatter it. The reduction is largest on the opening rounds of a resumed session, when a long history is replaced by a small working set. As the session continues the sidecar keeps the recent tail in full, so the prompt it sends grows while the original history grows slowly, and the per-round reduction falls: buyers measuring their own sessions have seen it go from about 92% on the first round to about 76% by the tenth. The dashboard's headline is cumulative and is dominated by those opening rounds.

So do not project a monthly saving from one request or from the headline. Read the per-request rows (or the X-Engrammatic-Preflight-Prompt-Tokens header) across a whole session of your own, and take the average over the session length you actually run. The effect is still largest exactly where it matters most, on cold and resumed sessions, which is where Help tells you to route.

Two endpoints on the host. /healthz is liveness: it answers 200 whenever the proxy process can serve, with ready, not_ready_reasons, upstream_ok, live_ready and the reporting fields in the body; the container healthcheck uses it, so a model outage never marks the sidecar unhealthy. /readyz is readiness: in shadow, 200 when the model is reachable; in live, 200 only when the model is reachable, calibration and the renderer attestation are in place and the live circuit is closed, else 503 with the reasons (the body's readiness_rule says which rule applied). Page on /readyz in live mode. Inside the body, upstream_ok=false means your model is unreachable (page your model server, not the sidecar); live_ready=false in live mode means the exact-token circuit is open (investigate compatibility, then restart the container); reporting_state other than current means the dashboard is stale or down, and reporting_detail says why (serving is unaffected; the reporter restarts itself up to five times an hour). Read reporting_state for exactly what it measures: the freshness of the snapshot, which the proxy writes and the dashboard process only serves. A reporter that has been killed shows within half a second, but one that is merely hung (stopped, or wedged) leaves the snapshot fresh, so the field stays current while the dashboard port itself times out. That is a true statement about your data and an incomplete one about your dashboard: if you want to alert on the dashboard being reachable, probe its port directly. Nothing about it should page anyone at night. Rejections with 503 and Retry-After under load are the admission limit working, not a fault: raise it only after the stage test.

vLLM 0.11.x with the generic chat renderer; 0.11.2 is the version the exact-count preflight was tested against. Other servers and versions are untested: shadow may run, live refuses when /tokenize does not answer the chat shape.

Docker is the supported path. For an evaluation on a host you control you can run the same entrypoint from a Python 3.11 virtual environment with the four pinned packages from the Dockerfile: PYTHONPATH=. ENGR_UPSTREAM_BASE_URL=… ENGR_MODE=shadow ENGR_MEMORY_MODE=… ENGR_ADAPTIVE_RECENT_TOKEN_CAP=… ENGR_ADAPTIVE_RETRIEVAL=… ENGR_WORKING_SET_TOKENS=… python -m engrammatic.sidecar_reporting. Pass every profile variable explicitly: the raw entrypoint defaults to dynamic and a 16,000-token ceiling, and the quickstart's checks (key readability, health, the smoke request) are then yours to do.

They are cut: the client sees a closed connection or a truncated stream, and in-memory checkpoints and counters are gone on restart. For a clean rollback move clients to the native URL first, let their in-flight responses finish, then stop the container. When the live circuit opens, responses already generating complete; queued and later live requests are refused with 503 until you investigate and restart.

One sidecar per model replica. Two replicas on one host are two sidecars: start the second with ENGR_PORT=8090 ENGR_REPORT_PORT=8091 ENGR_CONTAINER_NAME=engrammatic-sidecar-2 and its own ENGR_UPSTREAM_BASE_URL (the README has the full recipe). The admission limit is per process, so two sidecars in front of one vLLM would each admit their own limit. A two-card replica is one sidecar and two licences. Route each client to its replica's sidecar with your own ingress; the sidecar does not load-balance.

Per arm (native, your compaction, lexical, HDC) at the same admission level: (GPU-hours × your rate + sidecar host CPU/RAM + licence ÷ 8,760 × hours + operations) ÷ accepted tasks, where accepted means the task's own tests and constraints passed, not that HTTP returned 200. Report cold/resumed and warm strata separately, count censored or refused tasks in the denominator of offered work, and only count GPU-hours you can actually cancel or defer as savings.

The portal and Fleet Audit are configuration and sizing tools, not your runtime telemetry. Their capacity and money figures are modelled. The local dashboard marks estimated prompt counts separately from exact tokenizer counts; neither proves GPU-hour savings. A buying decision needs matched task measurements on your own stack.

There is intentionally no telemetry channel from your host to Engrammatic. Saved settings apply to future downloads only. Open the installed local dashboard, through an SSH tunnel if needed, to see runtime observations.

Restore the agent’s original model URL and original full history first, with native admission appropriate for that full context. Then stop/remove the sidecar container. No model weights change and no server schema needs migrating. Do not silently expand a refused compact request into the full prompt on the same live limit.

Conversation text is not written to disk by the sidecar. Reporting writes only a bounded aggregate snapshot in a private temporary directory, removed on normal shutdown. Abrupt host failure may leave that aggregate file; remove it during host cleanup. Disable host swap/core dumps if your policy prohibits operating-system copies of process memory.