Memory Without an LLM in the Loop

Eliminating external language models from the memory write path using local zero-shot ONNX models, FastEmbed, and deterministic civil date resolution to build sovereign, cost-predictable agent memory.

By The ebbmem team at Spectrayan
In-process ONNX memory ingestion pipeline showing dialogue turns processed locally through GLiNER, GLiREL, FastEmbed, and deterministic temporal resolution into an AES-256-GCM encrypted slab without external LLM API calls
By default, ebbmem executes entity recognition, relation scoring, temporal anchoring, and dense embedding in-process on standard CPU nodes.
On this page (8 sections)

Autonomous AI agents require persistent context across sessions to remain useful. Conventional memory solutions achieve persistence by inserting a cloud language model into the write path: packaging every turn, tool observation, or state update into an external prompt to extract structured JSON facts.

While functional for prototypes, this approach creates compounding API expenses, network latency delays, prompt drift, and privacy risks from transmitting raw conversations to third-party endpoints.

At Spectrayan, we engineered ebbmem to eliminate the language model from the memory write path. In its default configuration, ebbmem performs entity parsing, relation extraction, and vector embedding in-process using local zero-shot ONNX models and deterministic date resolution. Decoupling ingestion from model APIs makes agent context cheaper, private, reproducible, and operationally predictable.

1. The Hidden Tax of LLM-Driven Memory Extraction

In agent workflows, memory ingestion occurs continuously in the background as agents converse, observe tools, and execute tasks.

When ingestion relies on an external language model, every write operation incurs three operational bottlenecks:

  • Compounding Token Expenses: A single workflow may trigger dozens of state updates. If every turn requires an external model prompt to distill facts, ingestion token costs can quickly rival the cost of the agent's primary reasoning steps.
  • Network Round-Trip Delays: Transmitting prompts to cloud APIs, waiting in inference queues, and receiving completions adds latency. Even with asynchronous queues, backlog delays mean updated preferences are not immediately accessible for subsequent turns.
  • Data Egress Risks: Sending conversational data to third-party providers expands your compliance perimeter. Every external write represents data egress requiring data processing agreements and security reviews.

This pattern couples memory ingestion directly to third-party availability and billing. As an agent fleet scales, operational expenses multiply with message volume rather than underlying infrastructure.

2. Why Local Deterministic Extraction Makes Memory Cheaper

The primary operational benefit of eliminating the language model from the write path is removing the per-write model API bill.

In ebbmem, the ingestion pipeline runs locally on your host infrastructure. When an agent records a dialogue turn or tool output, ebbmem extracts entities, relationships, and temporal references using local zero-shot ONNX models and local embeddings running directly on the host CPU.

By replacing external API round-trips with in-process execution, memory ingestion transitions from a metered, variable expense into predictable infrastructure overhead. You can ingest high-frequency conversational turns and telemetry logs without monitoring token balances or anticipating invoice spikes.

Decoupling ingestion from language model billing fundamentally alters agent system economics. Autonomous fleets can continuously observe, index, and organize operational context without treating memory storage as a scarce, cost-prohibitive resource.

3. Privacy by Design: Keeping Conversational Data Inside Your Perimeter

Enterprise agent deployments face strict data protection obligations under GDPR, CCPA, and corporate governance standards. When memory systems depend on cloud language models for fact extraction, confidential dialogue, proprietary code, and personal identifiers cross organizational boundaries to third-party APIs.

ebbmem enforces privacy by design across the extraction lifecycle:

  • Zero Ingestion Data Egress: Entity parsing, relation extraction, and dense vector embeddings execute locally in-process. Raw text never leaves your host or container perimeter during extraction.
  • Per-Namespace Envelope Encryption: Memory traces, graph edges, and vector indices are sealed using authenticated AES-256-GCM envelope encryption. Plaintext Data Encryption Keys (DEKs) exist only in volatile memory and are zeroized on drop.
  • Customer Key Ownership: Encryption integrates with enterprise key management services, including AWS KMS, Google Cloud KMS, and HashiCorp Vault.
  • Verifiable Erasure: When deletion is requested, ebbmem physically zeroes memory slab buffers and issues a tamper-evident cryptographic receipt, providing auditable proof of physical destruction.

Keeping fact extraction inside your infrastructure perimeter satisfies data residency requirements and removes third-party data processor dependencies on the ingestion path.

4. Reproducibility & Predictability: The Case for Local Deterministic Parsing

Prompt-based memory extraction suffers from non-deterministic behavior. Large language models are generative systems designed for creative variation rather than strict software contracts.

When using prompt-based extraction, three recurring failure modes emerge:

  • Prompt Drift and Checkpoint Deprecation: Cloud providers update model checkpoints over time, causing identical prompts to yield altered keys or shifted confidence scores.
  • Formatting and Temperature Noise: Generative models can emit malformed JSON, add markdown fences, or omit required fields across turns.
  • Hallucinated Relationships: Prompted models often invent associative links that were never stated in the source dialogue, polluting downstream retrieval.

ebbmem solves this using specialized in-process models and deterministic algorithms:

  • GLiNER for Entity Recognition: Matches spans directly against entity types without prompting, using bidirectional attention to produce consistent boundaries and scores.
  • GLiREL for Relation Extraction: Scores relationship triples between extracted entity pairs, enforcing strict relational schemas without generative hallucination.
  • FastEmbed for Local Embeddings: Generates dense representations locally, ensuring identical text yields identical vector coordinates.
  • Deterministic Temporal Resolution: Resolves relative expressions (such as "yesterday" or "next Tuesday") using civil calendar date algorithms, mapping turn timestamps to exact civil dates (YYYY-MM-DD) without model arithmetic.

The Boundaries of Reproducibility

Technical honesty requires clarifying where reproducibility holds and where boundaries lie:

  • Where reproducibility holds: Given identical text, turn timestamps, and configuration weights, ebbmem's local ONNX and heuristic pipelines produce identical entity spans, relational triples, temporal resolutions, and vector embeddings across runs.
  • Where boundaries lie: Small zero-shot models excel at extracting explicitly stated facts, entities, relationships, and temporal anchors. They do not perform speculative multi-hop synthesis or psychological subtext analysis across months of history. For architectures requiring open-ended narrative synthesis, generative language models remain appropriate.

5. Honest Disclosure of Optional LLM Mode

While operating without a language model in the loop is ebbmem's default and recommended architecture, certain agent workflows benefit from generative capabilities.

For teams that specifically require open-ended narrative summarization, complex episodic reflection, or unstructured dialogue consolidation, ebbmem provides an optional language model extraction backend.

This mode is strictly opt-in and configured via explicit properties:

  • extractor.mode: "llm" activates structured JSON extraction using a configured language model provider.
  • extractor.require_llm: true requires an active language model, failing closed if the provider is unavailable.
  • reflect.require_llm: true requires a language model specifically for periodic background consolidation of episodic observations into semantic summaries.

ebbmem supports pluggable providers for this mode, including self-hosted models via Ollama, private inference endpoints via OpenAI-compatible APIs, and managed cloud services such as Google Gemini.

By offering this as a configuration toggle rather than an architectural requirement, ebbmem allows teams to operate entirely local and deterministic by default, enabling language model processing only where generative reasoning is explicitly desired.

6. The In-Process Architectural Reality: Running on Standard CPU Nodes

A common misconception is that local machine learning models require dedicated GPU clusters.

ebbmem is engineered to run efficiently on standard CPU hardware:

  • Quantized ONNX Weights: The default entity extraction pipeline uses quantized ONNX models (model_quantized.onnx), executing transformer inference on standard CPU cores via optimized ONNX Runtime execution providers.
  • Compact Embedding Dimensions: Local dense embeddings utilize compact vector spaces, minimizing computational overhead while delivering reliable semantic retrieval.
  • Unified Slab Storage: Rather than requiring separate vector, graph, and relational database clusters, ebbmem persists memory in a unified single-file slab format (bundle.mem).
  • Flexible Deployment Modalities: ebbmem embeds directly into application processes as a native Python extension (pip install ebbmem) or Rust crate (ebb-memory), runs as a standalone container via Docker Compose, or deploys into Kubernetes clusters via Helm charts.

Because models and storage operate in-process on standard virtual machine nodes, you avoid provisioning GPU instances or managing distributed database sprawl.

7. Conclusion: Sovereign, Predictable Memory for Autonomous Systems

Autonomous systems require reliable, cost-effective, and sovereign memory infrastructure. Relying on external language model API calls on every conversational write creates financial, latency, and compliance liabilities.

By establishing local zero-shot extraction as the default, ebbmem provides cost stability with zero per-write model API tokens, data sovereignty with zero third-party data egress, engineering predictability without prompt drift, and operational simplicity on standard CPU hardware with an embedded control plane store.

Agent memory should serve as a stable foundation for autonomous intelligence, not an unpredictable prompt chain.

To explore the architecture or test local in-process memory in your environment, review the documentation or launch the local Docker Compose configuration today.

Continue Reading

Two Kinds of “Dreaming” in Agent Memory: Offline Consolidation vs. Our Experimental Dream Pathway

Why the AI agent ecosystem currently uses “dreaming” for two distinct mechanisms—offline transcript consolidation and generative hypothesis construction—and why our experimental Dream pathway enforces strict source honesty.