LLMs Are Starting to Separate Knowledge Storage From Reasoning Compute

The fresh signal

Intern-S2-Mobius makes the knowledge–reasoning split explicit

Intern-S2-Mobius, released in mid-August, is interesting less because it is another 35-billion-parameter model than because of how its authors reorganize the Transformer. In the Mobius-v0 design, feed-forward computation associated with knowledge storage is consolidated into a globally shared Memory, while multiple self-attention-based Reasoners iteratively query that shared knowledge through hidden states.

The authors describe two consequences. First, a reasoning stage is no longer limited to the knowledge encoded in its local layer’s feed-forward block; shallow and deep stages can access the same shared repository. Second, part of the model’s deliberation happens through recurrent updates to latent hidden states rather than through a long visible chain of generated reasoning tokens.

The technical report says a 7B Mobius model trained from scratch reached a similar downstream score to a 7B Transformer baseline using 62.6% of the baseline’s training data. The released Intern-S2-Mobius model was continually pretrained from Qwen3.5-35B and, according to the authors, achieved similar downstream performance with nearly four times the end-to-end inference speed in their reported setup. Those are author-reported results, not independently reproduced measurements, and the speed gain is tied partly to shorter reasoning traces as well as the architecture itself.

The release should therefore not be read as proof that the Transformer has been replaced. It is a useful signal that a major model team now sees knowledge storage and reasoning computation as separable design objects.

This is also distinct from AIUpdateWatch’s recent analysis of Intern-S2 task scaling. Task scaling changes the diversity and difficulty of the training and reinforcement-learning environment. Mobius changes the internal machinery that carries knowledge and performs reasoning.

Conventional Transformers make the same stack remember and reason

A standard decoder Transformer alternates attention blocks with multilayer perceptron or feed-forward blocks. The division of labor is not perfectly clean, but a large body of interpretability and architecture work treats feed-forward networks as an important form of parametric memory: they help encode recurring associations, facts and local patterns. Attention, meanwhile, dynamically mixes information according to the current context.

That architecture is elegant because every layer has the same basic recipe. It is also potentially wasteful. A model may repeatedly spend dense matrix-multiplication capacity reconstructing static information that behaves more like a lookup than a new computation. The same parameters that make long-tail knowledge available also consume accelerator memory and bandwidth on every request, whether a prompt needs that knowledge or not.

Mixture-of-experts models already loosen one part of this constraint. They store a large number of feed-forward experts but route each token through only a small subset, creating conditional computation. That allows total parameter count to grow faster than active compute. The new memory architectures ask a complementary question: if some model capacity mainly stores reusable knowledge, why route it through neural computation at all? Why not introduce a dedicated lookup primitive?

That is the conceptual bridge between systems that otherwise look quite different. Mobius uses a shared learned memory inside the network. DeepSeek’s Engram uses deterministic conditional lookup. Apple’s hierarchical-memory work fetches small context-dependent blocks from much larger parametric banks. Limited-memory language models deliberately move factual knowledge into an external database. The common move is to stop treating every parameter as the same kind of intelligence.

DeepSeek’s Engram turns part of language modeling into conditional lookup

DeepSeek’s 2026 Engram work, published at ACL, frames the problem directly: mixture-of-experts provides conditional computation, but the Transformer lacks a native primitive for knowledge lookup.

Engram introduces what the authors call conditional memory. Its memory module modernizes n-gram embeddings so local token patterns can deterministically address a large embedding table. The lookup is effectively O(1) with respect to the size of that table. Instead of forcing early Transformer layers to reconstruct familiar local patterns through repeated dense computation, the model can retrieve a learned vector and devote more of the backbone to global composition.

The authors formulate a “sparsity allocation” problem: given a fixed parameter and compute budget, how much capacity should go to neural experts and how much to static memory? In their experiments, a 27B-parameter Engram configuration outperformed an iso-parameter, iso-FLOPs mixture-of-experts baseline across knowledge, reasoning, code and mathematics. They also report stronger long-context retrieval. Those figures are results from the paper’s own training and evaluation regime, not universal evidence that n-gram memory will dominate other architectures.

The systems implication may be more important than any one benchmark. Because Engram’s addresses are deterministic, the memory table can be prefetched from host memory instead of requiring the whole capacity to sit in expensive accelerator memory. The model begins to look less like one monolithic neural program and more like a compute engine attached to a very large learned store.

That distinction matters for hardware. High-bandwidth accelerator memory is scarce and expensive. Ordinary DRAM and future memory-expansion fabrics offer much more capacity at lower cost, but with different latency and bandwidth properties. An architecture that knows in advance which memory entries it will need can exploit that hierarchy more effectively than one whose every parameter participates in dense computation.

Hierarchical parametric memories push long-tail knowledge out of the core model

Apple researchers reached a related conclusion from a different starting point. Their ICLR 2026 work argues that forcing a compact model to memorize the entire long tail of world knowledge is inefficient, especially for devices with limited inference memory and compute.

Their design uses a comparatively small language model as an anchor for common knowledge and general reasoning, while larger hierarchical parametric memory banks hold information that is only occasionally needed. During training and inference, the system fetches a small context-dependent memory block rather than activating the entire bank.

In one reported comparison, a 160M-parameter model augmented with an 18M-parameter fetched memory block drawn from a 4.6B-parameter bank achieved performance comparable to a conventional model with more than twice the active parameters. The researchers scale their memory banks beyond 21B parameters and report that the approach works when memory is introduced during pretraining or attached later.

The important metric is therefore no longer simply total parameter count. A model might own tens of billions of memory parameters while touching only a small fraction per token. What matters for serving becomes active compute, active memory traffic, retrieval accuracy and the cost of the hierarchy that supplies those parameters.

This is one reason raw model-size comparisons are becoming less informative. AIUpdateWatch’s benchmark coverage already separates capability by task; architecture-aware comparisons increasingly need to separate total parameters, active parameters, memory parameters, context state and generated-token cost as well.

Other researchers want factual knowledge outside the weights entirely

The strongest version of the separation idea goes further than parametric memory. Limited Memory Language Models, presented at ICLR 2026, are trained to externalize entity-level factual knowledge into a database instead of storing as much of it as possible in opaque model weights.

The training recipe masks retrieved factual values from the language-model loss and teaches the model to perform targeted lookups. The authors’ goal is not merely smaller inference. Externalized facts can be inspected, edited or deleted without changing the entire neural network.

This resembles retrieval-augmented generation, but the boundary is different. Conventional RAG usually takes a pretrained model and adds retrieval at application time. Limited-memory models are pretrained around the assumption that certain facts live elsewhere. Retrieval is part of what the model learns to depend on rather than an optional wrapper bolted on after training.

That has advantages and new failure modes. Editable knowledge is attractive for freshness, provenance and deletion. But the system becomes dependent on database coverage, entity resolution and retrieval correctness. A fact omitted from the external store may be unavailable precisely because the training process discouraged the model from memorizing it. Knowledge modularity replaces one kind of uncertainty with another.

Recent auditing work on limited-memory models reinforces the point. In one causal study of deletion behavior, residual “remembered” answers were traced primarily to alternative retrieval paths rather than to the supposedly forgotten fact remaining in model weights. That is encouraging for controllability, but it also means the database graph becomes part of the model’s effective memory boundary.

Reasoning itself is becoming less tied to visible token generation

The separation trend is not only about where facts live. It also affects how reasoning consumes compute.

Mobius repeatedly refines continuous hidden states before decoding, a form of latent recurrent computation. Other 2026 architectures explore similar ideas. Memory-Efficient Looped Transformer, for example, is designed to let a model perform iterative reasoning in embedding space while keeping KV-cache memory roughly constant across reasoning loops by sharing and updating a cache rather than allocating a new one for every iteration.

This creates a second decoupling: reasoning depth need not equal the number of visible chain-of-thought tokens, and memory usage need not grow linearly with the number of internal reasoning iterations.

Memory Sparse Attention attacks a neighboring bottleneck from the long-context side. Its authors report an end-to-end memory architecture that scales from 16K to 100M tokens with less than 9% degradation in their evaluated long-context setting, using sparse attention, cache compression and memory parallelism. They report 100M-token inference on two A800 GPUs. Again, those are project results rather than an independently established production ceiling, but the design direction is consistent: memory capacity is becoming its own scaling dimension instead of being treated as synonymous with ordinary context length.

These designs should not be conflated. Conditional parametric memory, external factual databases, sparse long-context memory and recurrent latent reasoning solve different problems. What unifies them is architectural specialization.

Once memory and reasoning separate, model design becomes a systems co-design problem

Architectural specialization changes which hardware resource is scarce.

Dense Transformers make model weights, KV cache and matrix multiplication compete for accelerator memory and bandwidth. Conditional memories can shift some capacity into host DRAM. External knowledge bases can move facts onto CPUs, storage systems or network services. Recurrent reasoning can trade extra compute iterations for fewer visible tokens. Sparse attention can trade indexing complexity for lower long-context bandwidth.

The result is a model whose performance cannot be predicted from FLOPs alone. Latency depends on memory hierarchy, prefetch accuracy, interconnect bandwidth, cache locality, batch size and the ratio between attention and feed-forward work.

ICLR 2026 research from Amazon and the University of Wisconsin makes this broader point quantitatively. After training more than 200 models, the authors show that hidden size, the ratio of MLP parameters to attention parameters and grouped-query attention materially affect inference throughput even under similar training budgets. Their optimized designs achieved up to 42% higher throughput than the compared LLaMA-3.2 baseline in the paper.

That is a different result from Mobius or Engram, but it supports the same strategic shift: architecture is becoming a first-class scaling variable. “How many parameters?” and “how many training tokens?” are no longer sufficient questions for systems expected to serve billions of requests economically.

This also connects to AIUpdateWatch’s analysis of disaggregated prefill and decode. Serving systems are already splitting inference into phases with different hardware requirements. Model architectures are now beginning to split internal functions for similar reasons.

The hardest problem is deciding what should be memory and what should remain computation

Knowledge and reasoning are conceptually different, but real neural networks do not offer a clean human-readable boundary between them.

Some factual associations are useful ingredients of reasoning. Some apparently local patterns carry structural information that attention needs globally. A memory lookup can be fast but wrong. A shared memory can create interference between reasoning stages. An external store can become stale or incomplete. A latent reasoning loop can save output tokens while increasing internal compute and making intermediate decisions harder to inspect.

There is also no common evaluation standard for these architectures yet. Mobius’s nearly four-times speedup is reported against a specific baseline and serving configuration. Engram’s gains are measured against carefully matched MoE baselines built by the same research team. Apple’s memory models are much smaller than frontier production systems. MSA’s 100M-token result is a specialized research system, not evidence that arbitrary 100M-token prompts are now cheap or equally accurate.

Deployment tooling is another constraint. The released Intern-S2-Mobius weights can be served through LMDeploy, Transformers and vLLM configurations, but the architecture is still new enough that implementation compatibility is evolving. The project’s own GitHub issue tracker has already contained questions about correspondence between released code and the technical report. That is normal for an early architecture release, but it is another reason not to treat paper benchmarks as settled infrastructure numbers.

The core Transformer has survived many proposed replacements because its regularity maps extremely well onto accelerators and software stacks. A theoretically cleaner separation of memory and reasoning only wins if the added routing, lookup, caching and orchestration complexity pays for itself at real scale.

The durable signal will be memory-to-compute ratios, not another model-size record

The next evidence to watch is whether these ideas survive beyond individual papers and specialized releases.

Independent reproduction: whether third parties can reproduce Mobius-style throughput and Engram-style quality gains under comparable hardware and serving settings.
Frontier-scale adoption: whether large commercial model families begin publishing separate memory-capacity and reasoning-compute specifications.
Hardware co-design: whether CXL, large host-memory pools, memory-side accelerators or other heterogeneous systems become standard parts of model architecture rather than deployment afterthoughts.
Knowledge updates: whether modular memory actually makes factual updates, deletion and provenance easier without introducing unacceptable retrieval failures.
Active versus total parameters: whether model reporting shifts toward the amount of memory and compute touched per token rather than headline parameter count.
Latent reasoning economics: whether internal recurrent reasoning reduces end-to-end cost or merely moves computation out of visible token traces.

If this trend holds, the next generation of language models may look less like one giant homogeneous network and more like a composed system: a reasoning engine, one or more learned or external memory stores, specialized retrieval machinery and a runtime that decides how much of each resource a request deserves. That would be a deeper change than simply making the Transformer wider.

Sources