> ## Content Index > Fetch the complete content index at: https://blog.moravak.dev/llms.txt > Use this file to discover other available public pages before exploring further. # HNetBit: A Novel, More Efficient Architecture for Language Models - URL: https://blog.moravak.dev/hnetbit/ - Published: 2026-09-11T23:15:28.000Z - Updated: 2026-09-13T16:43:32.000Z - Description: Most AI research companies will disappear into thin air in a few years and, with them, hundreds of millions of dollars in loans, infrastructure, contracts and unfulfilled promises. - Author: Moravak - Tags: AI Most AI research companies will disappear into thin air in a few years and, with them, hundreds of millions of dollars in loans, infrastructure, contracts and unfulfilled promises. --- It has already been four years since the release of ChatGPT and the popularization (maybe even democratization) of LLMs. This golden era of AI has been the foundation for massive innovation and has pushed the boundaries of human efficiency. We have gone from a simple chatbot that could help with basic tasks to models that rival the knowledge of a human PhD in specific fields. This ridiculously fast progress does, however, come at an even more ridiculous cost. It is true that AI models get smarter with each release. It is also true that they get a lot more expensive and much harder to run on existing infrastructure—and that is the fatal flaw few are trying to solve, which might ultimately bankrupt the industry. In 2025, OpenAI's expenses reportedly hit $34 billion, while revenue barely reached $13 billion. If that wasn't eye-opening enough, the company estimated that its infrastructure and compute spending through 2030 will be in the neighborhood of a whopping $750 billion. The numbers just don't make sense. The industry is now completely immersed in a circular economy where the first entity to fall will drag everyone else down with it. ## What can we do to stop this? The obvious answer is to de-escalate, but that is impossible given the growing demand for AI. The other option is harder, slower, and a task long overdue for AI research companies: actually researching. You'd be amazed at how many solutions to the compute problem researchers have already put into practice. Labs like DeepSeek are adopting several of them, which translates directly into lower API prices and faster models. One of the most common methods is weight quantization, which I decided to take a step further by merging it with a dynamic chunking mechanism—an architecture I call **HNetBit**. My work is based on the papers [*Scalable MatMul-free Language Modeling*](https://arxiv.org/abs/2406.02528?ref=blog.moravak.dev) and [*Dynamic Chunking for End-to-End Hierarchical Sequence Modeling*](https://arxiv.org/abs/2507.07955?ref=blog.moravak.dev), and the code can be found in [this GitHub repository](https://github.com/jcv-dev/hnet%5Fbit%5Fbenchmarks/tree/main/hnet%5Fbit?ref=blog.moravak.dev). By using both techniques simultaneously, I achieved a middle ground that not only reaches results comparable to a Transformer, but also scales sub-linearly as parameter counts grow while using less memory during inference. These improvements do, of course, come at a cost—namely, training memory and training time. Although the impact of these disadvantages diminishes as model size scales, it is worth noting that they exist. Consequently, this architecture is not suitable for tiny models, where architecture overhead makes a standard Transformer the better choice. However, at parameter counts above 1B, HNetBit not only trains in a timeframe similar to a Transformer, but also achieves the same (if not better) BPB at a fraction of the memory footprint. ## But how? Two things make a modern LLM expensive: the weights it has to store and move, and the attention it has to compute. HNetBit attacks both and adds a third mechanism so the savings compound instead of canceling out. - **Kill the matmuls:** Every projection in HNetBit is a BitLinear layer. Instead of full-precision weights, the layer stores a scale factor and a matrix of values restricted to -1, 0, or +1\. That is 1.58 bits per weight instead of 16, turning matrix multiplication into simple sign flips, additions, or zero operations. Because you cannot backpropagate through a rounding operation, the forward pass uses ternary weights while the backward pass estimates gradients as if rounding never happened, updating a full-precision copy. That is the straight-through estimator, and it is the reason MatMul-free models can train at all. Activations are also quantized to 8 bits per token. You do lose precision, but you recover it with a higher learning rate—exactly as demonstrated in the MatMul-free paper. - **Kill the attention:** Transformers pay for attention twice: quadratically during training, and via a KV cache during inference that grows with every generated token. HGRN replaces attention with a gated linear recurrence. Each token updates a fixed-size state, which is the only element the model needs to carry forward. During training, the recurrence runs as a parallel scan for GPU efficiency; during inference, it operates in a single step, keeping memory flat regardless of context length. A short convolution before the recurrence provides a local window, compensating for recurrence's weakness with byte-level patterns. - **Let the model tokenize itself:** This is where H-Net enters the picture. Byte-level models are appealing because they eliminate the need for a tokenizer, but they carry a heavy cost: a passage a BPE model reads in 1,280 tokens takes roughly 4,000 bytes, increasing the workload at every position. HNetBit inserts routing modules between hierarchy stages. A router evaluates two consecutive hidden states, computes their cosine similarity, and determines whether they belong to the same chunk or a new one. Similar states are merged, while distinct ones trigger a boundary. Only boundary tokens advance to the next stage, and a dechunk layer expands the result back to full resolution using boundary probabilities as an exponential moving average. Because the router trains end-to-end in full precision, the model discovers its own variable-length tokenization rather than relying on a static vocabulary file. At 350M parameters, the hierarchy uses two routing stages, meaning the innermost, most expensive network processes only about 7% of the original positions—yielding roughly a 15x sequence compression where it matters most. The model learns bytes, then words, then phrases, with each level operating on fewer, richer units. ## Why it compounds Because chunking shrinks sequence length before reaching the largest layers, compute stops scaling directly with parameter count. Scaling from 150M to 350M parameters increases parameter count by 3x, yet training FLOPs grow by only 1.43x (with measured wall-clock scaling almost identically at 1.4345x). At 350M parameters, chunking removes 75% of the FLOPs the blocks would otherwise consume; at 150M, it still saves 47%. Meanwhile, because the recurrent state does not expand with context, prefill memory remains flat from 512 to 4,096 tokens, and decode memory remains flat from batch sizes 1 to 8\. A Transformer of equal scale incurs a quadratic penalty at every step. None of this is free. The hierarchy introduces extra buffers, the routers run in full precision, and the forward pass skips activation recomputation. As a result, smaller models burn more memory and time than a standard Transformer. However, while the overhead remains relatively constant, the savings scale alongside the model size. This makes HNetBit a billion-parameter-plus architecture rather than a 100M toy. Beyond that threshold, the curves cross, leaving a byte-level model that trains like a smaller network, remembers like an RNN, and exports at a fraction of the size.