isnaini azizah
However, this pivotal technology is showing its age. Many of the recent advancements in LLMs, such as the development of sophisticated reasoning models and their enhanced capacity to handle extensive input, are not seamless extensions of the original transformer design but rather workarounds addressing its inherent limitations. This has spurred a new wave of scientific and engineering inquiry into what comes next. While LLMs are here to stay, their underlying architecture is ripe for reinvention. MIT Technology Review has already foreshadowed this evolution, labeling the next generation of models as "LLMs+" in its "10 things that matter in AI" list for 2026. A new cohort of startups is emerging, poised to push the boundaries of this rapidly expanding technological frontier. While some will undoubtedly falter, they possess the agility and drive to explore uncharted territory with less to lose than established industry leaders.
The core strength of transformers lies in a mechanism called dense attention. This process encodes the meaning of a text block into a series of numerical representations by comparing every word (or token) with every other word through multiplication. This dense attention mechanism achieves remarkable accuracy in capturing textual meaning. However, its computational demands escalate dramatically with increasing text length. A document of 10,000 words, for instance, could necessitate approximately 50 million multiplications, a primary reason for the substantial energy consumption of LLMs. The financial implications are staggering; OpenAI, according to its president Greg Brockman, projects a $50 billion expenditure on computing power this year alone. The International Energy Agency further forecasts a doubling of electricity consumption by data centers globally by 2030.
Beyond the energy and cost concerns, transformers struggle with tasks that necessitate the comprehension of vast amounts of information. Their sequential, word-by-word processing limits their ability to maintain a large context window, hindering their capacity to ingest and retain extensive data – be it entire libraries, extensive codebases, or the outputs of other LLMs for agentic operations. Reasoning models, which employ a "scratch pad" approach akin to a chain of thought to process information, further exacerbate this data-handling challenge. As LLMs grow more sophisticated, transformers, once a revolutionary advantage, are becoming a significant bottleneck, with their core strength evolving into a limitation. The following explores four innovative approaches designed to overcome these transformer-related challenges, aiming to usher in an era of faster, more efficient, and potentially more intelligent LLMs.
01: Rethinking Attention
A direct strategy to enhance LLM speed and reduce costs involves fundamentally altering the attention mechanism. Replacing dense attention with sparse attention, which selectively computes relationships between word pairings rather than all of them, can drastically cut down computational overhead. While various sparse attention mechanisms have been proposed over the years, none have matched the semantic accuracy of dense attention. This paradigm may be shifting. Subquadratic, a Miami-based startup, asserts that it has developed the first sparse attention mechanism capable of rivaling mainstream LLMs in tasks like search and coding. This is a bold claim, met with some industry skepticism. Subquadratic’s model, SubQ, reportedly identifies the most relevant words within a given text dynamically, a capability that has garnered significant interest, with thousands joining its waitlist and plans for broad availability soon.
Manifest AI, a San Francisco-based startup, approaches the problem from a different angle, not by modifying attention but by replacing it entirely. They have developed a mechanism called power retention, which prioritizes and stores only the most critical information for a specific task, thereby preventing the data burden from overwhelming the LLM. Unlike attention mechanisms that track everything within their context window, and even sparse attention which retains a general overview, power retention operates by providing a rolling summary of the context. As new information is introduced, less relevant data is discarded. While the principle of retention has existed for a decade, Manifest AI claims to have refined these techniques to create models that can compete with transformer-based LLMs. They suggest that transformer models can be converted to power retention models with minimal retraining. To demonstrate this, they have transformed the open-source coding LLM StarCoder into PowerCoder, a power retention variant. They have also released Brumby, a model they claim is competitive with some versions of Alibaba’s popular open-source model, Qwen. Manifest AI aims for its power retention technology to become the go-to solution for LLMs tasked with processing massive datasets. Carles Gelada, the company’s cofounder and CTO, highlighted its potential for applications ranging from analyzing lengthy videos to developing agents capable of sustained task execution over weeks.
02: Making Models Smaller and More Flexible
Liquid AI, an MIT spinout based in Cambridge, Massachusetts, takes a hybrid approach, integrating its proprietary liquid neural networks with transformers to create what CEO Ramin Hasani calls LFMs (liquid foundation models). These models are notably smaller and more energy-efficient than most current LLMs. Liquid AI develops models for automotive manufacturers like Mercedes, designed to operate on the compact chips found in vehicles. Their latest models can even run on a Raspberry Pi, an inexpensive, low-power hobbyist computer. The company offers its models free of charge to organizations with annual revenues below $10 million, a policy that has contributed to nearly 34 million downloads.
Liquid neural networks, inspired by the structure of worm brains, are an evolution of convolutional networks, a type of neural network that predates transformers. Their key innovation is a mechanism that allows models to adapt their behavior in response to new information, enabling continuous learning – a capability absent in trained transformers, whose behavior remains fixed. While Liquid AI’s initial models were simpler, capable of controlling drones and vehicles, LFMs aim to scale this technology to compete with mainstream LLMs. Their newer models reportedly match the performance of rivals up to four times their size, including versions of Alibaba’s Qwen and Google’s open-source LLM Gemma. These recent LFMs are hybrid architectures, comprising 80% liquid neural networks and 20% transformers. This precise ratio was determined by another AI system developed by Liquid AI, which assists in designing all their models. This designer AI explores numerous combinations of neural networks – including liquid, convolutional, and transformers – to identify designs that optimize both performance and efficiency. Hasani views transformers as a stepping stone, envisioning future AI systems that emulate the brain’s efficient operation, powered by minimal energy.
03: Generating Text All at Once
The predominant method for LLM output generation involves producing text one word at a time, mirroring human speech and writing, but proving inefficient for computers. Generating text in larger chunks, such as entire sentences or paragraphs, is faster and more cost-effective. This is the approach adopted by Inception, a Palo Alto-based startup, which leverages a technique called diffusion. Diffusion models are widely recognized for their role in image and video generation, where they transform random pixel grids into coherent visuals by processing all pixels simultaneously. Inception has adapted this process for text generation, training its LLMs to convert random word strings into meaningful sentences. While these diffusion LLMs still utilize transformers for encoding meaning, their ability to generate entire text blocks at once allows transformers to perform more work with less effort. Stefano Ermon, Inception’s cofounder and CEO, explains that this simultaneous prediction of multiple tokens significantly enhances speed and cost-efficiency compared to conventional LLMs.
The challenge lay in applying a technology designed for visual data to text. Ermon points out that interpolating between colors in image generation is straightforward, but there isn’t a clear intermediate step between discrete words like "cat" and "dog." Ermon, also a researcher at Stanford University, along with two colleagues, developed the mathematical framework for diffusion models in text. Their initial diffusion model matched the performance of OpenAI’s 2019 LLM, GPT-2, but was ten times faster, prompting Ermon to found Inception. Their latest model, Mercury 2, reportedly rivals some of OpenAI’s 2023 GPT-4 models, achieving similar performance at ten times the speed. Ermon is optimistic about this approach’s scalability, emphasizing that "intelligence per dollar" will be the ultimate metric. Google is also exploring diffusion for text generation with its prototype LLM, Diffusion Gemma, a development Ermon views as validation of his belief that "This is the future."
04: Moving Beyond Words
Pathway, another Palo Alto startup, is pursuing perhaps the most radical departure by aiming to free LLMs from the limitations of language. Their LLM, dubbed Dragon Hatchling, has demonstrated exceptional performance on a benchmark involving over 250,000 complex Sudoku puzzles, solving more than 97% of them, a feat that eluded several leading LLMs. Pathway’s core argument is that despite their widespread success, LLMs still falter on certain problem classes, with Sudoku being one example. To achieve genuine, novel solutions to real-world problems, they contend that moving beyond transformers is essential. Transformers’ reliance on text processing makes them ill-suited for certain types of reasoning, as representing a Sudoku board purely through words is highly impractical.
Pathway’s solution involves replacing the transformer’s attention mechanism with a state space, a mathematical structure that compresses information into a more abstract representation rather than processing it word by word. This technique allows Dragon Hatchling to not only process and generate text but also to emulate forms of reasoning that bypass sequential language. This enhances efficiency and, theoretically, enables the model to tackle tasks inaccessible to other LLMs. Zuzanna Stamirowska, Pathway’s cofounder and CEO, draws parallels to complex cognitive processes like playing chess or performing mathematical calculations, which are not necessarily experienced as linear sentences. She argues that forcing reasoning through language imposes constraints. While acknowledging that a conventional LLM could learn to solve Sudoku by reading a book and writing code, Stamirowska emphasizes the aspiration for AI to tackle problems like curing cancer, which lack pre-existing instruction manuals. She posits that transformers represent an "engineering convenience" that became overly entrenched, and that further breakthroughs are inevitable, stating, "It started a religion, but it’s silly to think that a breakthrough won’t happen again."
