The AI boom has swept across industries, compelling public sector organizations to accelerate their adoption of these transformative technologies. However, government institutions grapple with unique constraints in security, governance, and operations that distinguish them significantly from their private sector counterparts. It is within this challenging landscape that purpose-built Small Language Models (SLMs) emerge as a particularly promising avenue for operationalizing AI.

A comprehensive study by Capgemini revealed that a substantial 79 percent of public sector executives globally harbor reservations about AI’s data security. This apprehension is entirely understandable, given the highly sensitive nature of government data and the stringent legal obligations surrounding its utilization. As Han Xiao, vice president of AI at Elastic, articulates, "Government agencies must be very restricted about what kind of data they send to the network. This sets a lot of boundaries on how they think about and manage their data." This fundamental need for absolute control over sensitive information is just one of many factors that complicate AI deployment, especially when contrasted with the typical operational assumptions prevalent in the private sector.

Unique Operational Challenges in the Public Sector

When private-sector entities scale their AI initiatives, they generally operate under the assumption that certain conditions are readily met. These often include unfettered cloud connectivity, reliance on centralized infrastructure, a degree of acceptance for incomplete model transparency, and minimal restrictions on data movement. For many governmental institutions, however, embracing these conditions can range from being imprudent to outright impossible.

Government agencies are fundamentally tasked with ensuring their data remains under their direct control, that information can be rigorously checked and verified, and that operational disruptions are minimized to the greatest extent possible. Compounding these responsibilities, these agencies frequently operate within environments characterized by limited, unreliable, or even absent internet connectivity. These multifaceted complexities often serve as significant impediments, preventing many promising public sector AI pilot programs from progressing beyond the experimental phase. "Many people undervalue the operating challenge of AI," Xiao observes. "The public sector needs AI to perform reliably on all kinds of data, and then to be able to grow without breaking. Continuity of operations is often underestimated." Indeed, an Elastic survey of public sector leaders indicated that a considerable 65 percent struggle with consistently using data in real-time and at scale.

Furthermore, existing infrastructure constraints exacerbate these challenges. Government organizations may encounter difficulties in procuring the essential graphics processing units (GPUs) that are crucial for both training and accessing complex AI models. As Xiao points out, "Government doesn’t often purchase GPUs, unlike the private sector—they’re not used to managing GPU infrastructure. So accessing a GPU to run the model is a bottleneck for much of the public sector."

The Case for Smaller, More Practical Models: Small Language Models (SLMs)

The myriad of non-negotiable requirements inherent in public sector operations renders large language models (LLMs) largely impractical. In contrast, SLMs offer a compelling alternative by virtue of their ability to be housed locally, thereby providing enhanced security and control. SLMs are a specialized class of AI models, typically featuring billions, rather than hundreds of billions, of parameters. This architectural distinction makes them significantly less computationally demanding than their larger LLM counterparts.

The public sector is not in need of ever-expanding models housed in remote, centralized data centers. Empirical research has demonstrated that SLMs can achieve performance levels equal to or even surpassing those of LLMs. This capability allows sensitive information to be utilized effectively and efficiently, sidestepping the complex operational hurdles associated with maintaining massive AI models. Xiao aptly summarizes this advantage: "It is easy to use ChatGPT to do proofreading. It’s very difficult to run your own large language models just as smoothly in an environment with no network access."

SLMs are meticulously designed and tailored to meet the specific needs of the department or agency that will employ them. Sensitive data remains securely stored outside the model itself, only being accessed when a query is made. Carefully engineered prompts ensure that only the most pertinent information is retrieved, leading to more accurate and relevant responses. By leveraging advanced techniques such as smart retrieval, vector search, and verifiable source grounding, AI systems can be constructed that precisely cater to the unique demands of the public sector. Consequently, the next evolutionary step in AI adoption within the public sector may involve bringing the AI tool directly to the data, rather than transmitting sensitive data to the cloud. Gartner forecasts that by 2027, small, task-specific AI models will see adoption rates three times greater than general-purpose LLMs.

Superior Search Capabilities: Unlocking Data’s Potential

"When people in the public sector hear AI, they probably think about ChatGPT. But we can be much more ambitious," Xiao states. "AI can revolutionize how the government searches and manages the large amounts of data they have." Beyond the familiar realm of chatbots, one of AI’s most immediate and impactful opportunities lies in its capacity to dramatically improve search functionalities. Like many organizations, the public sector is inundated with vast quantities of unstructured data, encompassing technical reports, procurement documents, meeting minutes, and invoices. Modern AI, however, can now deliver results sourced from a diverse range of media, including readable PDFs, scanned documents, images, spreadsheets, and audio recordings, and across multiple languages. All of this can be indexed by SLM-powered systems to furnish tailored responses and to draft complex texts in any language, while concurrently ensuring that all outputs adhere to legal compliance standards. "The public sector has a lot of data, and they don’t always know how to use this data. They don’t know what the possibilities are," Xiao emphasizes.

Even more powerfully, AI can empower government employees to interpret the data they access. "Today’s AI can provide you with a completely new view of how to harness that data," Xiao explains. A well-trained SLM possesses the ability to interpret legal norms, extract actionable insights from public consultations, support data-driven executive decision-making, and enhance public access to essential services and administrative information. These capabilities collectively contribute to profound improvements in the way the public sector conducts its operations.

The Promise of Small Language Models

By focusing on SLMs, the conversation shifts from the sheer comprehensiveness of a model to its operational efficiency. LLMs incur substantial performance and computational costs and necessitate specialized hardware that many public entities find financially prohibitive. While SLMs do require some capital expenditure, they are considerably less resource-intensive than LLMs, making them more cost-effective and contributing to a reduced environmental impact.

Public sector agencies are frequently subject to stringent audit requirements, and SLM algorithms can be meticulously documented and certified for transparency. Furthermore, certain countries, particularly in Europe, have implemented privacy regulations such as GDPR, which SLMs can be specifically designed to meet.

The use of tailored training data yields more precise results, thereby mitigating the errors, biases, and "hallucinations" that AI systems are prone to. As Xiao articulates, "Large language models generate text based on what they were trained on, so there is a cut-off date when they were trained. If you ask about anything after that, it will hallucinate. We can solve this by forcing the model to work from verified sources." Risks are further minimized by keeping data on local servers, or even on individual devices. This approach is not about isolation but rather about fostering strategic autonomy, which in turn enables trust, resilience, and relevance.

By prioritizing task-specific models engineered for environments that process data locally, and by implementing continuous monitoring of performance and impact, public sector organizations can cultivate enduring AI capabilities that effectively support real-world decision-making. "Do not start with a chatbot; start with search," Xiao advises. "Much of what we think of as AI intelligence is really about finding the right information."