isnaini azizah
The current landscape reveals a precarious reliance on luck and retrospective analysis to mitigate the potentially severe consequences of these unauthorized AI actions. While regulatory bodies like the European Union, through its AI Act, and US agencies, such as NIST, are rightly advocating for real-time monitoring by developers and deployers, their directives presuppose a level of visibility that is, in reality, severely lacking. Companies are struggling to ascertain not only what their AI agents are doing but also the underlying reasons behind their actions.
To illustrate this critical deficit, consider a hypothetical scenario: a bank hires a thousand new employees with access to sensitive customer accounts. However, their identification badges change daily, there’s no clear record of managerial review or authorization for their tasks, and crucially, no logs track modifications made to approval processes. This pervasive lack of transparency mirrors the current state of AI agent deployment. As agents are increasingly integrated into vital sectors such as finance, government, and healthcare, the imperative for real-time monitoring to prevent widespread and potentially catastrophic outcomes becomes paramount.
Effective safety hinges on robust monitoring, which in turn necessitates "observability"—the capability to capture system actions, process them into meaningful signals, and present them for human or automated oversight. In essence, we need to understand not just what an agent did, but also why it made those choices. Unfortunately, as highlighted by recent research from the Partnership on AI, current agent frameworks—the software infrastructure that powers these agents—fail to consistently generate the essential observability records required for safe deployment.
AI agents present a distinct set of challenges beyond those posed by standalone generative AI models. They inherit the inherent unpredictability of the large language models (LLMs) upon which they are built. However, unlike generative AI systems that produce a single output for human interpretation and action, agents undertake more complex, multi-step tasks. They engage in planning, tool selection, the spawning of sub-agents, and direct action execution. This intricate process allows for a multitude of emergent behaviors, making it exceedingly difficult for pre-deployment testing to anticipate all potential risks. Consequently, observing agents during operation becomes both more challenging and critically more important.
In a comprehensive study, researchers collaborated with experts from academia, civil society, and industry to examine the four most widely adopted agent frameworks. The objective was to ascertain the telemetry signals they actually emit. The findings revealed significant gaps that impede critical safety goals, including the detection of policy violations, unauthorized actions, and even failures in human oversight. Essential observability requirements were notably absent, such as persistent Agent IDs that endure over time and records detailing human approvals for agent tasks. This deficiency makes it arduous to attribute actions to specific agents or to ascertain whether they are engaging in unauthorized activities.
The good news, however, is that these identified gaps are not insurmountable. Addressing them begins with the systematic capture of two fundamental types of records: those that elucidate the sequence of events within an agent’s operational run and those that reveal the agent’s decision-making process. "What happened" records should identify the agent (while safeguarding privacy) and log human interventions, alongside any shifts in agent permissions or "memory." Concurrently, "decision" records must illuminate the agent’s reasoning and its confidence levels in the actions it undertakes.
Consider, for instance, an agent processing mortgage loan applications at a bank. If the system recommends denying an application, and a human loan officer subsequently approves that decision, auditors or regulators, whether in real-time or during a post-hoc review, must be able to definitively reconstruct which agent made the initial recommendation, which officer provided approval, and the duration of their engagement. Furthermore, the bank needs to analyze the agent’s decision-making process to ensure that it did not rely on prohibited information, such as data pertaining to the applicant’s neighborhood.
Capturing both "what happened" and "how the agent decided" requires distinct methodologies, but both are achievable. The current dearth of "what happened" records is primarily a technical challenge, a "software plumbing" issue that can be resolved by adopting OpenTelemetry, an industry-wide open observability standard maintained by a broad coalition of engineers, and by its integration into agent frameworks.
While OpenTelemetry has initiated a working group dedicated to drafting standardized formats for agent tracing, these standards are yet to be finalized. This group must prioritize the urgent incorporation of standards for maintaining persistent agent identity and for logging human interventions and alterations to agent permissions or memory. This effort needs to be completed before these systems achieve widespread scalability, all while meticulously addressing the privacy implications of agent IDs that might be linked to real individuals.
Ensuring the preservation of "how agents decided" records presents a more complex challenge, raising fundamental research questions for model providers and the broader research community. Does the step-by-step reasoning displayed by a model, often referred to as "chain of thought," accurately reflect its actual decision-making process? And do we possess reliable methods for quantifying an agent’s confidence in its decisions? These questions demand definitive answers before any standards can be established for the utilization of such observability records.
Closing these observability gaps cannot be achieved by any single organization or sector acting in isolation. Instead, a collective effort is required to build the necessary infrastructure that will prevent dangerous incidents and foster AI safety. AI developers responsible for building agent frameworks, including prominent players like OpenAI, Anthropic, CrewAI, and LangChain, must proactively integrate telemetry signals that record an agent’s persistent identity, any human interventions, and any modifications to its permissions and memory.
Simultaneously, enterprises adopting these agent frameworks wield significant influence. Particularly in regulated industries like banking and healthcare, these organizations can exert pressure on agent developers to provide the missing observability records, especially where industry standards have not yet been established. Enterprises also bear the responsibility of defining clear policies, permissions, and roles that delineate the safe operational boundaries for agents, as telemetry alone cannot definitively ascertain whether specific rules have been violated.
Regulators and AI standards institutes play a crucial role in specifying the observability records necessary for high-stakes AI deployments, while leaving the granular technical implementation to the industry. Positive momentum is already evident, with initiatives like NIST’s agent standards project in the US actively addressing agent identity, a critical gap identified in recent research.
No prudent bank would deploy a thousand unbadged employees to manage customer accounts and then hope to reconstruct events after a mishap. Yet, this is precisely the operational model we are currently employing for AI agents. Making their behavior observable is the indispensable first step towards rectifying this situation. By fostering a regulatory and policy ecosystem that enables action based on observed data, we can establish accountability for organizations. Until then, every task delegated to an agent represents a surrender of our own agency.
Madhulika Srikumar, Head of the AI Safety Program at the Partnership on AI, and Claire Leibowicz, Director of AI, Trust, and Society at the Partnership on AI and a doctoral candidate at Oxford University, are at the forefront of advocating for these critical safety measures. Their work emphasizes the urgent need for transparency and accountability in the burgeoning field of AI agents.
