Observability Usage of AI Governance Agents & Models

As organizations scale AI use cases, governance teams need a “single pane of glass” to monitor risk, performance, and compliance across AI assets. OneTrust allows customers to bring in information on model evaluation metrics, guardrails, usage, and other decision-critical data.

AI usage can be tracked at the AI agent or model level. A single AI agent invocation can result in multiple models being invoked, and these metrics help to understand trends and establish a baseline over a period. 

These metrics can also help provide a high-level overview of token consumption to help understand usage versus spend trends. Calculation of absolute cost (in $ or other currencies) incurred through AI systems is not available or intended.

Three critical data signals that are available to take advantage of are:

1. Invocation count (usage) - tracks the number of times a specific model or agent was invoked. 

2. Input token count - tracks the number of tokens sent to a model.

3. Output token count - tracks the number of output tokens generated by the foundation model(s).

Prerequisites

• You have access to OneTrust AI Governance with permissions to configure AI Automation. To deploy a lightweight worker node, follow the steps here. 

• Ensure the inventory of agents and models is imported through the AI automations.

• To collect observability metrics from Amazon Bedrock & AgentCore agents, each agent's IAM execution role must have the cloudwatch:PutMetricData permission.

Supported Platforms

Observability metric support varies by platform and by AI entity type.

Platform	AI entity	Metrics available
Amazon Bedrock & AgentCore	Agents and models	Invocation count, input token count, output token count
Azure Foundry	Agents and models	Invocation count, input token count, output token count
Databricks	Agents and models	Invocation count, input token count, output token count
Google Vertex (now Gemini Enterprise Agent Platform)	Agents	Invocation count

OneTrust AI Governance captures the identity associated with each Amazon Bedrock AgentCore agent, so governance teams can see which workload identity an agent uses and what type of outbound credential it relies on.

In Amazon Bedrock AgentCore, each agent has a workload identity managed by AgentCore Identity. When an agent is deployed to AgentCore Runtime, the service creates this identity automatically and associates it with the agent.

The agent uses its workload identity to obtain a workload access token, which authorizes it to retrieve outbound credentials (OAuth tokens or API keys) from credential providers stored in the AgentCore token vault.

The workload identity is separate from the agent's IAM execution role. The role governs AWS API access, while the workload identity governs access to stored credentials.

In OneTrust discovery data, "linked identity" refers to the workload identity associated with an agent runtime. In observed deployments, its name matches the runtime ID.

How Linked Identity Is Captured

When the AgentCore agent inventory is imported through AI automations, OneTrust collects the following attributes for each agent and displays them in the Linked identity section on the agent's record.

Attribute	Description
Credential type	Type of outbound credential the agent uses, such as OAUTH2 or API_KEY.
Linked identity name	Name of the agent's AgentCore workload identity (same as the runtime ID).
Workload linked identity	Amazon Resource Name (ARN) of the workload identity. The ARN is the unique identifier AWS assigns to the resource, and it includes the AWS service, region, and account ID.
Linked identity created date	Date and time the workload identity was created.
Linked identity last updated date	Date and time the workload identity was last changed.

To verify agent or model usage metrics, deploy the lightweight worker node through AI automations and ensure the inventory of agents and models is being imported through the AI automations. There are no additional configurations required. 

Once the feature is working in OneTrust AI Governance, observability metrics are surfaced directly on the agent or model record on the Details tab under Observability > Usage summary, allowing users to:

• Review the usage summary aggregated for the last 24 hours. 

• Correlate usage summary to evaluate the successful adoption of specific AI projects.

Known Limitations

• Usage data is aggregated to 24-hour windows since the last sync, and larger time windows are not currently supported. 

• The absolute cost incurred in monetary value (in $) is not provided or collected.

Note:

For model evaluation metrics, refer to AI Governance Model Evaluation Metrics.

To monitor and enforce thresholds for metrics reported, it is highly encouraged to set up custom policies and track individual metrics or a group of metrics against an organizationally defined threshold. A breach in these thresholds results in policy violations and allows deviations in a large agent inventory to surface.

For more information on creating custom policies, see AI Policies Feature Guide.