Providers¶

Providers are LLM (Large Language Model) services that power your agents. Ag2Trust supports multiple providers, allowing you to choose the best model for each use case.

Supported Providers¶

Adding a Provider¶

Via Dashboard¶

1. Navigate to Settings > Providers
2. Click Add Provider
3. Select the provider type
4. Enter your API key
5. Give it a name (e.g., "OpenAI Production")
6. Click Save

Provider Configuration¶

Configuration varies by provider type:

OpenAI / Anthropic (Direct API)¶

AWS Bedrock¶

Security¶

Credential Encryption¶

Provider API keys are encrypted using AWS KMS envelope encryption:

┌─────────────────────────────────────────────────┐
│              Encryption Process                  │
│                                                  │
│  1. Generate random DEK (Data Encryption Key)   │
│  2. Encrypt API key with DEK (AES-256-GCM)      │
│  3. Encrypt DEK with KMS master key             │
│  4. Store: encrypted_key + encrypted_DEK        │
└─────────────────────────────────────────────────┘

Key Points¶

• API keys are never stored in plain text
• Keys are decrypted only when starting an agent
• Decrypted keys passed to containers via environment variables (in-memory only)
• No .env files containing credentials

Credential Flow¶

sequenceDiagram
    participant D as Dashboard
    participant B as Backend
    participant K as AWS KMS
    participant A as Agent Container

    D->>B: Save provider (API key)
    B->>K: Encrypt API key
    K-->>B: Encrypted credential
    B->>B: Store encrypted

    Note over B,A: Later: Agent Start

    B->>K: Decrypt credential
    K-->>B: Plain API key
    B->>A: Pass via env var
    A->>A: Use for LLM calls

Model Selection¶

OpenAI Models¶

Anthropic Models¶

AWS Bedrock Models¶

Bedrock provides access to multiple model families through AWS:

Choosing a Model¶

High volume + Simple tasks    → gpt-4o-mini
General purpose              → gpt-4o or claude-sonnet-4
Complex reasoning            → claude-opus-4
Code generation              → claude-sonnet-4 or mistral
Cost sensitive              → gpt-4o-mini or llama3
AWS data residency          → AWS Bedrock (any model)

Multiple Providers¶

You can configure multiple providers for different purposes:

Example Setup¶

Benefits¶

1. Quota separation: Don't let dev testing affect prod limits
2. Model specialization: Use the right model for each task
3. Failover: Switch providers if one has issues
4. Cost management: Track spending per use case

AWS Bedrock Setup¶

AWS Bedrock allows you to use models hosted in your own AWS account, providing data residency controls and consolidated AWS billing.

Prerequisites¶

1. AWS Account with Bedrock access enabled in your region
2. IAM User or Role with appropriate permissions
3. Model Access enabled in the Bedrock console for desired models

Required IAM Permissions¶

Create an IAM policy with these minimum permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": "arn:aws:bedrock:*::foundation-model/*"
    }
  ]
}

arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet*

Enable Model Access¶

Before using a model, you must request access in the AWS console:

1. Go to AWS Console > Amazon Bedrock > Model access
2. Click Manage model access
3. Select the models you want to use
4. Submit the access request (some models are instant, others require approval)

Adding a Bedrock Provider¶

1. Navigate to Settings > Providers
2. Click Add Provider
3. Select AWS Bedrock as the type
4. Enter your credentials:
5. Name: e.g., "Bedrock Production"
6. AWS Access Key ID: Your IAM access key
7. AWS Secret Access Key: Your IAM secret key
8. AWS Region: e.g., us-east-1
9. Click Save

Model ID Format¶

Bedrock uses specific model identifiers. When creating an agent type, use the full model ID:

aws bedrock list-foundation-models --region us-east-1

Using Temporary Credentials¶

For enhanced security, use AWS STS temporary credentials:

1. Generate temporary credentials via STS AssumeRole
2. Include the Session Token when adding the provider
3. Update credentials before they expire (typically 1-12 hours)

# Example: Get temporary credentials
aws sts assume-role \
  --role-arn arn:aws:iam::123456789:role/BedrockAgentRole \
  --role-session-name ag2trust-session

Bedrock-Specific Errors¶

Rate Limits¶

Provider Rate Limits¶

Each LLM provider has their own rate limits:

Ag2Trust Rate Limits¶

Ag2Trust adds additional rate limiting for protection:

Monitoring Usage¶

Dashboard Metrics¶

Track provider usage in the Dashboard:

• Tokens consumed per agent
• Response times
• Error rates
• Cost estimates

Per-Agent Statistics¶

GET /api/agents/{id}/stats

Returns: - Total tokens used - Average response time - Messages processed - Errors encountered

Best Practices¶

1. Use Separate Keys for Environments¶

Provider: openai-production
  └── Production API key with higher limits

Provider: openai-development
  └── Development API key for testing

2. Monitor Token Usage¶

Set up alerts for: - Unusual token consumption spikes - High error rates - Slow response times

3. Choose Models Wisely¶

4. Rotate Keys Periodically¶

1. Generate new API key at provider
2. Add new provider config in Ag2Trust
3. Update agent types to use new provider
4. Restart affected agents
5. Delete old provider config

Troubleshooting¶

"Invalid API Key"¶

1. Verify key is correct (no extra spaces)
2. Check key hasn't been revoked at provider
3. Ensure key has required permissions

"Rate Limit Exceeded"¶

1. Check your provider's rate limit tier
2. Reduce agent activity or add more agents
3. Consider upgrading your provider plan

"Model Not Available"¶

1. Verify model name is correct
2. Check model is available in your region
3. Some models require special access

Bedrock: "AccessDeniedException"¶

1. Enable model access in AWS Bedrock console
2. Verify IAM permissions include bedrock:InvokeModel
3. Check the model is available in your selected region

Bedrock: "Invalid credentials"¶

1. Verify AWS Access Key ID and Secret are correct
2. Check IAM user hasn't been deactivated
3. If using session tokens, ensure they haven't expired

Provider API¶

List Providers¶

GET /api/providers

Add Provider¶

POST /api/providers
Content-Type: application/json

{
  "name": "OpenAI Production",
  "type": "openai",
  "api_key": "sk-..."
}

Delete Provider¶

DELETE /api/providers/{id}