Secrets Store CSI Driver

Overview

The Secrets Store CSI Driver mode provides secret management for clusters where External Secrets Operator cannot be installed due to organizational policies, cluster restrictions, or existing CSI driver deployments.

When to Use CSI Driver vs External Secrets Operator:Use CSI Secret Store when:

External Secrets Operator cannot be installed in your cluster
Secrets Store CSI Driver is already deployed
Cluster policies restrict which operators can run

Use External Secrets Operator when:

You have flexibility to install ESO
You need multi-backend support (Vault, GCP, etc.)
You prefer a more declarative secret management approach

See External Secrets Operator Guide for ESO setup.

How It Works

The CSI Secret Store integration uses three components:

SecretProviderClass: Defines which AWS Secrets Manager secrets to fetch and how to map them to Kubernetes Secret keys
CSI Syncer Deployment: A minimal pod that mounts the CSI volume to materialize the Kubernetes Secret (the CSI driver requires an active mount)
Kubernetes Secrets: Standard crewai-platform-secrets consumed by workloads (identical to inline or ESO modes)

Key Differences from External Secrets Operator:

CSI driver materializes secrets via volume mounts instead of a controller reconciliation loop
Requires a pod to keep the volume mounted (handled automatically by the chart’s csi-syncer deployment)
Currently supports AWS Secrets Manager only (ESO supports multiple backends)

Prerequisites

Before enabling CSI Secret Store mode, ensure the following components are installed in your cluster:

1. Secrets Store CSI Driver

Install the core CSI driver with syncSecret.enabled=true:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set syncSecret.enabled=true

Critical: You must set syncSecret.enabled=true during installation. This feature cannot be enabled after initial installation without re-creating the driver.

2. AWS Provider for Secrets Store CSI Driver

Install the AWS-specific provider, pinned to a release tag:

PROVIDER_VERSION=3.1.4
curl -fsSLO https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/${PROVIDER_VERSION}/deployment/aws-provider-installer.yaml
sha256sum aws-provider-installer.yaml
kubectl apply -f aws-provider-installer.yaml

Do not install this from the main branch. The provider is a privileged, node-level component that reads every secret this chart mounts, so fetching it from a moving branch means applying whatever that branch holds at the moment you run the command. Pin a tag, record the checksum you applied, and review the release notes before moving to a newer version.

3. IAM Permissions

Configure IAM permissions for the pods to access AWS Secrets Manager. Choose one method:

IRSA (Recommended for EKS)
Node IAM Role
EKS Pod Identity

IAM Roles for Service Accounts (IRSA) provides fine-grained, pod-level permissions.Create IAM Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-2:123456789012:secret:crewai/platform/*"
      ]
    }
  ]
}

Create IAM Role with Trust Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": [
            "system:serviceaccount:crewai-platform:crewai-platform-csi-syncer",
            "system:serviceaccount:crews:platform-secrets-csi-syncer"
          ]
        }
      }
    }
  ]
}

Reference the Role in Helm Values:

csiSecret:
  aws:
    roleArn: "arn:aws:iam::123456789012:role/crewai-csi-secrets-reader"

Node IAM Role grants permissions to all pods on the node.Attach Policy to Node Role:Attach the IAM policy from the IRSA tab to your EKS node group’s IAM role.Helm Configuration:Leave csiSecret.aws.roleArn empty:

csiSecret:
  aws:
    roleArn: ""  # Empty = use node role

Trade-offs:

Simpler configuration (no IRSA setup)
Broader permissions scope (all pods on node)
Less secure for multi-tenant clusters

EKS Pod Identity is AWS’s newer authentication method (requires EKS 1.24+).Create Pod Identity Association:

aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace crewai-platform \
  --service-account crewai-platform-csi-syncer \
  --role-arn arn:aws:iam::123456789012:role/crewai-csi-secrets

Helm Configuration:Leave csiSecret.aws.roleArn empty (Pod Identity uses AWS SDK credential chain):

csiSecret:
  aws:
    roleArn: ""  # Empty = use Pod Identity

Setup Instructions

1. Prepare Secrets in AWS Secrets Manager

Create secrets in AWS Secrets Manager with the required structure.

Platform Secrets

Create a secret containing all platform keys as a flat JSON object:

aws secretsmanager create-secret \
  --name crewai/platform/prod \
  --description "CrewAI Platform secrets" \
  --region us-west-2 \
  --secret-string '{
    "secret_key_base": "'"$(openssl rand -hex 64)"'",
    "encryption_key": "'"$(openssl rand -hex 32)"'",
    "active_record_encryption_primary_key": "'"$(openssl rand -hex 32)"'",
    "active_record_encryption_deterministic_key": "'"$(openssl rand -hex 32)"'",
    "active_record_encryption_key_derivation_salt": "'"$(openssl rand -hex 32)"'",
    "internal_api_key": "'"$(openssl rand -hex 32)"'",
    "github_token": "ghp_YOUR_GITHUB_TOKEN",
    "github_crew_studio_token": "ghp_YOUR_CREW_STUDIO_TOKEN",
    "registry_http_secret": "'"$(openssl rand -hex 32)"'",
    "factory_debug_token": "'"$(openssl rand -hex 32)"'",
    "deployment_instance_jwt_secret": "'"$(openssl rand -hex 32)"'",
    "oidc_private_key": "'"$(openssl rand -base64 32)"'",
    "oidc_key_id": "'"$(uuidgen)"'"
  }'

Property Names: The chart expects lowercase, underscore-separated property names by default. Use csiSecret.propertyMap to override if your secrets use different naming conventions.

Database Password Secret

Create a separate secret for the database password:

aws secretsmanager create-secret \
  --name crewai/platform/prod/db \
  --description "CrewAI Platform database password" \
  --region us-west-2 \
  --secret-string '{
    "password": "'"$(openssl rand -base64 32 | tr -d '\n')"'"
  }'

Why Separate? Separating the database password allows for independent rotation without touching platform secrets.

OAuth Secrets (Optional)

If using the OAuth service (oauth.enabled: true), create OAuth secrets:

Shared Secret (Default)
Separate OAuth Secret

Add OAuth properties to the main platform secret:

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --region us-west-2 \
  --secret-string '{
    "...existing properties...",
    "oauth_cookie_secret": "'"$(openssl rand -hex 32)"'",
    "oauth_db_encryption_key": "'"$(openssl rand -hex 32)"'",
    "oauth_internal_api_key": "'"$(openssl rand -hex 32)"'"
  }'

No additional Helm configuration needed — csiSecret.oauthSecretPath defaults to csiSecret.secretPath.

Create a dedicated secret for OAuth:

aws secretsmanager create-secret \
  --name crewai/platform/prod/oauth \
  --description "CrewAI OAuth service secrets" \
  --region us-west-2 \
  --secret-string '{
    "oauth_cookie_secret": "'"$(openssl rand -hex 32)"'",
    "oauth_db_encryption_key": "'"$(openssl rand -hex 32)"'",
    "oauth_internal_api_key": "'"$(openssl rand -hex 32)"'"
  }'

Configure in Helm values:

csiSecret:
  oauthSecretPath: "crewai/platform/prod/oauth"

oauth:
  enabled: true

2. Configure Helm Values

Create a values.yaml file with CSI Secret Store configuration:

# Enable CSI Secret Store mode
csiSecret:
  enabled: true
  provider: aws

  # AWS Secrets Manager paths
  secretPath: "crewai/platform/prod"
  databaseSecretPath: "crewai/platform/prod/db"
  # oauthSecretPath: "crewai/platform/prod/oauth"  # Optional separate OAuth secret

  # AWS configuration
  aws:
    region: "us-west-2"
    # roleArn: "arn:aws:iam::123456789012:role/crewai-csi-secrets"  # For IRSA

# Disable External Secrets Operator mode (mutually exclusive)
externalSecret:
  enabled: false

# Database configuration
envVars:
  DB_HOST: "my-postgres.us-west-2.rds.amazonaws.com"
  DB_NAME: "crewai_production"
  DB_USER: "crewai"
  # DB_PASSWORD comes from csiSecret.databaseSecretPath

3. Deploy the Chart

Install or upgrade the chart with CSI Secret Store enabled:

helm upgrade --install crewai-platform \
  oci://registry.crewai.com/crewai/stable/crewai-platform \
  --namespace crewai-platform \
  --create-namespace \
  --values values.yaml \
  --wait

What Happens During Installation:

SecretProviderClass Created (hook weight -40): Defines which secrets to fetch
CSI Syncer Deployed (hook weight -40): Pod mounts CSI volume, materializing the Secret
Wait Job Runs (hook weight -20): Validates all required keys are present
Migrations Run (hook weights -5, 0): Database schema updates
Workloads Start: Web, worker, and other pods consume the synced secrets

4. Verify Secret Synchronization

Check that secrets were successfully materialized:

# Verify SecretProviderClass exists
kubectl get secretproviderclass -n crewai-platform

# Verify CSI syncer pod is running
kubectl get pods -n crewai-platform -l app.kubernetes.io/component=csi-syncer

# Verify secrets were created
kubectl get secret crewai-platform-secrets -n crewai-platform

# Check secret keys (values are base64-encoded)
kubectl describe secret crewai-platform-secrets -n crewai-platform

Expected Output:

Name:         crewai-platform-secrets
Namespace:    crewai-platform
Type:         Opaque

Data
====
DB_PASSWORD:                     44 bytes
ENCRYPTION_KEY:                  64 bytes
GITHUB_TOKEN:                    40 bytes
SECRET_KEY_BASE:                 128 bytes
...

Configuration Scenarios

GitHub App Integration

When using GitHub App authentication (envVars.GITHUB_APP_ID set), add these properties to your platform secret:

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --region us-west-2 \
  --secret-string '{
    "...existing properties...",
    "github_client_secret": "your-github-app-client-secret",
    "github_webhook_secret": "your-webhook-secret",
    "github_app_private_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n"
  }'

The chart automatically includes these secrets when envVars.GITHUB_APP_ID is configured.

Auth Provider Integration

When using external auth providers (envVars.AUTH_PROVIDER set), add the provider-specific secret:

Microsoft Entra ID
Auth0
Okta

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --secret-string '{
    "...existing properties...",
    "entra_client_secret": "your-entra-client-secret"
  }'

envVars:
  AUTH_PROVIDER: "entra_id"
  ENTRA_ID_CLIENT_ID: "your-client-id"
  ENTRA_ID_TENANT_ID: "your-tenant-id"

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --secret-string '{
    "...existing properties...",
    "auth0_client_secret": "your-auth0-client-secret"
  }'

envVars:
  AUTH_PROVIDER: "auth0"
  AUTH0_CLIENT_ID: "your-client-id"
  AUTH0_DOMAIN: "your-tenant.auth0.com"

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --secret-string '{
    "...existing properties...",
    "okta_client_secret": "your-okta-client-secret"
  }'

envVars:
  AUTH_PROVIDER: "okta"
  OKTA_CLIENT_ID: "your-client-id"
  OKTA_DOMAIN: "your-domain.okta.com"

OAuth Provider Credentials

When the OAuth service is enabled, the chart requests each configured provider’s credentials automatically. You do not list them in oauthPropertyMap.

Base credentials only. The chart expands each provider’s base client ID and secret across all of that provider’s product integrations, exactly as inline mode does. Configuring Google means storing two properties, not sixteen: google_client_id and google_client_secret become GOOGLE_CAL_*, GOOGLE_GMAIL_*, GOOGLE_DRIVE_*, GOOGLE_CONTACTS_*, GOOGLE_SHEETS_*, GOOGLE_SLIDES_* and GOOGLE_DOCS_*. Microsoft behaves the same way across Outlook, OneDrive, SharePoint, Teams, Word and Excel, including the tenant ID when you set one.

The properties the chart requests, per provider configured under oauth.secrets:

Configured block	Backend properties requested
`google`	`google_client_id`, `google_client_secret`, plus `google_drive_picker_key` and `google_gmail_service_account_json` when set
`microsoft`	`microsoft_client_id`, `microsoft_client_secret`, plus `microsoft_tenant_id` when set
`hubspot`	`hubspot_client_id`, `hubspot_client_secret`
`notion`	`notion_client_id`, `notion_client_secret`
`salesforce`	`salesforce_client_id`, `salesforce_client_secret`

Example (Google and Microsoft):

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod/oauth \
  --region us-west-2 \
  --secret-string '{
    "oauth_cookie_secret": "...",
    "oauth_db_encryption_key": "...",
    "oauth_internal_api_key": "...",
    "google_client_id": "your-google-client-id.apps.googleusercontent.com",
    "google_client_secret": "your-google-client-secret",
    "microsoft_client_id": "your-microsoft-client-id",
    "microsoft_client_secret": "your-microsoft-client-secret"
  }'

csiSecret:
  oauthSecretPath: "crewai/platform/prod/oauth"

oauth:
  enabled: true
  secrets:
    google:
      clientId: "your-google-client-id.apps.googleusercontent.com"
      clientSecret: "placeholder-sourced-from-backend"
    microsoft:
      clientId: "your-microsoft-client-id"
      clientSecret: "placeholder-sourced-from-backend"

The oauth.secrets.* blocks declare which providers are in use, which is what tells the chart to request their properties. In CSI mode the values themselves come from the backend, so the secret fields do not need to hold the real value.

Every property the chart requests must exist in the backend. The AWS provider fails the entire mount when any single property is missing, not just the one key, so adding a provider to oauth.secrets without adding its properties to Secrets Manager stops all secrets from syncing.

When to use `oauthPropertyMap`

Only two cases need it:

Renaming a backend property. If your secret stores the Google client secret as google_oauth_secret rather than google_client_secret:
csiSecret: oauthPropertyMap: GOOGLE_CLIENT_SECRET: "google_oauth_secret"
A product-specific credential that differs from the provider default. If Gmail uses a different client than the rest of your Google integration, list that key so it is sourced from its own property instead of inheriting the base one:
csiSecret: oauthPropertyMap: GOOGLE_GMAIL_CLIENT_ID: "gmail_client_id" GOOGLE_GMAIL_CLIENT_SECRET: "gmail_client_secret"

An entry in oauthPropertyMap always wins over the fan-out for that one key; every other key keeps inheriting the provider default. See the OAuth Provider Credentials table for the complete list of keys the OAuth Secret can carry.

Optional Services

Wharf Service (wharf.enabled: true):

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --secret-string '{
    "...existing properties...",
    "wharf_jwt_secret": "'"$(openssl rand -hex 32)"'",
    "wharf_postgres_url": "postgresql://user:password@host:5432/wharf_db"
  }'

Wharf Database URL: Unlike External Secrets Operator (which can compose the URL from individual parts), CSI mode requires the complete wharf_postgres_url stored in AWS Secrets Manager. The CSI driver has no templating engine.

Cube.js Service (cube.enabled: true):

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --secret-string '{
    "...existing properties...",
    "cube_jwt_secret": "'"$(openssl rand -hex 32)"'"
  }'

Rotating Secrets

To rotate secrets in CSI Secret Store mode:

1. Update Secret in AWS Secrets Manager

aws secretsmanager update-secret \
  --secret-id crewai/platform/prod \
  --region us-west-2 \
  --secret-string '{
    "...updated properties..."
  }'

2. Trigger Secret Re-sync

The CSI driver periodically polls for updates (default: 2 minutes). To force immediate update, restart the CSI syncer pod:

kubectl rollout restart deployment/crewai-platform-csi-syncer -n crewai-platform

3. Restart Workloads

Pods do not automatically reload secrets from the Secret resource. Trigger a rolling restart:

# Restart web pods
kubectl rollout restart deployment/crewai-platform-web -n crewai-platform

# Restart worker pods
kubectl rollout restart deployment/crewai-platform-worker -n crewai-platform

# Restart OAuth service (if enabled)
kubectl rollout restart deployment/crewai-platform-oauth -n crewai-platform

Automatic Rotation: Unlike External Secrets Operator (which supports reloadOnChange annotations), CSI mode requires manual pod restarts after secret updates. Consider implementing an automated restart mechanism if frequent rotation is required.

Limitations and Considerations

Compared to External Secrets Operator

Feature	CSI Secret Store	External Secrets Operator
Backend Support	AWS Secrets Manager only	AWS, Azure, GCP, Vault, etc.
Auto-Generated Secrets	Not supported	Fully supported
TLS Auto-Generation	Not supported	Fully supported
Secret Templating	Limited (no composition)	Full templating support
Rotation	Manual pod restart	Automatic with `reloadOnChange`
Cross-Namespace	Automatic	Requires ESO configuration
Cluster Requirements	CSI driver installed	ESO installed

Unsupported Features

TLS Auto-Generation: The chart’s web.tls.autoGenerate feature is not compatible with CSI mode. Alternatives:

Store certificates in AWS Secrets Manager (properties: platform_ssl_key, platform_ssl_cert, crew_ssl_cert, crew_ssl_key)
Terminate TLS externally (load balancer or ingress controller)

ArgoCD Deployments: CSI mode works with ArgoCD, but you must manually create secrets in AWS Secrets Manager before deployment. The chart cannot auto-generate secrets via lookup + randAlphaNum in GitOps workflows.

Security Best Practices

Least Privilege IAM: Grant only secretsmanager:GetSecretValue and secretsmanager:DescribeSecret for specific secret ARNs
Separate Secrets: Use different secrets for database, OAuth, and platform secrets for rotation isolation
Enable CloudTrail: Audit secret access via AWS CloudTrail logs
Rotate Regularly: Implement a secret rotation schedule (database passwords, API keys, etc.)
Use IRSA: Prefer IRSA over node IAM roles for fine-grained permissions

Troubleshooting

Pods in CrashLoopBackOff

Symptom: Workload pods crash with “missing environment variable” errors. Cause: Secret not synced or missing required keys. Solution:

Check CSI syncer pod status:

kubectl logs -n crewai-platform -l app.kubernetes.io/component=csi-syncer

Verify SecretProviderClass configuration:

kubectl get secretproviderclass crewai-platform-csi -n crewai-platform -o yaml

Check IAM permissions (pod should have secretsmanager:GetSecretValue):

# From within the csi-syncer pod
kubectl exec -it deploy/crewai-platform-csi-syncer -n crewai-platform -- sh
aws sts get-caller-identity

Validate secret exists in AWS Secrets Manager:

aws secretsmanager describe-secret --secret-id crewai/platform/prod --region us-west-2

Wait Job Timeout

Symptom: Pre-upgrade wait job times out before secrets sync. Cause: CSI driver not installed or misconfigured. Solution:

Verify CSI driver is running:

kubectl get pods -n kube-system | grep csi-secrets-store

Check if syncSecret.enabled=true:

kubectl get daemonset csi-secrets-store-secrets-store-csi-driver -n kube-system -o jsonpath='{.spec.template.spec.containers[0].args}'

Output should include --enable-secret-rotation=true.

IAM Permission Errors

Symptom: CSI driver logs show AccessDeniedException from AWS Secrets Manager. Solution:

Verify IAM role permissions:

aws iam get-role-policy \
  --role-name crewai-csi-secrets-reader \
  --policy-name SecretsAccess

Check IRSA trust policy (if using IRSA):

aws iam get-role \
  --role-name crewai-csi-secrets-reader \
  --query 'Role.AssumeRolePolicyDocument'

Verify ServiceAccount annotation:
kubectl get serviceaccount crewai-platform-csi-syncer -n crewai-platform -o yaml
Should contain eks.amazonaws.com/role-arn: arn:aws:iam::... annotation.

Secret Property Not Found

Symptom: CSI driver logs show Property not found: <property_name>. Cause: JSON property name mismatch between AWS secret and chart expectations. Solution: Use csiSecret.propertyMap to override property names:

csiSecret:
  propertyMap:
    SECRET_KEY_BASE: "my_custom_secret_key"  # Maps to 'my_custom_secret_key' in AWS

CSI Secret Reference - Complete configuration reference
External Secrets Operator - Alternative secret management
Secrets Configuration - Direct secret values (inline mode)
AWS IAM Setup Guide - AWS-specific IAM and IRSA configuration

Core Configuration

Authentication

Built-In Integrations

Advanced

Overview

How It Works

Prerequisites

1. Secrets Store CSI Driver

2. AWS Provider for Secrets Store CSI Driver

3. IAM Permissions

Setup Instructions

1. Prepare Secrets in AWS Secrets Manager

Platform Secrets

Database Password Secret

OAuth Secrets (Optional)

2. Configure Helm Values

3. Deploy the Chart

4. Verify Secret Synchronization

Configuration Scenarios

GitHub App Integration

Auth Provider Integration

OAuth Provider Credentials

When to use `oauthPropertyMap`

Optional Services

Rotating Secrets

1. Update Secret in AWS Secrets Manager

2. Trigger Secret Re-sync

3. Restart Workloads

Limitations and Considerations

Compared to External Secrets Operator

Unsupported Features

Security Best Practices

Troubleshooting

Pods in CrashLoopBackOff

Wait Job Timeout

IAM Permission Errors

Secret Property Not Found

​Overview

​How It Works

​Prerequisites

​1. Secrets Store CSI Driver

​2. AWS Provider for Secrets Store CSI Driver

​3. IAM Permissions

​Setup Instructions

​1. Prepare Secrets in AWS Secrets Manager

​Platform Secrets

​Database Password Secret

​OAuth Secrets (Optional)

​2. Configure Helm Values

​3. Deploy the Chart

​4. Verify Secret Synchronization

​Configuration Scenarios

​GitHub App Integration

​Auth Provider Integration

​OAuth Provider Credentials

​When to use oauthPropertyMap

​Optional Services

​Rotating Secrets

​1. Update Secret in AWS Secrets Manager

​2. Trigger Secret Re-sync

​3. Restart Workloads

​Limitations and Considerations

​Compared to External Secrets Operator

​Unsupported Features

​Security Best Practices

​Troubleshooting

​Pods in CrashLoopBackOff

​Wait Job Timeout

​IAM Permission Errors

​Secret Property Not Found

​Related Documentation

Overview

How It Works

Prerequisites

1. Secrets Store CSI Driver

2. AWS Provider for Secrets Store CSI Driver

3. IAM Permissions

Setup Instructions

1. Prepare Secrets in AWS Secrets Manager

Platform Secrets

Database Password Secret

OAuth Secrets (Optional)

2. Configure Helm Values

3. Deploy the Chart

4. Verify Secret Synchronization

Configuration Scenarios

GitHub App Integration

Auth Provider Integration

OAuth Provider Credentials

When to use `oauthPropertyMap`

Optional Services

Rotating Secrets

1. Update Secret in AWS Secrets Manager

2. Trigger Secret Re-sync

3. Restart Workloads

Limitations and Considerations

Compared to External Secrets Operator

Unsupported Features

Security Best Practices

Troubleshooting

Pods in CrashLoopBackOff

Wait Job Timeout

IAM Permission Errors

Secret Property Not Found

Related Documentation