Overview
The CrewAI Platform chart can source every runtime secret from an external secret store via the External Secrets Operator. When enabled fully, your backend (AWS Secrets Manager, Vault, etc.) becomes the canonical source of truth — the chart neither generates random values nor reads any secret material fromvalues.yaml.
This page covers:
- How CrewAI’s chart consumes secrets — three modes, what each looks like
- Fresh install with ESO — clean deployment from day one
- Migration paths — moving an existing deployment onto ESO
- Backend key reference — what to populate, where
- The pre-upgrade wait Job — how the chart sequences sync
- Troubleshooting — what to do when things go wrong
This page applies to direct Helm and ArgoCD deployments. KOTS / Replicated deployments auto-populate the secrets stored in
crewai-secrets from license fields and do not need ESO. If you’re on KOTS and want to layer ESO on top, the same flags apply but you’ll need to coordinate with Replicated’s secret hydration.How the chart consumes secrets
The chart supports three modes, controlled by two values:externalSecret.enabled and externalSecret.manageAutoGenerated.
| Mode | externalSecret.enabled | externalSecret.manageAutoGenerated | Where secrets come from |
|---|---|---|---|
| A. Inline | false | (ignored) | Chart generates auto-gen values via lookup + randAlphaNum. Pre-set values in secrets: block override. |
| B. Partial ESO | true | false | ESO pulls DB, GitHub, SSL, AWS/Azure keys. Auto-gen keys are missing from the materialized Secret — pods CrashLoop. |
| C. Full ESO | true | true | ESO pulls every key the chart’s pods need from your backend. WHARF_POSTGRES_URL and CREWAI_OAUTH_API_KEY are derived inside ESO; everything else is fetched. |
manageAutoGenerated flag was introduced and is not a viable production state — it’s the historical default that ships an incomplete Secret. Mode C is what you want for any deployment that treats the external store as the source of truth.
For the rest of this page, “ESO” means Mode C unless explicitly noted.
Fresh install with ESO
If you are deploying CrewAI Platform for the first time and want ESO from day one, follow these steps in order.Prerequisites
- External Secrets Operator installed and healthy in the cluster (typically in the
external-secretsnamespace). - A
SecretStoreresource that can authenticate against your backend. The chart can render one viasecretStore.enabled: true(see Configuration Reference), or you can bring your own. - An empty path reserved in your backend for CrewAI secrets (e.g. AWS SM secret
crewai/platform/prod). For OAuth, optionally a second path (e.g.crewai/platform/prod/oauth).
1. Generate the auto-gen secret values
The chart generates these values for you on a non-ESO install vialookup + randAlphaNum. Even for a fresh ESO install, the easiest way to get reasonable defaults is to render them once locally:
Alternative: generate with openssl
Alternative: generate with openssl
If you prefer to generate independently of the chart, use these commands. The
| tr -dc 'A-Za-z0-9' | head -c N filter mirrors the chart’s randAlphaNum
output exactly — pure alphanumeric, no +/= from base64 or special characters
that would otherwise drift from the values a non-ESO chart install produces.2. Upload to your backend
Store the values as a JSON document atexternalSecret.secretPath. Property names must match the lowercased Secret key (SECRET_KEY_BASE → secret_key_base) unless you override via autoGeneratedPropertyMap.
3. Configure values
secrets: block. With externalSecret.enabled=true the chart skips crewai-secrets.yaml entirely; values there are silently ignored.
4. Install
helm install.
Migration paths
If you have an existing CrewAI deployment and are moving onto ESO, follow the path that matches your current state.Path 1 — From inline secrets (Mode A) to full ESO (Mode C)
You’re currently running withexternalSecret.enabled: false. The chart generates the auto-gen secrets and stores them in the in-cluster Secret <release>-secrets. You want to move to ESO.
This is the safest migration path. The existing Secret is annotated with helm.sh/resource-policy: keep — it survives the cutover, and ESO adopts it on first reconcile.
Step 1 — Extract the current values. They’re what your pods are currently using; reusing them avoids rotating session-encryption keys (which would invalidate all existing user sessions) or Active Record encryption keys (which would render encrypted database columns unreadable).
oauth.enabled: true, also extract <release>-oauth-secrets.
Step 2 — Upload to your backend. Map each Secret key to its lowercased property name (or whatever your autoGeneratedPropertyMap overrides specify). For OIDC_PRIVATE_KEY, use jq --rawfile to preserve newlines.
secrets: block from your values file — it has no effect with ESO enabled, and keeping it gives future readers a false sense that those values are wired in.
keep). The new ExternalSecret is created during the pre-upgrade hook, ESO adopts the existing Secret (sets its OwnerReference), the wait Job verifies completeness, and migrations run. Pods do not need to restart unless any backend value differs from the current in-cluster value.
Path 2 — From partial ESO (Mode B) to full ESO (Mode C)
You’re currently running withexternalSecret.enabled: true and manageAutoGenerated unset (or false). Your <release>-secrets materialized Secret is partial — it has the keys the old ExternalSecret requested (DB, GitHub, SSL, AWS/Azure) but is missing the auto-gen keys. Pods are likely already failing on missing env vars, or surviving only because the Secret was created by an even older non-ESO install and the old keys have been frozen in place via helm.sh/resource-policy: keep.
This path requires a one-time orphan-delete because Helm refuses to convert a regular resource into a hook resource in place. The new chart annotates the ExternalSecret as a pre-install,pre-upgrade hook; on first sync, Helm sees a same-name non-hook resource already exists and aborts with:
MISS, extract the value from the live cluster Secret (still alive thanks to keep):
--cascade=orphan strips the OwnerReference from the Secret instead of garbage-collecting it.
Path 3 — From values-file pre-set secrets to ESO
You may have been on a hybrid where you pre-set every secret in yoursecrets: block in values.yaml (the Pre-Set All Secrets pattern). Migrating to ESO is the same as Path 1 — extract the values from the cluster Secret (or directly from your values file), upload to the backend, then flip the flags. Delete the secrets: block once values are confirmed in the backend.
Backend key reference
Every property the chart will request from your secret store, with defaults and conditions.Main secret (at externalSecret.secretPath)
Pulled when manageAutoGenerated: true. Property names below are the defaults; override per-key via autoGeneratedPropertyMap.
| Property name | Maps to env var | Always required? |
|---|---|---|
password | DB_PASSWORD (at databaseSecretPath, not secretPath) | yes |
secret_key_base | SECRET_KEY_BASE | yes |
encryption_key | ENCRYPTION_KEY | yes |
active_record_encryption_primary_key | ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY | yes |
active_record_encryption_deterministic_key | ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY | yes |
active_record_encryption_key_derivation_salt | ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT | yes |
registry_http_secret | REGISTRY_HTTP_SECRET | yes |
factory_debug_token | FACTORY_DEBUG_TOKEN | yes |
deployment_instance_jwt_secret | DEPLOYMENT_INSTANCE_JWT_SECRET | yes |
oidc_private_key | OIDC_PRIVATE_KEY (multi-line PEM) | yes |
oidc_key_id | OIDC_KEY_ID (must rotate with oidc_private_key) | yes |
wharf_jwt_secret | WHARF_JWT_SECRET | only if wharf.enabled: true |
cube_jwt_secret | CUBE_JWT_SECRET | only if cube.enabled: true |
internal_api_key | CREWAI_PLUS_INTERNAL_API_KEY | yes |
github_token | GITHUB_TOKEN | yes (chart-managed) |
github_crew_studio_token | GITHUB_CREW_STUDIO_TOKEN | yes |
github_client_secret | GITHUB_CLIENT_SECRET | yes |
github_webhook_secret | GITHUB_WEBHOOK_SECRET_TOKEN | yes |
github_app_private_key | GITHUB_APP_PRIVATE_KEY (multi-line PEM) | yes |
platform_ssl_key | SSL_PRIVATE_KEY | yes |
platform_ssl_cert | SSL_CERTIFICATE | yes |
crew_ssl_key | CREW_SSL_KEY | yes |
crew_ssl_cert | CREW_SSL_CERT | yes |
aws_access_key_id / aws_secret_access_key | AWS_* | only if includes_aws_credentials: true |
azure_storage_access_key / azure_client_secret / entra_client_secret | AZURE_* / ENTRA_ID_CLIENT_SECRET | only if includes_azure_credentials: true |
Synthesized / auto-linked (do NOT put in the backend)
These appear in the materialized Secret but you do not populate them yourself:WHARF_POSTGRES_URL— whenwharf.enabled: true. The chart synthesizes this at materialization time aspostgres://<DB_USER>:<DB_PASSWORD>@<host>:<port>/<wharf_db_name>.DB_PASSWORDis pulled from your backend; everything else comes from chart values. RotatingDB_PASSWORDin your backend propagates here automatically on the next ESO refresh — no second key to keep in sync.CREWAI_OAUTH_API_KEY— whenoauth.enabled: true. Sourced from the same backend property asOAUTH_INTERNAL_API_KEY(defaultoauth_internal_api_keyatoauthSecretPath). The chart guarantees the two values are byte-identical, eliminating a silent OAuth-to-Rails 401 failure mode.
values.schema.json rejects attempts to override autoGeneratedPropertyMap.CREWAI_OAUTH_API_KEY or autoGeneratedPropertyMap.WHARF_POSTGRES_URL to enforce this invariant.
OAuth secret (at externalSecret.oauthSecretPath)
Rendered as a second ExternalSecret targeting <release>-oauth-secrets when both oauth.enabled and manageAutoGenerated are true. Falls back to secretPath if oauthSecretPath is empty.
| Property name | Maps to env var | Required |
|---|---|---|
oauth_cookie_secret | OAUTH_COOKIE_SECRET | yes |
oauth_db_encryption_key | OAUTH_DB_ENCRYPTION_KEY | yes |
oauth_internal_api_key | OAUTH_INTERNAL_API_KEY (also feeds CREWAI_OAUTH_API_KEY) | yes |
GOOGLE_*, MICROSOFT_*, HUBSPOT_*, NOTION_*, SALESFORCE_*) are not requested by default. Add them to oauthPropertyMap if you use those OAuth providers for end-user integrations:
UV_DEFAULT_INDEX (the private PyPI registry URL) deserves a special callout. In KOTS deployments it auto-derives from the Replicated license; in direct-Helm/ArgoCD deployments it must be set explicitly. With ESO enabled, only the autoGeneratedPropertyMap route works — values placed under secrets.UV_DEFAULT_INDEX are ignored because the inline Secret template does not render. Put the URL in your backend at externalSecret.secretPath (property uv_default_index) and list it in the map.The pre-upgrade wait Job
WhenmanageAutoGenerated: true and waitForSync: true (the default), the chart renders a pre-upgrade Job (<release>-wait-for-eso-<revision>) that blocks Helm until ESO has populated the materialized Secret with every required key.
This sequences ESO synchronization ahead of every pre-upgrade hook that depends on the auto-generated keys — Rails db:migrate, wharf migrate, oauth migrate. Without it, those migrations would race against the ExternalSecret being created during the same phase, hitting couldn't find key X in Secret.
Helm hook weight ordering when manageAutoGenerated: true:
Troubleshooting
Wait Job times out
<KEY_LIST> is exactly the set of property names missing from your backend. Add them and re-run helm upgrade.
If the dump shows ESO errors (e.g. secretsmanager:GetSecretValue Forbidden), the issue is in your SecretStore configuration — IAM role trust policy, IRSA annotation on the ServiceAccount, or the role’s permission boundary. Fix at the platform level; ESO reconciles automatically.
Wait Job hard-fails on TLS / connectivity
busybox.image.tag was pulled as a minimal variant that ships without OpenSSL. Pin to a known-good version:
Wait Job itself is broken (unrecoverable bug)
If the wait Job is producing unexpected failures and you need to unblock an upgrade immediately, two escape valves:- Chart-level disable (preferred)
- Per-upgrade bypass
Set in your values file:The wait Job stops rendering. Migrations still run as pre-upgrade hooks; they may CrashLoop briefly until ESO catches up. After the upgrade settles,
kubectl rollout restart deploy -n <ns> to refresh pod env caches.”ExternalSecret already exists” on first upgrade to a chart with hook-based ESO
--cascade=orphan.
Materialized Secret has only WHARF_POSTGRES_URL, everything else is missing
This was a real bug fixed in chart version that introduced mergePolicy: Merge on the ExternalSecret template. If you’re seeing it on a current version, your custom ExternalSecret manifest may have been hand-modified — check that spec.target.template.mergePolicy is Merge, not Replace (the default).
OIDC_PRIVATE_KEY mangled in the Secret
The most common cause is uploading the PEM to your backend with \n literals instead of real newlines. Round-trip verify:
openssl rejects the key, re-upload using jq --rawfile (which preserves newlines through JSON encoding) instead of any approach that involves shell string interpolation.
Secret has the key but it’s empty (0 bytes)
Related
- ArgoCD Deployment — using ESO with GitOps
- Configuration Reference: External Secret — every value with type and default
- Cloud Provider Identity — setting up IRSA for the ESO
SecretStore
