> ## Documentation Index > Fetch the complete documentation index at: https://enterprise-docs.crewai.com/llms.txt > Use this file to discover all available pages before exploring further. # External Secrets Operator (ESO) > Manage every CrewAI Platform secret through your existing secret store — AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, or HashiCorp Vault. ## Overview The CrewAI Platform chart can source **every** runtime secret from an external secret store via the [External Secrets Operator](https://external-secrets.io/). When enabled fully, your backend (AWS Secrets Manager, Vault, etc.) becomes the canonical source of truth — the chart neither generates random values nor reads any secret material from `values.yaml`. This page covers: * [How CrewAI's chart consumes secrets](#how-the-chart-consumes-secrets) — three modes, what each looks like * [Fresh install with ESO](#fresh-install-with-eso) — clean deployment from day one * [Migration paths](#migration-paths) — moving an existing deployment onto ESO * [Backend key reference](#backend-key-reference) — what to populate, where * [The pre-upgrade wait Job](#the-pre-upgrade-wait-job) — how the chart sequences sync * [Troubleshooting](#troubleshooting) — what to do when things go wrong This page applies to direct Helm and ArgoCD deployments. KOTS / Replicated deployments auto-populate the secrets stored in `crewai-secrets` from license fields and do not need ESO. If you're on KOTS and want to layer ESO on top, the same flags apply but you'll need to coordinate with Replicated's secret hydration. ## How the chart consumes secrets The chart supports three modes, controlled by two values: `externalSecret.enabled` and `externalSecret.manageAutoGenerated`. | Mode | `externalSecret.enabled` | `externalSecret.manageAutoGenerated` | Where secrets come from | | ------------------ | ------------------------ | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **A. Inline** | `false` | *(ignored)* | Chart generates auto-gen values via `lookup` + `randAlphaNum`. Pre-set values in `secrets:` block override. | | **B. Partial ESO** | `true` | `false` | ESO pulls DB, GitHub, SSL, AWS/Azure keys. Auto-gen keys are **missing** from the materialized Secret — pods CrashLoop. | | **C. Full ESO** | `true` | `true` | ESO pulls every key the chart's pods need from your backend. `WHARF_POSTGRES_URL` and `CREWAI_OAUTH_API_KEY` are derived inside ESO; everything else is fetched. | Mode A is the chart's default. Mode B existed before the `manageAutoGenerated` flag was introduced and is **not a viable production state** — it's the historical default that ships an incomplete Secret. Mode C is what you want for any deployment that treats the external store as the source of truth. For the rest of this page, "ESO" means Mode C unless explicitly noted. ## Fresh install with ESO If you are deploying CrewAI Platform for the first time and want ESO from day one, follow these steps in order. ### Prerequisites * External Secrets Operator installed and healthy in the cluster (typically in the `external-secrets` namespace). * A `SecretStore` resource that can authenticate against your backend. The chart can render one via `secretStore.enabled: true` (see [Configuration Reference](/reference/chart-values/external-secret)), or you can bring your own. * An empty path reserved in your backend for CrewAI secrets (e.g. AWS SM secret `crewai/platform/prod`). For OAuth, optionally a second path (e.g. `crewai/platform/prod/oauth`). ### 1. Generate the auto-gen secret values The chart generates these values for you on a non-ESO install via `lookup` + `randAlphaNum`. Even for a fresh ESO install, the easiest way to get reasonable defaults is to render them once locally: ```bash theme={null} helm template crewai-platform --values your-values.yaml \ | yq 'select(.kind == "Secret" and (.metadata.name | contains("-secrets"))) .data | to_entries | .[] | .key + ": " + (.value | @base64d)' \ | grep -E '^(SECRET_KEY_BASE|ENCRYPTION_KEY|ACTIVE_RECORD_ENCRYPTION_|REGISTRY_HTTP_SECRET|CREWAI_PLUS_INTERNAL_API_KEY|WHARF_JWT_SECRET|DEPLOYMENT_INSTANCE_JWT_SECRET|CUBE_JWT_SECRET|FACTORY_DEBUG_TOKEN|OIDC_PRIVATE_KEY|OIDC_KEY_ID|OAUTH_COOKIE_SECRET|OAUTH_DB_ENCRYPTION_KEY|OAUTH_INTERNAL_API_KEY)' ``` If you prefer to generate independently of the chart, use these commands. The `| tr -dc 'A-Za-z0-9' | head -c N` filter mirrors the chart's `randAlphaNum` output exactly — pure alphanumeric, no `+/=` from base64 or special characters that would otherwise drift from the values a non-ESO chart install produces. ```bash theme={null} # 64-char alphanumeric: SECRET_KEY_BASE, ACTIVE_RECORD_ENCRYPTION_* (×3), # DEPLOYMENT_INSTANCE_JWT_SECRET, FACTORY_DEBUG_TOKEN, WHARF_JWT_SECRET, # CUBE_JWT_SECRET, CREWAI_PLUS_INTERNAL_API_KEY, OAUTH_COOKIE_SECRET, # OAUTH_INTERNAL_API_KEY openssl rand -base64 64 | tr -dc 'A-Za-z0-9' | head -c 64 # 32-char alphanumeric: REGISTRY_HTTP_SECRET openssl rand -base64 32 | tr -dc 'A-Za-z0-9' | head -c 32 # 64-char hex: ENCRYPTION_KEY, OAUTH_DB_ENCRYPTION_KEY openssl rand -hex 32 # OIDC signing key (multi-line RSA PEM) openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 # OIDC key id echo "crewai-oidc-$(openssl rand -hex 8)" ``` ### 2. Upload to your backend Store the values as a JSON document at `externalSecret.secretPath`. Property names must match the lowercased Secret key (`SECRET_KEY_BASE` → `secret_key_base`) unless you override via `autoGeneratedPropertyMap`. ```bash theme={null} aws secretsmanager create-secret \ --name crewai/platform/prod \ --region us-west-2 \ --secret-string "$(jq -n \ --arg skb '' \ --arg ek '' \ --rawfile opk /path/to/oidc-private-key.pem \ '{secret_key_base: $skb, encryption_key: $ek, oidc_private_key: $opk}')" ``` See [Backend key reference](#backend-key-reference) below for the complete list. `OIDC_PRIVATE_KEY` is a multi-line PEM. Use `jq --rawfile` (as shown) to preserve newlines through JSON round-tripping. Hand-edited JSON or `--secret-string "$(cat file.pem)"` easily mangles the line breaks. ### 3. Configure values ```yaml theme={null} # your-values.yaml externalSecret: enabled: true manageAutoGenerated: true secretStore: my-secret-store secretPath: crewai/platform/prod databaseSecretPath: crewai/platform/prod/db # Optional — only if oauth.enabled and OAuth keys live at a different path oauthSecretPath: crewai/platform/prod/oauth # Optional — add extras like UV_DEFAULT_INDEX that aren't in the default set autoGeneratedPropertyMap: UV_DEFAULT_INDEX: uv_default_index ``` Do **not** populate the `secrets:` block. With `externalSecret.enabled=true` the chart skips `crewai-secrets.yaml` entirely; values there are silently ignored. ### 4. Install ```bash theme={null} helm install crewai-platform --version --values your-values.yaml -n ``` The chart's pre-upgrade flow (which also runs on first install) sequences ESO sync before pod startup. You'll see: ``` crewai-eso-wait-sa, Role, RoleBinding created (hook weight -50) crewai-external-secret created (hook weight -30) crewai-wait-for-eso- Job runs (hook weight -20) → polls until materialized Secret has every required key crewai-pre-upgrade-migration Job runs (hook weight -5) → runs Rails db:migrate against the now-complete Secret Main manifest applies, Deployments roll ``` If any backend property is missing, the wait Job times out with an explicit message naming the missing key. Add it to your backend, re-run `helm install`. ## Migration paths If you have an existing CrewAI deployment and are moving onto ESO, follow the path that matches your current state. ### Path 1 — From inline secrets (Mode A) to full ESO (Mode C) You're currently running with `externalSecret.enabled: false`. The chart generates the auto-gen secrets and stores them in the in-cluster Secret `-secrets`. You want to move to ESO. This is the **safest** migration path. The existing Secret is annotated with `helm.sh/resource-policy: keep` — it survives the cutover, and ESO adopts it on first reconcile. **Step 1 — Extract the current values.** They're what your pods are currently using; reusing them avoids rotating session-encryption keys (which would invalidate all existing user sessions) or Active Record encryption keys (which would render encrypted database columns unreadable). ```bash theme={null} NS= RELEASE= kubectl get secret -n "$NS" "$RELEASE-secrets" -o json \ | jq -r '.data | to_entries[] | "\(.key)\t\(.value | @base64d)"' \ > /tmp/existing-secrets.tsv ``` If `oauth.enabled: true`, also extract `-oauth-secrets`. **Step 2 — Upload to your backend.** Map each Secret key to its lowercased property name (or whatever your `autoGeneratedPropertyMap` overrides specify). For `OIDC_PRIVATE_KEY`, use `jq --rawfile` to preserve newlines. ```bash theme={null} # Build the AWS SM JSON, then: aws secretsmanager create-secret --name crewai/platform/prod --region us-west-2 \ --secret-string file:///tmp/platform.json ``` **Step 3 — Update values and upgrade.** ```yaml theme={null} externalSecret: enabled: true manageAutoGenerated: true secretStore: my-secret-store secretPath: crewai/platform/prod databaseSecretPath: crewai/platform/prod/db # oauthSecretPath: crewai/platform/prod/oauth # if oauth.enabled ``` **Delete the `secrets:` block from your values file** — it has no effect with ESO enabled, and keeping it gives future readers a false sense that those values are wired in. ```bash theme={null} helm upgrade --version --values your-values.yaml -n "$NS" ``` The inline Secret persists (via `keep`). The new ExternalSecret is created during the pre-upgrade hook, ESO adopts the existing Secret (sets its OwnerReference), the wait Job verifies completeness, and migrations run. Pods do not need to restart unless any backend value differs from the current in-cluster value. ### Path 2 — From partial ESO (Mode B) to full ESO (Mode C) You're currently running with `externalSecret.enabled: true` and `manageAutoGenerated` unset (or `false`). Your `-secrets` materialized Secret is partial — it has the keys the old ExternalSecret requested (DB, GitHub, SSL, AWS/Azure) but is **missing the auto-gen keys**. Pods are likely already failing on missing env vars, or surviving only because the Secret was created by an even older non-ESO install and the old keys have been frozen in place via `helm.sh/resource-policy: keep`. This path requires a **one-time orphan-delete** because Helm refuses to convert a regular resource into a hook resource in place. The new chart annotates the `ExternalSecret` as a `pre-install,pre-upgrade` hook; on first sync, Helm sees a same-name non-hook resource already exists and aborts with: ``` externalsecrets.external-secrets.io "crewai-external-secret" already exists ``` This step is **only required on the first upgrade** from a chart version that didn't use hooks for the ExternalSecret. Subsequent upgrades are fully idempotent — Helm handles the hook lifecycle automatically via `before-hook-creation,hook-failed`. **Step 1 — Confirm AWS SM has every key the new ExternalSecret will request.** This is where most migrations stall. Take inventory of what's in your backend vs. what the chart's full mode needs: ```bash theme={null} expected="secret_key_base encryption_key active_record_encryption_primary_key active_record_encryption_deterministic_key active_record_encryption_key_derivation_salt registry_http_secret factory_debug_token deployment_instance_jwt_secret oidc_private_key oidc_key_id wharf_jwt_secret github_token github_crew_studio_token github_client_secret github_webhook_secret github_app_private_key internal_api_key platform_ssl_key platform_ssl_cert crew_ssl_key crew_ssl_cert entra_client_secret" # if includes_azure_credentials actual=$(aws secretsmanager get-secret-value --secret-id crewai/platform/prod \ --region us-west-2 --query SecretString --output text | jq -r 'keys[]') for k in $expected; do echo "$actual" | grep -qx "$k" && printf " ok %s\n" "$k" || printf "MISS %s\n" "$k" done ``` For any `MISS`, extract the value from the live cluster Secret (still alive thanks to `keep`): ```bash theme={null} NS= kubectl get secret -n "$NS" -secrets -o json \ | jq -r ".data.\"\" | @base64d" ``` Upload it to the backend. **Step 2 — Orphan-delete the existing ExternalSecret(s).** This removes the CR but preserves the materialized Secret (which is what your running pods depend on). `--cascade=orphan` strips the `OwnerReference` from the Secret instead of garbage-collecting it. ```bash theme={null} kubectl delete externalsecret -n "$NS" -external-secret --cascade=orphan kubectl delete externalsecret -n "$NS" -oauth-external-secret --cascade=orphan 2>/dev/null || true ``` Running pods continue to function — they cached their env vars at startup and don't re-read the Secret. The Secret resource itself stays in the cluster with no owner; ESO will adopt it on first reconcile after the upgrade. **Step 3 — Update values:** ```yaml theme={null} externalSecret: enabled: true manageAutoGenerated: true # ← the new flag secretStore: my-secret-store # unchanged secretPath: crewai/platform/prod # unchanged databaseSecretPath: crewai/platform/prod/db # unchanged oauthSecretPath: crewai/platform/prod/oauth # if oauth.enabled ``` **Step 4 — Upgrade.** ```bash theme={null} helm upgrade --version --values your-values.yaml -n "$NS" ``` Helm now applies the hook-annotated ExternalSecret cleanly during the pre-upgrade phase, the wait Job confirms the Secret has every key, and migrations run. ### Path 3 — From values-file pre-set secrets to ESO You may have been on a hybrid where you pre-set every secret in your `secrets:` block in values.yaml (the [Pre-Set All Secrets](/configuration/argocd#recommended-pre-set-all-secrets) pattern). Migrating to ESO is the same as **Path 1** — extract the values from the cluster Secret (or directly from your values file), upload to the backend, then flip the flags. Delete the `secrets:` block once values are confirmed in the backend. ## Backend key reference Every property the chart will request from your secret store, with defaults and conditions. ### Main secret (at `externalSecret.secretPath`) Pulled when `manageAutoGenerated: true`. Property names below are the defaults; override per-key via `autoGeneratedPropertyMap`. | Property name | Maps to env var | Always required? | | -------------------------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------ | | `password` | `DB_PASSWORD` (at `databaseSecretPath`, not `secretPath`) | yes | | `secret_key_base` | `SECRET_KEY_BASE` | yes | | `encryption_key` | `ENCRYPTION_KEY` | yes | | `active_record_encryption_primary_key` | `ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY` | yes | | `active_record_encryption_deterministic_key` | `ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY` | yes | | `active_record_encryption_key_derivation_salt` | `ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT` | yes | | `registry_http_secret` | `REGISTRY_HTTP_SECRET` | yes | | `factory_debug_token` | `FACTORY_DEBUG_TOKEN` | yes | | `deployment_instance_jwt_secret` | `DEPLOYMENT_INSTANCE_JWT_SECRET` | yes | | `oidc_private_key` | `OIDC_PRIVATE_KEY` (multi-line PEM) | yes | | `oidc_key_id` | `OIDC_KEY_ID` (must rotate with `oidc_private_key`) | yes | | `wharf_jwt_secret` | `WHARF_JWT_SECRET` | only if `wharf.enabled: true` | | `cube_jwt_secret` | `CUBE_JWT_SECRET` | only if `cube.enabled: true` | | `internal_api_key` | `CREWAI_PLUS_INTERNAL_API_KEY` | yes | | `github_token` | `GITHUB_TOKEN` | yes (chart-managed) | | `github_crew_studio_token` | `GITHUB_CREW_STUDIO_TOKEN` | yes | | `github_client_secret` | `GITHUB_CLIENT_SECRET` | yes | | `github_webhook_secret` | `GITHUB_WEBHOOK_SECRET_TOKEN` | yes | | `github_app_private_key` | `GITHUB_APP_PRIVATE_KEY` (multi-line PEM) | yes | | `platform_ssl_key` | `SSL_PRIVATE_KEY` | yes | | `platform_ssl_cert` | `SSL_CERTIFICATE` | yes | | `crew_ssl_key` | `CREW_SSL_KEY` | yes | | `crew_ssl_cert` | `CREW_SSL_CERT` | yes | | `aws_access_key_id` / `aws_secret_access_key` | `AWS_*` | only if `includes_aws_credentials: true` | | `azure_storage_access_key` / `azure_client_secret` / `entra_client_secret` | `AZURE_*` / `ENTRA_ID_CLIENT_SECRET` | only if `includes_azure_credentials: true` | ### Synthesized / auto-linked (do NOT put in the backend) These appear in the materialized Secret but you do **not** populate them yourself: * **`WHARF_POSTGRES_URL`** — when `wharf.enabled: true`. The chart synthesizes this at materialization time as `postgres://:@:/`. `DB_PASSWORD` is pulled from your backend; everything else comes from chart values. Rotating `DB_PASSWORD` in your backend propagates here automatically on the next ESO refresh — no second key to keep in sync. * **`CREWAI_OAUTH_API_KEY`** — when `oauth.enabled: true`. Sourced from the *same backend property as* `OAUTH_INTERNAL_API_KEY` (default `oauth_internal_api_key` at `oauthSecretPath`). The chart guarantees the two values are byte-identical, eliminating a silent OAuth-to-Rails 401 failure mode. The chart's `values.schema.json` rejects attempts to override `autoGeneratedPropertyMap.CREWAI_OAUTH_API_KEY` or `autoGeneratedPropertyMap.WHARF_POSTGRES_URL` to enforce this invariant. ### OAuth secret (at `externalSecret.oauthSecretPath`) Rendered as a second `ExternalSecret` targeting `-oauth-secrets` when both `oauth.enabled` and `manageAutoGenerated` are true. Falls back to `secretPath` if `oauthSecretPath` is empty. | Property name | Maps to env var | Required | | ------------------------- | ------------------------------------------------------------ | -------- | | `oauth_cookie_secret` | `OAUTH_COOKIE_SECRET` | yes | | `oauth_db_encryption_key` | `OAUTH_DB_ENCRYPTION_KEY` | yes | | `oauth_internal_api_key` | `OAUTH_INTERNAL_API_KEY` (also feeds `CREWAI_OAUTH_API_KEY`) | yes | Provider client IDs/secrets (`GOOGLE_*`, `MICROSOFT_*`, `HUBSPOT_*`, `NOTION_*`, `SALESFORCE_*`) are **not** requested by default. Add them to `oauthPropertyMap` if you use those OAuth providers for end-user integrations: ```yaml theme={null} externalSecret: oauthPropertyMap: GOOGLE_CLIENT_ID: google_client_id GOOGLE_CLIENT_SECRET: google_client_secret MICROSOFT_CLIENT_ID: microsoft_client_id MICROSOFT_CLIENT_SECRET: microsoft_client_secret MICROSOFT_TENANT_ID: microsoft_tenant_id ``` `UV_DEFAULT_INDEX` (the private PyPI registry URL) deserves a special callout. In KOTS deployments it auto-derives from the Replicated license; in direct-Helm/ArgoCD deployments it must be set explicitly. With ESO enabled, **only** the `autoGeneratedPropertyMap` route works — values placed under `secrets.UV_DEFAULT_INDEX` are ignored because the inline Secret template does not render. Put the URL in your backend at `externalSecret.secretPath` (property `uv_default_index`) and list it in the map. ## The pre-upgrade wait Job When `manageAutoGenerated: true` and `waitForSync: true` (the default), the chart renders a pre-upgrade Job (`-wait-for-eso-`) that blocks Helm until ESO has populated the materialized Secret with every required key. This sequences ESO synchronization ahead of every pre-upgrade hook that depends on the auto-generated keys — Rails `db:migrate`, wharf migrate, oauth migrate. Without it, those migrations would race against the ExternalSecret being created during the same phase, hitting `couldn't find key X in Secret`. Helm hook weight ordering when `manageAutoGenerated: true`: ``` -50 SA / Role / RoleBinding (read-only on Secrets + ExternalSecrets) -30 ExternalSecret + OAuth ExternalSecret ← ESO begins reconciling -20 wait-for-eso-sync Job ← blocks until Secret has all keys -5 pre-upgrade-migration (Rails db:migrate) ← reads complete Secret 0 wharf-migrate-job, oauth-migrate-job ``` The wait Job uses BusyBox + raw HTTPS calls to the in-cluster Kubernetes API. Default timeout: 3 minutes. Successful syncs converge in under 30 seconds; the tight timeout surfaces broken syncs quickly without making operators wait. ## Troubleshooting ### Wait Job times out ``` Secret/-secrets still missing keys after 3 minutes: ``` Followed by a dump of the live ExternalSecret YAML. The `` is exactly the set of property names missing from your backend. Add them and re-run `helm upgrade`. If the dump shows ESO errors (e.g. `secretsmanager:GetSecretValue Forbidden`), the issue is in your `SecretStore` configuration — IAM role trust policy, IRSA annotation on the ServiceAccount, or the role's permission boundary. Fix at the platform level; ESO reconciles automatically. ### Wait Job hard-fails on TLS / connectivity ``` FATAL: cannot reach Kubernetes API. wget stderr was: wget: SSL not available ``` Your `busybox.image.tag` was pulled as a minimal variant that ships without OpenSSL. Pin to a known-good version: ```yaml theme={null} busybox: image: tag: "1.36" ``` ### Wait Job itself is broken (unrecoverable bug) If the wait Job is producing unexpected failures and you need to unblock an upgrade immediately, two escape valves: Set in your values file: ```yaml theme={null} externalSecret: waitForSync: false ``` The wait Job stops rendering. Migrations still run as pre-upgrade hooks; they may CrashLoop briefly until ESO catches up. After the upgrade settles, `kubectl rollout restart deploy -n ` to refresh pod env caches. For a one-off upgrade: ```bash theme={null} helm upgrade --values -n --no-hooks ``` Skips ALL hooks (including the wait Job) for this upgrade only. Subsequent upgrades run the wait Job again. Disabling the wait Job reverts to the pre-hook behavior where there is no ordering guarantee between ESO synchronization and pre-upgrade migrations. Use only as a temporary unblock while the wait Job is being patched. ### "ExternalSecret already exists" on first upgrade to a chart with hook-based ESO ``` pre-upgrade hooks failed: warning: Hook pre-upgrade ...external-secret.yaml failed: externalsecrets.external-secrets.io "-external-secret" already exists ``` You're upgrading from a chart version where the ExternalSecret was a regular manifest to one where it's a hook. This is the one-time migration documented in [Path 2](#path-2--from-partial-eso-mode-b-to-full-eso-mode-c). Orphan-delete the existing ExternalSecret(s), then re-run the upgrade. Running pods are unaffected because the materialized Secret is preserved by `--cascade=orphan`. ### Materialized Secret has only `WHARF_POSTGRES_URL`, everything else is missing This was a real bug fixed in chart version that introduced `mergePolicy: Merge` on the ExternalSecret template. If you're seeing it on a current version, your custom ExternalSecret manifest may have been hand-modified — check that `spec.target.template.mergePolicy` is `Merge`, not `Replace` (the default). ### `OIDC_PRIVATE_KEY` mangled in the Secret The most common cause is uploading the PEM to your backend with `\n` literals instead of real newlines. Round-trip verify: ```bash theme={null} aws secretsmanager get-secret-value --secret-id crewai/platform/prod --region us-west-2 \ --query SecretString --output text \ | jq -r .oidc_private_key \ | openssl rsa -check -noout # Expected: RSA key ok ``` If `openssl` rejects the key, re-upload using `jq --rawfile` (which preserves newlines through JSON encoding) instead of any approach that involves shell string interpolation. ### Secret has the key but it's empty (`0 bytes`) ``` GITHUB_CREW_STUDIO_TOKEN: 0 bytes ``` ESO fetched an empty string from your backend. Either the property exists with an empty value, or your backend treats absent properties as empty strings. Pods consuming such a value will fail when the value is actually needed. Populate the property — even if your deployment doesn't use the feature, an empty string is safer than failing at runtime. ## Related * [ArgoCD Deployment](/configuration/argocd) — using ESO with GitOps * [Configuration Reference: External Secret](/reference/chart-values/external-secret) — every value with type and default * [Cloud Provider Identity](/cloud-providers/aws) — setting up IRSA for the ESO `SecretStore`