> ## Documentation Index
> Fetch the complete documentation index at: https://enterprise-docs.crewai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common issues and solutions for CrewAI Platform deployments

## Overview

This guide provides solutions for common issues encountered when deploying and operating CrewAI Platform on Kubernetes. For additional support, generate a support bundle and contact your CrewAI representative.

## Diagnostic Commands

Before troubleshooting specific issues, gather diagnostic information:

```bash theme={null}
# Run comprehensive health check (verifies database, auth, Studio, tools, OAuth, LLM connections, and platform pods)
# Requires the auto-generated FACTORY_DEBUG_TOKEN — the endpoint returns 404 without it.
TOKEN=$(kubectl get secret <release-name>-secrets -n <namespace> \
  -o jsonpath='{.data.FACTORY_DEBUG_TOKEN}' | base64 -d)
curl -H "X-Factory-Debug-Token: $TOKEN" https://<your-factory-host>/health/debug

# Check all CrewAI Platform resources
kubectl get all -l app.kubernetes.io/name=crewai-platform

# View recent events
kubectl get events --sort-by='.lastTimestamp' | head -20

# Check pod status
kubectl get pods -o wide

# View logs for specific component
kubectl logs -l app.kubernetes.io/component=web --tail=100
kubectl logs -l app.kubernetes.io/component=worker --tail=100

# Check resource usage
kubectl top nodes
kubectl top pods
```

<Tip>
  For detailed health check results and component-specific troubleshooting, see the [Factory Health & Debug](/factory-health) guide.
</Tip>

## Common Issues

## Pod CrashLoopBackOff

**Symptoms:**

```bash theme={null}
kubectl get pods
# NAME                    READY   STATUS             RESTARTS   AGE
# crewai-web-xxx          0/1     CrashLoopBackOff   5          3m
```

**Common Causes:**

1. Missing required secrets
2. Database connection failure
3. Resource limits too restrictive
4. Invalid configuration

**Diagnosis:**

```bash theme={null}
kubectl logs POD_NAME
kubectl describe pod POD_NAME
```

## Pod Startup Failures

**Diagnosis:**

1. Check pod logs for errors
2. Verify resource limits are not too restrictive
3. Check secret availability
4. Verify image pull secrets are configured

```bash theme={null}
# Check pod status and logs
kubectl get pods
kubectl describe pod POD_NAME
kubectl logs POD_NAME
```

## Database Connection Issues

**Symptoms:** Logs show `could not connect to server: Connection refused`

**Diagnosis:**

If the application cannot connect to the database:

1. Verify `DB_HOST` is set correctly for external databases
2. Check database credentials in secrets
3. Ensure database allows connections from Kubernetes cluster
4. Verify database name and port configuration
5. Check security groups/firewall rules
6. If using Built-in Integrations (`oauth.enabled: true`) with an external database, ensure the OAuth database exists (default: `oauth_db`)
7. With an external database, ensure the Wharf database exists (default: `wharf`; Wharf is enabled by default)

```bash theme={null}
# Check database connectivity from a pod
kubectl run -it --rm debug --image=postgres:15 --restart=Never -- \
  psql -h DB_HOST -U DB_USER -d POSTGRES_DB

# Verify OAuth database exists (if oauth.enabled: true)
kubectl run -it --rm debug --image=postgres:15 --restart=Never -- \
  psql -h DB_HOST -U DB_USER -d oauth_db -c '\l'

# Verify Wharf database exists (enabled by default)
kubectl run -it --rm debug --image=postgres:15 --restart=Never -- \
  psql -h DB_HOST -U DB_USER -d wharf -c '\l'
```

## Storage/S3 Connection Issues

**Diagnosis:**

1. Verify AWS credentials are correct
2. Check bucket name and region
3. Ensure IAM permissions allow bucket access
4. For MinIO, verify endpoint URL is accessible

## Image Pull Errors

**Symptoms:** `ErrImagePull`, `ImagePullBackOff`

**Solutions:**

1. Verify image name and tag exist
2. Check pull secret configuration
3. Verify registry credentials
4. Check network connectivity to registry

## Ingress Not Accessible

**Symptoms:** Cannot reach application via ingress hostname

**Diagnosis:**

```bash theme={null}
# Check ingress status
kubectl get ingress
kubectl describe ingress crewai-ingress
```

**Solutions:**

1. Verify ingress controller is installed
2. Check ingress className matches your controller
3. Verify DNS points to ingress load balancer
4. Check TLS certificate configuration

## Out of Memory (OOMKilled)

**Symptoms:** Pods restarting with `OOMKilled`

**Solutions:**

1. Increase memory limits
2. Tune `WEB_CONCURRENCY` and `RAILS_MAX_THREADS`
3. Monitor actual memory usage
4. Check for memory leaks

```bash theme={null}
# Check memory usage patterns
kubectl top pods --sort-by=memory

# Increase memory limits in values.yaml
web:
  resources:
    limits:
      memory: "16Gi"  # Increase from 12Gi default
    requests:
      memory: "8Gi"

# Reduce concurrency to lower memory usage
envVars:
  WEB_CONCURRENCY: 2  # Reduce worker processes
  RAILS_MAX_THREADS: 3  # Reduce threads per process
```

## BuildKit Build Failures

**Symptoms:** Crew builds fail, BuildKit errors in logs

**Common Causes:**

1. Registry authentication issues
2. Insufficient BuildKit resources
3. Network connectivity problems

**Solutions:**

```bash theme={null}
# Check BuildKit pod status
kubectl get pods -l app.kubernetes.io/component=buildkit

# View BuildKit logs
kubectl logs -l app.kubernetes.io/component=buildkit --tail=100

# Verify registry authentication
kubectl get secret docker-registry -o yaml

# Increase BuildKit resources if needed
buildkit:
  resources:
    limits:
      cpu: "8"
      memory: "16Gi"

# Configure insecure registry if using internal registry
buildkit:
  registries:
    - hostname: "registry.internal.company.com"
      insecure: true
      http: true
```

## Third-Party OpenTelemetry Auto-Instrumentation Conflicts

**Symptoms:** Deployed crews crash on startup or get stuck in `Init`, with logs similar to:

```
ImportError: cannot import name '_create_exp_backoff_generator' from
'opentelemetry.exporter.otlp.proto.common._internal'
(/otel-auto-instrumentation-python/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py)

AwsEksResourceDetector failed: HTTP Error 403: Forbidden
AwsEc2ResourceDetector failed: timed out
SamplingRuleRecords is missing in getSamplingRules response: {'message': 'Missing Authentication Token'}
```

**Cause:** A cluster-wide OpenTelemetry Operator is auto-injecting language instrumentation into crew pods. On EKS this is most often the **Amazon CloudWatch Observability add-on** (Application Signals), but any OpenTelemetry Operator with auto-instrumentation enabled behaves the same way.

The operator's admission webhook mounts an instrumentation volume (`/otel-auto-instrumentation-python`) and prepends it to `PYTHONPATH`. CrewAI crew images already ship their own OpenTelemetry SDK (bundled with CrewAI and its dependencies). When two different OpenTelemetry versions are spliced together on `PYTHONPATH`, the namespace package resolves to incompatible modules and the crew fails to import, producing the `ImportError` above. The `AwsEksResourceDetector` and `getSamplingRules` lines are the injected AWS agent failing to authenticate. They are warnings, but they confirm injection is active.

The telltale sign is the path `/otel-auto-instrumentation-python` in the traceback. That directory is injected by the operator and is **not** part of the crew image (`/crew/.venv`).

**Solution:** Exclude the CrewAI namespaces in the **CloudWatch Observability add-on configuration**. This is the only reliable fix on current versions of the add-on.

<Warning>
  On the CloudWatch Observability add-on **v5.0.0 and later**, Application Signals "Auto Monitor" is enabled by default (`monitorAllServices: true`). Auto Monitor re-annotates every new pod at admission time, setting `cloudwatch.aws.amazon.com/auto-annotate-*: true` and forcing `instrumentation.opentelemetry.io/inject-*: true` back on. As a result, disabling injection with a **namespace annotation** or a **per-Deployment patch** does **not** work on these versions. Those annotations are silently overwritten. You must change the add-on configuration instead.
</Warning>

Apply an `exclude` block scoped to the CrewAI namespace, covering all languages (Auto Monitor injects Java, Python, Node.js, and .NET):

```bash theme={null}
aws eks update-addon \
  --cluster-name <cluster> \
  --addon-name amazon-cloudwatch-observability \
  --configuration-values '{"manager":{"applicationSignals":{"autoMonitor":{"monitorAllServices":true,"exclude":{"java":{"namespaces":["crewai-crews"]},"python":{"namespaces":["crewai-crews"]},"nodejs":{"namespaces":["crewai-crews"]},"dotnet":{"namespaces":["crewai-crews"]}}}}}}' \
  --resolve-conflicts PRESERVE
```

Alternatively, if you do not need CloudWatch APM auto-instrumentation anywhere, disable Auto Monitor entirely:

```bash theme={null}
aws eks update-addon \
  --cluster-name <cluster> \
  --addon-name amazon-cloudwatch-observability \
  --configuration-values '{"manager":{"applicationSignals":{"autoMonitor":{"monitorAllServices":false}}}}' \
  --resolve-conflicts PRESERVE
```

This is a cluster-scoped add-on setting, so it must be applied per cluster (once per environment) by whoever manages your EKS configuration.

**Validation:** The add-on update reports `IN_PROGRESS` while it rolls. Confirm the change took effect:

```bash theme={null}
# 1. Wait for the add-on to return to ACTIVE
aws eks describe-addon --cluster-name <cluster> \
  --addon-name amazon-cloudwatch-observability --query 'addon.status'

# 2. Confirm the exclude config persisted
aws eks describe-addon --cluster-name <cluster> \
  --addon-name amazon-cloudwatch-observability \
  --query 'addon.configurationValues' --output text

# 3. Recreate a crew pod, then confirm the injection annotations are gone
kubectl get pod <new-crew-pod> -n crewai-crews -o jsonpath='{.metadata.annotations}' \
  | tr ',' '\n' | grep -E 'cloudwatch.aws.amazon.com|instrumentation.opentelemetry.io'
```

Only pods created **after** the add-on returns to `ACTIVE` pick up the new configuration. A correctly excluded pod returns no `auto-annotate-*` or `inject-*` annotations in step 3, and its logs no longer show the `ImportError`.

### Sending Crew Telemetry to CloudWatch or Other Backends

Disabling auto-injection does **not** mean losing observability. The conflict only affects application-level *trace* injection. The other telemetry paths are unaffected, and crew traces can still reach external backends through the supported configuration:

* **Container logs** continue to flow to CloudWatch through your existing log agent (for example Fluent Bit). No instrumentation injection required.
* **Pod and container metrics** continue to flow through Container Insights and the CloudWatch agent. No injection required.
* **Crew execution traces** are emitted by each crew's own OpenTelemetry SDK. Rather than letting a third-party operator inject a conflicting agent, configure an external collector under **Settings → OpenTelemetry Collectors** in the platform. The platform forwards crew traces (and logs) to any OTLP-compatible endpoint, including CloudWatch's OTLP receiver, Datadog, Arize, or a generic OpenTelemetry collector. Provide the collector endpoint and any required headers; certificates are supported for TLS-secured endpoints.

This keeps the crew image's bundled OpenTelemetry stack intact while still delivering traces to your observability backend, without the `PYTHONPATH` collision that auto-injection causes.

## Persistent Volume Issues

**Symptoms:** Pods stuck in `Pending` state, PVC not binding

**Solutions:**

```bash theme={null}
# Check PVC status
kubectl get pvc

# Describe PVC for events
kubectl describe pvc crewai-postgres-data-crewai-postgres-0

# Check StorageClass availability
kubectl get storageclass

# Check if default StorageClass is set
kubectl get storageclass -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.metadata.name}{"\n"}{end}'

# Set default StorageClass if needed
kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```

## Authentication Provider Issues

**Symptoms:** Unable to log in, OAuth errors

**Solutions:**

### Entra ID Issues

```bash theme={null}
# Verify secrets are set
kubectl get secret crewai-secrets -o jsonpath='{.data.ENTRA_ID_CLIENT_ID}' | base64 -d
kubectl get secret crewai-secrets -o jsonpath='{.data.ENTRA_ID_TENANT_ID}' | base64 -d

# Check redirect URI configuration
# Must match: https://YOUR_HOST/auth/entra_id/callback

# Verify APPLICATION_HOST matches your domain
kubectl exec deploy/crewai-web -- printenv APPLICATION_HOST
```

### Okta Issues

```bash theme={null}
# Verify Okta configuration
kubectl get secret crewai-secrets -o jsonpath='{.data.OKTA_SITE}' | base64 -d
kubectl get secret crewai-secrets -o jsonpath='{.data.OKTA_CLIENT_ID}' | base64 -d

# Check redirect URI in Okta
# Must match: https://YOUR_HOST/auth/okta/callback
```

## Secret Management Issues

**Symptoms:** Pods fail to start, missing secret errors

**Solutions:**

```bash theme={null}
# Check if secrets exist
kubectl get secret crewai-secrets

# Verify secret contents (be careful with output)
kubectl describe secret crewai-secrets

# For external secret store issues
kubectl get secretstore
kubectl get externalsecret
kubectl describe externalsecret crewai-external-secret

# Check External Secrets Operator logs
kubectl logs -n external-secrets-operator deployment/external-secrets
```

### Expected Behavior: Pod Restarts After Secret Updates

**Observation:** Pods restart after running `helm upgrade` when secret values change

**This is expected behavior.** When you update secret values in your Helm values file (e.g., rotating credentials), the chart automatically triggers a rolling restart of all affected pods to ensure they pick up the new credentials. This is by design and prevents stale credentials from being used.

**What to expect:**

* Pods restart in a rolling fashion (no downtime)
* Each pod restarts once to load new secret values
* The restart happens automatically - no manual pod deletion needed

**To verify the restart was successful:**

```bash theme={null}
# Check that all pods are running with new secret values
kubectl get pods -l app.kubernetes.io/name=crewai-platform

# View pod age to confirm restart
kubectl get pods -o wide

# Check logs to verify application started correctly
kubectl logs -l app.kubernetes.io/component=web --tail=50
```

## Configuration Warnings

### RAILS\_MASTER\_KEY Warning

**Warning Message:**

```
⚠️  WARNING: RAILS_MASTER_KEY detected in your configuration.
    This key is automatically managed by the chart and should not be set manually.

    Please remove RAILS_MASTER_KEY from your values.
```

**Cause:** You have manually configured `RAILS_MASTER_KEY` in either `envVars` or `secrets` in your values file.

**Solution:**

The chart automatically manages `RAILS_MASTER_KEY` and does not require manual configuration. Remove this setting from your values file:

```yaml theme={null}
# Remove these lines from your values.yaml:
envVars:
  RAILS_MASTER_KEY: "..."  # Remove this

secrets:
  RAILS_MASTER_KEY: "..."  # Remove this
```

Then upgrade your deployment:

```bash theme={null}
helm upgrade crewai-platform \
  oci://registry.crewai.com/crewai/stable/crewai-platform \
  --values my-values.yaml
```

**Why This Matters:** The chart uses a different Rails configuration approach that doesn't require `RAILS_MASTER_KEY`. Setting it manually can cause configuration conflicts.

## Performance Issues

**Symptoms:** Slow response times, high latency

**Diagnostic Steps:**

```bash theme={null}
# Check resource utilization
kubectl top nodes
kubectl top pods

# Review slow query logs (if configured)
kubectl logs -l app.kubernetes.io/component=web | grep "Slow query"

```

**Solutions:**

1. **Scale web replicas:**

```yaml theme={null}
web:
  replicaCount: 4  # Increase from 2
```

2. **Increase resources:**

```yaml theme={null}
web:
  resources:
    limits:
      cpu: "8"
      memory: "16Gi"
    requests:
      cpu: "2000m"
      memory: "8Gi"
```

3. **Tune concurrency:**

```yaml theme={null}
envVars:
  WEB_CONCURRENCY: 4
  RAILS_MAX_THREADS: 10
```

4. **Add database read replicas** (configure in external database)

***

## Support and Resources

### Documentation

* **Installation Guide**: [Installation](installation)
* **Configuration Guide**: [Configuration](configuration)
* **Configuration Reference**: [Reference](reference)

### Generate Support Bundle

The support bundle collects comprehensive diagnostics for troubleshooting:

```bash theme={null}
# Install the support-bundle plugin (if not already installed)
kubectl krew install support-bundle

# Generate support bundle (automatically detects cluster specs)
kubectl support-bundle --load-cluster-specs

# Support bundle automatically includes:
# - All pod logs (web, worker, postgres, minio)
# - Optional component logs (OAuth, Wharf, Cube, BuildKit, Internal Registry)
# - Crew workload logs from configured namespaces
# - Resource configurations (deployments, statefulsets, services, ingresses)
# - Cluster state and node metrics
# - Event history
# - Health check results
# - Feature flag state (active flags and their configuration)
# - Secret names (not values)
```

**Automatic Component Detection:**

The support bundle automatically detects and collects logs from enabled components:

* **OAuth Service**: When `oauth.enabled: true`
* **Wharf Service**: When `wharf.enabled: true`
* **Cube Analytics**: When `cube.enabled: true`
* **BuildKit**: When `buildkit.enabled: true`
* **Internal Registry**: When `internalRegistry.enabled: true`
* **PostgreSQL**: When `postgres.enabled: true`
* **MinIO**: When `minio.enabled: true`

**Multi-Namespace Crew Logs:**

When using multi-organization namespace isolation, configure `supportBundle.crewLogNamespaces` to collect crew logs from additional namespaces:

```yaml theme={null}
supportBundle:
  crewLogNamespaces:
    - "crewai-org1"
    - "crewai-org2"
```

See [Global Configuration](/reference/chart-values/global#supportbundle-crewlognamespaces) for details.

The support bundle will be saved as a `.tar.gz` file that can be shared with CrewAI support for analysis.

<Note>
  Share the generated support bundle file with CrewAI support for faster issue resolution.
</Note>

### Quick Diagnostic Commands

```bash theme={null}
# Check all CrewAI resources
kubectl get all -l app.kubernetes.io/name=crewai-platform

# View recent events
kubectl get events --sort-by='.lastTimestamp'

# Component logs
kubectl logs -l app.kubernetes.io/component=web --tail=100 -f
kubectl logs -l app.kubernetes.io/component=worker --tail=100 -f
kubectl logs -l app.kubernetes.io/component=buildkit --tail=100

# Check resource usage
kubectl top nodes
kubectl top pods

# Describe problematic pod
kubectl describe pod <POD_NAME>
```

### Contact Support

For assistance with CrewAI Platform:

* **Customer Portal**: `https://enterprise.crewai.com/crewai`
* **Support Team**: Contact your CrewAI representative
* **Emergency Issues**: Generate and share support bundle with your support team
* **Release History**: `https://enterprise.crewai.com/crewai/release-history`