> ## Documentation Index
> Fetch the complete documentation index at: https://enterprise-docs.crewai.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Metrics Configuration
> Configuration for Prometheus metrics collection from web and worker pods.
The chart supports Prometheus metrics collection with two deployment topologies: **pull-based** (Prometheus scrapes pods) and **push-based** (OpenTelemetry Collector remote-writes to external endpoints).
## Prerequisites
Metrics collection is **disabled by default**. To enable it, set:
```yaml theme={null}
envVars:
ENABLE_PROMETHEUS: "true"
```
When enabled, the chart injects a `prometheus_exporter` sidecar into web and worker pods. Rails pushes measurements to `localhost:9394`, and the sidecar exposes them on the `/metrics` endpoint.
## Deployment Topologies
### Pull Mode (Default)
Recommended when Prometheus runs inside the same Kubernetes cluster.
**How it works:**
* Prometheus scrapes each pod's `:9394/metrics` endpoint
* Pods are discovered via `prometheus.io/scrape` annotations or ServiceMonitor
* No credentials required (cluster-internal traffic)
**When to use:**
* Prometheus Operator installed in cluster
* kube-prometheus-stack or similar monitoring stack
* Network policies allow inbound scraping
### Push Mode
Recommended when your observability platform lives outside the cluster or network policies prevent inbound scraping.
**How it works:**
* OpenTelemetry Collector sidecar runs next to each pod
* Collector scrapes `localhost:9394/metrics` (same pod, no network hops)
* Collector remote-writes to external endpoint (`metrics.push.url`)
* Supports Basic Auth and Bearer Token authentication
**When to use:**
* Grafana Cloud, Mimir, Datadog, or other external observability platforms
* Network policies block inbound traffic to pods
* Centralized metrics aggregation required
## Configuration Reference
### `metrics.*`
Top-level metrics configuration. Only takes effect when `envVars.ENABLE_PROMETHEUS: "true"`.
Metrics collection topology.
**Valid Values:**
* `"pull"` - Prometheus scrapes pods directly (default)
* `"push"` - OpenTelemetry Collector remote-writes to external endpoint
**Important:** The chart validates this value and fails rendering if set to anything other than `"pull"` or `"push"`.
Port where the prometheus\_exporter sidecar listens.
**Default:** `9394`
**Impact:**
* Used by Prometheus scrape annotations (`prometheus.io/port`)
* Used by ServiceMonitor endpoint configuration
* Used by Service port mapping
Resource requests and limits for the prometheus\_exporter sidecar.
**Applied to:** Both web and worker pods
**Default:**
```yaml theme={null}
metrics:
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 100m
memory: 128Mi
```
**When to adjust:**
* High-cardinality metrics (many unique label combinations)
* High scrape frequency (more frequent than 30s)
* Large number of metrics exported
### `metrics.pull.*`
Pull mode configuration. Ignored when `metrics.mode: "push"`.
Add Prometheus pod discovery annotations to web and worker pods.
**Default:** `true`
**Annotations added:**
```yaml theme={null}
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9394"
```
**When to disable:**
* Using ServiceMonitor exclusively (Prometheus Operator)
* Custom pod discovery configuration
* Annotations conflict with existing scrape config
#### `metrics.pull.serviceMonitor.*`
ServiceMonitor configuration for Prometheus Operator.
Create ServiceMonitor resource for Prometheus Operator.
**Prerequisites:**
* Prometheus Operator installed in cluster
* `monitoring.coreos.com/v1` API available
**Behavior:**
* Chart checks for `monitoring.coreos.com/v1` API before rendering
* ServiceMonitor is **not created** if API is unavailable (no error)
* ServiceMonitor targets both `web` and `worker` components automatically
Scrape interval for ServiceMonitor.
**Default:** `"30s"`
**Valid Values:** Any valid Prometheus duration (e.g., `"15s"`, `"1m"`, `"2m30s"`)
**Impact:** Higher frequency increases metric granularity but also increases resource usage.
Scrape timeout for ServiceMonitor.
**Default:** `"10s"`
**Important:** Must be less than `interval` (Prometheus requirement)
**When to increase:**
* Large number of metrics exported
* Network latency between Prometheus and pods
Additional labels to add to the ServiceMonitor resource.
**Purpose:** Prometheus Operator uses label selectors to determine which ServiceMonitors to watch. Your Prometheus custom resource typically requires specific labels on ServiceMonitor resources.
**Common Requirements:**
* **kube-prometheus-stack:** `release: kube-prometheus-stack`
* **Custom Prometheus Operator:** Check your Prometheus CR's `serviceMonitorSelector`
**Example:**
```yaml theme={null}
metrics:
pull:
serviceMonitor:
enabled: true
labels:
release: kube-prometheus-stack
team: platform
```
**How to find required labels:**
```bash theme={null}
kubectl get prometheus -o yaml | grep -A 5 serviceMonitorSelector
```
### `metrics.push.*`
Push mode configuration. Only used when `metrics.mode: "push"`.
Prometheus remote-write endpoint URL.
**Required when:** `metrics.mode: "push"`
**Format:** Full HTTP(S) URL including path
**Examples:**
* Grafana Cloud: `"https://prometheus-prod-01.grafana.net/api/prom/push"`
* Mimir: `"https://mimir.example.com/api/v1/write"`
* Datadog: `"https://api.datadoghq.com/api/v1/series"`
**Important:** The chart fails rendering if `metrics.mode: "push"` and this value is empty.
#### Providing the authentication Secret
The chart never creates or manages the Secret referenced by `basicAuth.passwordSecretRef`
or `bearerTokenSecretRef`. Those fields point at **any pre-existing Secret** in the release
namespace — how it gets there is up to you. Pick whichever matches how the rest of your
platform manages secrets.
**Option A — External Secrets Operator (recommended for production).**
If you already run ESO to sync from AWS Secrets Manager, GCP Secret Manager, Azure Key Vault,
Vault, etc., create an `ExternalSecret` in the release namespace:
```yaml theme={null}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: grafana-cloud-credentials
namespace: crewai
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager # your existing ClusterSecretStore / SecretStore
kind: ClusterSecretStore
target:
name: grafana-cloud-credentials # the Secret name the chart references
data:
- secretKey: password # the Secret key the chart references
remoteRef:
key: crewai/metrics/grafana-cloud-password
```
Then point the chart at it exactly as you would a manually-created Secret:
```yaml theme={null}
metrics:
push:
basicAuth:
passwordSecretRef:
name: grafana-cloud-credentials
key: password
```
Rotating the value in your upstream vault propagates to the Secret automatically. **Note:**
the shipper sidecar reads the Secret at container start via `env: valueFrom: secretKeyRef`
— it does not hot-reload. After a rotation, run
`kubectl -n crewai rollout restart deployment/crewai-web deployment/crewai-worker` to pick up
the new value.
**Option B — Sealed Secrets, SOPS, or any GitOps-managed Secret.**
Same shape. Commit an encrypted Secret manifest to your GitOps repo; the tooling materializes
a plain Kubernetes Secret in the namespace; the chart references it by name and key.
**Option C — Manual `kubectl` (quick / dev only).**
```bash theme={null}
kubectl create secret generic grafana-cloud-credentials \
--namespace crewai \
--from-literal=password='your-grafana-cloud-api-token'
```
Fast for a smoke test but avoid for production — the token lives in shell history and is not
rotated automatically.
Whichever path you pick, the Secret must exist in the same namespace as the CrewAI Platform
release. The chart looks up `/` at pod-start time via `envFrom` / `env`
references; no cross-namespace projection is done.
#### `metrics.push.basicAuth.*`
HTTP Basic Authentication for remote-write endpoint.
Basic auth username.
**Optional:** Only used with basic auth authentication
##### `metrics.push.basicAuth.passwordSecretRef.*`
Secret reference for basic auth password.
Name of Kubernetes Secret containing the password.
**Example:** `"grafana-cloud-credentials"`
Key name within the Secret containing the password value.
**Example:** `"password"`
**Complete Basic Auth Example:**
```yaml theme={null}
# Provide the Secret in the release namespace via any of the options above
# (ESO ExternalSecret, Sealed Secret, or the kubectl one-liner):
kubectl create secret generic grafana-cloud-credentials \
--from-literal=password='your-grafana-cloud-token'
# Configure chart
metrics:
mode: push
push:
url: "https://prometheus-prod-01.grafana.net/api/prom/push"
basicAuth:
username: "123456"
passwordSecretRef:
name: "grafana-cloud-credentials"
key: "password"
```
#### `metrics.push.bearerTokenSecretRef.*`
Bearer token authentication (alternative to basic auth).
Name of Kubernetes Secret containing the bearer token.
**Example:** `"datadog-api-key"`
Key name within the Secret containing the token value.
**Example:** `"token"`
**Bearer Token Example:**
```yaml theme={null}
# Provide the Secret in the release namespace via any of the options above
# (ESO ExternalSecret, Sealed Secret, or the kubectl one-liner):
kubectl create secret generic datadog-api-key \
--from-literal=token='your-datadog-api-key'
# Configure chart
metrics:
mode: push
push:
url: "https://api.datadoghq.com/api/v1/series"
bearerTokenSecretRef:
name: "datadog-api-key"
key: "token"
```
#### `metrics.push.shipper.*`
OpenTelemetry Collector sidecar configuration.
OpenTelemetry Collector container image.
**Default:** `"otel/opentelemetry-collector-contrib:0.108.0"`
**Factory / private-registry installs must allow this image.**
The default pulls from Docker Hub. Factory customers running behind a Replicated
proxy, an air-gapped mirror, or any private registry must either:
1. Mirror `otel/opentelemetry-collector-contrib:` into your registry and
override this field to point at the mirrored image, **or**
2. Add `docker.io/otel/opentelemetry-collector-contrib` to your Replicated image
allowlist / registry proxy so the default pull resolves.
See [Image Registry Configuration](/configuration/configuration#8-image-registry-configuration)
for how the chart resolves container images.
**When to override:**
* Air-gapped deployments with private registry
* Version pinning for stability
* Security scanning requirements
**Example:**
```yaml theme={null}
metrics:
push:
shipper:
image: "registry.company.com/otel/collector:0.108.0"
```
Resource requests and limits for the metrics-shipper sidecar.
**Default:**
```yaml theme={null}
metrics:
push:
shipper:
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
cpu: 100m
memory: 256Mi
```
**When to adjust:**
* High metric volume (increase memory)
* Network issues causing buffering (increase memory)
* Remote-write endpoint slow or rate-limited (increase memory for buffering)
## Architecture Details
### Sidecar Containers
When metrics are enabled, the chart injects sidecar containers into web and worker Deployments:
**Always injected (both modes):**
* `prometheus_exporter` - Exposes `:9394/metrics` endpoint (same image as main container, `CONTAINER_ROLE=prometheus`)
**Push mode only:**
* `metrics-shipper` - OpenTelemetry Collector that scrapes localhost and remote-writes
### Service Configuration
**Pull Mode:**
* **Web Service:** Adds metrics port (`9394`) to existing service
* **Worker Service:** Creates headless service with metrics port (worker pods don't have an existing service)
**Push Mode:**
* No Service changes (metrics ship directly to external endpoint)
### Pod Annotations
When `metrics.mode: "pull"` and `metrics.pull.annotations: true`, the chart adds:
```yaml theme={null}
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9394"
```
These annotations work with standard Prometheus Kubernetes service discovery configurations.
## Configuration Examples
### Pull Mode with ServiceMonitor
Recommended for clusters with Prometheus Operator:
```yaml theme={null}
envVars:
ENABLE_PROMETHEUS: "true"
metrics:
mode: pull
pull:
annotations: true
serviceMonitor:
enabled: true
interval: 30s
scrapeTimeout: 10s
labels:
release: kube-prometheus-stack
```
### Pull Mode with Annotations Only
For Prometheus instances using pod-based service discovery:
```yaml theme={null}
envVars:
ENABLE_PROMETHEUS: "true"
metrics:
mode: pull
pull:
annotations: true
serviceMonitor:
enabled: false
```
### Push Mode to Grafana Cloud
```yaml theme={null}
envVars:
ENABLE_PROMETHEUS: "true"
metrics:
mode: push
push:
url: "https://prometheus-prod-01.grafana.net/api/prom/push"
basicAuth:
username: "123456"
passwordSecretRef:
name: "grafana-cloud-credentials"
key: "password"
```
**Provide the Secret** — see [Providing the authentication Secret](#providing-the-authentication-secret)
for ESO / Sealed Secret / GitOps alternatives. Quick manual option:
```bash theme={null}
kubectl create secret generic grafana-cloud-credentials \
--namespace crewai \
--from-literal=password='your-grafana-cloud-api-token'
```
### Push Mode to Mimir
```yaml theme={null}
envVars:
ENABLE_PROMETHEUS: "true"
metrics:
mode: push
push:
url: "https://mimir.company.com/api/v1/write"
bearerTokenSecretRef:
name: "mimir-token"
key: "token"
```
### Disabled (Default)
```yaml theme={null}
envVars:
ENABLE_PROMETHEUS: "false"
```
No sidecars are injected, no Services are modified.
## Troubleshooting
### ServiceMonitor not picked up by Prometheus
**Symptoms:** ServiceMonitor created but no scrape targets in Prometheus
**Causes:**
1. Missing required labels on ServiceMonitor
2. Prometheus Operator not watching the namespace
3. Prometheus CR's `serviceMonitorSelector` doesn't match
**Solution:**
Check your Prometheus resource's selector:
```bash theme={null}
kubectl get prometheus -o yaml | grep -A 5 serviceMonitorSelector
```
Add matching labels to `metrics.pull.serviceMonitor.labels`.
### Push mode authentication failures
**Symptoms:** Metrics not appearing in remote system, collector logs show 401/403 errors
**Causes:**
1. Secret not created or in wrong namespace
2. Wrong secret key name
3. Expired credentials
**Solution:**
Verify secret exists and contains correct key:
```bash theme={null}
kubectl get secret grafana-cloud-credentials -o yaml
kubectl get secret grafana-cloud-credentials -o jsonpath='{.data.password}' | base64 -d
```
Check collector logs:
```bash theme={null}
kubectl logs -c metrics-shipper
```
### High memory usage in sidecars
**Symptoms:** Prometheus exporter or metrics-shipper OOMKilled
**Causes:**
1. High cardinality metrics (too many unique label combinations)
2. Slow remote-write endpoint causing buffering (push mode)
3. High scrape frequency
**Solution:**
Increase resource limits:
```yaml theme={null}
metrics:
resources:
limits:
memory: 256Mi # Increased from 128Mi
push:
shipper:
resources:
limits:
memory: 512Mi # Increased from 256Mi
```
## Security Considerations
* **Secrets:** Never commit authentication credentials to version control. Always use Kubernetes Secrets.
* **Network Policies:** Ensure network policies allow:
* Pull mode: Inbound traffic from Prometheus to pods on port 9394
* Push mode: Outbound traffic from pods to remote-write endpoint
* **RBAC:** No additional cluster permissions required for metrics collection
* **TLS:** Remote-write endpoints should use HTTPS. The collector validates certificates by default.