Metrics Configuration - CrewAI Platform Helm Chart

The chart supports Prometheus metrics collection with two deployment topologies: pull-based (Prometheus scrapes pods) and push-based (OpenTelemetry Collector remote-writes to external endpoints).

Prerequisites

Metrics collection is disabled by default. To enable it, set:

envVars:
  ENABLE_PROMETHEUS: "true"

When enabled, the chart injects a prometheus_exporter sidecar into web and worker pods. Rails pushes measurements to localhost:9394, and the sidecar exposes them on the /metrics endpoint.

Deployment Topologies

Pull Mode (Default)

Recommended when Prometheus runs inside the same Kubernetes cluster. How it works:

Prometheus scrapes each pod’s :9394/metrics endpoint
Pods are discovered via prometheus.io/scrape annotations or ServiceMonitor
No credentials required (cluster-internal traffic)

When to use:

Prometheus Operator installed in cluster
kube-prometheus-stack or similar monitoring stack
Network policies allow inbound scraping

Push Mode

Recommended when your observability platform lives outside the cluster or network policies prevent inbound scraping. How it works:

OpenTelemetry Collector sidecar runs next to each pod
Collector scrapes localhost:9394/metrics (same pod, no network hops)
Collector remote-writes to external endpoint (metrics.push.url)
Supports Basic Auth and Bearer Token authentication

When to use:

Grafana Cloud, Mimir, Datadog, or other external observability platforms
Network policies block inbound traffic to pods
Centralized metrics aggregation required

Configuration Reference

`metrics.*`

Top-level metrics configuration. Only takes effect when envVars.ENABLE_PROMETHEUS: "true".

string

default:"pull"

Metrics collection topology.Valid Values:

"pull" - Prometheus scrapes pods directly (default)
"push" - OpenTelemetry Collector remote-writes to external endpoint

Important: The chart validates this value and fails rendering if set to anything other than "pull" or "push".

number

default:"9394"

Port where the prometheus_exporter sidecar listens.Default: 9394Impact:

Used by Prometheus scrape annotations (prometheus.io/port)
Used by ServiceMonitor endpoint configuration
Used by Service port mapping

object

Resource requests and limits for the prometheus_exporter sidecar.Applied to: Both web and worker podsDefault:

metrics:
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
    limits:
      cpu: 100m
      memory: 128Mi

When to adjust:

High-cardinality metrics (many unique label combinations)
High scrape frequency (more frequent than 30s)
Large number of metrics exported

`metrics.pull.*`

Pull mode configuration. Ignored when metrics.mode: "push".

boolean

default:"true"

Add Prometheus pod discovery annotations to web and worker pods.Default: trueAnnotations added:

prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9394"

When to disable:

Using ServiceMonitor exclusively (Prometheus Operator)
Custom pod discovery configuration
Annotations conflict with existing scrape config

`metrics.pull.serviceMonitor.*`

ServiceMonitor configuration for Prometheus Operator.

boolean

default:"false"

Create ServiceMonitor resource for Prometheus Operator.Prerequisites:

Prometheus Operator installed in cluster
monitoring.coreos.com/v1 API available

Behavior:

Chart checks for monitoring.coreos.com/v1 API before rendering
ServiceMonitor is not created if API is unavailable (no error)
ServiceMonitor targets both web and worker components automatically

string

default:"30s"

Scrape interval for ServiceMonitor.Default: "30s"Valid Values: Any valid Prometheus duration (e.g., "15s", "1m", "2m30s")Impact: Higher frequency increases metric granularity but also increases resource usage.

string

default:"10s"

Scrape timeout for ServiceMonitor.Default: "10s"Important: Must be less than interval (Prometheus requirement)When to increase:

Large number of metrics exported
Network latency between Prometheus and pods

object

default:"{}"

Additional labels to add to the ServiceMonitor resource.Purpose: Prometheus Operator uses label selectors to determine which ServiceMonitors to watch. Your Prometheus custom resource typically requires specific labels on ServiceMonitor resources.Common Requirements:

kube-prometheus-stack: release: kube-prometheus-stack
Custom Prometheus Operator: Check your Prometheus CR’s serviceMonitorSelector

Example:

metrics:
  pull:
    serviceMonitor:
      enabled: true
      labels:
        release: kube-prometheus-stack
        team: platform

How to find required labels:

kubectl get prometheus -o yaml | grep -A 5 serviceMonitorSelector

`metrics.push.*`

Push mode configuration. Only used when metrics.mode: "push".

string

required

Prometheus remote-write endpoint URL.Required when: metrics.mode: "push"Format: Full HTTP(S) URL including pathExamples:

Grafana Cloud: "https://prometheus-prod-01.grafana.net/api/prom/push"
Mimir: "https://mimir.example.com/api/v1/write"
Datadog: "https://api.datadoghq.com/api/v1/series"

Important: The chart fails rendering if metrics.mode: "push" and this value is empty.

Providing the authentication Secret

The chart never creates or manages the Secret referenced by basicAuth.passwordSecretRef or bearerTokenSecretRef. Those fields point at any pre-existing Secret in the release namespace — how it gets there is up to you. Pick whichever matches how the rest of your platform manages secrets. Option A — External Secrets Operator (recommended for production). If you already run ESO to sync from AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, Vault, etc., create an ExternalSecret in the release namespace:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: grafana-cloud-credentials
  namespace: crewai
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager    # your existing ClusterSecretStore / SecretStore
    kind: ClusterSecretStore
  target:
    name: grafana-cloud-credentials   # the Secret name the chart references
  data:
    - secretKey: password              # the Secret key the chart references
      remoteRef:
        key: crewai/metrics/grafana-cloud-password

Then point the chart at it exactly as you would a manually-created Secret:

metrics:
  push:
    basicAuth:
      passwordSecretRef:
        name: grafana-cloud-credentials
        key: password

Rotating the value in your upstream vault propagates to the Secret automatically. Note: the shipper sidecar reads the Secret at container start via env: valueFrom: secretKeyRef — it does not hot-reload. After a rotation, run kubectl -n crewai rollout restart deployment/crewai-web deployment/crewai-worker to pick up the new value. Option B — Sealed Secrets, SOPS, or any GitOps-managed Secret. Same shape. Commit an encrypted Secret manifest to your GitOps repo; the tooling materializes a plain Kubernetes Secret in the namespace; the chart references it by name and key. Option C — Manual kubectl (quick / dev only).

kubectl create secret generic grafana-cloud-credentials \
  --namespace crewai \
  --from-literal=password='your-grafana-cloud-api-token'

Fast for a smoke test but avoid for production — the token lives in shell history and is not rotated automatically.

Whichever path you pick, the Secret must exist in the same namespace as the CrewAI Platform release. The chart looks up <name>/<key> at pod-start time via envFrom / env references; no cross-namespace projection is done.

`metrics.push.basicAuth.*`

HTTP Basic Authentication for remote-write endpoint.

string

default:""

Basic auth username.Optional: Only used with basic auth authentication

`metrics.push.basicAuth.passwordSecretRef.*`

Secret reference for basic auth password.

string

default:""

Name of Kubernetes Secret containing the password.Example: "grafana-cloud-credentials"

string

default:""

Key name within the Secret containing the password value.Example: "password"Complete Basic Auth Example:

# Provide the Secret in the release namespace via any of the options above
# (ESO ExternalSecret, Sealed Secret, or the kubectl one-liner):
kubectl create secret generic grafana-cloud-credentials \
  --from-literal=password='your-grafana-cloud-token'

# Configure chart
metrics:
  mode: push
  push:
    url: "https://prometheus-prod-01.grafana.net/api/prom/push"
    basicAuth:
      username: "123456"
      passwordSecretRef:
        name: "grafana-cloud-credentials"
        key: "password"

`metrics.push.bearerTokenSecretRef.*`

Bearer token authentication (alternative to basic auth).

string

default:""

Name of Kubernetes Secret containing the bearer token.Example: "datadog-api-key"

string

default:""

Key name within the Secret containing the token value.Example: "token"Bearer Token Example:

# Provide the Secret in the release namespace via any of the options above
# (ESO ExternalSecret, Sealed Secret, or the kubectl one-liner):
kubectl create secret generic datadog-api-key \
  --from-literal=token='your-datadog-api-key'

# Configure chart
metrics:
  mode: push
  push:
    url: "https://api.datadoghq.com/api/v1/series"
    bearerTokenSecretRef:
      name: "datadog-api-key"
      key: "token"

`metrics.push.shipper.*`

OpenTelemetry Collector sidecar configuration.

string

default:"otel/opentelemetry-collector-contrib:0.108.0"

OpenTelemetry Collector container image.Default: "otel/opentelemetry-collector-contrib:0.108.0"

Factory / private-registry installs must allow this image.The default pulls from Docker Hub. Factory customers running behind a Replicated proxy, an air-gapped mirror, or any private registry must either:

Mirror otel/opentelemetry-collector-contrib:<tag> into your registry and override this field to point at the mirrored image, or
Add docker.io/otel/opentelemetry-collector-contrib to your Replicated image allowlist / registry proxy so the default pull resolves.

See Image Registry Configuration for how the chart resolves container images.

When to override:

Air-gapped deployments with private registry
Version pinning for stability
Security scanning requirements

Example:

metrics:
  push:
    shipper:
      image: "registry.company.com/otel/collector:0.108.0"

object

Resource requests and limits for the metrics-shipper sidecar.Default:

metrics:
  push:
    shipper:
      resources:
        requests:
          cpu: 10m
          memory: 64Mi
        limits:
          cpu: 100m
          memory: 256Mi

When to adjust:

High metric volume (increase memory)
Network issues causing buffering (increase memory)
Remote-write endpoint slow or rate-limited (increase memory for buffering)

Architecture Details

Sidecar Containers

When metrics are enabled, the chart injects sidecar containers into web and worker Deployments: Always injected (both modes):

prometheus_exporter - Exposes :9394/metrics endpoint (same image as main container, CONTAINER_ROLE=prometheus)

Push mode only:

metrics-shipper - OpenTelemetry Collector that scrapes localhost and remote-writes

Service Configuration

Pull Mode:

Web Service: Adds metrics port (9394) to existing service
Worker Service: Creates headless service with metrics port (worker pods don’t have an existing service)

Push Mode:

No Service changes (metrics ship directly to external endpoint)

Pod Annotations

When metrics.mode: "pull" and metrics.pull.annotations: true, the chart adds:

prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9394"

These annotations work with standard Prometheus Kubernetes service discovery configurations.

Configuration Examples

Pull Mode with ServiceMonitor

Recommended for clusters with Prometheus Operator:

envVars:
  ENABLE_PROMETHEUS: "true"

metrics:
  mode: pull
  pull:
    annotations: true
    serviceMonitor:
      enabled: true
      interval: 30s
      scrapeTimeout: 10s
      labels:
        release: kube-prometheus-stack

Pull Mode with Annotations Only

For Prometheus instances using pod-based service discovery:

envVars:
  ENABLE_PROMETHEUS: "true"

metrics:
  mode: pull
  pull:
    annotations: true
    serviceMonitor:
      enabled: false

Push Mode to Grafana Cloud

envVars:
  ENABLE_PROMETHEUS: "true"

metrics:
  mode: push
  push:
    url: "https://prometheus-prod-01.grafana.net/api/prom/push"
    basicAuth:
      username: "123456"
      passwordSecretRef:
        name: "grafana-cloud-credentials"
        key: "password"

Provide the Secret — see Providing the authentication Secret for ESO / Sealed Secret / GitOps alternatives. Quick manual option:

kubectl create secret generic grafana-cloud-credentials \
  --namespace crewai \
  --from-literal=password='your-grafana-cloud-api-token'

Push Mode to Mimir

envVars:
  ENABLE_PROMETHEUS: "true"

metrics:
  mode: push
  push:
    url: "https://mimir.company.com/api/v1/write"
    bearerTokenSecretRef:
      name: "mimir-token"
      key: "token"

Disabled (Default)

envVars:
  ENABLE_PROMETHEUS: "false"

No sidecars are injected, no Services are modified.

Troubleshooting

ServiceMonitor not picked up by Prometheus

Symptoms: ServiceMonitor created but no scrape targets in Prometheus Causes:

Missing required labels on ServiceMonitor
Prometheus Operator not watching the namespace
Prometheus CR’s serviceMonitorSelector doesn’t match

Solution: Check your Prometheus resource’s selector:

kubectl get prometheus -o yaml | grep -A 5 serviceMonitorSelector

Add matching labels to metrics.pull.serviceMonitor.labels.

Push mode authentication failures

Symptoms: Metrics not appearing in remote system, collector logs show 401/403 errors Causes:

Secret not created or in wrong namespace
Wrong secret key name
Expired credentials

Solution: Verify secret exists and contains correct key:

kubectl get secret grafana-cloud-credentials -o yaml
kubectl get secret grafana-cloud-credentials -o jsonpath='{.data.password}' | base64 -d

Check collector logs:

kubectl logs <pod-name> -c metrics-shipper

High memory usage in sidecars

Symptoms: Prometheus exporter or metrics-shipper OOMKilled Causes:

High cardinality metrics (too many unique label combinations)
Slow remote-write endpoint causing buffering (push mode)
High scrape frequency

Solution: Increase resource limits:

metrics:
  resources:
    limits:
      memory: 256Mi  # Increased from 128Mi
  push:
    shipper:
      resources:
        limits:
          memory: 512Mi  # Increased from 256Mi

Security Considerations

Secrets: Never commit authentication credentials to version control. Always use Kubernetes Secrets.
Network Policies: Ensure network policies allow:
- Pull mode: Inbound traffic from Prometheus to pods on port 9394
- Push mode: Outbound traffic from pods to remote-write endpoint
RBAC: No additional cluster permissions required for metrics collection
TLS: Remote-write endpoints should use HTTPS. The collector validates certificates by default.

​Prerequisites

​Deployment Topologies

​Pull Mode (Default)

​Push Mode

​Configuration Reference

​metrics.*

​metrics.pull.*

​metrics.pull.serviceMonitor.*

​metrics.push.*

​Providing the authentication Secret

​metrics.push.basicAuth.*

metrics.push.basicAuth.passwordSecretRef.*

​metrics.push.bearerTokenSecretRef.*

​metrics.push.shipper.*

​Architecture Details

​Sidecar Containers

​Service Configuration

​Pod Annotations

​Configuration Examples

​Pull Mode with ServiceMonitor

​Pull Mode with Annotations Only

​Push Mode to Grafana Cloud

​Push Mode to Mimir

​Disabled (Default)

​Troubleshooting

​ServiceMonitor not picked up by Prometheus

​Push mode authentication failures

​High memory usage in sidecars

​Security Considerations

Prerequisites

Deployment Topologies

Pull Mode (Default)

Push Mode

Configuration Reference

`metrics.*`

`metrics.pull.*`

`metrics.pull.serviceMonitor.*`

`metrics.push.*`

Providing the authentication Secret

`metrics.push.basicAuth.*`

`metrics.push.basicAuth.passwordSecretRef.*`

`metrics.push.bearerTokenSecretRef.*`

`metrics.push.shipper.*`

Architecture Details

Sidecar Containers

Service Configuration

Pod Annotations

Configuration Examples

Pull Mode with ServiceMonitor

Pull Mode with Annotations Only

Push Mode to Grafana Cloud

Push Mode to Mimir

Disabled (Default)

Troubleshooting

ServiceMonitor not picked up by Prometheus

Push mode authentication failures

High memory usage in sidecars

Security Considerations