Wharf OTLP Trace Collector - CrewAI Platform Helm Chart

The Wharf service is an OTLP (OpenTelemetry Protocol) trace collector that receives, processes, and stores distributed tracing data from the CrewAI Platform. It provides observability into application performance and helps diagnose issues in distributed workflows.

Overview

Wharf operates as a separate microservice with its own database for storing trace data. When enabled, it automatically:

Receives OTLP trace data from the CrewAI Platform components
Stores traces in a dedicated PostgreSQL database
Provides a health check endpoint for monitoring
Exposes Prometheus metrics for observability
Configures automatic trace data retention

Prerequisites

Wharf is enabled by default in the chart. Before installing or upgrading, ensure the following requirements are met:

Wharf database: When using external PostgreSQL (postgres.enabled: false), the Wharf database must exist before installation. See Database Management below.
CrewAI version: Trace collection requires CrewAI >= 1.12. Versions >= 1.13 include the best support, with token usage and inputs captured in spans.

To opt out of trace collection, set wharf.enabled: false in your values.

Database Management

With internal PostgreSQL (postgres.enabled: true): The Helm chart automatically handles database initialization:

Auto-Creation: The Wharf database is created automatically during initial chart installation
Migrations: Database schema migrations run automatically during installation and upgrades
Database Readiness: Wharf waits for PostgreSQL to become available before proceeding
Isolation: Wharf uses a separate database from the main application for trace data isolation

With external PostgreSQL (postgres.enabled: false): When using a managed database service (AWS RDS, Google Cloud SQL, Azure Database), you must manually create the Wharf database before installing the chart. The database name should match postgres.wharfDatabase (default: wharf). If the database is missing, the migration will fail during installation or upgrade.

CREATE DATABASE wharf;

Grant the application user full privileges on the database:

GRANT ALL PRIVILEGES ON DATABASE wharf TO your_db_user;

Schema migrations still run automatically during each Helm upgrade.

Planning for external PostgreSQL

Never use the application database: The primary Rails database (for example, a database named like crewai_plus_production) is only for the CrewAI Platform application. Wharf must use a different logical database. Do not point Wharf at the Rails database.
Typical layout: Most teams reuse the same managed PostgreSQL instance they already use for the platform (for example, RDS, Aurora, Azure Database for PostgreSQL, Cloud SQL, or self-managed PostgreSQL) and add one additional empty database for traces—not a second server by default.
Naming: A descriptive name such as crewai_wharf helps operators see that the database is for trace storage. The chart default for postgres.wharfDatabase is wharf. Whatever name you create in PostgreSQL, postgres.wharfDatabase in your values must match it.
Separate PostgreSQL instance: Use a dedicated database server (not only a separate database name) when you need stronger isolation, independent backup and restore windows, or you expect very high trace volume and want to avoid load on the application database. That is optional and usually driven by governance or scale.
Network: When Wharf is enabled, the cluster needs the same kind of reachability to PostgreSQL as for the app (host, port, typically 5432), including firewalls and security groups, for the Wharf logical database.

Cloud SQL Auth Proxy Support (GCP)

When using Google Cloud SQL with Workload Identity, Wharf supports the Cloud SQL Auth Proxy sidecar:

Enabled: When cloudSqlProxy.enabled: true, a sidecar container is automatically added to Wharf pods
Purpose: Provides secure, encrypted connections to Cloud SQL instances using IAM authentication
Configuration: Uses the same cloudSqlProxy configuration as other services (web, worker, OAuth, job)

See Cloud SQL Auth Proxy Configuration for detailed configuration options.

Configuration

wharf.enabled

boolean

default:"true"

Enable or disable the Wharf OTLP trace collector service.Values:

true - Deploy Wharf trace collector (default)
false - Disable trace collection

When Enabled: The chart creates:

Wharf Deployment with configurable replicas
ClusterIP Service for OTLP trace ingestion
Dedicated PostgreSQL database for trace storage
Auto-injected environment variables for Rails integration

Image Configuration

wharf.image.host

string

default:""

Container registry hosting the Wharf image.Default: "" (empty) - Automatically uses global.imageRegistry valueFallback Behavior:When wharf.image.host is empty or not set, the chart uses global.imageRegistry. This allows you to configure the registry once at the global level.Example:

global:
  imageRegistry: "registry.company.com"

wharf:
  enabled: true
  image:
    host: "" # Automatically uses registry.company.com
    name: "proxy/crewai/crewai/wharf"
    tag: "0.1.2"

wharf.image.name

string

default:"proxy/crewai/crewai/wharf"

Wharf container image name.Default: "proxy/crewai/crewai/wharf" - Matches Replicated proxy path structurePath Transformation:When global.imageNamePrefixOverride is set, only the final component (wharf) is used with the override prefix.

wharf.image.tag

string

default:"0.1.2"

Wharf service version to deploy.Default: "0.1.2"Upgrade Consideration: Wharf handles database migrations automatically when the version changes.

wharf.image.pullPolicy

string

default:"IfNotPresent"

Image pull policy for Wharf container.Valid Values:

"IfNotPresent" - Pull only if image is not present locally
"Always" - Always pull image on pod start
"Never" - Never pull, use local image only

wharf.image.pullSecret

string

default:""

Image pull secret for Wharf image. If empty, uses the global image.pullSecret.Default Behavior: Inherits from image.pullSecret for consistency

Service Configuration

wharf.replicaCount

integer

default:"1"

Number of Wharf service replicas to deploy.Scaling Considerations:

Single replica is sufficient for most deployments
Multiple replicas can be used for high availability
Wharf is stateless and can scale horizontally

Default: 1

wharf.port

integer

default:"8080"

HTTP port for OTLP trace ingestion and health checks.Default: 8080Purpose: Receives OTLP traces and serves health check endpoint at /health

wharf.metricsPort

integer

default:"9090"

Prometheus metrics endpoint port.Default: 9090Purpose: Exposes service metrics in Prometheus format at /metricsAnnotations: The Deployment automatically includes Prometheus scrape annotations:

prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "9090"

Storage and Processing

wharf.config.storageType

string

default:"postgres"

Backend storage type for trace data.Default: "postgres"Supported Values:

"postgres" - PostgreSQL database storage (recommended)

Database Connection: Automatically configured using the shared database credentials from envVars.DB_USER and secrets.DB_PASSWORD

wharf.config.sinkType

string

default:"memory_batch"

Trace processing sink type.Default: "memory_batch"Purpose: Controls how traces are buffered and written to storageSupported Values:

"memory_batch" - Batch traces in memory before writing (better performance)

wharf.config.retentionWeeks

string

default:"13"

Trace data retention period in weeks.Default: "13" (approximately 3 months)Purpose: Automatically removes trace data older than the specified period to manage storageRecommendations:

Development: "4" (1 month)
Production: "13" (3 months) or longer based on compliance requirements
High-volume: Reduce retention to manage database size

Format: String representation of weeks (e.g., "13", "26", "52")

Resource Allocation

wharf.resources.limits.cpu

string

default:"500m"

CPU limit for Wharf pod.Default: "500m" (0.5 CPU cores)Sizing Guidelines:

Light tracing: "500m"
Moderate tracing: "1"
Heavy tracing: "2" or higher

wharf.resources.limits.memory

string

default:"512Mi"

Memory limit for Wharf pod.Default: "512Mi"Sizing Guidelines:

Light tracing: "512Mi"
Moderate tracing: "1Gi"
Heavy tracing: "2Gi" or higher

wharf.resources.requests.cpu

string

default:"100m"

CPU request for Wharf pod.Default: "100m" (0.1 CPU cores)Purpose: Guaranteed CPU allocation for scheduling

wharf.resources.requests.memory

string

default:"128Mi"

Memory request for Wharf pod.Default: "128Mi"Purpose: Guaranteed memory allocation for scheduling

Node Placement

wharf.nodeSelector

object

default:"{}"

Node selector labels for Wharf pod placement.Default: {} (no node restrictions)Example:

wharf:
  nodeSelector:
    node-type: observability
    topology.kubernetes.io/zone: us-west-2a

Use Cases:

Isolate observability workloads on dedicated nodes
Co-locate with other monitoring services
Comply with data residency requirements

Integration with CrewAI Platform

When Wharf is enabled, the following environment variables are automatically injected into the Rails application and worker pods:

WHARF_URL - Internal Kubernetes service URL for trace submission
WHARF_JWT_SECRET - Shared secret for authenticating trace submissions

These variables are configured automatically and do not require manual configuration. The Rails application uses these to send OTLP traces to the Wharf service.

Security

Wharf runs with security-hardened settings:

Non-root execution: Runs as user ID 1000
Read-only root filesystem: Prevents runtime modifications
Dropped capabilities: All Linux capabilities removed
No privilege escalation: allowPrivilegeEscalation: false

Monitoring

Health Checks:

Startup Probe: GET /health - Initial availability check (max 5 minutes)
Liveness Probe: GET /health - Service health monitoring (every 10 seconds)
Readiness Probe: GET /health - Traffic routing readiness (every 5 seconds)

Metrics:

Endpoint: http://wharf-pod:9090/metrics
Format: Prometheus exposition format
Auto-Discovery: Prometheus scrape annotations included

Example Configurations

Minimal Configuration (Default)

Wharf is enabled by default; no values are required to deploy it with default settings. The configuration below is shown for completeness:

wharf:
  enabled: true
  # All other settings use defaults

Custom Resource Limits

wharf:
  enabled: true
  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
    requests:
      cpu: "200m"
      memory: "256Mi"

Extended Retention with High Availability

wharf:
  enabled: true
  replicaCount: 2
  config:
    retentionWeeks: "26"  # 6 months
  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
    requests:
      cpu: "250m"
      memory: "512Mi"

Disabled (Opt-Out)

Set wharf.enabled: false to override the default and skip deploying the trace collector:

wharf:
  enabled: false  # Disable trace collection

GCP Cloud SQL with Cloud SQL Auth Proxy

# Enable Wharf with Cloud SQL Auth Proxy sidecar
wharf:
  enabled: true
  replicaCount: 1
  config:
    storageType: "postgres"
    retentionWeeks: "13"

# Cloud SQL Auth Proxy configuration (shared across all services)
cloudSqlProxy:
  enabled: true
  instanceConnectionName: "my-project:us-central1:crewai-db"
  port: 5432
  privateIp: true
  autoIamAuthn: true

# Database connection configuration
envVars:
  DB_HOST: "127.0.0.1"
  DB_PORT: "5432"
  DB_USER: "crewai-platform@my-project.iam"

# ServiceAccount with Workload Identity
serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: crewai-platform@my-project.iam.gserviceaccount.com

Documentation Index

​Overview

​Prerequisites

​Database Management

​Planning for external PostgreSQL

​Cloud SQL Auth Proxy Support (GCP)

​Configuration

​Image Configuration

​Service Configuration

​Storage and Processing

​Resource Allocation

​Node Placement

​Integration with CrewAI Platform

​Security

​Monitoring

​Example Configurations

​Minimal Configuration (Default)

​Custom Resource Limits

​Extended Retention with High Availability

​Disabled (Opt-Out)

​GCP Cloud SQL with Cloud SQL Auth Proxy

Overview

Prerequisites

Database Management

Planning for external PostgreSQL

Cloud SQL Auth Proxy Support (GCP)

Configuration

Image Configuration

Service Configuration

Storage and Processing

Resource Allocation

Node Placement

Integration with CrewAI Platform

Security

Monitoring

Example Configurations

Minimal Configuration (Default)

Custom Resource Limits

Extended Retention with High Availability

Disabled (Opt-Out)

GCP Cloud SQL with Cloud SQL Auth Proxy