#Kubernetes Deployment

Zephyr's Kubernetes deployment option allows you to run the Zephyr edge worker on your own Kubernetes infrastructure. This deployment model provides maximum control over your infrastructure while leveraging Zephyr's deployment and versioning capabilities.

#Overview

The Kubernetes edge worker is a containerized service that handles asset uploads and serves your deployed applications. It integrates with:

• S3-compatible storage (AWS S3, MinIO, Ceph, etc.) for asset storage
• Redis-compatible KV store for environment configuration and snapshots
• OpenTelemetry-compatible collectors for OTLP HTTP traces, metrics, and logs

#Configuration

The recommended way to configure the worker is using a JSON config file mounted as a Kubernetes Secret. This approach keeps all your configuration in one place and makes it easier to manage secrets securely.

Alternatively, you can use environment variables directly, though this is less recommended for production deployments.

#JSON Config File (Recommended)

Create a JSON file containing all your configuration. Worker-specific keys use the ZE_WORKER_ prefix; OpenTelemetry keys use the standard OTEL_ prefix:

// Example configuration
{
  "ZE_WORKER_PORT": "8080",
  "ZE_WORKER_JWT_SECRET": "your-secret-key",
  "ZE_WORKER_LOG_LEVEL": "info",
  "ZE_WORKER_DELIMITER": "-",
  "OTEL_SERVICE_NAME": "ze-k8s-workers",
  "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment.name=production",
  "OTEL_METRIC_EXPORT_INTERVAL": "60000",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "http://otel-collector.observability.svc.cluster.local:4318",
  "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer%20your-token",
  "ZE_WORKER_S3_ENDPOINT": "https://s3.amazonaws.com",
  "ZE_WORKER_S3_REGION": "us-east-1",
  "ZE_WORKER_S3_ACCESS_KEY_ID": "AKIAIOSFODNN7EXAMPLE",
  "ZE_WORKER_S3_SECRET_ACCESS_KEY": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "ZE_WORKER_S3_BUCKET": "zephyr-assets",
  "ZE_WORKER_S3_FORCE_PATH_STYLE": "false",
  "ZE_WORKER_S3_CONNECTION_TIMEOUT_MS": "5000",
  "ZE_WORKER_S3_REQUEST_TIMEOUT_MS": "30000",
  "ZE_WORKER_S3_UPLOAD_REQUEST_TIMEOUT_MS": "60000",
  "ZE_WORKER_S3_MAX_SOCKETS": "200",
  "ZE_WORKER_S3_SOCKET_ACQUISITION_WARNING_TIMEOUT_MS": "35000",
  "ZE_WORKER_REDIS_HOST": "redis.default.svc.cluster.local",
  "ZE_WORKER_REDIS_PORT": "6379",
  "ZE_WORKER_REDIS_PASSWORD": "your-redis-password",
  "ZE_WORKER_REDIS_TLS": "true",
  "ZE_WORKER_REDIS_DB": "0",
  "ZE_WORKER_REDIS_PREFIX_ENVS": "{ze-k8s-worker}:envs:",
  "ZE_WORKER_REDIS_PREFIX_SNAPSHOTS": "{ze-k8s-worker}:snapshots:",
  "ZE_WORKER_REPLICA_NAME": "us-east-1",
  "ZE_WORKER_REPLICAS_URLS": [
    "https://ze.eu.example.com",
    "https://ze.ap.example.com",
  ],
}

#Health Check Endpoints

The worker exposes the following health check endpoints for Kubernetes probes:

#Observability

The Kubernetes worker emits OpenTelemetry traces, metrics, and logs over OTLP HTTP. Enable telemetry with any OTEL_EXPORTER_OTLP_* endpoint or by explicitly setting a signal exporter such as OTEL_TRACES_EXPORTER=otlp.

If you set only OTEL_EXPORTER_OTLP_ENDPOINT, the worker treats it as the collector base URL and sends each signal to:

• Traces: <endpoint>/v1/traces
• Metrics: <endpoint>/v1/metrics
• Logs: <endpoint>/v1/logs

Use the signal-specific endpoint variables when your collector exposes separate routes:

{
  "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "http://otel-collector:4318/v1/traces",
  "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT": "http://otel-collector:4318/v1/metrics",
  "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "http://otel-collector:4318/v1/logs",
}

Signal-specific endpoint variables export only their configured signal unless a shared base endpoint is also set.

For collectors that require authentication, set OTLP headers using comma-separated key=value pairs. Percent-encode spaces and other special characters:

{
  "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer%20your-token",
  "OTEL_EXPORTER_OTLP_TRACES_HEADERS": "x-api-key=trace-key",
}

Store header values that contain tokens, API keys, or tenant credentials in a Kubernetes Secret.

Set OTEL_SDK_DISABLED=true to force telemetry off, even when endpoints are configured.

Worker stdout is structured JSON:

{
  "timestamp": "2026-06-29T04:31:14.288Z",
  "level": "info",
  "service": "ze-k8s-workers",
  "replica": "primary",
  "message": "Worker listening",
  "server.port": 8080,
  "otel.enabled": true
}

#Configuration Reference

Below is the complete reference for all available configuration options. These can be set in your JSON config file or as environment variables.

#Bootstrap Option

#Required Options

#Server Options

#OpenTelemetry Options

#S3 Options

#Redis Options

#Replication Options

#Using Environment Variables

While the JSON config file is recommended, you can also configure the worker using environment variables directly. This can be useful for simple setups or when integrating with external secret management systems.

Set ZE_WORKER_CONFIG_JSON to the path of your config file. If both are provided, values in the JSON file take precedence over environment variables.

env:
  - name: ZE_WORKER_JWT_SECRET
    valueFrom:
      secretKeyRef:
        name: ze-worker-secrets
        key: jwt-secret
  - name: ZE_WORKER_S3_ENDPOINT
    value: 'https://s3.us-west-2.amazonaws.com'
  - name: ZE_WORKER_S3_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: ze-worker-secrets
        key: s3-access-key
  # ... additional environment variables

#Scaling and Replication

Zephyr workers support two scaling modes that can be used independently or combined:

Check worker logs on startup to verify replication:

# Workers with REPLICAS_URLS set (replicate to other regions)
{"level":"info","message":"Replica mode enabled","zephyr.replica":"us-east-1"}
{"level":"info","message":"Replica fan-out configured","zephyr.replica.count":1,"zephyr.replica.urls":["https://ze.eu.example.com"]}

# Workers without REPLICAS_URLS (receive-only, share storage with primary)
{"level":"info","message":"Replica mode enabled","zephyr.replica":"us-east-2"}
{"level":"info","message":"No replica URLs configured; receive-only mode enabled"}

#Same-Region Scaling

Multiple workers sharing the same S3 bucket and Redis instance. All workers read/write to the same storage, so no data replication is needed.

All workers use identical config. Only ZE_WORKER_REPLICA_NAME differs:

// US East - Worker 1
{
  "ZE_WORKER_REPLICA_NAME": "us-east-1-worker-1",
  "ZE_WORKER_S3_BUCKET": "zephyr-us-east",
  "ZE_WORKER_REDIS_HOST": "redis-us-east.example.com",
}

#Multi-Region Replication

Each region has its own S3 and Redis. One primary worker per region replicates data to other regions' primaries via HTTP sync.

Each region lists other regions' load balancer URLs:

// US East
{
  "ZE_WORKER_REPLICA_NAME": "us-east-1",
  "ZE_WORKER_REPLICAS_URLS": [
    "https://ze.eu.example.com",
    "https://ze.ap.example.com",
  ],
  "ZE_WORKER_S3_BUCKET": "zephyr-us-east",
  "ZE_WORKER_REDIS_HOST": "redis-us-east.example.com",
}

#Combined: Multi-Region with Scaling

Multiple workers per region (shared storage) + cross-region replication. All workers in a region replicate to one worker per other region.

All workers have ZE_WORKER_REPLICAS_URLS pointing to other regions' load balancers:

#Troubleshooting

#Common Issues

#Worker fails to start

1. Verify all required configuration options are set
2. Check that S3 endpoint is reachable from within the cluster
3. Ensure Redis host is resolvable and accessible
4. Check the worker logs using kubectl

#Readiness probe failing

1. Check S3 bucket exists and credentials have read/write access
2. Verify Redis connection (host, port, password, TLS settings)
3. Review worker logs for specific error messages

#Telemetry is not reaching the collector

1. Confirm the collector accepts OTLP HTTP, not only OTLP gRPC
2. Verify the worker pod can resolve and connect to the collector service
3. Use the collector base URL for OTEL_EXPORTER_OTLP_ENDPOINT, for example http://otel-collector:4318
4. Include the full /v1/traces, /v1/metrics, or /v1/logs path when using signal-specific endpoints
5. Configure OTLP headers when the collector requires a bearer token, API key, or tenant header
6. Confirm OTEL_SDK_DISABLED is not set to true

#Logs are too verbose

Set ZE_WORKER_LOG_LEVEL to warn or error. You can keep OpenTelemetry enabled while reducing stdout and exported log volume.

#Assets not serving correctly

1. Confirm ZE_WORKER_DELIMITER matches your DNS configuration
2. Verify S3 bucket policy allows the worker to read objects
3. Check that the Ingress or load balancer is routing traffic correctly

#Getting Help

If you encounter issues with your Kubernetes deployment, reach out to your Zephyr account representative or contact us on Discord.