Skip to main content

Hyperspace Operations & Transport

Expert
Argo RolloutsOpenTelemetryJaegerPromQL

The observability pipeline is broken and HotROD's canary can't validate. Wire an OpenTelemetry Collector with spanmetrics to convert distributed traces into Prometheus metrics, then write PromQL queries that catch idle canaries, high error rates, and latency spikes.

Mission Objective

  • Automated rollout progression to HotROD version 1.76.0 driven by observability signals
  • OpenTelemetry Collector configured with an OTLP receiver for HotROD traces, a Spanmetrics connector converting traces to metrics, trace export to Jaeger, and metrics export to Prometheus
  • Canary analysis with three PromQL queries: traffic detection (at least 0.05 req/s to prevent idle canaries), error rate below 5%, and 95th-percentile latency below 1000ms

Key Learnings

  • Configure OpenTelemetry Collector pipelines
  • Spanmetrics connector (traces to metrics)
  • Detect idle canaries with traffic validation
  • Distributed tracing with Jaeger
The Story

After fixing the Zephyrian communications, word of your progressive release mastery spread across the galaxy. The Bytari, a highly advanced species from the Andromeda sector, were impressed.

They want to apply progressive delivery to their mission-critical service: HotROD (Hyperspace Operations & Transport, Rapid Orbital Dispatch), an interstellar ride-sharing service handling dispatch requests across thousands of star systems. Every millisecond of latency matters, and any error could strand travelers between dimensions.

A previous engineer started instrumenting HotROD with OpenTelemetry and configured Argo Rollouts for automated validation, but left the setup incomplete. The observability pipeline is broken. The Bytari don't use staging/production environments; they believe in single-environment progressive delivery validated purely by trace-derived metrics and automated health checks.

Your mission: fix the observability pipeline and canary validation. Make HotROD deployment-ready with proper distributed tracing.

Ready to start?

Launch in a preconfigured devcontainer

Open in Codespaces (opens in new tab)

Free GitHub account required

Walkthrough

  1. Open in GitHub Codespaces. The devcontainer is pre-configured and starts automatically. When you push from Codespaces, GitHub forks the repository to your account automatically.

    Prefer working locally? Clone the repo and open it in any editor that supports the Dev Containers specification (VS Code, JetBrains IDEs, and others). The devcontainer config will be detected automatically.

  2. Wait ~5-10 minutes for infrastructure to deploy. Port forwarding starts automatically after infrastructure is ready, keeping a terminal busy. Open a new terminal to run commands.

  3. Open the Ports tab and navigate to each service:

    • Port 30100: Argo CD (readonly / a-super-secure-password). Shows sync status. Use to refresh applications after pushing commits.
    • Port 30101: Argo Rollouts. Shows canary deployment progress and analysis status.
    • Port 30102: Prometheus. Explore available metrics and test PromQL queries. CLI tools work equally well if you prefer the terminal.
    • Port 30103: Jaeger. Shows distributed traces from HotROD to verify that tracing is working end-to-end.

  4. Fix the manifests in adventures/01-echoes-lost-in-orbit/expert/manifests/. Use the Argo Rollouts dashboard, Prometheus UI, and Jaeger UI to debug and validate your changes.

  5. Commit and push to trigger the deployment:

    git add adventures/01-echoes-lost-in-orbit/expert/manifests/
git commit -m "Fix configuration"
git push

    If pushing to a branch other than main, also update the ApplicationSet in appset.yaml to point to your branch.

    Refresh Argo CD apps:

    argocd app get hotrod --refresh
argocd app get otel --refresh

    If you changed HotROD, retry the rollout:

    kubectl argo rollouts retry rollout hotrod -n hotrod

    If you changed the OTel Collector config, restart it:

    kubectl rollout restart daemonset/collector -n otel

  6. Watch rollout progress. The rollout should progress automatically based on analysis metrics:

    kubectl argo rollouts get rollout hotrod -n hotrod --watch

  7. Run the smoke test to verify your solution:

    adventures/01-echoes-lost-in-orbit/expert/smoke-test.sh

Complete Your Challenge

  • When you push from Codespaces, GitHub forks the repository to your account automatically. If you are working locally, fork the repository on GitHub before pushing.
  • Verify your solution:
    adventures/01-echoes-lost-in-orbit/expert/smoke-test.sh
    If it passes, it generates a Certificate of Completion you can paste into the discussion.
  • Share your solutions in the challenge thread (opens in new tab) on community.offon.dev.

Completed the challenge? Share your achievement on LinkedIn (opens in new tab)

Toolbox

Helpful Documentation

Know someone who'd enjoy this?