Skip to main content

The Silent Canary

Intermediate
Argo RolloutsPromQL

A canary rollout is stuck and the Zephyrians are still waiting to communicate. Debug the broken progressive delivery system by writing PromQL health checks that let Argo Rollouts automatically validate and advance the deployment.

Mission Objective

  • Pod info version 6.9.3 deployed successfully in both staging and production environments
  • Rollouts automatically progress through canary stages based on health metrics
  • Two working PromQL queries in the AnalysisTemplate that validate application health during releases
  • All rollouts complete successfully

Key Learnings

  • Progressive delivery with Argo Rollouts
  • Canary deployments & automated analysis
  • Write PromQL queries for health validation
  • Kube-state-metrics for deployment decisions
The Story

After fixing the communication outage in Level 1, the Intergalactic Union welcomed a new species: the Zephyrians.

The communications team attempted to deploy their language files using a progressive delivery system, but the rollout is failing. The Zephyrians are still waiting to communicate with the rest of the galaxy.

A previous engineer configured automated canary deployments with health checks but left the setup incomplete.

Your mission: debug the broken rollout and bring the Zephyrians' voices online.

Ready to start?

Launch in a preconfigured devcontainer

Open in Codespaces (opens in new tab)

Free GitHub account required

Walkthrough

  1. Open in GitHub Codespaces. The devcontainer is pre-configured and starts automatically. When you push from Codespaces, GitHub forks the repository to your account automatically.

    Prefer working locally? Clone the repo and open it in any editor that supports the Dev Containers specification (VS Code, JetBrains IDEs, and others). The devcontainer config will be detected automatically.

  2. Wait ~5-10 minutes for infrastructure to deploy. After it deploys, the setup script starts port forwarding to the Argo Rollouts dashboard, keeping a terminal busy. Open a new terminal to run commands.

  3. Open the Ports tab and navigate to each service:

    • Port 30100: Argo CD (readonly / a-super-secure-password). Shows sync status. Use to refresh applications after pushing commits.
    • Port 30101: Argo Rollouts. Shows canary deployment progress and analysis status.
    • Port 30102: Prometheus. Explore available metrics and test PromQL queries. CLI tools work equally well if you prefer the terminal.

  4. Review and fix the configuration in adventures/01-echoes-lost-in-orbit/intermediate/manifests/.

    This challenge uses Kustomize under the hood: a base set of manifests with environment-specific overlays (staging, prod). Argo CD detects and applies these automatically, so you don't need to run Kustomize commands manually.

  5. Commit and push your changes to trigger the deployment:

    git add adventures/01-echoes-lost-in-orbit/intermediate/manifests/
git commit -m "Fix configuration"
git push

    If pushing to a branch other than main, also update the ApplicationSet in appset.yaml to point to your branch.

    Speed up Argo CD sync:

    argocd app get echo-server-staging --refresh
argocd app get echo-server-prod --refresh

  6. After Argo CD syncs, retry the rollouts:

    kubectl argo rollouts retry rollout echo-server -n echo-staging
kubectl argo rollouts retry rollout echo-server -n echo-prod

  7. Watch canary progress (should advance 33% to 66% to 100%):

    kubectl argo rollouts get rollout echo-server -n echo-staging --watch
kubectl argo rollouts get rollout echo-server -n echo-prod --watch

    In real-world progressive delivery, staging is updated first, validated, and then changes are promoted to production. This challenge skips that separation so you can focus on the canary rollout mechanics and health checks without managing two promotion steps.

  8. Run the smoke test to verify your solution:

    adventures/01-echoes-lost-in-orbit/intermediate/smoke-test.sh

Complete Your Challenge

  • When you push from Codespaces, GitHub forks the repository to your account automatically. If you are working locally, fork the repository on GitHub before pushing.
  • Verify your solution:
    adventures/01-echoes-lost-in-orbit/intermediate/smoke-test.sh
    If it passes, it generates a Certificate of Completion you can paste into the discussion.
  • Share your solutions in the challenge thread (opens in new tab) on community.offon.dev.

Completed the challenge? Share your achievement on LinkedIn (opens in new tab)

Toolbox

Helpful Documentation

Know someone who'd enjoy this?