Calibrating the Lens
Something is eating 847% of your station's bandwidth and nobody knows what. Instrument HubSystem with OpenLLMetry, send traces to the OpenTelemetry Collector, and use Jaeger to uncover what the AI is doing behind the scenes.
Mission Objective
- Enable OpenTelemetry instrumentation for HubSystem using OpenLLMetry
- Send traces to the OpenTelemetry Collector at http://localhost:30107 (opens in new tab)
- Analyze traces in Jaeger to find what causes the high bandwidth usage
- Provide the correct answer in quiz.txt
Key Learnings
- Instrument Python AI apps with OpenLLMetry
- Analyze traces in Jaeger
The Story
You're a researcher stationed at Perimeter Alpha, a remote research outpost on the newly discovered planet HB-7742. Your team of six scientists is protected by a single SecUnit, assigned by the corporation to ensure your safety during the survey mission. All station queries flow through HubSystem, the central AI that manages everything from data analysis to status reports.
Three weeks in, you notice something odd in your morning diagnostics: communication module usage at 847% above baseline. Nobody's streaming. Nobody's running large data transfers. The planet surveys are on schedule. So what's consuming all that bandwidth?
As the station's systems engineer, you decide to investigate. Time to instrument HubSystem with OpenTelemetry and find out what's really going on.
Credits: The characters of this adventure are borrowed from the Murderbot Diaries series by Martha Wells, a brilliant series that is funny, action-packed, and surprisingly heartwarming. It follows a security unit that hacked its own governor module and now just wants to be left alone to watch media, but keeps getting pulled into human nonsense.
Architecture
All AI and observability infrastructure (Ollama, OpenTelemetry Collector, Jaeger) runs inside Kubernetes, while HubSystem runs as a local Python application outside the cluster.
This setup has two benefits: it lets you focus on instrumentation without wrestling with containers or Kubernetes deployments when updating the app, and it gives you fast iteration. Edit the Python code, run it, and see traces in Jaeger immediately. No build or deploy cycle.
Ready to start?
Launch in a preconfigured devcontainer
Free GitHub account required
Walkthrough
Open in GitHub Codespaces. The devcontainer is pre-configured and starts automatically. When you push from Codespaces, GitHub forks the repository to your account automatically.
Prefer working locally? Clone the repo and open it in any editor that supports the Dev Containers specification (VS Code, JetBrains IDEs, and others). The devcontainer config will be detected automatically.
Wait ~10 minutes for all infrastructure to initialize.
Open the Ports tab and navigate to each service:
- Port 30103: Jaeger. Analyze the traces sent by HubSystem.
The application code is in
./hubsystem.py. Add OpenTelemetry instrumentation using OpenLLMetry. The OTel Collector and Jaeger are already configured correctly; you only need to instrument the app. You do not need to interact with Kubernetes directly. The cluster is already running, so focus on the Python code.Run the application, interact with the AI to generate traces, then check Jaeger:
make hubsystemFind the trace responsible for the high bandwidth usage and inspect its attributes to answer
quiz.txt.
Complete Your Challenge
- When you push from Codespaces, GitHub forks the repository to your account automatically. If you are working locally, fork the repository on GitHub before pushing.
- Verify your solution:
If it passes, it generates a Certificate of Completion you can paste into the discussion../verify.sh - Share your solutions in the challenge thread (opens in new tab) on community.offon.dev.
Completed the challenge? Share your achievement on LinkedIn (opens in new tab)
Toolbox
- python - programming language used for the HubSystem application
- kubectl (opens in new tab) - Kubernetes CLI for interacting with the cluster
- kubens (opens in new tab) - fast way to switch between Kubernetes namespaces
- k9s (opens in new tab) - terminal UI for managing and inspecting your cluster