Trace data is the most expensive signal most teams collect, and it is also the one they trim last because everyone fears losing the single request that explains an outage. This tutorial shows an orderly, evidence-led way to bring trace volume under control for the biggest producer in your estate, one stage at a time. It sits inside ObservabilityOps, the practice of running the observability system itself, as framed by the Ops Atlas page devoted to ObservabilityOps. You will need a working OpenTelemetry Collector deployment, access to your trace back end and one volunteer service team.
Before you start: know what you are trading
Two broad approaches exist. Head sampling chooses at the first span whether to keep a trace. It is cheap and stateless, but the choice is made before anyone knows whether the request will fail, so errors are not guaranteed to survive. Tail sampling waits for the whole trace and then decides, which lets you keep every error and every slow request. The cost, as the Atlas puts it by quoting the project documentation, is that it is stateful and difficult to operate. Keep both in mind while following the steps below.
Step 1: confirm that sampling is worth doing yet
Do not sample out of habit. The Atlas notes that OpenTelemetry guidance begins to suggest sampling at volumes around 1,000 traces per second. Below that, the effort and the risk of dropping something useful probably outweigh the saving. Measure your busiest service, record its peak rate and its share of the trace bill, and write both on the ticket. If a single producer accounts for most of the volume, you have found your first candidate.
Step 2: derive your rates from everything, first
Request counts, error rates and latency histograms should be computed from all spans before any are discarded. Otherwise a dashboard that says 200 requests per second may be reporting a one percent sample multiplied incorrectly, and every alert built on it will mislead. In the Collector this usually means placing a span-to-metrics stage ahead of the sampler in the pipeline order. Write down the order explicitly; a surprising number of incidents start when someone reorders processors during a tidy-up.
Step 3: check the metrics you are about to generate
Span-derived metrics have a known trap. If a per-instance identifier, such as a collector instance UUID, lands on the generated series, every replica multiplies the series count, and each restart creates a fresh set. The Atlas recommends aggregating that label away with a sum that excludes it, and specifically says that dropping labels at scrape time or raising the sample limit are the wrong remedies. Test this on a staging copy first: restart a collector and verify that the number of active series does not jump.
Step 4: route whole traces to one place
Tail sampling only works if every span of a trace reaches the same sampling instance. In a multi-replica setup this needs a two-tier arrangement: a first layer that load-balances using the trace identifier as the routing key, and a second layer that holds traces in memory long enough to decide. Plan memory carefully. The longer your decision window and the busier the service, the more traces sit in RAM. Start with a modest window, measure, and widen only if slow traces are being cut off before the decision is made.
If your teams are used to thinking of data pipelines as stages that feed models, the discussion of a stage-by-stage MLOps roadmap for data and engineering teams offers a comparable mental picture of ordered pipeline steps, even though the subject differs.
Step 5: write the keep rules in plain language first
Before touching configuration, agree on a short list a non-engineer could read:
- Keep every trace that contains an error.
- Keep every trace slower than an agreed threshold for that endpoint.
- Keep a small, fixed percentage of everything else for baselines.
Then translate the list into policies. The Atlas cites a social network that retains about a tenth of one percent of successful traces together with all failed ones. Treat that ratio as an illustration of what is possible and not a target; your baseline percentage should come from how often engineers genuinely need a normal trace for comparison. Record the reasoning, because the first question after the next incident will be why a particular trace is missing.
Step 6: roll out to one service and watch two numbers
Deploy to the volunteer team behind a flag that lets you switch back within minutes. Watch two figures for at least a week. The first is trace volume reaching the back end, which should fall to the expected fraction. The second is the share of incident investigations that still found the trace they needed. If engineers report missing evidence, adjust the rules before widening the rollout. Also keep an eye on the pipeline’s own export success rate, counting a failure only once retries are exhausted. A figure that sounds comfortable at 99.5 percent can still fall short of a 99.9 percent objective, a gap the Atlas demonstrates with a short calculation.
Step 7: document, then extend
Once the pilot is stable, publish a short page covering the pipeline order, the policies, the rollback procedure and the owner. Add the next largest producer, then the next. Revisit every quarter: services change shape, and a policy tuned for last year’s traffic may keep too much or too little. Fold the work into your maturity picture too. Per the Atlas ladder, introducing tail sampling for the biggest producer belongs to the last stage of its ninety-day plan, after ownership, standards and limits are already in place.
Communicating the change to engineers
Sampling changes what people see, so announce it before it lands. A short note in the team channel should say which service is affected, what is always kept, what proportion of the rest is kept, and where to complain if a trace they expected is missing. Hold a fifteen-minute walkthrough with the pilot team, using a real recent incident as the test case: would the policy have kept the trace that solved it? If the answer is no, adjust the rules in the room, not weeks later. People accept constraints they helped shape, and the habit of consulting them first keeps the observability owner from drifting into the role of a gatekeeper.
Version notes that save time
Collector configuration changes between releases. The Atlas observed, as of October 2026, that the Collector was at v0.163.1 and that a receiver named otlp had been renamed otlp_grpc at v0.144.0, a classic cause of broken upgrades. It also notes that the profiling signal reached public alpha in March 2026, that the Span Event API was deprecated the same month, and that semantic conventions had reached v1.44.0. Pin versions in source control, read the changelog before each bump, and test the full pipeline on staging with a synthetic trace that must come out the other end.
Troubleshooting a few common symptoms
- Traces arrive in fragments. Look for a queue hop without span links, a proxy that strips the traceparent header, or a service still emitting an older propagation format.
- Memory climbs steadily on the sampling tier. Shorten the decision window, add replicas or confirm the load balancer really routes by trace identifier.
- Error traces still go missing. Check that the keep-errors rule runs before the percentage rule and that the status field is being set by the instrumented code.
- Metrics no longer match the dashboard. Revisit Step 2 and confirm rates are generated before the sampler.
How to judge success
Define success before you start. A reasonable result is a clear drop in trace volume for the pilot service, no increase in the number of investigations blocked by missing data, and a unit cost that trends down over a month. The Atlas’s highest maturity stage is described in just those terms: unit cost falls while coverage holds. For a deeper look at why systematic pipelines pay off, see this reflection on the role operations discipline plays in machine learning projects, which makes a similar argument about repeatable, measurable stages.
Finish the exercise by writing a one-page summary: what you changed, the volume before and after, any evidence that went missing, and the next candidate. That page becomes the template for every later rollout.