Propagating Trace Context Across Subgraphs with OpenTelemetry

Linking a router span to its subgraph spans in one distributed trace requires the W3C traceparent header to flow from the Apollo Router into each subgraph and be adopted as the parent of that subgraph’s server span. This guide shows the exact router and subgraph configuration that makes a federated trace a single connected tree rather than disconnected fragments.

This is the implementation detail behind the broader Observability and Distributed Tracing in Federation guide.

When to use this pattern

  • You see a router trace and separate subgraph traces in your backend, with no parent-child link between them.
  • You need per-subgraph and per-resolver latency attributed under a single client operation.
  • You are standardising on OpenTelemetry/OTLP across the router and all @apollo/server subgraphs and want spec-compliant context propagation.

Prerequisites

How Context Propagation Works

OpenTelemetry context propagation is two halves on two sides of an HTTP call. On the router side, when the query planner issues a fetch to a subgraph, the router injects the active span context into the outgoing request as a W3C traceparent header (format: 00-<trace-id>-<span-id>-<flags>). On the subgraph side, the HTTP instrumentation must extract that header, rebuild the remote span context, and start its server span as a child of it. If either half is missing, the subgraph starts a brand-new root span and the trace fragments.

The router does this automatically once propagation is enabled in router.yaml. The subgraph does it automatically only if its OpenTelemetry SDK has the W3C trace-context propagator registered and HTTP auto-instrumentation active — which is the part teams most often miss.

It helps to be precise about the W3C format, because most propagation bugs are really format bugs. A traceparent header has four hyphen-separated fields: a two-digit version (00), a 32-hex-character trace id, a 16-hex-character parent span id, and two trace-flags digits where the low bit indicates whether the trace was sampled. When the router injects this header, the trace id is the id shared by the whole operation and the parent span id is the router’s fetch span — so the subgraph’s extracted parent is that fetch span, which is exactly the linkage you want. The optional tracestate header carries vendor-specific data and rides along automatically; you rarely touch it directly. The two ways this breaks are a subgraph configured for a different propagation format (B3 single/multi-header, used by some Zipkin setups, or Jaeger’s uber-trace-id) so it never looks for traceparent, and a sampling flag of 00 that tells the subgraph “the root chose not to record this,” which a correctly configured ParentBasedSampler will honour by also not recording. Both are configuration choices, not bugs in OpenTelemetry — which is why the fix is always in setup, never in code you write per request.

Anatomy of the traceparent header The traceparent value is split into four hyphen separated fields: a two character version, a thirty two character trace id shared by every span in the operation, a sixteen character parent span id that changes at each hop, and two flag characters carrying the sampling decision. traceparent: the four fields that stitch a trace together 00 4bf92f3577b34da6a3ce929d0e0e4736 00f067aa0ba902b7 01 version always 00 today trace id — never changes the same value in the router span and in every subgraph span; this is what groups them into one waterfall parent span id — changes every hop each service sends its own span id as the next hop's parent, which is what builds the nesting rather than a flat list flags — the sampling decision, made once at the head

Implementation Walkthrough

1. Enable propagation in the router

Tell the router to propagate W3C trace context on outgoing subgraph requests and to keep sampling decisions consistent across the trace.

# router.yaml
telemetry:
  exporters:
    tracing:
      common:
        service_name: apollo-router
        # honour the root's sampling decision so children aren't dropped
        parent_based_sampler: true
        sampler: 1.0                     # sample everything while verifying
      otlp:
        enabled: true
        endpoint: http://otel-collector:4317
        protocol: grpc
  instrumentation:
    spans:
      subgraph:
        attributes:
          subgraph.name: true            # label each fetch span
  # propagation block: which formats the router injects/extracts
  apollo:
    # (Apollo Studio FTV1 is separate; this section is OTLP propagation)

The router emits W3C traceparent on subgraph requests by default when tracing is enabled; parent_based_sampler: true is the critical line — without it a subgraph may independently decide not to sample and you lose the child spans.

2. Instrument the subgraph with @opentelemetry/sdk-node

Create a tracing bootstrap file that registers the W3C trace-context propagator and HTTP auto-instrumentation, then load it before anything else (so HTTP and GraphQL are patched at import time).

// tracing.ts  —  MUST be imported first, before @apollo/server / http
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import {
  W3CTraceContextPropagator,
} from '@opentelemetry/core';
import {
  ParentBasedSampler,
  TraceIdRatioBasedSampler,
} from '@opentelemetry/sdk-trace-base';

const sdk = new NodeSDK({
  resource: new Resource({
    // service.name MUST match what you expect to see in the backend
    [SemanticResourceAttributes.SERVICE_NAME]: 'products-subgraph',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4317', // same collector the router uses
  }),
  // Extract/inject W3C traceparent so the router's span becomes our parent
  textMapPropagator: new W3CTraceContextPropagator(),
  // Honour the router's sampling decision; only sample roots we originate
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(1.0),
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // the HTTP instrumentation is what reads the incoming traceparent
      '@opentelemetry/instrumentation-http': { enabled: true },
    }),
  ],
});

sdk.start(); // begins extracting context from inbound requests immediately

Wire it in ahead of the server. With ts-node/tsx use --import ./tracing.ts, or simply import './tracing'; as the very first line of your entrypoint:

// index.ts
import './tracing';                       // <-- first line, no exceptions
import { ApolloServer } from '@apollo/server';
import { startStandaloneServer } from '@apollo/server/standalone';
import { buildSubgraphSchema } from '@apollo/subgraph';
import gql from 'graphql-tag';

const typeDefs = gql`
  extend schema
    @link(url: "https://specs.apollo.dev/federation/v2.9", import: ["@key"])

  type Product @key(fields: "id") {
    id: ID!
    name: String!
  }
`;

const resolvers = {
  Product: {
    // resolver spans created here will nest under the router's fetch span
    __resolveReference: (ref: { id: string }) => ({ id: ref.id, name: 'Widget' }),
  },
};

const server = new ApolloServer({ schema: buildSubgraphSchema({ typeDefs, resolvers }) });
const { url } = await startStandaloneServer(server, { listen: { port: 4001 } });
console.log(`products-subgraph ready at ${url}`);

The auto-instrumentation patches the inbound HTTP server, so the moment the router’s request arrives carrying traceparent, the subgraph’s server span is opened as a child of the router’s fetch span — and every resolver/__resolveReference span nests beneath it.

3. Continue the trace into the subgraph’s own dependencies

Linking the router to the subgraph is the headline goal, but the trace is far more useful if it does not dead-end at the subgraph server span. The same auto-instrumentation set that extracts traceparent on the way in also injects it on the way out of the subgraph’s own HTTP and database calls — provided you enabled the relevant instrumentations. Because getNodeAutoInstrumentations() bundles instrumentation for http, common database drivers (pg, mysql2, mongodb, ioredis), and graphql, a subgraph that queries Postgres will, with no extra code, produce a span tree of fetch (router) → subgraph server → __resolveReference → pg.query. That last span is usually where a federated N+1 becomes obvious: instead of one pg.query span you see twenty identical ones in a row beneath a single reference resolution. Keeping these downstream spans inside the same trace is what turns “the products subgraph is slow” into “the products subgraph issues one query per product key instead of batching” — an actionable, code-level finding rather than a service-level one.

If a subgraph calls a downstream service that is not auto-instrumented (say, a hand-rolled fetch wrapper or a non-Node service), make sure the W3C propagator is what stamps the outbound request, so the next hop can adopt the context just as this subgraph did from the router. The chain only stays connected if every hop both extracts on entry and injects on exit using the same format.

One trace id, a chain of parent span ids The router span is the root. Two subgraph spans name the router span as parent, and a resolver span inside one subgraph names that subgraph span as its parent, producing three levels of nesting under a single trace id. trace id 4bf9…4736 — shared by all four spans router span (root) span id 00f0…02b7, no parent accounts subgraph span parent 00f0…02b7 products subgraph span parent 00f0…02b7, span id 9c2a… resolver span parent 9c2a… — nested one level deeper

Verification Steps

  1. Send a traced operation. Issue a query through the router so the planner fetches the subgraph:

    curl -s http://localhost:4000/ \
  -H 'content-type: application/json' \
  -d '{"query":"{ topProducts { id name } }"}' > /dev/null

  2. Confirm the header on the wire. Temporarily log inbound headers in the subgraph (or inspect with a debugging proxy). You should see a single traceparent per request:

    traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

    The middle segment is the router span id — that is the parent the subgraph will adopt.

  3. Check the backend trace tree. In your tracing UI (Tempo/Jaeger/Datadog), open the trace by id. You should see one tree: apollo-router → fetch (products) → products-subgraph server span → resolver spans. The trace.id is identical across all of them.

  4. Confirm shared trace id. Both processes report the same trace_id — if the subgraph shows a different trace id, the propagator is not registered or tracing.ts was not loaded first.

What a dropped traceparent actually looks like With propagation configured, one trace contains the router and subgraph spans. Without it, the subgraph starts a brand new trace id, so the backend shows two short unrelated traces instead of one and the correlation is unrecoverable. The symptom is not a missing trace — it is two short ones header propagated router span — 142 ms accounts — 38 ms products — 61 ms header dropped trace A — router span only, no children trace B — accounts, orphan root trace C — products, orphan root Because each orphan is a valid trace on its own, dashboards stay green and nothing errors. The tell is a router trace with no child spans — check that first when a waterfall looks too short.

Common Mistakes and Gotchas

  • Loading tracing.ts too late. If @apollo/server or http is imported before the SDK starts, the HTTP instrumentation never patches the server and the incoming traceparent is ignored. The bootstrap import must be the first statement in the process.
  • Forgetting parent_based_sampler. Without it (router) or ParentBasedSampler (subgraph), each service samples independently and you get half-trees — the router span sampled, the subgraph span dropped, or vice versa.
  • Mismatched propagator formats. The router emits W3C traceparent. If a subgraph is configured only for B3 or Jaeger propagation it won’t extract the header. Register W3CTraceContextPropagator (or a CompositePropagator including it) on the subgraph.
  • Different collector endpoints. If the router exports to one collector and a subgraph to another that writes to a different backend, the spans share a trace id but live in two stores and never appear as one tree. Point every service at the same trace pipeline.
  • Sampling at full volume in production. sampler: 1.0 and TraceIdRatioBasedSampler(1.0) are correct while verifying, but recording every trace in a high-throughput system is expensive. Once linkage is confirmed, lower the root ratio and rely on ParentBased/parent_based_sampler so the whole tree is recorded or dropped as a unit.

Frequently Asked Questions

Do I need to manually read the traceparent header in my resolvers?

No. The @opentelemetry/instrumentation-http auto-instrumentation extracts the header and establishes the remote parent context before your resolvers run, provided the SDK (with the W3C propagator) was started before the HTTP server was imported. Your resolver and __resolveReference spans are created within that context automatically.

Why is my subgraph creating a new root trace instead of a child span?

Either the W3C trace-context propagator is not registered on the subgraph SDK, or tracing.ts is loaded after @apollo/server/http, so HTTP auto-instrumentation never patched the inbound server. Register W3CTraceContextPropagator and make the tracing bootstrap the first import in the process.

Does this work the same with a manually composed Express subgraph?

Yes. As long as the OpenTelemetry HTTP instrumentation is active and the W3C propagator is registered before the Express app is created, the incoming traceparent is extracted regardless of whether you use startStandaloneServer or expressMiddleware.