Span profiles with Traces to profiles for Python
Span Profiles represents a major shift in profiling methodology, enabling deeper analysis of both tracing and profiling data. Traditional continuous profiling provides an application-wide view over fixed intervals. In contrast, Span Profiles delivers focused, dynamic analysis on specific execution scopes within applications, such as individual requests or specific trace spans.
This shift enables a more granular view of performance, enhancing the utility of profiles by linking them directly with traces for a comprehensive understanding of application behavior. As a result, engineering teams can more efficiently identify and address performance bottlenecks.
To learn more about Span Profiles, refer to Combining tracing and profiling for enhanced observability: Introducing Span Profiles.
Pyroscope integrates with distributed tracing systems supporting the OpenTelemetry standard. This integration lets you link traces with the profiling data and find resource usage for specific lines of code for your trace spans.
Note
- Only CPU profiling is supported at the moment.
- Because of how sampling profilers work, spans shorter than the sample interval may not be captured.
To use Span Profiles, you need to:
- Configure Pyroscope to send profiling data
- Configure a client-side package to link traces and profiles: Python
- Configure the Tempo data source in Grafana or Grafana Cloud to discover linked traces and profiles
Before you begin
Your applications must be instrumented for profiling and tracing before you can use span profiles.
- Profiling: Your application must be instrumented with Pyroscope’s Python instrumentation library. Refer to the Python guide for instructions.
- Tracing: Your application must be instrumented with OpenTelemetry traces. Refer to the OpenTelemetry guide for isntructions.
Configure the pyroscope-otel
package
To start collecting Span Profiles for your Python application, you need to include pyroscope-otel in your code.
This package provides a SpanProcessor
implementation, which connects the two telemetry signals (traces and profiles) together.
pip install pyroscope-otel
Next, create and register the PyroscopeSpanProcessor
:
# import span processor
from pyroscope-otel import PyroscopeSpanProcessor
# obtain a OpenTelemetry tracer provider
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
provider = TracerProvider()
# register the span processor
provider.add_span_processor(PyroscopeSpanProcessor())
# register the tracer provider
trace.set_tracer_provider(provider)
With the span processor registered, spans created automatically (for example, HTTP handlers) and manually will have profiling data associated with them.
View the span profiles in Grafana Tempo
To view the span profiles in Grafana Tempo, you need to have a Grafana instance running and a data source configured to link traces and profiles.
Refer to the data source configuration documentation to see how to configure the visualization to link traces with profiles.
Examples
Check out these demo applications for span profiles:
- Python example
- Other examples in multiple languages