Troubleshooting applications using the Java agent
If Application Observability doesn’t list your services, these are some of the most common causes:
No traffic
Make a few requests to the service to send data to Grafana Cloud.
Be patient
It can take up to 5 minutes until the data is visible in Application Observability.
Look for errors
Look for errors - either on the console or in Docker or Kubernetes logs, using Application Observability logs doesn’t make sense in this case.
If there are errors sending telemetry data, one of the parameters is usually wrong.
A 5xx
response code means that there’s something wrong with the Grafana Cloud OTLP Gateway.
Log agent data
If there are no errors in the logs, enable logging to check the application is sending data.
If the application isn’t sending data, look for the -javaagent
command line parameter to check if you loaded the Java agent.
Log exporter data
If the application is sending data to an data collector, Grafana Alloy or OpenTelemetry Collector, log the data from the data collector and check there are no errors forwarding the telemetry data to Grafana Cloud.
Java agent logging
To inspect the internals of the Java agent, enable the Java agent’s internal debug logging:
export OTEL_JAVAAGENT_DEBUG=true
Note
Only enable the Java agent debug logging when needed. The logs are extremely verbose and impact application performance.
Disable Java agent
Lastly, disable the Java agent to check the application runs successfully without instrumentation:
export OTEL_JAVAAGENT_ENABLED=false
OTLP debug logging
You can configure the OTLP debug logging can via environment variables, previous versions use logging
instead of console
:
Configuration | Result |
---|---|
OTEL_SPAN_EXPORTER=otlp,console | Log all spans, and also send spans to Loki. |
OTEL_METRICS_EXPORTER=otlp,console | Log all metrics, and also send metrics to Loki. |
OTEL_LOGS_EXPORTER=otlp,console | Log all logs. |
Note
You should disable debug logging when it isn’t needed as it produces many logs.
This produce log output like this:
Spans - look for LoggingSpanExporter
:
INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'WebController.withSpan' : 2337335133908c9ce6e0dfc7bda74d7c 8bfef4eaac83e8cb INTERNAL [tracer: io.opentelemetry.opentelemetry-extension-annotations-1.0:1.32.0-alpha] AttributesMap{data={thread.id=32, code.namespace=io.opentelemetry.smoketest.springboot.controller.WebController, code.function=withSpan, thread.name=http-nio-8080-exec-1}, capacity=128, totalAddedValues=4}
Metrics - look for LoggingMetricExporter
:
INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.21.0, attributes={container.id="048b9982e0b98cdc5579334bb1decc157ed1ebc23f391ebe306898898ec32fa4", host.arch="amd64", host.name="048b9982e0b9", os.description="Linux 6.2.0-39-generic", os.type="linux", process.command_line="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java -javaagent:/opentelemetry-javaagent.jar -Dgrafana.otel.use-tested-instrumentations=true io.opentelemetry.smoketest.springboot.SpringbootApplication", process.executable.path="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java", process.pid=1, process.runtime.description="BellSoft OpenJDK 64-Bit Server VM 25.312-b07", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_312-b07", service.instance.id="8231ca95-e9aa-474a-bd98-88349a9942ad", service.name="unknown_service:java", telemetry.auto.version="1.32.0", telemetry.distro.name="grafana-opentelemetry-java", telemetry.distro.version="0.32.0-beta.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.32.0-alpha, schemaUrl=null, attributes={}}, name=jvm.cpu.count, description=Number of processors available to the Java virtual machine., unit={cpu}, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1704964347622000000, epochNanos=1704964351627000000, attributes={}, value=12, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}
Logs - look for [scopeInfo:
and duplicated log body. The second line is for reference to see that it also contains HTTP request received
:
10:12:34.031 [docker-java-stream-636643381] INFO c.g.extensions.smoketest.SmokeTest - STDOUT: 2024-01-11T09:12:34.03Z INFO 'HTTP request received' : 2337335133908c9ce6e0dfc7bda74d7c 50d689015fd0a33c [scopeInfo: io.opentelemetry.smoketest.springboot.controller.WebController:] {thread.id=32, thread.name="http-nio-8080-exec-1"}
10:12:34.031 [docker-java-stream-636643381] INFO c.g.extensions.smoketest.SmokeTest - STDOUT: INFO [http-nio-8080-exec-1] io.opentelemetry.smoketest.springboot.controller.WebController: HTTP request received trace_id=