Documentation Grafana Cloud Instrument and send data Logs Troubleshoot write issues

Troubleshoot Cloud Logs write issues

Unfortunately, occasionally logs are rejected and write failures occur. This can be for a number of reasons; determining the reason for write failures can be challenging.

Because Loki has detailed information about write failures and can emit them through logs, Cloud Logs users can see write errors in a self-service fashion. However, there are limitations to what errors Loki can log. The following are example of errors that cannot be diagnosed with this feature:

• If the push request is too big (for example, 1GB). Grafana cloud would reject the request before it reaches Loki, so it would never be logged.
• Promtail internal issues, for example, Promtail running out of memory. Promtail issues wouldn’t be discovered because the request would never reach the Grafana Cloud to be logged by Loki.

To troubleshoot logs write issues:

1. Log in to your GrafanaCloud instance.
2. Click the menu icon, then click the Explore (compass) icon.
3. From the menu in the top left of the Explore page, select the grafanacloud-[instanceName]-usage-insights datasource.
4. Query the datasource for write issues.

• Use the Query Builder on the Builder tab.
• Or, click the Code tab and enter your query in the field. Example queries:

Copy

 {instance_type="logs"} |= "path=write"
 {instance_type="logs"} |= "push request failed"

The query returns log messages, which include information to help determine why the log write failed. The log messages include which stream discarded logs, for what reason, and provides valid values.

The following list contains the errors that can be discovered using this feature and the message displayed in the logs:

• Entry too far behind: “entry too far behind, oldest acceptable timestamp is: %s”
• Entry too far in the future: “entry for stream ‘%s’ has timestamp too new: %v”
• Entry too old: “entry for stream ‘%s’ has timestamp too old: %v, oldest acceptable timestamp is: %v”
• Duplicate label names: “stream ‘%s’ has duplicate label name: ‘%s’”
• Invalid labels: “Error parsing labels ‘%s’ with error: %s”
• Label name too long: “stream ‘%s’ has label name too long: ‘%s’”
• Label value too long: “stream ‘%s’ has label value too long: ‘%s’”
• Line too long: “Max entry size ‘%d’ bytes exceeded for stream ‘%s’ while adding an entry with length ‘%d’ bytes”
• Max label names per series: “entry for stream ‘%s’ has %d label names; limit %d”
• Missing labels: “error at least one label pair is required per stream”
• Rate limited: “Ingestion rate limit exceeded for user %s (limit: %d bytes/sec) while attempting to ingest ‘%d’ lines totaling ‘%d’ bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased”
• Stream limit: “Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased”