I’m trying to write Opentelemetry logs to Loki pushing them from my dotnet service directly to Loki with Opentelemetry packages, but ran into a problem. When I’m trying to get logs in grafana I get an error
could not write JSON response: 1:2: parse error: unexpected "=" in label set, expected identifier or "}"
After researching I realized that the problem was only with some of the log entries. Service have a metrics endpoint for prometheus. And when Prometheus collects metrics, the service writes a log entry like this:
RequestPath:/api/metrics RequestId:8000886a-0002-f400-b63f-84710c7967bb Microsoft.AspNetCore.Hosting.Diagnostics - Request finished HTTP/2 GET https://service/api/metrics - 200 - application/openmetrics-text;+version=1.0.0;+charset=utf-8 7.6416ms
I assume that loki use string “application/openmetrics-text;+version=1.0.0;+charset=utf-8” as common label and falling when I’m trying to get logs.
Does loki have some settings to do with this case? This label creates automatically by Opentelemetry package, so I don’t know what to do with it. I can disable Microsoft.AspNetCore.Hosting.Diagnostics logs but I think I need to resolve this issue on loki side or on the Otel package side.
I’m trying to write Opentelemetry logs to Loki pushing them from my dotnet service directly to Loki with Opentelemetry packages, but ran into a problem. When I’m trying to get logs in grafana I get an error
could not write JSON response: 1:2: parse error: unexpected "=" in label set, expected identifier or "}"
After researching I realized that the problem was only with some of the log entries. Service have a metrics endpoint for prometheus. And when Prometheus collects metrics, the service writes a log entry like this:
RequestPath:/api/metrics RequestId:8000886a-0002-f400-b63f-84710c7967bb Microsoft.AspNetCore.Hosting.Diagnostics - Request finished HTTP/2 GET https://service/api/metrics - 200 - application/openmetrics-text;+version=1.0.0;+charset=utf-8 7.6416ms
I assume that loki use string “application/openmetrics-text;+version=1.0.0;+charset=utf-8” as common label and falling when I’m trying to get logs.
Does loki have some settings to do with this case? This label creates automatically by Opentelemetry package, so I don’t know what to do with it. I can disable Microsoft.AspNetCore.Hosting.Diagnostics logs but I think I need to resolve this issue on loki side or on the Otel package side.
There seems to be an issue with empty attributes, and potentially some characters. See this post for the case with empty attributes: https://community.grafana.com/t/otlp-ingestion-from-otel-collector-to-loki-seems-to-corrupt-my-data/140588/3
It worked for me just now when using OpenTelemetry Collector, note the "attributes/empty" processor. It essentially removes the Empty() attributes from the data before it's sent to Loki.
The error itself seems to be happening on Grafana side.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
debug:
otlphttp/logs:
endpoint: "http://loki:3100/otlp"
tls:
insecure: true
otlp:
endpoint: "tempo:4317"
tls:
insecure: true
sending_queue:
num_consumers: 4
queue_size: 100
retry_on_failure:
enabled: true
prometheus:
endpoint: '0.0.0.0:8889'
processors:
batch:
memory_limiter:
# 80% of maximum memory up to 2G
limit_mib: 400
# 25% of limit up to 2G
spike_limit_mib: 100
check_interval: 5s
attributes/empty:
actions:
- key: "empty"
# match empty string
pattern: ^\s*$
action: delete
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp,debug]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus,debug]
logs:
receivers: [otlp]
processors: [memory_limiter, attributes/empty, batch]
exporters: [debug,otlphttp/logs]
I was struggling with the same error and I couldn't find issue in my .NET Web API with Open Telemetry. This problem was really strange for me, since I've not implemented anything custom regarding logging. I was only using Nuget packages for OpenTelemetry and the problem was occuring.
The problem was when I've added the following NuGet packages:
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.11.1" />
There is a bug in version 1.11.*
This problem is explained on GitHub: https://github.com/open-telemetry/opentelemetry-dotnet/issues/6138#issuecomment-2665815098
According to this, downgrade of OpenTelemetry Nuget Packages form 1.11 to 1.10 resolves the problem at this moment.
The solution is:
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.10.0" />
Of course it will be fixed in the future - see GitHub link and related issues.