Leandro Santiago
November 26th 2024
Multiple embedded Linux systems “on the edge”, mostly written in C++ and logging to Systemd/Journal.
I want to get some metrics from the C++ written applications, using Prometheus, to feed Grafana dashboards.
I don’t want to embed a Prometheus client inside the application.
It’d require lots of changes which might affect especially the application performance and stability.
On the other hand, I can easily write prometheus exporters in Go without affecting the rest of the system.
The solution I found was to write an exporter that uses the application logs as input to build the metrics.
I cannot afford to send all the logs to a central place to be processed (logstash, etc.).
My systems often have unreliable and slow internet connection.
I cannot store metrics locally.
I use Prometheus Remote Write to send them to a remote server to aggregate the data.
I can though rely on the fact that Journal supports structured logging and attach arbitrary information to my log lines.
You could call them metadata or tags.
Grok Exporter is listed in the Prometheus integration page: https://github.com/fstab/grok_exporter
It seems to be a “tail -f” on steroids, not leveraging Journal power.
Although I haven’t benchmarked, I suspect it does not perform well under high number of different log patterns, as they need to be matched on every log line.
I might be mistaken, though!
Grok Exporter comes with a large library of grok patterns, widely used in the industry, but they are not useful for me, as I am working with a custom application.
I can survive with simple Regexps instead of the fancy patterns
grok offers.
Grok Exporter is quite big and mature.
--------------------------------------------------
Language Files Lines Blank Comment Code
--------------------------------------------------
Go 65 12027 1027 2858 8142
Markdown 7 1723 413 0 1310
Bourne Shell 3 549 76 52 421
YAML 3 121 10 42 69
C 1 52 7 20 25
C/C++ Header 1 20 2 13 5
--------------------------------------------------
Total 80 14492 1535 2985 9972
--------------------------------------------------Finally, it describes itself as “Export Prometheus metrics from arbitrary unstructured log data”, whereas my logs are “semi-structured”.
go-journald-exporter is open source (GPL3) and
available on https://gitlab.com/leandrosansilva/go-journald-exporterIt is young and small.
------------------------------------------------
Language Files Lines Blank Comment Code
------------------------------------------------
Go 4 617 126 7 484
Markdown 1 122 37 0 85
YAML 1 34 4 10 20
------------------------------------------------
Total 6 773 167 17 589
------------------------------------------------In my C++ application, for each log statement, I also create a tag, so the log statements look like this:
const auto time = 34;
const auto command = "blah";
LOG("command_time") << "It took " << time
<< "ms to run command" << command;Where “command_time” is application defined.
The log library will speak to Journal directly instead via stdout/stderr.
The main advantage of it is using the structured log properties of Journal.
My C++ codebase is quite old so I want to use the existing log statements.
Output of journalctl -o json
And, in its basic usage, with:
It will connect to the user session, if run as a normal user, or to
the system session, if executed as root or any user in the
systemd-journal group.
A better way is using a configuration file:
Simplest config.yml
It will do it to simply create metrics for each distinct member of
the fields list.
That was my start point on development.
# HELP go_journald_exporter_without_unit
# TYPE go_journald_exporter_without_unit counter
go_journald_exporter_without_unit{field="MY_LOG_CLASS",value="command_time"} 1By using a configuration file, you can enable more powerful metrics by using custom patterns.
histograms: # a list of histograms
- name: command_time # metric name
# all named groups will become extra labels in the metric.
# A unique label called `value`, that can be parsed
# by `strconv.ParseFloat()`, is mandatory on histograms!
pattern: ^It took (?P<value>\d+)ms to run command (?P<command>\w+)$
buckets:
type: exponential # linear or exponential?
min: 1
max: 100
count: 5
condition:
field: MY_LOG_CLASS
value: command_time# HELP go_journald_exporter_command_time
# TYPE go_journald_exporter_command_time histogram
go_journald_exporter_command_time_bucket{MY_LOG_CLASS="command_time",command="blah",le="1"} 0
go_journald_exporter_command_time_bucket{MY_LOG_CLASS="command_time",command="blah",le="3.1622776601683795"} 0
go_journald_exporter_command_time_bucket{MY_LOG_CLASS="command_time",command="blah",le="10.000000000000002"} 0
go_journald_exporter_command_time_bucket{MY_LOG_CLASS="command_time",command="blah",le="31.6227766016838"} 0
go_journald_exporter_command_time_bucket{MY_LOG_CLASS="command_time",command="blah",le="100.00000000000004"} 1
go_journald_exporter_command_time_bucket{MY_LOG_CLASS="command_time",command="blah",le="+Inf"} 1
go_journald_exporter_command_time_sum{MY_LOG_CLASS="command_time",command="blah"} 34
go_journald_exporter_command_time_count{MY_LOG_CLASS="command_time",command="blah"} 1counters: # a list of counters
- name: command_time # metric name
# all named groups will become extra labels
# in the metric.
# I should not use the `time` in the metric,
# otherwise the number of metrics might explode,
# causing serious performance issues!
pattern: ^It took (\d+)ms to run command (?P<command>\w+)$
condition:
field: MY_LOG_CLASS
value: command_time(Note the extra command label)
# HELP go_journald_exporter_command_time
# TYPE go_journald_exporter_command_time counter
go_journald_exporter_command_time{MY_LOG_CLASS="command_time",command="blah"} 1You can of course have multiple counters and histograms in the same configuration file.
If you want to disable the default automatically generated counters, add to your config file:
Requirements:
go-journald-exporter against
grok_exporter.Leandro Santiago
social@setefaces.org
(BTW, I am open to new opportunities. Talk to me!)