Log, Transform, Monitor: Journals, CI and serverless databases
Hybrid Meetup #46 took place 2024-11-26 19:00 at Basislager and we had a nice mix of people, both online and offline, with different backgrounds in addition to two great presentations.
Go journald exporter
Leandro motivated the need for a tool that joins a legacy C++ application with modern event monitoring tool like Prometheus, through systemd, specifically journald: go-journald-exporter.
journald has a native protocol that allows processes to write directly to the log; go-journald-exporter will use go-systemd bindings to seek to the end of the journal, listen for specific messages and transform them to prometheus metrics, which the tool exposes through an HTTP endpoint (because Prometheus is pull-based).
This way, a lightweight tool of a few hundred lines of code, can offer better visibility into a fleet of legacy applications running low resource environments, where go is viable and a performant option, too.
More details in the slides.
From GitHub workflow JSON to postgres to grafana
Fedor turned GitHub CI run results (taken from the API, JSON format) into much more comprehensible visual aggregations with a small tool: gh-workflow-stats-action.
It reads data off the GitHub API, populates a postgres database and uses grafana postgres data source to aggregate and visualize metrics from CI runs.
For larger projects, GitHub API rate limiting needs to be accounted for, but it’s possible to turn CI output from a larger project into database tables over a weekend.
As a nice recursive twist, the tool used neon.tech to create a postgres database for storing the workflow run results in the cloud with one click: you get a DSN and you are ready to go. Neon splits storage and compute components of postgres and can unlock lots of useful features this way, like scaling and branching. With branching, the WAL gets written forward differently for different branches, which is useful for testing and staging environments.
On the side: neon.tech is hiring!
Thanks for the tools!
SLOC of the tools presented: 484 (go-journald-exporter) and 622 (gh-workflow-stats-action) — both help to turn streams of diagnostics into suitable formats for modern monitoring tools and are adaptable in other projects as well.
Thanks again to Leandro and Fedor for the inspiring talks and demos.
Misc
- kubernetes test grid: https://testgrid.k8s.io/sig-release
- another CI tool (written in Go): concourse
- more CI tools: https://argoproj.github.io/cd/, https://fluxcd.io/
- CI/CD Observability using OpenTelemetry
- go build infrastructure is open source, too: build, using a custom framework to model the different workflow steps leading to release artifacts
- if you are into the classic web, check out the tildeverse, e.g. https://tilde.club, https://tilde.town/, …
