Monitoring: Graphite, Grafana, alerts and notifications

Four different things watch the experiment, and they are easy to confuse. In short:

Tool	What it is	What it does here
Graphite	A time-series database with an HTTP API	Stores every number anyone measures: beam current, terminal voltage, board rates. WebDAQ writes to it and reads from it.
Grafana	A dashboard and alerting server	Draws those numbers as plots on the wall screen, and raises alerts when one leaves its range.
WebDAQ Stats page	Part of this application	Shows the current value of the metrics you selected, and copies them into each run’s `stats.csv`.
Telegram / Zulip / ELOG	Messaging and the logbook	Tell a person that something happened, and keep the record.

1. Why a time-series database at all

The accelerator, the target, the picoammeter and the digitizers each produce numbers continuously. Storing them inside the DAQ would mean every consumer — plots, alerts, run files, shift reports — reading a different file in a different format, and nothing at all being available from another machine.

Graphite solves exactly that. A metric is a dotted name with a value and a timestamp:

accelerator.terminal_voltage   3.412   1785191160
daq.rate1                      12874   1785191160

Anything can write one (carbon, the ingestion side, listens on TCP 2003), and anything can read it back over HTTP. The DAQ does not have to know who wants the number, and the analysis does not have to know which program produced it.

At LUNA the server is lunaserver. Its metric tree looks like this:

accelerator.charge              ancillary.rates
accelerator.column_current      daq.rate1 … rate3
accelerator.extraction_voltage  Germanium…
accelerator.terminal_voltage    12c12c.tetram…

What WebDAQ pushes

Board rates — events, pile-up, saturation, lost counts and file write rate per board and channel, pushed by the acquisition on the sampling cadence during a run — once a second unless it was changed on the Data rates page.
Beam current — every sample from the TetrAMM or the RBD 9103.

Both go to Carbon (port 2003 by default) of the one Graphite server set in Settings → Monitoring; the Stats page and Grafana read them back over HTTP (port 80 or 8080, depending on the install). The beam current has no address of its own any more: it follows that setting, under its usual names (tetramm.*, rbd9103.current).

Settings → Monitoring

One section holds both servers: Graphite (host, web port, Carbon port, metric prefix) and Grafana (URL, service-account token), each with a light saying whether it answers. One Save stores both.

Detect looks for the servers so nobody has to guess a port. Within a few seconds it tries the configured Graphite host, the Grafana host, the beam current’s old Graphite host (see below), localhost and lunaserver:

Graphite’s web API on ports 80, 8080, 8000, 8081 and 2080 (and the configured one) — it counts only if /metrics/find?query=*&format=json answers with a JSON list;
Carbon on 2003 (and the configured Carbon port) — an open port is all plaintext Carbon shows;
Grafana on 3000, 3001, 3002 and 3030 (and the configured one) — it counts only if /api/health answers with Grafana’s JSON, so WebDAQ’s own frontend on port 3000 is never mistaken for it.

Each find has a Use button that fills the fields; nothing is saved until you press Save. When nothing answers, start the local stack with scripts/monitoring.sh start (below) and detect again.

Note

Before this, conf/current.json had its own graphite_host/graphite_port for the beam current. If they name a different server from the Monitoring setting, the server logs a warning once at startup and leaves them in the file, but the Monitoring setting is what is used. The Monitoring section and the Current module section say where the current used to go.

Graphite time syntax

Ranges are relative: -10s, -30min, -6h, -1d. The minute unit is min — -30m is rejected with “Invalid offset unit ‘m’”. It is the mistake everyone makes once.

No Graphite or Grafana in the lab?

scripts/monitoring.sh start runs both on the DAQ PC in Docker, with their data in ~/webdaq-monitoring and Grafana already connected to Graphite:

scripts/monitoring.sh start                     # web 8080, Carbon 2003, Grafana 3001
scripts/monitoring.sh start --graphite-port 8081 --carbon-port 2013 --grafana-port 3002
scripts/monitoring.sh status | stop | remove    # remove keeps the data

Then press Detect in Settings → Monitoring and Use both finds (or type localhost, the web and Carbon ports above, and http://localhost:3001), and Save. Grafana’s first login is admin/admin.

2. The Stats page

Settings a shift crew actually needs: which numbers to watch, what to call them, and what they are measured in.

The light next to the title is the state of the Graphite server. Green means it answered a query for its metric tree; red means it did not, and every metric on the page will read N/A until it does. Grey is the check in flight. The light distinguishes a dead server from metrics that simply have no data — without it they look identical.

Adding a metric. Press Add metric. It opens on Active in the last hour: every metric Graphite has had a value for in the last hour, with its latest value, how long ago it came and a short trend — the DAQ’s own rates and Graphite’s bookkeeping left out. Type in the filter to narrow it down. For anything else, Browse all walks the whole tree (open accelerator, pick terminal_voltage) or searches it by name. Then give it:

a name — what the card and the CSV column will be called (“Terminal Voltage”), and
a unit — kV, uA, counts/s. Optional, but it is what makes a column readable a year later.

Both can be changed afterwards with the pencil on the card.

On each card: the latest value with its unit, the time of that reading, and a 30-minute trend line so you can see whether it is rising, falling or flat. The eye hides a metric without deleting it; the bin removes it.

The Graphite server card at the bottom of the page holds three things: the host, the port of the render API that WebDAQ reads, and the metric prefix for this experiment. The prefix names the campaign rather than a board — set it to ancillary.rates.12c12c and that campaign owns the subtree, so two campaigns can never merge series. WebDAQ’s own rates land under <prefix>.bo_<board>.ch_<channel>.totalRate, keyed by the board’s register id rather than its model name, so two identical boards stay apart. A live run picks up a change at its next stats interval.

Note which Graphite address is which: this page configures the render API, read over HTTP, used for every plot and every monitored value. The address in Settings → Current Module is Carbon’s plaintext ingest (usually port 2003), which measured values are pushed to. Mixing them up produces a page where everything reads N/A while the data is arriving perfectly well.

3. The run’s stats file

While a run with data saving is in progress, every enabled metric is sampled once a second and written to data/run<N>/stats.csv:

# LUNA DAQ statistics
# Run number: 1276
# Start time: 2026-07-28T01:09:00
# Format: CSV. The first column is the elapsed acquisition time in seconds;
# the rest are the metrics below, in this order. Missing samples are 0.
#
# Metric: Terminal Voltage | unit: kV | source: accelerator.terminal_voltage
# Metric: Board 0 rate | unit: counts/s | source: daq.rate1
#
Time [s],Terminal Voltage [kV],Board 0 rate [counts/s]
0.001,3.412,12874
1.005,3.411,12903

Points worth knowing:

It is plain CSV. The metadata is in # comment lines that every reader can skip: pandas.read_csv('stats.csv', comment='#') gives you the table directly.
Each column heading carries the name and unit you chose, and every metric is listed above with its Graphite path, so a column can always be traced back to its source.
The set of columns is frozen when the run starts. Adding a metric mid-run does not shift the columns under the rows already written; it appears in the next run.
A missing sample is written as 0. Graphite has ingestion lag, so a gap does not mean the instrument failed.

4. Grafana

Grafana reads the same Graphite data and does two things WebDAQ deliberately does not: long-term plots on the control-room screen, and alerting.

Use Grafana for: “how did the terminal voltage behave over the last three days”, “is the board rate drifting”, “email/notify me when the current drops below X”. Use the WebDAQ Stats page for: “what is the beam current right now, and put it in this run’s file”.

Alerts tied to the run

The Alerts page lists the alert rules from Grafana and lets you mark rules as auto-managed. An auto-managed rule is:

activated when a run starts, and
silenced when the run stops.

That solves the standard annoyance: a “beam current too low” rule that is right during a run and pure noise while you are setting up. Mark the rules that only make sense during data taking; leave the rest under Grafana’s own control.

The WebDAQ server talks to Grafana on the page’s behalf, so the address must be reachable from the server machine. Set it in Settings → Monitoring (default http://lunaserver:3000), together with a service-account token if your Grafana requires authentication; the token needs permission to read and edit alert rules. Both are stored in conf/grafana_settings.json and the token is never sent to the browser. Test alert-rule access reports how many alert rules the server can see.

5. Alerts: Telegram and Zulip

Settings → Notifications, in two halves: where a message can go, and what is worth sending one.

Destinations

	What to enter	Stored in
Telegram	Bot token (from `@BotFather`) and chat ID. The chat ID shows up in `https://api.telegram.org/bot<token>/getUpdates` after one message to the bot.	`conf/telegram_settings.json`
Zulip	Organisation URL, bot email, bot API key, stream and topic. Every alert lands in that one topic, so they stay together in a readable thread.	`conf/zulip_settings.json`

Test message sends one immediately, so a wrong token is found now rather than during the failure it was meant to report. Both secrets stay on the DAQ machine and are never sent to the browser; anyone holding one can post as the bot, so use an account created for this.

Alerts

Each alert is a rule: what to watch, when it counts as wrong, and which destinations carry it. A rule can go to Telegram, to Zulip, or to both, so a board failure can wake the whole shift while a beam-current dip only reaches one group.

Alert	Fires when	Settings
Board failure	A board sets its FAIL flag during a run.	— (once per board per run)
Board stopped producing data	A board reads no new data block for N seconds during a run, whether or not it raised the FAIL flag.	seconds
Beam current	The current read by the current monitor stays below (or above) a threshold.	threshold, direction, how long it must hold, only during a run
Graphite metric	A monitored value — terminal voltage, pressure, anything in Graphite — stays outside its range. Add one rule per metric.	metric path, threshold, direction, how long, how often to check

Two rules keep the messages worth reading:

One message per episode. A rule fires when its condition starts and stays quiet until it clears. A current sitting just below its threshold is one message, not one every two seconds. When it clears, a short “back in range” message follows, which can be switched off.
Unknown is not wrong. A Graphite server that cannot be reached, or a current monitor that is disconnected, leaves every rule exactly as it was. A hole in the monitoring never invents an alert, and never silently clears a real one. A rule in that state is marked nothing to measure, so it is not mistaken for a rule that is happy.
A flap is one message. The same subject of the same rule is not announced twice within a minute, so a value sitting exactly on its threshold cannot fill the chat or roll the night out of the history.
An alert that reached nobody says so. If every destination of a rule is off or unconfigured the message is still recorded, the bell turns amber, and the history row is badged reached nobody.

Rules live in conf/alerts.json. A DAQ machine that has never had rules starts with the one WebDAQ always had: board failures to Telegram. A board failure is written to the history even when no rule is configured for it — it is the event this whole system exists for, and a shift reading the history in the morning has to find it there.

A rule only works while the server is watching: the badge at the top of the panel says so, and offers Start watching when it is not (a backend started without main.py has no watcher). A rule whose destinations are all switched off is marked reaches nobody rather than looking armed.

A monitored beam current is read with a freshness horizon measured from the metric itself — the observed publication cadence and lag — rather than the fixed 30 s window a picoammeter gets. A value published every 10 s legitimately reads back tens of seconds old, and used to be reported as a dead readout.

Auto-restart, configured in the Acquisition setup card on the Overview, is what a board-failure message refers to: the run is stopped and started again after the delay, so a night shift does not lose hours to a board that hiccupped.

6. The Board Health tab

Board Health, a tab on the Dashboard next to Overview, shows what each board has actually done during this run, which is the question that follows “is it failed?”:

Data blocks read from the board, and events decoded from them, each with its current rate.
Bytes read from the board and handed to the file writer.
FAIL blocks, dropped blocks (the write queue was full — that is lost data) and read errors, highlighted when they are not zero.
A board that has read nothing for ten seconds is marked, which is the same condition the “board stopped producing data” alert watches, with your own delay.

Every counter belongs to the run: they start at zero when a run starts and are gone when it ends. A dash means the installed CaenDAQ build does not report that counter.

7. What happened while nobody was looking

A message is gone once it is read, so every alert and every recovery is also written on the server, in conf/alert_log.json (the most recent 500, kept across restarts).

The bell in the header carries the unread count. It turns amber when something reached nobody — a wrong token, a network outage, both destinations off.
Troubleshoot in the navigation is the full history: time, what happened, which board or value, which run, and which destinations actually received it. Filter by problems, recoveries or actions.

This is the page to open after a night shift, or after finding the DAQ stopped: it answers “what happened at 03:00” without anyone having to have been awake.

Recovery

The same page carries the fixes, each showing whether that part is healthy right now:

Action	Use it when
Reopen the boards	a board reads as disconnected after a link glitch (not during a run)
Reset the acquisition	a run failed or the boards are wedged (not during a run)
Reconnect the current monitor	the beam current reads as disconnected, or stops updating
Restart the statistics	`stats.csv` stopped being written mid-run. What was already collected is kept as `stats.csv.part1`
Re-check Graphite	after a Graphite outage, to clear the “unavailable” state it leaves behind

Nothing here runs by itself, and an action that would cost data is refused while a run is in progress and says so. Auto-restart remains the one thing that acts unattended, and only when switched on.

8. What to check when a number is missing

Symptom	Look at
Every metric reads `N/A`, light is red	Graphite host and port in Settings; `curl http://<host>/metrics/find?query=*` from the DAQ machine.
One metric reads `N/A`, others fine	Its path. Re-add it with the browser — a renamed metric keeps the old path in your configuration.
Light is green but the card is empty	Nothing is writing that metric. Check the source instrument, not WebDAQ.
`stats.csv` has all-zero columns	Same as above: the metrics were unreachable during that run. The header still tells you which they were.
Grafana plots but WebDAQ does not	Different port. Grafana often reads Graphite on 80 while the Stats page is configured for 8080, or the other way round.