Server architecture
This chapter is for whoever has to change the server. It describes what the backend is made of, which module owns which piece of state, and why the awkward parts are the way they are. How WebDAQ works covers the same system from the outside; read that first if you want the behaviour rather than the structure, and Directory structure for the file-by-file inventory.
1. The shape of the server
One process holds everything: the Flask application, the open digitizers, the acquisition, the spectra, the picoammeter, the background threads.
HTTP (JWT) in process
browser ─────────────► Flask routes ──────────► services ──────────► caendaq ──► boards
mobile (11 blueprints) (state, threads) (C++) .caendat
script │
├──► picoammeter / Graphite metric
├──► Carbon (rates, current)
└──► SQLite (run metadata)
The layering is conventional — routes validate and translate, services hold state and run threads, app/utils/ talks to hardware and to other servers — with one consequence worth stating: a service is a process-wide singleton reached through a get_*() function, because there is exactly one set of boards and one run. get_daq_manager(), get_caen_acquisition(), get_spy_manager(), get_alert_manager(), get_alert_log(), get_histogram_config(), get_grafana_client(), get_elog_client(), get_zulip_notifier(), get_board_scanner(), get_conversion_manager() all behave this way. Constructing a second one is a bug, not an option.
There is no acquisition process, no container and no socket between the boards and the file. caendaq is a C++/pybind11 library (server/native/caendaq) imported into this process; it opens the boards, reads them, writes .caendat and accumulates the online spectra in C++. Earlier releases ran XDAQ in Docker with a TCP “spy” socket; the vocabulary survives in module names (spy.py, spy_manager.py, /spy/status) and nowhere else.
Every relative path is relative to the process’s working directory. conf/, calib/, data/, cache/ and app.db belong to the experiment the server was started in, not to server/. This is the single most common source of “my configuration disappeared”.
2. Starting up: what main.py guarantees
main.py is short but almost none of it is optional, because each piece prevents a specific failure that was seen in a control room.
Step |
Why it is there |
|---|---|
|
Creating the app opens the CAEN boards over USB and the picoammeter socket. If the guard ran in |
|
Runs the Alembic migrations from the directory next to |
Signal handlers |
SIGINT and SIGTERM run |
|
|
|
Polls |
|
Starts the alert watcher here rather than in |
|
Waitress. Ten threads, every interface. |
ensure_schema_current is deliberately not called at import time: flask --app main.py db ... imports this module, and upgrading there would fire in the middle of the very commands meant to manage the database.
server.py is a separate Flask CLI app object for management commands such as flask --app server create-user. It does not serve anything.
3. The application factory
app/__init__.py is create_app(): config, SQLAlchemy, Flask-Migrate, JWT, CORS, eleven blueprints, one after_request hook, and experiment.set_flask_app(app).
That last line matters. Background threads — the auto-restart, the run-state listeners, the charge persistence — touch the database from outside a request, so they need an app to open a context against. The reference is handed to app/routes/experiment.py, and main.py sets it again for the case where the app is created there.
The after_request hook gzips large JSON and text responses. One energy spectrum is 32768 bins of JSON, a PSD map is the same again, and the dashboard fetches that for every channel on display every couple of seconds. The payload is mostly zeros and short repeated numbers, so it compresses by one to two orders of magnitude — the difference between a usable and an unusable connection over the VPN. Responses under 4 kB are left alone, compression level is 5 rather than 9 because this runs on the machine that is also taking the data, and Vary: Accept-Encoding is added so caches behave.
Sessions do not expire (JWT_ACCESS_TOKEN_EXPIRES = False). A control-room browser is logged in once and left open for a campaign, and an expiry mid-run logged the operator out with no warning. The trade-off is explicit: there is no token blocklist, so rotating JWT_SECRET_KEY is the only revocation there is. Acceptable for an instrument on a controlled network.
4. The HTTP surface
Blueprints carry no URL prefix; each route spells its own path. There are 187 reachable endpoints.
Blueprint |
Paths |
What it serves |
|---|---|---|
|
|
Issues and checks the JWT. |
|
|
Run start and stop, run metadata, notes and flags, adding and removing boards, data-saving settings, board status and diagnostics, auto-restart, and the superseded Telegram settings routes. |
|
|
Register reads and writes, polarity and channel enables, the board scan, online tuning, board info. |
|
|
Spectra and PSD maps, waveform traces and their per-board switch, the dashboard and ROI configuration, batch ROI integrals. |
|
|
The beam-current module: type, settings, connection, readings, history, accumulated and lifetime charge. |
|
|
The Graphite server and metric tree, the monitored metric list, the per-run |
|
|
Completed runs: summary list, detail, current log, |
|
|
ELOG settings, entries, attributes and fields, attachments, and the run draft. |
|
|
Grafana settings and alert rules. |
|
|
Transports and their tests, alert rules, the event log, the watcher, and the recovery actions. |
|
|
Per-channel energy calibration. |
app/routes/faraday.py is dead code. It defines a blueprint with /faraday/open and /faraday/close, and create_app() does not register it, so both return 404. Nothing calls them.
Older paths are kept working on purpose where the frontend has moved on — the Telegram settings under /experiment/ are the clearest case. A bookmark or a shift script pointing at them does not have to be rewritten.
Conventions across the surface: everything but /register and /login carries @jwt_required_custom (just flask_jwt_extended’s jwt_required(), re-exported from app/utils/jwt_utils.py so the identity casting lives in one place), bodies and replies are JSON, and the identity in the token is the user id as a string, cast back by get_current_user().
The status codes and the message strings are part of the contract, not incidental: the frontend decides what to show from them, and tests/test_routes.py exists to pin exactly that. A route that changes a 404 into a 400, or rewords a message the UI matches on, will break a page without breaking anything that looks like an API. Several routes also return why something did not happen rather than a bare failure — online tuning’s {saved, written, via, reason} is the pattern to copy, because the browser has to tell the operator whether the change reached the board or only the configuration file.
5. The database
Two models, in app/models/.
User is id, username, email and password hash. RunMetadata is the run record: run number, start and end time, notes, accumulated charge, target name, terminal and probe voltage, run type, a flag of good/unknown/bad, the owning user, and three provenance columns — board_info (the per-board snapshot from CaenAcquisition.board_info_all), software_versions (WebDAQ, caendaq, Python, platform) and sync_mode (daisy-chain or independent).
SQLite is the live source of truth. app/services/run_metadata_snapshot.py mirrors each row into data/run<N>/metadata.json after every commit, which is what makes a copied or archived run directory self-describing — the database stays behind on the acquisition machine, the run directory travels. Anything that mutates run metadata calls sync_run_metadata_file() after its commit, and there are several such places (run start and stop, notes, flags, the charge, the auto-restart), so adding another one means remembering this.
The snapshot is versioned (METADATA_SCHEMA_VERSION) and carries more than the row. It walks the run directory and records every file with its size and a role — raw-data, root, beam-current, statistics, board-configuration — grouped by role as well as listed, so a reader a year later does not have to infer what run_7_0000.caendat is from its extension. It is written to a temporary file in the same directory and moved into place, so a half-written metadata.json is never observable.
Schema changes go through Flask-Migrate (server/migrations/), and ensure_schema_current applies them at every start.
6. Acquisition: who owns the boards
Three modules, and the hand-off between them is the part to understand.
Module |
Owns |
|---|---|
|
The DAQ state — run number, saving, size limits, the board list, the run-state listeners, board-failure monitoring and auto-restart. Persists to |
|
The persistent probe connections, through the ctypes wrapper in |
|
The single in-process |
A board can only be open once, so starting a run moves it. start_run calls daq_mgr.release_digitizers() to close the probe connections, then caen_acq.configure(...) and .start(). Stopping does the reverse with reacquire_digitizers(), which retries three times a second apart — a CAEN link that was closed a moment ago is not always ready to reopen — and logs exactly which boards stayed shut, because those are the ones that will read “Disconnected” on the dashboard until Reset acquisition is used.
write_board_register() hides the hand-off from its callers: during a run the write goes through caendaq, between runs through the probe connection, and the caller never asks which.
conf/settings.json is written to a temporary file and moved into place under a lock. It is written from request threads and from background threads, and two interleaved writes once left invalid JSON — which the next start read as “no boards configured” and replaced with an empty board list.
Two more services sit beside these. services/board_scanner.py probes USB, optical and VME links in a background thread (a full optical sweep is 32 probes and a VME range can be hundreds, far too long for one request) and deliberately bypasses the container’s retry-three-times open, which is right for a board you know is there and ruinous for a sweep where most probes are expected to miss. services/sync_chain.py reads back what each board in a daisy chain actually needs — the master’s software trigger on TRG-OUT, the followers’ TRG-IN to TRG-OUT propagation, the start mode — and reports the differences rather than assuming them.
7. Board failures, auto-restart and diagnostics
No board is polled while a run is in progress. caendaq owns the digitizers then, so asking a board a question directly would conflict with the readout. What the monitoring thread in daq_manager.py watches instead is caendaq’s own board-FAIL signal: bit 26 of the aggregate header, counted per board as the data is decoded. The thread wakes once a second and acts only on a board’s first transition to failed, once per board per run.
The count of failing aggregates is all there is to report, and the code says so. There is no failure register behind the flag — typically it means a full internal buffer or a lost link — so the message reads “Board FAIL flag (37 data blocks)”. It used to be decoded as if it were register bits, producing labels like “Board Error (0x2)” that meant nothing to anyone.
A first failure sends the board-failure notification and, when auto-restart is enabled, schedules the restart after the configured delay (minimum five seconds, stored in conf/settings.json so it survives a restart). The delay is interruptible: stopping the monitoring cancels it.
perform_auto_restart() in app/routes/experiment.py is registered as the DAQ manager’s restart callback, and it does by hand what the Stop and Start buttons do together — which is why it is long. It notes which side channels were live, flags the failed run bad with an explanatory note, closes its current recording and its stats.csv, increments the run number, waits, then points both side channels at the new run before the acquisition starts, configures and starts caendaq, re-enables the spy and the monitoring, and records the new run with the failed one’s parameters carried over. If the new acquisition fails to start it closes the recordings it just opened, because no run follows them.
/experiment/board_diagnostics and /experiment/get_board_status expose the counters caendaq keeps per board for the run — blocks read, bytes read and written, blocks the write queue refused, CAEN read errors, events decoded, aggregates with the FAIL flag — which is what the dashboard’s Board Health tab shows and what the stall rule in section 13 watches. They are cumulative for the run and empty outside one, and a counter that the installed caendaq build does not provide reads None rather than failing the whole request.
8. Online spectra: what “spy” means now
caendaq accumulates per-channel spectra, PSD maps and waveform traces continuously in C++ while it decodes. app/utils/spy.py builds a ROOT object from the current snapshot on demand: no socket, no background collection thread, no second copy of the data stream, and no global channel numbering to get wrong. A spectrum request is addressed by (board index, channel), where the board index is the one caendaq handed out when the run was configured.
A request travels like this:
GET /histograms/<board>/<channel>[/<type>]
→ spy_manager.get_histogram() pick the type from the board's DPP if not given
→ ru_spy.histogram() numpy array from caendaq → ROOT TH1F (TH2F for PSD)
← apply calib/<name>_<id>.cal to the x-axis, then Rebin
→ TBufferJSON.ConvertToJSON()
← rendered in the browser by JSROOT
ReadoutUnitSpy.start() and .stop() only flip a flag — there is nothing left to spawn. What they do is make /spy/status honest about whether a run is live.
Two details exist to keep the operator informed rather than puzzled. There is always a histogram to draw, and its title says why it is empty: “No data yet — start a run”, “This board is not part of the current run”, “No waveform — switch waveforms on for this board”. A blank canvas or a stale plot tells you nothing, and an empty plot labelled “Default Histogram” tells you less. And the statistics box is switched off everywhere, because it covers the part of the spectrum people look at and its mean and RMS over raw ADC bin indices mean nothing.
The calibration is cached by file mtime, so editing a .cal file takes effect without a restart and without re-parsing it on every poll. 1-D spectra are rebinned by SpyManager; the 2-D PSD map is rebinned by the route.
BuilderUnitSpy in the same module is a placeholder: the single-process DAQ does not produce coincidence histograms yet, and it returns an empty one that says so.
9. Online tuning
Online tuning is not a service. It is one route plus an allowlist.
POST /digitizer/<id>/setting/<setting> {"value": 1234, "online": true}
GET /digitizer/<id>/online_registers
POST without online is the plain configuration edit: the value lands in conf/<name>_<id>.json and applies at the next run. With online it is also written to the board, so the effect shows up in the trace and the spectrum immediately. The configuration is updated either way, even when the board write is refused or fails — so what the tuner shows and what the next run will apply can never drift apart. The response says what happened: saved, written, via (run or probe), and a reason when the write did not go through.
app/utils/safe_registers.py decides what may be written. It is an allowlist, not a denylist: per-channel DPP parameters — thresholds, gates, shaping and trapezoid times, DC offset, fine gain, the algorithm-control registers that select the probe — are listed by their channel-0 address (0x10??, with the live address normalised to that form), and anything unrecognised is left to the configuration file. A denylist would mean that a register added to a dump later quietly became writable mid-run.
The registers kept out are the ones that describe the shape of the readout: record length, pre-trigger, input dynamic range, events per aggregate, aggregate and board configuration, the channel enable mask, acquisition control. Changing one of those under a live acquisition produces buffers the decoder does not expect — a corrupted run at best, a stopped board at worst. They are named in the module anyway, so a refusal can say why rather than “not in the list”.
GET /digitizer/<id>/online_registers returns the board’s firmware and the safe register names, so the tuner can mark which fields are tunable before anything is changed instead of discovering it one write at a time.
10. The histogram dashboard and the ROIs
services/histogram_config.py owns conf/histograms.json: which histograms exist, their names, their ROIs, their zooms and the dashboard settings.
This used to live in the browser and be persisted to the web server’s frontend/cache/, which put it in the one process that knows nothing about boards, runs or the run directory. Three things followed, and moving it fixed all three: the DAQ server could not record a run’s ROIs because it could not see them, every browser kept its own dashboard, and the definitions were declared three times in TypeScript and drifted. On first start the store seeds itself once from frontend/cache/histogram-configs.json and friends if they are still there; WEBDAQ_FRONTEND_CACHE pointed at an empty directory switches that off. Writes are atomic and serialised, because the dashboard saves on every ROI edit and every zoom.
services/roi_analysis.py does two jobs with that configuration. compute_integrals() answers the whole dashboard in one call behind GET /histograms/roi_integrals: the spectrum is read once per board and channel and every ROI on it is integrated from that one copy. The browser used to ask for one integral per ROI per tick, so sixteen histograms with three ROIs each meant forty-eight round trips and forty-eight spectrum reads every few seconds. Results are {gross, background, net} rather than a bare number.
write_run_snapshot() drops roi.json into the run directory when a run ends. It is wired in through register_run_hook(), called at import of app/routes/histograms.py, which registers a listener on the DAQ manager’s run state — so it happens whether or not a browser is still open. POST /histograms/run/<n>/roi_snapshot writes one on demand.
11. Beam current and charge
Four interchangeable modules present the same interface to app/routes/current.py: utils/tetramm.py (TCP), utils/rbd9103.py (serial), utils/graphite_current.py (a monitored accelerator metric, scaled into µA) and utils/mock_current.py (test mode). Nothing downstream can tell which is in use, which is the point — the per-run current.txt, the readout and the accumulated charge all work the same way.
The charge integration follows the DAQ’s own run state, not the client. _on_run_state_changed() in current.py is registered with daq_mgr.add_run_state_listener(), so a closed browser, a run stopped by the auto-restart and a run that saves no data all leave the figure correct. It also decides which run the charge belongs to, at the moment the run starts: leaving that to /current/start — which the dashboard calls only when data is being saved — once let a run started with saving off keep the previous run’s number and overwrite that run’s charge on stop. A 12-hour run was left reading 100 µC. A run that saves nothing now claims no run number and stores nothing. The lifetime total keeps counting either way, because it answers how much beam the target has seen.
On stop the run’s charge is written into its metadata row from the same listener, and the log is closed — a stop issued while the monitor was unreachable used to leave current.txt open, and later readings were appended to a finished run.
If the server is restarted mid-run, the module picks the integration back up at import rather than sitting there switched off.
12. Graphite: two endpoints, two directions
The one thing to get right here: the ingest and the read API are different servers on different ports, configured in different files.
Configured in |
Used for |
|
|---|---|---|
Carbon plaintext ingest, port 2003 |
|
What the picoammeter controllers and caendaq push. Nothing can be read back from it. |
Graphite render API, HTTP |
|
What the Stats page, the current history and the alert rules read. |
utils/carbon.py owns the push side, and it is guarded rather than plain: one connection per push for all metrics (Carbon’s protocol takes any number of newline-terminated lines, so a four-channel picoammeter costs one handshake instead of four), a one-second handshake deadline, and a circuit breaker that opens after three consecutive failures and stays open for a minute. The loops that call it also own the buffers the web layer reads and the charge integration the run metadata depends on, so a dead collector must never hold anything up — and the breaker is the part that matters. A host refusing connections fails in microseconds, but a host that is powered off or behind a firewall dropping SYNs never answers at all, and then every push waits out the connect timeout. At a 0.5 s sampling interval that is not a hiccup, it is a permanent stall. Board rates are pushed by caendaq itself, configured through CaenAcquisition.set_graphite() from conf/stats.json.
utils/graphite.py is the render-API client. services/graphite_reader.py wraps it for everything that plots history and rebuilds the client whenever conf/stats.json changes, so editing the server in the UI takes effect without a restart. Taking the address from the file the Stats page configures means an operator sets it once; the current plot used to query port 2003 and report “no data”. services/stats_manager.py holds the monitored metric list and the loop that samples them into data/run<N>/stats.csv, with the column set frozen at run start.
13. Alerts, notifications and recovery
services/alerting.py holds the rules in conf/alerts.json and one watcher thread that evaluates them on a two-second tick. A rule is a type (board_failure, stalled_buffers, beam_current, graphite_metric), a threshold, how long the condition must hold, and which transports carry it.
Two properties keep the alerts worth reading:
One message per episode. A rule fires when its condition becomes true and stays quiet until it clears. A beam current sitting just below its threshold is one message, not one per tick.
Unknown is not false. A Graphite server that cannot be reached, or a picoammeter whose samples have gone stale, yields no value at all — and a gap in the monitoring path must neither invent an alert nor clear a real one.
Every measurement is taken through a small set of callables (AlertSources), so the whole engine is testable without hardware, without Graphite and without a run. The watching itself is deliberately dull: one thread, one pass per tick, and Graphite rules throttle themselves further with their own poll_seconds.
Rules are normalised and validated on the way in, and a bad one is rejected with AlertConfigError rather than stored and silently never fired. Each type carries its own defaults and the parameters the UI needs to render it, which is where to add a new kind of alert: a RULE_DEFAULTS entry, a source callable, and an _evaluate_* branch. Most types also take during_run_only, because a beam current below threshold means nothing when nobody is measuring.
services/alert_log.py writes every alert and every recovery to conf/alert_log.json, capped at the most recent 500 events. A Telegram message is gone once it has been read, and a shift arriving at 08:00 needs to be able to ask what the DAQ did at 03:00 — including across the restart that is often part of the story. Nothing in this module may raise into its caller: failing to record an alert must never stop it being sent.
services/telegram_notifier.py and services/zulip_notifier.py are the two transports. Their credentials live in conf/telegram_settings.json and conf/zulip_settings.json and are returned to the browser masked.
services/recovery.py names the handful of things that actually recover a session — reopen the board connections, reset the acquisition, reconnect the current monitor, restart the statistics collection, re-check Graphite — reports for each whether it currently looks needed, and runs one on request. Nothing in it runs by itself. The operator presses it, so a recovery never happens behind their back mid-measurement; the one exception in WebDAQ is auto-restart, which is opt-in and says so in the logbook entry it writes.
The watcher is normally started by main.py. POST /notifications/watcher exists because a backend launched another way — a bare flask run, some other WSGI server — would leave the rules configured and never evaluated, and the settings page that reports this has to be able to fix it.
14. Grafana, ELOG and the run report
services/grafana_client.py exposes only Grafana’s alert-rule API: list, read, update, silence, activate. The address and the service-account token stay in conf/grafana_settings.json and never reach the browser, and the narrow surface is deliberate because the token may carry far more rights than the Alerts page needs. Which rules are auto-managed is remembered in the frontend, which silences them on stop and activates them on start.
services/elog_client.py reads and writes a PSI ELOG logbook through py_elog, imported lazily so a server without the package still starts and reports a clear reason. Authentication is one shared service account in conf/elog_settings.json; attribution is separate, with ELOG’s Author attribute filled from the WebDAQ user who is posting, so entries still carry a real name.
services/run_report.py turns a finished run into a draft entry — the database row, metadata.json, the logged current, the accelerator readings and the ROIs, assembled into body text and attributes. The result is a draft the operator edits before posting; nothing here is authoritative. Attributes are matched by keyword rather than exact name, because every LUNA logbook names its fields slightly differently and the composer learns the real field list from the logbook itself.
15. Completed runs and conversion
services/run_data.py is the read side: it turns data/run<N>/ into a file inventory, the current as a time series, an integrated charge, and a RUReader conversion to ROOT. It only reads, and writes ROOT output; the acquisition path is untouched.
ConversionManager runs RUReader as a subprocess, one conversion per run at a time, writing run<N>.root next to the data it was made from. A large run takes minutes, so the request only starts the job and the dashboard polls status.
Two details are there because of the binary rather than the data. The installed RUReader is not necessarily built from the pinned submodule, and an older one exits with “Unknown option” instead of converting, so capabilities() asks it once with --help and only sends the flags it advertises — which also lets the UI offer exactly those options. And a RUReader built against a different ROOT fails with a linker error that says nothing about the real cause, so _annotate_log() prepends the explanation and the fix to the log the dashboard shows. RUREADER_BIN and RUREADER_ENV_PREFIX override the binary and the environment it is run in.
app/routes/data.py joins all that with what the database knows, so a run list can show what is actually on disk — whether there is data, whether there is a current log, whether it has been converted, how many files and how many bytes — rather than only what was intended.
16. A run, start to finish
Starting a run is partly orchestrated by the client, and it is worth knowing which parts.
The browser (components/run-control-buttons.tsx) pushes the acquisition settings, then starts the side channels — /current/start/<run> and /stats/run/<run>/start, only when data is being saved — then calls /experiment/start_run, then writes the run metadata and activates the auto-managed Grafana alerts. The side channels may not stop the run. They were once awaited bare, so a picoammeter that was connected but not sampling answered 4xx, threw, and the DAQ was never started at all: the operator saw “Failed to start the run” and lost beam time to a monitor. They are now caught, and the toast says which one did not start.
POST /experiment/start_run then does this, in order:
Refuse if a run is already in progress or no boards are configured.
prepare_run_start()— createdata/, and when saving,data/run<N>/, and copy each board’sconf/<name>_<id>.jsoninto it. Only the board configurations are copied. Calibration files are not —calib/*.calis read live from the working directory, so a run directory carries the settings it was taken with but not the calibration it was displayed with.release_digitizers(), thencaen_acq.configure(boards, run, out_dir, max_file_bytes, write=save)andcaen_acq.start(). A failure at either step reacquires the probe connections before returning, so a failed start does not leave the boards closed.set_running_state(True)— which stamps a timezone-aware ISO 8601 start time (without the offset the browser parsed it ambiguously and the run timer could start negative) and notifies the run-state listeners.Enable the on-demand spy and start board-failure monitoring.
Record the run in the database, if saving.
set_save_data(false) is a real mode: configure(write=False) runs decode-only, so spectra, rates and charge all work and nothing is written.
POST /experiment/stop_run unwinds it — stop monitoring, stop the spy, stop the acquisition, reacquire the digitizers, write the end time, increment the run number if data was saved, and set_running_state(False). The listeners do the rest: the charge is persisted and the current log closed by current.py, roi.json is written by roi_analysis.py. Metadata failures are logged and never fail the stop, but they are logged — a silent pass there once hid a missing migration.
The run-state listeners are the extension point for anything that must follow a run. Register with daq_mgr.add_run_state_listener(fn) rather than adding a step to the route: a listener fires for a run stopped by the auto-restart and for a run whose browser has gone away, and a route step does not.
17. Configuration files
All of them are JSON in conf/, relative to the working directory, written by the server and safe to read or edit by hand when it is stopped.
File |
Owner |
Holds |
|---|---|---|
|
|
Run number, saving, size limits, the board list, auto-restart and its delay. |
|
|
One board’s full register dump. The single source of truth for the board: the CAEN dashboard edits it, caendaq applies it, and Acquisition Control (0x8100) in it decides how a run starts. |
|
|
Histograms, ROIs, zooms, dashboard settings. |
|
|
The current module in use, the Carbon ingest address, the lifetime charge. |
|
|
Per-device settings for the two picoammeters. |
|
|
The Graphite render API, the metric subtree, the monitored metrics and their units, the sampling cadence. |
|
|
The alert rules. |
|
|
The last 500 alert and recovery events. |
|
the two notifiers |
Bot credentials. |
|
|
Addresses and service tokens. |
A missing file is not an error: each owner writes a sensible default on first use. An unreadable conf/settings.json is moved aside with a timestamp rather than replaced, so the board configuration in it can be repaired instead of lost.
18. Test mode and the test suite
TEST_FLAG=True substitutes the simulated digitizer and the simulated picoammeter. It is not a stub layer bolted on the side: utils/board_defaults.py fabricates a register dump with the same register set, the same JSON shape and plausible values, so the CAEN dashboard, the synchronisation controls and online tuning all have something real to act on; utils/mock_current.py writes exactly the same current.txt format and produces a deliberately beam-like signal — a slow decay with ripple, drift and the occasional dip — so plots and charge integrals look like something an operator would recognise. Database and metadata paths are unchanged.
server/tests/ is plain unittest, no extra packages, no hardware, no Graphite and no ELOG; HTTP responses are faked where a server would be needed. tests/run_tests.sh runs the suite from a scratch working directory so a real setup’s conf/ and data/ are never touched — the exception is test_routes.py, which drives the app the way the frontend does and therefore reads the directory it is started from.
The convention for new tests is to test the rule rather than the implementation: “a structural register is refused mid-run” survives a refactor, “the structural list has 16 entries” does not.
19. The frontend as a client
The web interface has no privileged path into the DAQ. It is one API client among several, which is why a browser crash cannot take a run with it, and it is worth knowing the three places where it does more than draw.
src/lib/api.ts is the single Axios instance, with NEXT_PUBLIC_API_URL as its base and two interceptors. The request interceptor attaches the JWT from the auth store. The response interceptor catches a dead session centrally — 401 from flask-jwt-extended for missing or expired, 422 for malformed — clears the token and redirects to the login page with the current path as next. Without it, every polling component would start getting 401s and, because most of them swallow errors to survive a brief server hiccup, the page would simply stop updating and look as though the DAQ had died. Calls to /login and /register are exempt, since a 401 there is the login form’s business.
src/app/api/server-control/route.ts runs in the Next.js process, not the browser, and is how Start an Experiment works: it resolves the conda environment’s prefix (asking conda rather than trusting conda run -n luna python, which on some versions resolves python from the inherited PATH), runs scripts/check_db.sh against the chosen working directory’s app.db, creates the default user if there is none, and spawns server/main.py with that directory as its cwd and its own pid as LUNA_LAUNCHER_PID. Every step has a hard timeout so a hung conda or sqlite call cannot wedge the request, and a graceful exit of the launcher takes the backend with it — a SIGKILL is covered from the other side by the watchdog in section 2.
Two things in that preflight are worth knowing when a start behaves oddly. It copies migration revisions the repository has and the measurement directory does not, adding only — anything already there wins, so a project whose history diverged keeps it. And it probes the database with plain sqlite3 reads first, skipping both the conda-wrapped flask db upgrade and the create-user script when there is nothing to do, because importing the app is the single most expensive step of a start and on a warm boot neither is needed.
src/app/api/cache/route.ts keeps the state that genuinely belongs to a browser rather than to the DAQ: the visualisation channel selection, the waveform view configuration, the list of recent working directories. The histogram dashboard used to live here too, and section 10 explains why it does not any more.
Everything else is conventional: Zustand stores for client state, React Query for polling, and JSROOT for drawing the TBufferJSON payloads the histogram endpoints return.
20. Working in this codebase
A short list of things that are easy to get wrong here.
Doing this |
Do it this way |
|---|---|
Adding an endpoint |
Put it in the blueprint that owns the subject, decorate with |
Needing a service |
Call its |
Needing to react to a run |
Register a run-state listener, not a step in the start or stop route. |
Touching the database off a request |
Open a context on the app reference handed to |
Opening a file |
Relative paths are relative to the working directory. Anything that must follow the installation has to be anchored to |
Writing a config file |
Temporary file plus |
A side effect at run start |
It must not be able to fail the run. Catch it, let the run proceed, and say which part did not start. |
Adding an alert |
A |
Reporting a failure |
Say what broke and what to press. The alert log, the recovery actions and the empty-histogram titles all exist because “something went wrong” is not a message anyone can act on at 03:00. |
Changing the schema |
|
Changing C++ |
|