# Run data and conversion What a run leaves on disk, what each file is for, and how to turn it into something an analysis can read. The run directory is the unit of data: a copy of `data/run1276/` is a complete, self-describing measurement, with the board configuration it was taken with and a record of the beam that produced it. --- ## 1. The run directory Everything a run writes goes into `data/run/`, relative to the working directory the server was started from. The directory is created when the run starts, and only when **Save data** is on — a run with saving off still decodes, fills spectra and publishes rates, it just writes nothing. | File | Written | What it is | |---|---|---| | `run__0000.caendat` | during | The data. One unified binary stream for every board, with the same header layout the XDAQ-era files had, so the same converter reads old and new runs. | | `run__0001.caendat`, … | during | The next parts, when a size limit is set. The header is written once, in the first file. | | `_.json` | at start | The board's complete register dump, copied out of `conf/` (with time stamp and energy recording switched on if a dump had them off). This is what makes the run reproducible: thresholds, gates, trapezoid parameters, start mode, as they were. | | `_.cal` | at start | The board's energy calibration (offset and slope per channel), copied out of `calib/`. | | `calib_history.jsonl` | on change | One line per calibration changed during the run: time, board, channel, new and previous coefficients, and the peaks it came from. | | `coincidences.json` | at start | The coincidence gates and per-channel time offsets the online spectra used. | | `histograms.json` | at start | The histogram ranges, rebinning and regions of interest in force. | | `register_history.jsonl` | on change | One line per register written to a board during the run (Tuner, online settings): time, board, address and value. | | `coincidences_history.jsonl` | on change | The coincidence gates and time offsets each time they were changed (or aligned) during the run. | | `settings_history.jsonl` | on change | The same for an SIS3316: its complete settings each time they were changed during the run. A new histogram divider also restarts that board's spectra. | | `current.txt` | during | The beam current, one row per sample, with the time relative to the start of the log. The first two lines name the start time and the columns. | | `stats.csv` | during | The accelerator readings you selected on the Stats page, one row per sampling tick. | | `stats.csv.part1`, … | during | Earlier rows of the same run, kept aside when the statistics were restarted mid-run (see [Troubleshooting](troubleshooting.md)). | | `metadata.json` | at every change | The run's record — times, target, voltages, charge, flag, notes, the boards and the software versions — written again whenever anything changes. | | `roi.json` | at stop | Every region of interest defined at the time, with its counts. | | `run.root` | on request | The converted file, produced by RUReader from every `.caendat` in the directory. Besides the data it holds the run record (`RunMetadata`, and each field under `Run/`), every configuration file above verbatim (`RunFiles/`), the beam current and the charge (`Current/`) and the monitored metrics (`Stats/`): the file alone says how its data was taken. | **The database is the live record; `metadata.json` is the copy that travels.** SQLite (`app.db`) is what the Logbook reads and writes. After every change the same information is written into the run directory, so a directory copied to another machine still knows its target, its charge and the firmware that took it. --- ## 2. The data file `.caendat` is the raw CAEN aggregate stream with a small header in front of it. The header names the boards — sampling period, time-tag period, DPP version, channel count and register id for each — and then the aggregates follow as the boards produced them. There is no trailer, which is deliberate: a run killed by a power cut is still readable up to the point where it stopped. Two consequences worth knowing on shift: * **One file holds every board.** The boards are interleaved in one stream rather than written side by side, which is what makes a synchronised run a single object with one time origin. * **Nothing is ever overwritten.** Starting a run in a directory that already has data writes `_1`, `_2`, … rather than replacing what is there, and the dashboard asks before reusing a run number at all. --- ## 3. Converting to ROOT Conversion is offline and does not touch the run: it reads the `.caendat` files and writes `run.root` next to them, using RUReader. Open **Logbook → the run → Convert**. The tab shows what the installed RUReader can do, because the options are read from the converter itself rather than assumed; a build that does not offer a flag does not show it. Start the conversion and leave the page if you like — it runs on the server, in the background, and the tab reports progress and the last lines of the converter's log when it finishes. A conversion that fails leaves the `.caendat` files untouched. The log tail is the first thing to read; a truncated final aggregate from an interrupted run is the usual cause, and `--ignore-fail` (passed automatically when the installed converter supports it) is what gets the rest of the data out. --- ## 4. Checking a run that was interrupted ```bash cd server python scripts/check_run_integrity.py data/run1276 ``` The script walks the file structure without converting anything and says whether the header, the board definitions and the aggregate chain are consistent. Use it after a crash, a full disk or a power cut, before deciding whether a run is worth converting. --- ## 5. Reading the run back in the Logbook **Logbook → the run** opens the run's own page, which reads the directory rather than only the database: * **Overview** — times, duration, target, voltages, run type, accumulated charge, the flag, and the file inventory with sizes. * **Notes** — the free-text record, editable after the fact. * **Beam Current** — `current.txt` plotted, with the charge integrated over the whole run (start to stop, see [current and charge](current-and-charge.md)), from every sample rather than the drawn points. * **Stats** — one plot per metric from `stats.csv`, each labelled with the alias and unit you gave it on the Stats page. A final row still being written is dropped rather than plotted, because half a number drawn as a value looks like a cliff that never happened. * **Boards** — the register dumps copied in at run start, so you can see what the run was taken with without opening the files. * **Convert** — section 3 above. --- ## 6. Housekeeping **Run numbers** live in `conf/settings.json` and advance on Stop, only when saving. Set the number by hand on the dashboard when a campaign needs to continue an older series — the dashboard warns before reusing a directory that already holds data. **Disk.** A rate of a few MB/s is normal for a busy chain; the dashboard shows the current write rate, and **Board Health** shows the bytes each board has handed to the writer. If the write queue cannot keep up, the blocks it refuses are counted as *Dropped* on Board Health — that counter is lost data, and it is the one number on that page that should always be zero.