API reference¶

Pipeline¶

class probixi.Probixi¶

Self-calibrating probabilistic peak finder and indexer.

Observe the detector noise on a slice of seed frames, calibrate a probabilistic detector from it (peak/noise scale kappa, peak prior, the pixel/radial/panel mean blend, a variance scale pinning the background to N(0, 1), and the matched-filter detection threshold), then stream frames to emit indexed, merge-ready results.

Parameters:

list_file (Union[str, Path]) – CrystFEL .lst list, .geom geometry and .cell unit-cell inputs. cell_file may be omitted for peak-only use
geometry_file (Union[str, Path]) – CrystFEL .lst list, .geom geometry and .cell unit-cell inputs. cell_file may be omitted for peak-only use
cell_file (Union[str, Path, None]) – CrystFEL .lst list, .geom geometry and .cell unit-cell inputs. cell_file may be omitted for peak-only use
noise_mode (Literal['per_frame', 'online']) – Whether the running noise model keeps updating per frame (“online”, tracking slow drift) or resets each frame.
warmup_frames (int) – Frames observed before the dead-pixel mask is committed.
finder_kappa (float) – Pre-calibration detector defaults; calibrate overrides kappa and the peak prior with learned values.
posterior_threshold (float) – Pre-calibration detector defaults; calibrate overrides kappa and the peak prior with learned values.
candidate_threshold (Optional[float]) – Pre-calibration detector defaults; calibrate overrides kappa and the peak prior with learned values.
matched_filter (bool) – Use the multi-scale matched filter (the recommended operating point).
mf_scales (tuple[float, ...]) – Matched-filter kernel scales and the fallback threshold (learned by calibrate when target_noise_peaks is set).
mf_threshold (float) – Matched-filter kernel scales and the fallback threshold (learned by calibrate when target_noise_peaks is set).
flux_variance (bool) – Replace the frozen variance floor with a learned photon-transfer curve so dim/low-flux shots are whitened against their own Poisson noise. Opt-in; intended for XFEL/SFX or jet-intensity-variable data.
seed (Optional[SeedConfig]) – Optional indexer configuration objects.
refine (Optional[RefineConfig]) – Optional indexer configuration objects.
cell_match (Optional[CellMatchConfig]) – Optional indexer configuration objects.
integrate (Optional[IntegrateConfig]) – Optional indexer configuration objects.
device (Optional[device]) – Torch device and frame dtype.
dtype (dtype) – Torch device and frame dtype.

threshold_calibration¶

The fitted matched-filter threshold result, set by calibrate.

Type:: ThresholdCalibration or None

property metadata: Metadata¶: Parsed run metadata (frames, geometry, cell).

property geometry: dict¶: Detector geometry dict (beam_center, clen, pixel_size, wavelength, …).

property target_cell: CellParams¶: Target unit cell that accepted orientations must match.

property noise: NoiseModel¶: The live noise model (built lazily on the first frame seen).

property finder: PeakFinder¶: The peak finder over the noise model (built lazily on the first frame).

frames(start=None, stop=None, batch_size=1, prefetch=2)¶

Stream frames off disk as device tensors, prefetching reads.

Parameters:

start (Optional[int]) – Half-open absolute frame range over the run; full run if omitted.
stop (Optional[int]) – Half-open absolute frame range over the run; full run if omitted.
batch_size (int) – Frames stacked per yielded tensor (1 yields single frames).
prefetch (int) – Batches read ahead on a background thread to overlap I/O.

Returns:

Frames on the configured device and dtype.

Return type:

Iterator[Tensor]

show_frame(frame, *, path, peaks=True, predictions=True, update_noise=False, **render_kwargs)¶

Render one frame with its detected peaks and indexed reflections.

Resolves frame to an absolute index, reads it off disk, runs peak finding (and indexing, when a target cell is configured) on that single frame, and writes an overlay image. Call calibrate() first for production-quality detection.

Parameters:

frame (Union[int, str, tuple]) – Absolute frame index, an "image_filename//event" string, or a (filename, event) pair.
path (Union[str, Path]) – Output image path.
peaks (bool) – Overlay detected peaks.
predictions (bool) – Overlay the predicted lattice when the frame indexes.
update_noise (bool) – Fold this frame into the live noise model (off by default so recalling a frame does not perturb pipeline state).
**render_kwargs – Forwarded to probixi.io.render_frame() (title, cmap, vmax_pct, …).

Returns:

The written image path.

Return type:

Path

fit_noise(frames)¶

Observe frames to warm the noise model (builds it on the first).

Returns:: Self, for chaining.
Return type:: Probixi

noise_diagnostics(path, *, frames=None, start=None, stop=None, batch_size=15, **kwargs)¶

Write a noise-model diagnostic GIF, driving the live model.

Streams frames (or a [start, stop) slice of the run) through the pipeline’s live noise model, snapshotting after every batch_size frames, and animates the running mean background, its radial profile, and the per-batch drift. The model is built lazily (with the geometry-aware bad-pixel mask) if it has not been seeded yet.

Because it drives the live model, every frame is folded into the running stats: run it before calibrate to watch warmup, or on a fresh Probixi to inspect a run’s drift. One mean-image snapshot is retained per batch, so prefer a bounded slice for long runs.

Parameters:

path (Union[str, Path]) – Output .gif path.
frames (Optional[Iterable[Tensor]]) – Explicit frames; defaults to the [start, stop) run slice.
start (Optional[int]) – Half-open frame range when frames is not given.
stop (Optional[int]) – Half-open frame range when frames is not given.
batch_size (int) – Frames folded in between animation snapshots.
**kwargs – Forwarded to the noise model’s diagnostics (fps, cmap, dpi, max_radius, figsize).

Returns:

The written GIF path.

Return type:

Path

calibrate(n_seed=32, seed_frames=None, eigen_modes=0, target_noise_peaks=5.0, threshold_opts=None, **opts)¶

Calibrate the detector on seed frames, then freeze the learned params.

Warms the noise model, then learns the peak/noise scale kappa and peak prior (EM mixture) and the mean blend + variance scale that pin the background z to N(0, 1). With target_noise_peaks set (the default), also calibrates the matched-filter threshold so a signal-free frame yields at most that many noise blobs, and installs it on the finder. With flux_variance enabled on the pipeline, also fits the photon-transfer curve.

Parameters:

n_seed (int) – Leading frames to calibrate on when seed_frames is not given.
seed_frames (Optional[Iterable[Tensor]]) – Explicit calibration frames; overrides n_seed.
eigen_modes (int) – If > 0, also fit this many low-rank background modes (XFEL/SFX).
target_noise_peaks (Optional[float]) – Matched-filter operating point (expected noise blobs per signal-free frame). None skips threshold calibration. 5 ~ mf_threshold 5.5.
threshold_opts (Optional[dict]) – Extra keyword arguments forwarded to calibrate_threshold.
**opts – Extra keyword arguments forwarded to calibrate_noise.

Returns:

The applied noise calibration.

Return type:

CalibrationResult

property beamstop_min_res: float | None¶: Effective low-resolution cutoff (A) excluding the beam center, or None.

peak_stream(frames, start_index=0, update_noise=True, estimate_scale=True)¶

Open a lazy stream of per-frame peak results.

Parameters:

frames (Iterable[Tensor]) – Frames to search; the first triggers lazy model construction.
start_index (int) – Absolute index assigned to the first frame (so results carry true run indices when starting partway through).
update_noise (bool) – Fold each frame into the running noise model as it passes.
estimate_scale (bool) – Estimate the per-frame fluence scale (when a calibration reference exists) so index_stream can attach it to each solution. Set False for peak-only use to skip the per-frame regression and avoid accumulating scales that are never consumed.

Returns:

Lazy, composable stream of PeakResult (torch-resident).

Return type:

PeakStream

index_stream(frames, batch_size=8, start_index=0, update_noise=True)¶

Open a lazy stream of indexing solutions over frames.

Runs the full pipeline per frame: detect peaks, lift to reciprocal space, seed and refine an orientation whose cell matches the target, then predict and integrate the lattice.

Parameters:

frames (Iterable[Tensor]) – Frames to process.
batch_size (int) – Frames per batched refinement pass.
start_index (int) – Absolute index assigned to the first frame.
update_noise (bool) – Fold each frame into the running noise model as it passes.

Returns:

Lazy stream of IndexResult, one per indexed frame.

Return type:

IndexStream

scale_stream(frames, start_index=0)¶

Infer the per-frame relative intensity scale, frame by frame.

Each frame is regressed against the background reference frozen at calibrate (so online drift-tracking does not contaminate the scale), yielding g_f and its uncertainty – the shot-to-shot fluence scale that dominates at an XFEL and that initializes downstream per-pattern scaling. Requires calibrate to have been run.

Parameters:

frames (Iterable[Tensor]) – Frames to scale (single 2-D frames).
start_index (int) – Absolute index assigned to the first frame.

Yields:

FrameScale – (frame_index, scale, sigma, offset) per frame.

__init__(list_file, geometry_file, cell_file=None, noise_mode='online', warmup_frames=16, finder_kappa=10.0, posterior_threshold=0.5, candidate_threshold=None, matched_filter=True, mf_scales=(1.0, 1.6, 2.4), mf_threshold=5.0, flux_variance=False, flux_var_floor=0.15, seed=None, refine=None, cell_match=None, integrate=None, device=None, dtype=torch.float32)¶

Streams and results¶

class probixi.indexer.IndexStream¶

Lazy, composable stream of per-frame :class:`IndexResult`s.

A torch-iterable produced by Indexer.index_stream(). Operators (map/filter/tap) compose lazily; terminals (collect, to_stream, count, …) drive the underlying generator once.

The stats funnel (frames/hits/indexed) is shared across composed operators, so counts stay live no matter how the stream is wrapped.

__init__(source, stats=None)¶

map(fn)¶

Return type:: IndexStream

filter(predicate)¶

Return type:: IndexStream

tap(fn)¶

Return type:: IndexStream

enrich_gate(alpha=0.001)¶

Drop solutions whose predicted spots are not backed by image signal.

Parameters:: alpha (float) – Max chance probability to accept a frame (a per-frame false-discovery level). Smaller is stricter.
Returns:: Filtered stream.
Return type:: IndexStream

to_stream(writer)¶

Return type:: int

to_db(path, *, geometry, cell=None, geometry_file=None, files=None, panel='0')¶

Drain the stream into a DuckDB database at path.

Convenience terminal wrapping DuckDBOffloader: opens the database, writes the geometry/panels/cell metadata tables, and streams every result into the frames/reflections/peaks tables.

Parameters:

path (str or Path) – Destination .duckdb file (overwritten if present).
geometry (dict) – Indexer geometry dict (beam_center, clen, …).
cell (Optional[CellParams]) – Target unit cell, written to the cell table.
geometry_file (str or Path, optional) – Geometry file whose text is stored verbatim.
files (Optional[dict]) – Loader file map; enables the non-indexed frame backfill.
panel (str) – Fallback panel name for out-of-panel peaks/reflections.

Returns:

Number of indexed frames written.

Return type:

int

collect()¶

Return type:: list[IndexResult]

count()¶

Return type:: int

for_each(fn)¶

Return type:: None

class probixi.indexer.IndexResult¶

Indexing solution for a single frame.

frame_index¶

Index of the source frame.

Type:: int

n_peaks¶

Number of peaks fed to the indexer.

Type:: int

n_indexed¶

Number of peaks explained by the solution.

Type:: int

rmsd¶

Root-mean-square q-residual over indexed peaks.

Type:: float

A¶

(3, 3) reciprocal-to-lab matrix, A = U @ B.

Type:: Tensor

U¶

(3, 3) crystal orientation (rotation).

Type:: Tensor

B¶

(3, 3) cell-only reciprocal basis.

Type:: Tensor

cell¶

Unit cell recovered from B.

Type:: CellParams

indexed_mask¶

(N,) bool mask of which peaks were indexed.

Type:: Tensor

hkl¶

(N, 3) integer Miller indices assigned to each peak.

Type:: Tensor

positions¶

(N, 2) observed peak (row, col) positions, aligned with hkl.

Type:: Tensor

intensities¶

(N,) integrated peak intensity, aligned with positions.

Type:: Tensor

sigmas¶

(N,) 1-sigma uncertainty on each intensity, aligned with positions.

Type:: Tensor

loss_history¶

Per-iteration refinement loss (excluded from repr).

Type:: Tensor

predicted_hkl¶

(M, 3) Miller indices of all reflections predicted to diffract on this frame, or None if prediction/integration was not run.

Type:: Tensor, optional

predicted_positions¶

(M, 2) predicted detector (row, col) positions.

Type:: Tensor, optional

predicted_intensities¶

(M,) box-integrated intensity per predicted reflection (snapped to the observed centroid where a peak coincides).

Type:: Tensor, optional

predicted_sigmas¶

(M,) 1-sigma uncertainty on each predicted intensity.

Type:: Tensor, optional

predicted_peak¶

(M,) max background-subtracted pixel in each box (spot height; the CrystFEL stream’s peak column).

Type:: Tensor, optional

predicted_background¶

(M,) mean per-pixel noise background under each box (the CrystFEL stream’s background column; informational only).

Type:: Tensor, optional

diffraction_limit¶

Per-crystal resolution limit (nm^-1) from integrated I/sigma; reflections beyond it are dropped and it is written as the stream’s diffraction_resolution_limit.

Type:: float, optional

enrichment¶

Bright-rate of predicted spots over the background bright-rate (see spot_enrichment); ~1 is a noise indexing, >>1 a real lattice.

Type:: float, optional

n_bright¶

Predicted spots landing on above-threshold image signal.

Type:: int, optional

enrich_p¶

Chance probability of this many bright predicted spots under the noise-null (spot_enrichment); the gate keeps frames with small enrich_p. Self-calibrating: no enrichment cut to choose.

Type:: float, optional

scale¶

Per-frame relative intensity scale (background fluence vs the calibration reference), if inferred; an initialization for downstream per-pattern scaling/merging.

Type:: float, optional

scale_sigma¶

1-sigma uncertainty on scale.

Type:: float, optional

__init__(frame_index, n_peaks, n_indexed, rmsd, A, U, B, cell, indexed_mask, hkl, positions, intensities, sigmas, loss_history, predicted_hkl=None, predicted_positions=None, predicted_intensities=None, predicted_sigmas=None, predicted_peak=None, predicted_background=None, diffraction_limit=None, enrichment=None, n_bright=None, enrich_p=None, scale=None, scale_sigma=None, mosaicity=None, profile_radius=None)¶

Output writers¶

class probixi.DuckDBOffloader¶

Write ``IndexResult``s to a DuckDB database.

A relational alternative to the CrystFEL .stream: run metadata lands in small geometry/panels/cell tables, every file-event becomes a row in frames (flagged indexed or not, with its per-frame statistics), and the integrated reflections and searched peaks are stored keyed by the frame’s frame_id().

Same interface as DataOffloader, so it drops into the same driver loop:

with DuckDBOffloader(out, geometry=geom, cell=cell, files=files) as off:
    stream.to_stream(off)

or, more directly, via to_db().

When files is supplied every file-event is enumerated, so frames that never indexed are recorded with indexed = FALSE and null statistics; the indexed rate is then AVG(indexed::INT) over frames. Without files only indexed frames are written.

Parameters:

path (Union[str, Path]) – Destination .duckdb file. Overwritten if it exists.
geometry (dict) – Indexer geometry (beam_center, clen, pixel_size, wavelength, adu_per_photon, and – when available – panels).
cell (Optional[CellParams]) – Target unit cell; written to the cell table and used for any symmetry labels.
geometry_file (Union[str, Path, None]) – Geometry file whose text is stored verbatim in geometry.geometry_file.
files (Optional[dict]) – Loader file map, used both to resolve a global frame index to its source file/event and to enumerate the non-indexed frames.
frame_range (Optional[tuple[int, int]]) – Half-open [lo, hi) global-frame-index range this writer is responsible for. When set, the non-indexed backfill is restricted to this range – required when only a sub-range is processed (--start / --stop) or when several writers each cover a disjoint block (the multi-GPU path), so the whole dataset is not marked non-indexed by every writer. None backfills every file-event.
indexer_name (str) – Recorded for provenance parity with the stream writer (unused in the DB).
panel (str) – Fallback panel name for peaks/reflections outside every geometry panel.

__init__(path, geometry, *, cell=None, geometry_file=None, files=None, frame_range=None, indexer_name='probixi', panel='0')¶

write(result)¶

Buffer one indexed frame (its stats, reflections and peaks).

Return type:: None

write_peaks(result)¶

Buffer one frame’s peak-search result (peaks-only; no indexing).

Records a frames row with indexed = FALSE and the peak count, and the searched peaks in the peaks table; the reflections table stays empty. result is a PeakResult.

Return type:: None

class probixi.DataOffloader¶

Write IndexResult``s to a CrystFEL-style ``.stream file.

Use as a context manager; the instance is itself the per-frame writer expected by IndexStream.to_stream (__call__ forwards to write):

with DataOffloader(out, geometry=geom, cell=cell, files=files) as off:
    stream.to_stream(off)

Please be aware that this functionality may be deprecated in the future.

Parameters:

path (Union[str, Path]) – Destination stream file.
geometry (dict) – Indexer geometry (beam_center, clen, pixel_size, wavelength, and – when available – the parsed panels).
cell (Optional[CellParams]) – Target unit cell; supplies the symmetry labels (lattice_type, centering, unique_axis). Per-crystal cell parameters are recovered from each solution’s orientation matrix.
geometry_file (Union[str, Path, None]) – Geometry file embedded verbatim in the stream header when present.
files (Optional[dict]) – Loader file map, used to resolve a global frame index back to its source file and within-file event number.
indexer_name (str) – Value written for Generated by. The per-crystal indexed_by is always fromfile so CrystFEL parses it as a known indexing method.
panel (str) – Fallback panel name when a peak falls outside every geometry panel (or when the geometry carries no panel definitions).

__init__(path, geometry, *, cell=None, geometry_file=None, files=None, indexer_name='probixi', panel='0')¶

write(result)¶

Append one indexed frame to the stream as a CrystFEL chunk.

Return type:: None

class probixi.PeakOffloader¶

Write peak-search results as CrystFEL-readable CXI files.

Exports peaks in the CXI layout so that indexamajig --peaks=cxi can re-index probixi’s peaks with its own engine. One .cxi is written per source HDF5 file; each external-links the raw image stack and stores the peak arrays under /entry_1/result_1 indexed by event.

Alongside the .cxi files it writes peaks.lst and a companion geometry file (the input geometry plus peak_list/peak_list_type), so the output directory is drop-in for indexamajig.

Use as a context manager; the instance is the per-frame writer expected by PeakStream.for_each (__call__ forwards to write()):

with PeakOffloader(out_dir, geometry=geom, geometry_file=gf,
                   files=meta.files) as off:
    for result in pipeline.peak_stream(frames):
        off.write(result)

Parameters:

path (str | Path) – Output directory (created if absent).
geometry_file (Union[str, Path, None]) – Input geometry file; copied verbatim with peak-list keys appended.
files (Optional[dict]) – Loader file map (filename -> H5Info), used to resolve frames to their source file/event and to external-link the raw image stack.

__init__(path, *, geometry_file=None, files=None)¶

Multi-GPU¶

probixi.run_data_parallel(list_file, geometry_file, cell_file, output, *, devices=None, start=None, stop=None, batch_size=8, seed_frames=32, target_noise_peaks=5.0, noise_mode='online', warmup_frames=16, flux_variance=False, flux_var_floor=0.15, panel='0', enrich_gate=False, enrich_alpha=0.001, threads_per_worker=None, keep_parts=False, quiet=False)¶

Run indexing across several devices on one node and merge the outputs.

Return type:: Path

Utilities¶

probixi.io.frame_id(filename, event)¶

Return type:: str