Detection Efficiency¶
This guide explains how pycWB computes detection efficiency—the probability of recovering an injected signal as a function of its parameters—and how efficiency curves are used to characterize search sensitivity.
Overview¶
Detection efficiency measures the fraction of simulated signals recovered by the search pipeline. pycWB reports efficiency as a function of:
Signal amplitude: the injected \(h_{rss}\) (
sim_hrss). Only fixed-\(h_{rss}\) populations are supported; target-SNR or SNR-scaled injections are rejected by the efficiency actions.Waveform: one curve per injected waveform name (
sim_name).
The key metrics are hrss50 and hrss90—the root-sum-squared strain amplitude at which 50% and 90% of injections are recovered, respectively.
Computing Efficiency¶
Efficiency computation follows these steps (as implemented by the
postprocess.plot_efficiency actions):
Match simulations:
match_simulationswithhow: rightwrites one row per injection (uniquesim_sim_idx). Recovered injections carry the matched trigger’s columns (non-nullid); missed injections keep null trigger columns and stay in the denominator.Score simulations: ranking values are joined to the matched rows by trigger
id, either from a pre-scored SIM catalog (scored_file) or by scoring withmodel_file. Injections without a trigger, or whose trigger was removed by the prediction cuts, have no score.Apply the IFAR threshold: an injection is detected if it is recovered and the inclusive empirical background tail of its score satisfies
\[\frac{N_{bkg}(\rho_{bkg} \geq \rho)}{T_{bkg}} \le \frac{1}{\text{IFAR}}\]where the background values are the
ranking_parcolumn ofbkg_catalog(the scored FAR background), \(T_{bkg}\) islivetimein seconds and IFAR is parsed fromifar.Count injections per waveform and per injected amplitude: for each
sim_nameand each distinctsim_hrssvalue (no amplitude binning):\[\epsilon(h_{rss}) = \frac{N_{detected}(h_{rss})}{N_{injected}(h_{rss})}\]where \(N_{injected}\) counts unique
sim_sim_idx. Withexclude_vetoed: true, injections flaggedsim_vetoed_cat0,sim_vetoed_cat1,sim_vetoed_cat2orsim_across_segmentsare removed from numerator and denominator; with the defaultfalsethey remain and count as missed unless detected.The plotted error bars are the binomial standard error:
\[\sigma_\epsilon = \sqrt{\frac{\epsilon (1 - \epsilon)}{N_{injected}}}\]Fit efficiency curve: a cWB sigmoid (
logNfit) is fitted to the per-amplitude efficiencies of each waveform (see hrss50 and hrss90).
Efficiency Workflow Steps¶
Matching and scoring simulations (evaluate_efficiency writes the scored
SIM catalog used as scored_file; its threshold only affects the
returned summary, the fraction of scored rows with xgb_prob at or above
it):
- id: sim_eval_match
name: Match SIM Evaluation Catalog
action: postprocess.matching.match_simulations
inputs:
catalog_file: ${paths.sim_eval_catalog}
simulation_file: ${paths.sim_eval_simulations}
args:
how: right # one row per simulation for efficiency
window_buffer: 0.0
outputs:
output_file: tmp://sim_eval_matched_right.parquet
- id: sim_efficiency_score
name: Score SIM Evaluation Triggers
action: postprocess.evaluate.evaluate_efficiency
inputs:
catalog_file: ${paths.sim_eval_catalog}
model_file: ${paths.model_file}
config_file: ${paths.config_file}
args:
threshold: 0.5
outputs:
output_file: ${paths.sim_eval_scored}
Computing efficiency vs. \(h_{rss}\) per waveform at a fixed IFAR
(bkg_catalog is the scored_catalog written by evaluate_far_rho, and
livetime is the livetime of that FAR background):
- id: waveform_hrss_curves_100yr
name: Sensitivity Curves At 100-Year IFAR
action: postprocess.plot_efficiency.compute_efficiency_vs_hrss_by_waveform
inputs:
sim_catalog: ${paths.sim_eval_catalog}
matched_file: "@sim_eval_match.matched_file"
bkg_catalog: ${paths.bkg_far_scored}
model_file: ${paths.model_file}
config_file: ${paths.config_file}
args:
livetime: "@bkg_split.far.livetime.seconds"
ranking_par: rhor
scored_file: ${paths.sim_eval_scored}
ifar: 100yr
use_unique_sim: true
exclude_vetoed: false
outputs:
output_file: ${paths.output_dir}/simulations/efficiency_vs_hrss_by_waveform_100yr.png
fit_parameters_file: ${paths.output_dir}/simulations/fit_parameters_by_waveform_100yr.csv
hrss50 and hrss90¶
The per-waveform actions
(compute_efficiency_vs_hrss_by_waveform()
and
compute_hrss50_by_waveform_csv())
fit the cWB sigmoid
pycwb.modules.statistics.sigmoid_fit.logNfit() to the points
\((\log_{10} h_{rss}, \epsilon)\) with Minuit
(pycwb.modules.statistics.sigmoid_fit.fit()). With
\(y = \pm(\log_{10} h_{rss} - \log_{10} h_{rss}^{50})\) (the sign set by
the orientation flag), the fitted curve is
and \(\epsilon = 0.5\) at \(y = 0\). When \(\beta_+ y > 1\) the code uses \(s = \sigma \beta_+ e\) and \(y = 1\). Both orientation flags are tried and the fit with the lower \(\chi^2\) is kept; the \(\chi^2\) is unweighted (residuals divided by the standard deviation of the efficiency values), not binomially weighted.
hrss50 is the fitted parameter \(10^{\log_{10} h_{rss}^{50}}\) (bounded to \(10^{-25}\)–\(10^{-19}\)), with
hrssErfrom its fit error.hrss10 and hrss90 are the amplitudes where the fitted curve crosses 0.1 and 0.9, root-found only inside the sampled \(h_{rss}\) range; they are NaN when the curve does not reach that level there.
The fit is not attempted when all efficiencies are below 0.5 (status
above_sampled_range,hrss50empty,bound= largest \(h_{rss}\)), all are above 0.5 (below_sampled_range), or fewer than three amplitudes are available (skipped).
compute_hrss50() (and its
plotting alias
plot_efficiency_vs_hrss())
instead pools all injections in matched_right_file into one curve and
reports only hrss50, by linear interpolation in \(\log h_{rss}\) between
the two amplitudes that bracket 50% efficiency. It returns no value when 50%
is not bracketed, and computes no hrss90.
Efficiency by Waveform Type¶
Efficiency is computed separately for each injected waveform name
(sim_name), characterizing the search’s sensitivity to different signal
morphologies (for example sine-Gaussian bursts at various central frequencies
and Q-factors, or white-noise bursts), provided each population has fixed
\(h_{rss}\) values.
Grouping is automatic; there is no filter argument. For plotting, the Q-factor
and frequency are parsed from the waveform name (native ..._Q<q>_... /
..._<f>Hz... names or cWB SG<f>Q<q> / SGE<f>Q<q> names); names
without a Q-factor are drawn in a Q = 0 panel.
compute_efficiency_by_waveform()
reports one efficiency per waveform, pooled over all amplitudes, together
with the fraction recovered by cWB regardless of the IFAR threshold.
Visualization¶
Efficiency figures are rendered by the helpers in
pycwb.modules.postprocess.efficiency_plots and included in the
HTML report (Post-Production Workflow Guide) through the plots of each
simulation_runs entry. The plots are:
Efficiency vs. \(h_{rss}\) by waveform: one panel per Q-factor, one curve per frequency, binomial error bars, the fitted sigmoid overlaid (when the fit succeeded) and a dotted line at each fitted hrss50 (
compute_efficiency_vs_hrss_by_waveform)Pooled efficiency vs. \(h_{rss}\) with the interpolated hrss50 (
compute_hrss50/plot_efficiency_vs_hrss)Per-waveform bar chart of detection efficiency (
compute_efficiency_by_waveform)
Configurable Thresholds¶
Parameter |
Default |
Description |
|---|---|---|
|
|
IFAR threshold for detection (see IFAR duration syntax) |
|
|
Comma-separated IFARs for |
|
|
Statistic used for both the background calibration and the injection scores |
|
none |
Pre-scored SIM catalog, joined by |
|
required |
Background livetime of |
|
|
Remove CAT0/1/2-vetoed and across-segment injections from the denominator |
|
|
Must be |
Interpreting Efficiency Results¶
hrss50 represents the amplitude at which the search is 50% efficient—a common figure of merit for burst searches.
hrss90 is often quoted as the “sensitive range” of the search.
Flat efficiency at high amplitude: loud signals should be recovered (efficiency → 1). With the default
exclude_vetoed: false, vetoed and across-segment injections, and injections whose trigger fails the prediction cuts, count as missed, so a plateau below 100% is expected when such injections exist. A plateau below 100% withexclude_vetoed: truepoints to a pipeline problem.Efficiency at low amplitude: Should approach the false-alarm probability (not zero) due to accidental coincidences with background triggers.
Statistical uncertainty: Binomial error bars shrink with more injections. For precise hrss50/hrss90, aim for at least several hundred injections per waveform type.
Read efficiency curves¶
Read each curve together with its eligible injection count, uncertainty and
ranking threshold. Include missed eligible injections in the denominator and
account for the exclude_vetoed setting when interpreting the plateau.
Compare waveform families using their frequency content and chosen amplitude
coordinate. Inspect the fit status column (fit_status in the hrss50 CSV)
before quoting hrss50 or hrss90; the sampled amplitudes must cover the requested
crossing. Add trials or amplitude points where the curve is poorly determined.
See also: XGBoost Classification · Background Estimation · Injection Infrastructure
Next: Postproduction Reports — assembling the final reports
Apply the method: Run Ranking and Efficiency on a Study describes the study workflow; Choose an Analysis Task routes other production tasks to their guides.
Manual simulation summary paths¶
For deterministic polarization generators whose waveform support is independent
of scheduling/arrival fields, waveform support caching is enabled by default
to reuse measured source epochs and sample counts. Use
--no-cache-waveform-support to disable it (or pass
cache_waveform_support=False in Python). All model parameters, including distance,
remain in the key by default. Model-specific exclusions can be saved in the
injection configuration. For the physical-distance CBC benchmark, for example:
injection:
cache_waveform_ignore_par: [distance, distance_index, factor_index]
# Existing generator and population settings follow here.
--cache-waveform-ignore-par distance,distance_index,factor_index replaces the
configured list; --cache-waveform-ignore-par '' clears it. The list does not
enable caching by itself. Each excluded parameter must leave source waveform
support unchanged for the selected generator. Generator identity and sample
interval cannot be excluded. Python callers can override the same list through
build_simulation_summary(..., cache_waveform_support=True,
cache_waveform_ignore_par=[...]); None inherits the configuration and [] clears
it. Original parameters are still passed to the generator and retained in rows.
The lookup recomputes the detector arrival delays for each injection and keeps every simulation row, distance, identity and veto flag. Only timing metadata is cached, in memory for the current build; waveform arrays are not retained. Direct-strain generators and generators returning parameter updates are evaluated on each row. Disable caching for generators that change their support with scheduling fields or explicitly excluded parameters.
pycwb simulation-summary --work-dir /production defaults to
/production/config/user_parameters.yaml and writes
/production/catalog/simulations.parquet. An explicitly supplied config
or --output path is relative to the caller’s directory. Relative paths
inside the config (DQ, frames, waveform inputs) are resolved from the
production directory, consistently with batch setup and execution.
IFAR duration syntax¶
IFAR accepts positive numeric seconds (including scientific notation) or
positive durations with s, day, wk, mo, and yr suffixes,
such as 100yr. Existing presets retain their exact historical values;
1mo is 30 days, whereas the historical 6mo is half a Julian year.
Unknown or nonpositive values fail explicitly, including in MDC scoring.
Choosing a simulation association rule¶
Native match_simulations associates waveform/trigger interval overlaps
within the same trial and scheduled job, then chooses unique pairs. This is
the practical default for mixed waveform families and extended signals:
requested injection GPS time can denote a waveform endpoint, while trigger
GPS time is a reconstructed detector arrival. A blanket 0.1-second cut on
these two scalar columns does not reproduce cWB’s detector-time cut.
Use ranking_par: rho_alt (or pycwb match-simulations --ranking-par
rho_alt) when choosing unique events with the cWB pp_irho=1 statistic.
The default rho remains available for native analyses. Rank selection
alone does not make the association algorithm cWB-equivalent.
For exact cWB postproduction comparison, use import_cwb_simulation on
ROOT/MDC truth: it uses cWB’s recorded injection association and applies
time_window: 0.1 after unique selection. Failed winners stay missed;
a quieter candidate is not substituted. For native detector-time recovery
cuts, first establish compatible injected/reconstructed detector timing
and injection identity. Do not silently substitute a geocentric GPS cut.
Dense overlapping injections remain potentially ambiguous under interval
association; controlled populations with adequate separation are preferable.
Keeping background and sensitivity selections consistent¶
Use the same ranking statistic for FAR, sensitivity, and the report. The
standard workflow uses rhor and supplies its already-scored SIM catalog:
args:
ranking_par: rhor
scored_file: ${paths.sim_eval_scored}
ifar: 100yr
These options are supported by the per-waveform efficiency, hrss curves,
hrss50, and hrss50 CSV actions. For compute_hrss50, supply the injection
truth via matched_right_file. Scores are joined by event ID; missed
injections and events removed by prediction cuts stay in the denominator.
Without scored_file, model scoring applies the same user ranking hooks
and prediction cuts as score_catalog. Probability ranking remains the
backward-compatible default, ranking_par: xgb_prob.
Calibration uses inclusive empirical background tail counts, including ties.
No background exceedances means empirical FAR zero, not a measured infinite
exposure. Results identify ranking_par, ranking_threshold, and
ifar_convention. The legacy prob_threshold field is populated only
for probability ranking. Plot labels identify the selected statistic.