calibrain.UncertaintyCalibrator#

class calibrain.UncertaintyCalibrator(uncertainty_estimator, metric_evaluator)[source]#

Uncertainty Calibrator#

This class connects UncertaintyEstimator and MetricEvaluator with an isotonic-regression based re-calibration of nominal coverages.

It uses:

  • UncertaintyEstimator’s aggregated membership/curve utilities to evaluate empirical coverage for any nominal coverage level c.

  • MetricEvaluator.calibration_metrics_4 (derived from the demos in metric_evaluation.py) to summarize AAD, ASD, etc. using the default calibration metrics.

  • Experiment-level train/test splits provided by the caller to avoid overfitting entire sources.

Conceptual model#

For each nominal coverage level c (e.g. 0.5, 0.9):

g(c) = empirical coverage when building CIs with nominal coverage c.

On the training experiments we learn g(c) via isotonic regression, then we numerically invert this curve to obtain a re-calibrated nominal coverage c_cal(c) ≈ g^{-1}(c). On a held-out evaluation split we build intervals using c_cal(c) and measure empirical coverage again.

The final goal is that after recalibration the empirical coverage is closer to the original nominal coverage grid defined in your UncertaintyEstimator (self.nominal_coverages).

Methods

apply_calibration(nominal_levels)

Apply learned calibration to new nominal coverage levels.

calibrate([x_true, x_hat, posterior_std, ...])

Run the full calibration procedure on the provided posterior statistics.

evaluate_with_mapping(*, test_data[, ...])

Evaluate pre- and post-calibration curves on test_data using an already-fitted mapping (from fit_mapping).

fit_mapping(*, train_data[, free_interval_type])

Fit an isotonic nominal-recalibration mapping from train_data only.
__init__(uncertainty_estimator, metric_evaluator)[source]#
Parameters:
calibrate(x_true=None, x_hat=None, posterior_std=None, *, train_data=None, test_data=None, verbose=False, fit=True, free_interval_type='full_cov')[source]#

Run the full calibration procedure on the provided posterior statistics.

Parameters:

  • x_true (np.ndarray, optional) – Ground-truth sources used both for training and evaluation when train_data is not supplied. Shape (n_sources, n_times) or (n_sources,).

  • x_hat (np.ndarray, optional) – Posterior mean estimates paired with x_true.

  • posterior_std (np.ndarray, optional) – Posterior standard deviation paired with x_true.

  • train_data (dict, optional) – Dictionary containing x_true, x_hat, and posterior_std arrays for the training experiments. When provided, the positional arguments are ignored for training.

  • test_data (dict, optional) – Dictionary containing x_true, x_hat, and posterior_std arrays for the evaluation experiments. If omitted, the training data is also used for evaluation.

  • verbose (bool, optional) – If True, print progress information.

  • fit (bool, optional) – If False, skip isotonic regression fitting and report only the raw (pre-calibration) coverage curve on the evaluation data. The returned post_calibration block mirrors the pre-calibration results with an identity mapping.

Returns:

Dictionary containing:

  • pre_calibration and post_calibration metadata (nominal coverages, empirical coverages, metrics, CI bounds).

  • train_empirical_coverages for diagnosing how the calibration map behaves on the training split.

  • calibration_metric_names listing the metrics evaluated.

Return type:

dict

fit_mapping(*, train_data, free_interval_type='full_cov')[source]#

Fit an isotonic nominal-recalibration mapping from train_data only.

This is intended for experiments like “post_fixed”, where you fit once (e.g., at the default setting) and reuse the same mapping over many evaluation datasets (e.g., an SNR/NNZ sweep).

Returns:

  • train_curve: pre-calibration curve on the training split

  • recalibrated_nominal_coverages: c_recalibrated on the nominal grid

Return type:

dict with

evaluate_with_mapping(*, test_data, free_interval_type='full_cov')[source]#

Evaluate pre- and post-calibration curves on test_data using an already-fitted mapping (from fit_mapping).

Return type:

dict

apply_calibration(nominal_levels)[source]#

Apply learned calibration to new nominal coverage levels.

Parameters:

nominal_levels (array-like) – Original nominal coverage levels.

Returns:

nominal_levels_recalibrated – Recalibrated nominal coverage levels to use when constructing credible intervals (i.e. in place of the original nominal levels).

Return type:

ndarray