calibrain.MetricEvaluator#

class calibrain.MetricEvaluator(ue, *, nominal_coverages=None, evaluation_metrics=None, calibration_metrics=None, logger=None)[source]#

Evaluator aligned with the updated UncertaintyEstimator.

Supported settings#

  • fixed

  • eeg_free

  • meg_free

Supported modes#

  • pointwise

  • aggregated

Conventions#

  1. fixed: x_true, x_hat have shape (N,T)

  2. eeg_free: x_true, x_hat have shape (N,3,T) error metrics use amplitude representation ||x_i(t)||_2

  3. meg_free: x_true is 3D truth in the retained local 3D basis, shape (N,3,T) x_hat is reduced 2D posterior mean, shape (N,2,T) or flat (2N,T) posterior covariance is reduced 2D, shape (2N,2N) error metrics use reduced-coordinate amplitude representation after projecting truth to 2D via the same V_tan basis used by the inverse

Notes

  • Aggregated mode always means time-average.

  • EMD expects coords already aligned with the source subset being evaluated.

  • Calibration metrics are:

    • max_underconfidence_deviation

    • max_overconfidence_deviation

    • mean_absolute_deviation

    • mean_signed_deviation

  • For MEG, pass V_tan = lf_free_meg[“Q_basis”] whenever possible to avoid basis mismatches.

Methods

calibration_curve(*, x_true, x_hat, ...[, ...])

calibration_metrics_4(nominal, empirical)

emd(*, x_true, x_hat, coords, setting[, ...])

evaluate_all(*, x_true, x_hat, ...[, mode, ...])

mae(*, x_true, x_hat, setting[, mode, ...])

mean_posterior_std(*, posterior_uncert, setting)

mse(*, x_true, x_hat, setting[, mode, ...])

rmae(*, x_true, x_hat, setting[, mode, ...])

rmse(*, x_true, x_hat, setting[, mode, ...])

__init__(ue, *, nominal_coverages=None, evaluation_metrics=None, calibration_metrics=None, logger=None)[source]#
static calibration_metrics_4(nominal, empirical)[source]#
Return type:

Dict[str, float]

mse(*, x_true, x_hat, setting, mode='pointwise', V_tan=None, L_free_Mx3N=None)[source]#
Return type:

float

mae(*, x_true, x_hat, setting, mode='pointwise', V_tan=None, L_free_Mx3N=None)[source]#
Return type:

float

rmse(*, x_true, x_hat, setting, mode='pointwise', V_tan=None, L_free_Mx3N=None)[source]#
Return type:

float

rmae(*, x_true, x_hat, setting, mode='pointwise', V_tan=None, L_free_Mx3N=None)[source]#
Return type:

float

mean_posterior_std(*, posterior_uncert, setting, mode='pointwise', n_times=None)[source]#
Return type:

float

emd(*, x_true, x_hat, coords, setting, mode='pointwise', V_tan=None, L_free_Mx3N=None, eps=1e-12)[source]#
Return type:

float

calibration_curve(*, x_true, x_hat, posterior_uncert, setting, mode='aggregated', V_tan=None, L_free_Mx3N=None, free_interval_type='full_cov')[source]#
Return type:

Dict[str, Any]

evaluate_all(*, x_true, x_hat, posterior_uncert, setting, mode='aggregated', coords=None, V_tan=None, L_free_Mx3N=None, compute_emd=False, free_interval_type='full_cov')[source]#
Return type:

Dict[str, Any]