Metric Evaluation#

ME#

class calibrain.metric_evaluation.MetricEvaluator(confidence_levels=None, metrics=None, logger=None)[source]#

Bases: object

__init__(confidence_levels=None, metrics=None, logger=None)[source]#: Initialize the MetricEvaluator with confidence levels, metrics, and a logger. :type confidence_levels: ndarray :param confidence_levels: Array of confidence levels to evaluate metrics against. :type confidence_levels: np.ndarray, optional :type metrics: list[str] :param metrics: List of metric names (method names) to evaluate. :type metrics: list[str], optional :type logger: Logger :param logger: Logger instance for logging debug and error messages. :type logger: logging.Logger, optional

mean_calibration_error(empirical_coverage, **kwargs)[source]#

Calculate the area under the curve (AUC) deviation, which measures the average calibration error. :type kwargs: :param kwargs: Additional keyword arguments that may be needed for metric calculations:

‘empirical_coverage’: np.ndarray, empirical coverage values.

Returns:: The AUC deviation value.
Return type:: float

max_underconfidence_deviation(empirical_coverage, **kwargs)[source]#

Calculate the maximum positive deviation from the confidence levels (). :type empirical_coverage: :param empirical_coverage: Empirical coverage values. :type empirical_coverage: np.ndarray :type kwargs: :param kwargs: Additional keyword arguments that may be needed for metric calculations:

‘empirical_coverage’: np.ndarray, empirical coverage values.

Returns:: The maximum positive deviation value.
Return type:: float

max_overconfidence_deviation(empirical_coverage, **kwargs)[source]#

Calculate the maximum negative deviation from the confidence levels. :type empirical_coverage: :param empirical_coverage: Empirical coverage values. :type empirical_coverage: np.ndarray :type kwargs: :param kwargs: Additional keyword arguments that may be needed for metric calculations:

‘empirical_coverage’: np.ndarray, empirical coverage values.

Returns:: The maximum negative deviation value.
Return type:: float

mean_absolute_deviation(empirical_coverage, **kwargs)[source]#

Calculate the mean absolute deviation from the confidence levels.#

kwargsdict: Additional keyword arguments that may be needed for metric calculations: - ‘empirical_coverage’: np.ndarray, empirical coverage values.

returns:: The maximum absolute deviation value.
rtype:: float

mean_signed_deviation(empirical_coverage, **kwargs)[source]#

Calculate the mean signed deviation from the confidence levels.#

kwargsdict: Additional keyword arguments that may be needed for metric calculations: - ‘empirical_coverage’: np.ndarray, empirical coverage values.

returns:: The mean signed deviation value.
rtype:: float

mean_posterior_std(cov, **kwargs)[source]#

Calculate the mean posterior standard deviation. :type kwargs: :param kwargs: Additional keyword arguments that may be needed for metric calculations:

‘cov’: np.ndarray, covariance matrix for uncertainty metrics.

Returns:: The mean posterior standard deviation.
Return type:: float

emd(x, x_hat, orientation_type, subject, fwd_path, **kwargs)[source]#

Compute Earth Mover’s Distance (EMD) between true and estimated source activations. Adapted from BSI-ZOO Parameters: - x : (n_sources, n_times) or (n_sources, 3, n_times)

Ground truth source time courses.

x_hatsame shape as x
Estimated source time courses.
orientation_typestr
‘fixed’ or ‘free’ for orientation modeling.
subjectstr
Subject ID used to locate the forward model.

Returns: - float

Earth Mover’s Distance between normalized source distributions.

jaccard_error(x, x_hat, orientation_type=None, **kwargs)[source]#

TODO: To be checked! Calculate Jaccard error between true and estimated active source sets.

Parameters:

x (np.ndarray) – True source activations
x_hat (np.ndarray) – Estimated source activations
orientation_type (str, optional) – ‘fixed’ or ‘free’ for orientation modeling
**kwargs (dict) – Additional arguments

Returns:

Jaccard error (1 - Jaccard index) between active source sets

Return type:

float

mse(x, x_hat, orientation_type, **kwargs)[source]#

euclidean_distance(x, x_hat, orientation_type, subject, nnz, fwd_path, **kwargs)[source]#: adapted from BSI-ZOO

f1(x, x_hat, orientation_type, **kwargs)[source]#: adapted from BSI-ZOO

accuracy(x, x_hat, orientation_type, **kwargs)[source]#

Calculate accuracy between true and estimated active source sets.

Accuracy = (TP + TN) / (TP + TN + FP + FN) where: - TP: True Positives (correctly identified active sources) - TN: True Negatives (correctly identified inactive sources) - FP: False Positives (incorrectly identified as active) - FN: False Negatives (missed active sources)

Parameters:

x (np.ndarray) – True source activations (n_sources, n_times) or (n_sources, 3, n_times)
x_hat (np.ndarray) – Estimated source activations (same shape as x)
orientation_type (str) – ‘fixed’ or ‘free’ for orientation modeling
**kwargs (dict) – Additional arguments

Returns:

Accuracy score between 0.0 and 1.0 (higher is better)

Return type:

float

evaluate_and_store_metrics(current_results_dict, metric_suffix='', **kwargs)[source]#

Evaluate metrics and update the results dictionary.

Parameters:

current_results_dict (dict) – Dictionary to store the results of the metrics.
metric_suffix (str, optional) – Suffix to add to metric keys (e.g., “_all_sources”, “_active_indices”).
kwargs (dict) – Additional keyword arguments that may be needed for metric calculations: - ‘empirical_coverage’: np.ndarray, empirical coverage values. - ‘cov’: np.ndarray, covariance matrix for uncertainty metrics.