ECE

class hybrid_learning.concepts.train_eval.kpis.aggregating_kpis.ECE(n_bins=10, threshold=0.5, threshold_discard=0.0, only_class=None, max_prob=True, adaptive=False, class_conditional=False, aggregation='expectation')[source]

Bases: AggregatingKpi

Expected calibration error. The formula first calculates per bin the absolute difference between the fraction of predictions that are correct (accuracy) and the mean of the predicted probabilities in the bin (confidence). The final expected calibration error then is the sum of these bin-wise differences, each weighted by the fraction of predictions in the bin and total number of predictions:

\[ECE = \sum_{b=1}^{B} \frac{N_b}{N} | text{acc}(b) - text{conf}(b) |\]

for bins \(B\), and \(N, N_b\) the total number of predictions respectively the predictions falling into bin \(b\). Bins are here selected to be equally sized with respect to lower and upper bound.

For details see [Naeini2015] and [Nixon2019]

Naeini2015

Naeini, Mahdi Pakdaman, Cooper, Gregory F, and Hauskrecht, Milos. Obtaining well calibrated probabilities using bayesian binning. In AAAI, pp. 2901, 2015. http://europepmc.org/article/PMC/4410090

Nixon2019

Nixon, Jeremy, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring Calibration in Deep Learning. In CVPR WS, pp. 38-41. 2019. https://arxiv.org/abs/1904.01685

Public Data Attributes:

Inherited from : py: class:Module

dump_patches

This allows better BC support for load_state_dict().

T_destination

alias of TypeVar('T_destination', bound=Mapping[str, Tensor])

Public Methods:

update(outputs, labels)

Called every batch.

reset()

Called once at the beginning of a new epoch.

value()

Shall return the aggregated metric value as a scalar.
Inherited from : py: class:AggregatingKpi

update(outputs, labels)

Called every batch.

reset()

Called once at the beginning of a new epoch.

value()

Shall return the aggregated metric value as a scalar.

value_and_reset()

Shorthand for subsequent calls to value and to reset.

forward(outputs, labels)

Calculate KPI without gradient.
Inherited from : py: class:Module

forward(outputs, labels)

Calculate KPI without gradient.

register_buffer(name, tensor[, persistent])

Adds a buffer to the module.

register_parameter(name, param)

Adds a parameter to the module.

add_module(name, module)

Adds a child module to the current module.

get_submodule(target)

Returns the submodule given by target if it exists, otherwise throws an error.

get_parameter(target)

Returns the parameter given by target if it exists, otherwise throws an error.

get_buffer(target)

Returns the buffer given by target if it exists, otherwise throws an error.

apply(fn)

Applies fn recursively to every submodule (as returned by .children()) as well as self.

cuda([device])

Moves all model parameters and buffers to the GPU.

xpu([device])

Moves all model parameters and buffers to the XPU.

cpu()

Moves all model parameters and buffers to the CPU.

type(dst_type)

Casts all parameters and buffers to dst_type.

float()

Casts all floating point parameters and buffers to float datatype.

double()

Casts all floating point parameters and buffers to double datatype.

half()

Casts all floating point parameters and buffers to half datatype.

bfloat16()

Casts all floating point parameters and buffers to bfloat16 datatype.

to_empty(*, device)

Moves the parameters and buffers to the specified device without copying storage.

to(*args, **kwargs)

Moves and/or casts the parameters and buffers.

register_backward_hook(hook)

Registers a backward hook on the module.

register_full_backward_hook(hook)

Registers a backward hook on the module.

register_forward_pre_hook(hook)

Registers a forward pre-hook on the module.

register_forward_hook(hook)

Registers a forward hook on the module.

state_dict([destination, prefix, keep_vars])

Returns a dictionary containing a whole state of the module.

load_state_dict(state_dict[, strict])

Copies parameters and buffers from state_dict into this module and its descendants.

parameters([recurse])

Returns an iterator over module parameters.

named_parameters([prefix, recurse])

Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.

buffers([recurse])

Returns an iterator over module buffers.

named_buffers([prefix, recurse])

Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.

children()

Returns an iterator over immediate children modules.

named_children()

Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.

modules()

Returns an iterator over all modules in the network.

named_modules([memo, prefix, remove_duplicate])

Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.

train([mode])

Sets the module in training mode.

eval()

Sets the module in evaluation mode.

requires_grad_([requires_grad])

Change if autograd should record operations on parameters in this module.

zero_grad([set_to_none])

Sets gradients of all model parameters to zero.

share_memory()

See torch.Tensor.share_memory_()

extra_repr()

Set the extra representation of the module

Special Methods:

__init__([n_bins, threshold, ...])

Init.
Inherited from : py: class:AggregatingKpi

__init__([n_bins, threshold, ...])

Init.
Inherited from : py: class:Module

__init__([n_bins, threshold, ...])

Init.

__call__(*input, **kwargs)

Call self as a function.

__setstate__(state)

__getattr__(name)

__setattr__(name, value)

Implement setattr(self, name, value).

__delattr__(name)

Implement delattr(self, name).

__repr__()

Return repr(self).

__dir__()

Default dir() implementation.
Parameters

  • n_bins (int) –

  • threshold (float) –

  • threshold_discard (float) –

  • only_class (int) –

  • max_prob (bool) –

  • adaptive (bool) –

  • class_conditional (bool) –

  • aggregation (str) –

__init__(n_bins=10, threshold=0.5, threshold_discard=0.0, only_class=None, max_prob=True, adaptive=False, class_conditional=False, aggregation='expectation')[source]

Init.

Parameters
reset()[source]

Called once at the beginning of a new epoch. Shall reset the aggregating statistics.

update(outputs, labels)[source]

Called every batch. Shall be used to update the aggregating statistics.

Parameters

  • outputs (Tensor) – Output tensors of shape (BATCH x H x W);

  • labels (Tensor) – Label tensors of shape (BATCH x H x W);

value()[source]

Shall return the aggregated metric value as a scalar.

Returns

tensor containing the given metric value.

Return type

Tensor

adaptive: bool

Whether to use adaptive binning. Adaptive means the upper and lower bounds of bins are calculated from the complete final results such that each bin contains 1/n_bins samples.

aggregation: str

Aggregation specifier. Must be one of "mean", "expectation", or "max".

class_conditional: bool

Whether to bin each class separately.

correct_sum: Tensor

Buffer collecting the total amount of true positives and true negatives per bin.

count: Tensor

Total amount of predictions per bin.

max_prob: bool

Only consider the predicted class (equals maximal probability class for threshold 0.5)

n_bins

Number of bins. Upper and lower bounds are chosen according to the setting in adaptive.

only_class: int

Compute ECE only for a single class, condition on the predicted class

prob_sum: Tensor

Buffer collecting the sum of probabilities for each bin.

threshold: float

Threshold for the selecting positive class.

threshold_discard: float

If not None, discard all values below this threshold.

training: bool