scnn.activations

Generate activation patterns for convex reformulations of neural networks.

Overview:

This module provides functions for generating gate vectors and activation patterns for ReLU or threshold activation functions. These patterns are used when forming (subsampled) convex reformulations of neural networks with those activations. Gate vectors are also required for Gated ReLU networks.

An activation pattern is a vector of the form,

\[d_i = 1(X w \geq 0),\]

where \(1(z \geq 0)\) is an element-wise indicator function whose i’th element is one when \(z_i \geq 0\) and zero-otherwise and \(w \in \mathbb{R}^d\) is a “gate vector”. Forming the convex reformulation of a neural network with ReLU activations requires enumerating the activation patterns a single ReLU or threshold neuron can take on,

\[\mathcal{D} = \left\{ d = 1(X w \geq 0) : w \in \mathbb{R}^d \right\}.\]

In practice, :math`mathcal{D}` can approximated sampling vectors \(w \sim P\) according to some distribution \(P\) and then computing the corresponding pattern \(d_i\).

scnn.activations.compute_activation_patterns(X: ndarray, G: ndarray, filter_duplicates: bool = True, filter_zero: bool = True, bias: bool = False, active_proportion: float | None = None) → Tuple[ndarray, ndarray]

Compute activation patterns corresponding to a set of gate vectors.

Parameters:

X – an (n x d) data matrix, where the examples are stored as rows.
G – an (d x m) matrix of “gate vectors” used to generate the activation patterns.
filter_duplicates – whether or not to remove duplicate activation patterns and the corresponding
filter_zero – whether or not to filter the zero activation pattern and corresponding gates. Defaults to True.
bias – whether or not a bias should be included in the gate vectors.
active_proportion – force each gate to be active for active_proportion`*n of the training examples using a bias term. This feature is only supported when `bias == True.

Returns:

D, an (n x p) matrix of (possibly unique) activation patterns where
\(p \leq m\)
G, a (d x b) matrix of gate vectors generating D.

scnn.activations.generate_index_lists(d: int, order: int) → List[List[int]]

Generate lists of all groups of indices of order up to and including order.

Parameters:

d – the dimensionality or maximum index.
order – the order to which the lists should be generated. For example, order=2 will generate all singleton lists and all lists of pairs of indices.

Returns:

List of list of indices.

scnn.activations.sample_convolutional_gates(rng: Generator, d: int, n_samples: int) → ndarray

Generate convolutional gate vectors by random sampling.

Parameters:

rng – a NumPy random number generator.
d – the dimensionality of the gate vectors.
n_samples – the number of samples to use.

Returns:

G - a \(d \times \text{n_samples}\) matrix of gate vectors.

scnn.activations.sample_dense_gates(rng: Generator, d: int, n_samples: int) → ndarray

Generate dense gate vectors by random sampling.

Parameters:

rng – a NumPy random number generator.
d – the dimensionality of the gate vectors.
n_samples – the number of samples to use.

Returns:

G - a \(d \times \text{n_samples}\) matrix of gate vectors.

scnn.activations.sample_gate_vectors(seed: int, d: int, n_samples: int, gate_type: typing_extensions.Literal[dense, feature_sparse, convolutional] = 'dense', order: int | None = None) → ndarray

Generate gate vectors by random sampling.

Parameters:

seed – the random seed to use when generating the gates.
d – the dimensionality of the gate vectors.
n_samples – the number of samples to use.
gate_type –
the type of gates to sample. Must be one of:
- ’dense’: sample dense gate vectors.
- ’feature_sparse’: sample gate vectors which are sparse in
  specific features.
order – the maximum order of feature sparsity to consider. Only used for gate_type=’feature_sparse’. See sample_sparse_gates() for more details.

Notes

It is possible to obtain more than n_samples gate vectors when: gate_type=’feature_sparse’
if len(sparsity_indices) does not divide evenly into n_samples. In: this case, we use the ceiling function to avoid sampling zero gates for some sparsity patterns.

scnn.activations.sample_sparse_gates(rng: Generator, d: int, n_samples: int, sparsity_indices: List[List[int]]) → ndarray

Generate feature-sparse gate vectors by random sampling.

Parameters:

rng – a NumPy random number generator.
d – the dimensionality of the gate vectors.
n_samples – the number of samples to use.
sparsity_indices – lists of indices (i.e. features) for which sparse gates should be generated. Each index list will get n_samples / len(sparsity_indices) gates which are sparse in every feature except the given indices.

Notes

It is possible to obtain more than n_samples gate vectors if
len(sparsity_indices) does not divide evenly into n_samples. In this case, we use the ceiling to avoid sampling zero gates for some sparsity patterns.

Returns:: G - a \(d \times \text{n_samples}\) matrix of gate vectors.