Contents

• ikkuna.export.subscriber
  • Module contents

ikkuna.export.subscriber

Module contents

class ikkuna.export.subscriber.Subscriber(subscriptions, message_bus)

Base class for receiving and processing activations, gradients and other stuff into insightful metrics.

__init__(subscriptions, message_bus)

Parameters:subscriptions (list(Subscription)) –

_add_publication(topic, type='DATA')

Parameters:

• topic (str) – Topic name
• type (str) – DATA or META

__del__()

If for whatever reason a subscriber ceases to exist before the interpreter ends, delete the registered topics

publications

kinds

subscriptions

message_bus

compute(message_or_bundle)

This is where the magic happens. Subclasses should override this method so that they can compute their metric upon reception of their desired messages or do whatever else they want. If interested in plotting, they should then use their backend property to plot the metric (if they display line plots) and their message_bus to publish a new message with the metric.

Parameters:message_or_bundle (ikkuna.export.messages.Message or ikkuna.export.messages.MessageBundle) –

If the subscriber uses a SynchronizedSubscription, a bundle is received in each call, otherwise just a Message.

Messages can either be NetworkMessage if the Subscriber is not interested in actual training artifacts, or ModuleMessage

receive_message(message)

Process a single message received from an MessageBus.

process_messages(message_or_bundle)

Callback for processing a single Message or a MessageBundle object with NetworkMessages or ModuleMessages in it.

Parameters:message_or_bundle (ikkuna.export.messages.Message or ikkuna.export.messages.MessageBundle) – The exact nature of this package is determined by the ikkuna.export.subscriber.Subscription attached to this ikkuna.export.subscriber.Subscriber. Raises:ValueError – If the received MessageBundle object is not complete()

class ikkuna.export.subscriber.Subscription(subscriber, kinds, tag='default', subsample=1)

Specification for a subscription that can span multiple kinds and a tag.

_tag

Tag for filtering the processed messages

Type:str

_subscriber

The subscriber associated with the subscription

Type:ikkuna.export.subscriber.Subscriber

counter

Number of times the Subscription was called for each pair of module and kind or meta data identifier. Since one ikkuna.export.subscriber.Subscriber is associated with only one configuration of ikkuna.export.messages.MessageBundle, this will enable proper subsampling of message streams.

Type:dict(tuple(ikkuna.utils.NamedModule, str) or str, int)

kinds

List of string identifiers for different message kinds. These are all the message kinds the subscriber wishes to receive

Type:list(str)

_subsample

Factor for subsampling incoming messages. Only every subsample-th message will be processed.

Type:int

__init__(subscriber, kinds, tag='default', subsample=1)

Parameters:

• subscriber (ikkuna.export.subscriber.Subscriber) – Object that wants to receive the messages
• kinds (list) – List of message kinds to process
• tag (str) – tag for filtering messages
• subsample (int) – Number of messages to ignore before processing one. Note that this number if applied to every kind, regardless of frequency. So if subsample = 10, every tenth weights message would be processed, but also only every tenth epoch_finished message.
• tag – Optional tag for filtering messages.

counter

kinds

_handle_message(message)

Process a newly arrived message. Subclasses should override this method for any special treatment.

Parameters:message (ikkuna.export.messages.Message) –

handle_message(message)

Callback for receiving an incoming message.

Parameters:message (ikkuna.export.messages.ModuleMessage) –

class ikkuna.export.subscriber.SynchronizedSubscription(subscriber, kinds, tag='default', subsample=1)

A subscription which buffers messages and publishes a set of messages, each of a different kind, when one round (a train step) is over. This is useful for receiving several kinds of messages in each train step and always have them be processed together.

_new_round(round_idx)

Start a new round of buffering, clearing the previous cache and resetting the record for which kinds were received in this round.

Parameters:round_idx (int) – Global train step of the new round Raises:RuntimeError – If not all desired kinds have been received for all identifiers yet in the current round

_handle_message(message)

Start a new round if a new sequence number is seen.

class ikkuna.export.subscriber.PlotSubscriber(subscriptions, message_bus, plot_config, backend='tb')

Base class for subscribers that output scalar or histogram values per time and module

_backend

Plotting backend

Type:ikkuna.visualization.Backend

__init__(subscriptions, message_bus, plot_config, backend='tb')

Parameters:

• ylims (tuple(int, int)) – Optional Y-axis limits
• plot_config (dict) – Configuration parameters for plotting. Relevant keys are title, xlabel, ylabel and ylims. Which of them are actually used depends on the Backend

backend

The backend to use for plotting

Type:ikkuna.visualization.Backend

compute(message_or_bundle)

This is where the magic happens. Subclasses should override this method so that they can compute their metric upon reception of their desired messages or do whatever else they want. If interested in plotting, they should then use their backend property to plot the metric (if they display line plots) and their message_bus to publish a new message with the metric.

Parameters:message_or_bundle (ikkuna.export.messages.Message or ikkuna.export.messages.MessageBundle) –

If the subscriber uses a SynchronizedSubscription, a bundle is received in each call, otherwise just a Message.

Messages can either be NetworkMessage if the Subscriber is not interested in actual training artifacts, or ModuleMessage

class ikkuna.export.subscriber.CallbackSubscriber(kinds, callback, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1)

Subscriber class for subscribing to ModuleMessages and running a callback with them.

__init__(kinds, callback, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1)

Parameters:

• subscription (Subscription) –
• message_bus (ikkuna.export.messages.MessageBus) –
• callback (function) – A function that accepts as many parameters as the Subscription delivers messages at once.

compute(message_or_bundle)

This is where the magic happens. Subclasses should override this method so that they can compute their metric upon reception of their desired messages or do whatever else they want. If interested in plotting, they should then use their backend property to plot the metric (if they display line plots) and their message_bus to publish a new message with the metric.

Parameters:message_or_bundle (ikkuna.export.messages.Message or ikkuna.export.messages.MessageBundle) –

If the subscriber uses a SynchronizedSubscription, a bundle is received in each call, otherwise just a Message.

Messages can either be NetworkMessage if the Subscriber is not interested in actual training artifacts, or ModuleMessage

class ikkuna.export.subscriber.BatchedSVCCASubscriber(dataset_meta, n, forward_fn, freeze_at=10, batch_size=256, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

A subscriber which at some interval halts training, propagates some dataset through the net, records activations for all modules and stores them. At the next checkpoint, SVCCA similarity is computed between every layer at time step t and the same layer at t-1.

Note

It seems that despite the fact that SVCCA runs at least twice as fast on the GPU, this subscriber is not faster than on the CPU. I’m not sure why that is, since I’m even caching input data to avoid repeatedly copying to the GPU

Warning

Currently, this subscriber uses Numpy for SVCAA. For some reason, the activations produced by networks lead to highly ill-conditioned covariance matrices and PyTorch’s SVD handles them differently than Numpy does. SVD does not converge in that case, and robust_cca_similarity must be used, which defeats the whole idea of speeding up the process.

__init__(dataset_meta, n, forward_fn, freeze_at=10, batch_size=256, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

Parameters:

• dataset_meta (ikkuna.utils.DatasetMeta) – Dataset to load data from for retrieving activations
• n (int) – Number of datapoints to randomly sample
• freeze_at (float) – Similarity threshold after which a layer is frozen. Values >= 1 will turn off freezing
• batch_size (int) – Batch size to use for forward passes. Can be set to some value if the entire data at once would be too larger. Otherwise use SVCCASubscriber

_module_complete_previous(module)

Check if activations for module are completely buffered from the previous step.

_module_complete_current(module)

Check if activations for module are completely buffered from the current step.

_record_activations_previous(module, data)

Record activations into the previous buffer

_record_activations_current(module, data)

Record activations into the current buffer

compute(message)

A NetworkMessage with the identifier self_similarity will be published.

class ikkuna.export.subscriber.HistogramSubscriber(kind, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, backend='tb')

A Subscriber which subsamples training artifacts and computes histograms per epoch. Histograms are non-normalized.

compute(message)

Note

Since the histogram is computed by the visualization backend (there’s no practical way around it), this subscriber does not publish a ModuleMessage

class ikkuna.export.subscriber.LossSubscriber(message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

A Subscriber that plots the training loss at every time step.

compute(message)

This is where the magic happens. Subclasses should override this method so that they can compute their metric upon reception of their desired messages or do whatever else they want. If interested in plotting, they should then use their backend property to plot the metric (if they display line plots) and their message_bus to publish a new message with the metric.

Parameters:message_or_bundle (ikkuna.export.messages.Message or ikkuna.export.messages.MessageBundle) –

If the subscriber uses a SynchronizedSubscription, a bundle is received in each call, otherwise just a Message.

Messages can either be NetworkMessage if the Subscriber is not interested in actual training artifacts, or ModuleMessage

class ikkuna.export.subscriber.MeanSubscriber(kind, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

compute(message)

Compute the average of a quantity. A ModuleMessage with the identifier {kind}_mean is published.

class ikkuna.export.subscriber.MessageMeanSubscriber(kind, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

Compute the mean over all messages with scalar data of a given kind in one train step. This is useful for displaying summary statistics.

compute(message)

A ModuleMessage with the identifier {kind}_message_mean is published.

class ikkuna.export.subscriber.NormSubscriber(kind, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb', order=2)

compute(message)

Compute the norm of a quantity. A ModuleMessage with the identifier {kind}_norm{order} is published.

class ikkuna.export.subscriber.RatioSubscriber(kinds, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb', absolute=True)

A Subscriber which computes the average ratio between two quantities. The dividend will be the first element of the kinds property, the divisor the second. Therefore it is vital to pass the message kinds to the Subscription object in the correct order.

__init__(kinds, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb', absolute=True)

Parameters:absolute (bool) – Whether to use absolute ratio

compute(message_bundle)

The ratio between the two kinds is computed as the ratio of L2-Norms of the two Tensors. A ModuleMessage with the identifier {kind1}_{kind2}_ratio is published.

class ikkuna.export.subscriber.SVCCASubscriber(dataset_meta, n, *args, **kwargs)

Simplified BatchedSVCCASubscriber subclass which does not use a ChunkedDict but instead regular dicts and thus assumes that the entire test data can be propagated through the model in one go and batching is unnecessary.

class ikkuna.export.subscriber.SpectralNormSubscriber(kind, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

__init__(kind, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

Parameters:kind (str) – Message kind to compute spectral norm on. Doesn’t make sense with kinds of non-matrix type.

For other parameters, see PlotSubscriber

compute(message)

The spectral norm computation is taken from the Pytorch implementation of spectral norm. It’s possible to use SVD instead, but we are not interested in the full matrix decomposition, merely in the singular values.

A ModuleMessage with the identifier {kind}_spectral_norm is published.

class ikkuna.export.subscriber.TestAccuracySubscriber(dataset_meta, forward_fn, batch_size, message_bus=<ikkuna.export.messages.MessageBus object>, frequency=100, ylims=None, tag='default', subsample=1, backend='tb')

Subscriber which can compute the accuracy on a given data set (generally the test/validation set).

_dataset_meta

Dataset to compute accuracy over

Type:ikkuna.utils.DatasetMeta

_data_loader

Loader on the dataset

Type:torch.utils.data.DataLoader

_frequency

Number of batches to ignore before computing accuracy once

Type:int

_forward_fn

Bound method on the model to push data through and get predictions

Type:function

__init__(dataset_meta, forward_fn, batch_size, message_bus=<ikkuna.export.messages.MessageBus object>, frequency=100, ylims=None, tag='default', subsample=1, backend='tb')

Parameters:

• dataset_meta (ikkuna.utils.DatasetMeta) – Test dataset
• forward_fn (function) – Bound version of torch.nn.Module.forward(). This could be obtained from the train.Trainer.model property.
• batch_size (int) – Batch size to use for pushing the test data through the model. This doesn’t have to be the training batch size, but must be selected so that the forward pass fits into memory.
• frequency (int) – Inverse of the frequency with which to compute the accuracy

compute(message)

Compute accuracy over the entire test set.

A NetworkMessage with the identifier test_accuracy is published.

class ikkuna.export.subscriber.TrainAccuracySubscriber(message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

Subscriber which computes the batch accuracy.

__init__(message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

For parameters see PlotSubscriber

compute(message)

Compute accuracy over the current train batch.

A NetworkMessage with the identifier train_accuracy is published.

class ikkuna.export.subscriber.VarianceSubscriber(kind, message_bus=<ikkuna.export.messages.MessageBus object>, tag='default', subsample=1, ylims=None, backend='tb')

A Subscriber which computes the variance of quantity for the current batch.

compute(message)

Compute the variance of a quantity. A ModuleMessage with the identifier {kind}_variance is published.