/Users/hmamin/incendio/incendio/callbacks.py:26: UserWarning: Accio not available.
  warnings.warn('Accio not available.')

%load_ext autoreload
%autoreload 2
%matplotlib inline

/Users/hmamin/anaconda3/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)

# Used in notebook but not needed in package.
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader

from htools import assert_raises

/Users/hmamin/anaconda3/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)

Model

BaseModel allows models to freeze/unfreeze layers and provides several methods for weight diagnostics. It should not be instantiated directly, but used as a parent class for a model. Like all PyTorch models, its children will still need to call super().__init__() and implement a forward() method.

class BaseModel[source]

BaseModel() :: Module

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in
a tree structure. You can assign the submodules as regular attributes::

    import torch.nn as nn
    import torch.nn.functional as F

    class Model(nn.Module):
        def __init__(self):
            super(Model, self).__init__()
            self.conv1 = nn.Conv2d(1, 20, 5)
            self.conv2 = nn.Conv2d(20, 20, 5)

        def forward(self, x):
            x = F.relu(self.conv1(x))
            return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their
parameters converted too when you call :meth:`to`, etc.

:ivar training: Boolean represents whether this module is in training or
                evaluation mode.
:vartype training: bool

class SimpleModel(BaseModel):
    
    def __init__(self, dim):
        super().__init__()  
        self.fc1 = nn.Linear(dim, 2)
        self.fc2 = nn.Linear(2, 1)
        
    def forward(self, x):
        x = F.leaky_relu(self.fc1(x))
        return self.fc2(x)

class GroupedModel(BaseModel):
    
    def __init__(self, dim):
        super().__init__()  
        g1 = nn.Sequential(
             nn.Linear(dim, 8),
             nn.LeakyReLU(),
             nn.Linear(8, 4),
             nn.LeakyReLU()
        )
        g2 = nn.Linear(4, 1)
        self.groups = nn.ModuleList([g1, g2])
        
    def forward(self, x):
        for group in self.groups:
            x = group(x)
        return x

snet = SimpleModel(2)
snet.freeze()
for n in range(5):
    snet.unfreeze(n_layers=n)
    unfrozen = [x[1] for x in snet.trainable()]
    print('Unfrozen', unfrozen)
    assert sum(unfrozen) == n
    assert not any(unfrozen[:-n])

Unfrozen [False, False, False, False]
Unfrozen [False, False, False, True]
Unfrozen [False, False, True, True]
Unfrozen [False, True, True, True]
Unfrozen [True, True, True, True]

snet.freeze()
with assert_raises(AttributeError) as ar:
    for n in range(3):
        snet.unfreeze(n_groups=n)

As expected, got AttributeError('SimpleModel' object has no attribute 'groups').

gnet = GroupedModel(2)
gnet.freeze()
n_unfrozen = [0, 2, 6]
for n, nu in zip(range(3), n_unfrozen):
    gnet.unfreeze(n_groups=n)
    unfrozen = [x[1] for x in gnet.trainable()]
    print('Unfrozen', unfrozen)
    assert sum(unfrozen) == nu

Unfrozen [False, False, False, False, False, False]
Unfrozen [False, False, False, False, True, True]
Unfrozen [True, True, True, True, True, True]

gnet.freeze()
for n in range(7):
    gnet.unfreeze(n_layers=n)
    unfrozen = [x[1] for x in gnet.trainable()]
    print('Unfrozen', unfrozen)
    assert sum(unfrozen) == n
    assert not any(unfrozen[:-n])

Unfrozen [False, False, False, False, False, False]
Unfrozen [False, False, False, False, False, True]
Unfrozen [False, False, False, False, True, True]
Unfrozen [False, False, False, True, True, True]
Unfrozen [False, False, True, True, True, True]
Unfrozen [False, True, True, True, True, True]
Unfrozen [True, True, True, True, True, True]

optim = variable_lr_optimizer(snet, 2e-3)
print(optim)

with assert_raises(ValueError) as ar:
    optim = variable_lr_optimizer(snet, [3e-3, 1e-1])
    optim

Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 0.001
    lr: 0.002
    weight_decay: 0
)
As expected, got ValueError(Received more learning rates than layer groups.).

update_optimizer(optim, 1e-3, 0.5)
assert len(optim.param_groups) == 1
assert optim.param_groups[0]['lr'] == 1e-3

lrs = [1e-3, 3e-3]
optim = variable_lr_optimizer(gnet, lrs)
print(optim)
assert [group['lr'] for group in optim.param_groups] == lrs

update_optimizer(optim, 2e-3, lr_mult=1/3)
print([group['lr'] for group in optim.param_groups])
assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 3)

Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 0.001
    lr: 0.001
    weight_decay: 0

Parameter Group 1
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 0.001
    lr: 0.003
    weight_decay: 0
)
[0.0006666666666666666, 0.002]

optim = variable_lr_optimizer(gnet, 1e-3, lr_mult=0.5)
print([group['lr'] for group in optim.param_groups])
assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 2)

[0.0005, 0.001]

optim = variable_lr_optimizer(snet, 2e-3)
print(optim)

with assert_raises(ValueError) as ar:
    optim = variable_lr_optimizer(snet, [3e-3, 1e-1])
    optim

Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 0.001
    lr: 0.002
    weight_decay: 0
)
As expected, got ValueError(Received more learning rates than layer groups.).

update_optimizer(optim, 1e-3, 0.5)
assert len(optim.param_groups) == 1
assert optim.param_groups[0]['lr'] == 1e-3

lrs = [1e-3, 3e-3]
optim = variable_lr_optimizer(gnet, lrs)
print(optim)
assert [group['lr'] for group in optim.param_groups] == lrs

update_optimizer(optim, 2e-3, lr_mult=1/3)
print([group['lr'] for group in optim.param_groups])
assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 3)

Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 0.001
    lr: 0.001
    weight_decay: 0

Parameter Group 1
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 0.001
    lr: 0.003
    weight_decay: 0
)
[0.0006666666666666666, 0.002]

optim = variable_lr_optimizer(gnet, 1e-3, lr_mult=0.5)
print([group['lr'] for group in optim.param_groups])
assert np.isclose(optim.param_groups[1]['lr'], optim.param_groups[0]['lr'] * 2)

[0.0005, 0.001]

Trainer

handle_interrupt[source]

handle_interrupt(meth)

Decorator for Trainer.fit() method that allows the user to
interrupt training with Ctrl+c while still running the
`on_train_end` method for each of its callbacks. Without this,
cutting training short would lose that functionality, so we
couldn't do things like uploading to S3.

Arguments
---------
meth: callable
    The method to decorate.

Returns
-------
callable: The wrapped method.

class Trainer[source]

Trainer(net, dl_train, dl_val, criterion, mode:('binary', 'multiclass', 'regression'), out_dir, optim=None, optim_type='Adam', eps=0.001, last_act=None, threshold=0.5, metrics=None, callbacks=None, device=device(type='cpu')) :: LoggerMixin

Mixin class that configures and returns a logger.

Examples
--------
class Foo(LoggerMixin):

    def __init__(self, a, log_file):
        self.a = a
        self.log_file = log_file
        self.logger = self.get_logger(log_file)

    def walk(self, location):
        self.logger.info(f'walk received argument {location}')
        return f'walking to {location}'

class PredictionExaminer[source]

PredictionExaminer(trainer)

Examine model predictions. This lets us view rows where the model was
very confidently wrong, barely wrong, confidently right, barely right, or
just random rows.