[K3AE4] - Denoiser and classifier model

Episode 4 : Construction of a denoiser and classifier model

Objectives :

• Building a multiple output model, able to denoise and classify
• Understanding a more advanced programming model

The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
The use of a GPU is often indispensable.

What we're going to do :

• Defining a multiple output model using Keras procedural programing model
• Build the model
• Train it
• Follow the learning process

Data Terminology :

• clean_train, clean_test for noiseless images
• noisy_train, noisy_test for noisy images
• class_train, class_test for the classes to which the images belong
• denoised_test for denoised images at the output of the model
• classcat_test for class prediction in model output (is a softmax)
• classid_test class prediction (ie: argmax of classcat_test)

Step 1 - Init python stuff

1.1 - Init

import os
os.environ['KERAS_BACKEND'] = 'torch'

import keras

import numpy as np
from skimage import io
import random

from modules.AE4_builder    import AE4_builder
from modules.MNIST          import MNIST
from modules.ImagesCallback import ImagesCallback

import fidle

# Init Fidle environment
run_id, run_dir, datasets_dir = fidle.init('K3AE4')

FIDLE - Environment initialization

Version              : 2.3.0
Run id               : K3AE4
Run dir              : ./run/K3AE4
Datasets dir         : /gpfswork/rech/mlh/uja62cb/fidle-project/datasets-fidle
Start time           : 03/03/24 21:11:56
Hostname             : r6i0n6 (Linux)
Tensorflow log level : Warning + Error  (=1)
Update keras cache   : False
Update torch cache   : False
Save figs            : ./run/K3AE4/figs (True)
keras                : 3.0.4
numpy                : 1.24.4
sklearn              : 1.3.2
yaml                 : 6.0.1
skimage              : 0.22.0
matplotlib           : 3.8.2
pandas               : 2.1.3
torch                : 2.1.1

1.2 - Parameters

prepared_dataset : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
dataset_seed : Random seed for shuffling dataset. 'None' mean using /dev/urandom
scale : % of the dataset to use (1. for 100%)
latent_dim : Dimension of the latent space
train_prop : Percentage for train (the rest being for the test) batch_size : Batch size
epochs : Nb of epochs for training
fit_verbosity is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch

scale=0.1, epochs=20 => 2' on a laptop

prepared_dataset = './data/mnist-noisy.h5'
dataset_seed     = None

scale            = .1

train_prop       = .8
batch_size       = 128
epochs           = 10
fit_verbosity    = 1

Override parameters (batch mode) - Just forget this cell

fidle.override('prepared_dataset', 'dataset_seed', 'scale')
fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')

** Overrided parameters : **
scale                : 1
** Overrided parameters : **
epochs               : 20

Step 2 - Retrieve dataset

With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)

clean_train,clean_test, noisy_train,noisy_test, class_train,class_test = MNIST.reload_prepared_dataset(
                                                                                    scale      = scale, 
                                                                                    train_prop = train_prop,
                                                                                    seed       = dataset_seed,
                                                                                    shuffle    = True,
                                                                                    filename   = prepared_dataset )

Loaded.
rescaled (1).
Seeded (None)
Shuffled.
splited (0.8).
clean_train shape is :  (56000, 28, 28, 1)
clean_test  shape is :  (14000, 28, 28, 1)
noisy_train shape is :  (56000, 28, 28, 1)
noisy_test  shape is :  (14000, 28, 28, 1)
class_train shape is :  (56000,)
class_test  shape is :  (14000,)
Blake2b digest is    :  55a32efaea5b81b1cd9b

Step 3 - Build models

builder = AE4_builder( ae={ 'latent_dim':10 }, cnn = { 'lc1':8, 'lc2':16, 'ld':100 } )

model = builder.create_model()

model.compile(optimizer='rmsprop', 
              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
              loss_weights={'ae':1., 'classifier':1.},
              metrics={'classifier':'accuracy'} )

# keras.utils.plot_model(model, "multi_input_and_output_model.png", show_shapes=True)

Step 4 - Train

20' on a CPU
1'12 on a GPU (V100, IDRIS)

# ---- Callback : Images
#
fidle.utils.mkdir( run_dir + '/images')
filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'

encoder = model.get_layer('ae').get_layer('encoder')
decoder = model.get_layer('ae').get_layer('decoder')

callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)

chrono = fidle.Chrono()
chrono.start()

history = model.fit(noisy_train, [clean_train, class_train],
                 batch_size      = batch_size,
                 epochs          = epochs,
                 verbose         = fit_verbosity,
                 validation_data = (noisy_test, [clean_test, class_test]),
                 callbacks       = [ callback_images ]  )

chrono.show()

Epoch 1/20

/gpfswork/rech/mlh/uja62cb/local/fidle-k3/lib/python3.11/site-packages/keras/src/backend/common/backend_utils.py:88: UserWarning: You might experience inconsistencies accross backends when calling conv transpose with kernel_size=3, stride=2, dilation_rate=1, padding=same, output_padding=1.
  warnings.warn(

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 21s 42ms/step - classifier_accuracy: 0.3181 - loss: 2.2338 - val_classifier_accuracy: 0.7614 - val_loss: 1.0077
Epoch 2/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.6843 - loss: 1.1433 - val_classifier_accuracy: 0.8201 - val_loss: 0.7553
Epoch 3/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.7423 - loss: 0.9513 - val_classifier_accuracy: 0.8395 - val_loss: 0.6768
Epoch 4/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.7737 - loss: 0.8651 - val_classifier_accuracy: 0.8444 - val_loss: 0.6433
Epoch 5/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.7852 - loss: 0.8223 - val_classifier_accuracy: 0.8541 - val_loss: 0.6199
Epoch 6/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.7911 - loss: 0.7952 - val_classifier_accuracy: 0.8532 - val_loss: 0.6100
Epoch 7/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.7998 - loss: 0.7654 - val_classifier_accuracy: 0.8624 - val_loss: 0.5909
Epoch 8/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8041 - loss: 0.7550 - val_classifier_accuracy: 0.8620 - val_loss: 0.5810
Epoch 9/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8067 - loss: 0.7476 - val_classifier_accuracy: 0.8651 - val_loss: 0.5702
Epoch 10/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8102 - loss: 0.7336 - val_classifier_accuracy: 0.8651 - val_loss: 0.5745
Epoch 11/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8134 - loss: 0.7207 - val_classifier_accuracy: 0.8674 - val_loss: 0.5683
Epoch 12/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 42ms/step - classifier_accuracy: 0.8123 - loss: 0.7251 - val_classifier_accuracy: 0.8659 - val_loss: 0.5666
Epoch 13/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.8147 - loss: 0.7266 - val_classifier_accuracy: 0.8699 - val_loss: 0.5603
Epoch 14/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.8167 - loss: 0.7173 - val_classifier_accuracy: 0.8701 - val_loss: 0.5613
Epoch 15/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 18s 42ms/step - classifier_accuracy: 0.8196 - loss: 0.7014 - val_classifier_accuracy: 0.8716 - val_loss: 0.5595
Epoch 16/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8193 - loss: 0.7087 - val_classifier_accuracy: 0.8697 - val_loss: 0.5514
Epoch 17/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8216 - loss: 0.6949 - val_classifier_accuracy: 0.8724 - val_loss: 0.5464
Epoch 18/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8230 - loss: 0.6989 - val_classifier_accuracy: 0.8718 - val_loss: 0.5516
Epoch 19/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8223 - loss: 0.6912 - val_classifier_accuracy: 0.8726 - val_loss: 0.5433
Epoch 20/20
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 19s 43ms/step - classifier_accuracy: 0.8237 - loss: 0.6959 - val_classifier_accuracy: 0.8726 - val_loss: 0.5520
Duration :  373.47 seconds

Save model weights

os.makedirs(f'{run_dir}/models', exist_ok=True)

model.save_weights(f'{run_dir}/models/model.weights.h5')

Step 5 - History

fidle.scrawler.history(history,  plot={'Loss':['loss', 'val_loss'],
                                 'Accuracy':['classifier_accuracy','val_classifier_accuracy']}, save_as='01-history')

Saved: ./run/K3AE4/figs/01-history_0

Saved: ./run/K3AE4/figs/01-history_1

Step 6 - Denoising progress

imgs=[]
for epoch in range(0,epochs,4):
    for i in range(5):
        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
        img      = io.imread(filename)
        imgs.append(img)      

fidle.utils.subtitle('Real images (clean_test) :')
fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')

fidle.utils.subtitle('Noisy images (noisy_test) :')
fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')

fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')

fidle.utils.subtitle('Noisy images (noisy_test) :')
fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)

fidle.utils.subtitle('Real images (clean_test) :')
fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)

Real images (clean_test) :

Saved: ./run/K3AE4/figs/02-original-real

Noisy images (noisy_test) :

Saved: ./run/K3AE4/figs/03-original-noisy

Evolution during the training period (denoised_test) :

Saved: ./run/K3AE4/figs/04-learning

Noisy images (noisy_test) :

Real images (clean_test) :

Step 7 - Evaluation

Note : We will use the following data:
clean_train, clean_test for noiseless images
noisy_train, noisy_test for noisy images
class_train, class_test for the classes to which the images belong
denoised_test for denoised images at the output of the model
classcat_test for class prediction in model output (is a softmax)
classid_test class prediction (ie: argmax of classcat_test)

7.1 - Reload our model (weights)

builder = AE4_builder( ae={ 'latent_dim':10 }, cnn = { 'lc1':8, 'lc2':16, 'ld':100 } )

model = builder.create_model()

model.load_weights(f'{run_dir}/models/model.weights.h5')

7.2 - Let's make a prediction

Note that our model will returns 2 outputs : denoised images from output 1 and class prediction from output 2

outputs = model.predict(noisy_test, verbose=0)

denoised = outputs['ae']
classcat = outputs['classifier']

print('Denoised images   (denoised_test) shape : ', denoised.shape)
print('Predicted classes (classcat_test) shape : ', classcat.shape)

Denoised images   (denoised_test) shape :  (14000, 28, 28, 1)
Predicted classes (classcat_test) shape :  (14000, 10)

7.3 - Denoised images

i=random.randint(0,len(denoised)-8)
j=i+8

fidle.utils.subtitle('Noisy test images (input):')
fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')

fidle.utils.subtitle('Denoised images (output):')
fidle.scrawler.images(denoised[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')

fidle.utils.subtitle('Real test images :')
fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')

Noisy test images (input):

Saved: ./run/K3AE4/figs/05-test-noisy

Denoised images (output):

Saved: ./run/K3AE4/figs/06-test-predict

Real test images :

Saved: ./run/K3AE4/figs/07-test-real

7.4 - Class prediction

Note: The evaluation requires the noisy images as input (noisy_test) and the 2 expected outputs:

• the images without noise (clean_test)
• the classes (class_test)

# We need to (re)compile our resurrected model (to specify loss and metrics)
#
model.compile(optimizer='rmsprop', 
              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
              loss_weights={'ae':1., 'classifier':1.},
              metrics={'classifier':'accuracy'} )


# Get an evaluation
#
score = model.evaluate(noisy_test, [clean_test, class_test], verbose=0)

# And show results
#
fidle.utils.subtitle("Accuracy :")
print(f'Classification accuracy : {score[1]:4.4f}')

fidle.utils.subtitle("Few examples :")
classid_test  = np.argmax(classcat, axis=-1)
fidle.scrawler.images(noisy_test, class_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=classid_test, save_as='04-predictions')

Accuracy :

Classification accuracy : 0.8726

Few examples :

Saved: ./run/K3AE4/figs/04-predictions

fidle.end()

End time : 03/03/24 21:18:48
Duration : 00:06:52 302ms
This notebook ends here :-)
https://fidle.cnrs.fr

---