[K3AE5] - Advanced denoiser and classifier model

Episode 5 : Construction of an advanced denoiser and classifier model

Objectives :

• Building a multiple output model, able to denoise and classify
• Understanding a more complex advanced programming model

The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
The use of a GPU is often indispensable.

What we're going to do :

• Defining a multiple output model using Keras procedural programing model
• Build the model
• Train it
• Follow the learning process

Data Terminology :

• clean_train, clean_test for noiseless images
• noisy_train, noisy_test for noisy images
• class_train, class_test for the classes to which the images belong
• denoised_test for denoised images at the output of the model
• classcat_test for class prediction in model output (is a softmax)
• classid_test class prediction (ie: argmax of classcat_test)

Step 1 - Init python stuff

1.1 - Init

import os
os.environ['KERAS_BACKEND'] = 'torch'

import keras

import numpy as np
from skimage import io
import random

from modules.AE5_builder    import AE5_builder
from modules.MNIST          import MNIST
from modules.ImagesCallback import ImagesCallback

import fidle


# Init Fidle environment
run_id, run_dir, datasets_dir = fidle.init('K3AE5')

FIDLE - Environment initialization

Version              : 2.3.0
Run id               : K3AE5
Run dir              : ./run/K3AE5
Datasets dir         : /gpfswork/rech/mlh/uja62cb/fidle-project/datasets-fidle
Start time           : 03/03/24 21:13:30
Hostname             : r6i1n1 (Linux)
Tensorflow log level : Warning + Error  (=1)
Update keras cache   : False
Update torch cache   : False
Save figs            : ./run/K3AE5/figs (True)
keras                : 3.0.4
numpy                : 1.24.4
sklearn              : 1.3.2
yaml                 : 6.0.1
skimage              : 0.22.0
matplotlib           : 3.8.2
pandas               : 2.1.3
torch                : 2.1.1

1.2 - Parameters

prepared_dataset : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
dataset_seed : Random seed for shuffling dataset
scale : % of the dataset to use (1. for 100%)
latent_dim : Dimension of the latent space
train_prop : Percentage for train (the rest being for the test) batch_size : Batch size
epochs : Nb of epochs for training
fit_verbosity is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch

prepared_dataset = './data/mnist-noisy.h5'
dataset_seed     = None

scale            = .1

train_prop       = .8
batch_size       = 128
epochs           = 10
fit_verbosity    = 1

Override parameters (batch mode) - Just forget this cell

fidle.override('prepared_dataset', 'dataset_seed', 'scale')
fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')

** Overrided parameters : **
scale                : 1
** Overrided parameters : **
epochs               : 30

Step 2 - Retrieve dataset

With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)

clean_train,clean_test, noisy_train,noisy_test, class_train,class_test = MNIST.reload_prepared_dataset(
                                                                                    scale      = scale, 
                                                                                    train_prop = train_prop,
                                                                                    seed       = dataset_seed,
                                                                                    shuffle    = True,
                                                                                    filename   = prepared_dataset )

Loaded.
rescaled (1).
Seeded (None)
Shuffled.
splited (0.8).
clean_train shape is :  (56000, 28, 28, 1)
clean_test  shape is :  (14000, 28, 28, 1)
noisy_train shape is :  (56000, 28, 28, 1)
noisy_test  shape is :  (14000, 28, 28, 1)
class_train shape is :  (56000,)
class_test  shape is :  (14000,)
Blake2b digest is    :  6a924cdded42f36c5bd2

Step 3 - Build model

builder = AE5_builder( ae   = { 'latent_dim':10 }, 
                       cnn1 = { 'lc1':8,  'lc2':16, 'ld':100 }, 
                       cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 } )

model = builder.create_model()

model.compile(optimizer='rmsprop', 
              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
              loss_weights={'ae':1., 'classifier':1.},
              metrics={'classifier':'accuracy'} )

Step 4 - Train

20' on a CPU
1'30 on a GPU (V100, IDRIS)

# ---- Callback : Images
#
fidle.utils.mkdir( run_dir + '/images')
filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'

encoder = model.get_layer('ae').get_layer('encoder')
decoder = model.get_layer('ae').get_layer('decoder')

callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)

chrono = fidle.Chrono()
chrono.start()

history = model.fit(noisy_train, [clean_train, class_train],
                 batch_size      = batch_size,
                 epochs          = epochs,
                 verbose         = fit_verbosity,
                 validation_data = (noisy_test, [clean_test, class_test]),
                 callbacks       = [ callback_images ] ) 


chrono.show()

Epoch 1/30

/gpfswork/rech/mlh/uja62cb/local/fidle-k3/lib/python3.11/site-packages/keras/src/backend/common/backend_utils.py:88: UserWarning: You might experience inconsistencies accross backends when calling conv transpose with kernel_size=3, stride=2, dilation_rate=1, padding=same, output_padding=1.
  warnings.warn(

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.5006 - loss: 1.7636 - val_classifier_accuracy: 0.8238 - val_loss: 0.7497
Epoch 2/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.7972 - loss: 0.8180 - val_classifier_accuracy: 0.8598 - val_loss: 0.6047
Epoch 3/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8277 - loss: 0.7056 - val_classifier_accuracy: 0.8717 - val_loss: 0.5547
Epoch 4/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.8453 - loss: 0.6459 - val_classifier_accuracy: 0.8791 - val_loss: 0.5276
Epoch 5/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.8525 - loss: 0.6154 - val_classifier_accuracy: 0.8852 - val_loss: 0.5120
Epoch 6/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.8588 - loss: 0.5956 - val_classifier_accuracy: 0.8874 - val_loss: 0.5025
Epoch 7/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.8634 - loss: 0.5731 - val_classifier_accuracy: 0.8869 - val_loss: 0.4978
Epoch 8/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.8684 - loss: 0.5632 - val_classifier_accuracy: 0.8915 - val_loss: 0.4828
Epoch 9/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.8711 - loss: 0.5546 - val_classifier_accuracy: 0.8909 - val_loss: 0.4853
Epoch 10/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 45ms/step - classifier_accuracy: 0.8756 - loss: 0.5351 - val_classifier_accuracy: 0.8915 - val_loss: 0.4800
Epoch 11/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 21s 47ms/step - classifier_accuracy: 0.8761 - loss: 0.5316 - val_classifier_accuracy: 0.8892 - val_loss: 0.4850
Epoch 12/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 21s 48ms/step - classifier_accuracy: 0.8771 - loss: 0.5254 - val_classifier_accuracy: 0.8934 - val_loss: 0.4753
Epoch 13/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 23s 52ms/step - classifier_accuracy: 0.8794 - loss: 0.5177 - val_classifier_accuracy: 0.8946 - val_loss: 0.4723
Epoch 14/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 22s 51ms/step - classifier_accuracy: 0.8832 - loss: 0.5081 - val_classifier_accuracy: 0.8807 - val_loss: 0.5106
Epoch 15/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8786 - loss: 0.5122 - val_classifier_accuracy: 0.8909 - val_loss: 0.4782
Epoch 16/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8844 - loss: 0.5016 - val_classifier_accuracy: 0.8871 - val_loss: 0.4902
Epoch 17/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8838 - loss: 0.4998 - val_classifier_accuracy: 0.8945 - val_loss: 0.4729
Epoch 18/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8856 - loss: 0.4968 - val_classifier_accuracy: 0.8951 - val_loss: 0.4748
Epoch 19/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8873 - loss: 0.4882 - val_classifier_accuracy: 0.8933 - val_loss: 0.4807
Epoch 20/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8886 - loss: 0.4837 - val_classifier_accuracy: 0.8960 - val_loss: 0.4725
Epoch 21/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8896 - loss: 0.4774 - val_classifier_accuracy: 0.8926 - val_loss: 0.4781
Epoch 22/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8918 - loss: 0.4819 - val_classifier_accuracy: 0.8909 - val_loss: 0.4870
Epoch 23/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8933 - loss: 0.4729 - val_classifier_accuracy: 0.8933 - val_loss: 0.4754
Epoch 24/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8924 - loss: 0.4712 - val_classifier_accuracy: 0.8945 - val_loss: 0.4798
Epoch 25/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8909 - loss: 0.4693 - val_classifier_accuracy: 0.8935 - val_loss: 0.4838
Epoch 26/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8943 - loss: 0.4653 - val_classifier_accuracy: 0.8931 - val_loss: 0.4763
Epoch 27/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8967 - loss: 0.4570 - val_classifier_accuracy: 0.8959 - val_loss: 0.4721
Epoch 28/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8915 - loss: 0.4724 - val_classifier_accuracy: 0.8928 - val_loss: 0.4847
Epoch 29/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8928 - loss: 0.4699 - val_classifier_accuracy: 0.8930 - val_loss: 0.4792
Epoch 30/30
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
438/438 ━━━━━━━━━━━━━━━━━━━━ 20s 46ms/step - classifier_accuracy: 0.8935 - loss: 0.4650 - val_classifier_accuracy: 0.8924 - val_loss: 0.4805
Duration :  606.07 seconds

Save model weights

os.makedirs(f'{run_dir}/models', exist_ok=True)

model.save_weights(f'{run_dir}/models/model.weights.h5')

Step 5 - History

fidle.scrawler.history(history,  plot={'Loss':['loss', 'val_loss'],
                                 'Accuracy':['classifier_accuracy','val_classifier_accuracy']}, save_as='01-history')

Saved: ./run/K3AE5/figs/01-history_0

Saved: ./run/K3AE5/figs/01-history_1

Step 6 - Denoising progress

imgs=[]
for epoch in range(0,epochs,4):
    for i in range(5):
        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
        img      = io.imread(filename)
        imgs.append(img)      

fidle.utils.subtitle('Real images (clean_test) :')
fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')

fidle.utils.subtitle('Noisy images (noisy_test) :')
fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')

fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')

fidle.utils.subtitle('Noisy images (noisy_test) :')
fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)

fidle.utils.subtitle('Real images (clean_test) :')
fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)

Real images (clean_test) :

Saved: ./run/K3AE5/figs/02-original-real

Noisy images (noisy_test) :

Saved: ./run/K3AE5/figs/03-original-noisy

Evolution during the training period (denoised_test) :

Saved: ./run/K3AE5/figs/04-learning

Noisy images (noisy_test) :

Real images (clean_test) :

Step 7 - Evaluation

Note : We will use the following data:
clean_train, clean_test for noiseless images
noisy_train, noisy_test for noisy images
class_train, class_test for the classes to which the images belong
denoised_test for denoised images at the output of the model
classcat_test for class prediction in model output (is a softmax)
classid_test class prediction (ie: argmax of classcat_test)

7.1 - Reload our best model

builder = AE5_builder( ae   = { 'latent_dim':10 }, 
                       cnn1 = { 'lc1':8,  'lc2':16, 'ld':100 }, 
                       cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 } )

model = builder.create_model()

model.load_weights(f'{run_dir}/models/model.weights.h5')

7.2 - Let's make a prediction

Note that our model will returns 2 outputs : denoised images from output 1 and class prediction from output 2

outputs = model.predict(noisy_test, verbose=0)

denoised = outputs['ae']
classcat = outputs['classifier']

print('Denoised images   (denoised_test) shape : ', denoised.shape)
print('Predicted classes (classcat_test) shape : ', classcat.shape)

Denoised images   (denoised_test) shape :  (14000, 28, 28, 1)
Predicted classes (classcat_test) shape :  (14000, 10)

7.3 - Denoised images

i=random.randint(0,len(denoised)-8)
j=i+8

fidle.utils.subtitle('Noisy test images (input):')
fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')

fidle.utils.subtitle('Denoised images (output):')
fidle.scrawler.images(denoised[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')

fidle.utils.subtitle('Real test images :')
fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')

Noisy test images (input):

Saved: ./run/K3AE5/figs/05-test-noisy

Denoised images (output):

Saved: ./run/K3AE5/figs/06-test-predict

Real test images :

Saved: ./run/K3AE5/figs/07-test-real

7.4 - Class prediction

Note: The evaluation requires the noisy images as input (noisy_test) and the 2 expected outputs:

• the images without noise (clean_test)
• the classes (class_test)

# We need to (re)compile our resurrected model (to specify loss and metrics)
#
model.compile(optimizer='rmsprop', 
              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
              loss_weights={'ae':1., 'classifier':1.},
              metrics={'classifier':'accuracy'} )


# Get an evaluation
#
score = model.evaluate(noisy_test, [clean_test, class_test], verbose=0)

# And show results
#
fidle.utils.subtitle("Accuracy :")
print(f'Classification accuracy : {score[1]:4.4f}')

fidle.utils.subtitle("Few examples :")
classid_test  = np.argmax(classcat, axis=-1)
fidle.scrawler.images(noisy_test, class_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=classid_test, save_as='04-predictions')

Accuracy :

Classification accuracy : 0.8924

Few examples :

Saved: ./run/K3AE5/figs/04-predictions

fidle.end()

End time : 03/03/24 21:24:18
Duration : 00:10:48 843ms
This notebook ends here :-)
https://fidle.cnrs.fr

---