[K3LADYB1] - Prediction of a 2D trajectory via RNN

Artificial dataset generation and prediction attempt via a recurrent network, using Keras 3 and PyTorch

Objectives :

• Understanding the use of a recurrent neural network

What we're going to do :

• Generate an artificial dataset
• dataset preparation
• Doing our testing
• Making predictions

Step 1 - Import and init

1.1 - Python

import os
os.environ['KERAS_BACKEND'] = 'torch'

import keras
import numpy as np
from math import cos, sin
import random

import fidle

# Init Fidle environment
run_id, run_dir, datasets_dir = fidle.init('K3LADYB1')

FIDLE - Environment initialization

Version              : 2.3.0
Run id               : K3LADYB1
Run dir              : ./run/K3LADYB1
Datasets dir         : /gpfswork/rech/mlh/uja62cb/fidle-project/datasets-fidle
Start time           : 03/03/24 21:07:18
Hostname             : r3i7n8 (Linux)
Tensorflow log level : Warning + Error  (=1)
Update keras cache   : False
Update torch cache   : False
Save figs            : ./run/K3LADYB1/figs (True)
keras                : 3.0.4
numpy                : 1.24.4
sklearn              : 1.3.2
yaml                 : 6.0.1
matplotlib           : 3.8.2
pandas               : 2.1.3
torch                : 2.1.1

1.2 - Parameters

# ---- About dataset
#
max_t        = 1000
delta_t      = 0.01
features_len = 2


sequence_len = 20
predict_len  = 5

# ---- About training
#
scale         = .2       # Percentage of dataset to be used (1=all)
train_prop    = .8       # Percentage for train (the rest being for the test)
batch_size    = 32
epochs        = 5
fit_verbosity = 1        # 0 = silent, 1 = progress bar, 2 = one line per epoch

Override parameters (batch mode) - Just forget this cell

fidle.override('scale', 'train_prop', 'sequence_len', 'predict_len', 'batch_size', 'epochs')

** Overrided parameters : **
scale                : 1
train_prop           : 0.8
sequence_len         : 20
predict_len          : 5
batch_size           : 32
epochs               : 10

Step 2 - Generation of a fun dataset

2.1 - Virtual trajectory of our ladybug

def ladybug_init(s=122):
    
    if s>0 : random.seed(s)
    ladybug_init.params_x = [ random.gauss(0.,1.) for u in range(8)]
    ladybug_init.params_y = [ random.gauss(0.,1.) for u in range(8)]
    
def ladybug_move(t):

    [ax1, ax2, ax3, ax4, kx1, kx2, kx3, kx4] = ladybug_init.params_x
    [ay1, ay2, ay3, ay4, ky1, ky2, ky3, ky4] = ladybug_init.params_y
    
    x = ax1*sin(t*(kx1+20)) + ax2*cos(t*(kx2+10)) + ax3*sin(t*(kx3+5)) + ax4*cos(t*(kx4+5))
    y = ay1*cos(t*(ky1+20)) + ay2*sin(t*(ky2+10)) + ay3*cos(t*(ky3+5)) + ay4*sin(t*(ky4+5)) 

    return x,y

2.2 - Get some positions, and build a rescaled and normalized dataset

# ---- Get positions
#
ladybug_init(s=16)
x,y = 0,0
positions=[]
for t in np.arange(0., max_t, delta_t):
    x,y = ladybug_move(t)
    positions.append([x,y])

# ---- Build rescaled dataset
#
n = int( len(positions)*scale )
dataset = np.array(positions[:n])

k = int(len(dataset)*train_prop)
x_train = dataset[:k]
x_test  = dataset[k:]

# ---- Normalize
#
mean = x_train.mean()
std  = x_train.std()
x_train = (x_train - mean) / std
x_test  = (x_test  - mean) / std

print("Dataset generated.")
print("Train shape is : ", x_train.shape)
print("Test  shape is : ", x_test.shape)

Dataset generated.
Train shape is :  (80000, 2)
Test  shape is :  (20000, 2)

2.3 - Have a look

An extract from the data we have: the virtual trajectory of our ladybug
And what we want to predict (in red), from a segment (in blue)

fidle.scrawler.serie_2d(x_train[:1000], figsize=(12,12), lw=1,ms=4,save_as='01-dataset')

Saved: ./run/K3LADYB1/figs/01-dataset

k1,k2 = sequence_len, predict_len
i = random.randint(0,len(x_test)-k1-k2)
j = i+k1

fidle.scrawler.segment_2d( x_test[i:j+k2], x_test[j:j+k2],ms=6, save_as='02-objectives')

Saved: ./run/K3LADYB1/figs/02-objectives

2.4 - Prepare sequences from datasets

# ---- Create sequences and labels for train and test
#
xs_train, ys_train=[],[]
all_i = np.random.permutation( len(x_train) - sequence_len - 1 )

for i in all_i:
    xs_train.append( x_train[ i : i+sequence_len ] )
    ys_train.append( x_train[ i+sequence_len+1 ]   )
    
xs_test, ys_test=[],[]
for i in range( len(x_test) - sequence_len - 1):
    xs_test.append( x_test[ i : i+sequence_len ] )
    ys_test.append( x_test[ i+sequence_len+1 ]   )

# ---- Convert to numpy / float16
    
xs_train = np.array(xs_train, dtype='float16')
ys_train = np.array(ys_train, dtype='float16')
xs_test  = np.array(xs_test,  dtype='float16')
ys_test  = np.array(ys_test,  dtype='float16')

fidle.utils.subtitle('About the splitting of our dataset :')

print('Number of sequences : ', len(xs_train))
print('xs_train shape      : ',xs_train.shape)
print('ys_train shape      : ',ys_train.shape)

fidle.utils.subtitle('What an xs look like :')
fidle.utils.np_print(xs_train[10] )
fidle.utils.subtitle('What an ys look like :')
fidle.utils.np_print(ys_train[10])

About the splitting of our dataset :

Number of sequences :  79979
xs_train shape      :  (79979, 20, 2)
ys_train shape      :  (79979, 2)

What an xs look like :

[[ 0.052 -0.497]
 [ 0.077 -0.357]
 [ 0.086 -0.199]
 [ 0.081 -0.028]
 [ 0.067  0.149]
 [ 0.047  0.324]
 [ 0.024  0.49 ]
 [ 0.003  0.642]
 [-0.013  0.771]
 [-0.019  0.874]
 [-0.013  0.946]
 [ 0.009  0.984]
 [ 0.049  0.986]
 [ 0.109  0.952]
 [ 0.188  0.883]
 [ 0.288  0.782]
 [ 0.406  0.651]
 [ 0.541  0.498]
 [ 0.69   0.325]
 [ 0.85   0.142]]

What an ys look like :

[ 1.185 -0.23 ]

Step 3 - Create a model

model = keras.models.Sequential()
model.add( keras.layers.InputLayer(shape=(sequence_len, features_len)) )
model.add( keras.layers.GRU(200, return_sequences=False, activation='relu') )
model.add( keras.layers.Dense(features_len) )

model.summary()

model.compile(optimizer='rmsprop', 
              loss='mse', 
              metrics   = ['mae'] )

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape              ┃    Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ gru (GRU)                       │ (None, 200)               │    122,400 │
├─────────────────────────────────┼───────────────────────────┼────────────┤
│ dense (Dense)                   │ (None, 2)                 │        402 │
└─────────────────────────────────┴───────────────────────────┴────────────┘

 Total params: 122,802 (479.70 KB)

 Trainable params: 122,802 (479.70 KB)

 Non-trainable params: 0 (0.00 B)

Step 4 - Train the model

4.1 Add Callbacks

os.makedirs(f'{run_dir}/models',   mode=0o750, exist_ok=True)
save_dir = f'{run_dir}/models/best_model.keras'

savemodel_callback = keras.callbacks.ModelCheckpoint( filepath=save_dir, monitor='val_mae', mode='max', save_best_only=True)

4.2 - Train it

Need 3' on a cpu laptop

chrono=fidle.Chrono()
chrono.start()

history=model.fit(xs_train,ys_train,
                  epochs  = epochs, 
                  verbose = fit_verbosity,
                  validation_data = (xs_test, ys_test),
                  callbacks = [savemodel_callback])

chrono.show()

Epoch 1/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 0.0452 - mae: 0.1203 - val_loss: 0.0019 - val_mae: 0.0350
Epoch 2/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 0.0014 - mae: 0.0287 - val_loss: 0.0017 - val_mae: 0.0343
Epoch 3/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 8.9036e-04 - mae: 0.0230 - val_loss: 8.2342e-04 - val_mae: 0.0204
Epoch 4/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 6.7123e-04 - mae: 0.0200 - val_loss: 3.1158e-04 - val_mae: 0.0134
Epoch 5/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 5.3428e-04 - mae: 0.0179 - val_loss: 0.0011 - val_mae: 0.0248
Epoch 6/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 4.5709e-04 - mae: 0.0164 - val_loss: 5.6828e-04 - val_mae: 0.0201
Epoch 7/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 3.9540e-04 - mae: 0.0153 - val_loss: 1.1927e-04 - val_mae: 0.0084
Epoch 8/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 3.5160e-04 - mae: 0.0145 - val_loss: 1.0853e-04 - val_mae: 0.0084
Epoch 9/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 3.1917e-04 - mae: 0.0137 - val_loss: 5.1418e-04 - val_mae: 0.0191
Epoch 10/10
2500/2500 ━━━━━━━━━━━━━━━━━━━━ 105s 42ms/step - loss: 2.9676e-04 - mae: 0.0133 - val_loss: 1.5461e-04 - val_mae: 0.0098
Duration :  1047.59 seconds

fidle.scrawler.history(history,plot={'loss':['loss','val_loss'], 'mae':['mae','val_mae']}, save_as='03-history')

Saved: ./run/K3LADYB1/figs/03-history_0

Saved: ./run/K3LADYB1/figs/03-history_1

Step 5 - Predict

5.1 - Load model

loaded_model = keras.models.load_model(f'{run_dir}/models/best_model.keras')
print('Loaded.')

Loaded.

5.2 - Make a 1-step prediction

A simple prediction on a single iteration

s=random.randint(0,len(x_test)-sequence_len)

sequence      = x_test[s:s+sequence_len]
sequence_true = x_test[s:s+sequence_len+1]

sequence_pred = loaded_model.predict( np.array([sequence]), verbose=fit_verbosity )

print('sequence shape      :',sequence.shape)
print('sequence true shape :',sequence_true.shape)
print('sequence pred shape :',sequence_pred.shape)

fidle.scrawler.segment_2d(sequence_true, sequence_pred, save_as='04-one-step-prediction')
fidle.scrawler.multivariate_serie(sequence_true, predictions=sequence_pred, labels=['Axis=0', 'Axis=1'],save_as='05-one-step-prediction-2axis')

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 40ms/step
sequence shape      : (20, 2)
sequence true shape : (21, 2)
sequence pred shape : (1, 2)

Saved: ./run/K3LADYB1/figs/04-one-step-prediction

Saved: ./run/K3LADYB1/figs/05-one-step-prediction-2axis

5.3 - Make n-steps prediction

A longer term prediction, via a nice iteration function
We will perform predictions to iteratively build our prediction.

def get_prediction(dataset, model, iterations=4):

    # ---- Initial sequence
    #
    s=random.randint(0,len(dataset)-sequence_len-iterations)

    sequence_pred = dataset[s:s+sequence_len].copy()
    sequence_true = dataset[s:s+sequence_len+iterations].copy()

    # ---- Iterate 
    #
    sequence_pred = list(sequence_pred)

    for i in range(iterations):
        sequence   = sequence_pred[-sequence_len:]
        prediction = model.predict( np.array([sequence]), verbose=fit_verbosity )
        sequence_pred.append(prediction[0])

    # ---- Extract the predictions    
    #
    prediction = np.array(sequence_pred[-iterations:])

    return sequence_true,prediction

An n-steps prediction :

sequence_true, sequence_pred = get_prediction(x_test, loaded_model, iterations=5)

fidle.scrawler.segment_2d(sequence_true, sequence_pred, ms=8, save_as='06-n-steps-prediction-norm')
fidle.scrawler.multivariate_serie(sequence_true, predictions=sequence_pred, hide_ticks=True, labels=['Axis=0', 'Axis=1'],save_as='07-n-steps-prediction-norm')

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 17ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 15ms/step

Saved: ./run/K3LADYB1/figs/06-n-steps-prediction-norm

Saved: ./run/K3LADYB1/figs/07-n-steps-prediction-norm

fidle.end()

End time : 03/03/24 21:24:50
Duration : 00:17:33 626ms
This notebook ends here :-)
https://fidle.cnrs.fr

---