[GRAD1] - Linear regression with gradient descent¶
Low level implementation of a solution by gradient descent. Basic and stochastic approach.Objectives :¶
- To illustrate the iterative approach of a gradient descent
What we're going to do :¶
Equation : $ Y = X.\Theta + N$
Where N is a noise vector
and $\Theta = (a,b)$ a vector as y = a.x + b
We will calculate a loss function and its gradient.
We will descend this gradient in order to find a minimum value of our loss function.
$ \triangledown_\theta MSE(\Theta)=\begin{bmatrix} \frac{\partial}{\partial \theta_0}MSE(\Theta)\\ \frac{\partial}{\partial \theta_1}MSE(\Theta)\\ \vdots\\ \frac{\partial}{\partial \theta_n}MSE(\Theta) \end{bmatrix}=\frac2m X^T\cdot(X\cdot\Theta-Y) $
and :
$\Theta \leftarrow \Theta - \eta \cdot \triangledown_\theta MSE(\Theta)$
where $\eta$ is the learning rate
Step 1 - Import and init¶
import numpy as np
import sys
import fidle
from modules.RegressionCooker import RegressionCooker
# Init Fidle environment
#
run_id, run_dir, datasets_dir = fidle.init('GRAD1')
# ---- Instanciate a Regression Cooker
#
cooker = RegressionCooker(fidle)
FIDLE - Environment initialization
Version : 2.3.0 Run id : GRAD1 Run dir : ./run/GRAD1 Datasets dir : /gpfswork/rech/mlh/uja62cb/fidle-project/datasets-fidle Start time : 03/03/24 21:02:59 Hostname : r3i6n3 (Linux) Tensorflow log level : Info + Warning + Error (=0) Update keras cache : False Update torch cache : False Save figs : ./run/GRAD1/figs (True) numpy : 1.24.4 sklearn : 1.3.2 yaml : 6.0.1 matplotlib : 3.8.2 pandas : 2.1.3
FIDLE 2020 - Regression Cooker
Version : 0.1 Run time : Sunday 03 March 2024, 21:02:59
Step 2 - Get a dataset¶
X,Y = cooker.get_dataset(1000000)
cooker.plot_dataset(X,Y)
Dataset :¶
X shape : (1000000, 1) Y shape : (1000000, 1) plot : 1000 points
X : mean= 5.000 std= 2.886 min= 0.000 max= 10.000 Y : mean= -32.046 std= 36.285 min=-187.090 max= 134.849
Step 3 : Data normalization¶
X_norm = ( X - X.mean() ) / X.std()
Y_norm = ( Y - Y.mean() ) / Y.std()
cooker.vector_infos('X origine',X)
cooker.vector_infos('X normalized',X_norm)
X origine : mean= 5.000 std= 2.886 min= 0.000 max= 10.000 X normalized : mean= 0.000 std= 1.000 min= -1.732 max= 1.732
Step 4 - Basic descent¶
theta = cooker.basic_descent(X_norm, Y_norm, epochs=200, eta=0.01)
Basic gradient descent :¶
With :
with :
epochs = 200
eta = 0.01
epochs :
#i Loss Gradient Theta
0 +19.258 -8.546 +1.273 -4.187 -0.013
20 +8.913 -5.705 +0.850 -2.796 -0.220
40 +4.302 -3.809 +0.567 -1.866 -0.359
60 +2.247 -2.543 +0.379 -1.246 -0.451
80 +1.331 -1.698 +0.253 -0.832 -0.513
100 +0.923 -1.133 +0.169 -0.555 -0.554
120 +0.741 -0.757 +0.113 -0.371 -0.581
140 +0.660 -0.505 +0.075 -0.248 -0.600
160 +0.624 -0.337 +0.050 -0.165 -0.612
180 +0.608 -0.225 +0.034 -0.110 -0.620
200 +0.600 -0.150 +0.022 -0.074 -0.626
Visualization :
Loss :
Step 5 - Minibatch descent¶
theta = cooker.minibatch_descent(X_norm, Y_norm, epochs=10, batchs=20, batch_size=10, eta=0.01)
Mini batch gradient descent :¶
With :
with :
epochs = 10
batchs = 20
batch size = 10
eta = 0.01
epochs :
#i Loss Gradient Theta
0 +0.572 -6.435 +3.642 -0.011 -0.670
1 +0.899 -9.341 -0.417 -0.074 -0.673
2 +0.728 -6.996 +0.732 +0.028 -0.604
3 +0.595 +2.165 +4.488 -0.013 -0.601
4 +0.524 +5.066 +1.364 -0.031 -0.631
5 +0.340 +0.935 +4.302 +0.004 -0.603
6 +0.919 +0.448 -0.489 +0.033 -0.598
7 +0.826 -10.115 +7.453 +0.032 -0.634
8 +1.391 +7.885 -9.292 +0.017 -0.639
9 +0.535 +1.411 -4.537 +0.017 -0.613
Visualization :
Loss :
fidle.end()
End time : 03/03/24 21:03:03
Duration : 00:00:05 659ms
This notebook ends here :-)
https://fidle.cnrs.fr
