[GRAD1] - Linear regression with gradient descent¶
Low level implementation of a solution by gradient descent. Basic and stochastic approach.Objectives :¶
- To illustrate the iterative approach of a gradient descent
What we're going to do :¶
Equation : $ Y = X.\Theta + N$
Where N is a noise vector
and $\Theta = (a,b)$ a vector as y = a.x + b
We will calculate a loss function and its gradient.
We will descend this gradient in order to find a minimum value of our loss function.
$ \triangledown_\theta MSE(\Theta)=\begin{bmatrix} \frac{\partial}{\partial \theta_0}MSE(\Theta)\\ \frac{\partial}{\partial \theta_1}MSE(\Theta)\\ \vdots\\ \frac{\partial}{\partial \theta_n}MSE(\Theta) \end{bmatrix}=\frac2m X^T\cdot(X\cdot\Theta-Y) $
and :
$\Theta \leftarrow \Theta - \eta \cdot \triangledown_\theta MSE(\Theta)$
where $\eta$ is the learning rate
Step 1 - Import and init¶
import numpy as np
import sys
import fidle
from modules.RegressionCooker import RegressionCooker
# Init Fidle environment
#
run_id, run_dir, datasets_dir = fidle.init('GRAD1')
# ---- Instanciate a Regression Cooker
#
cooker = RegressionCooker(fidle)
FIDLE - Environment initialization
Version : 2.3.2 Run id : GRAD1 Run dir : ./run/GRAD1 Datasets dir : /lustre/fswork/projects/rech/mlh/uja62cb/fidle-project/datasets-fidle Start time : 22/12/24 21:20:37 Hostname : r3i6n0 (Linux) Tensorflow log level : Info + Warning + Error (=0) Update keras cache : False Update torch cache : False Save figs : ./run/GRAD1/figs (True) numpy : 2.1.2 sklearn : 1.5.2 yaml : 6.0.2 matplotlib : 3.9.2 pandas : 2.2.3
FIDLE 2020 - Regression Cooker
Version : 0.1 Run time : Sunday 22 December 2024, 21:20:37
Step 2 - Get a dataset¶
X,Y = cooker.get_dataset(1000000)
cooker.plot_dataset(X,Y)
Dataset :¶
X shape : (1000000, 1) Y shape : (1000000, 1) plot : 1000 points
X : mean= 5.005 std= 2.886 min= 0.000 max= 10.000 Y : mean= -36.064 std= 42.175 min=-223.947 max= 173.076
Step 3 : Data normalization¶
X_norm = ( X - X.mean() ) / X.std()
Y_norm = ( Y - Y.mean() ) / Y.std()
cooker.vector_infos('X origine',X)
cooker.vector_infos('X normalized',X_norm)
X origine : mean= 5.005 std= 2.886 min= 0.000 max= 10.000 X normalized : mean= -0.000 std= 1.000 min= -1.734 max= 1.730
Step 4 - Basic descent¶
theta = cooker.basic_descent(X_norm, Y_norm, epochs=200, eta=0.01)
Basic gradient descent :¶
With :
with :
epochs = 200
eta = 0.01
epochs :
#i Loss Gradient Theta
0 +20.845 -8.910 +0.957 -4.366 -0.010
20 +9.718 -5.948 +0.639 -2.915 -0.165
40 +4.759 -3.971 +0.426 -1.946 -0.269
60 +2.548 -2.651 +0.285 -1.299 -0.339
80 +1.563 -1.770 +0.190 -0.867 -0.385
100 +1.124 -1.182 +0.127 -0.579 -0.416
120 +0.928 -0.789 +0.085 -0.387 -0.437
140 +0.841 -0.527 +0.057 -0.258 -0.451
160 +0.802 -0.352 +0.038 -0.172 -0.460
180 +0.785 -0.235 +0.025 -0.115 -0.466
200 +0.777 -0.157 +0.017 -0.077 -0.470
Visualization :
Loss :
Step 5 - Minibatch descent¶
theta = cooker.minibatch_descent(X_norm, Y_norm, epochs=10, batchs=20, batch_size=10, eta=0.01)
Mini batch gradient descent :¶
With :
with :
epochs = 10
batchs = 20
batch size = 10
eta = 0.01
epochs :
#i Loss Gradient Theta
0 +0.199 -4.963 -3.053 +0.010 -0.504
1 +1.057 -1.864 -0.011 +0.036 -0.435
2 +0.871 -8.460 +1.489 +0.036 -0.443
3 +1.095 +0.758 +7.706 +0.053 -0.424
4 +0.460 +4.356 +1.412 +0.018 -0.477
5 +0.819 +5.324 +3.128 +0.011 -0.513
6 +1.048 +4.922 -2.358 -0.011 -0.504
7 +0.867 +2.918 +7.562 -0.009 -0.495
8 +0.639 -2.459 -1.839 +0.000 -0.486
9 +0.611 -1.844 +0.078 +0.011 -0.502
Visualization :
Loss :
fidle.end()
End time : 22/12/24 21:20:41
Duration : 00:00:04 128ms
This notebook ends here :-)
https://fidle.cnrs.fr
