Global minimum, trap

Question

Hi,

how do I avoid being trapped in a local minimum with the regression?

Thanks

Answer 1

Linear régression does not have local minimum issue. It is a convex problem

Comments

Sorry, I don't understand. You say there cannot be a landscape with different convex locations?

---

I am saying that if you use a linear regression, you will never have any issue with respect to local minimum because this is a convex optimization algorithm. That does not mean that your data landscape is convex.

---

Let me ask differently then. If I have multiple parameters and I try to optimise them, what do I do?

---

Well you can try a linear regression if your data is simple enough. But it may not work properly. Then you're going to need more powerful algorithms (like neural network). But there is no magical recipe if that's what you're looking for (otherwise it would be much easier)

---

But aren't we not talking about neural networks now? I am confused now.

---

I misunderstood your question then. I assumed you were talking about linear regression. Then yes with the neural networks, local minimas are a big issue. But we will see in future sessions a few methods which are used to somewhat avoid them (still not 100% sure)

---

I try again. Sorry.
With y=m*x +b there is only one solution, but if I have y(x1, x2, x3) dependent on multiple parameters, I don't think there is only one solution to minimise. Correct?

---

In this case it is a linear regression because you're linearly dependent with respect to your parameter. So it is convex (you're optimizing with respect to parameters not with respect to your inputs) and therefore only one solution.

---

Sorry again. Even if it is polynomial of higher order? x1,x2, x3 are my different observation parameters. Maybe my naming was bad here.

---

If it's polynomial, it's polynomial with respect to input. But Y=MX+B is always linear wrt parameters M.

---

Yes, I can follow, but a polynomial can have more than one solution? Or still not? Things become complicated when multidimensional?

---

Well if your optimisation is polynomial (non-linear) wrt parameters then yeah you can have multiple solution. But in your example, you're polynomial wrt inputs and linear wrt parameters so only one solution with this algorithm.

---

What does wrt mean, please?

---

With respect to

---

To clarify, how should I have indicated my function as multi-dim?
If I write f(para1, para2, para3) you cannot see IMHO you cannot see if it is non-linear.

---

No I don't know if it is non-linear, but I assumed it was because 99.9999% of the time in deep learning it is non linear. Otherwise the linear regression will do just fine.