Source: google-ml-course

Reducing loss in ML

Aim: compute model parameters (weights) such that loss is minimal

• For which parameters does the loss converge? (Reach a minimum)
• How to get these optimal model parameters? –> which direction do we go in the parameter space?
• One possibility: compute the gradient and steer the model parameters based on the gradient –> gradient descent , variations of gradient descent
• In neural networks : backpropagation

Gradient: derivative of the loss function with respect to the model parameters $$ \begin{align} \dfrac{\partial L}{\partial \mathbf{w}} \end{align} $$