Optimization Basics

This lecture covers the fundamentals of optimization, starting with logistic regression and maximizing the margin. It delves into derivatives of linear and non-linear functions, convex vs. non-convex functions, and the process of minimizing convex functions using algorithms like gradient descent. The lecture also explores the influence of step size and starting points in optimization, the concept of conjugate gradient, and second-order methods like Newton's method. Practical Python implementations and potential instabilities in optimization are discussed, along with the importance of convex functions in having a global minimum.