AMS 553.761: Nonlinear Optimization I

Nonlinear Optimization I 553.761
TTh 4:30pm - 6:30pm, WHITEHEAD HALL 304

GENERAL INFORMATION

Instructor : Amitabh Basu
Office Hours : Wednesday 5:00--6:30pm, or email for appointment. Office Hours will be in my office : Whitehead 202B.
Email : basu [dot] amitabh [at] jhu [dot] edu

Teaching Assistant : Yashil Sukurdeep and Lingyao Meng will be the TAs for our class.

Yashil's email is yashil.sukurdeep [at] jhu [dot] edu.
Lingyao's email is lmeng2 [at] jhu [dot] edu.
Yashil's office hours will be on Thursdays from 2:45pm--4:15pm.
Lingyao's office hours will be on Mondays from 6:30pm--8:30pm.

Notes/Texts : I will use lecture slides prepared by Daniel P. Robinson for the course. They will be periodically posted here.

Introduction. Slides without "Notes"
Background and basics. Slides without "Notes". MATLAB demo file
Optimality conditions. Slides without "Notes"
Newton's method. Slides without "Notes". MATLAB demo file
Line Search Methods. Slides without "Notes".
Convexity. Slides without "Notes"
Conjugate Gradient
Trust Region Methods. Slides without "Notes".
Least Squares. Slides without "Notes".
Smooth Convex Optimization
Nonsmooth Convex Optimization: Sections 4.1 and 4.4 from Notes on Convexity
Stochastic Gradient Descent. Thanks to Hongda Qiu for figuring out the right constant Q in the strongly convex analysis!
Coordinate Minimization. Slides without "Notes".
Second Order Methods. Slides without "Notes".
Linear Programming.

Other useful textbooks and resources

Optimization:

J. Nocedal and S. Wright, Numerical Optimization, Second Edition, Springer, (2006)
A. R. Conn, N.I.M Gould, and Ph. L. Toint, Trust Region Methods, SIAM, Philadelphia, PA, (2000)
R. Fletcher, Practical Methods of Optimization, 2nd Edition, Wiley, Chichester and New York, (1987)
D. P. Bertsekas, Nonlinear Programming, Second Edition, Athena Scientific, Belmont, MA, (1999)
A. Ruszcynski, Nonlinear Optimization, Princeton University Press, Princeton, NJ (2006)
Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer Academic Publishers, Norwell, MA, (2004)

Basic Numerical Analysis:

S. D. Conte and C. de Boor, Elementary Numerical Analysis: an algorithmic approach, International Series in Pure and Applied Mathematics, McGraw-Hill Book Company, (1972).

MATLAB Tutorial from Daniel P. Robinson.

Syllabus : The syllabus with list of topics to be covered is available HERE.
This course considers algorithms for solving various important nonlinear optimization problems and, in parallel, develops the supporting theory. Primary focus will be on unconstrained optimization. Topics will include: necessary and sufficient optimality conditions; gradient, Newton, and quasi-Newton based line-search, trust-region, and cubic regularization methods; linear and nonlinear least squares problems; and linear and nonlinear conjugate gradient methods. Special attention will be paid to the large-scale case and will include topics such as limited-memory quasi-Newton methods, stochastic gradient descent.

EXAM AND GRADING INFORMATION

There will one in-class Midterm and one in-class Final exam. In addition, I will regularly (approx. every two week) post homework assigments here. You will be asked to hand in some of the HW problems which will be graded (approximately every two weeks). Seriously attempting ALL the homework problems is imperative for your success in the class, and they will give an indication of the kind of problems on the tests.

Homeworks

HW I.
Please hand in your solutions in class on Thursday, September 19, 2019.
HW II.
There was a major error in Algorithm 13 from the "Line Search Methods" slides, which is the topic of Problem 2.4. I have updated the slides with a correct algorithm. Thanks to Genevieve Starke for catching this!
For problem 2.4, one needs to assume that the function f has a Lipschitz continuous gradient. This was not explicitly stated earlier. I have added this to the statement of the problem now. Thanks to Thomas Athey for catching this!
Please hand in your solutions in class on Tuesday, October 8, 2019.
HW III.
Please hand in your solutions in class on Thursday, October 31, 2019.
There was a typo in the statement of Lemma 2.2 in the ``Conjugate Gradient" lecture notes. In the expression for the gradient of 'f', there was a x^0 which should be \bar x. This typo has now been fixed. Thanks to Hongda Qiu for catching this.
HW IV.
Please hand in your solutions in class on Thursday, December 5, 2019.

Midterms and Final

The Midterm will be held on Tuesday, October 8, 2019 in class. The syllabus for the midterm will be everything that was covered from the beginning of the semester to the lecture on Thursday, October 3, 2019.
The Final Exam will be held from 9am -- 12 noon on Friday, December 13, 2019.
The syllabus for the final is all the material covered during the course of the semester.

Grades

Your final grade will be based on the following weightage :
Homework - 35%
Midterm - 30%
Final - 35%
Blackboard will be used to post the HW and exam grades. You should be able to access your grades there.