Syllabus for Math 589, Section 002, Spring 2024

Location and Times

Math 589, Section 001, meets in Social Sciences 308, TuTh 2:00PM - 3:15PM.

Course Description

According to the catalogue:

Course Prerequisites or Co-requisites

As determined by the GIDP Program in Applied Mathematics Graduate Student Handbook.

Browse to Core Courses for addition information.

Instructor and Contact Information

Information	Data
Instructor	Professor Marek Rychlik
Office	Mathematics 605
Telephone	1-520-621-6865
Email	rychlik@arizona.edu
Instructor Homepage/Web Server	http://alamos.math.arizona.edu
Course Homepage	http://alamos.math.arizona.edu/math589
Course Homepage (Mirror)	http://marekrychlik.com/math589

Course Format and Teaching Methods

The course format is that of a conventional lecture, with in-class discussion and additional web-delivered content. All lectures will be recorded and available on Zoom and Panopto.

Written homework will be assigned regularly and graded using Gradescope.

In addition, the course incorporates required programming assignments. Numerical experimentation is essential to understanding and using the course subject matter. The assignments will be graded by an autograder implemented in Gradescope.

Course Objectives

1. Understand and apply various numerical methods for optimization and optimal control.
2. Analyze and solve complex optimization problems in both constrained and unconstrained settings.
3. Explore Monte-Carlo algorithms and their applications in inference and learning.
4. Investigate neural network algorithms and their implementation in modern programming frameworks.

Generative AI use IS permitted or encouraged

In this course you are welcome and expected to use generative artificial intelligence/large language model tools, e.g. ChatGPT, Dall-e, Bard, Perplexity. Using these tools aligns with the course learning goals such as developing writing and programming skills, and ability to effectively use available information. Be aware that many AI companies collect information; do not enter confidential information as part of a prompt. LLMs may make up or hallucinate information. These tools may reflect misconceptions and biases of the data they were trained on and the human-written prompts used to steer them. You are responsible for checking facts, finding reliable sources for, and making a careful, critical examination of any work that you submit. Your use of AI tools or content must be acknowledged or cited. If you do not acknowledge or cite your use of an AI tool, what you submit will be considered a form of cheating or plagiarism. Please use the following guidelines for acknowledging/citing generative AI in your assignments:

• MLA
• APA

Absence and Class Participation Policy

Importance of attendance and class participation

Participating in course and attending lectures and other course events are vital to the learning process. As such, attendance is required at all lectures and discussion section meetings. Students who miss class due to illness or emergency are required to bring documentation from their healthcare provider or other relevant, professional third parties. Failure to submit third-party documentation will result in unexcused absences.

Missed Exams

Students are expected to be present for all exams. If a verifiable emergency arises which prevents you from taking an in-class exam at the regularly scheduled time, the instructor must be notified as soon as possible, and in any case, prior to the next regularly scheduled class. Make-up exams and quizzes will be administered only at the discretion of the instructor and only under extreme circumstances. If a student is allowed to make up a missed exam, (s)he must take it at a mutually arranged time. No further opportunities will be extended. Failure to contact your instructor as stated above or inability to produce sufficient evidence of a real emergency will result in a grade of zero on the exam. Other remedies, such as adjusting credit for other exams, may be considered.

COVID-19 related policies

As we enter the new semester, the health and wellbeing of everyone in this class is the highest priority. Accordingly, we are all required to follow the university guidelines on COVID-19 mitigation. Please visit http://www.covid19.arizona.edu for the latest guidance.

UA policies

The UA"s policy concerning Class Attendance, Participation, and Administrative Drops is available at: http://catalog.arizona.edu/2015-16/policies/classatten.htm The UA policy regarding absences for any sincerely held religious belief, observance or practice will be accommodated where reasonable, http://policy.arizona.edu/human-resources/religious-accommodation-policy . Absences pre-approved by the UA Dean of Students (or Dean Designee) will be honored. See: http://uhap.web.arizona.edu/policy/appointed-personnel/7.04.02

Required Texts or Readings

Required Textbook

None. See Background reading for a list of texts worth studying along with taking the course.

Assignments and Examinations

Notes on exam administration

All examinations are planned to be administered during the class time, either in person or on Zoom.

If, due to unforseen circumstances, they cannot be held in person, they are held on Zoom using the "gallery view" mode.The exam papers for not in-person tests will be distributed on-line by D2L and collected electronically using D2L "dropbox" feature.

Exam/assignment listing with date and grade contribution

Exam or Assignment	Date	Grade contribution
Midterm 1	February 13 (Tuesday), 2:00pm - 3:15pm	20%
Midterm 2	April 4 (Thursday), 2:00pm - 3:15pm	20%
Final Examination	May 6 (Monday), 3:30pm - 5:30pm	30%
Homework	Written and Programming, administered via Gradescope	30%

Homework Assignments

Written homework consists of approximately twelve assignments equally contributing to the grade, each worth 30/12 = 2.5% of the grade. The assignments are posted on line at this link: Homework. The assignment papers are collected via Gradescope, which is cloud-based software for semi-automatic grading. Things to keep in mind:

• The student will submit all graded work through Gradescope. Gradescope assignments, by their nature, are answer-oriented. The opportunity for partial credit is somewhat limited.
• Homework may not be submitted through e-mail, no exceptions.
• Due dates will be adjusted in Gradescope only as needed and the student is expected to check regularly for those adjustments. Due dates in D2L will not be updated.

Written homework is assigned regularly throughout the semester, for a total of approximately 80 problems. Two types of homework will be assigned:

1. Homework which consists of selected exercises in the required textbook.
2. Some custom homework will be composed by the instructor. Some of the custom problems will require programming.

Homework submission requirements

Using Gradescope for grading differs from other grading systems. Mainly, it uses AI to allow the instructor to accurately grade a larger number of problems than it would be possible otherwise. Some grading is completely automated (e.g., solutions to problems with a numerical answer). More comples answers may be grouped automatically by using Machine Learning, OCR and image analysis. However, it is possible to completely confuse the system by improperly structuring the submitted document. Therefore, please read the instructions below carefully and re-visit them as needed. Note that Gradescope supports automatic regrade requests which you can use if all fails.

The solutions must be structured in such a way that Gradescope can read them and that its 'AI' can interpret them. Your homework must be submitted as a PDF document, even if you use scanner or phone to capture images. Two typical workflows will be as follows:

1. Download the blank assignment (also called a 'template') from Gradescope.
2. Read and understand exactly what answers you need to provide. The space to enter the answer is a blue box, and marked with a label such as 'Q1.1' ("Question 1, part 1").
3. Work out the problem on "paper" (real or virtual), to obtain the answers. They must fit in the designated boxes in the 'template'. The size of the box is a hint from the instructor about the size of the answer (typically a number or a math formula) when entered by hand, using regular character size.
4. The recommended way to fill out the 'template' is paperless, by using suitable software and hardware (digital pen or tablet). I use a free program Xournal for this and it works great. You need to use it in combination with a digital pen or a tablet. It can produce a PDF easily, ready for submission to Gradescope.
5. You can also print the assignment on (real) paper, fill out the answers and scan the marked up document back to PDF format. However, the position of the boxes must be exactly (to a fraction of an inch) as in the original. Also, you may encounter a variety of "quality control" issues, especially if you are using a digital camera to scan the paper solution. All issues can be solved by a mix of the right hardware and software, but may not be the best time investment. The least troublesome way to scan is to use a real, flatbed scanner, e.g. in the library.
6. Upload the resulting document (a PDF of the 'template' marked up with your answers) to Gradescope. Your PDF must contain your name and student id in designated spaces. The Gradescope 'AI' will look for your name and student id, to properly associate it with your account.
7. After grading, the grade will be transmitted to D2L (Brightspace) and will be added to your 'Final Calculated Grade' automatically.
8. Do not reduce handwriting size! Reduce the size of your answer using
   • closed form expressions;
   • appropriate math functions, e.g., absolute value, min and max.
9. Under no circumstances write outside the provided space (boxes). Gradescope, and the grader only considers the content of the designated boxes.
10. IMPORTANT! Do not insert pages in the solution template. This will confuse Gradescope, and will result in reduced score and/or will require re-submission. However, you are encouraged to submit scratchwork. You should create pages at the end of the document. Similarly, if you run out of space in the template for your solution, you can continue the solution on a newly created page at the end of the document, adding a note in the template: "Solution continued on page 13" where page 13 will contain the continuation.

Programming and Software

Programming in Python and MATLAB is an important part of the course. Programming assignments in the first parts of the course will be in Python, and later in the course they will approximately alternate between MATLAB and Python.

Additionally, for illustrating some aspects of the course, I will be using these programs (easy to download and free to use):

• A computer algebra system (CAS) called Maxima.
• The maxima GUI wxMaxima is a separate program and this is the preferred way to use Maxima if you are a beginner. During the previous semester students successfully applied Maxima to doing their homework, and I will provide examples of Maxima calculations in class.
• Automatic Row Reduction Generator - a program I wrote in Common Lisp for training students in various flavors of Gaussian elimination. It simulates row reduction, as it would be performed on paper by hand, featuring:
  • various pivoting strategies
  • row echelon and reduced row echelon forms
  • rational arithmetic with exact answers
  • Floating-point and rational complex numbers

Final Examination

The final examination is scheduled for: May 6 (Monday), 3:30pm - 5:30pm.

The time, data and general exam rules are set by the University and can be found at these links:

• Final Exam Regulations
• Final Exam Schedule

Grading Scale and Policies

The student in the class normally receives a letter grade A, B, C, D or E.

The cut-offs for the grades are:

Grade	% Range
A	90%+
B	80-90%
C	70-80%
D	60-70%
E	0-60%

Normally, individual tests and assignments will not be "curved". However, grade cut-offs may be lowered at the end of the semester (but not raised!) to reflect the difficulty of the assignments and other factors that may cause abnormal grade distribution.

The grade will be computed by D2L and the partial grade will be updated automatically by the system as soon as the individual grades are recorded.

General UA policy regarding grades and grading systems is available at https://catalog.arizona.edu/policy-type/grade-policies

Safety on Campus and in the Classroom

For a list of emergency procedures for all types of incidents, please visit the website of the Critical Incident Response Team (CIRT):

https://cirt.arizona.edu/case-emergency/overview Also watch the video available at https://arizona.sabacloud.com/Saba/Web_spf/NA7P1PRD161/common/learningeventdetail/crtfy000000000003560

Classroom Behavior Policy

To foster a positive learning environment, students and instructors have a shared responsibility. We want a safe, welcoming and inclusive environment where all of us feel comfortable with each other and where we can challenge ourselves to succeed. To that end, our focus is on the tasks at hand and not on extraneous activities (i.e. texting, chatting, reading a newspaper, making phone calls, web surfing, etc).

Threatening Behavior Policy

The UA Threatening Behavior by Students Policy prohibits threats of physical harm to any member of the University community, including to one's self. See: http://policy.arizona.edu/education-and-student-affairs/threatening-behavior-students .

Accessibility and Accommodations

Our goal in this classroom is that learning experiences be as accessible as possible. If you anticipate or experience physical or academic barriers based on disability, please let me know immediately so that we can discuss options. You are also welcome to contact Disability Resources (520-621-3268) to establish reasonable accommodations. For additional information on Disability Resources and reasonable accommodations, please visit http://drc.arizona.edu/ .

If you have reasonable accommodations, please plan to meet with me by appointment or during office hours to discuss accommodations and how my course requirements and activities may impact your ability to fully participate. Please be aware that the accessible table and chairs in this room should remain available for students who find that standard classroom seating is not usable. Code of Academic Integrity Required language: Students are encouraged to share intellectual views and discuss freely the principles and applications of course materials. However, graded work/exercises must be the product of independent effort unless otherwise instructed. Students are expected to adhere to the UA Code of Academic Integrity as described in the UA General Catalog. See: http://deanofstudents.arizona.edu/academic-integrity/students/academic-integrity http://deanofstudents.arizona.edu/codeofacademicintegrity .

UA Nondiscrimination and Anti-harassment Policy

The University is committed to creating and maintaining an environment free of discrimination, http://policy.arizona.edu/human-resources/nondiscrimination-and-anti-harassment-policy . Our classroom is a place where everyone is encouraged to express well-formed opinions and their reasons for those opinions. We also want to create a tolerant and open environment where such opinions can be expressed without resorting to bullying or discrimination of others.

Additional Resources for Students

UA Academic policies and procedures are available at: http://catalog.arizona.edu/2015-16/policies/aaindex.html Student Assistance and Advocacy information is available at: http://deanofstudents.arizona.edu/student-assistance/students/student-assistance

Confidentiality of Student Records

http://www.registrar.arizona.edu/ferpa/default.htm .

Subject to Change Statement

Information contained in the course syllabus, other than the grade and absence policy, may be subject to change with advance notice, as deemed appropriate by the instructor.

Significant Dates (from the Registrar's Website)

Graduate

      
Course Timeline
Classes are held on Monday, Wednesday, and Friday from 12:00 PM to 12:50 PM.


    

        

            
Month
            
Week
            
Topic
        
    
    

        
        

            
January
            
Week 1
            
Introduction to the course and Finite-Dimensional Optimization: Unconstrained Optimizations (Overview)
        
        

            
Week 2
            
Minimizing Quadratic Functions, Least Square Minimization
        
        

            
Week 3
            
Gradient Descent and Backtracking
        
        

            
Week 4
            
Steepest Descent
        
        
        

            
February
            
Week 5
            
Newton, Quasi-Newton, and Cubic-Newton Methods
        
        

            
Week 6
            
Equality Constrained Minimization: Primal-Dual Newton Method
        
        

            
Week 7
            
Interior Point Methods
        
        

            
Week 8
            
Linear Programming: Simplex Method
        
        
        

            
March
            
Week 9
            
L1-Regularization, Sparsity, and Compressed Sensing
        
        

            
Week 10
            
Optimal Control: Introduction and Basic Principles
        
        

            
Week 11
            
Dynamic Programming
        
        

            
Week 12
            
Monte-Carlo Algorithms: Overview and Direct Sampling Methods
        
        
        

            
April
            
Week 13
            
Importance Sampling and Stochastic Algorithms for Optimization
        
        

            
Week 14
            
Markov-Chain Monte-Carlo (MCMC) and Ising Model
        
        

            
Week 15
            
Gibbs Sampling, Metropolis-Hastings, and Exact Sampling
        
        

            
Week 16
            
Variational Algorithms and Neural Networks Algorithms
        
        
        

            
May
            
Week 17
            
Course Review and Final Exam Preparation
        
    



Academic Calendar: Spring 2024 - Graduate - Regular Academic Session


    

        

            
Date
            
Event
        
    
    

        

            
10/1/2023
            
Shopping Cart available.
        
        

            
1/9/2024
            
Last day for students to add to or drop from a waitlist.
        
        

            
1/10/2024
            
First day of Spring classes; UAccess available for registration; first day to add classes for audit (instructor signature required).
        
        

            
1/15/2024
            
Martin Luther King Day, no classes.
        
        

            
1/17/2024
            
Last day to use UAccess for adding classes, changing classes, or changing sections.
        
        

            
1/18/2024
            
Instructor approval required on a Change of Schedule form to ADD or CHANGE classes.
        
        

            
1/23/2024
            
Last day for a refund.
        
        

            
1/24/2024
            
Beginning today, students may completely withdraw from all classes in the term.
        
        

            
2/6/2024
            
Last day for department staff to add or drop in UAccess; last day to drop without a grade of W; last day to change from credit to audit or vice versa with only an instructor's signature; last day to change from pass/fail to regular grading or vice versa with only instructor approval.
        
        

            
2/7/2024
            
Instructor's and dean's signatures required to change from pass/fail to regular grades or vice versa; W period begins.
        
        

            
3/3/2024
            
Last day to make registration changes without the dean's signature.
        
        

            
3/4/2024
            
Instructor's and dean's signatures required on all Change of Schedule forms to ADD or CHANGE classes; Spring recess begins.
        
        

            
3/10/2024
            
Spring recess ends.
        
        

            
3/26/2024
            
Last day for students to withdraw from a class online through UAccess; last day to change to/from audit with only instructor approval; last day for instructors to administratively drop students.
        
        

            
3/27/2024
            
Instructor's and Graduate College dean's permission required to withdraw from a class or to change to/from audit.
        
        

            
5/1/2024
            
Last day to request a complete withdraw from all classes in the term; last day of class.
        
        

            
5/2/2024
            
Reading day, no classes.
        
        

            
5/3/2024
            
Final exams begin.
        
        

            
5/9/2024
            
Final exams end; final grades available in UAccess as soon as posted by the instructor.
        
        

            
5/10/2024
            
Degree award date.
        
    


    

Material Covered

Module 1: Finite-Dimensional Optimization (over continuous variables)

1. Unconstrained Optimizations (Iterative Methods)
   • Minimizing Quadratic Functions, Least Square Minimization
   • Gradient Descent and Backtracking
   • Steepest Descent
   • Newton, Quasi-Newton, and Cubic-Newton Methods
2. Equality Constrained Minimization
   • Primal-Dual Newton Method
3. Interior Point Methods
4. Linear Programming
   • Simplex Method
5. L1-Regularization, Sparsity, and Compressed Sensing

Module 2: Optimal Control (Example of Infinite-Dimensional Optimization)

1. Introduction to Optimal Control
   • Basic Principles and Applications
2. Dynamic Programming
   • Concepts and Computational Strategies

Module 3: Inference & Learning

1. Monte-Carlo Algorithms
   • Direct Sampling: Rejection Method, Pebble Game, Mapping Techniques
   • Importance Sampling
   • Direct Brute-Force Sampling
   • Stochastic Algorithms for Optimization
2. Markov-Chain Monte-Carlo (MCMC)
   • Introduction to the Ising Model as a Running Example of Graphical Model
   • Gibbs Sampling and Metropolis-Hastings Algorithms
   • Exact Sampling for Special Cases (e.g., Trees) with a Link to Dynamic Programming
3. Algorithms for Inference & Learning
   • Variational Algorithms: Mean-Field Theory, Belief Propagation/Message Passing with an Example of the Ising Model
   • Neural Networks Algorithms: Training/Learning and Executing/Inference
     • Primer on TensorFlow, PyTorch, or Julia-Flux

Bonus Module: Oscillation Phenomena and Structural Conditions

1. Understanding Oscillation Phenomena
   • Theoretical Foundations and Practical Implications
2. Structural Conditions for Numerical Discretization
   • Ensuring Faithful Capture of Continuous Problems