Math 577-1, Big Data and Neural Computing, Fall 2018

Information TypeData
Meeting TimeMWF, 11:00—11:50
Meeting Room MATH 124
InstructorProfessor Marek Rychlik
OfficeMathematics 605
Homepage (Mirror)

Office Hours

PersonnelDay(s) of the WeekHourRoomComment
Marek RychlikM1:00pm—1:50pmMathematics 605 Regular Office Hours in my office
Marek RychlikW12:00pm—12:50pmMathematics 605 Regular Office Hours in my office
Marek RychlikF12:00pm—12:50pmMathematics 605 Regular Office Hours in my office

Required Texts

Information Theory, Inference and Learning Algorithms, Edition: first, Author: David J.C. McKay, ISBN: 978-0521642989, e-ISBN: ???, Publisher: Cambridge University Press, Date published: October 6, 2003, Pages: 640, required (book downloadable for free!).

Required Examinations And Other Grade Components

  • Five homework assignments, worth 16% of the course grade each, for a total of 80%
  • A take-home Final Project worth 30%.

Homework, and take-home final, shall be submitted as a typed paper, with the exception of these graphs and figures which cannot be easily drawn with software. The work shall be submitted electronically, as a PDF document, through D2L, using the Dropbox feature of D2L.

The student is expected to be comfortable with scientific computing and MATLAB software. Most of the homework will involve writing short (50-200 lines) programs in MATLAB. Some most used MATLAB features will be covered in class.


  1. General mathematical background representative of a first year graduate student in mathematics, applied mathematics or a related scientific field.
  2. Familiarity with scientific programming.


Homework is assigned throughout the semester. Two types of homework problems will be assigned:

  1. Homework which consists of selected exercises in the book.
  2. Custom homework assignments created by the instructor and involving a programming component in MATLAB.

Extra Credit Assignments

There will be some.


Overall Course Objectives and Expected Learning Outcome

  1. Learn connectionist approach to programming, in particular neural computing with various types of neural networks: Hopfield, soft-max, multi-layer, pattern recognition, probabilistic, convolutional, recurrent, long-short memory, self-organizing etc.
  2. Learn how to scale up computer code to Big Data.
  3. Learn mathematics related to neural programming and big data, including Bayesian inference, optimization with stochastic gradient type methods, and Monte Carlo.
  4. Learn image and other signal processing basics.
  5. Solve representative problems of machine learning with neural networks and MATLAB.

Course Outline

Week Dates Topics Sections Covered
1Aug 20—Aug 24 Perceptron, binary classification, Bayesian inference. Prior and posterior likelihood. Constructing maximum likelihood decoders. 39.1, 39.2, 39.3, 39.4
2Aug 27—Aug 31 Learning by steepest descent. Barzilai-Borwain formula and variable learning rate. Bayes formula with marginalization. Soft-max and decoding of lossy 7-bar display codes. 39.1, 39.2, 39.3, 39.4
3Sep 3—Sep 7 Multi-layer neural networks. The patternnet architecture. Soft-max layer. Backpropagation as the fundamental training algorithm. 44.1, 44.2, 44.3, 44.4
4Sep 10—Sep 14 Principal Component Analysis (PCA, KLT). The MNIST data set and its protocol. Binary data formats. Multi-layer neural networks. The patternnet architecture. Soft-max layer. 44.1, 44.2, 44.3, 44.4
5Sep 17—Sep 21 Logistic regression and multi-class logistic regression. Discussion of coordinate-free calculations and Frechet derivative. The derivative, the gradient and the second derivative of the multi-class logistic regression network. 44.1, 44.2, 44.3, 44.4, paper written by instructor
5Oct 15—Oct 19 Using Hopfield networks as associative memory. Convergence of Hopfield network. Improving weights with Hebb rule. 42.4, 42.5, 42.6, 42.7, 42.8, 42.9
5Oct 22—Oct 26 Message Passing. Parallel programming in MATLAB. Threads, workers, labs, threads. Synchronization of threads with barriers. Collective communications, gop (Global Operation). The spmd (Single Program Multiple Data) block. Implementation of the soldier counting algorithm. 16.1, 16.2, 16.3, 16.4
5Oct 29—Nov 2 More Parallel programming in MATLAB. Modeling with Gaussian Processes. Applications to "Plastic" challenge data. 16.1, 16.2, 16.3, 16.4, 45.1, 45.2, 45.3
5Nov 5—Nov 9 Modeling with Gaussian Processes. The Bayesian point of view on splines. The Gibbs distribution for sampling from a Gaussian process. Filtering Gaussian process, e.g. integration. 45.1, 45.2, 45.3, 45.4, 45.5, 45.6, 45.7

Course Policies

Attendance Policy

Students are expected to attend every scheduled class and to be familiar with the University Class Attendance policy as it appears in the General Catalog. It is the student's responsibility to keep informed of any announcements, syllabus adjustments or policy changes made during scheduled classes.

Expected Classroom Behavior

Students are expected to behave in accordance with the Student Code of Conduct and the Code of Academic Integrity. The guiding principle of academic integrity is that a student's submitted work must be the student's own. University policies can be found at

Threatening Behavior

See No prohibited behavior will be tolerated.

Administrative Drop

Students who miss the first two class meetings will be administratively dropped unless they have made other arrangements with the instructor.

Missed Exams

Students are expected to be present for all exams. If a verifiable emergency arises which prevents you from taking an in-class exam at the regularly scheduled time, the instructor must be notified as soon as possible, and in any case, prior to the next regularly scheduled class. Make-up exams and quizzes will be administered only at the discretion of the instructor and only under extreme circumstances. If a student is allowed to make up a missed exam, (s)he must take it at a mutually arranged time. No further opportunities will be extended. Failure to contact your instructor as stated above or inability to produce sufficient evidence of a real emergency will result in a grade of zero on the exam. Other remedies, such as adjusting credit for other exams, may be considered.

Accessibility and Accommodations

Disabled students must register with Disability Resources and be identified to the course instructor through the University's online process in order to use reasonable accommodations.

It is the University's goal that learning experiences be as accessible as possible. If you anticipate or experience physical or academic barriers based on disability, please let me know immediately so that we can discuss options. You are also welcome to contact Disability Resources 520-621-3268 to establish reasonable accommodations.

Please be aware that the accessible table and chairs in this room should remain available for students who find that standard classroom seating is not usable.

Policy on the grade of "I" (incomplete)

The grade of "I" will be awarded if all of the following conditions are met:

  • The student has completed all but a small portion of the required work.
  • The student has scored at least 50% on the work completed.
  • The student has a valid reason for not completing the course on time.
  • The student agrees to make up the material in a short period of time.
  • The student asks for the incomplete before grades are due, 48 hours after the final exam.


Changes to the Syllabus

The information contained in the course syllabus, other than the grade and absence policies, is subject to change with reasonable advance notice, as deemed appropriate by the instructor.