Math 577-1, Information and Coding, Fall 2016

Information TypeData
Meeting TimeMWF, 11:00—11:50
Meeting Room MATH 514
InstructorProfessor Marek Rychlik
OfficeMathematics 605
Homepage (Mirror)

Office Hours

PersonnelDay(s) of the WeekHourRoomComment
Marek RychlikM9:30am—10:20amMathematics 605 Regular Office Hours in my office
Marek RychlikM2:00pm—2:50pmMathematics 605 Regular Office Hours in my office
Marek RychlikW2:00pm—2:50pmMathematics 605 Regular Office Hours in my office

Required Texts

Information Theory, Inference and Learning Algorithms, Edition: first, Author: David J.C. McKay, ISBN: 978-0521642989, e-ISBN: ???, Publisher: Cambridge University Press, Date published: October 6, 2003, Pages: 640, required (book downloadable for free!).

Required Examinations And Other Grade Components

  • Five homework assignments, worth 16% of the course grade each, for a total of 80%
  • A take-home Final Project worth 30%.

Homework, and take-home final, shall be submitted as a typed paper, with the exception of these graphs and figures which cannot be easily drawn with software. The work shall be submitted electronically, as a PDF document, through D2L, using the Dropbox feature of D2L.

The student is expected to be comfortable with scientific computing and MATLAB software. Most of the homework will involve writing short (50-200 lines) programs in MATLAB. Some most used MATLAB features will be covered in class.


  1. General mathematical background representative of a first year graduate student in mathematics, applied mathematics or a related scientific field.
  2. Familiarity with scientific programming.


Homework is assigned throughout the semester. Two types of homework problems will be assigned:

  1. Homework which consists of selected exercises in the book.
  2. Custom homework assignments created by the instructor and involving a programming component in MATLAB.

Extra Credit Assignments

There will be some.


Overall Course Objectives and Expected Learning Outcome

  1. Learn the foundations of information theory.
  2. Learn the principles of data compression.
  3. Learn the foundations of finite fields (Galois fields) with emphasis on its computational aspects.
  4. Learn the foundations of error correcting codes with emphasis on codes of algebraic origin.
  5. Learn how to apply MATLAB to representative problems science and technology, such as compressing images and text, reliably transmitting information over noisy channels, and handling statistical inference in "big data" systems.

Course Outline

Week Dates Topics Sections Covered
1Aug 22—Aug 26 Introduction to Information Theory. Provability, Entropy, and Inference. More about Inference. 1, 2, 3
2Aug 29—Sep 2 Introduction to MATLAB. Image representation and image processing. Communications Toolkit and Galois field arithmetic. Implementation of Hamming (7,4) code. MATLAB documentation, RTG error correcting codes slides
3Sep 5Labor Day - no class.
3Sep 7—Sep 9 Data Compression. The Source Coding Theorem. I.4
4Sep 12—Sep 16 Sketch of the proof of Source Coding Theorem. I.4
5Sep 19—Sep 23 Elias-Shannon-Fano coding and Arithmetic coding. Symbol Codes. Stream Codes. Codes for Integers. I.5, I.6, I.7
6Sep 26—Sep 30 I.6, I.7
7Oct 3—Oct 7 Probabilities and Inference. An Example Inference Task: Clustering. IV.20, IV.21
8Oct 10—Oct 14 Maximum Likelihood and Clustering. Exact Marginalization. IV.22, IV.24
9Oct 17—Oct 21 Neural networks. Introduction to Neural Networks. V.39
10Oct 24—Oct 28 Neural networks. Introduction to Neural Networks. The Single Neuron as a Classifier. Capacity of a Single Neuron. V.39, V.40, V.41
11Oct 31—Nov 4 Supervised Learning in Multilayer Networks. V.42, V.44
12Nov 11 Veteran's Day - no class.
13Nov 14—Nov 18
14Nov 21—Nov 23
14Nov 24—Nov 27 Thanksgiving recess.
15Nov 28—Dec 2
16Dec 5—Dec 7
Finals WeekDec 14 (Wednesday)Final Exam, 10:30 pm - 12:30 pm (regular room).

Course Policies

Attendance Policy

Students are expected to attend every scheduled class and to be familiar with the University Class Attendance policy as it appears in the General Catalog. It is the student's responsibility to keep informed of any announcements, syllabus adjustments or policy changes made during scheduled classes.

Expected Classroom Behavior

Students are expected to behave in accordance with the Student Code of Conduct and the Code of Academic Integrity. The guiding principle of academic integrity is that a student's submitted work must be the student's own. University policies can be found at

Threatening Behavior

See No prohibited behavior will be tolerated.

Administrative Drop

Students who miss the first two class meetings will be administratively dropped unless they have made other arrangements with the instructor.

Missed Exams

Students are expected to be present for all exams. If a verifiable emergency arises which prevents you from taking an in-class exam at the regularly scheduled time, the instructor must be notified as soon as possible, and in any case, prior to the next regularly scheduled class. Make-up exams and quizzes will be administered only at the discretion of the instructor and only under extreme circumstances. If a student is allowed to make up a missed exam, (s)he must take it at a mutually arranged time. No further opportunities will be extended. Failure to contact your instructor as stated above or inability to produce sufficient evidence of a real emergency will result in a grade of zero on the exam. Other remedies, such as adjusting credit for other exams, may be considered.

Accessibility and Accommodations

Disabled students must register with Disability Resources and be identified to the course instructor through the University's online process in order to use reasonable accommodations.

It is the University's goal that learning experiences be as accessible as possible. If you anticipate or experience physical or academic barriers based on disability, please let me know immediately so that we can discuss options. You are also welcome to contact Disability Resources 520-621-3268 to establish reasonable accommodations.

Please be aware that the accessible table and chairs in this room should remain available for students who find that standard classroom seating is not usable.

Policy on the grade of "I" (incomplete)

The grade of "I" will be awarded if all of the following conditions are met:

  • The student has completed all but a small portion of the required work.
  • The student has scored at least 50% on the work completed.
  • The student has a valid reason for not completing the course on time.
  • The student agrees to make up the material in a short period of time.
  • The student asks for the incomplete before grades are due, 48 hours after the final exam.


Changes to the Syllabus

The information contained in the course syllabus, other than the grade and absence policies, is subject to change with reasonable advance notice, as deemed appropriate by the instructor.