CS 839: Mathematical Principles of Reinforcement Learning

Course Information

Course Description

This is a PhD-level course on the theory of reinforcement learning (RL). While it is impossible to cover all important topics in RL, the central theme of this course is to understand the mathematical principles behind classic RL ideas and algorithms. Topics include (tentative):

• Fundamentals: Markov Decision Processes (MDPs), value iteration, policy iteration
• Classic RL theory: fitted Q-iteration, error propagation, importance sampling, policy gradient
• Exploration and exploitation: optimism, pessimism
• Modern topics

Schedule

Grading

The grading for the course will be based on (tentative, subject to ±10% changes):

• 3-4 Homework Assignments ~ 50%
• Course Project ~ 50%

Problem sets and deadlines are posted on the course webpage (see schedule above), and solutions should be submitted to Gradescope. The deadlines on the schedule are tentative; please refer to Gradescope for the exact time.

Late Homework Policy

Homeworks must be submitted by the posted due date. You are allowed up to 6 total late days for the homeworks throughout the entire semester. These will be automatically deducted if your assignment is late.

After your late days are used up, late penalties will be applied:

• Up to 24 hours late: 33% penalty
• Up to 48 hours late: 66% penalty
• Beyond 48 hours: no credit

We will track all your late days and any deductions will be applied in computing the final grades. If you are unable to turn in HWs on time, aside from permitted days, then do not enroll in the course.

Course Project

See Project Topics & References for a list of suggested papers.

Students will complete a course project, which includes:

• A short project proposal
• A project report
• A final presentation

The project can take one of the following forms:

• Reproduce and understand theoretical analysis from an existing paper
• Work on novel research questions related to RL theory (discuss with me first)
• Start with reproducing, then try to extend — recommended for most students

Prerequisites

This is a theory-oriented course. Students should have a solid background in:

• Linear algebra
• Probability and statistics
• Machine learning fundamentals (e.g., CS 760)
• Mathematical maturity and proof-writing experience

Prior exposure to reinforcement learning is helpful but not required.

Academic Integrity and Collaboration Policy

Homework assignments and project must be completed individually. Cheating and plagiarism will be dealt with in accordance with University procedures (see the UW-Madison Academic Misconduct Rules and Procedures).

You are encouraged to discuss ideas, approaches, and techniques broadly with your peers or the instructor, but not at a level of detail where specific solutions are described by anyone.

If you have any questions about this policy, please ask the instructor before you act.

AI Policy

The use of artificial intelligence (AI) tools and applications (including, but not limited to, Copilot, ChatGPT, Gemini, Claude and others) for course assignments and assessments does not support the learning objectives of this course and is prohibited. However, you may use AI tools to help understand lecture content.

Using AI for homework assignments or project is a violation of the course's expectations and will be addressed through UW-Madison's academic misconduct policy, specifically UWS 14.03(1)(b): Uses unauthorized materials or fabricated data in any academic exercise.

Resources

• Reinforcement Learning: An Introduction by Sutton and Barto
• Reinforcement Learning: Theory and Algorithms by Agarwal, Jiang, Kakade, and Sun