Machine learning is promising to revolutionize every domain. Beyond the media hype, what are the basic foundations? Are the theoretical underpinnings irrelevant given the success of deep learning? What does success even mean? In addition to covering the core concepts, the course aims to ask such critical questions and foster a healthy debate among the students. Assignments will include exploring failure modes of popular algorithms, in addition to traditional problem-solving type questions. The core concepts covered include: linear models, kernel methods, probabilistic models, spectral methods (matrices and tensors), neural networks representation theory, non-convex optimization, generalization in deep neural networks, causality etc. The course assumes students are comfortable with analysis, probability, statistics, and basic programming.

- Lectures on Tu/Th at 1pm-2:25pm in Moore B270
- Recitations on Wed at 3pm-4pm in Annenberg 107
- Office hours on Tu at 3pm-4pm in Annenberg 243 and Th at 5pm-6pm in Annenberg 106
- We will be using Piazza for discussion forums and announcements [link]
- We will be using Moodle for managing homeworks and grades [link]

- 7 Assignments (35% of final grade) See the Grading Policy below
- Project (50% of final grade) See the Project Grade Decomposition below
- Scribing (10% of final grade)
- Participation in class and on Piazza (5% of final grade)

Assignments will be due at 4pm on Friday via Moodle. Students are allowed to use up to 48 late hours. Late hours must be used in units of hours. Specify the number of hours used when turning in the assignment. Late hours cannot be used on the projects. There will be no TA support over the weekends.

Peer grading will be used in this class. After your assignment submission, you will be assigned to grade three different assignments of your classmates.
**20%** of your assignment grade will be based on the accuracy of your grading your peers' assignments.

In order to preserve anonymity in the assignment grading process, please
include a **cover page** on your submission which your name is clearly written.
**Do not** write your name in any of the pages besides your cover page.

Project Grade Decomposition

- Proposal Report: 20%
- Proposal Presentation: 10%
- Final Report: 50%
- Final Presentation: 20%

**Homeworks**: (taken from CS 1) It is common for students to discuss ideas for the homework assignments. When you are helping another student with their homework, you are acting as an unofficial teaching assistant, and thus must behave like one. Do not just answer the question or dictate the code to others. If you just give them your solution or code, you are violating the Honor Code. As a way of clarifying how you can help and/or discuss ideas with other students (especially when it comes to coding and proofs), we want you to obey the "50 foot rule". This rule states that your own solution should be at least **50 feet away **. If you are helping another students but cannot without consulting your solution, don't help them, and refer them instead to a teaching assistant.

**Projects**: Students are allowed to collaborate fully within their project teams, but no collaboration is allowed between teams.

Ehsan Abbasi | eabbasi@caltech.edu |

Gautam Goel | ggoel@caltech.edu |

Sahin Lale | alale@caltech.edu |

Lecture 1: | Introduction. |

Lecture 2: | Optimization: Non-convex analysis. |

Lecture 3: | Optimization: Competitive problems. |

Lecture 4: | Spectral Methods: Matrix methods. |

Lecture 5: | Spectral Methods: Tensor methods. |

Lecture 6: | Representation theory: Neural Networks |

Lecture 7: | Generalization theory: VC and Radamacher bounds |

Lecture 8: | Generalization theory: Mystery in deep networks? |

Lecture 9: | Midterm: Presentation of project proposals. |

Lecture 10: | Midterm: Presentation of project proposals. |

Lecture 11: | Robustness in ML |

Lecture 12: | Generative models: likelihood-based |

Lecture 13: | Generative models: GANs |

Lecture 14: | Active learning |

Lecture 15: | Domain Adaptation |

Lecture 16: | Lifelong learning |

Lecture 17: | TBD |

Lecture 18: | TBD |

Lecture 19: | Final presentation of projects |

Lecture 20: | Final presentation of projects |

Lecture videos can be found in the following YouTube playlist

Lecture 1 covered logistics, basic probability and statistics background Lecture 1 Slides.

For details Intro. Probability, Intro. Statistics- Lecture 2 Slides Lecture 2 Notes
- Lecture 3 Slides Lecture 3 Notes
- Lecture 4 Slides Lecture 4 Notes
- Lecture 5 Slides Lecture 5 Notes
- Lecture 6 Slides
- Lecture 7 Slides
- Lecture 8 Slides Lecture 8 Notes
- Lecture 11 Slides