The course will introduce AI, with a focus on neural networks in PyTorch.
- Edited notebook 1.2 to be clearer, and added exercises at the end of notebooks 1.1 and 1.2.
- Changes to the last part (Sec. 6 and onwards) of the LLM notes.
- Clarified that the code in the notebook in week 8 is non-examinable. The statement at the top of that notebook now reads: "The code in this notebook is non-examinable!! We're doing some slightly weird stuff to get the nice visualisations!! Its the concepts illustrated in the notebook that are examinable!! i.e. only the stuff covered in the associated video is examinable."
- Added a note on extra exercises to this README.
- Added links to practice exams to this README.
- Laurence Aitchison [laurence.aitchison@bristol.ac.uk] (unit director)
- This course runs in TB2.
Linear algebra, basic statistics, basic Python.
The course is taught in a "flipped classroom model". So there are no in-person lectures (we wouldn't be able to schedule them anyway given the wide range of cohorts on the course). Instead, there are extensive pdf notes, iPython notebooks and videos. There is a weekly 2-hour lab/problem class. In that session, you can work on the question sheets and/or iPython notebooks, and ask TAs/me any questions on that material, or on any other part of the course.
Course materials for each week will be posted here on Monday. Course materials each week will include some combination of:
- Notes
- There are pdf notes for every week of taught material.
- The notes are often long-ish, but that's only because I'm trying to be thorough (e.g. I'm trying to include all the steps in derivations). So the notes shouldn't in general be very dense.
- The pdf notes will usually, but not always include questions.
- All questions will have worked answers.
- Notebooks
- Each week, there will usually, but not always be iPython notebooks, discussing practical aspects of training neural networks.
- These iPython notebooks occasionally have exercises.
- The concepts in the iPython notebooks are examinable (see Exam section below for details on how code will be examined).
- You don't learn code (well) by just reading. Even if I haven't done exercises, open up your own Colab notebook alongside and type the code in to follow along. Try the things that I'm doing. Try something a bit different!
- Videos
- There are always videos of me going through the pdf notes, and may sometimes be videos of me going through the iPython notebooks.
- Watching the videos isn't necessary!!! The videos just go through (slowly and thoroughly) the material in the underlying pdf notes / iPython notebook. So if you read + understand the underlying notes / notebooks, the videos are not necessary.
- To reiterate, the exam is based on the notes/notebooks, not the videos.
- Labs/problem classes 11am-1pm on Thursdays in QUEENS BLDG F.101a/b/c. These start in week 1 (16th January):
- TAs will be at all of these sessions. I will be at most of the sessions.
- In these sessions, you can work on the exercises in the pdf notes + CoLab notebooks for that week.
- You can ask TAs/me about these exercises (or anything else in the course).
- (Note that more "organised" material in this course is not really possible, as the course is taken so widely that some students have clashes with these sessions).
All the notebooks are going to be in PyTorch. These days, you need to know PyTorch if you're doing AI:
- PyTorch is currently the leading AI library, both for academic research, and industry.
- Fantastic docs + tutorials + libraries (better than any other library)
- Pythonic and hackable: you can look at + modify the source for just about anything
- PyTorch is great for learning AI, because it exposes alot of the underlying concepts, and doesn't hide them under abstractions.
- As we'll see, you can use PyTorch as GPU-enabled-numpy, so it is useful for many applications outside of traditional AI.
Other points:
- You won't need to install PyTorch. All the iPython notebooks in the taught material are presented using Google Colab.
- These exams are in the standard TB2 assessment period.
- The exam will be a 3 hour, 20 question, multiple choice, i 8000 n-person exam.
- The exam is primarily based on the material in the pdf notes.
- But the material in the CoLab notebooks is examinable. I may ask in-detail about the material in the first week's CoLab notebooks (as you can see from Q1 and Q2 of the sample exam). For instance, you should know what
t.ones(2,3,4)orA.mean(2)does (along with the material in the GPU and intro to Python notebooks). I won't ask questions that hinge on your knowledge of syntax, such as "how do you construct a tensor of zeros:t.zeros(1)(2)ort.zeros(1.2)ort.zeros[[1],[2]]. But I might ask questions that hinge on your ability to read and interpret syntactically valid code. - You aren't expected to know the material in the rest of the CoLab notebooks in that level of detail. Â Specifically, the material in the rest of the CoLab notebooks may be examined in a couple of ways:
- The CoLab notebooks can illustrate examinable concepts (e.g. Q7 and Q10 in the sample exam).
- You may need to read more neural-network-y code, but where all information required would be clearly available in the code itself. For instance, I could ask for a verbal description of the following neural network layer:
nn.Sequential(nn.Linear(in_features=100, out_features=200), nn.ReLU()). I could give then several possible descriptions, with the correct description being: "A linear layer that takes vectors with 100 features and returns feature vectors with 200 features, followed by a relu nonlinearity". But I wouldn't ask you to do that fornn.Sequential(nn.Linear(100, 200), nn.ReLU())as the code doesn't tell you what argument isin_featuresand which isout_features, and you really should to look it up to be sure.
- Things have changed quite alot since I took over the course, so the previous EMAT31530 exam is relevant, but nothing before that.
- The exam will focus on the concepts rather than requiring you to learn specific math equations / PyTorch functions.
- Instructions for MCQ exams
- The formula sheet in the sample exam is the same as that in the paper.
- I'm going to try hard to avoid needing memorisating equations in questions (see practice exams for details).
- The Methods of Artificial Intelligence Exam summer examination permits the use of a non-programmable calculator. We recommend the Casio FX-85 GT CW (solar-powered) or Casio FX-83GT CW (battery-powered).
- Please be advised that the use of a programmable calculator will be treated as an examination offence (and is likely to incur a penalty under the University Assessment Regulations). Your calculator will be visually checked by an invigilator during each exam.
- For students who do not own or are unable to obtain a non-programmable calculator prior to their exam, please contact the SEMT school office for further advice: semt-student-enquiries@bristol.ac.uk (e-mail enquiries) or Ada Lovelace Building room G1 (in-person enquiries; 9am-4pm Monday-Friday).
- If you are unsure whether your calculator is programmable or not, our recommendation is to use Google to search for your calculator model, and whether it is programmable or non-programmable. This usually provides a clear answer. Additionally, calculator brand websites often list this information under the product description for each model. If you are concerned and want someone to double check, the SEMT school office can offer support - see the contact details above.
Note that these previous exams are shorter (2 hour , 15 questions), whereas the exam this year is longer (3 hours , 20 questions). The questions this year are exactly the same type and difficulty as the questions in previous years: the exam is just a bit longer.
There are two problems:
- The course will introduce to the basics of how to train modern AI systems. But it is also important for you to understand the capabilities of modern state-of-the-art AI systems.
- At the same time, you're going to have questions about the course material. You can of course ask on the Teams, and in the TA-led problem classes. But what if you want feedback immediately?
The solution is: try asking modern AI systems! Specifically, for this kind of thing (programming, math deep learning), people reckon:
Are all pretty good. There is a feeling that Claude is the best for programming, buts its pretty close. I have been using Claude to help me to e.g. understand recent research in deep learning, so these models are definitely at the point where they can help you learn! Note that:
- The university doesn't have a subscription to Claude. So you can consider getting a personal subscription (I have one). Other providers tend to have more/better free models available though.
- If you want to ask about something specific that turns up e.g. in the lecture notes, you can attach the lecture notes to your message to Claude!
- These models are pretty good in 99% of cases. But like me/a TA, Claude can be wrong. When these models are wrong, it is called a "hallucination" in deep learning research. Some thoughts on that:
- These models do not by default not have access to the internet (sometimes they do, but atm, you have to turn that on using a "search" or similar button). So if you're asking about something super specific (e.g. something that only turns up in a few research papers), it may "misremember". Try attaching some material (e.g. a paper on the topic) to your message to Claude!
- These models can hallucinate if you ask leading questions. For instance, if its impossible to do <task> in <Python library>, but you ask it "How do I do <task> in <Python library>" anyway, it will often try anyway and give you a wrong solution. Try to ask open-ended, non-leading questions.
- Be careful about which model you're asking. Most of the providers have some good, expensive models and some less good, but cheaper models. These platforms will often give free users some access to the good model, but quickly force you onto the less good model as you continue using it.
These models can supercharge your learning if you ask it about things you're confused about as you're learning. As I am sure you are aware, they can also write code, solve exercises etc. for you. I don't think we have any research on this, but that is likely to be detrimental to your learning. And of course, in the exam, you won't have access to these models! So be careful. My advice is: use these models to help you learn more quicker, and think harder and deeper. Definitely don't switch off brain and have them do stuff for you!
Finally, there's alot that these models can't do. Despite the hype, it can't by itself build large programs/systems, nor can it write (interesting) research papers, and it definitely can't to any mechanical/civil engineering. But they can dramtically accelerate experts as they do those things. And they can definitely help you learn those things.
Overall, these AI systems are some of the most powerful tools humanity has developed, and so learning to use them effectively is a critical skill!
- The little book of deep learning
- 3Blue1Brown (YouTube Channel)
- Deep learning book
- Andrej Karpathy's course on building neural networks, from scratch, in code.
- Harvard Annotated Transformer
- Usually searching Google for a given topic will give a bunch of blog posts on the subject (often in places like Towards Data Science)
The exercises in the pdf notes and Colab notebooks are supposed to be complete. But I have had a query about extra exercises.
If you find yourself in this position, the best exercise is to go through my Colab notebooks, looking mainly at my text (i.e. not the code). Then close my notebook, and reproduce the code yourself. If you can do that, you'll understand everything in the course really deeply. And if you can reproduce the code in the non-examinable backprop notebook, let me know and I can chat to you about PhDs :-).
To download the notes, use the download button highlighted with the red circle:
We have students from quite a few different cohorts on this course. So I'm going to try to go over at least some of the prerequistes here. Even if the notes look easy, please do skim through to the end, and check out the exercises.
In any case, there will definitely be material you haven't seen in notebook 1.2 (Introduction to PyTorch) and notebook 1.3 (GPUs with PyTorch).
| Notes/Notebook | Video (if available) |
|---|---|
| notes | notes video 1 (24:57) |
| notes video 2 (13:30) | |
| notes video 3 (23:43) | |
| notes video 4 (27:27) | |
| Notebook 2: Linear Regression | N/A |
| Notes/Notebook | Video (if available) |
|---|---|
| notes | notes video 1 (21:56) |
| notes video 2 (24:31) | |
| notes video 3 (22:41) | |
| Notebook 3: Linear Classification | N/A |
This week, we introduce NNs! This week is a bit unusual, as the notes are very short, with no exercises, but I expect the Python notebooks to be more challenging. That's because mathematically defining NNs is somewhat straightforward. However, understanding how PyTorch implements NNs requires a bit more thought, especially if you're going to understand things well enough to do modifications/tweaks for research/real-world problems!
| Notes/Notebook | Video (if available) |
|---|---|
| notes | notes video 1 (19:32) |
| Notebook 4.1: Python classes | N/A |
| Notebook 4.2: NNs in PyTorch | N/A |
| Notebook 4.3: NNs with torch.nn | N/A |
| Notebook 4.4: NN exercises | N/A |
The key bit of magic in PyTorch is the ability to compute gradients of the loss through arbitrary compute graphs. PyTorch does this using the backprop algoritm, and the backprop algorithm to compute gradients is at the heart of all modern AI. This might be a bit tougher than other weeks, as it involves going back-and-forth between the math and the code. But once you really understand backprop, you'll really know (and be able to reason about) what goes on in modern AI.
Note that Notebook 5.2 is "formally" non-examinable, but should be super-useful in reinforcing the examinable material in the notes!
| Notes/Notebook | Video (if available) |
|---|---|
| notes | notes video 1 (16:55) |
| notes video 2 (23:40) | |
| notes video 3 (9:39) | |
| notes video 4 (7:00) | |
| notes video 5 (13:36) | |
| Notebook 5.1: Compute Graph | N/A |
| Notebook 5.2: Backprop | N/A |
Computing the gradients using backprop was the hard part. But there still are a few tricks to using these gradients to quickly train NNs. That's what this week is all about.
| Notes/Notebook | Video (if available) |
|---|---|
| notes | notes video 1 (21:17) |
| notes video 2 (7:13) | |
| notes video 3 (23:16) | |
| notes video 4 (24:00) | |
| Notebook 7.1 torch.optim | N/A |
| Notebook 7.2 NNs for MNIST | N/A |
This week is a bit unusual, because I'd advise looking at the CoLab notebook first. That notebook introduces overfitting and cross-validation in linear models. The pdf notes then go on to talk about overfitting in neural networks.
Note that there are no exercises this week.
| Notes/Notebook | Video (if available) |
|---|---|
| Notebook 8 Overfitting | notebook video (16:47) |
| notes | notes video 1 (17:01) |
| notes video 2 (21:59) |
This week we'll look at architectures for neural networks for images.
There are no exercises in the pdf notes this week, but there are exercises in the CoLab notebooks.
| Notes/Notebook | Video (if available) |
|---|---|
| notes | notebook video 1 (28:33) |
| notebook video 2 (11:27) | |
| notebook video 3 (18:24) | |
| Notebook 9.1 Convolutions | N/A |
| Notebook 9.2 CNNs | N/A |
This week we'll take a very high-level look at modern large language models.
There are no exercises in the pdf notes this week, but there are exercises in the CoLab notebooks.
| Notes/Notebook | Video (if available) |
|---|---|
| notes | notebook video 1 (8:30) |
| notebook video 2 (12:04) | |
| notebook video 3 (23:17) | |
| notebook video 4 (21:02) | |
| Notebook 10.1 Strings | N/A |
| Notebook 10.2 Tokenization | N/A |
| Notebook 10.3 Attention | N/A |
There will be 2 hour revision sessions on 11am-1pm on Thursdays in QUEENS BLDG F.101a/b/c as usual.