Week 6 Learning Material

Structuring Machine Learning Projects - Week 1

This week covered the following topics:

• Why ML Strategy
• Orthogonalization
• Single Number Evaluation Metric
• Satisficing and Optimizing Metric
• Train/Dev/Test Distributions
• Size of the Dev and Test Sets
• When to Change Dev/Test Sets and Metrics?
• Why Human-level Performance?
• Avoidable Bias
• Understanding Human-level Performance
• Surpassing Human-level Performance
• Improving your Model Performance
• Andrej Karpathy Interview

The quiz for this week was particularly interesting. Andrew compares the testing method to a "flight simulator". Rather than just asking questions related to the topics covered it provides a fictional scenario and asks the actions you would take as you train an LLM for a customer. It walks through situations and asks you to make decisions on parameter tuning, changing training methods, adjusting training sets etc. I found this method of testing much more effective in connecting the underlying intuitions with real world use.

Applied Techniques

No progress on writing or training anything yet. Just more investigation of the Hugging Face community and what people are working on to gain some knowledge on how the community is tackling problems.

I came across this post this week on how security researcher Sean Heelan used o3 to experiment in finding a 0 day that he himself discovered and actually finding another 0 day. o3 found the new 0 day 1 in 100 tries which I would wager is better than human level performance if you had 100 kernel experts looking at the same code base. Fascinating work.