A few months ago I decided to train Karpathy’s nonGPT on one of my WhatsApp group chats. This is a group of nine friends, and we have been chatting (quite intensively) for the past ten(!) years. Wh...

When I started learning about transformers, I couldn’t find a discussion of the transformer architecture that was easy for me to follow. I assume it is mainly because of my background in physics, a...

In this post I’m going to introduce PAdam, my new variant of Adam optimizer which allows for any $p$-norm to be used as a regularizer. Update: We have now published a paper on PAdam which can be f...

A second post about generalization toy models. In this post I aim to study the effect of sparsity inducing regularization schemes on generalization. Logic and challenges The first post focused on ...

As part of my attempts at better understanding Lasso regularization (more on that to come soon!), I’ve been thinking more broadly about regularization in ML recently. One thing I was able to unders...

I’ve been thinking about Lasso regression lately, and more specifically, about regularization. I wanted to see if there was a way to reformulate Lasso regularization as a quadratic convex problem. ...

Often times, we wish to fit a line to data. For many of us, the first method that springs to mind is least squares. But what exactly is the “correct” least squares test? This post will tackle the s...

I’m freshening up my statistics, and trying to make a better sense of the Central Limit Theorem (CLT). There are quite a few derivations out there, including a comprehensive post by Terence Tao. Ye...