Having trouble with an MLP not converging? Try updating fewer parameters with each step (only those with largest expected loss reduction). This is a simple mechanism for provably reducing the (unexpected) curvature of the mean-squared-error loss function.

When mean squared error starts increasing precipitously during MLP training, it's often because the dimension of data (/ rank / number of nonzero principal components) starts to collapse inside the model (green line in gif). I'm currently working on strategies to avoid this! 🤔🤓

Promising theory I'm working on: the best preconditioner for MLP's isn't just zero mean unit variance. It's projecting onto the principal components and *then* unit variance. This maximizes the radial symmetry of the data in the 2-norm, minimizing global curvature of MSE loss.

Just finished implementing a truncated linear regression (reduces outputs with SVD then regresses). Will be working this into my neural network "conditioning" function to improve fit performance next!

More than two years later, the last chapter of my dissertation is finally published! 😍 dl.acm.org/doi/10.1145/35… As far as I know, this is now the state-of-the-art for monotonicity preserving spline interpolation software (1D). Fast, reliable, and twice differentiable.😎

Voice mode on dog walks☕️😌 a great way to start the day!

Step 3: Do a rep of gratitude every day. No matter who you are, this is proven to help your mind and your body. And when it feels like there is absolutely nothing you can do, you can do this one thing. Write down one thing you’re grateful for, every day.