When in doubt, train for longer. Here are some cursive transformer samples from a little past 100k steps. The model crosses t's and dots its i's. This involved downsampling stroke data as aggressively as possible, running an A100 for several hours and using a smaller context.

Tried a stroke tokenizer that discretizes (x,y) stroke coordinates instead of (dx,dy)->(d_r, d_theta). Even with high-res binning, wiggles are visible. This approach is def inferior to the (d_r, d_theta) tokenizer we've been using. One of those things we needed to verify

Better downsampling method is leading to additional quality improvements.

Slowly increasing dataset size to reduce overfitting.

With a few modifications to the sampling code last night, we were able to start doing custom generations. Model supports uppercase, lowercase, digits, and some punctuation. It's not perfect, but it's improving rapidly, and we expect to be writing entire paragraphs soon!

Jules Verne, "Paris in the Twentieth Century," 1863