redesigned the mini shoggoth body to fit the stereo camera inside. also added a little spike at the tip to make it feel more alien. my roommate also made me notice that it looks a lot like a little green men head with a weird giger-like mandible when held upside down,

sim2real achieved! mini shoggoth can now (slowly) track hands. to what purpose i do not yet know stack is PPO + mujoco + dynamics randomization + yolo11n

The Worldwide LeRobot hackathon is in 2 weeks, and we have been cooking something for you… Introducing SmolVLA, a Vision-Language-Action model with light-weight architecture, pretrained on community datasets, with an asynchronous inference stack, to control robots🧵

worked on having gpt4o drive open-loop motor primitives like waving or grabbing from stereo vision. special tokens like <distance: 10cm> get streamed straight into the transcript stream, which gpt4o reacts too. next up is integrating the closed-loop hand tracking. main challenge

approaching the end of v1 here :) worked on some new primitives, and integrating them all together. making the gpt4o realtime api consistent in its use of the primitives was harder than i thought. i will now start writing the blogpost and cleaning the vibecode a bit for

Learning where to look, learning what to see. Work on Continuous Thought Machines continues to be exciting! As a teaser, take a look at our approach to "Foveated Active Vision". Check it out 👇 (details in the thread) pub.sakana.ai/ctm

played around with action smoothing due to control instabilities in the PPO policy. see how the raw predicted action diverges as it goes OOD. happy to find that exponential moving average is extremely effective here!

in dexterity tests, shoggoth mini reveals strong passive grasping generalization

just found out the white in human eyes is adaptive as the contrast makes gaze direction obvious, which helps in communicating intent much better than other animals sciencedirect.com/science/articl…

i've been working on a blogpost about the project these past few days, hopefully released by the end of the week! here's a cute animation of the system design

wrapped up the blogpost and code for the shoggoth mini project! will release everything tomorrow 👀