i am giving a guest lecture on scaling up vibe checks for your custom LLM pipelines, which is essential to constructing a good fine-tuning dataset!! super stoked for it, and excited to be in the company of the awesome folks lecturing ☺️

sign up here: maven.com/s/course/76981…

scores super high on faithfulness (h/t valerie)

Ethan Mollick Why does everyone see these tools then say 'prompt engineering is dead'? It's just been emphasized even more! And how do we evaluate the prompt is better? 'Automatically generates good prompts' --how do we know it's good? It 'works pretty well' --what tests did you perform?

a chief reason for fine-tuning is to get rid of slop in the outputs of your pipelines

Agreed. daniel bashir is a fantastic interviewer. I remember being so impressed by all the preparation that went into my episode. I’ve never seen anything like it

i hate when i have a UX idea that i think is really good on a sunday night so i rush to implement it & get bogged down with debugging my bad typescript & emerge with a working solution hours later only to find that it was a bad idea & i should have just gone to bed early

We will talk about our work, “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences”, at HEAL (well, at least I will be there 😎). Excited to chat with all the amazing people working in this space!

Elicit Grant Sanderson Ian Goodfellow Vik Paruchuri Shishir Patil 9/ Scaling up Vibe Checks by Shreya Shankar

I think she highlights an important fact that evals are going to be tricky because these models can freely improvise responses. I liked the interface and designs that she mentioned at the back and I think manual evaluation of generated…

underrated use case of chatgpt's new memory feature 😉 i said 'no yapping; remember this'

Shreya has been a delight to work with! If you are doing anything related to LLMOps, reach out to her!🙂