huge fan of using AI and copilots to help write common data science code patterns but this marketing is so funny. like 1 million lines of data science code can be one Jupyter notebook with the cell outputs

Why is finding the right dataset for analytics use-cases still time-consuming?

We surveyed ~90 practitioners on why, what, and how of dataset search, and surface key challenges and desired system capabilities 🚀!

paper: madelonhulsebos.com/assets/dataset…
talk: HILDA 2024 ACM SIGMOD’24✨!

would be very curious to know if and how people are using chatgpt memory & custom instructions features to actually teach the LLM things

gpt 4o is way worse than gpt 4 at latex-wrangling for me. e.g., it gives bullets in md format instead of itemize env. i thought about abusing the memory feature (“remember that bullets in latex should start with \begin{itemize}”) but im lazy af and just switched back to gpt4

i am giving a guest lecture on scaling up vibe checks for your custom LLM pipelines, which is essential to constructing a good fine-tuning dataset!! super stoked for it, and excited to be in the company of the awesome folks lecturing ☺️

sign up here: maven.com/s/course/76981…

scores super high on faithfulness (h/t valerie)

Ethan Mollick Why does everyone see these tools then say 'prompt engineering is dead'? It's just been emphasized even more! And how do we evaluate the prompt is better? 'Automatically generates good prompts' --how do we know it's good? It 'works pretty well' --what tests did you perform?

a chief reason for fine-tuning is to get rid of slop in the outputs of your pipelines

Agreed. daniel bashir is a fantastic interviewer. I remember being so impressed by all the preparation that went into my episode. I’ve never seen anything like it

i hate when i have a UX idea that i think is really good on a sunday night so i rush to implement it & get bogged down with debugging my bad typescript & emerge with a working solution hours later only to find that it was a bad idea & i should have just gone to bed early

We will talk about our work, “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences”, at HEAL (well, at least I will be there 😎). Excited to chat with all the amazing people working in this space!

Elicit Grant Sanderson Ian Goodfellow Vik Paruchuri Shishir Patil 9/ Scaling up Vibe Checks by Shreya Shankar

I think she highlights an important fact that evals are going to be tricky because these models can freely improvise responses. I liked the interface and designs that she mentioned at the back and I think manual evaluation of generated

underrated use case of chatgpt's new memory feature 😉 i said 'no yapping; remember this'

Shreya has been a delight to work with! If you are doing anything related to LLMOps, reach out to her!🙂