🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

It is a couple years into LLM-world, and I continue to see lots of bias and ethics papers that study jailbreaks. I have yet to see *one* paper with 1) engagement with real world harms and 2) even a description of what a jailbreak is and is not. This is not science guys #ACL2024

Nice article on deficiencies in MMLU-like benchmarks in mass-market The Economist . Points out they are out of date, sloppily built, often wrong, leaked into LLM training data, unstable, and gamed by vendors. Wonder if business/govt types will stop treating these as gospel...

Chatted to stranger on train. I said LLMs impressive and exciting, he thought I meant AGI and extinction risk. Said I worked on AI in healthcare, he assumed this meant medical diagnosis. Bit frustrating, but I guess reflects the way media talks about AI to general public.

An example where AI literally harmed people: an AI generated mushroom identification book!

New blog: The latest/trendiest tech isnt always appropriate ... I saw this very strongly in the late 2010s with LSTMs (which do not work well for data-to-text), and continue to see this in 2024 (GPT4 is not always the best approach)... ehudreiter.com/2024/08/26/the…

Sometimes the latest technology is *not* appropriate for an NLG task. I saw this very strongly in the late 2010s with LSTMs (which do not work well for data-to-text), and continue to see this in 2024 (GPT4 is not always the best approach). Researchers/devs need to be open-minded

Honoured and excited to be giving a keynote on evaluation at NLPCC in China. Also look foward to meeting people in the Chinese NLP community, who are doing great work tcci.ccf.org.cn/conference/202…

We are organizing a shared task on (dis)agreements among annotators in lexical semantics. The task provides human judgements over seven languages - a useful resource for some of you looking at the fundamentals of disagreements such as their complexity and underlying causes.

A story about fraud in the AI research community: On September 5th, Matt Shumer, CEO of OthersideAI, announces to the world that they've made a breakthrough, allowing them to train a mid-size model to top-tier levels of performance. This is huge. If it's real. It isn't.

New blog: One-day class on NLG evaluation ehudreiter.com/2024/09/09/one… In early Sept I ran a one-day class on evaluation. I summarise what I did in this class and give links to my presentations, in case this is useful to other people.

Call for participation: if you plan to attend INLG 2024 in Tokyo, join us on Sept 23 for the 2nd Workshop on Practical LLM-assisted Data-to-Text Generation! Website: practicald2t.github.io (1/4)

On holiday in Shetland with Ann. I don't usually post holiday pictures, but this is 50m from our self catering flat, and is the kind of scenery i love

I worry that: (A) At a superficial level, LLMs can do amazing human-like things (B) Many NLP "evaluations" of LLMs are meaningless, and community doesnt seem to care Therefore (C) Extravagent claims are made for LLMs based on garbage evals, and taken at face value

Congratulations to Allmin Pradhap Singh Susaiyah for passing his PhD defence! I helped to supervise Allmin (who is at Eindhoven), it was a pleasure to work with him.

Really interesting to see this analysis of how people actually use chatgpt for news

The widely-used wordfreq database of English word frequencies will no longer be updated.

The proceedings for the 17th International Natural Language Generation Conference has been published by siggen_acl and is now available: aclanthology.org/events/inlg-20… I am looking forward to seeing everyone in person next week in Tokyo! #nlg #nlp #inlg2024

LinkedIn is using content to train AI models, but not in Europe! (linkedin.com/help/linkedin/… , click on Can I Opt Out). Shows benefits of strong data protection laws.

New blog: How AI can help reform UK NHS ehudreiter.com/2024/09/23/how… The UK government wants to reform the UK health system by digitisation, shifting care to communities, and focusing on prevention. I think there is a lot of potential for AI to help with this...

UK govt wants to reform the National Health Service to shift care to GPs and community care, and to focus on prevention of illness. AI can help with this if AI/Med researchers focus on these topics, instead of being fixated on improving diagnoses in hospitals.