This is weirdly not understood enough. At a very high level, hallucinations are due to a generative model “filling the blanks” which corresponds to its capability to generalize.

A lot of people don’t realize that LLMs, and generally ML, relies on fundamental hypotheses to work and generalize Flaws like prompt injections, hallucinations, or adversarial examples exist due to these fundamental hypotheses. Aka these are inherent to current model designs!!!

Programming languages are also like this. Python hides away a lot of low level complexity (relinquishing control and, sometimes, efficiency) but is very close to English. We trade off needing to specify some things for not being able to specify them (easily) when needed (6/9)

Attacks such as this highlight the importance of system-level security analysis at all stages of model deployment, starting as the design of the architecture, and extending towards as late as the actual deployment of the model and how different user queries are batched together.

At least part of the problem is people stubbornly refusing to apply the lessons of the past w/r/t appsec and safety-critical systems to AI systems because "✨AI is different✨". Whether that last is a true or simply convenient belief is up for debate.

this

I'm a PhD student at University of Oxford and I think I'm living in a fairytale :-) Foxes playing around in the snow at Magdalen College this morning — absolutely magical!

I strongly recommend turning this off. It's unbelievable that they quietly enabled this while everyone was focused on their 'Recall' AI feature. Now they're collecting and using everyone's Microsoft Word and Excel data to train their AI models.

Incredible opportunity right there!

Congrats Nicolas Papernot !

Does anyone know companies hiring for entry level roles (in Canada/remote)? And I mean *real* entry level, not degree + 2 certs + 3 years experience “entry level”. Not just cybersecurity, any entry level roles at all, in any area.

Dropbox is looking for a senior ML engineer to join our threat intelligence and product trust & safety team, link in reply. DM me if you want to know more

A lot of what I did in the early days of AIRT was actually just building in Security process…

DM is ok, but Chris is fun to work with :D

Okay. Homoglyph jokes are too subtle for this audience. I understand, it’s fine

Congrats Varun Chandrasekaran and well deserved as always !

Wanted to share that I just finished my MBA. Two graduate degrees, my undergrad and a CISSP. Now all I have to do is start attending RSA… ;)

One of the best examples of unintentional data poisoning: replicating a common pattern of public code that was used as training data for an LLM. Workarounds are everywhere because they are often fast solutions. Actual fixes can often take longer and thus may be less frequent

Totally agree. Application design is primordial in LLM/agentic security because of how hard it is to actually make the model itself safe. By the way, isn’t that documented as “the golden rule” in the Nvidia Guardrails’ documentation’s LLM security guidelines Rich Harang (@ BlackHat / DEF CON) ?

Some of the most brilliant minds I know are building one benchmark after another instead of finding more principled ways of understanding behaviours. Is this what science has come to?