Do you? open.spotify.com/track/5ARrtxJL…

NEW Scientific Reports article exploring influence of gender, personality, and music preferences on music-induced body movement. Collaboration between Petri Toiviainen & Emily #Carlson with IIIT Hyderabad colleagues Yudhik #Agrawal & Vinoo AR nature.com/articles/s4159…

We love listening to music while playing games, doing house chores, or when working on a paper ✍️🎶 Can we have AI curate music while reading a book? 📖 A fun project with jaidev and Vinoo AR turned into a ISMIR Conference #ISMIR2022 Brave New Idea Award winning paper! 🧵 1/9

Imagine we are reading the first Harry Potter book 🪄 Wouldn't it be nice to have the appropriate movie soundtrack play as we read through various chapters? When Harry discovers that he's a wizard, leads Gryffindor to win at Quidditch, to finally facing Voldemort. 2/9

How? We divide chapters into segments, the movie into scenes, and soundtracks into parts. Then, we align chapter segments with movie scenes, and identify the soundtrack parts that play during these scenes. Thus, the movie acts as a bridge between the book and the music. 3/9

Book↔️Movie↔️Music Previous work provides a coarse alignment between book chapters and movie scenes. We obtain finer matches by scoring narrative paragraphs in the chapter with movie frames using CLIP! Music in the movie scene is matched to the soundtracks parts via Shazam. 5/9

Weaving the soundtrack Chapter segments matched with soundtrack parts enable us to play music composed for the same story. For the unmatched chapter segments, we fill in music by retrieving soundtracks from the same movie closest to the emotion conveyed in the book. 6/9

10 participants read a couple book chapters with music. Some opinions: "The music provided insight into the tone of the chapter [...] beyond imagination, provided a soundtrack to what was read" "When the tension built in the plot, the music transitioned to match it" 7/9

Sonus Texere! Appropriate to the Harry Potter theme, we also made up a spell to create such music 🪄😉 Watch jaidev explain our work in a video ismir2022program.ismir.net/poster_182.html read the paper arxiv.org/abs/2212.01033 or experience a chapter (spoilers!) auto-book-soundtrack.github.io 8/9

This was a cheerful collaboration with Jaidev (jaidev), my first independent study student IIIT Hyderabad, and Vinoo (Vinoo AR) who I approached thanks to a wonderful comms team initiative to write blog posts on faculty blogs.iiit.ac.in/vinoo-alluri/! Fin! 9/9

ISMIR Awards! This year's Brave New Idea Award is for: Jaidev Shriram, Makarand Tapaswi, and Vinoo Alluri for "Sonus Texere! Automated Dense Soundtrack Construction for Books Using Movie Adaptations"

Scientists are surely having fun with AI entering the picture but nothing can replace the experience of reading a physical book, says Vinoo Alluri, who is currently researching on how music changes the experience of reading a book.  newindianexpress.com/cities/hyderab… via The New Indian Express