Live Coding with Machine Learning (Magenta.js)

I made a project for a Hackathon last month where the theme was Music and AI (I didn’t win but I did place). It had to incorporate AI/Machine Learning into some type of musical project. I created a GUI that allows you to send drum patterns via OSC into Sonic Pi in real time as well as generate extensions of those drum patterns using a Machine Learning model called a Drum RNN (Recurrent Neural Network) which I got from the Google Magenta Project. You can also live code additional loops in Sonic Pi to accompany the drum patterns that are made in the GUI.

I finally got around to making a video of a Live Coding session I did with it:

Here is a link to the Github repo I made: https://github.com/mrbombmusic/sonic-pi-drum-rnn-gui
Here is a video I made which explains how it works: https://youtu.be/9GHz_-xZPQs

It requires installing the p5.js-OSC library to make it work locally on your computer.

I’m interested to see if others can get it up and running on their computers. Please let me know if you are able to get it working or have any questions.

I would also be interested in getting this onto the web to make it more accessible for others to use. I know this would require some type of web based OSC library but I haven’t been able to find much about. If anyone is knows more about how to do that, I’d love to know more. Ideally, it would be great to have some type of function in Sonic Pi that could be used in tandem with this instead of just copying and pasting all the code needed to have it work, but that is just wishful thinking at this point.

7 Likes

Looks like fun!
I’d love to see if I can get this up and running soon, and have a play with it. I may not get to that for a little while though - pretty busy for the next few weeks :sweat_smile:

(P.S. - Any chance you can add a license to your project? :crossed_fingers: Anyway, great work @mrbombmusic, looking forward to trying it out :slight_smile:)

1 Like

License added!

Let me know how it goes.

2 Likes

This is so excellent. But its not working for me:
Runtime Error: [buffer 1, line 89] - NoMethodError
Thread death ±-> :live_loop_playDrumPatterns
undefined method []' for nil:NilClass workspace_one:89:in block (3 levels) in __spider_eval’

Please help. Thank you.

@feline - what version of Sonic Pi are you using?

3.3.1 on windows,
Thanks

Have done some brief tests and I can see the same result.
No OSC messages appear to arrive in Sonic Pi, despite setupOsc referring to port 4560.

@mrbombmusic - I notice in my browser console that I get a whole pile of CORS errors:

Could this be why nothing is sent through? Not entirely sure, or how to fix it if so.

Maybe it works on Mac, I tried Windows and Linux. Same CORS errors.
Anyway, techno machine works and it was great fun to play with.
The beauty is in not knowing what will happen, like fractals.

Thanks,

1 Like

Hi there. Sorry to see it isn’t working for you.

The error message you are getting seems to be saying that there isn’t any drum pattern to play yet. The way it works is by sending the pattern made in the GUI grid to an array in Sonic Pi. I seem to recall getting similar errors when I would try running Sonic Pi prior to sending any drum patterns from the GUI. That would explain the ‘[ ]’ part at the beginning of the error message and why the error is coming from the playDrumPatterns live loop which is where the array for the drum pattern would play.

I would recommend making a pattern in the GUI and sending it to Sonic Pi before you actually run the Sonic Pi code. You might need to try hitting run a few times in Sonic Pi.

Also be aware that this was built in version 3.2.0 and I have not tried running it on any other versions. Someone else also raised an issue in the Github which might have also been related to using a different version. At some point, I will try to run it on 3.3 and see if it works for me.

I also made it on a Mac and can not speak to how it might work on other operating systems.

@ethancrawford I can’t speak to the CORS errors as I don’t even really know what that means. It is beyond my level of understanding. I didn’t originally make this project with the intention of it being used outside of my computer. Its great to see others trying to use it, but unfortunately I don’t have all the technical know how to trouble shoot why it may not work. If you are not seeing any OSC messages in Sonic Pi, I am wondering if it has something to do with the p5.js OSC library?

Sure. I’m not terribly familiar with it either as you may have guessed :joy: my guess was also that it might be to do with the p5.js OSC library. If that’s indeed the case, then if it or a different JS OSC library can be made to work, data might make it through to Sonic Pi :slightly_smiling_face:

@mrbombmusic / @feline - see the following for a little fix which should hopefully prevent the errors from appearing when starting the Sonic Pi script.
The only small procedure change in my update is that it would now necessary to start the Sonic Pi script first, before using the drum GUI.
I tested this on v4 beta on Windows, but it should work the same in 3.3.1 too.
Let me know what you think :slightly_smiling_face:

Sonic pi receiver.rb no errors,

and console in Chrome : polling-xhr.js:157 GET http://127.0.0.1:8081/socket.io/?EIO=4&transport=polling&t=Nu50l70 net::ERR_CONNECTION_REFUSED

and node bridje.js follows with
node:dgram:917
throw new ERR_SOCKET_DGRAM_NOT_RUNNING();
It must be P5js-osc issue.

How are you starting your web server/accessing the web GUI?
I’ve been able to get things working on Windows and macOS (with my proposed patch that I linked above) by doing the following:

(In my case, the incoming port that Sonic Pi reported in the preferences panel was not 4560, so I changed that from 4560 to the value it did say, in the line shown above in the script.js file).

  • Starting the Sonic Pi receiver script
  • Creating beats in the web GUI, pressing send beat, and selecting a kit

…and I had sound.

1 Like

Yes it works. It was npm install in p5js-osc. It said to do force audit, I complied and that was a mistake.
Once I left it alone, it all worked.
Never give up.
Thank you Ethan. And now the music.
Happy New Year!

2 Likes

Great to hear you got it working!
Happy new year to you also :grinning_face_with_smiling_eyes: