Live Coding with Machine Learning (Magenta.js)

mrbombmusic · October 5, 2020, 2:02am

I made a project for a Hackathon last month where the theme was Music and AI (I didn’t win but I did place). It had to incorporate AI/Machine Learning into some type of musical project. I created a GUI that allows you to send drum patterns via OSC into Sonic Pi in real time as well as generate extensions of those drum patterns using a Machine Learning model called a Drum RNN (Recurrent Neural Network) which I got from the Google Magenta Project. You can also live code additional loops in Sonic Pi to accompany the drum patterns that are made in the GUI.

I finally got around to making a video of a Live Coding session I did with it:

Here is a link to the Github repo I made: GitHub - mrbombmusic/sonic-pi-drum-rnn-gui: A GUI that uses Magenta Drum RNN to create and generate drum patterns to send to Sonic Pi via OSC
Here is a video I made which explains how it works: Sonic Pi DrumRNN GUI - YouTube

It requires installing the p5.js-OSC library to make it work locally on your computer.

I’m interested to see if others can get it up and running on their computers. Please let me know if you are able to get it working or have any questions.

I would also be interested in getting this onto the web to make it more accessible for others to use. I know this would require some type of web based OSC library but I haven’t been able to find much about. If anyone is knows more about how to do that, I’d love to know more. Ideally, it would be great to have some type of function in Sonic Pi that could be used in tandem with this instead of just copying and pasting all the code needed to have it work, but that is just wishful thinking at this point.

ethancrawford · October 5, 2020, 2:10am

Looks like fun!
I’d love to see if I can get this up and running soon, and have a play with it. I may not get to that for a little while though - pretty busy for the next few weeks

(P.S. - Any chance you can add a license to your project? Anyway, great work @mrbombmusic, looking forward to trying it out )

mrbombmusic · October 5, 2020, 2:39am

License added!

Let me know how it goes.

feline · December 27, 2021, 6:21am

This is so excellent. But its not working for me:
Runtime Error: [buffer 1, line 89] - NoMethodError
Thread death ±-> :live_loop_playDrumPatterns
undefined method []' for nil:NilClass workspace_one:89:in block (3 levels) in __spider_eval’

Please help. Thank you.

ethancrawford · December 27, 2021, 6:51am

@feline - what version of Sonic Pi are you using?

feline · December 27, 2021, 7:18am

3.3.1 on windows,
Thanks

ethancrawford · December 28, 2021, 2:39am

Have done some brief tests and I can see the same result.
No OSC messages appear to arrive in Sonic Pi, despite setupOsc referring to port 4560.

@mrbombmusic - I notice in my browser console that I get a whole pile of CORS errors:

Could this be why nothing is sent through? Not entirely sure, or how to fix it if so.

feline · December 28, 2021, 9:45pm

Maybe it works on Mac, I tried Windows and Linux. Same CORS errors.
Anyway, techno machine works and it was great fun to play with.
The beauty is in not knowing what will happen, like fractals.

Thanks,

mrbombmusic · December 29, 2021, 12:20am

Hi there. Sorry to see it isn’t working for you.

The error message you are getting seems to be saying that there isn’t any drum pattern to play yet. The way it works is by sending the pattern made in the GUI grid to an array in Sonic Pi. I seem to recall getting similar errors when I would try running Sonic Pi prior to sending any drum patterns from the GUI. That would explain the ‘[ ]’ part at the beginning of the error message and why the error is coming from the playDrumPatterns live loop which is where the array for the drum pattern would play.

I would recommend making a pattern in the GUI and sending it to Sonic Pi before you actually run the Sonic Pi code. You might need to try hitting run a few times in Sonic Pi.

Also be aware that this was built in version 3.2.0 and I have not tried running it on any other versions. Someone else also raised an issue in the Github which might have also been related to using a different version. At some point, I will try to run it on 3.3 and see if it works for me.

I also made it on a Mac and can not speak to how it might work on other operating systems.

@ethancrawford I can’t speak to the CORS errors as I don’t even really know what that means. It is beyond my level of understanding. I didn’t originally make this project with the intention of it being used outside of my computer. Its great to see others trying to use it, but unfortunately I don’t have all the technical know how to trouble shoot why it may not work. If you are not seeing any OSC messages in Sonic Pi, I am wondering if it has something to do with the p5.js OSC library?

ethancrawford · December 29, 2021, 12:49am

Sure. I’m not terribly familiar with it either as you may have guessed my guess was also that it might be to do with the p5.js OSC library. If that’s indeed the case, then if it or a different JS OSC library can be made to work, data might make it through to Sonic Pi

ethancrawford · December 29, 2021, 5:55am

@mrbombmusic / @feline - see the following for a little fix which should hopefully prevent the errors from appearing when starting the Sonic Pi script.
The only small procedure change in my update is that it would now necessary to start the Sonic Pi script first, before using the drum GUI.
I tested this on v4 beta on Windows, but it should work the same in 3.3.1 too.
Let me know what you think

github.com/mrbombmusic/sonic-pi-drum-rnn-gui

Make Sonic Pi receiver script more robust

mrbombmusic:master ← ethancrawford:master

opened 05:48AM - 29 Dec 21 UTC

ethancrawford

+29 -41

Previously, the script would generate several errors. One seemed to be due to c…alling an invalid method on a hash. Others seemed to be due to certain time state variables that sometimes may have not been initialised properly. These have now been fixed - the Sonic Pi script should (hopefully) no longer generate errors when getting started. However, it also will not play any sounds until at least both of the following are done (in any order): - the user has sent the appropriate beats with the 'Send Beat' and/or 'Send Generated Beat' buttons, depending on which type(s) of beat they wish to play - the user has selected a drum kit with the kit dropdown selector.

feline · December 29, 2021, 7:11am

Sonic pi receiver.rb no errors,

and console in Chrome : polling-xhr.js:157 GET http://127.0.0.1:8081/socket.io/?EIO=4&transport=polling&t=Nu50l70 net::ERR_CONNECTION_REFUSED

and node bridje.js follows with
node:dgram:917
throw new ERR_SOCKET_DGRAM_NOT_RUNNING();
It must be P5js-osc issue.

ethancrawford · December 29, 2021, 1:49pm

How are you starting your web server/accessing the web GUI?
I’ve been able to get things working on Windows and macOS (with my proposed patch that I linked above) by doing the following:

starting the node bridge according to the instructions at GitHub - genekogan/p5js-osc: OSC for p5.js with examples
starting a python web server similarly to how someone did it as described in Sending OSC to Sonic Pi. · Issue #2 · mrbombmusic/sonic-pi-drum-rnn-gui · GitHub
loading the web GUI in a browser at http://localhost:8000/GUI Code/
making sure that on the following line, the port number used to send OSC outwards from script.js was edited to match the number labeled Incoming OSC port in the Sonic Pi IO settings (on my macOS machine it was different):

github.com

mrbombmusic/sonic-pi-drum-rnn-gui/blob/9a544efb5f4da6ad1f55b1c157211a25dbf6af32/GUI Code/script.js#L4

    
      
          var socket;
          var isConnected;
          
          
setupOsc(12004, 4560);
          
          

          
let sequencer = new Nexus.Sequencer('#sequencer', {
            columns: 16,
            rows: 9,
            size: [600, 200]
          });
          
          
let genSequencer = new Nexus.Sequencer('#genSequencer', {
            columns: 16,

(In my case, the incoming port that Sonic Pi reported in the preferences panel was not 4560, so I changed that from 4560 to the value it did say, in the line shown above in the script.js file).

Starting the Sonic Pi receiver script
Creating beats in the web GUI, pressing send beat, and selecting a kit

…and I had sound.

feline · December 30, 2021, 9:55am

Yes it works. It was npm install in p5js-osc. It said to do force audit, I complied and that was a mistake.
Once I left it alone, it all worked.
Never give up.
Thank you Ethan. And now the music.
Happy New Year!

ethancrawford · December 30, 2021, 3:01pm

Great to hear you got it working!
Happy new year to you also

Topic		Replies	Views
Sonic Pi Live Coding with Algorithmic Music Generator Creations & Ideas	7	1158	May 31, 2022
We’re Making an AI Assistant for Sonic Pi and We’re Looking for Code Submissions Creations & Ideas	2	808	December 18, 2021
Live coding performance video. My first! Creations & Ideas	13	400	June 4, 2024
Typing to Sonic Pi drum pattern code generator Creations & Ideas	3	1649	April 26, 2019
I did a Drum Pad Kit in Python using Sonic Pi's OSC messages Creations & Ideas	3	81	November 2, 2024

Live Coding with Machine Learning (Magenta.js)

Related topics