I think this is similar to what I experienced with my attempts to use SP for Monome applications (developement stopped; code for Open Stage Control and Monome grid).
This is kind of hard to reproduce with a simple setup because IMHO the sheer amount of OSC messages and interaction seems to be the reason. There is for sure space for improvements of the code but as far as I see the performance breaks down to early - at least for that kind of application.