Hi there,
are you running on linux or Windows?
If it’s Windows, it’s quite likely you have all the energy saving preferences turned on.
These preferences essentially turns parts of the computer off as much as possible to save on energy consumption. Unfortunately whilst turning the CPU on and off can often save energy in a way that typical users won’t notice (whether a particular computation takes <1ms or 20-30ms is not typically noticeable) it can have a dramatic impact on live audio manipulation. When the CPU is sleeping and conserving energy, it’s not able to manipulate and generate audio in real time. Of course, computers aren’t really ‘real time’ in that they firstly make operations discrete and then batch a bunch of similar operations together for computational efficiency (to make better usage of caches etc.). However, when the CPU sleeps for longer than the acceptable time duration for a typical batch of operations, things can go awry.
Could you try and turn off the energy saving preferences (if you have permission) and see if that helps things?