I’ve been trying out Frederic’s SOURCE sampler (Very cool btw), and i’ve encountered an odd bug,
after loading some sounds and then playing a few notes i left it for about an hour, when i came back
the sound was extremely distorted and noisy, rebooting with “sudo init 6” had no effect but shutting it down with “sudo init 0” and then disconnecting and reconnecting power fixed the problem. A few days later i experienced the same problem on another ElkPi loaded with SOURCE. Unfortunately (and quite foolishly) i didn’t save any logs or check if other plugins were affected at the time and so far i haven’t been able to recreate the bug. I’ve appended the --log-level=debug, --log-flush-interval=5 and --timing-statistics to the sushi service file and i am aware of the dmesg command, is there anything else i should do?
Hi @Jonathan, thanks for posting 
I’m supper happy to see SOURCE being loaded in another ElkPi! Also I’ve seen the instructions might need improvement so great that you got it working.
This distortion issue I also experience it from time to time. As you say, power cycling with reconnecting power fixes it. I think at some point I wrote in the forums about that, but never did any further research so I’ll appreciate if you find a solution and share it.
Cheers,
frederic
I’m sorry that this is a bit rushed but I’m leaving for a few days and i wanted to post this before i left.
I’ve experienced the bug again and this time i was able to get logs and other stuff.
The attached file contains a script to save logs to directories, the log directories (times aren’t correct but they seem to be in order), 2 recordings of the distortion/noise (was only able to record with a smartphone in a pair of headphones, sorry), a copy of /udata/source in the “2021-06-25_05:05:43” log directory (Freesound api keys censored), and notes showing what was done/tried, and in what order.
source-debug.zip
TL;DR.
- Even after stopping source, sushi, and sensei, and then starting sushi with other configs like the config_play_lv2_jx10.json the distortion remained.
- After rebooting with the source, sushi, and sensei services disabled, and then starting sushi with the jx10 config, the synth is now only partially distorted/noisy.
- Shutting the pi down and then disconnecting and reconnecting the power fixes the distortion/noise completely.
Since this seams to break the audio for every plugin loaded after it and at least partially survives soft reboots, I think someone from the elk team should take a look.
Hi Jonathan and thanks for the very detailed logs. I have looked at them and listened to the audio files you provided. They definitely sound very strange, some kind of modulated noise in there. Do you hear the noise only when running sushi or does it remain even after closing sushi? I understand it does persist when running sushi again.
The most interesting one to me is the one in 2021-06-25_05:05:43, In that one you can se in the captured /proc/xenomai/sched/stat that there has been a lot of mode switches in the realtime thread. I cannot see that in any of the other logs.
Though the fact that it remains even after restarting sushi (or even after soft rebooting the board?) makes me suspect it could be a lower level issue (drivers, cpld or even audio codecs). I’ll pass the logs along to one of our firmware experts.
Thanks for responding, I’m very happy someone’s taking a look at this.
When using source and the jx10 plugin the noise only occurs when playing a sound, with the vst2 multi track guitar synth and vst3 fx configs the noise was constant until i stopped sushi.
The mode switches are rather interesting, every time i hit a key MSW increases by 1, once i release the key the MSW again increases by 1, also sometimes after booting without touching anything MSW will increase by 1 every few minutes while other times (tested after a complete shutdown) MSW will increase by 1 approximately every 0.5 to 3 seconds randomly (after 40 minutes it was already at 240).
I’ve also figured out what seams to be a reliable way to cause the bug, just run this python script on the elk system and reboot with “sudo init 6”.
I also recorded some better audio.
Source Clean
Source Bad reboot 1
Source Bad reboot 2
Source Bad reboot 3
lv2 jx10 Bad reboot 3
Thanks for the audio files.
I noticed something in the sushi logs. For every midi key event, there is also a call to the vst host callback with opcode audioMasterProcessEvents, i.e. the plugin wants to send an event out from the plugin. Now this isn’t handled by sushi, but if logging in debug mode it logs an entry, which, if done in the process callback could cause a mode switch.
Why does the plugin pass through midi data, and if you set log level to “info” do you still get the mode switches for every key press/release?
My working hypothesis is that the plugin does something bad in the realtime thread and that this in turn messes up the driver or audio codec/cpld code. But I’ll see if I have time to try to reproduce.
Hi @Jonathan,
This seems like a driver issue. But it is caused most likely because of those MSWs (I guess). And the soft reboot will not work because the CPLD won’t be reset by that and anything you run afterwards will be distorted. You need to do a hard-power reset which seems to be fixing the issue. I need soome info to debug this.
- How often or how quickly can you produce this issue after a fresh reboot (power on) ?
- Can you send the ‘dmesg’ log that you see after you have got this distortion.
- It would be interesting to see if it happens without the MSWs occuring.
Thanks,
Nitin
@Gustav,
I did not make the plugin so i have no idea why it passes through midi data.
After setting sushi’s log level to info, pressing keys no longer created “vst host callback” messages or increase MSW, but MSW will still increase on its own as described in my earlier post.
Also the spam.py script doesn’t seam to be necessary, starting from an unplugged elkpi with sushi log level set to info, without touching any keys, connect power, ssh in and reboot with “sudo init 6”, pressing keys now causes noisy audio.
@Nitin,
After powering on the elkpi and then rebooting with “sudo init 6”, there is no point where i am able to play notes and the audio is not noisy, so far this is completely repeatable.
Here are the dmesg logs, Before reboot, After reboot, After reboot load lv2 jx10 config.
Also the amount of noise changes every time i reboot, in this example the noise and volume remains constant until the note ends (I’m not holding the keys btw), while in this example the amount of noise increases when i hit the key harder (higher velocity).
Hi everyone,
Sorry I missed this conversation. Are this experiments being done using SOURCE (the plugin) or using another plugin?
I’m no expert RT-audio programmer and it is possible that some unexpected mode switches are happening in SOURCE, although I did some things to avoid that and hopefully it should not be happening.
The problem can be replicated 100% of the time with the SOURCE plugin, just start sushi with the plugin and reboot with “sudo init 6”.
I also haven’t tried replicating this problem with any other plugin yet.
If you could try replicating this problem on your end that would be great, if you could also try a debug build as well that would be even better.
Perhaps we can get some attention back on this issue.