We had some code in our product that uses std::thread and other related classes to create and run new real-time threads. Rather than change all of our code to use the Twine worker pool, I changed all of the calls from std::thread to the pthread-compatible calls in Xenomai (__cobalt_pthread_create and related). I’ve got this up and running, but I’m still seeing the MSW field climbing when running sushi.
I am assuming that the “MSW” field is “mode switches” and CSW might be “context switches”, and I don’t know what the XSC field is supposed to represent. On my threads, they stay in lock-step. CSW seems to occur twice for every MSW, and XSC seems to occur 3 times for every MSW. The numbers will continue to climb while the program is running.
I don’t see that we are making any system calls on the additional threads, and I’m assuming that “wait”-ing will not cause a mode switch. Should I be concerned about these mode switches, and do you have any other ideas what may be causing them?
Thanks. Yes, we’ve been using those instructions already. Unfortunately, gdb does not break until I play a note (the application is a virtual instrument). And yet, when I run outside of gdb, the MSW field is incrementing even without playing a note.
Any suggestions as to why we are not hitting the SIGXCPU break during this period?
When I play a note, the program hits some other one-time mode switches, which cause a gdb break. I have not been successful in using the “ignore” command in gdb to skip breaking on the first few mode switches. I did this:
(gdb) catch signal SIGXCPU
Catchpoint 1 (signal SIGXCPU)
(gdb) ignore 1 2
Will ignore next 2 crossings of breakpoint 1.
(gdb) run -r --debug-mode-sw -c ./config_play.json
Starting program: /usr/bin/sushi_b64 -r --debug-mode-sw -c ./config_play.json
The program runs, and I connect my USB MIDI keyboard by typing “aconnect 16 128” in a separate terminal.
But when I go back to the 1st terminal, where gdb is running, I see this:
Program terminated with signal SIGXCPU, CPU time limit exceeded.
The program no longer exists.
I’m not sure what I’m doing wrong, can you give me any advice?
it states: you are not allowed to call any operating system functions from your real-time processing callback.
one of the items that is mentioned as disallowed is Any thread synchronization primitives, like mutexes, semaphores, etc.
Does this include the cobalt calls for mutexes wait/signal, etc.?
Yesterday I tried to go a little deeper into my mode switching issue. I removed all of the “worker” code from my additional real-time thread, leaving only the bare minimum for wait/signal handling. The mode switches continue to accumulate.
I looked at a few other sites to see if I could find out what might be going on. I found this site:
This page gives the advice of adding this call for the current thread: pthread_set_mode_np(0, PTHREAD_WARNSW, nullptr);
So I figured I would try that. Do you think it would be necessary to add this call for new real-time threads?
That call to set the PTHREAD_WARNSW flag is actually done by SUSHI when you activate the --debug-mode-sw flag, but only on its main thread. So you need to call it on any new RT threads if you want to spote MSWs there.
Thanks for the confirmation that this code is required for all real-time Xenomai threads, in order to catch mode switches in the debugger.
Unfortunately, when I tried adding this line of code in my threads, and setting catch signal SIGXCPU in gdb, the debugger was still not breaking at the mode switches, which are still accumulating according to the stat file.
So, I guess it is not the “magic bullet” I was looking for.
Maybe there is something else I can try? If you have any other suggestions, please let me know.
Hi @rick,
this is weird, we never came to a situation where MSWs were not detected by gdb in that way.
A couple of suggestions:
Make a deliberate syscall into one of your threads (e.g. write to file) so that you can verify that you’re properly setting the PTHREAD_WARNSW flag for each thread
As an alternative to spot syscalls, you can run the plugin with the dummy frontend. In that case, you would need to specify MIDI notes in the Events section. With the dummy frontend, you can attach strace to one of the RT threads and monitor syscalls happening there. You won’t be able to access any Xenomai functions with this frontend, though.
I resolved an issue with the pthread_setmode_np() call, and now I can use gdb to catch the SIGXCPU signal.
I have found that our product has a few mutexes which are current used across the audio and the ui threads (to prevent shared access). Is there any approved solution for having a mutex like that? Or is the answer simply “don’t do it”!
I see now that my previous success was not real. The breakpoint that was caught earlier (mutex between audio and ui threads) with SIGXCPU was on the main sushi thread.
In my own cobalt threads I have tried adding code with the following, and none of them trigger a break: malloc/free fopen/fprintf/fclose std::cout << writing something to standard out.
And so it seems to me that the SIGXCPU mechanism is not working for my own cobalt threads. Do you think that maybe I need to add more than simply a call to pthread_set_mode_np(0, PTHREAD_WARNSW, nullptr); on each thread?
Also, do you know whether std::atomic might trigger a mode switch?
That one should be enough, of course it has to be called after the thread has started in the thread local context. Can you try to check the return status (verify it’s 0) and, in case, pass a pointer to an int variable for the last argument instead of nullptr and verify that the mode bit is set on that one?
That depends on which type the std::atomic is guarding and on the architecture. You can verify it at compile time using std::atomic_is_lock_free(). Typically, primitive types such as int, floats and even structs with two of them sometimes, are lock-free. More complex types are implemented by the compiler using mutexes, and those will cause mode switches.
No worries! This is an interesting deep discussion, happy to find out what is the culprit.
That one should be enough, of course it has to be called after the thread has started in the thread local context. Can you try to check the return status (verify it’s 0) and, in case, pass a pointer to an int variable for the last argument instead of nullptr and verify that the mode bit is set on that one?
The return status is always 0. I printed out the value of the mode and it is 0x0000. That’s because the mode returned is the previous value of the mode. If I run the setmode command again, then the value returned is 0x4000, which is PTHREAD_WARNSW.
I had tried to add my deliberate syscall test code after the setmode() but before the typical wait() loop in my thread. If, instead, I add my code after the wait loop which contains __cobalt_pthread_cond_wait() [meaning that the thread has been signaled and is finished waiting], then gdb gets the SIGXCPU signal and breaks properly. So, that is a small victory.
I also verified that this signal is hit in gdb if I only call setmode() once, rather than twice.
The next step, of course, is to remove the test code altogether and run again. In this case, the signal is never hit, and if I watch /proc/xenomai/sched/stat I can see the MSW numbers incrementing.
