I skipped the part where you said you would build your own hat. My bad.
That said, do you really need a microprocessor for transmitting multiple streams of audio over wireless. It will work, but for that application do you need the power of the hardware device?
It sounds like you could use a DAC->wireless without much in between. Now if you could multiplex the wireless, want to have some type of digital mixing, or EQ that might suggest something more powerful.
The benefit of the Raspberry PI is that it streamlines development.