To be very clear: the basic requirements to do low-latency performance without modifying Elk itself are to use a Raspberry Pi 3B / 3B+ together with the Elk Pi board, as we write on the website.
In addition to that, it is relatively simple to adapt Elk Audio OS running on Raspberry Pi 3B/3B+ to work with another audio codec but still keep in mind that this requires some expertise with Linux driver development and is not for anyone. Still, it is way simpler than starting from scratch.
Supporting other Raspberry Pis like the Zero is a pretty difficult job. The Compute Module version of the 3B / 3B+ is relatively simple, though, as it has the same SOC and main components.