Homomorphic encryption (HE) allows computations on encrypted data, leaking neither the input nor the computational output. While the method has historically been infeasible to use in practice, due to recent advancements, HE has started to be applied in real-world applications. Motivated by the possibility of outsourcing heavy computations to the cloud and still maintaining end-to-end security, in this paper, we use HE to design a basic audio conferencing application and demonstrate that our design approach (including some advanced features) is both practical and scalable. First, by homomorphically mixing encrypted audio in an untrusted, honest-but-curious server, we demonstrate the practical use of HE in audio communication. Second, by using multiplication operations, we go beyond the purely additive audio mixing and implement advanced example features capable of handling server-side mute and breakout rooms without the cloud server being able to extract sensitive user-specific metadata. Whereas the encryption and decryption times are shown to be magnitudes slower than generic AES encryption and roughly ten times slower than Signal's AES implementation, our solution approach is scalable and achieves end-to-end encryption while keeping performance well within the bounds of practical use. Third, besides studying the performance aspects, we also objectively evaluate the perceived audio quality, demonstrating that this approach also achieves excellent audio quality. Finally, our comprehensive evaluation and empirical findings provide valuable insights into the tradeoffs between HE schemes, their security configurations, and audio parameters. Combined, our results demonstrate that audio mixing using HE (including advanced features) now can be made both practical and scalable.