I thought I was this close today (fingers very close together). I messed around with ByteArray conversion to Bytes, big and little endian, compressed and non-compressed...and finally got it working!
In the end, I just had to ensure my ByteArray was parsed, byte-by-byte into a Bytes class, and fed to the OGG parser. That, and I had an error in my sound filenames that missed some folders. But it worked. I could include a packed audio file in the game assets, and load OGGs from it at runtime!
OSX and Linux weren't having any of that, though. Seg faults and malloc errors no matter what I tried. And it appears to be inside the lime.ndll again. The more I look at it, it seems something inside lime.ndll is incorrectly freeing allocated memory whenever I try to read bytes from a larger file of bytes (as opposed to a file as a whole).
I started back down the path where I rebuild lime.ndll from source, and hit the same roadblocks. Missing header files, unsupported commands. It just isn't compiling.
The one thing I noticed, however, is that if I manually download the source for one of the libraries it depends on, I can pass that step fine and crash at the next one. So I'm suspicious I incorrectly pulled the necessary library source from github. Probably the wrong versions.
I guess that's where I'll start tomorrow. Assuming that fixes the compile issue, that will let me debug the ndll to see more specifically what is crashing. No guarantee I'll know how to fix it, though.