Canonically, they are only usually a different sex, and there's no canonical "explanation" for those that aren't (fannish speculation has generally been that those people are gay or trans, but I kind of think that may be overly simplistic; I kind of like it not meaning anything specific, just being a thing that happens sometimes).

Also, I'd agree about the human imagination thing as a limit--plus I don't think people ever have daemons that take forms they wouldn't know about?