Ivan and By are two-three generations younger than Ezar, so it makes sense for social attitudes to have changed.

Ezar was in his 40s, which is rather late by any standard. Either he was just about to marry (after some delay or something), or he was planning not to and using military service as a smokescreen. I do not think he was anywhere near the throne genealogically; he was merely a Vorbarra and married a woman with better connections.