I don't think Barrayar's smaller, just considerably less populated. It took Earth a long time to reach several billion people, and that's without whole continents full of inedible plants.

It'd be easy to have a relatively unified culture, since it started roughly that way, with any pocket that retained a separate identity at all being considered weird.