yeah, they usually start you with the shittiest fief

it's income, i guess? plus if you build it up a bunch they'll start giving you better troops when you recruit (which, in original M&B at least, could be good or bad; sometimes they'd offer archers, which I DID NOT WANT)