It's less about being away from the field than it is too much for one person to officiate without way too many stops for reviews.
But on the topic of actually having people on the field, how would handle a pitcher headhunting if the ump is in New York? Wait for a phone call? A light behind the plate that says someone is ejected?