I don't have a problem with him going as ultimately he is the manager and responsible for the team performance. However with the poor hitting and injuries, I can't see anyone really doing much better.
Managers just really don't matter that much in baseball and I'll stick with it.