Optimize according to your mental model; profile to check that your mental model matches reality.

If you optimize infrequently, or haven't profiled code like this recently, or haven't profiled this specific codebase recently, then your mental model is probably due a refresh.