Olive trunk removal?

I would like to take off this section of straight trunk but was unsure if this late in spring was ok to make such a big cut. The tree seems healthy with buds at the end of every branch but 60% of the foliage is on that trunk. Opinions or advice would be appreciated. Thanks.

You say the tree has buds now?
It hasn't began growing yet this season?

No, it has. That came off wrong, I meant that it's growing well.