No reason to be hasty. But when you see something that detracts from the tree, then it's time to resolve that issue. Mine was in keeping too heavy apex branching. Now I'm happier with the direction. Yours could be fine in that respect, just hard to see. I'm just surmising that this may be the case, as you are saying that there is no strength down low. I think that yours is such a fine trunk, that after you do some trimming, you should open up the view into the canopy so we can see those branch lines. I think that this is a key for this specie, as otherwise, they just look like a big ol mop.
Take your time, will be watching for updates!