No, I think you're OK so far. If you removed too much the tree is going to tell you soon enough, just don't remove any more. You need to let a lot of growth develop to fatten up the trunk.