Just parroting what I've heard from other posts regarding fruiting... Most people prevent from fruiting to allow the tree to use that energy developing other areas. I have wanted to get something that produces some fruit but i'm afraid I would want to go against that same advice to see the fruit!
Edit: I guess your question was if you should remove the fruit, not if you should prevent it! I would think it depends on how much energy it will expend trying to sustain the fruit vs if it was removed.