Fruit is mostly removed to avoid overly weakening the tree for the most part. Some trees, like Crabapples flower so profusely that the resulting fruit load can draw extreme resources from the plant. In some cases that can actually lead to decline and death over time. Thinning the fruit can help keep the tree strong.
That thinning is done in the spring, as the fruit just begins to grow. Thinning blossoms can achieve the same thing.
Additionally, thinning is done to balance out the visual load. Too many blooms and fruit in one part of the tree, not enough in another, etc. That thinning is done in the spring as well.