Pea gravel is typically too large for any but the largest of trees. In addition, it is mostly impermeable stone, so it is sortof like planting your tree in ball-bearings. The reason people use pumice, lava, akadama, kanuma, haydite, etc, is that they all absorb water and keep the soil moist without it being soggy. How many posts have you read on this site in the last week about trees losing branches or dying outright because the soil in the pot is too compact so that it holds water like a sponge and the roots rot?
Peat moss is usually too fine to be used in a pot for the same reason.
Peat moss + pea gravel would not be a good mix - which is why no one uses it. If it is cheap and readily available everywhere and no one uses it - trust me, there is a reason.