I'm newer to bonsai than to trees in the ground by allot, but I would guess you're probably on the right track. For a tree to fruit you need others to pollinate it (depending on the variety- some can self pollinate, others require different varieties of similar species to pollinate), and you need pollinators, i.e. bees, flies, butterflies, hummingbirds, etc., as the species is evolved to. Weather can play a roll as late spring frosts or hot dry periods will kill flowers before they can produce.
Complications that can arise from being a bonsai might include...
Size and age: for some species a certain trunk diameter is necessary to move enough nutrients up to the branches to produce fruit.
Root base: can't get those nutrients without enough roots.
Nutrients: an all aggregate/inorganic soil may not provide the tree enough nutrients to fruit.
Working/Stress levels: Part of bonsai is keeping the tree stunted by keeping it slightly stressed, as well as continuous trimming to maintain appearance, both of which can reduce a tree's ability to produce. Also, what
@Tums said.
So, as rule of thumb, that perfectly manicured tree in the aggregate substrate in the tiny pot growing tiny leaves is not likely to fruit. Hence fruiting trees in shows are such an exciting deal, and often defoliated. Maybe just let it go for a year and see where it flowers and how much.
