If that's what you mean, I use chicken grit, which is crushed granite, for one of the components of my mix. I've never planted anything in that alone, and don't know anyone who has.
Given that it can't absorb anything, I would think pure ballast would function simply as an anchor. In all other ways, you'd effectively be growing your tree hydroponically (if that's a word.)
When you mentioned "ballast" I thought at first of a function of a soil mix, as in weight to counterbalance the weight of the tree.