I had quite a few bouganvillea in my landscape in SoCal. I don't see any problems. Because the bark is naturally light-colored, you are just more aware of the micro-flora that exists on most woody plants. If you had an oak tree you would never even notice it
Yes, with bouganvillea their bark on older sections of the plant develops flaky nodules that brush off easily when handled. Generally you want to leave them alone, because they will continue to occur naturally, and they give the tree the character of age. They are quite dry and crispy and crunch under your hand (for lack of a better description).