Vance, just kill the moss and algae with the vinegar. I wouldn't try to remove it. It will sluff off. What I do is get a little bowl of vinegar, and use a cotton ball. And I use tweezers to hold the cotton ball. And then give the moss and algae a good soaking.
My JBP had moss creeping up the trunk pretty badly, but the vinegar got it.
If you look carefully at the second picture, you can see a lot of moss/algae on the trunk. That picture was taken in June.
The first picture is the more recent picture, and sometime between June and now (I don't remember exactly when) I finally got around to killing the moss. It took two applications. The first one got most of it. After the moss turned yellow then brown, I could see the areas I had missed. The second application cleaned that up. I didn't try to pick the moss off. That would, as you said, ruin the bark.