Yes,
When you repot, never just "slip pot", always clean out the bulk of the old soil. You want as much as is reasonable the entire contents of the pot to be the same, uniform media. No zones of old media of significant size. A little old media is no problem.
Problem is, you already disturbed the root system when you "slip potted" it last Saturday. IF you have the better soil you mentioned in stock on hand, you probably should repot again right now. Otherwise, if you do not have that "better soil" on hand, it is best to leave it alone. Generally myself and most bonsai growers try to avoid repotting trees too often. Once every two to three years is normal for younger trees, and once every 5 to 10 years for older trees, particularly for conifers. Azalea, Gardenia and Citrus, I would only repot once every 2 to 3 years. My gardenia, when I had one, I would only repot every 5 years or so.
(I mentioned citrus, because, gardenia is actually in the citrus family).
You should get your "better soil" in stock. If the tree seems to have recovered while waiting for you to get the new media, just leave it alone until next year. If the tree has continued to decline, you can do an emergency repot. Thoroughly clean out the old, bad media, and pot it up in the new mix.