Olives are Mediterranean trees. They like to be watered deeply, but infrequently. My olive trees in my landscape get no supplemental watering - so they go 6 months+ with no water. They almost like the soil to dry out completely between waterings. I can see algae on your soil which means that your soil - even at the surface - is not drying out. You need to put your tree in full sun and avoid watering it until the soil is dry. Do not keep it in a pan of evaporative water. They are almost desert plants.
Additionally, because of our poor water in California, you have to watch salt buildup in your soil. The best way to avoid this is to not water frequently and shallowly, but infrequently and deeply - so the excess salts are flushed from the pot.