There are several different sub species/species of olive.
African olive tends to have smaller leaves and often does not have the silver back of leaves. Down here all olives are pest species but African olive spreads more then European olive so it is possible your seedling is African olive.