Do you have alot of dandelions?
I do sometimes get aphids, but they don't bother my trees any, and we have a nice population of lady bugs every year that seem to keep them in check. Sometimes they bother my rose bushes, but i actally planted those to attract the aphids away from other plants.