The “olive” may be a variety of Mountain Mahogany - cercocarpus. The leaves and twigs look about right and they are very common in the nursery industry here. There is also a native variety or varieties in Colorado, depending on your specific location.
Big thank you to
@Colorado for his help here! He maybe preppy punk from the wrong end of the state, but his tree knowledge outpaces mine by miles.
After obsessing over it for some time I've come to the conclusion that it is in fact a mountain mahogany, curl leaf species to be exact, cercocarous ledifolius. This species doesn't naturally happen on this side of the mountains often, so I was completely unfamiliar with them. Excited because I'm looking for a solid entry for the Native Tree challenge.