There are plenty of woody weeds so having a trunk does not preclude weedy status. Just check your state's lists of environmental weed species to see how common woody weeds are.
Possible weeds and possible self sowing species will vary in different parts of the world and, often, from state to state. Adding a location to your personal profile will cut down the number of wild guesses on any ID and also help members to give more accurate help for your specific climate.
It's still a bit early to be able to give a good ID on this one.
I was going to suggest cotoneaster as possible self seeder but the leaves are opposite so can't be cotoneaster.
Viburnum leaves are opposite so that's still a possibility but I'd like to see it grow a bit bigger before going further.