See I did try a few months ago leaving him outside for about 4 days but he went aweful then too, dropping leaves and stuff. So I was too worried to leave him out any longer
I see you have MANY threads on this one tree. Please stop making new threads every time you have an issue. By continuing to post in the same thread, people do not have to ask the same questions time and again. If I look back, the "do not grow this indoors" has been discussed at least twice before with regards to this elm. So.. If you put it outside, and you do not see an improvement after 3 days, do not decide it is the wrong advice, but ASK.
Trees do NOT like to be moved around. So moving it outside is pressure to perform. Then, after a few days you move it in again. And again the tree needs to change its ways. As trees need weeks to fully adjust to real changes in placement but respond to stress with a matter of days, you are putting the tree through lots of problems.
Move it outside to a place where it only receives sun for a few hours in the early morning. The rest of the day something like dappled shade somewhere, untill you see lots of foliage again (e.g., all branches have started to grow). Then you can move it to more sun. Mine is in full sun, all day. The trunk of trees can get sunburn and summersun is not a good idea if there are only few leaves. Water enough to keep the soil damp. Besides that, do nothing. Just wait. If the situation gets worse, update. In 2 weeks you should see swelling of buds.