Yoga means union, and we see this as the union between mind and body.
Yoga isn't just movement, it also consists of breathing, meditation, concentration and much more. The philosopher Patanjali is said to be the philosophical founder of yoga and according to him there are eight limbs to yoga, which must be fulfilled to achieve the perfect union.
When doing physical yoga you are working on the asana part.
Read more about the different yoga styles here: