An elementary system with gears and excited by unbalance mass has been constructed for analyzing the interaction between torsion and bending vibration in rotor dynamics. For this system, only the interaction caused primarily by unbalance mass has been investigated. The stability and bifurcation characteristics of the system have been studied by numerical computation based on Hopf bifurcation and Floquet theory. The results show that the interaction between torsion and bending vibrations can affect the stability and bifurcation of the unbalance response, in particular the onset speed of instability. In addition to the above, the interaction also affects the steady-state response. To investigate the influence of unbalance mass, the periodic solution and its stability have been studied near the first bending critical speed of the decoupled system. All the results show that the coupling of torsion and bending vibrations can have a significant influence on the nonlinear dynamics of the whole system.