This paper presents a human-robot interface, which incorporates Kalman filters (KFs) and adaptive multispace transformation (AMT), to track movements of the human hand and control the robot manipulator. This system employs one inertial measurement unit and a 3-D camera (Kinect) to determine the orientation and translation of the human hand, and uses KFs to estimate these. Although KFs can estimate the translation, the translation error increases in a short period of time when the sensor fails to sense hand movement, including handshaking. Therefore, a method to correct the translation error is required. In this paper, the change rate of the human hand is used to determine the posture of the robot. An overdamping strategy is also employed to eliminate the effect of movement sensing failure. Given that a human operator has difficulty operating with high precision due to perceptive and motor limitations, an AMT method is proposed to assist the operator in improving the accuracy and reliability of determining the movement of the robot. The human-manipulator interface is then experimentally tested in a laboratory environment. The results indicate that the system based on the human-manipulator interface can successfully control the robot manipulator.