In this paper, we present a new real-time depth estimation method using the stereo color camera and the ToF depth sensor. First, we obtain the initial depth information from the ToF depth sensor. Exploiting the initial depth information to narrow the disparity range by performing 3-D warping from the position of the ToF camera to the position of the stereo camera due to accelerating the algorithm. We construct the cost volume by calculating intensity difference and truncated absolute difference of gradients. After narrowing the disparity range, we aggregate the cost volume. Experimental results show that the proposed method can represent the disparity detail and improve the quality in the vulnerable areas of stereo matching.