This paper brings out a robust pixel-wise object tracking algorithm which is based on the K-means clustering algorithm. In order to achieve the robust object tracking under complex condition (such as wired objects, cluttered background), a new reliability-based K-means clustering algorithm is applied to remove the noise background pixel (which is neigher similar to the target nor the background samples) from the target object. According to the triangular relationship among an unknown pixle and its two nearest cluster centers (target and background), the normal pixel (target or background one) will be assigned with high reliability value and correctly classified, while noise pixels will be given low reliability value and ignored. A radial sampling method is also brought out for improving both the processing speed and the robustness of this algorithm. According to the proposed algorithm, we have set up a real video-rate object tracking system. Through the extensive experiments, the effectiveness and advantages of this reliability-based K-means tracking algorithm are confirmed.