The online target specific feature based state estimation method has proved its applicability in video-based multiple objects tracking. This paper proposes a multi-modal tracking approach by coupling a distance based tracker with an appearance based tracking method. This method is applicable for trajectory formation of multiple objects with complex random motion structure. Proximity measurement scheme is applied to introduce structural context information in tracking-by-detection framework. The multiple-instance framework is formulated to incorporate spatial-temporal information of a target, to select significant features and to establish the statistical correlation between a prior model of the target and its recent observation. The proposed approach improves tracking performance significantly by reducing the number of fragmented trajectories and ID switches. The quantitative, as well as qualitative performance of the proposed method, is evaluated on six benchmark video sequences with the challenging environment like random movement between objects and partial occlusion. The proposed approach performs better than other state-of-the-art methods used for multiple objects tracking.