In this paper, we evaluate the downlink performance of Transparent mode (T-mode) and Non-Transparent mode (NT-mode) in a two-hop cellular system based on IEEE 802.16j. In particular, we evaluate the performance in terms of the system capacity, optimal resource allocation, and outage probability using Monte Carlo simulation with various system parameters such as different Frequency Reuse Factors (FRFs) and the distance between Base Station (BS) and Relay Station (RS). To analyze the Signal to Interference and Noise Ratio (SINR) of the access and relay links, an SINR model is introduced for cellular multihop systems considering intra- and inter-cell interferences. Then, we present a method of optimal resource allocation for the Access Zone (AZ) and Relay Zone (RZ) to maximize the system capacity. Consequently, the simulation results provide an insight into choosing the appropriate RS position and optimal resource allocation. Through numerical examples, it is found that the FRFs of two and three are good choices to achieve the highest capacity with low outage in T- and NT-modes, respectively.