Short-term origin–destination (OD) flow prediction is vital for operations planning, control, and management in urban railway systems. While the entry and exit passenger demand prediction problem has been studied in various studies, the OD passenger flow prediction problem receives much less attention. One key challenge for short-term OD flow prediction is the partial observability of the OD flow information due to trips having not been completed at a certain time interval. This paper develops a novel deep learning architecture for the OD flow prediction in urban railway systems and examines various mechanisms for data representation and for dealing with partial information. The deep learning framework consists of three main components, including multiple LSTM networks with an attention mechanism capturing short/long-term temporal dependencies, a temporally shifted graph matrix for spatiotemporal correlations, and a reconstruction mechanism for partial OD flow observations. The model is validated using smart card data from Hong Kong’s Mass Transit Railway (MTR) system and compared with state-of-the-art prediction models. Experiments are designed to examine the characteristics of the proposed approach and its various components. The results show the superior performance (accuracy and robustness) of the proposed model and also the importance of partial observations of OD flow information in improving prediction performance. In terms of data representation, predicting the deviation of OD flows performs consistently better than predicting OD flows directly.