Competitive karuta is an official Japanese card game and is described as “martial art on the tatami.” Recently, competitive karuta has attracted a great deal of attention among young people. One of characteristic rules of competitive karuta is that there is no referee; therefore players must judge themselves even if the difficult situation arises. Consequently, the players sometimes get into an argument over their judgement, which disrupts the other matches in the room because all the matches proceed in parallel. In this paper, we propose a system that judges the player who took a card first in a competitive karuta match. Our system measures motion data when players take a card by using a wrist-worn accelerometer and gyroscope, and estimates the times when the players touched the card. From the evaluation experiments, 69.2% of rounds were estimated without error and 99.0% of rounds were estimated within 20-ms error. When our system was introduced on the close game, the accuracy of deciding the player taking a card was 75%.