Journal of Systems Research

Industrial systems are increasingly threatened by cyberattackswith potentially disastrous consequences. To counter suchattacks, industrial intrusion detection systems strive to timelyuncover even the most sophisticated breaches. Due to its criticality for society, this fast-growing field attracts researchersfrom diverse backgrounds, resulting in 130 new detectionapproaches in 2021 alone. This huge momentum facilitatesthe exploration of diverse promising paths but likewise risksfragmenting the research landscape and burying promisingprogress. Consequently, it needs sound and comprehensibleevaluations to mitigate this risk and catalyze efforts into sustainable scientific progress with real-world applicability. Inthis paper, we therefore systematically analyze the evaluationmethodologies of this field to understand the current stateof industrial intrusion detection research. Our analysis of609 publications shows that the rapid growth of this researchfield has positive and negative consequences. While we observe an increased use of public datasets, publications stillonly evaluate 1.3 datasets on average, and frequently usedbenchmarking metrics are ambiguous. At the same time, theadoption of newly developed benchmarking metrics sees littleadvancement. Finally, our systematic analysis enables us toprovide actionable recommendations for all actors involvedand thus bring the entire research field forward.