2024 · 17 citations · 69 references
ComputingComputer ScienceCybersecurityComputer SecurityMachine LearningNetwork SecurityAnomaly DetectionComputer IntrusionData AnalysisIntrusion AnalysisIntrusion DetectionAnomaly Detection BenchmarkingBenchmark DatasetsDesign PatternsExperimental Bias CorrectionImpact AnalysisMachine Learning For Intrusion DetectionNetwork Intrusion DetectionNetwork-based Intrusion Detection
Synthetically generated benchmark datasets are vitally important for machine learning and network intrusion research. When producing intrusion datasets for research, providers make complex, subtle and sometimes unwary decisions that can affect data utility. Unfortunately, examining network data is difficult, so these decisions are rarely audited. We perform an in-depth manual analysis of seven highly-cited benchmark datasets, discovering six suspect design patterns, which we term ‘data design smells’. We formulate six heuristics to measure the prevalence of these issues. These design choices, if not properly accounted for, can introduce severe experimental bias, which we demonstrate with four concrete examples. We then conduct a systematic impact analysis of the wider literature that relies on these datasets. Our results suggest that bad design smells correlate with poor data diversity, murky labelling and poorly-defined generalisation criteria. Worryingly, we find that improper usage of these datasets can weaken their utility as benchmarks which, in turn, biases downstream intrusion detection research. We conclude with some recommendations for using and creating NIDS datasets to help alleviate these issues.
69
SMOTE: Synthetic Minority Over-sampling Technique
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence Hall et al. · Journal of Artificial Intelligence Research · 2002 · 31.2K citations · Full text
A detailed analysis of the KDD CUP 99 data set
Mahbod Tavallaee, Ebrahim Bagheri, Wei Lu et al. · 2009 · 4.8K citations · Full text
Domain randomization for transferring deep neural networks from simulation to the real world
Josh Tobin, Rachel Fong, Alex Ray et al. · 2017 · 2.8K citations