System Call-Based Behavioral Detection of Malware on Routers

Alex Duff, Ni An, Gaurav Naik, Steven Weber, Spiros Mancoridis
Drexel Issac L. Auerbach Cybersecurity Institute, Pennsylvania, United States

Keywords: IoT botnets, Malware detection, PCA, machine learning

The Internet of Things (IoT) introduced new targets and attack vectors for malicious actors who infect insecure devices with malware in order to form large botnets that can launch distributed denial of service (DDoS) attacks. These botnets comprise various infected devices such as Internet-connected cameras and home routers. Our research focuses on the unsolved problem of creating robust malware detection to secure home routers. Other research shows that, on traditional computer systems, system call-based behavioral analysis techniques are effective at detecting the presence of malware and other anomalies, but routers are specialized systems with a much narrower scope of functions and responsibilities. This research demonstrates two different approaches to behavioral malware detection on home endpoint routers through the observation of kernel-level system calls on home routers, one-class support vector machines and principal component analysis, and compares their effectiveness.