FAAD:an unsupervised fast and accurate anomaly detection method for a multi-dimensional sequence over data stream

作     者：Bin LI Yi-jie WANG Dong-sheng YANG Yong-mou LI Xing-kong MA 

作者机构：Science and Technology on Parallel and Distributed Processing Laboratory College of ComputerNational University of Defense Technology Block Chain Research Institute of LianLian Pay 

出 版 物：《Frontiers of Information Technology & Electronic Engineering》 (信息与电子工程前沿（英文版）)

年 卷 期：2019年第20卷第3期

页      面：388-404页

核心收录：

学科分类：0810[工学-信息与通信工程] 0808[工学-电气工程] 0809[工学-电子科学与技术（可授工学、理学学位）] 08[工学] 0839[工学-网络空间安全] 0804[工学-仪器科学与技术] 0835[工学-软件工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：Project supported by the National Key R&D Program of China(No.2016YFB1000101) the National Natural Science Foundation of China(Nos.61379052 and 61502513) the Natural Science Foundation for Distinguished Young Scholars of Hunan Province,China(No.14JJ1026) the Specialized Research Fund for the Doctoral Program of Higher Education,China(No.20124307110015) 

主　　题：Data stream Multi-dimensional sequence Anomaly detection Concept drift Feature selection 

摘      要：Recently, sequence anomaly detection has been widely used in many fields. Sequence data in these fields are usually multi-dimensional over the data stream. It is a challenge to design an anomaly detection method for a multi-dimensional sequence over the data stream to satisfy the requirements of accuracy and high speed. It is because:(1) Redundant dimensions in sequence data and large state space lead to a poor ability for sequence modeling;(2) Anomaly detection cannot adapt to the high-speed nature of the data stream, especially when concept drift occurs, and it will reduce the detection rate. On one hand, most existing methods of sequence anomaly detection focus on the single-dimension sequence. On the other hand, some studies concerning multi-dimensional sequence concentrate mainly on the static database rather than the data stream. To improve the performance of anomaly detection for a multi-dimensional sequence over the data stream, we propose a novel unsupervised fast and accurate anomaly detection(FAAD) method which includes three algorithms. First, a method called information calculation and minimum spanning tree cluster is adopted to reduce redundant dimensions. Second, to speed up model construction and ensure the detection rate for the sequence over the data stream, we propose a method calledrandom sampling and subsequence partitioning based on the index probabilistic suffix tree. Last, the method called anomaly buffer based on model dynamic adjustment dramatically reduces the effects of concept drift in the data stream. FAAD is implemented on the streaming platform Storm to detect multi-dimensional log audit data.Compared with the existing anomaly detection methods, FAAD has a good performance in detection rate and speed without being affected by concept drift.