Massive data clustering by multi-scale psychological observations

作     者：Shusen Yang Liwen Zhang Chen Xu Hanqiao Yu Jianqing Fan Zongben Xu Shusen Yang;Liwen Zhang;Chen Xu;Hanqiao Yu;Jianqing Fan;Zongben Xu

作者机构：National Engineering Laboratory of Big Data Analytics Xi'an Jiaotong University Department of Mathematics and Statistics University of Ottawa Center for Statistics and Machine Learning Princeton University Industrial Artificial Intelligent CenterPazhou Laboratory 

出 版 物：《National Science Review》 (国家科学评论(英文版))

年 卷 期：2022年第9卷第2期

页      面：43-51页

核心收录：

学科分类：0710[理学-生物学] 12[管理学] 0711[理学-系统科学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 08[工学] 081104[工学-模式识别与智能系统] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：supported by the National Key R&D Program of China (2020YFA0713900) the National Natural Science Foundation of China (U1811461 and 11690011) support of the Natural Sciences and Engineering Research Council of Canada (RGPIN-2016-05024) support of the National Science Foundation (DMS-1662139 and DMS-1712591) 

主　　题：massive data clustering psychological observation Weber–Fechner law cognitive interpretability computational scalability 

摘      要：Clustering is the discovery of latent group structure in data and is a fundamental problem in artificial intelligence,and a vital procedure in data-driven scientific research over all disciplines.Yet,existing methods have various limitations,especially we ak cognitive interpretability and poor computational scalability,when it comes to clustering massive datasets that are increasingly available in all domains.Here,by simulating the multi-scale cognitive observation process of humans,we design a scalable algorithm to detect clusters hierarchically hidden in massive datasets.The observation scale changes,following the Weber-Fechner law to capture the gradually emerging meaningful grouping structure.We validated our approach in real datasets with up to a billion records and 2000 dimensions,including taxi trajectories,single-cell gene expressions,face images,computer logs and audios.Our approach outperformed popular methods in usability,efficiency,effectiveness and robustness across different domains.