Using Memory in the Right Way to Accelerate Big Data Processing
Using Memory in the Right Way to Accelerate Big Data Processing作者机构：Department of Computer Science and Technology Tsinghua University Beijing 100084 China Intel Labs China Beijing 100091 China
出 版 物：《Journal of Computer Science & Technology》 (计算机科学技术学报（英文版）)
年 卷 期：2015年第30卷第1期
页 面：30-41页
核心收录：
学科分类：0810[工学-信息与通信工程] 0808[工学-电气工程] 08[工学] 080401[工学-精密仪器及机械] 0804[工学-仪器科学与技术] 080402[工学-测试计量技术及仪器] 0835[工学-软件工程] 081002[工学-信号与信息处理] 0701[理学-数学] 0811[工学-控制科学与工程] 081201[工学-计算机系统结构] 0812[工学-计算机科学与技术（可授工学、理学学位）]
主 题：big data key/value pair architecture awareness performance measurement
摘 要：Big data processing is becoming a standout part of data center computation. However, latest research has indicated that big data workloads cannot make full use of modern memory systems. We find that the dramatic inefficiency of the big data processing is from the enormous amount of cache misses and stalls of the depended memory accesses. In this paper, we introduce two optimizations to tackle these problems. The first one is the slice-and-merge strategy, which reduces the cache miss rate of the sort procedure. The second optimization is direct-memory-access, which reforms the data structure used in key/value storage. These optimizations are evaluated with both micro-benchmarks and the real-world benchmark HiBench. The results of our micro-benchmarks clearly demonstrate the effectiveness of our optimizations in terms of hardware event counts; and the additional results of HiBench show the 1.21X average speedup on the application-level. Both results illustrate that careful hardware/software co-design will improve the memory efficiency of big data processing. Our work has already been integrated into Intel distribution for Apache Hadoop.
维普期刊数据库