Evaluation on ChatGPT for Chinese Language Understanding

作     者：Linhan Li Huaping Zhang Chunjin Li Haowen You Wenyao Cui 

作者机构：School of Computer ScienceBeijing Institute of TechnologyBeijingChina 

出 版 物：《Data Intelligence》 (数据智能（英文）)

年 卷 期：2023年第5卷第4期

页      面：885-903页

学科分类：0502[文学-外国语言文学] 050201[文学-英语语言文学] 05[文学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

主　　题：Language Model ChatGPT ChatBIT Chinese Language Understanding Artificial intelligence 

摘      要：ChatGPT has attracted extension attention of academia and industry.This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets.Experiments indicate that ChatGPT achieved competitive results in sentiment analysis,summary,and reading comprehension in Chinese,while it is prone to factual errors in closed-book QA.Further,on two more difficult Chinese understanding tasks,that is,idiom fill-in-the-blank and cants understanding,we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning.This paper further analyses the possible risks of using ChatGPT based on the results.Finally,we briefly describe the research and development progress of our ChatBIT.