以支撐搜索服務(wù)為主的網(wǎng)絡(luò)操作系統(tǒng)研制

秦偉俊

摘 要：隨著信息技術(shù)和互聯(lián)網(wǎng)的發(fā)展，搜索引擎作為支撐互聯(lián)網(wǎng)服務(wù)的一個重要的基礎(chǔ)設(shè)施，其顯著的特點就是超大規(guī)模，主要體現(xiàn)在數(shù)據(jù)存儲與數(shù)據(jù)處理兩個方面。該課題以網(wǎng)絡(luò)操作系統(tǒng)新型體系結(jié)構(gòu)為目標(biāo)，突破大規(guī)模分布式數(shù)據(jù)共享與管理、高效資源調(diào)度、虛擬資源管理、大規(guī)模分布式系統(tǒng)運維等關(guān)鍵技術(shù)，提出了高性能存儲引擎與存儲系統(tǒng)設(shè)計、MapReduce環(huán)境中基于虛擬分區(qū)的負(fù)載平衡策略、異構(gòu)Hadoop集群環(huán)境中性能驅(qū)動的數(shù)據(jù)平衡方法、云計算資源計量方法、分布式計算引擎設(shè)計、數(shù)據(jù)密集型計算系統(tǒng)設(shè)計、基于歷史信息比例因子動態(tài)調(diào)整的作業(yè)完成時間預(yù)測算法、空閑節(jié)點評估調(diào)度算法、MapReduce作業(yè)輸入負(fù)載均衡策略、應(yīng)用遠程虛擬運營與應(yīng)用開放平臺設(shè)計、超大規(guī)模數(shù)據(jù)中心多層次網(wǎng)絡(luò)互聯(lián)設(shè)計、大規(guī)模機群系統(tǒng)監(jiān)控管理技術(shù)、大規(guī)模機群系統(tǒng)軟件遠程部署技術(shù)、低功耗高密度服務(wù)器和機柜設(shè)計技術(shù)，以及機器與機群管理系統(tǒng)等多項技術(shù)成果。利用百度強大的互聯(lián)網(wǎng)服務(wù)作為支撐，基于超大規(guī)模的云計算數(shù)據(jù)中心，設(shè)計并實現(xiàn)具有自主知識產(chǎn)權(quán)的新一代網(wǎng)絡(luò)操作系統(tǒng)。該系統(tǒng)將以支持搜索業(yè)務(wù)為核心，以數(shù)據(jù)中心的集群平臺為服務(wù)支撐，為搜索引擎服務(wù)提供超大規(guī)模數(shù)據(jù)存儲和數(shù)據(jù)處理能力，同時為基于搜索的開放應(yīng)用服務(wù)提供運營支撐。

關(guān)鍵詞：網(wǎng)絡(luò)操作系統(tǒng) 搜索服務(wù) 數(shù)據(jù)存儲引擎 資源調(diào)度 系統(tǒng)監(jiān)控 Hadoop MapReduce

Abstract： With the development of information technology and Internet， search engine plays an important role of internet service infrastructure with the feature of large scale dimensions， which focus on two aspects of data storage and data processing. With the aim of new architecture of network operating system， this project breaks through key technologies in large-scale distributed data sharing and management， efficient resource scheduling， virtual resource management， large-scale distributed system maintenance， and proposes several technical achievements including high performance storage engine and system design， load balance strategy based on virtual partition in MapReduce environment， performance-driven data balancing method in heterogeneous Hadoop cluster environment， resource measurement method in cloud computing， distributed computing engine design， system design of data intensive computing， job finishing time schedule algorithm based on historical information ratio factor dynamic adjustment， idle node evaluation schedule algorithm， MapReduce job injection load balance strategy， application remote virtual operation and platform design， multi-level network design for large-scale data center， large-scale cluster system monitoring and management， large-scale cluster system software remote deployment， low-energy， high-density server cabinet design， and clustering management system. With the support of internet search service from Baidu Inc.， the project design and implement new generation of network operating system with independent intellectual property rights based on large-scale cloud computing data center， which aims to provide technical supports for search services and clustering platform in data center， large-scale data storage and processing capacity for search engine and operational supports for upper open application services.

Key Words： Network Operating System； Search Service； Data Storage Engine； Resource Scheduling； System Monitoring； Hadoop； MapReduce

閱讀全文鏈接（需實名注冊）：http：//www.nstrs.cn/xiangxiBG.aspx？id=65523&flag=1