面向大数据分析的分布式文件系统关键技术.pdf

《面向大数据分析的分布式文件系统关键技术.pdf》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、万方数据计算机研究与发展D()I：10．7544／issnl000一1239．2014．20120863JournalofComputerResearchandDevelopment5l(2)：382—394，2014面向大数据分析的分布式文件系统关键技术周江1’2’3王伟平3孟丹3马灿3古晓艳12．3蒋杰41(中国科学院计算技术研究所计算机应用研究中心北京100190)2(中国科学院大学北京100049)3(中国科学院信息工程研究所北京100093)4(腾讯公司数据平台部广东深圳518057)(zhoujiang@ncic．ac．cn)KeyTechnologyinDis

2、tributedFileSystemTowardsBigDataAnalysisZhouJian91·2～，WangWeipin93，MengDan3，MaCan3，GuXiaoyanl·2～，andJiangJie41(Cbmp猁PrA户户￡站n￡io”Res阳rc^CP”fer，，”盯甜“把o，Co打2户“fi靠gTPf^”ozogy，C^抽P5PAf(【dPmyo，SfiP行fes，BP巧砌g100190)2(U”i乞圯rsif了D，C矗i"P5PAf口4emyo，SfiPnfPs，BP巧i理g100049)3(j”“if“fPo，J对如rm丑￡io九E"gi7jP已

3、rj”g，C^i以已5PAcⅡdPⅢyo厂SfiP门fPs，BP巧i行g100093)4(DnfnPZnf如r埘DP户口r￡研P珂f，T_村fP竹fj行c．，S矗P行z^P，2，G“口行gdu，zg518057)AbstractWiththearrivalofbigdataperiod，dataanalysisandprocessingarebecomingamoreimportanttechnol。gywhichthedatacenterandInternetcompaniesdependon．Massdatastorageisahotspottopicinbigdata

4、analysiswiththeexpansionofinformationandVarietyofdatastructure．TraditionaIdistributedfilesystemsarelackofthenewdemandsinscalab订ity，reliabilityandperformance．Inthispaper，aclusterfilesystemtowardsbigdataanalysisisdesigned，whichisnamedClover．Cloverusesthenamespacemanagementbasedondirectorysha

5、rdingandconsistenthashingtosolvetheproblemofmetadataextension．Itprovidesmetadataconsistencyfordistributedtransactionsthroughamodifiedtwo—phasecommitprotoc01．ⅣIoreover，CloverpresentsahighlyaVailablemechanismbasedonthesharedstoragep001．ItachievesmetadatareliabilitywithhotstandbyandgIobaIstat

6、erecoverymechanism．TheevaIuationresuhsreVeaIthatCloVercouldimproVemetadataperformancelinearlywiththeaveragevaluefrom5．13％to159．32％byaddingonemetadataserVer．Namespacemanagementanddistributedtransactionswouldcausethedegradationofperformanceonmultiplemetadataservers，buttheinfluenceisnegligibl

7、e(1essthan10％)．ComparingwithHDFS，ClovercouldkeepthesimilarthroughputandquicklyrecoverfrommetadataserVerfailures．PracticalapD“cationtestsshowthatCloverissuitableforbuildinghighscalableandhighaVailablestoragesystem．Keywordsbigdata；massdatastorage；distributedfile