面向非对称环境的大数据系统优化机制

《面向非对称环境的大数据系统优化机制》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、万方数据江苏大学硕士学位论文ABSTRACTInthecurrentbigdatasystem，almostallofnodesconfigurethesamehardwareorhaveagoodLoadbalance．Throughasinglereplicationandsplitstrategy,wouldn’tconsiderthedifferencesbetweenstorageandcomputeresourcesofeachnode，andalsohavealackofstrategythatnodewitchisrunningcouldadapttodynamicchan

2、gesofinthebigdatasystem．Weneedtoupgradedynamictothisgeneralcaseconstantlyinthebigdatasystem，meanwhileresearchanddesignofthebigdatasystemoptimizationmechanismfortheasymmetricenvironment，throughadjustthereplicationandslipdynamicllytoadaptresourcesbetweennodes，andloaddynamicchange．Thepaperanalyzessom

3、estrategieslikedataaccessffequency，storagemode，dataprocessingmode，resourceofnode，datamigrationpoliciesinthebigdatasystem，fromthedifferencesbetweenstorageandcomputeresourcesofeachnode，designedthestructureofbigdatasystemforasymmetricenvironment．wedesigntheoptimizationstrategyofdataaccessbalanceforas

4、ymmetricenvironment，throughtheamountofdataaccessandjudgetheheatofthedata；Increasingthereplicationofthedatastateidentification，byconvertingdatabetween”active”and”sleep”statetodynamicallyadjustthenumberofreplications．reducethereplicationsofthecolddataandimprovethereplicationsofthehotdata．Dynamically

5、assigningtheresourcesofstorageandprocessorofcolddatatoimproveefficiencythetaskandimprovetheutilizationofstorageresourcesforbigdatasystems．Anddesignaccessbalanceoptimizationstrategy,bydynamicadjustmentalgorithmoftheheartbeatintervalandtaskmigrationalgorithmtostatisticandmeasurementprocessorandstora

6、geresourceofthenodeaftervirtualization；whenanodehaslotsoftasks，throughmakingareasonablechoiceandthenodewhichsavethe”sleep”replication，andtomakingataskstomigrationandrunsuccessfully；toadaptthedifferencesbetweenresourcesofcomputeandstorageandimprovetheefficiencyofIII万方数据面向非对称环境的大数据系统优化机制thetask．Mean

7、whilebaseontheHadooptoimplementtheprototypesystemforthedataaccessbalanceoptimizationstrategyforasymmetricenvironment，andusetheHadoopbenchmarktoolsandEmailsecurityteststandarddatasetstotestandanalyzetheprototypesy