结构大数据架构26.pdf

《结构大数据架构26.pdf》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、解構大數據架構大數據系統的伺服器與網路資源規劃“Howtoeatanelephant–onebyteatatime”CPLi李俊邦EnterpriseTechnologistEnterpriseSolutions&Alliances,GreaterChinaDell議程1.不同的伺服器角色1.Manager2.NameNodes3.EdgeNodes4.DataNodes2.HadoopCluster設計3.Etu+Dell4.Futures/Roadmap5.Questions?2ServerRoles-Manager•系統安裝圖形介面/主控台•大多安裝在EdgeNode•常見版本–

2、ClouderaManager–ApacheAmbari3ServerRoles–NameNodes•存放HDFS的metadata•JobManagerforYARNdata-processingframework•Primary–Heartbeatsfromdatanodes–10thheartbeatisablockreportfromwhichitgeneratesmetadata•Standby–Checksineveryhourtomirrormetadata/blockmap–Notahot-spare–requiresmanualfail-over•HighAvaila

3、bility(HA)canbeaddedinsomedistributions–ResultsinadedicatedHAnodethatactsasawitnesstotheNameNodecluster4ServerRoles-EdgeNodes•資料進出Hadoop叢集的主要端口•可擴展•Hadoop叢集裡唯一的多網段節點Corporate NetworkData NetworkData NetworkPowerEdge R730 –Name NodeData NetworkPowerEdge R730 –Standby Name NodeCorporateData Networ

4、kPowerEdge R730 –Edge Node(s)Data NetworkPowerEdge R730 –HA NodeData NetworkPowerEdge R730XD –Data Nodes5ServerRoles-DataNode•HDFS的主要存放處•執行YARN資源管理所指定的資料處理•主要屬性–記憶體›標配64GB›更多服務（Impala/Spark)需要更多記憶體–很多的本地硬碟(JBOD/Non-RAIDmode)›SFF(2.5”)forperformance-basedworkloads›LFF(3.5”)forcapacity-centricwork

5、loads–CPUs–legacyrecommendationof1:1core:spindleratio›SSDs,fasterHDD(10K+),andin-memoryworkloadsmakethislessofanissue›10and12corearethebestpracticedefault6HadoopClusterDesignHadoopClusterDesign–HardwareConsiderations8HadoopClusterDeployment–InstallationBestPractices•Usepre-built,assembled&cabled

6、racksfromvendor•自動佈署工具(ex:OpenCrowbar)•Purchasenodesinstandardsizegroupsforeasycapacitygrowthandordering,notinsinglenodeincrements–Commonincrementsare½orfullrackforeasydeploymentandsizing•Foreachtypeofhardware,purchasesparecomponentstokeeponsiteforeasy,rapidrepair9CoreHadoopUseCases資料處歸檔分析理高硬碟／C

7、PU比高硬碟／CPU比高核心數記憶體使用低記憶體使用中等記憶體使用高DWoffload法規需求ETLoffload市場分析EDH詐欺預防長期歸檔質量分析網路分析ITLog分析10CommonHadoopUseCasetoEcosystemToolMapping11HadoopUseCasetoRatioMapping歸檔資料處理分析1:2:11:4:12:8:1CPU(Cores):Memory(GB):Disk(數量)–DataNode12N