Hadoop集群测试报告.docx

《Hadoop集群测试报告.docx》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、测试报告一、集群设置1.服务器配置CPU24内存128G带宽1024M磁盘44T磁盘吞吐预计100M/s2.Hadoop服务部署HADOOP-12-151NameNode、Balancer、HiveGateway、SparkGateway、ResourceManager、ZkServerHADOOP-12-152DataNode、SNN、HFS、HiveGateway、WebHCat、Hue、ImpalaDeamon、CMServerMonitor、CMActivityMonitor、CMHostMoni

2、tor、CMEventServer、CMAlertPublisher、OozieServer、SparkHistoryServer、SparkGateway、NodeManager、JobHistoryServer、ZkServerHADOOP-12-153DataNode、HiveGateway、HiveMetastore、HiveServer2、ImpalaCatalog、ImpalaStateStore、ImpalaDeamon、SparkGateway、NodeManager、ZkServerHA

3、DOOP-12-154DataNode、HiveGateway、ImpalaDeamon、SparkGateway、NodeManager、Sqoop2HADOOP-12-155DataNode、HiveGateway、ImpalaDeamon、SparkGateway、NodeManager、ZkServerHADOOP-12-156DataNode、HiveGateway、ImpalaDeamon、SparkGateway、NodeManager、ZkServer3.hadoop参数设置yarn　ya

4、rn.nodemanager.resource.memory-mb108544yarn.scheduler.minimum-allocation-mb4096yarn.scheduler.maximum-allocation-mb32768yarn.scheduler.increment-allocation-mb4096yarn.nodemanager.resource.cpu-vcores24yarn.nodemanager.vmem-pmem-ratio3.1　　mapreduce　mapreduc

5、e.map.memory.mb4096mapreduce.reduce.memory.mb8192mapreduce.map.java.opts3072mapreduce.reduce.java.opts6144mapreduce.task.io.sort.mb1536mapreduce.task.io.sort.factor100mapreduce.reduce.shuffle.parallelcopies15mapreduce.reduce.shuffle.memory.limit.percent0.

6、08mapreduce.reduce.shuffle.input.buffer.percent0.9mapreduce.job.max.split.locations15mapreduce.map.failures.maxpercent5mapreduce.reduce.failures.maxpercent5　　dfs　dfs.namenode.handler.count35dfs.datanode.handler.count3dfs.datanode.max.transfer.threads4096一

7、、基准测试1.HDFS读写的吞吐性能1.1连续10次执行如下写操作，其性能见图示：cd/opt/cloudera/parcels/CDH-5.4.9-1.cdh5.4.9.p0.19/jarshdfsadminhadoopjarhadoop-test-2.6.0-mr1-cdh5.4.9.jarTestDFSIO-write-nrFiles10-fileSize1000-resFile/tmp/TestDFSIO_results.log其具体数据见表格：No.Throughputmb/secAverage

8、IOratemb/secTestexectimesec121.35004922.3262176573.077226.8747127827.9325618761.257330.7341465631.8610172354.897425.5561660628.2618198454.782531.2516602432.1100196862.441630.4715471931.563486154.819729.673414431.390