培乐园-海量数据之架构和处理6

《培乐园-海量数据之架构和处理6》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、5.Technology5.Technology•Hardware•Datastructure•Algorithm•Distribution&Cloud5.Technology:remember5.Technology:computingPlatformCommunicationSchemeDatasizePeer-to-PeerTCP/IPPetabytesVirtualClustersMapReduce/MPIPeta,TeraHPCClustersMPI/MapReduceTerabytesMulticoreMultithreadingGigab

2、ytesGPUCUDAGigabytesFPGAHDLGigabytes5.Technology:storage•Change:–TapeisDead–DiskisTape–FlashisDisk–RAMLocalityisKing•Distributed:–DistributedDB–DistributedMemorySystem–DFS5.Technology:network•1000MbEthernet•1GbEthernet•10GbEthernetasthebackbonenetwork•NetworkSwitch?5.Technology:

3、more•HadoopStack•NoSQL&NewSQL•MPI,Spark,Mesos•HadoopDB,Storm,S4,Kafka,RonHadoop•FLASHSSD,Memory,GPU,参考•GFS/MapReduce/Bigtable•Hadoop/Hive•Google,Facebook,Amazon,…..•Datawarehouse,Machinelearning,….•…………•很多示意图/架构图来源于学术/交流/互联网，未指明，抱歉•Thanks☺问题•Howtoprocess:–100BillionWebpages•Extr

4、actingFeatures•PageRank–600millionusers•SocialNetwork,multicast•Recommendation–1trillionlogs•CTRprediction•SPAM/Frauddetection•Learningtorank–1millionmachines•Management•Automation北京培乐园科技咨询有限公司:http://www.peileyuan.com