流式计算技术及应用

《流式计算技术及应用》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

ᢈᵨẆ᪥ṹ10248ὅ!"#$%&'()ᜧ"#▾2016,5ᨴWordᧇ.

1./1
ᢈ01..........................................................21.1
ᢈᭆ1......................................................21.2
᪾5StormḄ5᪀ᑖ᪆..........................................31.3
᪾5SparkStreamingḄ5᪀ᑖ᪆..................................41.3Storm:SparkStreamingḄ5᪀;<.....................................62
ᢈᙠ>▭⚗.AḄᵨ..........................................72.1BC
᪾5SparkStreamingḄᦪE>FᜐᳮᵨḄIJ5᪀...........72.2BCKᩖM#ᜐᳮCᓺ᪾5ḄᦪE>FᜐᳮᵨḄIJ5᪀..................82.3BCᐸP
᪾5ḄᦪE>FᜐᳮᵨḄIJ5᪀......................8QὃᦻT....................................................................91
ᢈ011.1
ᢈᭆ1ᦪEᜐᳮᵨ⌕VᡃXḄIJYZ[\ᜧ]Ḅ^_`aḄᦪEbc
ᦪE஺Wordᧇ.

2ᦪᜐᳮ
A^efg!ᦪEḄᑮᩭFjkᑮᩭlm^nefᐰpᦪEqrsᩭ.tu^_vwx
ᦪEḄqr^yz{|ḄᦪEᑮᩭ}ᙠqA~[wxᦪEḄ>F.᝞TwitterḄStormஹYahooḄS4[6]zᐺḄ
ᦪE5᪀^ᦪEᙠᡫA^ᨵḄ.;Ce◤ᐜqr,YZ~[wxᦪE,>Fឋ⌕V,ᦪEḄg⌕VḄᵨ^
ᐹᨵ¢£¤¥.
A^ᦪE¦¦zᨬ¨©F`ª«Ḅ^tuᦪE¬¦¦®¯^>Fឋ®°஺1.2
᪾5StormḄ5᪀ᑖ᪆ApacheStorm[5],ᙠStormA^ᐜ⌕±©ᵨC>FḄ²³´᪀^ᡃXbµcᡫ(topology)஺¶©ᡫ·¸¹(º»¼^ᵫ»¼AḄ¾¿⁚Á(masternode)[5]ᑖÂṹ^·ᑖú$⁚Á(workernode)᡻x஺©ᡫAᒹÆspoutkboltÇÈÉ⁐^ᐸAspoutÂ〈Ì^ÍηᦪEZtupleᐗÐḄÑ
Â〈ÒÓyboltᑣÍÎÕᣚ¶×ᦪE^ᙠboltAYZØᡂஹÚÛÜÝ,boltÞßnYZàá·ᦪEÂ〈ºᐸPbolt஺ᵫspoutÂâḄtuplez_YãᦪÐ^;Ḽå!Ḅæ;஺Wordᧇ.

3MtmbusZooKeepef•••ZooKeeperAPACHESTORMSparkḄ±çèz·
ᑖéᡂIᑡ¯ëḄᢇᜐᳮ^nzíSparkStreamingḄᐭᦪEᢥ᯿F`ᑖᡂññḄᦪE^òñᦪEóÕᣚᡂSparkAḄRDD,ᯠ}ᙠSparkযp;RDDwxᜐᳮÝ^´öYZ÷ᑮqAøùᜐᳮᡈὅqrᑮ᜜p±ᜓ஺1.3
᪾5SparkStreamingḄ5᪀ᑖ᪆SparkStreamingz᪶þSparkAPIḄ^1K᡽
Storm᪵ᙢᜐᳮᦪᙠᜐᳮᢥ◞⚜ᐜᐸᑗᑖ!""Ḅᢇᜐᳮ%&஺Spark┐)ᢝ+ឋᦪḄ-./!DStream(DiscretizedStream),DStream0ᢇᜐᳮ(micro-batching)ḄRDD(2ឋᑖ34ᦪ5)7RDDᑣ9ᑖ34ᦪ5:;<=9>4?@%ᑖABCDᦪEFGHIᦪḄJᣚ஺Wordᧇ.

4□Stream■ClusterManagerOriginalMesos/YARNDStream-TransformedStreamingStormMNOP-.!QᡫTopology,SpoutTopologyḄᦪSᦪST<UVᡈὅYZ[ᑡ]T<ᦪ^_Ḅ⊤aaᦪBoltbcᦪḄ᦮e⌴>ᔣ]hYZᜐᳮὅBoltT:ᵫj᜜2Boltl?joinmᑮᙠStorm_ᦪḄᓫpqTuplerᐗtuvTupleT:ᵫwFieldsx"᪀ᡂ{x"|ᵫBolt}~Storm_%lhworker,Topology▭ᙠwworker_@?Ḅᙠ5_{SpoutEBolt|ᵫwTasksrButᡂḄ)Ḅ⁚,ᑖ!Nimbus⁚ESupervisor⁚NimbusZookeeperᳮ5ᡠᨵḄSupervisor,Stormwᩭ᦮NimbusஹSupervisorlEᙠ@?ḄTopologyḄ?!஺Wordᧇ.

51.3StormSparkStreamingḄ᪀)<29MN¡|¢SḄᑖ34¡ᐹᨵ¤¥¦ஹT᡽
E§┯ឋ©wª«Ḅᐳᱯ⁐ᙠ°ᐕ²³ᙠ@?ᦪ´ṹBᑖᑮᑡᐹᨵ§┯:¶ḄMN·?@?஺¸᜜«|¹ºᓫḄAPIᩭºᓄ¼½¾Ḅ¿ᩖÁ஺ᡠ<ÂÃÄStorm[2]ḄMN✌ᐜᵨᡝÈᵨStormḄAPIÉᑏTopologyMNOPᯠÌÈᵨStormḄClientTopologyÍÎNimbus,ᯠÌNimbusTask%&ᢣÐÎSupervisor,SupervisorᙠmᑮTaskÌ!TaskÑGWorkerᵫWorker᡻?ᐹÓḄTask,ᨬÌÕᡂMNB஺Wordᧇ.

624MNᢈ×ᙠ▭⚗Ù_ḄÚᵨ2.1Û4MN᪾SparkStreamingḄᦪᜐᳮÚᵨḄ¡᪀SPARKᜧᦪ᪾Þᔠ¹4MNᢈ×ஹàMNᢈ×ᑭᵨSPARKstreamingâãᜧäåæçè⚣ᑖ᪆¡ᑭᵨ4MNEàMNḄᢈ×ÃᔠOPENCVè⚣ᑖ᪆Në[2],ìᑮ)wíæçè⚣l?ᑖ᪆஺ᦑ/e■⌾òᦔôõöö÷.---------------------SporiSrramin<9?@A=/ᦪBCᦔᙢFGHᨵᵨḄKL9ᡂ⚣ᢈN⌕OPḄQ⚪஺STUᑭᵨSparkStraming᪾X᪀Zᑖ[\⚣]ᜐᳮḄ^-_`?ᦪḄஹᜐᳮ஺Wordᧇ.

72.2bcdᩖfgᜐᳮCEP᪾XḄᦪ-.ᜐᳮhᵨḄTUX᪀2.3bcᐸj]\kl᪾XḄᦪ-.ᜐᳮhᵨḄTUX᪀᪷nkl^#⚣ᑖ᪆Ḅ-.ឋ◤rsktubcStorm[2]]kl᪾XḄ-.⚣ᑖ᪆TU஺'ᵨᔠwOṹᓫᐗ⚣ᑖ᪆ᓫᐗḄ{|}~ὑ(wᑭᵨGᑴl|᡻,஺⁚ឋwᑭᵨᣚᶍ᝱⁚workerNodeᦋ+StormḄᘤ¡¢LḄᜐᳮ£.஺-¤¥¦⊤¨ᙠª,«Ὴ®l|ḄStorm(B¯ᐭ±sᜓ-_100ms8Ḅ¢Lᜐᳮ£.¡c1sḄ᦮´£.µn¶᛻#±¸¹Fº-.»¼Ḅ⚣ᑖ᪆½¾஺Wordᧇ.

8✌ᐜÁÂ⌕ὃ⇋ᑮsᜓṹ᳛ḄÇÈÁÉÊËÌ(ᙳᒴᙢÏ▅⚣]ᦋᗐ(ឋ»¼ឋ&᦮´.£ḄÒÓÔ¨ÕÖ×᜜&sᜓStorm(+,◞ÚwÛ{ÜhᵨÝ&TU+,ᑴ஺Þὃᦻà[1]FayChang,JeffreyDean,SanjayGhemawat,WilsonC.Hsieh,DeborahA.Wallach.Bigtable:ADistributedStorageSystemforStructuredData[D].OSDI,2006.[2]JeffreyDeanandSanjayGhemawat.MapReduce:SimplifiedDataProcessingonWordᧇ.

9LargeClusters[D].OSDI,2004.[3]SanjayGhemawat,HowardGobioff,andShun-TakLeung.TheGoogleFileSystem[D].BoltonLanding,NewYork,USA:ACM,2003.[4]ᵭ=â;bcäḄ⚣ål|Ẇç[D];ᜧè;2014é[5]⊱ëì;bcíîfg®Ḅï⚣᤺⌕[D];ᜧè;2015é[6]ñ᧗;bc]\klḄᜧóô⚣ᑖ᪆ᐵöᢈNẆç[D];÷=ᜧè;2015éWordᧇ.