7.0 Understanding the Parallelism of a Storm Topology

《7.0 Understanding the Parallelism of a Storm Topology》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、UnderstandingtheParallelismofaStormTopologyWhatmakesarunningtopology:workerprocesses,executorsandtasksStormdistinguishesbetweenthefollowingthreemainentitiesthatareusedtoactuallyrunatopologyinaStormcluster:1.Workerprocesses2.Executors(threads)3.TasksHereisa

2、simpleillustrationoftheirrelationships:A workerprocess executesasubsetofatopology.Aworkerprocessbelongstoaspecifictopologyandmayrunoneormoreexecutorsforoneormorecomponents(spoutsorbolts)ofthistopology.Arunningtopologyconsistsofmanysuchprocessesrunningonman

3、ymachineswithinaStormcluster.An executor isathreadthatisspawnedbyaworkerprocess.Itmayrunoneormoretasksforthesamecomponent(spoutorbolt).A task performstheactualdataprocessing—eachspoutorboltthatyouimplementinyourcodeexecutesasmanytasksacrossthecluster.Thenu

4、mberoftasksforacomponentisalwaysthesamethroughoutthelifetimeofatopology,butthenumberofexecutors(threads)foracomponentcanchangeovertime.Thismeansthatthefollowingconditionholdstrue: #threads≤#tasks.Bydefault,thenumberoftasksissettobethesameasthenumberofexecu

5、tors,i.e.Stormwillrunonetaskperthread.ConfiguringtheparallelismofatopologyNotethatinStorm’sterminology"parallelism"isspecificallyusedtodescribetheso-called parallelismhint,whichmeanstheinitialnumberofexecutor(threads)ofacomponent.Inthisdocumentthoughweuset

6、heterm"parallelism"inamoregeneralsensetodescribehowyoucanconfigurenotonlythenumberofexecutorsbutalsothenumberofworkerprocessesandthenumberoftasksofaStormtopology.Wewillspecificallycalloutwhen"parallelism"isusedinthenormal,narrowdefinitionofStorm.Thefollowi

7、ngsectionsgiveanoverviewofthevariousconfigurationoptionsandhowtosettheminyourcode.Thereismorethanonewayofsettingtheseoptionsthough,andthetablelistsonlysomeofthem.Stormcurrentlyhasthefollowing orderofprecedenceforconfigurationsettings: defaults.yaml 

8、yaml