资源描述:
《1#large-scale incremental processing using distributed transactions and notifications》由会员上传分享,免费在线阅读,更多相关内容在教育资源-天天文库。
1、Large-scaleIncrementalProcessingUsingDistributedTransactionsandNotificationsDanielPengandFrankDabekdpeng@google.com,fdabek@google.comGoogle,Inc.Abstractlelismofthecomputation;alldocumentsfinishonepro-cessingstepbeforestartingthenext.Forexample,whenUpdatinganindexo
2、fthewebasdocumentsaretheindexingsystemiswritinginvertedlinkstothecur-crawledrequirescontinuouslytransformingalargerenthighest-PageRankURL,weneednotworryaboutrepositoryofexistingdocumentsasnewdocumentsar-itsPageRankconcurrentlychanging;apreviousMapRe-rive.Thistas
3、kisoneexampleofaclassofdatapro-ducestephasalreadydetermineditsPageRank.cessingtasksthattransformalargerepositoryofdataNow,considerhowtoupdatethatindexafterrecrawl-viasmall,independentmutations.Thesetaskslieinaingsomesmallportionoftheweb.It’snotsufficienttogapbetw
4、eenthecapabilitiesofexistinginfrastructure.runtheMapReducesoverjustthenewpagessince,forDatabasesdonotmeetthestorageorthroughputrequire-example,therearelinksbetweenthenewpagesandthementsofthesetasks:Google’sindexingsystemstoresrestoftheweb.TheMapReducesmustberuna
5、gainovertensofpetabytesofdataandprocessesbillionsofup-theentirerepository,thatis,overboththenewpagesdatesperdayonthousandsofmachines.MapReduceandandtheoldpages.Givenenoughcomputingresources,otherbatch-processingsystemscannotprocesssmallup-MapReduce’sscalabilitym
6、akesthisapproachfeasible,datesindividuallyastheyrelyoncreatinglargebatchesand,infact,Google’swebsearchindexwasproducedforefficiency.inthiswaypriortotheworkdescribedhere.However,WehavebuiltPercolator,asystemforincrementallyreprocessingtheentirewebdiscardstheworkdo
7、neinprocessingupdatestoalargedataset,anddeployeditearlierrunsandmakeslatencyproportionaltothesizeoftocreatetheGooglewebsearchindex.Byreplacingatherepository,ratherthanthesizeofanupdate.batch-basedindexingsystemwithanindexingsystemTheindexingsystemcouldstorethere
8、positoryinabasedonincrementalprocessingusingPercolator,weDBMSandupdateindividualdocumentswhileusingprocessthesamenumberofdocumentsperday,whiletransactionstomaintainin