Multi-Query Optimization for Parallel Data

《Multi-Query Optimization for Parallel Data》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Multi-QueryOptimizationforParallelData owSystemsPeterAlvaro,NeilConway,andAndrewKrioukovMay14,2009Abstractandeachworkernodeisassignedoneormoretasks.Whenaworkercompletesatask,HadoopassignstheExistingparalleldata owsystemsarestrictlyreac-workeranothertasktoexecute,usingsomesimpletiveinth

2、eiroptimizations.Atbest,suchapproachesheuristicsthattrytoplacecomputationsclose"toapproximatetheoptimalstrategy,missingopportuni-theirinputdata.However,noglobalanalysisisper-tiestooptimizeacrossmultiplequeriesandresched-formedtoattempttocolocatedataandcomputation,ulequeriestoimprovelo

3、cality.Weproposethreetopredicttheoptimalscheduleforjobsandtasks,ortechniquesthatimprovequeryexecutionperformancetoshareworkbetweensimilarjobs.byutilizinghigh-levelknowledgeoftheworkload.WhenMap-Reduceisusedastheexecutionplat-Thersttechniquepredictivelyreplicatesdatatoim-formforadeclar

4、ativequerylanguagesuchasHiveproveaggregatereadbandwidthandincreaselocal-orPig,therearemoreopportunitiesforoptimiza-ity.Thesecondtechniquereorderssimilarqueriestion[13].However,currentsystemsimplementonlytoimprovecacheperformance.Thethirdtechniquesimple,conservativeoptimizations:forexam

5、ple,Pigschedulesmultiplequeriesinparalleltoimprovere-canpushdistributiveoralgebraicaggregateevalua-sourceutilization.Weevaluatethesetechniquesus-tionbeneathjoins[14],andHiveappliesprojectioningApacheHiveonAmazonEC2andshowperfor-andselectionpushdown[19].Inparticular,noneofmanceimproveme

6、ntsof5%to50%.thesesystemscurrentlyperformsmultiple-queryop-timization.1IntroductionTheprimitivestateofqueryoptimizationforthesesystemsisonlypartlyduetotheirrelativeimma-Paralleldata owsystemssuchasMap-Reduce[6]andturity.Thereisalsoaphilosophicaldierencebe-Hadoop[2]haverecentlyexperien

7、cedasurgeinpop-tweenthesesystemsandparallelrelationaldatabases.ularity.ThesesystemsareincreasinglyusedfordataTheoptimizationsappliedbytraditionaldatabaseswarehousingandanalytics,eitherdirectlyorthroughrequireanaccuratemodelthatpredictstheruntimetheuseofahigh-levelquerylanguagethatisc