Runtime Code Generation in Cloudera Impala.pdf

《Runtime Code Generation in Cloudera Impala.pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、RuntimeCodeGenerationinClouderaImpalaSkyeWanderman-MilneNongLiskye@cloudera.comnong@cloudera.comAbstractInthispaperwediscusshowruntimecodegenerationcanbeusedinSQLenginestoachievebetterqueryexecutiontimes.Codegenerationallowsquery-specificinformationknownonlyatruntime,suchascolumntypesan

2、dexpressionoperators,tobeusedinperformance-criticalfunctionsasiftheywereavailableatcompiletime,yieldingmoreefficientimplementations.WepresentClouderaImpala,anopen-source,MPPdatabasebuiltforHadoop,whichusescodegenerationtoachieveupto5xspeedupsinquerytimes.1IntroductionClouderaImpalaisano

3、pen-sourceMPPdatabasebuiltfortheHadoopecosystem.HadoophasproventobeaveryeffectivesystemtostoreandprocesslargeamountsofdatausingHDFSandHBaseasthestoragemanagersandMapReduceastheprocessingframework.ImpalaisdesignedtocombinetheflexibilityandscalabilitythatisexpectedfromHadoopwiththeperform

4、anceandSQLsupportofferedbycommercialMPPdatabases.Impalacurrentlyexecutesqueries10-100xfasterthanexistingHadoopsolutionsandcomparablytocommercialMPPdatabases[1],allowingenduserstoruninteractive,exploratoryanalyticsonbigdata.Impalaisbuiltfromgrounduptotakemaximaladvantageofmodernhardware

5、andthelatesttechniquesforefficientqueryexecution.Impalaisdesignedforanalyticworkloads,ratherthanOLTP,meaningit’scommontoruncomplex,long-running,CPU-boundqueries.RuntimecodegenerationusingLLVM[3]isoneofthetech-niquesweuseextensivelytoimproveexecutiontimes.LLVMisacompilerlibraryandcollect

6、ionofrelatedtools.Unliketraditionalcompilersthatareimplementedasstand-aloneapplications,LLVMisdesignedtobemodularandreusable.ItallowsapplicationslikeImpalatoperformJITcompilationwithinarunningprocess,withthefullbenefitsofamodernoptimizerandtheabilitytogeneratemachinecodeforanumberofarch

7、itectures,byexposingseparateAPIsforallstepsofthecompilationprocess.ImpalausesLLVMtogeneratefully-optimizedquery-specificfunctionsatruntime,whichofferbetterper-formancethangeneral-purposeprecompiledfunctions.Thistechniquecanimproveexecutiontimesby5xormoreforrepresentativeworkloads.Inth