SHUG3_Impala_tech_talk

《SHUG3_Impala_tech_talk》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Impala:AModernSQLEngineforHadoopJamesKinley,SolutionsArchitectCloudera,Inc.Agenda•Goals•UserviewofImpala•Impalainternals•ComparingImpalatoothersystemsImpalaOverview:Goals•General-purposeSQLqueryengine:oworksforbothanalyticalandtransactionalworkloadsowillsupportq

2、ueriesthattakefrommillisecondstohours•RunsdirectlywithinHadoop:oreadswidelyusedHadoopfileformatsotalkstowidelyusedHadoopstoragemanagersorunsonsamenodesthatrunHadoopprocesses•Highperformance:oC++insteadofJavaoruntimecodegeneration(LLVM)ocompletelynewexecutionengi

3、nethatdoesn'tbuildonMapReduceUserViewofImpala:Overview•Runsasadistributedserviceincluster:oneImpaladaemononeachnodewithdata•UsersubmitsqueryviaODBC/JDBCtoanyofthedaemons•Queryisdistributedtoallnodeswithrelevantdata•Ifanynodefails,thequeryfails•ImpalausesHive'sme

4、tadatainterface:connectstoHive'smetastore•Supportedfileformats:ouncompressed/lzo-compressedtextfilesosequencefilesandRCFilewithsnappy/gzipcompressionoAvrodatafilesoParquetcolumnarformat(moreonthatlater)UserViewofImpala:SQL•SQLsupport:opatternedafterHive'sversion

5、ofSQLoessentiallySQL-92,minuscorrelatedsubqueriesoINSERTINTO...SELECT...oonlyequi-joins:nonon-equijoins,nocrossproductsoORDERBYrequiresLIMIToLimitedDDLsupport(CREATETABLE)•Functionallimitations:onocustomUDFs,fileformats,orSerDesonobeyondSQL(buckets,samples,trans

6、forms,arrays,structs,maps,xpath,json)oonlyhashjoins;joinedtablehastofitinmemory:beta:ofsinglenode(broadcast)GA:aggregatememoryofallexecutingnodes(partitioned)UserViewofImpala:ApacheHBase•HBasefunctionality:ousesHive'smappingofHBasetableintometastoretableopredi

7、catesonrowkeycolumnsaremappedintostart/stoprowopredicatesonothercolumnsaremappedintoSingleColumnValueFilters•HBasefunctionallimitations:ononested-loopjoinsoalldatastoredastextImpalaArchitecture•Twobinaries:impaladandstatestored•Impaladaemon(impalad)ohandlesclien

8、trequestsandallinternalrequestsrelatedtoqueryexecutionoexportsThriftservicesforthesetworoles•Statestoredaemon(statestored)oprovidesnameserviceandmetadatadistributiono