欢迎来到天天文库
浏览记录
ID:39714791
大小:1.66 MB
页数:73页
时间:2019-07-09
《ebay big data mining》由会员上传分享,免费在线阅读,更多相关内容在学术论文-天天文库。
1、Large-ScaleClick-streamandtransactionlogmininginpracticeUweMayer,NishParikh,GyanitSinghOctober6-9,2013.BIGDATASCIENCEBestPracticesKeyIdeas•BigDataSets•BigDataProperties•Challengesinworkingwithbigdata•PracticalSolutions•LeveragingHadoop•CaseStudies3Typeso
2、fDataUsedinthisTutorial•Click-streamlogs–PetaByteScale•TransactionalData–TeraByteScale–Morethan½Bitemsforsale4BESTPRACTICESUSEDINPRESENTEDCASESTUDIES•DataCleaning–Takingcareofbaddata–Importanceofdomainknowledge•DataSampling–Reservoirsampling•De-duplicati
3、on•Normalization•HandlingIdiosyncrasiesoflong-taildata•UnderstandingTractabilityofAlgorithms•Efficiencyatscale•Bucketingdataintherightway•BiasRemoval–Systembias–Platformbias–Userbias•Handlingcurseofdimensionality5MoreDataisGood6Butitneedstobeusedcarefull
4、y7QUERYSUGGESTIONSAtScaleoverHadoopQuerySuggestionsonthewebQuerySuggestionsateBay•Enableuserstobroadenornarrowsearches.•Leaduserstorelatedproductsorbrands.•Optimizethebuyingexperience.QuerySuggestionAlgorithms•Variousalgorithmsinliterature–Agglomerativec
5、lustering–QuerySimilarityMeasures(Linguistic,Latent)–QueryFlowGraphs•Ourapproachprimarilybasedonusertrails.Challenges•Large-scaledata–100M+users.–30TB+click-streamlogs.–1B+usersessions.–Severalbillionsearches.•NoisyData–Robots–APICalls–Crawlers,spiders–T
6、oolsandscripts–UserBiasQuerySuggestionsforthequery‘calculator’.Challenges•LongTail•DynamicInventorySuggestionsaremoreusefulfortailqueries.HADOOPTOTHERESCUEHadoopClusterateBay(Oneofseveral)•Nodes–CentOS464Bit–IntelDualHexCoreXeon2.4GHz–72GBRAM–2*12(24TB)H
7、DD–SSDforOS•Network–TOR1Gbps–CoreSwitchesuplink40Gbps•Cluster–532n–1008n–4000+cores–24000vCPUs–5–18PBMobius–ComputationPlatformApplicationClickStreamVisualizerMetricsDashboardResearchProjectsLayerMobiusStudio(Eclipseplugin)MobiusGenericJavaDatasetAPIQuer
8、yLanguageLayerLowlevelDatasetaccessAPIeBayInfra-HadoopClusterstructure&DataSourceLayereBayData(Logs,Tables)Sundaresanetal.ScalableStreamProcessing&MapReduce,HadoopWorld,2009.DataCleaning•Dataiscleanedduringtheprocessingpha
此文档下载收益归作者所有