ebay big data mining

《ebay big data mining》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Large-ScaleClick-streamandtransactionlogmininginpracticeUweMayer,NishParikh,GyanitSinghOctober6-9,2013.BIGDATASCIENCEBestPracticesKeyIdeas•BigDataSets•BigDataProperties•Challengesinworkingwithbigdata•PracticalSolutions•LeveragingHadoop•CaseStudies3Typeso

2、fDataUsedinthisTutorial•Click-streamlogs–PetaByteScale•TransactionalData–TeraByteScale–Morethan½Bitemsforsale4BESTPRACTICESUSEDINPRESENTEDCASESTUDIES•DataCleaning–Takingcareofbaddata–Importanceofdomainknowledge•DataSampling–Reservoirsampling•De-duplicati

3、on•Normalization•HandlingIdiosyncrasiesoflong-taildata•UnderstandingTractabilityofAlgorithms•Efficiencyatscale•Bucketingdataintherightway•BiasRemoval–Systembias–Platformbias–Userbias•Handlingcurseofdimensionality5MoreDataisGood6Butitneedstobeusedcarefull

4、y7QUERYSUGGESTIONSAtScaleoverHadoopQuerySuggestionsonthewebQuerySuggestionsateBay•Enableuserstobroadenornarrowsearches.•Leaduserstorelatedproductsorbrands.•Optimizethebuyingexperience.QuerySuggestionAlgorithms•Variousalgorithmsinliterature–Agglomerativec

5、lustering–QuerySimilarityMeasures(Linguistic,Latent)–QueryFlowGraphs•Ourapproachprimarilybasedonusertrails.Challenges•Large-scaledata–100M+users.–30TB+click-streamlogs.–1B+usersessions.–Severalbillionsearches.•NoisyData–Robots–APICalls–Crawlers,spiders–T

6、oolsandscripts–UserBiasQuerySuggestionsforthequery‘calculator’.Challenges•LongTail•DynamicInventorySuggestionsaremoreusefulfortailqueries.HADOOPTOTHERESCUEHadoopClusterateBay(Oneofseveral)•Nodes–CentOS464Bit–IntelDualHexCoreXeon2.4GHz–72GBRAM–2*12(24TB)H

7、DD–SSDforOS•Network–TOR1Gbps–CoreSwitchesuplink40Gbps•Cluster–532n–1008n–4000+cores–24000vCPUs–5–18PBMobius–ComputationPlatformApplicationClickStreamVisualizerMetricsDashboardResearchProjectsLayerMobiusStudio(Eclipseplugin)MobiusGenericJavaDatasetAPIQuer

8、yLanguageLayerLowlevelDatasetaccessAPIeBayInfra-HadoopClusterstructure&DataSourceLayereBayData(Logs,Tables)Sundaresanetal.ScalableStreamProcessing&MapReduce,HadoopWorld,2009.DataCleaning•Dataiscleanedduringtheprocessingpha