JiRong_Wen20130708

《JiRong_Wen20130708》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、SensingtheDynamicsoftheWeb–AWebPerspectiveforBigDataJi-RongWen(RenminUniversityofChina)JointworkwithZhichengDou(MSRA)BigWebData•Webdata–Biggest“free”datasource–4Vs:Volume,Velocity,Variety,andVeracity•Grandchallenge:makesenseofWebdata•Doweusethewebdatawell?

2、–Stateoftheart:searchengines–AninformationretrievalmethodologyAnExample•Whowillwinthe2012USpresidentialelection?•“2012presidentialpolls”–Millionsofpages/sites–Changingeverymin/sec–Differentdatasources–Differentdataformats–HighlynoisyTheidea:WebSensor•Theph

3、ysicalworld•Sensor:aconverterthatmeasuresaphysicalquantityandconvertsitintoasignalwhichcanbereadbyanobserverorbyan(todaymostlyelectronic)instrument(Wikipedia)•Thevirtualworld(theWeb)•WebSensor:a“softsensor”layerontopoftheWebWebSensorWebDefinitionofWebSenso

4、r•AWebSensorisaconverterthatcontinuouslydiscover,extractandaggregatestructuredinformationaroundatopicfromtheWeb–Programmable“focusedcrawler”•AWebSensorΩ=(ν,θ,φ,ω)–ν:webvariable–Θ:programtoproducethewebvariable–φ:properties–ω:constraintsofa“functionwell”Web

5、SensorWebVariable•Aroundatopic•Structured,self-defined•Timeseries•Statistical9/16/201212:55PMATimeSeriesforBillGates’numberof9/17/201212:55PMfollowersontwitter.com9/18/201212:55PM9/19/201212:55PM9/20/201212:55PM9/21/201212:55PMWebSensorVersionDateData…3569

6、/21/201212:55PM82935003559/20/201212:55PM82790293549/19/201212:55PM82634753539/18/201212:55PM82501383529/17/201212:55PM82368013519/16/201212:55PM8226399…ATimeSeriesforwebsentimentson“MicrosoftSurface”Query:MicrosoftSurface9/21/20124:55PM9/20/20124:55PM9/19

7、/20124:55PM9/18/20124:55PM9/18/20124:55PMWebSensorProgram•Basicfunctions–Datacollecting–Dataextraction/mining–Dataaggregation•Processinglogic–ProgramminglanguageWebSensorIDE(PSWSEX)WebSensorPlatformClientApplicationsWebSensorProgrammingUserApplicationsWebS

8、ensorManagementDataVisualizationWindowsGUIWizardsOfficeInternetExplorerWebSensorServiceWebSensor…Configuration:Title,Description,Tag,ExecutionFrequency,…WebSensorScheduling&ManagementWebSensorSDKSensorScriptW