Large-Scale Parallel Statistical Forecasting Computations in R.pdf

《Large-Scale Parallel Statistical Forecasting Computations in R.pdf》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

1、Large-ScaleParallelStatisticalForecastingComputationsinRMurrayStokelyFarzanRohaniEricTassoneAbstractWedemonstratetheutilityofmassivelyparallelcomputationalinfrastructureforstatisticalcomputingusingtheMapReduceparadigmforR.Thisframeworkallowsuserstowritecom-putationsinahigh-l

2、evellanguagethatarethenbrokenupanddistributedtoworkertasksinGoogledatacenters.Resultsarecollectedinascalable,distributeddatastoreandreturnedtotheinteractiveusersession.Weapplyourapproachtoaforecastingapplicationthattsavarietyofmodels,prohibitingananalyticaldescriptionofthestat

3、isticaluncertaintyassociatedwiththeoverallforecast.Toovercomethis,wegeneratesimulation-baseduncer-taintybands,whichnecessitatesalargenumberofcomputationallyintensiverealizations.Ourtechniquecuttotalruntimebyafactorof300.Distributingthecomputationacrossmanymachinespermitsanalyst

4、stofocusonstatisticalissueswhileansweringquestionsthatwouldbeintractablewithoutsignicantparallelcomputationalinfrastructure.Wepresentreal-worldperformancecharacteristicsfromourapplicationtoallowpractitionerstobetterunderstandthenatureofmassivelyparallelstatisticalsimulationsin

5、R.KeyWords:StatisticalComputing,R,forecasting,timeseries,parallelism1.IntroductionLarge-scalestatisticalcomputinghasbecomewidespreadatInternetcompaniesinrecentyears,andtherapidgrowthofavailabledatahasincreasedtheimportanceofscalingthetoolsfordataanalysis.Signicantprogresshasbe

6、enmadeindesign-ingdistributedsystemstotakeadvantageofmassiveclustersofsharedmachinesforlong-runningbatchjobs,butthedevelopmentofhigher-levelabstractionsandtoolsforinteractivestatisticalanalysisusingthisinfrastructurehaslagged.Itisparticularlyvitalthatanalystsareabletoiteratequi

7、cklywhenengagedindataexploration,modeltting,andvisualizationontheseverylargedatasets.Supportinginteractiveanalysisofdatasetsthatarefarlargerthanavailablememoryanddiskspaceonasinglemachinerequiresahighdegreeofparallelism.AtGoogle,parallelismisimplementedusingsharedclustersofcom

8、moditymachines[5].Thispaperdescribesastatisticalcomput