资源描述：
《Learning from narrated instruction videos》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。
1、LearningfromnarratedinstructionvideosJean-BaptisteAlayracyPiotrBojanowskiNishantAgrawalJosefSivicENSInriaIIITHyderabadInriaIvanLaptevSimonLacoste-JulienyInriaInriaAbstractWeaddresstheproblemofautomaticallylearningthemainstepstocompleteacertaintask,suchaschangingacartire,f
2、romasetofnarratedinstructionvideos.Thecontributionsofthispaperarethree-fold.First,wedevelopajointmodelforvideoandnaturallanguagenarrationthattakesadvantageofthecomplementarynatureofthetwosignals.Second,wecollectanannotateddatasetof57Internetin-structionvideoscontainingmorethan
3、350,000framesfortwotasks(changingcartireandCardioPulmonaryResuscitation).Third,weexperimentallydemonstratethattheproposedmodelautomaticallydiscovers,inanunsupervisedmanner,themainstepstoachieveeachtaskandlocatethemwithintheinputvideos.There-sultsfurthershowthattheproposedmodel
4、outperformssingle-modalitybaselines,demonstratingthebenefitsofjointmodelingvideoandtext.1IntroductionMillionsofpeoplewatchnarratedinstructionvideos1tolearnnewtaskssuchasassemblingIKEAfurnitureorchangingaflatcartire.Whatifmachinescouldlearnthesetasksfrommyriadsofin-structionvideo
5、sontheInternet?Suchautomaticcognitiveabilitywouldenableconstructingvirtualassistantsandsmartrobotsthatwouldhelppeopletoachievenewtasksinunfamiliarsituations.Inthiswork,weconsiderinstructionvideosanddevelopamethodthatlearnsasequenceofsteps,aswellastheirtextualandvisualrepresent
6、ations,requiredtoachieveacertaintask.Forexample,givenasetofnarratedinstructionvideosdemonstratinghowtochangeacartire,ourmethodautomaticallydiscoversconsecutivestepsforthistasksuchasloosenthenutsofthewheel,jackupthecar,removethesparetireandsoonasillustratedinFigure1.Inaddition,
7、themethodlearnsthevisualandlinguisticvariabilityofthesestepsfromnaturalvideos.arXiv:1506.09215v2[cs.CV]2Jul2015Discoveringkeystepsfrominstructionvideosisahighlychallengingtask.First,linguisticexpres-sionsforthesamestepcanhavehighvariabilityacrossvideos,forexample:“...Loosenupt
8、hewheelnutjustalittlebeforeyoustartjackingthecar...”and“...St
