multidimensional hyperpresentations in a distributed multimedia system

《multidimensional hyperpresentations in a distributed multimedia system》由会员上传分享，免费在线阅读，更多相关内容在教育资源-天天文库。

ITAD-730-CP-93-1893rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993MULTIDIMENSIONALHYPERPRESENTATIONSINADISTRIBUTEDMULTIMEDIASYSTEMEarlRennison,KateFinnInformation,Telecommunications,andAutomationDivisionSRIInternationalMenloPark,California,U.S.A.{rennison,finn}@erg.sri.comDooHyunKim,Young-HwanLimElectronicsandTelecommunicationsResearchInstitute,Daejeon,KoreaAbstractWehavedefinedamodelformultidimensionalhyperpresentations,inwhichtime-varyingmultimediapresentationsaredynamicallylinkedtogether.Inourmodelahyperlink’scontextcanchangeovertime.Thus,atanytimeduringacontinuouspresentation,ausercaninteractivelyaccesscontext-specificinformation.Howthehyperlinksaretraverseddependsuponuserinteractionand/orcontext-sensitivetriggers.Thisgeneralizedmodelcanbeappliedtosolvemanyissuesassociatedwithmultimediainformationrepresentationandaccess.1.INTRODUCTIONVideocanbeaverypowerfultoolforcommunicatinginformationandideas.Unfortunately,videoisaserialmedium,sothatsearchingforinformationentailscumbersomenavigationamongnonsequentialsegments.Digitalinteractivevideohasmadethenavigationprocesseasier,buthasnotsolvedtheproblemoffindinginformationofinterest,suchasaparticularvideosegment.Incontrast,hypermediasystemsprovideeffectivetechniquesforpreciselynavigatingthroughandaccessinginformationspaces,thusenablinguserstoquicklyfindinformationofinterestviaamorenaturalinterface[1;2].Attemptstomergetherichnessofvideowiththenavigability/accessibilityofhypermedia,however,havenotprovidedadequatesupportformultidimensionaldata.Thetermmultidimensionalisusedinthispapertoincludethefullspectrumofmedia:multipleaudioandvideotracks(layering);multiplechannels(suchasleftandrightaudio,video,andmouseinputs);3-Dgraphicsand3-Daudio;andtime-varyinginformation.Thetermhyperpresentationreferstotheconceptoflinkingtime-basedmultimediapresentationstogether,basedonthetime-varyingcontextofthepresentation.Forexample,considerascenariowheretheresultsofresearchprojectsarestoredinadatabaseasmultidimensionalhyperpresentations.Apersonbrowsingthroughsuchadatabasecouldinitiallybepresentedwithanoverviewofaselectedresearchprojects.Theoverviewcouldbeaconcisevideocontainingadescriptionofeachofthekeyresultsofeachproject.If,whileviewingtheinformation,thepersonfindsatopicofinterest,heorshecouldpressa“TellMeMore”buttonthatwouldalterthevideotoshowmoredetailedinformationonthespecifictopicbeingdiscussedatthetimethebuttonwaspressed.Orfurther,thepersoncoulduseamousetoclickonaspecificitemthatheorsheseesonthescreen;thevideowouldfreeze,andanotherwindowwouldpopup,showingavideothatrelatestotheitemselected.ThisexampleofhyperpresentationisillustratedinFigure1.1 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993Userselects“TellMeMore”UserclicksonvisualitemOverviewSubjectBUserentersreturnSubjectAUserentersreturnOverviewSubjectAOverviewSubjectBOverviewTimeFigure1.ExampleofHyperPresentationAlthoughthistypeoffunctionalityandinteractionmayofferanintuitivemethodofinformationaccess,currentsupportservicesforthistypeofinteractionarenotsufficient.Toadequatelyaddressthisproblem,aneedexistsfor(1)amodeltoexpressthemultidimensionalrelationshipsbetweenrelatedmedia,aswellasmodelinguserinteractions,and(2)mechanismsforexecutingthespecifiedrelationships,suchashyperlinks,inrealtime.Wehavedefinedamodelformultidimensionalhyperpresentations,inwhichmultimediapresentationsarelogicallylinkedsoastooccureithersequentiallyorsimultaneously.Thismodelincludesthetime-basedlinkingofmultimediapresentations,basedonthecontextsofthepresentations,wherebyeachpresentation’scontextisdefinedbymarkersandsegmentsdelineatedovertime.Thegeneralizedmodelwehavedefinedcanbeappliedtosolvemanyoftheissuesassociatedwithmultimediainformationrepresentationandaccess.2.MuXDISTRIBUTEDMULTIMEDIASYSTEMInthissection,wewillpresentaverybriefandhigh-leveloverviewofthemultimediadataprocessingmodelemployedandtheMuXdistributedmultimediaI/Osystemthatusesthemodel.AmoredetaileddescriptionoftheMuXmodelisprovidedbyBakeretal.[3].2.1MuXMultimediaDataProcessingModelTheMuXmodelcomprisesastreamlayer,amultimediapresentationlayer,andahyperpresentationlayer,asshowninFigure2.Thestreamlayerprovidesservicessimilartothoseofavideorouterusedinvideoproductionstudios,wheresourcesanddestinationsmaybelocalorremotefilesordevices(suchasmicrophones,musicalinstruments,videocameras,displays,andaudiospeakers).Themultimediapresentationlayerservicesarebasedontheconceptofaprogrammablemediamultiplexerormediamixer,suchasanaudiomixerusedinconcertproductions,oraswitchorvideoeditorusedinavideoproductionstudio.Inthislayer,themediamultiplexertakesinputfromavarietyofsources,mixesthemaccordingtocontrollableparameters,anddirectstheresulttoanoutputportordestination.Thehyperpresentationlayerisageneralizationofthelinksdescribedabove,wherethedocument2 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993HyperPresentationLayerMultimediaPresentationLayerStreamLayerFigure2.MultimediaDataProcessingModelmayincludetime-basedmediasuchasaudioandvideo,andthelinksaredynamicandvaryovertime.Intheparagraphsthatfollowwedescribetheservicesprovidedbythesethreelayersingreaterdetail.Thestreamlayerprovidesservicestoaccessamultimediadatastreamfromafile,adevice,oranetworkconnection,orfromthehigherlayers;performsprocessingonthestream(e.g.,compressionorformatconversionusingfilters);deliversmultimediadatatoadestinationconsistingofeitherafile,adevice,oranetworkconnection,ortothehigherlayers,inatimelymanner;andallowstheselectionofaninputfromoneormorestreamsanddistributionoftheinputtooneormoredestinationsusingaswitch.Toprovidetheseservices,severalabstractionsandmechanismshavebeendefined,includingastream,source,destination,filter,andswitch.ThesemechanismsareillustratedinFigure3.Themultimediapresentationlayerbuildsonthestreamlayer.Amultimediapresentationisacollectionofstreamsthatarecoordinatedwithrespecttotimeandspace.Streamswithinapresentationaresynchronizedandhavesharedpresentationcontrol.Therearelogicalgroupingsofmediastreamsforintegrationandmedia-specificpresentationcontrol.Thestreamsofdissimilarmedia(e.g.,auralandvisual)aresynchronizedandpresentedsimultaneously.Streamsofsimilarmediacanbecut,reordered,processed,andmixedtoformanewstream:suchstreamsaregroupedtogetherasachannelforpresentationorfurtherprocessing.Keycomponentsofamultimediapresentationincludetracks,channels,andpresentations.Tracksaretime-orderedstreamsthatmaybereserializedviaasetofclipsoracliplist.Channelsareagroupoftracksthataremixedtogethertoformanewstream.Apresentationisthecompletesetofsynchronizedchannels.Toprovidethesemediaintegrationandsynchronizationservices,thefollowingmechanismshavebeendefined:logicaltimesystem(LTS),cue,presentationcontext,FilterFilterPipeSwitchFilterPipeFilterStreamStreamSourceDestinationFigure3.StreamLayerMechanisms3 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993mixer,clip,track,channel,andpresentation.TherelationshipsamongthesemechanismsareillustratedinFigure4.Thepresentation,channel,track,andclipmechanismsareusedprimarilytospecifytimingandsynchronizationrelationshipsandaredescribedinfurtherdetailinSubsection2.3.PresentationLTSPCChannelLTSPCTrackPCStreamClipListMixerPCStreamTrackPCStreamClipListFigure4.MultimediaPresentationMechanismsApresentationcontext(PC)specifiesparametersthatdefinehowmediastreamsaremixed,integrated,andpresentedtotheenduser.PCsarespecifictothemediumassociatedwithastream.ThepresentationcontextvalueassociatedwithaPCcanvaryovertime,offeringconsiderableflexibilityindefininghowmediastreamsareintegratedandmixed.Forexample,aPCforanaudiostreamwouldspecifythevolumeatwhichthatstreamispresented.Thevolumecanbespecifiedasafunctionoftime,asshowninFigure5.Presentationcontextsalsodefineparameterslikeoriginandextentofvideowindows.Thehighestlevelofthethree-layermodelisthehyperpresentationlayer.Asnotedabove,theabstractionofthehyperpresentationlayeristhatofadynamicnetworkofmultimediapresentationsconnectedbylinks,formingmultidimensional(timeandspace)multimediapresentations.Links,orhyperlinks,thatconnectmultimediapresentationstogetheraredynamicinthattheychangeovertimeandhavefixedlifetimesduringwhichtheycanbeactivatedVolume=“interval(0,3)*40+interval(3,5)*(40+(time-3)*(40/2))+Vinterval(5,9)*80+100ointerval(9,14)*(80-(time-9)*(80/5)”80lu60m40e202468101214TimeFigure5.Time-BasedPresentationContext4 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993[4;5;6;7;1].Withinthecontextofthismodel,hyperlinkscanbeassociatedwithtracks,channels,andpresentations,andcanlinkthemtoanyoftheaforementionedcomponents,asillustratedinFigure6.A“publication”bindstogetherasetofpresentationslinkedtogetherwithPublicationPresentationHyperLinkPresentationChannelHyperLinkChannelTrackHyperLinkTrackClipClipFigure6.HyperLinksbetweenMultimediaPresentationComponentshyperlinks.Thehyperpresentationlayer,includinghyperlinks,isthefocusofthispaperandwillbedescribedindetailinsubsequentsections.2.2TheMuXMultimediaI/OSystemWehaveimplementedadistributedmultimediaI/Osystem,calledMuX,basedonthemultimediadataprocessingmodeldescribedintheprevioussection.MuXsupportsasetofdistributedmultimediaI/Oservices,includingtheintegralservicesofmediatranslation,compression,andsynchronization.Itprovidesnetwork-transparentaccesstostoredmultimediadata,real-timemultimediaI/Odevices,andmultimediaprocessing.TheMuXsystemsupportstheroutinganddeliveryofreal-timemultimediadata;real-timevideo,audio,andgraphicsacquisition;andmediaprocessingandoutput.Tofacilitateitssupportofadistributedenvironment,theMuXsystemisbasedontheclient-serverparadigm.ThesystemconsistsofamultimediaI/Oserver[8];apresentationmanager;theMuXLibclientlibrary,whichembodiestheMuXapplicationprogrammer’sinterface(API)[9];andascriptinglanguage[10].Inthisenvironment,theclientapplicationspecifiesthetimingandsynchronizationrelationshipsamongmediastreams,andthemultimediaI/Oserversexchangereal-timedatadirectlyamongthemselvestomeetthesespecifications.Thesystemservicesaredefinedaccordingtoobject-orientedconcepts,bothindesignandinimplementation(usingC++).Thoughmostoftheservicesareimplementedintheserver,theclientlibraryutilizesadistributedobjectmessagingservicethatallowsobjectstosendmessagestoremoteobjectsinanetwork-transparentfashion,givingtheapplicationstheillusionthatobjectsarebeingexecutedlocallywheninfacttheymaybeexecutedonanothermachine.Thus,applicationprogrammerscandevelopdistributedapplicationsthattransparentlyutilizereal-timeresourcesonmultiplemachines,withouthavingtodealwiththecomplexitiesofinterprocesscommunicationandcoordinationofresources.5 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July19932.3TreatmentofTimeThenotionoftimeiscentralbothtothehyperpresentationimplementationandtotheMuXmodelingeneral.Timemaybespecifiedinseveraldifferentways,includingabsoluterealtimeorwall-clocktime;relativerealtimespecifiedinsecondsand/ormicrosecondsfromastarttime(whichmaybeeitherabsoluteorrelative);ortimespecifiedviatheSMPTEtimecode*(e.g.,00:07:05.10).Sincetheplayouttimesofmultimediapresentationsarenotknowninadvanceandaredeterminedbyuserinteraction,itisextremelyusefultospecifytimingrelationshipsintermsofrelativetime.Forthisreason,alltimingspecificationsaredefinedineitherrelativerealtimeorSMPTEtimecode,whichisinherentlyrelative.TheMuXsystemsupportsfourmethodsforspecifyingtimingandsynchronizationrelationshipsbetweenmedia:(1)hierarchicalrelationshipsspecifiedbytherelationshipbetweenpresentations,channels,andtracks,wheretracktimesarerelativetoachannel’stime,andchanneltimesarerelativetoapresentation’stime;(2)timingrelativetoatime-lineusingeitheralogicaltimesystem(LTS)[8]orarelativereal-timesystem(RRTS)[11;12];(3)relationshipsbetweenstreamsusingcues,e.g.,“before,”“after,”“starts,”and“finishes”[13;14;15];and(4)directinteractionswiththeuserviatheapplication.ThesetimingandsynchronizationspecificationsareillustratedinFigure7.Notethatduringexecutiontherelativetimesystemiselastic,meaningthatitcanstretchandshrinkbychangingthespeedofthepresentation.Guimaraesetal.[15]havedevelopedasetoftoolsfordefiningtemporalrelationshipsbetweenmediatypes,andasystemthatsupportsperiodic,time-event,andstate-changeactions;theirworkisbasedpartiallyonAllen’ssetofthirteentemporalrelationshipsbetweenpairsofobjects[16].Similarrelationshipshavealsobeenexplored[1;2;4;6;7;15;17;18;19;20;21].IntheMuXsystem,thegraphicalspecificationofrelationshipsisnotexplicit;rather,itisimplicit,expressingthestart-timeandend-timeordurationbymeansofacuethatisassociatedwithanothercomponent.ThetimingspecificationsemployedbyMuXarefundamentaltohyperlinktriggerdetectionandwillbediscussedinfurtherdetailinthefollowingsection.3.HYPERPRESENTATIONIMPLEMENTATION“Classical”hypertextisdiscussedintermsoflinksandnodes[22].However,thedefinitionsoflinksandnodeshavevariedamongauthors,researchers,andapplications[5;12;23].Forexample,nodescouldbethecurrentandsubsequentpointsofreference,andalinkcouldbeadefinitionofthewaytotraversefromonenodetoanother.Nodescouldcontainbitsofinformation,andlinkscoulddefinehowthesebitsofinformationrelatetooneanother.InthecontextoftheMuXmodelandsystem,wedefineahyperpresentationnodetobeamediumstreamorasetofmediastreamsassociatedwithaspanoftime;andahyperlinktobeasetofparametersandactionscontrollingthetransitionbetweentwohyperpresentationnodes.IntermsofthemodeldefinedinSection2.1,thehyperpresentationnodemaycorrespondtoatrack,achannel,orapresentation.*TheSMPTE(SocietyofMotionPictureandTelevisionExperts)timecodeisan80-bitstandardforrepresentingtimespecifiedin::.,andincludesothersynchronizationanduser-specifiedinformation.6 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993UserControl“Play”“Start”HyperLink“Play”4“Stop”PresentationAPresentationB33DisplayChannelDisplayChannelVideoTrackVideoTrackUserControl4232311AudioChannelAudioChannelAudioTrackAudioTrack2322211TimeTimeLEGEND1Hierarchicalrelationships2Relativetotimeline3Relationshipbetweentwostreamsusingcue4UsercontrolFigure7.TimingandSynchronizationSpecification3.1ClassesofHyperlinksTherearemanywaystocategorizeclassesofmultimediahyperlinks.Forexample,hyperlinksmightbecategorizedbywhethertheyconnectnodesthatoperatesimultaneouslyornodesthatoperatesequentially;orbywhetherornotthehyperlinksreturntheusertotheoriginatingnode.BrondmoandDavenport[4],inthe“ElasticCharles”project,usedlinkswhosephysicalappearancedenotedthetypeofmediatheyconnect:alinktoavideosegment,forexample,wasindicatedbyamicon,ormotionicon.Ogawaetal.[7]categorizelinksasuserdriven,timedriven,orconditiondriven.Broadlyspeaking,wehaveclassifiedthelinksinMuXaccordingto•Theiractivationperiods•Theircontextsensitivity•Theplasticityoftheiractivationareas.**Anothertermfor“activationareas”is“hotspots,”definedbyMichonas:“scalable,polygonalregionswithinimagesthattriggeractionsonpointereventssuchasmouseenterormousebuttonpresses.Fromtheusersperspectivetheybehaveastransparentbuttonsthatexactlyconformtoanobjectinavideoimage...Oncehotspotsaredefined,theyareautomaticallyscaledtoregistercorrectlywithavideoimageeveniftheimageisresized,moved,orstretched.”[6,p.365].7 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993Withrespecttoactivationperiods,somelinksareactivethroughoutthe“life”oftheirassociatedtrackorchannel.Forexample,afixed“TellMeMore”buttoncanbepresentedonthemainwindowatalltimes(itslocationandappearancedon’tchange,althoughwhatitpointstomight).Ontheotherhand,somelinksareactiveonlyduringveryspecificportionsofapresentation,suchasalinkassociatedwithaparticularperson’simagewithinavideo;thatlinkisactiveonlywhenthepersonisonscreen.Wedenotethespanoftimethatalinkisactiveasasegment.Segmentsaregenerallyassociatedwiththeunderlyingcontentofthepresentation.ThenotionofasegmentissimilartotheconceptofstratificationdescribedbyAguierreSmithandDavenport[24].TheTellMeMorebuttonmentionedaboveisanexampleofacontext-sensitivelink:thelocationpointedtobythelinkmayvaryaccordingtothecurrentpositioninapresentation.Other,lessdynamiclinksmayalwayspointtothesameplace,regardlessofwhentheyareactivated.TheTellMeMorebuttonisalsoanexampleofalinkwithanonchangingactivationarea:itslocation,extent,andappearancedonotchangeovertime(althoughaTellMeMorebuttoncouldconceivablychangefromdarktolighttoindicatethatitpointstomoreinformation).However,ahighlychangeablehyperlinkcouldbeassociatedwithamovingobject,suchasanairplanechangingdirectionandapproachingthecamerawithinavideo.Theactivationareaforthislinkishighlyplastic;itcanchangeitslocation,extent,andappearance(theoutlineoftheplane).Aspecialtypeoflinkistheautomaticcontext-sensitivetrigger,whichenablesanauthortolinkpiecesofinformationtogetherautomatically,andalsoenablesuserstodeterminewhichoftheseautomaticlinksistraversedwhiletheyviewapresentation,basedonthecontextinwhichtheviewingtakesplace.Forexample,a“MoreDetailon‘Time’”buttonmaybeofferedwhileauserisviewingapresentationonthispaper.Iftheviewerselectsthisbuttonpriortoinitiatingthepresentation,andapointinthepresentationisreachedwheretheconceptof“time”isintroduced,thepresentationwouldautomaticallypauseandalinktoarelatedpresentationwouldbeactivated.3.2HyperlinkParametersOnemayassumethat,ataminimum,linksneedtoidentifywheretheypoint,andwhatactioncausesthemtobeactivated.Linkscanalsocontainmuchotherinformation.Inthissection,welistandbrieflydefinethecategoriesofhyperlinkinformationusedintheMuXsystem,andpresentsomesimpleexamples.InSection4,wewillpresentmoredetailedexamplesofhyperlinkusage.Identifier:Auniquecharacterstringthatcanbeusedinexpressionsandprocedurestorefertoaparticularlink;optional;e.g.,“FirstJump,”“143,”“videoIntro.”ControlStream:Thetypeofstreamcontainingthetriggeringevent:e.g.,mouse,microphone,keyboard.TriggerEvent:Theeventthatactivatesthehyperlink,e.g.,,mouseWithin,leftButtonDown,“jump”or“link”(spoken);morethanonetriggercanactivatealink[6;15].Action:Whathappenswhenthelinkisactivated;multipleactionscanbetakenwhenthelinkisactivated;e.g.,stoppresentationofcurrentvideoandaudiotracks,presentspecifiedvideoincurrentwindow,opennewwindowandpresentspecifiedvideoinit.8 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993StartTime:Theinitialpointatwhichthehyperlinkisactive;thetimeisexpressedasrelativetothetrack,channel,orpresentationinwhichthehyperlinkisdefined.Duration:Thelengthoftimeinwhichthehyperlinkisactive.Appearance:Thephysicalformtakenbythehyperlink’sactivationarea;e.g.,dashedorsolidline;opaqueortransparent;circle,square,ormicon[4;6];outlinedorinvisible.Origin:Thelocationoftheupper-leftcornerofthephysicalactivationarea,expressedasthepixelspecificationofthexandycoordinates(andzcoordinates,for3-Dgraphicsandanimation).Thespecificationoftheoriginofhyperlink’sactivationareainwhichthephysicallinkchangeslocationovertimecanberelativelycomplex.Extent:Theshapeorboundingregionofthephysicalactivationarea,definedbyitsmaximumlengthandheight(anddepth,if3-D).Aswiththeorigin,theextentcanchangeovertime.Visuallinkscancontainanyoftheseparameters;audiolinksdonotcontaintheparametersforappearance,origin,orextentbecausetheyhavenophysicalactivationarea.Notethatdurationcanbeexpressedeitherasanexplicitlengthoftime,orasthedifferencebetweenasuppliedendtimeandthestarttime.3.3HyperLinkMechanismsThekeyobjectiveofthehyperlinkmechanismsistodetectwhenaneventinthecontrolstreammeetsthetriggerconditionsspecifiedbythehyperlink.Sincetheunderlyingmediumormediastream(s)associatedwithahyperpresentationnodemaychangeovertime,aswithvideo,thesemechanismsshouldexecuteinrealtimeandbecloselytiedtothepresentationofthemediumormediastream(s).Thereareacoupleofdifferentapproachestoachievingthisobjective.Onemethodistomarktheindividualbitsassociatedwiththehyperpresentationmediastreamtoindicatethattheyareassociatedwithaparticularactivationarea.Forexample,eachpixelofeachframeofavideostreamcouldbemarkedtoindicatethatitisassociatedwithanactivationareaonwhichahyperlink’striggerisbased.Whilethismethodmaybeeffectiveindeterminingexactlywhenanareahasbeenselected,itwouldrequireasignificantamountsofstoragespaceforspecificationandtimeforanauthortospecifywhichpixelsareassociatedwithanactivationarea.Thus,untilhigh-speedimagerecognitionmechanismsareavailabletodoon-lineprocessingofvideostreams,thismethodisnotviableforvideo.Analternativeapproachistospecifythehyperlinkregionsofinterestasaformalexpression.Dependingupontheircomplexity,expressionscanbeusedtodescribethelocationandshapeofactivationareasastheychangeovertime.Forexample,consideranactivationareaassociatedwithanobjectthatentersavideoclipfromoneside,graduallymovesacrossthescreen,andexitsontheotherside.Thismovementcaneasilybeexpressedasalinearfunctionoftime,wherebyanauthorwouldonlyberequiredtospecifythelocationandsizeoftheactivationareaattwolocationsinthevideosequence.9 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993Giventhecurrentcomputingcapabilitiesofworkstationsandtheauthoringflexibilityaffordedbythelattermethod,wehavebasedourhyperlinktriggermechanismsonformalexpressionsexpressedasfunctionsoftime.Themechanismsthatembodythespecificationoftheparametersthatdefinehyperlinknodes,executethehyperlinktriggerdetectioninreal-time,andperformthenecessaryoperationstoinstantiatethehyperlinkincludethefollowing:ConditionContext:Aconditioncontextdescribesaneventthattriggersahyperlink.Itspecifiesparameterssuchastriggerevent,andactivationareaoriginandextent.Eachoftheseparametersmaybespecifiedasanabsolutevalueorasaformalexpressionstatedasafunctionoftime.Aconditioncontextisresponsibleforevaluatingtheseexpressionstodeterminetheirvalueatspecificinstantsoftime.ControlStream:Acontrolstream,asnotedabove,isthemechanismthatactivatesahyperlinkwhenitmeetsaconditioncontext.Comparator:Acomparatoristheprimarymechanismfordetectingalinkortriggerconditionandexecutingtheactionsnecessarytolinkthepresentationstogether.Toperformthisoperation,acomparatorhasassociatedwithitacontrolstream(s)andacondition.Duringoperation(betweenthestarttimeandtheendtime),acomparatorcontinuouslycomparestheinputfromthecontrolstreamtotheassociatedconditionthatactivatesahyperlinkortrigger.Whensuchaconditionismetbythecontrolstream,thecomparatorexecutesanactiontohyperlinkthepresentations.Action:Anactionisasetofprogrammaticallydefinedoperationsthatperformahyperlinktraversal.Whenacomparatordeterminesthataconditionismet(e.g.,thatabuttonhasbeenmouseselectedatthelocationofahyperlink),thecomparatorexecutestheactionassociatedwiththehyperlink.Inaddition,ahyperlinkmechanismbundlesthesemechanismstogetherandassociateswiththemtheappearanceoftheactivationarea,andthetemporalaspectsofstarttimeandduration.ThemechanismsareillustratedinFigure8.HYPERLINK•StartTimeConditionContext•DurationActionControlStream•••ControlStreamComparatorFigure8.HyperPresentationMechanisms10 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993Notethatakeyissueintheimplementationofreal-timehyperlinktriggerdetectioniscoordinatingthetemporalrelationshipsbetweenthepresentationmechanisms(i.e.,themechanismsthatpresentwhattheuserseesandhears)andthetriggerdetectionmechanisms.Thisissueisfurthercomplicatedbytheuseofpresentationcontexts.Asabasisforthissynchronizationcoordination,thehyperlinkmechanismusesthestarttime,whichisrelativetothepresentationmedium,andtheduration.Specifically,thestarttimeanddurationareusedtoregistera“cue”withthemediumstreamwithwhichthehyperlinkisassociated,asillustratedpreviouslyinFigure7.Thecuemechanismkeepsthehyperlinkmechanismsinsynchwiththetimingofthepresentationbysendingupdatemessagesastothetimingstatusofthepresentationinrelationtothehyperlink,suchas“start,”“stop,”“pause,”and“changeofspeed.”Thehyperlinkpassesthesemessagestothecomparator,whichusesthemtocorrelatethetimingofthecontrolstreamwiththetimingspecifiedbytheconditioncontext.3.4HyperPresentationScriptingandtheMuXAPIAnintegralsupportfacilityforhyperpresentationsisascriptinglanguagethatallowsuserstodirectlyorindirectlyspecifytherelationshipsandhyperlinksbetweenmedia[1;6;2].TheMuXsystemprovidesascriptingtoolkitthatallowsuserstoscripthyperpresentations.TheMuXscriptinglanguage[10]isanobject-orientedlanguagethatprovidesfacilitiestodefinepublicationsthatmayconsistofmultiplepresentations(andtheirassociatedchannels,tracks,andsoon)andhyperlinksbetweenthepresentations.Theactiondefinitionsinthescriptspecifyexecution-orientedoperationssuchas“play,”“pause,”and“stop.”Thereisatightrelationshipbetweenthescriptingtoolkitandtheclientlibrary,whoseinterfaceisdefinedbytheMuXAPI[9].Theclientlibraryprovidesfacilitiestoconstructobjectsontheserverandsettheobjectparameters.Theobjectsconstructedareprimarilycontrolstructuresusedforexecutingreal-timepresentations.Aftertheobjectshavebeenconstructedandthecontrolstructuresdefined,theAPIdefinesexecutionoperations,suchas“play,”“pause,”and“stop,”thatallowapplicationstocontroltheviewingofmultimediapresentations.Thesesameoperationscanalsobeusedbythescriptlanguagetodefinehyperlinkactions.Thus,ahyperpresentationscriptcanmodeluserinteractionsanddefinetheappropriateresponses.3.5HyperPresentationSystemOverviewTogainanoverviewofhowtheMuXsystemimplementsahyperpresentation,considertheinteractions,illustratedinFigure9,betweentheMuXsystemcomponents(theapplication,scriptingtoolkit[10],clientlibrary[9],andserver)toloadascriptandexecuteapresentation.Thisoperationisinitiatedbytheapplication,whichinstructsthescripttoolkittoloadapublication.Thepublicationdefinedbyascriptspecifiestheconstructionandparametersofstream,multimediapresentationandhyperpresentationobjects.Messagestoconstructtheseobjectsarepropagatedthroughthesystem,asillustratedintheleftmostcolumninFigure9.Afterconstructionoftheobjectsthatdefineapublication,theapplicationcanstartthehyperpresentationbysendinga“play”messagetothe“root”presentationspecifiedbythepublication.AsshowninthesecondcolumnfromtheleftinFigure9,thisplaymessagebypassesthescripttoolkitandisrelayedtotheappropriatepresentationobjectintheserver.Thispresentationobjectinturnsendsmessagestoitsassociatedchannel,trackandstreamobjectstoexecutethepresentation.Iftheusertriggersahyperlinkwhilethepresentationisplaying,thistriggeringeventisdetectedbythecomparatorassociatedwiththehyperlink.Whenthistriggeringeventisdetectedfromthecontrolstream(whichmaybegeneratedfromauser’smouseinputs),theactionobjectassociatedwiththehyperlinksendsamessagetotheclientlibrary,asillustratedinthethird11 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993columnfromtheleftinFigure9.Thismessageisreceivedbyaproxyactionobjectontheclientside,whichcallsacallbackfunction(registeredwiththeactionobjectwhenthepublicationwasconstructed).Thiscallbackfunctioncontainstheoperationsthatexecuteahyperlink,suchas“pause”and“play”messagestopresentationobjects,asillustratedintherightmostcolumninFigure9.4.HYPERPRESENTATIONSINACTIONInthissection,wepresentseveralexamplescenarios,describerepresentativehyperlinksandtheiruses,andincludethescriptlanguageusedtodefinethehyperlinks.Ourexampleapplicationisasfollows:imaginethatwevideotapeoneoftheauthorsofthispaper,Earl,givingaverbalpresentationonHyperPresentationinMuX.WealsotapetheMuXdevelopmentteaminaction,atatypicaldesignmeeting.ThevideotapeincludesvariousshotsofEarl,standinginfrontofawhiteboard,talkingdirectlytothecamera,talkingaboutMuXClientApplicationLoadScriptPlayPresentationScriptToolkitDefinitionofHyperLinkExecution•Streams•ActionCallback•Presentations•HyperlinksClientLibraryCreate,SetParamsPlayCallbackReceivedPause,Play•Streams•Presentation•Action•Presentation•Presentations•HyperlinksNetworkMuXServerHyperPresentationCreate,SetParamsTriggerDetected•HyperLink•Comparator•ConditionContext•ActionMultimediaPresentationCreate,SetParamsPlayPlay•Presentations•Presentation•Presentation•Channels•Channels•Channels•Tracks•Tracks•TracksStreamCreate,SetParamsPlayInputReceivedPlay•Sources•Streams•Stream•Streams•Streams•DestinationsFigure9.MuXClient-ServerInteractionstoExecuteaHyperPresentation12 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993viewgraphs,showingscreenshots,etc.Thevideotapeisdigitized,andtheaudioandvideoarestoredasseparatefiles.Next,wedevelopahyperpresentationstoryboard,inwhichwedividethevideoandaudiointosections:Abstract,Introduction,RelatedResearch,etc.Foreachsection,wemarkthebeginningandendingframesofthecorrespondingvideoandaudiotracks.Ageneralapplicationcanbeusedtoprovideagraphicaluserinterfaceforon-linetalks.ButtonsontheGUIinclude“Return,”theubiquitous“TellMeMore,”andtypicalVCRcontrolbuttons(Pause,Play,Rewind,FastForward,etc.).Thereisalsoasmallsetofradiobuttons,labelled“AudioOnly,”“VideoOnly,”and“Audio-Video.”NowwewritetheMuXscriptforourpresentation:wedefinetheinputandoutputstreams,theaudioandvideochannels,thevarioustracks,andtheclipswithineachtrack.Notethatthetrackslistedwillincludefilesfortheopeninggraphics,theviewgraphs,andthebuttonsontheGUI,aswellastheaudioandvideofiles.Themainpresentationwillincludeonlyahigh-leveloverviewofthetalk:perhapsagraphiccontainingopeningtitles,theintroduction,andafewoverviewsentencesfromeachsection.Ateachtracklevel,however,wehavealsodefinedanumberofhyperlinks,whichcanbeusedthroughoutthepresentationtogetaccesstomoredetailedinformation,referbacktoothersections,orjumparound.Followingaresomeexamplesofhyperlinkdefinitionsfromthescriptsforthevarioustracks.4.1ExampleOne:ContinuouslyActive,Context-Insensitive,NonplasticHyperlinkTheviewgraphtrackcontainstheactualgraphicsusedtoproducetheviewgraphs.Wehavecreatedhyperlinkactivationareasforbulletlistitemsoneachviewgraph.Forexample,intheviewgraphentitled“MuXObjectClassHierarchy,”therearesixbulletitems;eachhasacorrespondingactivationarea(seeFigure10).Mouseclickingonanyoftheseareastakestheusertoanexpandedpresentationonthatparticularsubject.Thehyperlinkdefinitionforthechannellookslikethefollowing:channelLink=hyperlink[controlStreammouse;triggerleftButtonUp;startTimecue[startTimeObjectClassVG];endTimecue[endTimeObjectClassVG];action{pauseObjectClassVG;playchannelPresentation;};appearancebullet;origin(40,40);extent(5,5);]Inthisexample,“channelPresentation”isthenameofapresentationthathasbeendefinedwithinthepublication,whichshowstheappropriateviewgraphandpresentsthecorrespondingaudioexplanation.Thelink“channelLink”isactivewheneverandaslongasthetrackObjectClassVGisbeingpresented:inotherwords,itiscontinuouslyactiveandcontextinsensitive,anditsshapedoesnotchange.13 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993MuXObjectClassHierarchy•Publication•Presentation•Channel•Track•Hyperlink•StreamAudioOnlyVideoOnlyVCRControlsTellMeMore...Audio-VideoFigure10.ExampleHyperPresentationwithNonplasticHyperLink4.2ExampleTwo:ContinuouslyActive,Context-Insensitive,NonplasticHyperlinkFromourradiobuttonpanel,wehaveselectedAudio-Video.WearewatchingapresentationonSection5ofthispaper;itconsistsofavideoofEarltalkingaboutthefocusofourfutureeffortsontheMuXproject.Whenhegetstothepartaboutcontext-basedqueries,weclickthe“TellMeMore”button,andchangeourradiobuttonselectionfromAudio-VideotoAudioOnly(seeFigure11]).Thepresentationpauses,stoppingtheflowofaudioinformationandfreezingtheAudioOnlyVideoOnlyVCRControlsTellMeMore...Audio-VideoFigure11.HyperPresentationExamplewithContext-SensitiveHyperLinkcurrentframeofEarl.Avoice-overdiscussionbegins,providingfurtherdetailonthetopicofcontext-basedquerying.Partofthescriptforthe“TellMeMore”buttonlookslikethefollowing:14 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993moreButton=hyperlink[controlStreammouse;triggerleftButtonUp;startTimecue[startTimefutureDirPres];endTimecue[endTimefutureDirPres];appearancegraphic;origin(512,512);extent(20,5);action{if(interval(t1,t6))pausefutureDirPres;if(interval(t1,t3))playAbstractPres;if(interval(t3,t4))playIntroPres;if(interval(t4,t6))playDetailedFutureDirPres;};Becausethetargetpresentationofthe“TellMeMore”buttonchangesovertime,thehyperlinkdefinitionspecifiesthetargetvalueforeachdiscretetimeinterval,asshowninthegraphinFigure12.FUTUREDIRPRESINTROPRESABSTRACTPRESt0t1t2t3t4t5t6Figure12.ActivePeriodsofaHyperLinkThe“playFutureDirPres”functionisexecutedonlyduringthespecifiedtimeinterval.Whenexecuted,itpausesthecurrentpresentation,determinestheradiobuttonselection,andplaystheappropriatetrack(s)—inthiscase,onlytheaudiotrack.The“moreButton”linkiscontinuouslyactiveandcontextsensitive,butitsappearanceandshapeneverchange.4.3ExampleThree:Context-Insensitive,PlasticHyperlinkAttheendofthehigh-levelpresentation,butbeforetheclosingtitlesandmusic,weshowavideooftheMuXdevelopmentteam.Althoughmostoftheparticipantsaresittingaroundthetable,individualsarecontinuallyjoiningorleavingthemeeting,goinguptothewhiteboardtowriteideasdown,orturningtowardsthecurrentspeaker.Throughoutthisclip,therearefaintoutlinesdrawnaroundthevariousindividuals.Movingthemousecursorintooneoftheseshapespopsupasmallwindowcontainingtheperson’sname,title,andcontributiontotheMuXproject.Movingthecursoroutsidetheshapecausesthepop-upwindowtodisappear.Asampledefinitionofoneofthesehyperlinkswouldbeasfollows:15 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993earlLink=hyperlink[controlStreammouse;triggermouseWithin;startTimecue[startTimemuxTeamPres];endTimecue[endTimemuxTeamPres];action{playearlPres};appearanceoutline;origin(“interval(t1,t2)*time*((x2-x1)/(t2-t1))+interval(t2,t3)*time*((x3-x2)/(t3-t2))+interval(t4,t3)*time*((x4-x3)/(t4-t3))”,“interval(t1,t2)*time*((y2-y1)/(t2-t1))+interval(t2,t3)*time*((y3-y2)/(t3-t2))+interval(t4,t3)*time*((y4-y3)/(t4-t3))”);extent(“interval(t1,t2)*5+interval(t2,t3)*10+interval(t3,t4)*15”,“interval(t1,t2)*15+interval(t2,t3)*30+interval(t3,t4)*45”);]Inthiscase,theoriginsandextentsofhyperlinksaredefinedwithanexpressionandarebrokendownintothevaluesthatapplywithinthediscretetimeintervals.Thehyperlink“earlLink”isanexampleofalinkthatisactiveonlyduringspecificportionsofaclip.Itisnotcontextsensitive,sinceitalwaysdisplaysthesamepop-up,regardlessofwhenitisactivated.Also,itsoriginandextentchangefrequentlyovertime,makingitsactivationareahighlyplastic.5.FUTUREDIRECTIONSThecurrentimplementationofhyperpresentationswithinMuXhasservedasaproofofconcept.Futuredevelopmentwillcontinue,expandingtheflexibilityandpowerofthehyperpresentationcomponentinseveralareas.Belowwedescribesomefuturedirectionsforresearchanddevelopment.First,wewouldliketoaddagraphicaluseinterface,sothatauthorscaninteractwiththesysteminamorenaturalmode,andwillnothavetolearnthesyntacticidiosyncraciesofthescriptinglanguageandAPI.Wehavedevelopedmock-upsofseveralversionsofaGUI,butasofthispublicationhavenotimplementedthem.Thecharacteristicsofnodesandlinks,andthenode-linkrelationshipitself,needtobefurtherrefinedforthemultimediaenvironment.AspointedoutbyLaurel,Oren,andDon[25],theapplicationofhypermediatomultimediarequiresnewconventions.Weplantoinvestigatemorefluidtransitionsbetweentwohyperpresentationnodeswhenahyperlinkistraversed.Onecannotjust“jumpto”avideosegmentwithoutriskingadverseeffects,becausevideoimpliesatemporaldimensionaswellastheoccupationofa2-dimensionalspace.Forexample,onemustspecifywhichframetojumptowhengoingtoavideofile:thefirst,current,orsomeotherframeofthespecifiedvideo?Andwhenavideopresentationisexited(viahyperlinkorotherwise),atwhatframeorinwhatstateisitleft?Thesameconsiderationsholdforaudiodata,graphicaldata,etc.InadditiontoGUI-basedauthoringmethods,weplantoinvestigatemethodsforeasingtheauthor’sburdenincreatinghyperpresentationscripts.Ourfirststepwillbetoautomatetheprocessofgeneratingtime-varyingexpressionsbasedinitiallyongraphicalinteractionsand16 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993lateronimageanalysis.Wealsoplantoinvestigatetherelationshipbetweenhyperpresentationauthoringsystemsandmedialoggingsystemssuchasvideologgers.Finally,weplantoaddressissuesrelatingtotheintegrationofcontext-basedquerieswithanauthoringsystem[1;6;7;19].6.ACKNOWLEDGEMENTSThisworkwasconductedwiththecooperationandsupportoftheElectronicsandTelecommunicationsResearchInstitute,Korea,whichissponsoringSRIProject2026,“IntegratedMultimediaI/OInterfaceforIntelligentComputing.”Theauthorswouldliketoacknowledgethegracioussupportofthefollowingpeoplewhocontributedtodevelopingthesystemdescribedinthispaper:RustiBaker,MichaelBrown,AlanDowning,SooHyoungOhe,JoshuaNicholas,ScottRandolph,NagendraSiravara,andDr.DongHoSong.TheauthorswouldalsoliketothankLouSchreier,MikeDavisandAlanDowningfortheirassistanceinpreparingthispaper.7.REFERENCES[1]Zellweger,P.,“Towardamodelforactivemultimediadocuments,”inMultimediaInterfaceDesign,eds.M.BlattnerandR.Dannenberg,ACMPress,NewYork,1992.[2]Buchanan,M.C.,andZellweger,P.,“SchedulingMultimediaDocumentsUsingTemporalConstraints,”presentedat3rdInternationalWorkshoponNetworkingandOperatingSystemsforDigitalAudio-Video,SanDiego,California,November1992.[3]Baker,R.,Downing,A.,Finn,K.,andRennison,E.,“MultimediaProcessingModelforaDistributedMultimediaI/OSystem,”presentedat3rdInternationalWorkshoponNetworkingandOperatingSystemsforDigitalAudio-Video,SanDiego,California,November1992.[4]Brondmo,H.P.,andDavenport,G.,“CreatingandviewingtheElasticCharles-ahypermediajournal,”inHypertext:StateoftheArt,eds.R.McAleeseandC.Green,Intellect,Ltd.,GreatBritain,1990.[5]Halasz,F.,“ReflectionsonNoteCards:SevenIssuesfortheNextHypermediaSystems,”CommunicationsoftheACM,Vol.34,No.12,1988,pp.836-852.[6]Michon,B.,“Highlyiconicinterfaces,”inMultimediaInterfaceDesign,eds.M.BlattnerandR.Dannenberg,ACMPress,NewYork,1992.[7]Ogawa,R.,Harada,K.,andKameko,A.,“Scenario-basedHypermedia:AModelandaSystem,”inHypertext:Concepts,Systems,andApplications,eds.A.Rizk,N.Steitz,andJ.Andre,CambridgeUniversityPress,GreatBritain,1990.[8]Rennison,E.,Baker,R.,Kim,D.-H.,andLim,Y.-H.,“MuX:AnXCo-ExistentTime-BasedMultimediaI/OServer,”TheXResource,Issue1,Winter1992,pp.213-33.[9]Finn,K.,Downing,A.,Randolph,S.,Rennison,E.,andKim,D.-H.,ApplicationProgrammerInterfaceforMuX,ITAD-2026-TR-93-104,SRIInternational,MenloPark,California,May1993.[10]Finn,K.,andRennison,E.,MuXScript:AScriptingLanguageforSpecifyingTime-BasedMultimediaandHypermediaPresentations,ITAD-2026-TR-92-147,SRIInternational,MenloPark,California,October1992.[11]Rennison,E.,Brown,M.,Downing,A.,Finn,K.,andRandolph,S.,TheMuXMultimediaI/OSystem,ITAD-2026-TR-93-178,SRIInternational,MenloPark,California,May1993.[12]Halasz,F.andConklin,J.,“IssuesintheDesignandApplicationofHypermediaSystems,”TutorialatCHI’90,Seattle,Washington,April1990.17 3rdAustralianMulti-MediaCommunications,Applications,andTechnologyWorkshopWollongong,NewSouthWales,14-15July1993[13]Little,T.T.D.,andGhafoor,A.,“SynchronizationandStorageModelsforMultimediaObjects,”JournalonSelectedAreasofCommunications,Vol.8,No.3,April1990.[14]Blakowski,G.,Hubel,J.,andLangrehr,U.,“ToolsforSpecifyingandExecutingSynchronizedMultimediaPresentations,”SecondInternationalWorkshoponNetworkingandOperatingSystemSupportforDigitalAudioandVideo,Heidelberg,Germany,November1991.[15]Guimaraes,N.,Correia,N.,andCarmo,T.,“Programmingtimeinmultimediauserinterfaces,”ProceedingsoftheACMSymposiumonUserInterfaceSoftwareandTechnology,15-18November,Monterey,California,ACMPress,NewYork,1992,pp.125-134.[16]Allen,J.,“MaintainingKnowledgeaboutTemporalIntervals,”CommunicationsoftheACM,26(1),November1983,pp.832-843.[17]Buchanan,M.C.,andZellweger,P.,“Specifyingtemporalbehaviorinhypermediadocuments,”ProceedingsoftheACMConferenceonHypertext,30November-4December,Milano,Italy,ACMPress,NewYork,1992,pp.262-271.[18]Fountain,A.,Hall,W.,Heath,I.,andDavis,H.,“MICROCOSM:Anopenmodelforhypermediawithdynamiclinking,”inProceedingsoftheACMConferenceonHypertext,1990,pp.298-311.[19]Ogawa,R.,Tanaka,E.,Taguchi,D.,andHarada,K.,“Designstrategiesforscenario-basedhypermedia:Descriptionofitsstructure,dynamics,andstyle,”ProceedingsoftheACMConferenceonHypertext,30November—4December,Milano,Italy,ACMPress,NewYork,1992,pp.71-80.[20]Rhyne,J.,andWolf,C.,“Toolsforsupportingthecollaborativeprocess,”UIST’92,15-18November1992,pp.161-170.[21]Sherman,M.,Hansen,W.,McInerny,M.,andNeuendorffer,T.,“Buildinghypertextonamultimediatoolkit:AnoverviewofAndrewToolkit,”inProceedingsoftheACMConferenceonHypertext,CambridgeUniversityPress,GreatBritain,1990,pp.13-24.[22]Conklin,J.,“Hypertext:AnIntroductionandSurvey,”IEEEComputer,1987,pp.17-41.[23]Nielsen,J.,Hypertext&Hypermedia,AcademicPress,Inc.,SanDiego,California,1990.[24]AguierreSmith,T.G.,andDavenport,G.,“TheStratificationSystem:ADesignEnvironmentforRandomAccessVideo,”ThirdInternationalWorkshoponNetworkandOperatingSystemSupportforDigitalAudioandVideo,SanDiego,California,November1992.[25]Laurel,B.,Oren,T.,andDon,A.,“IssuesinMultimediaInterfaceDesign:MediaIntegrationandInterfaceAgents,”inProceedingsofCHI’90,Seattle,Washington,April1990.18