Automated evaluation of essays and short answers

《Automated evaluation of essays and short answers》由会员上传分享，免费在线阅读，更多相关内容在行业资料-天天文库。

1、AUTOMATEDEVALUATIONOFESSAYSANDSHORTANSWERSJillBursteinClaudiaLeacockRichardSwartzAutomatedEvaluationofEssaysandShortAnswersJillBurstein(jburstein@etstechnologies.com)ClaudiaLeacock(cleacock@etstechnologies.com)RichardSwartz(rswartz@etstechnologies.com)ETSTechnologies,Inc.ASubsidiaryofEdu

2、cationalTestingServicePrinceton,NJhttp://www.etstechnologies.comAbstractEssayquestionsdesignedtomeasurewritingability,alongwithopen-endedquestionsrequiringshortanswers,arehighly-valuedcomponentsofeffectiveassessmentprograms,buttheexpenseandlogisticsofscoringthemreliablyoftenpresentabarri

3、ertotheiruse.ExtensiveresearchanddevelopmenteffortsatEducationalTestingService(ETS)overthepastseveralyears(seehttp://www.ets.org/research/erater.html)innaturallanguageprocessinghaveproducedtwoapplicationswiththepotentialtodramaticallyreducethedifficultiesassociatedwithscoringthesetypesof

4、assessments.Thefirstofthese,e-rater™,isasoftwareapplicationdesignedtoproduceholisticscoresforessaysbasedonthefeaturesofeffectivewritingthatfacultyreaderstypicallyuse:organization,sentencestructure,andcontent.Thee-ratersoftwareis"trained"withsetsofessaysscoredbyfacultyreaderssothatitcanac

5、curately"predict"theholisticscoreareaderwouldgivetoanessay.ETSimplementede-rateraspartoftheoperationalscoringprocessfortheGraduateManagementAdmissionsTest(GMAT)in1999.Sincethen,over750,000GMATessayshavebeenscored,withe-raterandreaderagreementratesconsistentlyabove97%.Thee-raterscoringcap

6、abilityisnowavailableforusebyinstitutionsviatheInternetthroughtheCriterionOnlineWritingEvaluationserviceathttp://www.etstechnologies.com/criterion.Theserviceisbeingusedforbothinstructionandassessmentbymiddleschools,highschools,andcollegesintheU.S.ETSTechnologiesisalsoconductingresearchth

7、atexploresthefeasibilityofautomatedscoringofshort-answercontent-basedresponses,suchasthosebasedonquestionsthatappearinatextbook'schapterreviewsection.Ifsuccessful,thisresearchhasthepotentialtoevolveintoanautomatedscoringapplicationthatwouldbeappropriateforevaluatingshort-