Advances in technology—including artificial intelligence (AI), as well as computer- and Web-based technologies—have led to the development of exciting instructional and testing applications for teaching writing. Many new capabilities are appearing, yet most are just enhancements or improvements to processes and methods that have been around for a while. Equipment such as wireless classroom setups and hand-held devices offer exciting possibilities, but some of the radical changes in writing instruction are stemming from the development and relatively widespread use of computerized essay correction technology.
Interest in this technology has been powered largely by the many voices raised in concern about the weak writing of American students and the need for improvement in this area. As part of this movement, more and more standardized tests such as the SAT, GRE, and ACT are including essay components.
Most of these standardized tests have been computer-scored for a number of years in all areas, except the short answer and essay sections. Tests like Michigan's MEAP, for example, are mainly machine-scored, with human scorers correcting the few brief essay or short answer sections—a fairly complicated and expensive process.
Scoring essays effectively and providing helpful feedback is a far different proposition from machine-scoring multiple-choice tests. As more tests are including essay and other open-ended writing components, better methods are constantly being sought to cut scoring time and expense.
The top testing companies provide an array of essay-scoring products. Educational Testing Services (ETS) offers e-rater, Pearson Knowledge Technologies has Intelligent Essay Assessor (IEA), and Vantage Learning offers IntelliMetric. There are a few other offerings, but these are the three leading artificial intelligences, technologies, or engines that are used in a wide number of tests and that are spreading into writing instruction from about fourth grade through college levels.
How They Work
The three technologies mentioned above are basically engines or core programs that are generally bundled in with other elements by their owners and/or licensed to other companies. e-rater, for example, is part of ETS's Criterion online service, IEA is part of Pearson Knowledge Technologies' Summary Street, and IntelliMetric is part of Vantage Learning's My Access!. Core technologies such as these have also been licensed to other companies, including Prentice Hall; Holt, Rinehart and Winston; and others.
These technologies are also often adapted for other purposes. e-rater, for example, grades essays and has two offshoots—one that specializes in grammar and mechanics (Critique) and one that handles grading open-ended questions (c-rater).
Each of these proprietary technologies uses a different approach and produces different results or a different slant to essay grading. e-rater emphasizes natural language processing for scoring grammar and organization. IEA uses Latent Semantic Analysis and other machine learning and statistical methods for scoring. These methods emphasize content coverage in scoring but also include grammar, mechanics, style, and organization, as needed. Vantage Learning doesn't discuss its MY Access! technology other than to state that the product uses artificial intelligence technologies.
All of the technologies have been extensively tested, and all of them approach or equal human scorers in reliability. All produce holistic grades from 1 to 4 or 1 to 6—similar to those produced by human graders— as well as provide tutorial advice intended to guide student writers as needed. All have similar limitations, which will be discussed later.
The technologies interact with students in roughly the same way; i.e., all are Web-based and the essays are computer-entered. The Web-based platform offers a serious advantage, enabling students to use the services in any location where they can use a computer—with their class, at home, in the library, or in the lab.
Students enter essays in response to prompts based on writing topics the computer software can score. The following is an 11th grade writing prompt from the Intelligent Essay Assessor Demonstration page: "Please write a structured essay on the Great Depression and the New Deal."
Currently, a limited number of prompts are available. Although companies are developing more every year, the limited availability is due to the expense and length of time involved in developing scoring models for each prompt.
To make a specific prompt machine-scorable, up to 300 students write an essay in response to that prompt. These initial essays are scored by two specially trained humans. (If they disagree, a third person evaluates the essays.) The essays and scores are fed into computers that develop scoring models that work for that prompt only. This entire process is repeated for each new prompt.
The cost of computerized essay-grading programs varies greatly and generally is determined by specific-use situations. The price may be determined as a per-student subscription or the fee may be based on student use as determined by a hit or bundles of hits. For example, Criterion from ETS is priced at $15 per student per year, while Holt Online Essay Scoring is priced at $1 per hit (per essay scored once) in batches of 50 hits.
Doubts and Limitations: Frankenstein in the Classroom?
There are many questions surrounding the use of computerized essay scoring.
How can computers accurately score the nuances of the written word and/or something as complex as an essay? What standards should be used, since experts rarely agree about the nature of good writing? Should the mechanics be emphasized, or should the emphasis be placed on the structure and quality of the content and thought involved?
Even developers admit that computers can't assess creative, innovative, or individualistic writing. Another important element that computers can't score is the context of the essay or the audience to which an essay is addressed.
By being unable to deal with these and other elements, will computerized essay scoring somehow force students into some mold of monstrous uniformity? Will it leave the creative side undeveloped?
In addition to these concerns, computerized essay scoring programs are limited by the material fed to them to create scoring models, as well as the limited number of available topics or prompts. And, it's not all that hard to deceive essay grading technologies; they can be fooled. Finally, when it gets right down to it, many educators just don't trust computers and don't want to let them influence student lives any more than they already do.
Any of these perceived disadvantages, however, can be overcome in the classroom by preparing students for the differences that computerized essay scoring programs create in testing situations and by not relying exclusively on computerized scoring.
There are advantages to the computerized systems, too. Computerized essay grading and Web-based writing training systems provide instant and immediate feedback. Students can make improvements and resubmit their work, each time honing the quality of the essay. With some systems, teachers can add comments or change the grade, if desired. The programs can make more writing possible by enabling teachers to spend less time correcting essays.
As a former high school English teacher with classes of 25 to 35 students, I get excited at the thought of computers handling some of the incredibly lengthy processes of scoring 130 or so student essays. The length and tediousness of correction has, by itself, led to less writing instruction in many schools because teachers just don't have enough time to correct all of the essays that should be assigned.
A Web-based training system would make far more writing practice possible, even if careful teacher oversight and other measures are required. The improved focus, understanding, and skills that students gain in these writing situations will be beneficial as they carry over into other areas.
Today, computerized essay scoring applications are in rapid development by a number of respected companies that have long produced testing, textbook, and other products. Among the top technologies in current use are e-rater, Intelligent Essay Assessor, and IntelliMetric.
Educational Testing Service, Rosedale Road, Princeton, NJ 08541; Phone: 609/921-9000; Internet: http://www.ets.org/criterion/.
Criterion 3.0, developed by ETS, is powered by an improved version of e-rater, v2.0. Criterion gives students self-paced writing practice using writing assignments created with parameters and guidelines that instructors choose. The program provides 44 prompts for grades 4 to 8 and 29 prompts for high school, all developed in much the same way as those used with other essay scoring engines.
Recent updates to Criterion allow the program to automatically provide a holistic score and feedback on instructor-created topics without any additional customization or training. Elementary and secondary school teachers can tailor writing topics to their own lesson plans or state standards and provide students with a risk-free automated environment to improve their writing skills.
Tennessee's Knox County schools use Criterion in the middle and high schools. County educators felt that Criterion helped their students become better writers and gave them valuable practice in taking standardized tests. Knox County also collaborated with ETS to develop unique topics in alignment with state testing. As a result, student writing is compared to Tennessee state standards, as well as the national standards embedded in the Criterion topics.
Additional improvements to Criterion 3.0 include enhancements to the scoring engine, newly developed grade-specific scoring models, scoring feedback and reports based on a set of 12 explicitly stated features common to most writing assessment evaluations, and the application of the same scoring criteria to every essay. In addition, Writer's Handbooks are available with vocabulary-appropriate feedback messages for elementary, middle, high school/college, and English language learners.
Diana Schmelzer, principal of University High School in Irvine, Calif., believes that the use of Criterion in her school saves each teacher who assigns essays approximately 60 hours of grading for nearly 180 essays per topic.
To read more about Criterion in action, see the article "Turning Out Better Writers: Practice Makes Perfect, and Computer-Based Writing Makes for More Practice" in the May/June 2004 issue of MultiMedia & Internet@Schools.
Intelligent Essay Assessor
Pearson Knowledge Technologies, 4940 Pearl East Circle, Suite 200, Boulder, CO 80301; Phone: 303/545-9092; Internet: http://www.knowledge-technologies.com/.
Intelligent Essay Assessor (IEA) was developed by the former Knowledge Analysis Technologies, now a Pearson company known as Pearson Knowledge Technologies. IEA technology focuses on the understanding of the subject matter that goes into creating an essay.
Though IEA focuses on meaning and content, it can also measure mechanics, styles, and more. This technology is used in Pearson Knowledge Technologies' Summary Street and is also used by NCS Pearson; Prentice Hall; Holt, Rinehart and Winston; and others.
Summary Street is a particularly interesting application of IEA in that it guides student summary writing, working with reading comprehension as well as writing. The program was developed in conjunction with the Colorado Literacy Tutor project and is being used by schools throughout Colorado.
Summary Street compares student summaries to the original text, providing feedback about accuracy, comprehensiveness, and length, as well as problems like redundancy, poor spelling, and extraneous sentences. The program works well, but, as with other similar programs, it isn't perfect. Summary Street doesn't evaluate structure well, can't recognize creativity or metaphor, and doesn't evaluate style.
Prentice Hall's Online Essay Scorer [http://www.phschool.com/language_arts/] uses IEA technology to provide summaries of misspelled words, grammatical errors, and redundant sentences, in addition to a holistic score. An online demonstration is available, illustrating the program's instant scoring and feedback; sample essays are also posted online.
Holt, Rinehart and Winston [http://www.hrw.com/language/] incorporates IEA into its Elements of Language and Holt Online Essay Scoring programs to help students prepare for standardized tests and other classroom uses.
The Elements of Language program includes an interactive online textbook and Web site. The textbook provides reading and writing activities as well as diagnostic and chapter tests. The Web site includes a language center with interactive vocabulary instruction and links to online writing resources and reference materials.
At the time this article was written, Holt Online Essay Scoring offered 58 prompts, with new prompts to be added continually. The program includes seven types of prompts for middle and high school: expository, persuasive, how-to, descriptive, narrative, writing about literature, and writing about nonfiction. An additional prompt—biographical narrative—is offered on the high school track. Teachers cannot add their own prompts to this program.
Vantage Learning, 110 Terry Drive Suite 100, Newtown, PA 18940; Phone: 800/230-2213; Internet: http://www.vantagelearning.com/.
MY Access! [http://www.gomyaccess.com], based on the IntelliMetric scoring engine, is a prompt-driven, Web-based writing environment that scores student essays written in English, Spanish, or Chinese. A Japanese language component is in development.
The program's instant diagnostic instruction helps students improve their writing and helps teachers shape instruction to student needs. The idea is that more frequent opportunities to write with immediate scoring and feedback will encourage improvement.
According to Vantage Learning, the IntelliMetric scoring system is as accurate as human scorers. An adaptation of this technology, IntelliMetric ShortStuff, is used to score very short, open-ended essay responses.
MY Access! provides students with a suite of writer's tools that can be turned on or off by the teacher. The tools include checklists, a dictionary/thesaurus, word banks, graphic organizers, a writer's journal, rubrics, a spell-checker, a grammar and style editor called MY Editor, and more. The program enables two-way communication; students can read and write comments and respond to teacher comments. Charts and graphs are created as students work, monitoring their progress.
When this article was written, MY Access! contained approximately 200 prompts: Ten prompts will be added each year to each of the four levels. The levels include upper elementary, middle school, high school, and higher education. The prompts cover five areas: informative, narrative, literary, text-based/informative, and persuasive. Teachers can add prompts to MY Access! and students can respond using the tools, but these essays cannot be scored.
This program has been used in Birmingham High School in the Los Angeles Unified School District in a partnership that brought mobile assessment carts into the school. Each cart contained 25 Apple iBook computers loaded with MY Access! writing instruction software. One year of use produced a significant increase in the number of Birmingham students who passed the California High School Exit Examination (CAHSEE).
"It's really a voyage of discovery," said Catherine Jo Foss, assistant principal. "The kids log on, pull up their portfolios, write, rewrite, submit, rewrite, get their scores, and then do it all over again in order to move their score from a two to a three or a three to a four. It's like they're playing the game of writing. And they love to win."
Vantage has licensed or sold the IntelliMetric scoring technology to several other companies for programs such as the ACT and College Board exams. CTB/McGraw-Hill has used the technology in its new Writing Roadmap program [http://www.ctb.com/mktg/roadmap/overview.jsp?CONTENT%3C%3Ecnt_id=10134198673270923&FOLDER%3C%3Efolder_id=9852723696559579], which allows teachers to assign essays in four writing genres or styles—descriptive, expository, narrative, and persuasive.
With Writing Roadmap, students write essays based on the prompts, following directions from the online writing checklist. Both analytic and summary scores are given almost instantaneously so that students can get results and teachers can tailor instruction to demonstrated class needs. Student essays are kept in individual online writing portfolios for later teacher or student access. As with most of the other essay scoring programs discussed here, Writing Roadmap is aligned to state standards through the use of rubrics, etc., which use the most common elements in national and state standards.
Into the Future
The challenge that remains for writing instruction lies in motivating all of our students to write. Greatly improved feedback and computer use in language arts classrooms should motivate some students, but many will continue to fail to see the writing taught in English classes as a relevant part of their lives.
Sometimes I think that, instead of trying to convince students to join our curriculum, the best approach might be to begin our curriculum where many of the students already are—using e-mail, blogging, and working with other informal computer applications.
One problem is that these informal applications are difficult to guide and monitor in school settings. Perhaps the developing AI technologies could help in this regard, supporting teachers in guiding and monitoring these nontraditional applications and helping us show all of our students that good writing is an important part of everyday life. The students could then move to work with more formal writing situations on the way to becoming mature, fluent, and frequent writers.
Charles Doe has been teaching for 34 years, including 20 years as a Title I reading specialist and 5 years as a media specialist. In addition to presenting and writing articles, he has been involved with computers in education for 12 years. He is a long-time reviewer for MultiMedia & Internet@Schools magazine.
Communications to the author may be addressed to Charles Doe, Media Specialist, Hastings Area Schools, 232 W. Grand, Hastings, MI 49058; E-mail: firstname.lastname@example.org.
try it out
A Few Online Essay Scoring Demos