Learning through writing at Boise State University
BSU Writing Center & Writing Across the CurriculumNumbers 102 & 103 - February &March 2000
An effective approach to grading:
Primary Trait Analysis
Though Word Works has in the past published a few pieces that touch on responding to student writing, we have so far said very little about evaluating and grading writing. We have avoided the subject for a number of reasons, the main one involving the uneasy truce teachers must maintain between writing and grading. Many faculty who teach writing or use writing assignments dislike having to figure out grades for student papers: so many papers have such mixtures of strengths and weaknesses they are hard to fit into any grading slot without doing some injustice. Another reason many teachers dislike grading is that grades don't contribute very much to learning. It's true that a grade is a form of feedback, students who take the time to analyze their work when they get it back can learn a lot about why it received the grade it did. But for most students, grades alone are not a very informative way to identify strengths and weaknesses in writing.
One system that is proving to be effective is Primary Trait Analysis (PTA). It is a system of evaluating and grading designed to show students what specific strengths their work exhibits and what weaknesses they need to work on. It is a point system, similar in some ways to other point-based grading systems. The differences are that (1) the point scale is divided into several categories of equal weight, and (2) the points in the scale are linked to specific descriptions of strengths and weaknesses.
PTA is not new. It first became well known from a 1977 article by Richard Lloyd Jones. More recently it has been promoted by Barbara Walvoord and Virginia Johnson Anderson in their 1998 book, Effective Grading. This issue of Word Works is greatly indebted to both these sources, and to a workshop conducted by Walvoord and colleagues from the University of Notre Dame at the 1998 IUPUI Assessment Institute in Indianapolis. The institute was attended by a team sent from BSU to gather information toward developing an assessment program for the Core Curriculum.
Because PTA scales tend to be pretty long, we won't show you a complete scale right here. By the time you got through it you might not want to read the rest of what we have to say. We have appended a couple of sample complete scales on pages 4-8. Here is an abbreviated one, three excerpts from a scale devised to evaluate and grade science reports.
Science report--abbreviated example
The complete scale breaks down the parts of the report thus:
II. Materials and methods
III. Etc. (separate categories for Findings, Discussion, Conclusion, References)
Here is the detailed set of level descriptions from three of the categories. The numbers are points the report can earn in each category, from 4 to 1. The points for all categories are added up and converted to a letter grade.
Introduction A
4 History well researched. Major contributions presented with discrimination and balance. Controversies outlined and weighed.
3 History adequately outlined. Role of major contributions recognized. Relative merit of conflicting opinions somewhat unclear.
2 Historical outline present. Contextual development and relative merit of contributions unrecognized or ragged. Presentation of conflicting ideas absent.
1 Historical outline absent or garbled. Contributions listed as in a diary; consideration of merit absent. Notions of conflicting ideas ignored.
Introduction B
4 Hypothesis clearly recognized or well crafted and elegantly stated in testable form. Hypothesis cleverly embedded in context.
3 Hypothesis recognized or well stated. Contextual connections evident.
2 Hypothesis detectable but may not be stated in testable form. Contextual connections tenuous.
1 Hypothesis undetectable or garbled so as to violate scientific principles. Context absent or ignored.
Materials & Methods B
4 Procedures clear, need no interpretation. Appropriate details present.
3 Procedures easily interpreted. Relevant information dominates.
2 Procedures unclear but interpretable. Irrelevant information interferes.
1 Procedures scrambled. Irrelevant information predominates. Reads more like a bad diary.
If you glance at the two sample scales on pages 4-8, you may feel daunted. It looks like a lot of work, writing out all those descriptions. But the work is not so hard if it's done in small steps; it doesn't all have to be done at once. The sample scales were developed over a period of time, as the instructors gradually clarified their expectations. Here are some hints for making the project doable, without taking up a huge amount of time.
Treat your PTA scale as an ongoing draft, always in need of more work. As you use it you will find various ways to fine-tune (or completely rethink) the scale. In particular;
A PTA scale offers benefits both for students and instructors. Some of the prominent advantages are:
There is no corresponding list of disadvantages. The only disadvantage of PTA scales is that they take a long time to write. But when you consider that they can be developed gradually in easy stages, and put to use when only partially developed, you can see that this is not much of a disadvantage.
PTA scales are useful beyond the individual classroom. Academic departments at some universities use them across all sections of a course or curriculum to assess the overall effectiveness of a program. (Such assessment is distinct, please note, from faculty evaluation. It is always designed so as not to identify individual classes or instructors.)
The BSU freshman composition program is using an assessment strategy related to PTA. Student folders are selected at random, anonymously, across all sections of ENGL 101 and 102. Each year the assessment team looks at just two traits, rotating through the complete list of traits over several years. The Core Curriculum Committee is trying to encourage other BSU departments that offer core courses to set up analogous programs. If your department is interested, please contact Marcia Belcheir, Assessment Coordinator, at x1117, or Rick Leahy, Core Curriculum Committee chair, at x3585.
PTA scales can also be used for other projects besides writing assignments. The scale for the architectural plan beginning on page 7 is an example.
If you've been interested enough to read this far, you might like to look at a couple of complete scales. On the following pages you'll find two sample scales, one from a technical communication class and one from an architecture class.
Sample scale for a technical communication memo
Rhetorical situation; writer-reader relationship; honesty (ethos, pathos)
5-6 The document takes into account who the readers are and how the readers probably feel about the subject. It shows awareness of how the document is going to affect primary and secondary readers. It responds accurately to the requirements of the assignment. Text and graphics are honest; they don't distort facts or withhold important information the writer does not want the reader to know. The subject line is informative, specific, and concise.
3-4 The document is adequate in its relationship to the rhetorical situation. There may be noticeable but not damaging lapses in tone. In some minor way it may not quite answer the assignment. It may have some minor lapse in honesty, perhaps inadvertent. The subject line is adequately informative but not as specific or concise as it could be.
1-2 The document is significantly flawed in its relationship to the rhetorical situation. It may be written in a tone that is inappropriate or even offensive to the audience. It may miss the point of the assignment. It may be obviously dishonest, by misrepresenting or omitting facts, using misleading graphics, or creating a bias for or against some idea. The subject line is too vague or confusing to be informative.
Choice of information and ideas; accuracy, conciseness, comprehensiveness (logos)
5-6 The document contains all the information and analysis the reader needs, and nothing that the reader does not need (taking into account, however, the varying needs of different readers). The information is accurate. The analysis and conclusions are logical and clearly related to the facts. General assertions are backed up with sufficient reasons, evidence, and specific examples. When appropriate, the writer draws creative, imaginative, and inventive conclusions from the facts.
3-4 The document contains adequate information, but readers feel they need to know more. There may be extraneous information, but not enough to cause serious confusion. There may be minor inaccuracies in the information. The analysis and conclusions make sense in light of the facts presented, but they may appear unimaginative; they only point out the obvious. General assertions are mostly backed up with reasons, evidence, and specifics, but not to the reader's complete satisfaction.
1-2 The document may be missing important information, or have so much extraneous material that it confuses the reader. The information may be inaccurate. It may make assertions and reach conclusions that are not warranted by the facts, not supported by specific examples, or not developed with clear reasoning. It may do an inadequate job of drawing conclusions from the facts; in other words, the reader can easily see meaning in the facts that the author failed to point out.
Organization, visual elements; accessibility; professional appearance
5-6 The document design is clear, attractive, and professional. Information and ideas are easily accessible to the reader through predictable organization, advance organizers, paragraph divisions, section headings, etc. Headings and paragraph indentations follow a clear hierarchical scheme. Lists are used where possible but not over-used. The document follows the proper conventions of format. Graphics are clear and placed near the text that refers to them. Pages look balanced and open, with appropriate use of white space. In a case assignment, the writer has effectively rethought and restructured the material into a proper technical organization.
3-4 The document is adequately designed and executed, but there may be a few lapses in the design or the professional appearance. The reader may have some difficulty finding different parts of the document and kinds of information. The hierarchy of headings and paragraph indentations may be somewhat unclear or confusing. Lists are generally used well, though there may be an over- or under- use of them. Graphics are mostly clear and related well to the text, though there may be some problems. Pages look balanced and open for the most part, but they may be somewhat inconsistent. In a case assignment, the writer has for the most part restructured the material, but some information may be placed inappropriately.
1-2 The document design is confusing or looks unprofessional. The reader has serious trouble finding different parts of the document and kinds of information. Advance organizers may not be used where needed; paragraphing may fail to give clear indications of changes in topic; section headings may be missing or misleading. There may be no clear hierarchy of headings or paragraph indentations, or there may be a confusing one. Graphics may be unclear or hard to relate to the text. Pages look crowded or sloppy, or too much white space may make the document look choppy and disunified. In a case assignment, the writer has followed the material as found in the assignment rather than restructuring it into an effective technical organization.
Coherence, clarity, correctness
5-6 The writing style is direct, literal (rather than figurative or metaphoric), and unambiguous. Words are kept simple and are used with precision. Sentences are grammatically correct and stylistically effective, particularly in the use of emphasis points, subordination, parallel structure, and the given-new contract. Punctuation is correct and is used effectively for separation and emphasis. There are no spelling errors. In a case assignment, the writer's own words and sentences are used whenever possible, rather than copying the phrasing of the case directly into the document.
3-4 The writing style is mostly clear but may be occasionally too indirect, figurative, ambiguous, or vague. There may be a few words that are misused or too much like jargon, but they don't interfere significantly with reading. There may be a few sentences that are ineffective because of problems with sentence emphasis, subordination, parallel structure, or the given-new contract. There may be a noticeable number of spelling or sound-alike errors (no more than an average of two per page). In a case assignment, the writer may have composed original sentences for the most part, but there is too much reliance on the language of the case.
1-2 The writing style is too indirect, overly figurative, ambiguous, or vague. Words are inappropriately used, or there may be too much jargon for the intended audience. Sentences have errors that seriously interfere with easy reading. There may be several stylistic flaws in sentence emphasis, subordination, parallel structure, and the given-new contract. Punctuation is haphazard and shows no sense of how punctuation tells readers how to read. There are errors in spelling and confusion of sound-alike words. In a case assignment, the writer lifts whole sentences out of the case instead of composing original ones.
Scoring scale for technical documents
Rhetorical situation; writer-reader relationship; honesty (ethos, pathos)
6 5 4 3 2 1
Choice of information and ideas; accuracy, comprehensiveness (logos)
6 5 4 3 2 1
Organization, visual elements; accessibility; professional appearance
6 5 4 3 2 1
Coherence, clarity, correctness
6 5 4 3 2 1
Total points _______ Grade ________
A = 23-24; A- = 22; B+ = 21; B = 19-20; etc.
Sample scale for an architectural plan
Program: Plan
4 The assigned program is carefully analyzed and developed. The architect has not omitted any portion of the program and has in fact added to the program.
3 The architect provides some insight or depth of understanding to the assigned program. However, the internal logic and character of the work needs to be more clearly established and developed.
2 The development of the program is generalized and lifeless. Mainly surface relationships are provided. The program has not been developed much beyond the level of bubble diagram.
1 The architect communicates no real understanding or development of the assigned program.
Clarity of concept and design objectives
4 The architect's concept is organized and unified and has logical transitions between the urban and intimate scale.
3 The design objective is mainly clear to the viewer because the architect has tried to order his/her objectives. The link between the urban and architectural realms is not fully explained graphically.
2 Although there may be some attempt at presenting design objectives in a thoughtful manner, the work is confused and disjunctive.
1 The project has no discernible concept.
Style
4 The architect demonstrates a quality of imagination and rigor that results in a distinctive project. The work shows a personal exploration.
3 The architect includes refining details, but a portion of the work remains general. The overall composition is pleasing.
2 The architect does not invest himself or herself in the work. The style seems bland, guarded, flat, and not very interesting.
1 The architect demonstrates no recognizable individualistic or historical style.
Development of the small scale; detailed information
4 Character, detail and scale are clearly expressed in plan and section.
3 Some details are thoughtful and vivid. However, the character of the plan and/or section is not developed.
2 Simplistic details are used in a typical way. Repetition of these details distracts from the work. The plan and section together describe a reasonable, believable building, but little information about or attention to detail is developed.
1 Development of the character of the plan and/or section is limited and immature.
Development of the urban scale
4 The development of the urban scale shows a confident control of the project and communicates a clear parti. The work "reads" smoothly from urban scale to the intimate scale. Coherent development at this level makes the project clear and easy to understand.
3 The architect shows some control in the development of an urban parti, and has only a few elements at the urban scale that are awkward or perfunctory.
2 The architect has definite problems with parti: in simplistic terms, the big idea. Most of the urban plan is simplistic in conception, and immature in its development.
1 There is no discernible urban idea. All is perfunctory.
Knowledge of construction
4 There are no obvious errors in construction. The architect shows he/she is familiar with the building materials and their appropriate use.
3 A few errors in construction practices appear in the project, showing the
architect is still learning about the building materials that were chosen. These errors do not substantially detract from the overall impression of the work.
2 Errors or omissions in the use of the chosen building materials are so numerous that they are distracting to the viewer.
1 Errors or omissions in standard building practices are serious enough and frequent enough to interfere with meaning.
Graphic presentation
4 The project is presented in a complete and compelling manner.
3 The project is compelling but incomplete.
2 Required drawings are missing, and the presented work is not legible due to the lightness of the drawings or the haphazard method of presentation.
1 Little effort was invested in the graphic communication of the assigned project.
Lloyd-Jones, Richard. "Primary Trait Scoring." Evaluating Writing. Ed. Charles R. Cooper and Lee Odell. Urbana, IL: NCTE, 1977, 33-66.
Walvoord, Barbara, and others. "Using the Grading Process for Departmental and General Education." 1998 Assessment Institute, Indiana University-Purdue University at Indianapolis, 8-11 November 1998.
Walvoord, Barbara E., and Virginia Johnson
Anderson. Effective Grading: A Tool for
Learning and Assessment. San
Francisco: Jossey-Bass, 1998.