Tuesday, September 21, 2010

Skills and Knowledge, and Evil Standardized Tests

In a recent op-ed in the New York Times, Susan Engel, the director of the teaching program at Williams College decries the awful state of the reliance of our educational system on standardized tests.  I am very sympathetic to this view, but for different reasons than Engel.  She sees the rote memorization that current standardized tests assess as trivial, and suggests:
Instead, we should come up with assessments that truly measure the qualities of well-educated children: the ability to understand what they read; an interest in using books to gain knowledge; the capacity to know when a problem calls for mathematics and quantification; the agility to move from concrete examples to abstract principles and back again; the ability to think about a situation in several different ways; and a dynamic working knowledge of the society in which they live.
And a response by Jonah Lehrer on his blog Frontal Cortex over at Wired: http://www.wired.com/wiredscience/2010/09/what-are-we-measuring-in-school
Lehrer says that the tests are bad, because knowledge is fleeting, and what really matters are perseverance and diligence.  These are the traits of successful students (highly correlated with grades) and employees, why don't we measure these directly?
I was moved to write a response, because I feel both of these pieces represent misconceptions of the nature of the difference between knowledge,  skills, and traits, and the problems with assessment.
First, I really think that Engel has mostly the right idea, and I am very sympathetic to her criticism.  She seems to accept that testing is inevitable, and that we need some metrics of success in schools.  Further, her critique is not of all testing, but rather of the particular form of our current tests which values convenience and ease of interpretation over what we actually value in our children.  However, I feel that she falls into the trap of separating knowledge from skills, and saying that what we really want from education is skills (true enough) and that we can do this without resorting to teaching boring factual knowledge (untrue).
Lehrer cites his own experience an organic chemistry class, in which the professor noted that students will forget all of the material, but that the class (and grades in the class) was a way to identify those students who had enough grit to stay up late and cram tons of facts into their heads for a limited amount of time.  The class was therefore not just a class on organic chemistry, but rather, a class on "learning how to learn."  Unfortunately for the hopes of many in higher education, cognitive psychologists have found that "learning how to learn" any general thing, is not really possible: skills of close reading, critical thinking, and abstract thought are quite specific to the particular background knowledge of the topic.
In Daniel Willingham’s book “Why Don’t Student’s Like School” he devotes a chapter to the evidence behind his claim that “factual knowledge precedes skill.” We think that knowledge is fleeting, or trivial, but we have that impression because once the knowledge begins to be used, it is thought of as skill. But most of the things we think of as skill are based on a foundation of factual knowledge. Lehrer may think that he simply flushed down everything that he learned in organic chemistry, but his skill as a science writer is informed by some of those facts, whether he knows it or not. Likewise with Engel’s suggestion of the elevation of noble skills rather than trivial memorized facts. The skills she imagines :
1) the ability to understand what they read;
Reading comprehension depends critically on background knowledge of the reader. Efforts to independently assess reading skill inevitably find that those who have more background knowledge in the topic area understand more. See Recht and Leslie (1988) for a study which compared “good” readers and “poor” readers on topics which they had background knowledge or not.  Here is Daniel Willingham on why
reading is not a skill.
2) the capacity to know when a problem calls for mathematics and quantification;
Again, this can be surprisingly specific to ones area of expertise and background knowledge.
3) the agility to move from concrete examples to abstract principles and back again;
Again, many studies in cognitive psychology have shown that abstract thinking ability is strikingly specific. Professionals in one domain, which you might think helps them be better “abstract thinkers” or “critical thinkers” are shown to be just merely average when tested on a task outside of their domain which requires abstract thought.
4) the ability to think about a situation in several different ways;
This again is specific on the background knowledge. People are generally limited to relating situations to things they already know. This is limited by background knowledge.
5) and a dynamic working knowledge of the society in which they live.
Here we have knowledge, but not rote memorization, rather, a dynamic working knowledge of society. But again, what makes this knowledge dynamic and working as opposed to static and trivial? For example, what if Engel wanted high school students to be able to reason about race inequity in our current society. Wouldn’t this depend on whether they knew the history of the civil rights movement? Or demographic facts about our country?
But I agree that the emphasis on standardized tests is ill-conceived.  And I agree with Engel's suggestion that we get students interested in using books as a way to gain knowledge.  For me, this is the real tragedy of our current crop of education reform: a totally backward and ham-handed approach to motivation and interest, both from the perspective of the teachers as well as the students.
The current regime of high stakes testing is not a problem because knowledge is trivial, but because constant narrow testing is actually a terrible way to motivate students to get this knowledge. What should we be doing? Getting kids excited about reading. Teaching them interesting content in science, social studies, literature, etc. And yes, improving their vocabulary, and their general background knowledge.  If we did that, I think we would find their test scores magically rising.
Further, we need to recognize that basing their pay on students' test scores is also a terrible way to motivate teachers.  For many teachers (myself included), a primary challenge is to instill as much knowledge, while maintaining motivation and interest.  This is in the context of the fact that our brains were not meant to think (most of our brain power and sophistication is devoted to perception and moving).  One problem with high stakes testing is that a too-strong incentive leads to limited learning, and paradoxically, lower motivation.   Among the first to observe how too large an incentive limits learning was  Edward Tolman, in his classic paper "Cognitive Maps in Rats and Men" (1948).  
If rats are too strongly motivated in their original learning, they find it very difficult to relearn when the original path is no longer correct.
Tolman ends his paper with the following words:
We must, in short, subject our children and ourselves (as the kindly experimenter would his rats) to the optimal conditions of moderate motivation and of an absence of unnecessary frustrations, whenever we put them and ourselves before that great God-given maze which is our human world. I cannot predict whether or not we will be able, or be allowed, to do this; but I can say that, only insofar as we are able and are allowed, have we cause for hope.
In my next post, I will discuss some more about which theory of learning and motivation we have chosen instead of Tolman (and the psychologists who have followed him).


The Triscuit's Dad said...

Interesting post, Cedar, and I think right on with your critique. The question I'm left with: what exactly do you think we should do to assess? You suggest that if we focus on motivating with interesting content and improving skills & knowledge, test scores will "magically rise". But what measurements should we use? For example, there's an "active learning" lit that rejects a lot of the assumptions of the assessment crowd. I focus on writing myself, as a kind of uber-skill that allows students to retain and build on their knowledge, but is that practical?

Cedar said...

Hey Rich! Thanks for reading.
Yeah, I guess that is the question, isn't it? What do we test?

First, what I would like to see is an expansion of what we consider valuable in education. Because reading and math are considered "core" subjects, they are the ones that get tested, they are the ones that determine which schools are "failing" etc. So, first thing, I would like to see things get more complex, not simpler, by adding more content. Not necessarily more skills, but more interesting content, thereby improving knowledge. My point is that improving knowledge is teaching skills. This would make "reading" scores go up.
Second, in the content areas that we expand (history, social studies, politics/civics, science, etc) we carefully consider what we are teaching students, but not run away because it is controversial politically. Choosing a canon of, for example, American history for every elementary student to learn becomes really controversial very fast. But the point of a lot of the cognitive psychology research is that you need some sort of body of facts before you can do reasoning. I don't think it totally vital that you get the canon exactly right, but you need to acknowledge the role of background knowledge.
My impression of what little I know of the active learning literature (at the K-12 level) is that it tends to either mistake what the kids are actually learning (my kids made pulleys in their preschool at Mills, and I really doubt they learned anything about the physics, despite the grand designs of the project - not that they didn't have fun, I was totally satisfied with that experience for them), or it doesn't take into account all of the background knowledge that kids who end up in active learning scenarios in this country tend to have. For example, my kids might do well in a something like a montessori pure exploration sort of curriculum, but part of the reason for that is that they get a lot of background knowledge from books that I read them, or books that they read at home. I guess what I feel about active learning is that what seems to be kids creating their own new knowledge can actually be reliant on previous background knowledge. I for one have had less success with active learning situations with students who know relatively less.

The question of writing is a great one that I grapple with frequently myself. Is writing a skill? Is it worth it to spend time on writing skills, grammar and academic style when, for example, a student has a limited vocabulary, or uses certain words wrong? I think teaching writing is totally worth it, but I think that you should also consider that when you teach the content of your class, you are also teaching writing. Writing is a great activity, because I think it makes students actively use the concepts, and have to struggle with the way that they logically fit together. It also (if done well) encourages meta-thinking and perspective- taking, which is good for learning too. I think getting practice writing is well worth it, and an excellent skill to work on, but I think we should also remind ourselves that what often looks like writing skill is also dependent on deep background knowledge in the content area, as well as practice.

Cedar said...

I should add that I think learning should be more active in general, but that I don't think that necessarily excludes use of factual knowledge.

Georgina said...

Teachers should have a voice in the education policy debate currently taking place on a state and national level. Through the VIVA Project -- http://vivateachers.org -- teachers can share and collaborate on their own ideas about what should be done about education reform.

Visit the website and make teachers voices heard. It's that simple and that incredibly important.