Skills and Knowledge, and Evil Standardized Tests

In a recent op-ed in the New York Times, Susan Engel, the director of the teaching program at Williams College decries the awful state of the reliance of our educational system on standardized tests.  I am very sympathetic to this view, but for different reasons than Engel.  She sees the rote memorization that current standardized tests assess as trivial, and suggests:

Instead, we should come up with assessments that truly measure the qualities of well-educated children: the ability to understand what they read; an interest in using books to gain knowledge; the capacity to know when a problem calls for mathematics and quantification; the agility to move from concrete examples to abstract principles and back again; the ability to think about a situation in several different ways; and a dynamic working knowledge of the society in which they live.

And a response by Jonah Lehrer on his blog Frontal Cortex over at Wired: http://www.wired.com/wiredscience/2010/09/what-are-we-measuring-in-school

Lehrer says that the tests are bad, because knowledge is fleeting, and what really matters are perseverance and diligence.  These are the traits of successful students (highly correlated with grades) and employees, why don't we measure these directly?

I was moved to write a response, because I feel both of these pieces represent misconceptions of the nature of the difference between knowledge,  skills, and traits, and the problems with assessment.

First, I really think that Engel has mostly the right idea, and I am very sympathetic to her criticism.  She seems to accept that testing is inevitable, and that we need some metrics of success in schools.  Further, her critique is not of all testing, but rather of the particular form of our current tests which values convenience and ease of interpretation over what we actually value in our children.  However, I feel that she falls into the trap of separating knowledge from skills, and saying that what we really want from education is skills (true enough) and that we can do this without resorting to teaching boring factual knowledge (untrue).

Lehrer cites his own experience an organic chemistry class, in which the professor noted that students will forget all of the material, but that the class (and grades in the class) was a way to identify those students who had enough grit to stay up late and cram tons of facts into their heads for a limited amount of time.  The class was therefore not just a class on organic chemistry, but rather, a class on "learning how to learn."  Unfortunately for the hopes of many in higher education, cognitive psychologists have found that "learning how to learn" any general thing, is not really possible: skills of close reading, critical thinking, and abstract thought are quite specific to the particular background knowledge of the topic.

In Daniel Willingham’s book “Why Don’t Student’s Like School” he devotes a chapter to the evidence behind his claim that “factual knowledge precedes skill.” We think that knowledge is fleeting, or trivial, but we have that impression because once the knowledge begins to be used, it is thought of as skill. But most of the things we think of as skill are based on a foundation of factual knowledge. Lehrer may think that he simply flushed down everything that he learned in organic chemistry, but his skill as a science writer is informed by some of those facts, whether he knows it or not. Likewise with Engel’s suggestion of the elevation of noble skills rather than trivial memorized facts. The skills she imagines :

1) the ability to understand what they read;
Reading comprehension depends critically on background knowledge of the reader. Efforts to independently assess reading skill inevitably find that those who have more background knowledge in the topic area understand more. See Recht and Leslie (1988) for a study which compared “good” readers and “poor” readers on topics which they had background knowledge or not.  Here is Daniel Willingham on why reading is not a skill.

2) the capacity to know when a problem calls for mathematics and quantification;
Again, this can be surprisingly specific to ones area of expertise and background knowledge.

3) the agility to move from concrete examples to abstract principles and back again;
Again, many studies in cognitive psychology have shown that abstract thinking ability is strikingly specific. Professionals in one domain, which you might think helps them be better “abstract thinkers” or “critical thinkers” are shown to be just merely average when tested on a task outside of their domain which requires abstract thought.

4) the ability to think about a situation in several different ways;
This again is specific on the background knowledge. People are generally limited to relating situations to things they already know. This is limited by background knowledge.

5) and a dynamic working knowledge of the society in which they live.
Here we have knowledge, but not rote memorization, rather, a dynamic working knowledge of society. But again, what makes this knowledge dynamic and working as opposed to static and trivial? For example, what if Engel wanted high school students to be able to reason about race inequity in our current society. Wouldn’t this depend on whether they knew the history of the civil rights movement? Or demographic facts about our country?

But I agree that the emphasis on standardized tests is ill-conceived.  And I agree with Engel's suggestion that we get students interested in using books as a way to gain knowledge.  For me, this is the real tragedy of our current crop of education reform: a totally backward and ham-handed approach to motivation and interest, both from the perspective of the teachers as well as the students.

The current regime of high stakes testing is not a problem because knowledge is trivial, but because constant narrow testing is actually a terrible way to motivate students to get this knowledge. What should we be doing? Getting kids excited about reading. Teaching them interesting content in science, social studies, literature, etc. And yes, improving their vocabulary, and their general background knowledge.  If we did that, I think we would find their test scores magically rising.

Further, we need to recognize that basing their pay on students' test scores is also a terrible way to motivate teachers.  For many teachers (myself included), a primary challenge is to instill as much knowledge, while maintaining motivation and interest.  This is in the context of the fact that our brains were not meant to think (most of our brain power and sophistication is devoted to perception and moving).  One problem with high stakes testing is that a too-strong incentive leads to limited learning, and paradoxically, lower motivation.   Among the first to observe how too large an incentive limits learning was  Edward Tolman, in his classic paper "Cognitive Maps in Rats and Men" (1948).  

If rats are too strongly motivated in their original learning, they find it very difficult to relearn when the original path is no longer correct.

Tolman ends his paper with the following words:

We must, in short, subject our children and ourselves (as the kindly experimenter would his rats) to the optimal conditions of moderate motivation and of an absence of unnecessary frustrations, whenever we put them and ourselves before that great God-given maze which is our human world. I cannot predict whether or not we will be able, or be allowed, to do this; but I can say that, only insofar as we are able and are allowed, have we cause for hope.

Amen.

In my next post, I will discuss some more about which theory of learning and motivation we have chosen instead of Tolman (and the psychologists who have followed him).