Lots of Research, No Evidence

This is a long one, sorry.

I.

In 2014, NCTM published Principles to Actions, containing “eight essential, research-based Mathematics Teaching Practices,” and, no, I have no idea why “Mathematical Teaching Practices” is capitalized.

I found the research support for these practices to be thin. But after recently expressing this opinion, I was pointed to Enhancing Classroom Practice with Research behind Principles to Actions, a research companion to the original. Because of my earlier skepticism, I was particularly interested to see what they cited in Chapter 7, which is all about “productive struggle.”

Behold, the book.

That chapter contains 37 citations. Having read the chapter, what I propose to do here is go through each of those citations and see how they do, or don’t, support the claim, and in this way learn something about how these authors think about research.

II.

We should start by clarifying what “productive struggle” means. Here is an example the book gives:

“When middle school students who know how to find the areas of rectangles and triangles but not of more complex shapes are given the task shown…they will experience struggle because there is no clear path to a solution.”

Teachers should teach like that, not showing students how to dissect a complex shape into parts, but letting them struggle in a search for a solution.

To flesh out the picture a bit, the authors offer three ways that teachers can engineer productive struggle:

• Assign work they can’t do on their own: Assign work that students “may not be able to solve the task independently but can solve … with the assistance of peers or questions from the teacher.”

• Don’t let kids struggle on math that isn’t the focus: They give an example of a kid struggling with division on something like that area problem. Go ahead and help them with division, because that math is not central to the lesson.

• Focus students on sense making: Students should be focused on verbal, conceptual knowledge rather on their procedural knowledge.

That said, the authors claim that students will benefit in five ways:

1. They’ll feel a sense of accomplishment

2. They’ll have stronger knowledge and understanding

3. Their achievement will be high

4. Their achievement will be improved

5. They will attain mastery and long-term retention of the material

With that, let’s turn to those 37 citations, and see which of them provide evidence of these five benefits.

III.

Three of the citations are for popular science books, one of which is Malcolm Gladwell’s Outliers, another is Make It Stick, and the third is Creativity, Inc., a book about Pixar and creativity.

Brown, Peter C., Henry L. Roediger III, and Mark A. McDaniel. Make It Stick. Cambridge, Mass.: Belknap Press of Harvard University Press, 2014.

Catmull, Ed, and Amy Wallace. Creativity, Inc: Overcoming the Unseen Forces That Stand in the Way of True Inspiration. New York: Random House, 2014.

Gladwell, Malcolm. Outliers: The Story of Success. New York: Little Brown and Company, 2008.

Five of the citations are studies or summaries relating to Carol Dweck’s mindset research:

Dweck, Carol S. Mindsets and Math/Science Achievement. New York: Carnegie Corporation of New York Institute for Advanced Study, 2008.

Mindset Works. Brainology Curriculum Guide/or Teachers: Introductory Unit. Mindset Works, 2014. https://www. mindsetworks.com/

Mindset Works. "Do's and Don'ts of Praise." mindsetkit.org. 2015. https://www.mindsetkit.org/topics/praise-process- not-person/dos-donts-of-praise

Mueller, Claudia M., and Carol S. Dweck. "Praise for Intelli- gence Can Undermine Children's Motivation and Performance." Journal ofPersonality and Social Psychology 75, no. I (I998): 33.

Blackwell, Lisa S., Kali H. Trzesniewski, and Carol Sorich Dweck. "Implicit Theories of lntelligence Predict Achievement across an Adolescent Transition: A Longitudinal Study and an Intervention." Child Development 78, no. 1 (2007): 246-63.

Mindset don’t provide evidence for the productive struggle claims, though. They might support the idea that struggle is inevitable in learning and that having a growth mindset is helpful for overcoming those difficulties. What they don’t provide is evidence that it’s important for teachers to engineer moments of struggle for students.

The reason they’re citing Make it Stick is because there is research suggesting that interleaved practice is both more effective and more difficult for students. But this is about a method for practicing skills, which is procedural knowledge, which is not what they’re talking about, so it shouldn’t count as evidence of anything except that, in the broadest possible sense, sometimes difficult things are good for you.

Another five papers are from mathematicians and educational thinkers whose work is either quite old or who are better thought of as philosophers or theorists. They don’t really provide evidence in the contemporary sense:

Vygotsky, Lev. "Interaction between Learning and Development." Readings on the Development ofChildren 23, no. 3 (1978): 34-41.

Dewey, John. Democracy and Education: An Introduction to the Philosophy of Education. New York: Macmillan, 1926.

Dewey, John. The Quest/or Certainty: A Study ofthe Relation of Knowledge and Action. New York: Minton, Balch, and Company, 1929.

Piaget, Jean. Science of Education and the Psychology of the Child. Translated by D. Coltman. New York: Orion Press, 1970.

Polya, George. How to Solve It: A New Aspect of Mathematical Method. Princeton, N.J.: Princeton University Press, 1945.

That’s thirteen. They also cite two NCTM books and the Common Core State Standards:

National Council of Teachers of Mathematics (NCTM). Principles and Standards for School Mathematics. Reston, Va.: NCTM,2000.

- - . Principles to Actions: Ensuring Mathematical Success for All. Reston, Va.: NCTM, 2014.

National Governors Association Center for Best Practices (NGA Center) and Council of Chief State School Officers (CCSSO). Common Core State Standards for Mathematics. Washington, D.C.: NGA Center and CCSSO, 2010. http:// www.corestandards.org.

The Teaching Gap is a very interesting book about cultural differences between American, German, and Japanese schools, and one of the differences it finds is that in the US teachers tend to provide didactic procedural instruction followed by repetitive practice. That’s very interesting, and definitely relevant for this discussion, but it’s not evidence that things are better if you engineer productive struggle:

Stevenson, Harold, and James W. Stigler. The Learning Gap: Why Our Schools Are Failing and What We Can Learn from Japanese and Chinese Education. New York: Touch- stone, 1992.

Stigler, James W., and James Hiebert. The Teaching Gap: Best Ideas from the World's Teachers for Improving Education in the Classroom. New York: Simon and Schuster, 1999.

- - . "Closing the Teaching Gap." Phi Delta Kappan 91, no. 3 (2009): 32-37.

Hiebert, James, and James W. Stigler. "A World of Difference." Journal of Staff Development 25, no. 4 (2004): 10-15.

Lewis, Catherine C. Educating Hearts and Minds: Reflections on Japanese Preschool and Elementary Education. Cambridge, U.K.: Cambridge University Press, 1995.

That brings us to twenty-one. A number of books and papers are cited not for the studies but for quotes and broader theoretical support, things like “effective teachers have high expectations for their students” and “teachers who need to offer support should be careful to maintain the cognitive demand of the task.” None of which is evidence for productive struggle:

Stein, Mary Kay, Margaret S. Smith, Marjorie Henningsen, and Edward A. Silver. Implementing Standards-Based Mathematics Instruction: A Casebook for Professional Development. 2nd ed. New York: Teachers College Press, 2009.

Hattie, John, and Helen Timperley. "The Power of Feedback." Review of Educational Research 77, no. I (2007): 81-112.

Stein, Mary Kay, Barbara W. Grover, and Marjorie Henningsen. "Building Student Capacity for Mathematical Thinking and Reasoning: An Analysis of Mathematical Tasks Used in Reform Classrooms." American Educational Research Journal 33, no. 2 (1996): 455-88.

Kilpatrick, Jeremy, Jane Swafford, and Bradford Findell, eds. Adding It Up: Helping Children Learn Mathematics. Wash- ington, D.C.: National Academy Press, 2001.

Smith, John P., III. "Efficacy and Teaching Mathematics by Telling: A Challenge for Reform." Journal for Research in Mathematics Education 27, no. 4 (1996): 387-402.

Smith, Margaret Schwan. "Reflections on Practice: Redefining Success in Mathematics Teaching and Learning." Mathematics Teaching in the Middle School 5, no. 6 (2000): 378-86.

Cangelosi, James S. Classroom Management Strategies: Gaining and Maintaining Students' Cooperation. Hoboken, N.J.: John Wiley & Sons, 2007.

Franke, Megan Loef, Thomas Carpenter, Elizabeth Fennema, Ellen Ansell, and Jeannie Behrend. "Understanding Teachers' Self-Sustaining, Generative Change in the Context of Professional Development." Teaching and Teacher Education 14, no. 1 (1998): 67-80.

Weinert, Franz E., Friedrich-W. Schrader, and Andreas Helmke. "Quality of Instruction and Achievement Outcomes." International Journal of Educational Research 13, no. 8 (1989): 895-914.

Harris, Douglas N., and Tim R. Sass. "Teacher Training, Teacher Quality and Student Achievement." Journal ofPublic Economics 95, no. 7 (2011): 798-812.

Wayne, Andrew J., and Peter Youngs. "Teacher Characteristics and Student Achievement Gains: A Review." Review of Educational Research 73, no. I (2003): 89-122.

Which leaves us with just five studies, two of which are case studies about teacher discourse moves that don’t provide evidence for our claims:

Carter, Susan. "Disequilibrium and Questioning in the Primary Classroom: Establishing Routines That Help Students Learn." Teaching Children Mathematics 15, no. 3 (2008): 134-37.

Williams, Steven R., and Juliet A. Baxter. "Dilemmas of Discourse-Oriented Teaching in One Middle School Mathematics Classroom." Elementary School Journal 97, no. I (1996): 21-38. 

And now, finally, we get to the heart of things. Warshauer’s 2011 dissertation is where the term “productive struggle” is first used. It’s built on Hiebert & Grouws and Hiebert & Wearne, who instead called it “constructive struggle.”

Warshauer, Hiroko Kawaguchi. "The Role of Productive Struggle in Teaching and Learning Middle School Mathematics." PhD diss., University of Texas at Austin, 2011.

Hiebert, James, and Douglas A. Grouws. "The Effects of Classroom Mathematics Teaching on Students' Learning." In Second Handbook of Research on Mathematics Teaching and Learning, edited by Frank K. Lester, Jr., pp. 371-404. Charlotte, N.C.: Information Age, and Reston, Va: National Council of Teachers of Mathematics, 2007.

Hiebert, James, and Diana Wearne. "Developing Understanding through Problem Solving." In Teaching Mathematics through Problem Solving: Grades 6-12, edited by Harold L. Schoen and Randall I. Charles, pp. 3-13. Reston, Va: National Council of Teachers of Mathematics, 2003.

If there’s evidence for productive struggle, it has to be there.

IV.

Warshauer’s dissertation is not a study of the efficacy of productive struggle for producing any of those five things. But I really like it! She’s doing deep observation of teacher/student interactions and trying to impose some sort of useful structure on it. This is the sort of qualitative work that I love, the attempt to make sense of the huge variety of interactions that happen in a math classroom. It’s not evidence of any of the stuff we’re looking for here, but it’s still interesting.

She cites those other two papers:

“Some mathematics educators, researchers, and theoreticians, however, have written about aspects of student struggle as potentially beneficial and promising toward learning mathematics with understanding (Hiebert & Grouws, 2007; Hiebert & Wearne, 2003)”

We’re at the end of the line, folks. It’s now or never.

The 2007 citation is of a chapter in an NCTM research handbook, “The Effects of Classroom Mathematics Teaching on Students’ Learning.” I am sorry to say that it is, again, not a study. However, it does indeed claim that struggle has benefits.

So, what do they say?:

Although little, if any, research has tried to isolate and test the effects of struggle on the development of students’ conceptual understandings, many findings suggest that some form of struggle is a key ingredient in students’ conceptual learning.

And I’ll stop here. There is little, if any, research that supports the benefits of teaching for productive struggle.

V.

Research is authority. At least that’s what a lot of people want from it. We want research to tell us how the world is, what you can believe and what you can’t. “Studies show,” carries weight in a conversation—people are more likely to disagree with your interpretation of a study than to dismiss it out of hand. At the end of the day, we’re a society that pays a ton of people to just try to figure stuff out and write about it. Even when we don’t really know how they’re doing it.

While this chapter didn’t provide any evidence in support of its claims, it does provide evidence that many people support its claims.

And I don’t think that’s a crazy thing to do! I think a lot of these education publications basically use citations to show that the perspective of the author is supported by others, and this really does have some epistemic value. Wouldn’t you like this essay a little bit more if 37 people with PhDs said I was on the right track?

So I don’t think it’s crazy, and a lot of it is actually helpful for figuring out who you are, who you read, where you’re coming from and what it represents. But it’s not quite the same thing as a study that (say) compares lessons that were built for productive struggle versus those that weren’t, is it? And that study is just not there.

I think this line is blurred across not just education but society at large. It’s a fundamental question: are you saying “studies show…” to cite evidence or muster support? Both are reasonable! But which one is it? And which one should we care about more? And I think the answer is: evidence.