Kapur & Bielaczyc (2012) was very interesting for me to read. The research design was very interesting and well done, what a huge project this was!
The authors could show that their main hypothesis was supported in most, but not all, of the classes. i.e. the students that learned under the PF (productive failure) condition outperformed students that learned under the DI (direct instruction) condition on post-tests. This is despite the fact that only 2 PF groups in the most-capable school were able to solve the problems on their own during the generation phase but most DI students were able to solve them. So the main hypothesis of short-term failure leading to long-term success has some support.
However, there are a couple of things the authors did not discuss which could have affected their results. One major one is that the PF students all worked collaboratively in small groups while all DI students worked individually. This is a huge difference is learning conditions that was not taken into consideration. Could it be the collaborative learning that lead to the eventual success of the PF students?
Specifically, the students in the PF condition were all exposed to multiple RSMs. All groups came up with various strategies for trying to solve the problem, whereas with the DI students, only 1 method (1 RSM) was used. This opens up the question of whether it could be exposure to multiple RSMs that lead to the eventual success of the DF students.
One related question is if the PF students were GIVEN the various RSMs, would they have had the same eventual success? Or is it student generation of the RSMs that leads to success? (I suspect the latter, but this is not addressed in the study.) This is a big factor, because the groups that generated the most RSMs were the ones who were eventually the most successful. If all groups were given the same RSMs, what would be the effect on learning outcomes? The big question for me is, is it exposure to multiple RSMs which explains the eventual success of the PF groups? Is the key variable “exposure to multiple RSMs”, or is it “initial failure”? What is the effect of initial failure? How about initial success instead?
For me, the results can be at least partially explained by the fact that students who collaborated were all exposed to multiple RSMs (most of which failed). They were then taught the canonical RSM, which they all learned and could use well to solve the problems in the post-test. Another question is what learning outcomes would have been if the students had instead been exposed to RSMs which were initially successful? This experimental design was set up so that students would initially fail (2 groups did not, they succeeded). What if it was set up so that students would initially succeed?
I cannot agree that the results of the study are based solely on PF students first experiencing failure. Confounding variables are collaboration (PF) vs. non-collaboration (DI), multiple RSMs (PF) vs. one RSM only (DI), and the absence of the study of the possible effect of successful initial RSMs on eventual success. Nevertheless, I think this is a very interesting study that raises many questions about how we can best support and structure (or unstructure) student learning.