This course will become read-only in the near future. Tell us at if that is a problem.

Getting started (target date: Jan 20)

To get started in this course, send an email message to Greg Wilson to say 'hi'; he'll send you back these three chapters from the book Making Software:

  1. Mark Guzdial: "Why Is It So Hard to Learn to Program?"
  2. Michele A. Whitecraft and Wendy M. Williams: "Why Aren't More Women in Computer Science?"
  3. Andrew Begel and Beth Simon: "Novice Professionals: Recent Graduates in a First Software Engineering Job"

Please also have a look at this post on the Software Carpentry blog, which compares Greg's attempt to teach online with these research-based best practices. For your first task, write a blog post of your own that compares what you do (or what's been done to you :-) with those best practices, then add a comment below with a link to your blog post.



Links (extracted from people's comments)

  1. Comments by Greg Wilson are in italics with the initials 'GW'; please add lots of your own.
  2. Some ideas for things people could tackle as their third task are gathered together at the bottom; please add more.

Common Threads

One theme came up repeatedly is that these recommendations are made with a traditional (battery-farmed) learning model in mind, and it's hard sometimes to relate them to free-range learning. In particular, we typically have brief or intermittent contact with our learners through one-day workshops, drop-in lessons spaced over a few weeks, or self-paced online learning. GW hasn't found any work (yet) on what teaching and learning practices work best in those situations—who'd like to go looking for some as their third task in this course?

Greg Wilson

  • Background
    • Teaching scientists how to program (and do other things with computers) for 14 years.
    • Mostly crash courses (9-5, five days straight)
  • Space learning over time.
    • Nope: it's mostly been contract teaching, so we cram.
    • Can see that people's brains are full around the middle of day 3.
    • Shifting to two-day workshops + six weeks of online follow-up.
  • Interleave worked examples with problems.
    • Don't do it.
    • Doubt students would do the exercises: most are surfing for solutions to specific problems of their own rather than diving deep.
  • Use graphics.
    • Yes, in slides.
  • Connect concrete and abstract.
    • Try to, but no idea how to tell if we succeed.
  • Use pre-questions to introduce a new topic.
    • Nope. On reflection, I think most learners would find this frustrating.
  • Use quizzes to re-expose students to key content.
    • Don't; should; doubt students would do them.
  • Teach students to use delayed judgments to identify content that needs further study.
    • Can't.
  • Use quizzes to identify content that needs to be learned.
    • Don't.
  • Ask deep explanatory questions.
    • Don't.

Terri Ko

  • Background
    • Didn't absorb as much when first using a computer for note-taking—had to re-train herself to read & comprehend what she was typing.
    • Has taught in university classrooms.
  • Space learning over time.
    • Mentees pretty much "learn and do".
    • Don't come back to subjects much (that she notices).
  • Interleave worked examples with problems.
    • Yes in the classroom, not so much as a mentor.
    • Tends to throw Google Summer of Code students in at the deep end.
  • Use graphics.
    • Classroom yes; almost never as a mentor.
    • Web doesn't currently make this easy.
  • Connect concrete and abstract.
    • Try, but it's hard.
    • Game dev students seem best at it—not sure what conclusion to draw.
  • Use pre-questions to introduce a new topic.
    • Rarely in classroom, occasionally when mentoring (to get an idea of learners' background knowledge).
  • Use quizzes to re-expose students to key content.
    • Yes in class, never with mentees (but do ask them to write up learning summaries periodically).
  • Teach students to use delayed judgments to identify content that needs further study.
    • Nope (and some classroom learners seem upset by the idea).
  • Use quizzes to identify content that needs to be learned.
    • Yes in the classroom
      • GW: is it used for assessment, or to identify content that needs to be learned?
    • No with free-range learners.
  • Ask deep explanatory questions.
    • Occasionally in the classroom, but there it's mostly the students doing the asking.
  • Reflections
    • Many of these ideas would feel forced outside the classroom.
    • But gamification does seem to work (with some people, in some contexts).
    • EPW: gamificaiton has worked to keep me engaged in things online and I've often wondered if Software Carpentry couldn't improve on "interleaving" and "rexposing through quizes" by having a weekly problem sent out to a list with an online system for checking the answer and recording how many a user has answered successfully like Project Euler (

Neil Chue Hong

  • Background
    • Teaching scientists and engineers (typically at the graduate or post-grad level).
  • Space learning over time.
    • Learners are typically working at 200% of capacity, so there's usually no time for training at all, much less for revisiting things.
  • Interleave worked examples with problems.
    • Always.
  • Use graphics.
    • Lots (though coming up with a good visualization is often hard).
    • JP: Using flow charts to add a visual element is a great idea!
  • Connect concrete and abstract.
    • Skipped.
  • Use quizzing to promote learning.
    • Yes.
  • Help students allocate study time efficiently.
    • As above, this is very hard.
    • Helps to take learners out of their usual environment.
  • Ask deep explanatory questions.
    • Not clear what the "deep questions" are in software development.
    • GW: would you like to work with other course participants to try to compile some for your third task?
  • Reflections
    • Would like to try using quizzes more.
    • Main focus has to be on finding time.

Mr Steve

  • Space learning over time.
    • Not good at this.
  • Interleave worked examples with problems.
    • Have thought about giving students examples to read, but have rarely done it.
    • Instead, ask students who are writing good code to share their work with others.
    • Mentions "40 mathematical shapes" problem (turtle geometry).
    • Also thinking about Etoys Challenge: give students a fixed set of tiles for building programs, ask them to solve a particular task.
    • Quotes Minsky: "You don't really know something unless you know it multiple ways."
  • Use graphics.
    • Yup (though more with math than programming).
  • Connect concrete and abstract.
    • Can only think of one example.

Luis Pedro Coelho

  • Background
    • Teaching programming to scientists
  • Space learning over time.
    • Did well with Python programming
    • Poorly with other things like version control—only came up once.
  • Interleave worked examples with problems.
    • Did some (mostly reviewing homework exercises in class).
  • Use graphics.
    • Not much.
  • Connect concrete and abstract.
    • Tried to—had both a lecture and a practical class, and tried to focus more on concepts in the former and examples in the latter.
      • GW: But what the recommendations are suggesting is "make the connections explicitly and frequently".
  • Use quizzing to promote learning.
    • Multiple choice homework every week, ungraded in-class quiz by show of hands at the end of week 1.
    • Thinks it worked well.
    • GW: But was this assessment, or to aid learning?
  • Help students allocate study time efficiently.
    • Nope.
  • Ask deep explanatory questions.
    • Refrained from doing this—was focusing on programming skill, rather than computational thinking.

Leopoldo Teixeira

  • Background
    • Interested in teaching at the NGO where he volunteers.
    • Not much experience teaching, so will discuss experience as a learner.
  • Space learning over time.
    • Rarely done in class, but did it on his own before exams.
  • Interleave worked examples with problems.
    • Used in most programming classes he took, but sometimes there wasn't time to solve the problems in class.
    • And students often skip classes devoted to problem solving.
    • Echoes GW's experience: problem-solving tutorials at U of Toronto are mostly empty.
    • EPW: I combine presentation of material and problem-solving together during the same periods and have typically had near %100 attendence.
  • Use graphics.
    • Agrees with the recommendation, but has seen a lot of bad slides in his life :-)
  • Connect concrete and abstract.
    • His research group recommends doing a concrete example before introducing the abstract concept.
    • Doesn't comment on how well this has worked for him.
  • Use quizzing to promote learning.
    • Hasn't seen it at all.
  • Help students allocate study time efficiently.
    • Not in general, though some teachers did help him specifically.
    • But as per #2, students often don't seem interested.
  • Ask deep explanatory questions.
    • Has been asked to reflect on the big picture
    • GW: Not clear if this is the same as instructor explicitly asking questions that required "deep answers".


  • Background
    • Thoughts as both learner and facilitator.
  • Space learning over time.
    • Difficult in free-range setting where students can drop in and out.
    • Remind them using RSS, etc.?
    • Has noticed that websites aimed at helping people learn new life skills often use reminders in various forms.
  • Interleave worked examples with problems.
    • Can offer examples, but no guarantee people will work through them.
  • Use graphics.
    • As a learner, usually prefer text, but want visuals to help with difficult content.
  • Connect concrete and abstract.
    • Seems straightforward.
    • GW: do you regularly do this, and/or has it regularly been done for you by your instructors?
  • Use quizzing to promote learning.
    • Difficult in a free-range setting.
  • Help students allocate study time efficiently.
    • Haven't seen any good suggestions to ensuring that students follow up.
  • Ask deep explanatory questions.
    • Can be asked, but no way to ensure students follow up.
  • Reflections
    • Facilitators' inability to require students to do something in free-range setting undermines many of these recommendations.

Julie Pichon

  • Background
    • Wondered if "free-range" specifically means "online".
      • GW answered "No, but many people doing free-range learning now are online."
    • Teaches sporadically as a volunteer, usually to groups of 6-10.
  • Space learning over time.
    • Difficult in a 5-6 week course, but most concepts build on each other.
  • Interleave worked examples with problems.
    • Regularly shows students short working programs, ask them to figure out what they do.
    • Is this review rather than worked examples?
    • Thinks alternating is important (and thinks she should go into more detail than she does).
  • Use graphics.
    • Doesn't really do this—is usually projecting code.
    • GW: which raises an interesting question: does the visual nature of programming environments like Scratch help learning as much as diagrams?
  • Connect concrete and abstract.
    • Not doing, or not doing well—students have trouble re-using ideas in new contexts.
    • EPW: I like this way of evaluating whether you're being successful with this or not. I do it a lot in class, but now that you mention this criteria for success I'd say that 1/3 of my students have a very difficult time re-using ideas in new contexts, which suggests that I definitely have more work to do.
  • Use pre-questions to introduce a new topic.
    • Doesn't do this.
    • Wonders if previewing an unknown programming concept would lead to confusion.
  • Use quizzes to re-expose students to key content.
    • Doesn't do this either (though asking students to figure out what small programs do might count as quizzes?)
  • Teach students to use delayed judgments to identify content that needs further study.
    • Nope.
  • Use quizzes to identify content that needs to be learned.
    • Doesn't do this—considering that most students don't take time to do homework, unlikely it would work.
  • Ask deep explanatory questions.
    • Only does this a little bit.

Andrew Cox

  • Space learning over time.
    • Hadn't considered doing this, but many skills build on each other.
    • Thinks it's often enough to know that a particular tool or library exists (then go find the docs).
    • Notes that "...few studies have examined acquisition of complex bodies of structured information."
  • Interleave worked examples with problems.
    • Points out that one of the problems with learning to program is knowing if your code is well-formed, and thinks examples help with this.
  • Use graphics.
    • Speculates that videos may have the same beneficial effect as graphical diagrams.
  • Connect concrete and abstract.
    • Doesn't really know how to connect this.
  • Use quizzing to promote learning.
    • Likes the idea, will give it a try
    • But how do we validate that the students' work is correct?
    • Just checking output isn't good enough.
  • Help students allocate study time efficiently.
    • "...the key takeaway for me."
    • Likes the idea of having students rate how much they know about the question before answering it.
  • Ask deep explanatory questions.
    • Thinks that having students write blog posts answering deep questions will work.
  • Reflections
    • Do classroom methods apply to learning software development? Absolutely.
    • How does learning programming differ from learning history?
      • Most problems are unique.
      • Wide range of correct answers.
      • Need to be aware of a broad range of technologies as well as fundamentals.
      • GW: I actually think most historians would claim all these are true of their field :-)
    • Hardest thing about online learning is validating what the student has actually done or learned.
    • Andrew's OK with this: it's up to them to learn.

Duane Griffin

  • Quotes Greg Wilson's observations ("students won't revisit material" and "no practical way to assess")
  • Thinks student motivation is the key to the first.
  • Thinks peer assessment might help with the second (mentions "a suitably modded MeetBot")
  • On the teaching side, has always used unstructured mentoring.

Ethan White

  • Background
    • "Figuring out how to teach well as a professor at a research university is largely a self-study affair."
    • Currently teaching programming and data management to biologists.
    • Read stuff in a textbook or watch some online videos before class.
    • 5-10 minutes of review.
    • 45 minutes on exercises.
  • Space learning over time.
    • Doing OK, but not systematic.
  • Interleave worked examples with problems.
    • Doing well.
    • Using Online Python Tutor.
  • Use graphics.
    • Yes, and Online Python Tutor does graphical representations of running programs.
  • Connect concrete and abstract.
    • Yes.
  • Use quizzing to promote learning.
    • Not convinced this makes sense in this context.
    • Does have one mid-way quiz intended to provide delayed re-exposure.
    • EPW: To be clear, I'm not convinced that using pre-quizzing to introduce new topics makes sense in this context. I think quizzing to re-expose students is a good idea and I'd like to try doing more of it than the single mid-way quiz.
  • Help students allocate study time efficiently.
    • Yes, mostly for students who ask questions.
    • "By looking at their code and talking to them it typically becomes clear where the 'illusion of knowing' is creeping in and causing them problems and I think I do a fairly good job of breaking that cycle and helping them focus on what they still need to learn."
  • Ask deep explanatory questions.
    • Gives students a large individual project as a capstone.
    • GW: This is not quite the same as explicitly asking questions that require deep explanatory answers—I wonder if it has the same effect?
    • EPW: Yes, I agree. I guess like Luis Pedro Coelho my feeling is that the concept of "deep explanatory questions" is a little strange in a skills course. As such, the analog for me of being pushed to think deeply about a concept is being pushed to tackle problems that: 1) aren't defined for you; and 2) are more complex than any sort of homework problem.

Heather Payne

  • Background
    • Teaching people (primarily women, and soon girls) how to program in a free-range setting.
  • Space learning over time.
    • Run workshops twice a month, priced accessibly, so that people can come frequently.
    • Hit the same fundamental ideas in different contexts, which sort of meets this criterion.
  • Interleave worked examples with problems.
    • Yes: alternate 15-minute "theory" explanations with 15-30 minute practical work.
  • Use graphics.
    • We have slides.
  • Reflections
    • Think it's more important to teach in a way that inspires passion.
    • Use an inspiring space and build a community around what you're doing.
    • GW: I don't think anyone would argue for the contrary :-)
    • "I wonder what would happen if we spent more time thinking about the things that are often ignored: things like people and space and community and fun and creativity and a-ha moments."
    • GW: would you like to go look for an answer to that question as your third task, and share your findings with us?

Gregory Brown

  • Background
    • Mendicant U's courses are entirely project-based and discourse-oriented.
    • Expect 15-20 hr/week for 3 weeks.
  • Space learning over time.
    • Doesn't really fit MU's format.
    • Running follow-up courses probably isn't practical.
  • Interleave worked examples with problems.
    • Hard to provide those with open-ended projects.
    • "...many of the problems students are trying to solve fall outside of the nicely packaged set of examples that we've been compiling over time. This is to some extent the nature of the course..."
  • Use graphics.
    • MU courses are entirely text (email, IRC, etc.)—would like to improve on this.
    • Have to consider limited/intermittent connectivity, etc.
  • Connect concrete and abstract.
    • We do it into both directions.
    • Would be happy to share thoughts.
  • Use pre-questions to introduce a new topic.
    • "We provide a review of entrance exam problems for each of our students, and through that process, we do get an opportunity to talk with students about things...that will resurface during our sessions. However, our entrance exams aren't especially representative of the format and contents of our courses."
  • Use quizzes to re-expose students to key content.
    • Have informal reviews rather than formal quizzes.
    • Students can always revisit old reviews, but even over three weeks, it seems like students are only able to keep track of a limited amount of information.
  • Teach students to use delayed judgments to identify content that needs further study.
    • Hard to do (largely as a result of the compressed timeline).
  • Use quizzes to identify content that needs to be learned.
    • "To the extent that we don't use 'tests''s because we have the time and resources to do even better...The only discomfort I have with that claim is that I haven't yet found a way to evaluate/measure whether our approach is objectively better than formal testing for the goals that we are trying to accomplish."
  • Ask deep explanatory questions.
    • This is the whole basis of our approach.
  • Reflections
    • Hard to apply IES recommendations because our philosophy is so different.
    • Hope to find other ideas through this course.

Eric Gric

  • Limited, unsystematic experience as teacher (one-on-one or one-on-few)
  • Discusses the mentoring model at Mendicant University.
  • "Course completion rate has increased as we have more closely integrated the entrance exam and pre-course mentoring with the core course."
  • Difficulty applying many recommendations because of one-off nature of Mendicant encounters.
  • Interleaving worked examples and problem-solving exercises fits his intutions.


  • A university student, developing challenges for the Trisfera P2PU course.
  • Reflections based primarily on experience as a learner.


  • Is providing an after-school webmaking class
  • Background is web development, not education
  • Space learning over time: had 13 weeks with about 1.5 hours week
  • Interleave worked examples: "Our students were presented with a short presentation at the beginning of every sessions and were asked to find examples of what they were shown."
  • Graphics: not much.
  • Quizzing: "I noticed that the students would warm up and be ready for new concepts once they spent 10 minutes trying to answer something on their own."
  • Connect abstract and concrete: related concepts to real life (seemed most appropriate with high school students).
  • Ask deep explanatory questions: again, relate to real life.

Ideas for Task #3

  • What teaching and learning practices have been shown to work well in our free-range learning contexts?
  • What are the "deep questions" in programming that we should be raising and answering?
  • What effect does inspiring passion, a cool environment, etc., have on what and how much people actually learn?
  • Please add more here.

Task Discussion

  • Kerri   22 januari 2012 13:41

    Here are my thoughts on the IES reccomendations and how they have or could be applied to the afterschool program I am involved with and its possible expansion:

  • Rogelio Moreno   21 januari 2012 18:30

    I'm pretty late, but I thought "better late than never". So here is my blog post. I also found interesting the idea that perhaps, sometimes, it is not that the student is not able to learn, but that the method the teacher is using is not adequate for that student; even though the teacher believes that his way of explaning is crystal clear.

  • Luis Pedro Coelho   20 januari 2012 09:03

    My contribution to this discussion focuses on what I did for my Programing for Scientists class.

  • Duane Griffin   20 januari 2012 07:05

    I've posted on my experiences teaching and learning, meagre though they are.

  • Heather Payne   20 januari 2012 02:03

    2:02 am, and I just finished my write up. Took a different route, probably. Enjoy! Happy to hear everyone's thoughts!

  • Andrew Cox   19 januari 2012 17:38

    I am working on a web application SkillBonsai and have posted my thoughts in that context: Do Research-based Study Methods Apply to Learning Webcraft?

    I've been interested in learning and skill acquisition and the best way to rapidly learn a new skill or to improve an existing skill for the last couple years. I posted my thoughts around SkillBonsai back in April.

    I've read a number of books on the brain and skill mastery, but so far nothing on research related to formal education, so I'm very eager to discuss the experiences of those in the group.

  • Eric G   19 januari 2012 14:28

    I posted my thoughts about the IES recommendations in relationship to Mendicant University's mentoring program here.

    Like Ethan and Terri, I found it somewhat difficult to extend the recommendations to our students, but thought it was a useful exercise especially in terms of how to teach beginners.

  • Ethan White   19 januari 2012 07:56

    I've posted about my experience teaching programming related skills in the classroom (graduate/upper-level undergraduate classes teaching biologists how to program and use databases) and how it fits in with the IES report on my blog.

    While I think I'm doing an OK job of following the recommendations in the classroom I'm not at all clear how to extend a lot of these things to free-range learners; both online and for local folks who don't have the time to take a full class.

  • Neil Chue Hong   19 januari 2012 07:01

    I've posted some of my thoughts from our experience of teaching more "experienced" students (PhD up to professors) here:

  • Leopoldo Teixeira   19 januari 2012 05:42

    I have posted my thoughts about the IES report and my experience as a student at

  • Terri   19 januari 2012 00:39

    I've blogged my thoughts about how the IES report applies to my teaching here:

    In short, though, I felt throughout like the IES report was a great look at what works in the traditional classroom, but some of that stuff seems unlikely to work the same way when students aren't forced to sit in a room X hours a day, and with free-range learners some of it seems like it would be a turn-off since they're trying to get away from a traditional classroom experience.  That said, though, some of the underlying ideas are things I could clearly do *better* with my free-range students if I knew to seize upon / set up opportunities where they wouldn't seem forced.

    I wrote what I said without reading everyone else's posts, so now I'm looking fowards on catching up to what the rest of you think!

  • Gregory Brown   15 januari 2012 18:27

    I've posted my reflections on how my teaching practices relate to IES Report as well. Because Mendicant University is such a messy and weird structure (at least when compared to traditional classroom learning), it was a bit of a stretch to apply these concepts to what I've been doing.

    That said, the process of forming this mapping gave me a whole lot to think about, and broke free a lot of thoughts and questions that I think will be relevant to this course. While much of my post ended up being mostly noodling about ideas for how to improve MU, I think there are some general concepts here that we can discuss together for sure.

  • Julie Pichon   15 januari 2012 16:51

    I also blogged my thoughts regarding the report's recommendations, and what I'm doing and not doing (...though many "things from the report I'd like to try" ended up making their way in there too).

  • Greg Wilson   15 januari 2012 17:13
    Reageer op:   Julie Pichon   15 januari 2012 16:51

    Hi Julie; thanks for your post. To answer the question you raise at the start, no, we're not restricting our attention to online learners—that's just the particular kind of free-range learner I've been focusing on most recently, so that's what I talked about in my post. I'm equally interested in other kinds of non-traditional learners; I just don't have as much personal experience to draw on (but I'm really excited to hear about yours).



  • Julie Pichon   16 januari 2012 14:24
    Reageer op:   Greg Wilson   15 januari 2012 17:13

    Thank you for the clarifications Greg! I'm not used to hearing "free-range" to describe people.

  • Greg Wilson   16 januari 2012 14:27
    Reageer op:   Julie Pichon   16 januari 2012 14:24

    I like the term because it lets us refer to traditional lecture-in-classroom learners as "battery-farmed students" :-)

  • Jaelle   15 januari 2012 15:58

    I've posted my comments here:

  • MrSteve   10 januari 2012 21:31

    Further thoughts on recommendations 3 & 4 are posted here

    It would be great if we could have a curated set of artifacts that helped "Connect and integrate abstract and concrete representations of concepts."  The sorting dance algorithms are one good example.

  • Laura B   2 januari 2012 10:30

    I've put up my thoughts here.  I'm working my way through the other articles.

  • Greg Wilson   2 januari 2012 07:19

    Duane Griffin has blogged his thoughts about the research-based best practices part of our first challenge. Please comment here or there, and please keep the posts coming.  (Oh, and when you do post something, please add a comment to this challenge to help others find it.)

  • Greg Wilson   2 januari 2012 07:12

    MrSteve has blogged his thoughts about research-based best practices.

  • R.T. Lechow   1 januari 2012 15:09

    The bit where "Miller's subjects didn’t define iterations, but set operations" was kinda awesome.

  • MrSteve   28 december 2011 13:48

    Would it be okay to post these documents in Google Docs (with private sharing only among classmates) so we can highlight and comment on them?

  • Greg Wilson   2 januari 2012 07:17
    Reageer op:   MrSteve   28 december 2011 13:48

    The short answer is, I don't know, but I suspect not --- I'd welcome comments from someone with a stronger background in copyright than I have. Here's my thinking:

    1. The copyright is held by O'Reilly (the publisher).
    2. But the "fair use" provisions in American (and other) copyright laws allow academics to make a small number of copies of small amounts of material for teaching purposes.
    3. However, those laws were framed when copying meant photocopying...
    4. it's not clear if circulating electronic copies (as I've already done) is legal...
    5. ...and putting something on Google Docs stretches that ever further. (Remember, it's not really private: we're giving Google a copy, and they're allowing us to use it in return.)

    On the other hand, having a shared wiki-like copy we could all comment on would be helpful (and fun). I think anyone re-using copyrighted material in non-traditional teaching and learning is going to run into questions like this; what do you all think the right answer is?