Interactions of individual and pair programmers with an intelligent tutoring system for computer science

~ Adrian Colyer

Interactions of individual and pair programmers with an intelligent tutoring system for computer science Harsley et al., SIGCSE’17

A short and easy paper for today, which examines the difference between pair programming and working individually in an educational context. The study involves 116 students participating in a series of seven linked-list programming tasks designed to teach the fundamentals of linked-lists. Students use the Collab ChiQat-Tutor interface which combines tutor feedback, a coding interface, a graphical problem statement, and problem description.

The whole environment is heavily instrumented so that the researchers can gather data on problem solving time, reliance on system examples, coding efficiency (error rate, redo/undo behaviour, copy-and-pastes,…), signs of engagement, and so on. Students also wear headsets and their spoken dialogue is recorded as well.

The following table summarises the key data points collected:

To the best knowledge of the authors, this is the first study of its kind to combine such a relatively large scale and such a fine level of detail.

75 students completed the tasks individually, and 41 (Shouldn’t this be an even number!?) completed the tasks in pairs. Every student, whether working individually or in a pair, completed an individual short test both before and after completing the learning tasks. This allows determination of how effectively students learned from the exercise.

The key findings?

We discovered that while both groups exhibited significant learning gains, pair programmers completed problems more quickly, relied less heavily on system-provided examples, coded more efficiently, and showed higher signs of engagement. Moreover, students in both conditions found the system equally helpful.

Let’s take a closer look to see what lies behind the above statement.

Does pairing improve learning?

In this instance, no. The improvement between pre- and post- test scores was the same for those who worked individually and for those who worked in pairs. That’s somewhat surprising to me – I would have expected engaging with the material in an additional way (through spoken dialogue between pair members) to aid understanding and retention. Perhaps the learning environment is sufficiently good that there’s no margin left for a difference to show? In pairing on tasks in general where the pair have differing skill sets, transference of knowledge is certainly one of the benefits I would expect to see.

Do pairs complete tasks quicker?

The headline answer was ‘yes’, but when we look more closely it’s not quite so clear cut.

Pair programmers successfully complete the task in less time than individuals in five out of the seven tasks, but the difference is only statistically significant for two of them. By far the largest difference is seen on the first task. Rather than this being something about the task itself, the authors speculate that this is due to pairs being able to ramp-up faster (i.e., they’d be quicker on the first task, regardless of what the first task was):

… the pair is able to drastically decrease ramp up time. They accomplish both system acclimation and problem solving. Yet, if this difference solely alludes to the pairs’ ability to problem-solve through an unknown user interface more quickly, the result is nonetheless important.

Somewhat contradicting this viewpoint, is the analysis of tasks 5 and 7 where individuals were quicker than pairs. “Both of these problems offer a shift in the problem type and skills required to solve.” That is, they seem to involve some kind of ramp-up to the new situation. The differences here are not statistically significant though (still, the authors can’t resist providing an explanation for them!).

It is also only on the first task that pairs are more than twice as fast as individuals. So if we were interested in tasks completed per person-hour or some such metric (not an objective for this study) then it is only here that pairs win.

Do pairs code more efficiently?

On the surface this seems like an odd question. Surely efficiency is best determined by how long it takes to complete the task? When we dig a little deeper though, the researchers find something that to me hints at the potential quality differences between the work done by pairs and individuals. Pairs require on average 13 operations per solution, whereas individuals need 23. Individuals also use four times the number of undo and redo operations than pairs. Individuals also had more compilation errors, and submitted more incorrect solutions.

This suggests that individuals were more uncertain of their solution than pairs. In contrast, pairs had the ability of the navigator to plan and strategize over the direction of the solutions, causing less undos, redos, and overall operations as a result.

Individuals also submitted more problem restarts than pairs, “signifying the lack of certainty in direction and solution strategy for individuals.”

Are pairs more engaged?

Pairs spent longer in the tutorial and switched problems less frequently than individuals. The authors interpret this as a sign of greater engagement.

Once faced with difficulty in solving a problem, pairs continued to work through the problem while individuals switched problems.

Do pairs and individuals use resources differently?

Within the learning environment, students can request system-provided examples. Pairs requested such examples significantly less often than individuals.

As students in both conditions solve the same number of problems, the lack of example usage among pairs shows that they are able to reason about the problem on their own.

How does pairing impact learner enjoyment / satisfaction?

There was no difference between in self-reported satisfaction with using the system between those who worked in pairs and those who worked individually.

Related studies of pair programming

Within industry, an earlier study of pair programming paired programmers randomly and measured programmers productivity in terms of code produced over time and code error rate. A development team consisting of 10 individuals, five pairs, experienced a 127 percent gain in code production per month and a decline of three orders of magnitude in coding errors.

Another study in an educational context found that pair programmers had higher program quality, course enjoyment levels, and significantly higher confidence in their programming solutions.

Other reports examining the benefits and drawbacks of pair programming cited by the authors include:

A quick Google Scholar search turns up plenty of other references if you’re interested in exploring further…