Using sparklines to aid reviewing

When assessing proposal reviews, quite often there’s a much deeper story than the average grade alone will tell you. Expectnation adds sparklines to the summary of each proposal to quickly illustrate the variety of reviewing opinion.

Here are some real world examples, drawn from the reviews for XTech 2007.

High fliers

Reviewers are pretty unanimous about this proposal, as 4.0 is the top grade in the scoring system used for this conference. The lack of variance suggests that we ought to ensure that all the assigned reviewers put in their scores — this could be just one person voting.

These next two are the pretty typical distribution for highly graded proposals: mostly top marks and a few average ones.

Middle ranking

Some of the more interesting trends show up when the scoring isn’t biased to either the top or bottom ends of the scale.

The reviewers are unanimous about this paper’s soundness. It’s not made anyone go “wow”, but there’s nothing to grumble about either.

This proposal clearly polarises opinion, and is such stands a better chance than the unanimous 3.0 above. Perhaps the subject matter or approach is controversial or timely.

Another proposal that divides the reviewers’ opinions. It’s also worth checking here that we don’t either have a pathologically strict reviewer, or at the other end, a reviewer with a wild passion for the cause this paper advocates.

Low scoring

If the quality of submissions is good, we don’t have to give low graded proposals much attention, but the sparklines could alert us to potential oversights.

This proposal seems to polarise opinion between “rubbish” and “ok”, so it’s interesting to me to check out the subject matter and see if we’re missing something with potential.

These proposals were not received at all well.