I recently wrote an post called Anonymous review is amazing, talking about our experience with anonymous review at !!Con (it was excellent! I was surprised and delighted!). There was a discussion on the PyCon organizers list today about whether PyCon should do anonymous review, and I started thinking about this a little more carefully.
I’m going to make a few assumptions up front: our goal as conference organizers is to have
- a process that is as unbiased as possible
- speakers who will be engaging
- who come from diverse backgrounds
- where some are new speakers, and some are more experienced
Let’s talk about whether anonymous review will help us with these things!
Is anonymous review less biased?
Yes.
Firstly, people believe that anonymous review is less biased.
One of our !!Con’s speakers, Katherine Ye, told us:
Thank you so much for [anonymizing everything]! It’s a relief to know that I wasn’t picked for gender, race, age, or anything like that.
Kenneth Hoxworth said of RailsConf’s anonymous review process:
It gave me courage that I wasn’t going up against big names.
It’s really important for people to have confidence in a conference’s review process. Nobody wants to put time into a proposal if they’re going to be dismissed because of their gender or age or race, or just because they’re not famous enough. People also worry about not being accepted on their own merit.
Anonymous review helps us build confidence, and that’s really valuable.
Anonymous review is also actually less biased. This study by Kathryn McKinley shows that, in peer-reviewed scientific articles, both men and women express systemic bias against women, and double-blind reviewing removed that bias. (thanks to Lindsey Kuper for the link!)
They found nepotism and gender bias were significant factors in the evaluation process. To be judged as good as their male counter parts, female applicants had to be 2.5 times more productive.
Will anonymous review help my conference’s diversity?
Maybe. EuroPython has an anonymous review process, and recently very few of their announced speakers were women. This is because very few women applied to give talks. You can’t accept talks that don’t exist!
A more effective way to diversify your speaker pool is through active outreach. I don’t know of any evidence to show that anonymous review helps you attract a more diverse range of speakers. (is there some? I would love to know.)
Will anonymous review help me get inexperienced speakers?
Maybe.
On one hand, we have
It gave me courage that I wasn’t going up against big names.
On the other hand, Douglas Napoleone pointed out:
An anonymous system has an inherent bias towards very well written proposals. Those people whom have given the most talks are those whom are best at writing proposals which are best at getting through selection committees. It becomes a feedback loop which cuts out the very speakers we want most. Knowing that a person is a new speaker with a decent proposal is key when comparing them against a proposal by someone whom has given a talk at the last 8 python conferences.
PyCon’s approach is to actively encourage new speakers to apply and work with them to write better proposals, and that’s been successful.
Florian Gilchner wrote about eurucamp’s experience with anonymous review here:
We found that newcomers don’t write worse proposals than seasoned speakers. Quite the contrary, we found that many proposals that are submitted to many conferences are unspecific and dull and would only fly by having a big name attached. Anonymous CFPs are very good at weeding out copy-pasta. We didn’t accept quite a few people that would have been really shiny on the program.
and
Every year, we have at least one person we take huge bets on and get very good talks out of that. Most of the time, it’s someone would [would lose out] in a direct and open battle.
But will my speakers be good?!
This is probably the scariest part. We did anonymous review for !!Con, and our speakers were very good. Our main hope was that if somebody wrote a proposal about an interesting topic, then they could give an engaging 10-minute talk. This worked. It’s relevant here that our talks were all lightning talks.
We also had an anonymizer, who did an amazing job reviewing videos and telling us his impressions. This meant that we had to trust his judgement (which I do! and our speakers were great!), but having only one person watching talks introduces bias.
I’d be worried about doing anonymous review if I was organizing a conference where the talks were longer. (though it’s been done successfully!)
So should you do anonymous review?
Anonymous review takes extra time. You should think about what benefits you hope that it’ll bring, and what your alternatives are. There’s some excellent discussion on the comments to a draft of this post. Go read the whole thing.
Some other things you can spend time on:
- doing outreach to get more applications from under-represented communities
- giving new speakers feedback on their proposals and helping them do a better job
- writing up a really good call for speakers (see JSConf EU’s!)
- running brainstorming sessions to help people come up with ideas
I would do it again for !!Con, since the response to it was super positive and the talks were good. I find the bias-reduction argument pretty compelling. Nepotism and accepting your friends’ talks are really hard to fight against. Judging speaker quality still worries me!