EA - Distillation Contest - Results and Recap by Aris Richardson

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Distillation Contest - Results and Recap, published by Aris Richardson on July 26, 2022 on The Effective Altruism Forum. This post: Announces the winning submissions to the Distillation Contest Gives further insight into the scoring process for the contest Examines the effectiveness of our advertising strategies Gives a brief impact estimate of the contest Shares my advice for community builders who are planning to run contests Notes: A huge thank you to Akash and all of the judges for this contest! This wouldn’t have been possible without their work. I’m definitely not perfect! I imagine there are better ways to advertise, run, and score a contest, so I wanted to be transparent about my process so that other people could make suggestions if they have ideas for improvements. Want to use the materials from the Distillation Contest to run your own version? I’m in the process of creating a platform for EA contests and will soon have a demo site up! I’m planning to upload all of the Distillation contest materials into a “bundle” so that anyone can host their own contest easily. Once the site is up (hopefully in the next couple of weeks), feel free to use anything there and iterate to make better resources. If you have your own EA contest resources and would like to make them available to other people, I’d love to add them to the notion! Winners The submission winning first place in the Distillation Contest is Understanding Selection Theorems by UC Berkeley’s Adam Khoja – distilling John Wentworth's Selection Theorems: A Program for Understanding Agents. Our second-place winner is The Geometry of Adversarial Perturbations by Gabriel Wu from Harvard University – a distillation of Universal Adversarial Perturbations. We’ve granted 15 other prizes, six $500 awards, and nine $250 awards, plus three honorable mentions. The six $500 winners are listed below, with their distillations linked to their names. You can find the other finalists, and their distillations, on the EA Berkeley distillation contest winners page. Callum McDougal, Cambridge Jasper Day, University of Edinburgh (has not yet given permissions to share submission) Harrison Gietz, Louisiana State University Sasha Sato, UC Berkeley Chinmay Deshpande, Harvard University Yash Dave, UC Berkeley Scoring Each submission was scored by two judges (of which we had five total, all of whom actively work in the alignment space). Our rubric took into account the submission’s Depth of Understanding, Clarity of Presentation, Concision/Length, Originality of Insight, Accessibility, and two extra-subjective measures: X-Factor and Subjective Rating, explained below. X-Factor Some submissions may end up scoring low despite being amazing because they are exceptional for a quality that’s missed by the factors listed above (they make really great applications of the material, they synthesize multiple sources, they have unexpectedly unique and useful dimensions, etc.). You can grant as many additional points to a distillation as you’d like when you score with this X-Factor zone. Most papers will not have an X-factor effect, so do not feel required to give out X-Factor points. Subjective Rating Ignore the rubric. Assume you just had to rate the submission on a scale from 1-10 (including decimals). What rating would you give this submission? Once the judges had scored their submissions, we found the average score for that judge and then divided each submission score by the average to get an adjusted score. If the average score was 40, a score of 50 would turn into a 1.25. There were 19 submissions that were rated above average by both of their judges. Given that this was already above the number of submissions we said we’d award, we moved on to mostly comparing these submissions to one another. We also looked at “controversial” submission...