Via the INDUCTIVE mailing list, I learned of the Journal of Interesting Negative Results in Natural Language Processing and Machine Learning
It is becoming more and more obvious that the research community in general, and those who work NLP and ML in particular, are biased towards publishing successful ideas and experiments. Insofar as both our research areas focus on theories “proven” via empirical methods, we are sure to encounter ideas that fail at the experimental stage for unexpected, and often interesting, reasons. Much can be learned by analysing why some ideas, while intuitive and plausible, do not work. The importance of counter-examples for disproving conjectures is already well known. Negative results may point to interesting and important open problems. Knowing directions that lead to dead-ends in research can help others avoid replicating paths that take them nowhere. This might accelerate progress or even break through walls!
That’s healthy thinking, although the site/project/journal seems very new, not much up there yet. However it does have a page of links to other such journals, events, forums and articles in favour of documenting scientific failures. Listed in there is an upcoming AAA-08 Workshop, What Went Wrong and Why: Lessons from AI Research and Applications.
The second workshop will continue our analysis of failures in research. In addition to examining the links between failure and insight, we would like to determine if there is a hidden structure behind our tendency to make mistakes that can be utilized to provide guidance in research.