Natural Questions: a Benchmark for Question Answering Research – “We present the Natural Questions corpus, a question answering dataset. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations, 7,830 examples with 5-way annotations for development data, and a further 7,842 examples 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.”

Center for Data Innovation: “Google has released Natural Questions, a dataset of 300,000 questions and human annotated answers to advance natural language understanding in computers. The questions come from anonymized Google search queries with both short and long answers sourced from Wikipedia. Along with the training dataset, Google has released a dataset of 16,000 questions where each question has answers from five different annotators to help researchers test their question and answer systems…”