Amazon Science – “Product search algorithms, like the ones that help customers place orders through Alexa, aim at returning the products that are most relevant to users’ queries, where relevance is usually interpreted as “anything that satisfies the users’ need”. A common way to estimate customers’ satisfaction is to rely on the judgment of human annotators. (We annotate a very small fraction of 1% of interactions.) We’ve noticed that customers frequently engage with voice shopping results that annotators label as irrelevant. At this year’s ACM Web Search and Data Mining (WSDM) conference, in February, we will present a systematic analysis of this phenomenon…”