Vice: “After years of development, AI is now driving cars on public roads, making life-changing assessments for people in correctional settings, and generating award-winning art. A longstanding question in the field is whether a superintelligent AI could break bad and take out humanity, and researchers from the University of Oxford and affiliated with Google DeepMind have now concluded that it’s “likely” in new research. The paper, published last month in the peer-reviewed AI Magazine, is a fascinating one that tries to think through how artificial intelligence could pose an existential risk to humanity by looking at how reward systems might be artificially constructed…

The paper envisions life on Earth turning into a zero-sum game between humanity, with its needs to grow food and keep the lights on, and the super-advanced machine, which would try and harness all available resources to secure its reward and protect against our escalating attempts to stop it. “Losing this game would be fatal,” the paper says. These possibilities, however theoretical, mean we should be progressing slowly—if at all—toward the goal of more powerful AI…”