Business Insider – The reason should worry the entire tech industry. [unpaywalled] “AI models and tools look great when researchers measure success with their own wonky benchmarks. When paying customers try this technology in the real world, things can get ugly. That’s what happened when an IT executive at a pharmaceutical company tried out Microsoft’s Copilot AI features, according to a recent research note from Morgan Stanley. The chief information officer of this pharma company paid extra to have 500 employees use Office 365 Copilot in the fourth quarter of 2023 and first quarter of 2024. This is Microsoft‘s much-heralded AI upgrade to its popular suite of productivity software. After six months, the exec canceled the upgrade because the AI tools weren’t good enough to be worth the extra money. In fact, he compared the slide-generation capability of Microsoft’s AI tools to “middle school presentations,” according to a transcript of a call with the Morgan Stanley analysts that was included in their research note. “The price is double,” the executive, identified only as Greg, said. “And we really just do not see the value we’re getting out of those tools worth double.” The E3 version of Microsoft’s 365 software suite costs about $34 per user a month. Adding Copilot AI features costs another $30 per user a month. For 500 employees, that would roughly add an extra $180,000 a year. These new Microsoft tools are considered some of the premier examples of powerful artificial intelligence in action in the real world. Investors have bid up Microsoft and other big tech shares massively, betting that this product and similar offerings will catch on with paying customers. If a large pharma company can’t see $180,000 of value in these tools, that’s a problem that should worry the entire tech industry…”