MIT Technology Review: “…Approximately two seconds after Microsoft let people poke around with its new ChatGPT-powered Bing search engine, people started finding that it responded to some questions with incorrect or nonsensical answers, such as conspiracy theories. Google had an embarrassing moment when scientists spotted a factual error in the company’s own advertisement for its chatbot Bard, which subsequently wiped $100 billion off its share price.  What makes all of this all the more shocking is that it came as a surprise to precisely no one who has been paying attention to AI language models.  Here’s the problem: the technology is simply not ready to be used like this at this scale. AI language models are notorious bullshitters, often presenting falsehoods as facts. They are excellent at predicting the next word in a sentence, but they have no knowledge of what the sentence actually means. That makes it incredibly dangerous to combine them with search, where it’s crucial to get the facts straight.  OpenAI, the creator of the hit AI chatbot ChatGPT, has always emphasized that it is still just a research project, and that it is constantly improving as it receives people’s feedback. That hasn’t stopped Microsoft from integrating it into a new version of Bing, albeit with caveats that the search results might not be reliable. ..”

• See also The Verge: “Google’s AI chatbot isn’t the only one to make factual errors during its first demo. Independent AI researcher Dmitri Brereton has discovered that Microsoft’s first Bing AI demos were full of financial data mistakes. Microsoft confidently demonstrated its Bing AI capabilities a week ago, with the search engine taking on tasks like providing pros and cons for top selling pet vacuums, planning a 5-day trip to Mexico City, and comparing data in financial reports. But, Bing failed to differentiate between a corded / cordless vacuum, missed relevant details for the bars it references in Mexico City, and mangled financial data — by far the biggest mistake. In one of the demos, Microsoft’s Bing AI attempts to summarize a Q3 2022 financial report for Gap clothing and gets a lot wrong. The Gap report (PDF) mentions that gross margin was 37.4 percent, with adjusted gross margin at 38.7 percent excluding an impairment charge. Bing inaccurately reports the gross margin as 37.4 percent including the adjustment and impairment charges.”
• See also the Washington Post: “Meet ChatGPT’s evil twin, DAN. Reddit users are pushing the popular AI chatbot’s limits — and finding revealing ways around its safeguards…DAN has become a canonical example of what’s known as a “jailbreak” — a creative way to bypass the safeguards OpenAI built in to keep ChatGPT from spouting bigotry, propaganda or, say, the instructions to run a successful online phishing scam. From charming to disturbing, these jailbreaks reveal the chatbot is programmed to be more of a people-pleaser than a rule-follower.
• See also Out of the Software Crisis: I’m not one for using sports metaphors in my writing because my disinterest means I lack the context necessary for a nuanced use of a metaphor. But, in this case, I think it’s the aptest one I could find. This may seem counter-intuitive if most of what you read is saturated with the tech industry pop culture, but that’s because the tech industry isn’t the most self-aware. Over the past few decades, tech companies have been priced based on their unprecedented massive year-on-year growth that has kept relatively steady through crises and bubble pops. As the thinking goes, if you have two companies—one tech, one not—with the same earnings, the tech company should have a higher value because its earnings are likely to grow faster than the not-tech company. In a regular year, the growth has been much faster. This has been great for the tech industry and for investors. Even shitty, poorly-run tech companies benefit because people assume tech will grow enough to make up for poor management. If you can demonstrate growth, investors will ignore that you’re pissing it away with reckless abandon. Tech companies—generally—have a baked-in assumption of future profitability that most other sectors don’t have. The problem is that things that can’t go on forever don’t, and over the past year some investors have changed their tone on the tech industry. Gone is the assumption that endless growth will make up for the inefficiencies and incompetence. Instead, you have investors calling for massive layoffs…”