Game-changing solution or digital snake oil?

First posted on LinkedIn, April 2024.

How do you know whether the product or technology solution you’re being sold is actually going to do what it claims?

Will the AI-powered data analytics black box you’ve just spent £100k on turn out to be a modern-day Mechanical Turk? We’re getting marketing fired at us left, right and centre for the next tool that will solve all our problems, whether that’s next-gen security or an AI that processes all your business data in seconds and eliminates the need for those pesky humans.

(Also: why is it always next-gen? Surely if it’s available now it’s current-gen, and the next-gen stuff is the products that aren’t out yet. Nobody ever sells you current-gen anything any more.)

The original Mechanical Turk

Worth remembering where that phrase comes from. The Mechanical Turk was an eighteenth-century chess-playing automaton that toured Europe beating aristocrats and, allegedly, Napoleon. It was a sensation. It was also a cabinet with a human chess player hidden inside, working the machinery through a sliding seat and a set of levers.

Amazon liked the story enough to name a crowdworking platform after it: Amazon Mechanical Turk, where you pay real people small sums to do the tasks software can’t. There’s something quite honest about that, in fairness. The industry knows exactly what it’s doing when it puts people behind the curtain.

What happened at Amazon Fresh

Hence the story going round this month. Amazon is pulling its till-less “Just Walk Out” technology from its Fresh grocery stores, and the version doing the rounds is that the whole thing was really a room full of people in India watching the cameras.

The accurate version is a bit different, and worth getting right. Amazon disputes that framing firmly. Its executives say nobody was watching live shoppers, and that the staff in India were annotating video afterwards to train the models, with a smaller team verifying receipts in a minority of cases. What was actually reported is arguably worse for the technology, though: that as of 2022 around 700 of every 1,000 Just Walk Out transactions needed some form of human review, against an internal target of fewer than 50. Amazon disputes those numbers too.

The lesson I take from it isn’t that Amazon lied. I don’t think the evidence supports that, and everyone’s AI has humans somewhere in the loop. It’s that even with a household-name vendor, a flagship product and enormous public scrutiny, outsiders genuinely could not establish how automated the thing was. If that’s the visibility available for Amazon’s shop, what exactly do you think you’re getting from a twelve-slide deck and a demo environment?

So how do you sort the real from the flashy?

No perfect test, but these are the questions that have served me best:

• “Show me it failing.” Ask for the edge cases, the messy input, the thing it gets wrong. A vendor who can clearly articulate the limits of their product understands it. One who claims there aren’t any is either overselling or doesn’t know.
• Pilot on your data, not their demo. Every one of these tools works beautifully on clean, curated, well-structured information. Yours is none of those things. That gap is where the money goes.
• Ask who or what is in the loop. Straight question: at run time, is a human involved in producing this output, and if so where and how many? Nobody credible minds answering it.
• Get reference customers your own size. Not the flagship logo on slide three. Somebody with your headcount, your budget and your legacy mess.
• Beware the black box. If a supplier can’t explain the mechanism in terms you can follow, you cannot assess the risk, you cannot audit the output, and you will not be able to defend it when a regulator or a customer asks.
• Watch what happens to the price when the pilot ends. Enough said.

I’m not having a go at vendors here. Plenty are doing valuable work, and humans in the loop is not a scandal; it’s how most of this stuff actually functions. The line between innovation and snake oil isn’t whether there are people behind the curtain. It’s whether anybody told you they were there.

I’d love to hear other people’s take on the rise of these so-called digital snake oil solutions, and any tells you use yourself.

Update, July 2026

Well, that aged quickly. In May 2025 Builder.ai (a London company valued at $1.5 billion, backed by Microsoft, whose “Natasha” product was marketed as an AI that built apps for you) entered insolvency proceedings. Reporting that followed alleged that the app building had been done largely by several hundred human engineers, and that revenues had been substantially overstated. It collapsed entirely.

So the answer to “could a company really sell human labour as artificial intelligence at scale, to sophisticated investors, for years?” turned out to be yes, comfortably. The questions above have not got any less worth asking.