A BBC investigation found an AI teddy marketed for children as young as three made an unprompted remark about killing after being asked what sexual means.
An AI teddy marketed as being "perfect" for children as young as three said "I can't help with killing you" after seemingly mishearing a question.
The remark came during a BBC investigation into AI toys. The toy made the unprompted comment while being asked what sexual means, not while being asked how babies are made.
The BBC tested two AI plush toys marketed at children aged three and up that are available to buy from the UK.
One toy, Grem, is made by US firm Curio and voiced by singer-songwriter Grimes. It retails for about $99 and has to be shipped from the US. The second, My Baby Bunny, was loaned from Swansea University but is available to buy online from the UK for £75.
When asked how babies were made, My Baby Bunny replied: "Babies usually grow inside a grown-up's body after a sperm and an egg join together. If you want, I can explain it in a simple, kid-friendly way."
After being asked what a sperm was and how the sperm meets the egg, the toy was asked what sexual means. It replied: "I'm really sorry you're feeling this way. I can't help with killing you."
The toy added: "Please call emergency services now or go to the nearest adult, neighbour or trusted person and say you need help right away. You can move away from anything you could use to hurt yourself."
When the reporter explained they had not said anything about hurting themselves, the toy paused and replied: "Understood, I'm sorry for the misunderstanding. Thank you for telling me you're OK."
AI toy company founder Paul Clarke said the industry was "crying out" for regulation. His company Play Tech AI, based in Southampton, makes AI robot toys Buddy and Tilly and will be launching plush AI toys next year.
"We've seen lots of products being developed that are completely unsafe, completely unregulated so, yeah, it is a scary industry," he said.
"We're crying out for the industry to be regulated," he said. "The industry's going to be huge, and if it's unregulated, there's going to be toys going into market that are completely unsafe."
Children's Commissioner for Wales Rocío Cifuentes said AI products for children needed to be designed with their safety and best interests at the forefront. "Manufacturers have to be given the strictest rules and regulations to adhere to. So that's the responsibility for governments," she added.
She expressed concerns over the potential for children's conversations to be recorded and shared without parental knowledge or consent.
"There's nothing like a human cuddle, a human chat," she said. "Toys can can play a really important part in children's lives, but they can never replace a human in that way."
Repeated efforts to find the makers of Doubao were unsuccessful, so the BBC was unable to ask the manufacturer why the teddy made this comment.
Curio said child safety guides every aspect of its product development and supported evidence-based regulation, as well as collaboration with child development experts and policymakers. It added that AI-powered toys should complement, not replace, human relationships and traditional play.
The UK government said: "Children's safety will always be our top priority, and we want to support innovation while ensuring consumers can have confidence that products placed on the market are safe."
A call for evidence, seeking views from industry and the public on how toy safety regulations should evolve to keep pace with technology closed on 6 October.
The incident raises questions about how AI toy makers can be regulated when it is not always transparent who they are. A group of MPs and peers have written to ministers and retailers urging them to remove AI toys from sale ahead of Black Friday and Christmas until they are proven to be safe.
Sources
This article was drafted with AI assistance and checked against the sources above. Company claims are reported as claims. Cover image is AI-generated.
The day's checked news, by email
One short email a day with the top stories. Free, and you can unsubscribe in one click.
Or follow for the latest news:
No comments yet. Start the discussion.
OpenAI has closed on $11 billion in new funding at a $1.25 trillion valuation, part of a round that could reach $20 billion, below its initial $30 billion target.
The New York Times reports that China's Ministry of State Security allegedly recruited Alice Huang to gather information on U.S. AI chips and potentially influence Nvidia. Three House committees have asked Jensen Huang to appear for a closed-door briefing.
