Notion confirms AI chat transcripts exposed after Fireworks AI breach

A cyberattack at Fireworks AI exposed some of its customers’ data, the company has confirmed. Fireworks AI, a platform that developers use to access and use open AI models, said an attacker breached its cloud provider to gain access to a Fireworks development system, which allowed access to some of its customers’ data.

The company said in its latest security advisory that the cybercriminals took Fireworks’ customers’ billing information and some datasets. Fireworks touts tens of thousands of customers worldwide using its platform, including legal tech company Harvey and AI company Cursor.

At least one confirmed affected customer is online document editor and notetaker Notion, which said in a disclosure that the breach affected around 1% of its own customers.

Notion said that Fireworks alerted the company to a security incident and later confirmed that the hackers had stolen some Notion customer data. On November 3, Notion said it disconnected its systems from Fireworks, deleted the data stored in its systems, and stopped all connectivity between the two.

Notion confirmed this week that the hackers took chat transcripts of its Notion AI customers who used the “Insights” feature. The company said that the transcripts include “any input to or output from the model that was written to the chat,” but added that much of the data was unattributable and could not be determined from which customer it was obtained.

On the evening of November 2, Fireworks told Notion that malicious actors had accessed a small dataset. Notion initially determined that it contained synthetic testing data, meaning artificially created material used for testing, rather than actual customer information.

By the morning of November 3, Notion disconnected its systems from Fireworks, deleted all data from the provider’s system, and stopped communication between the two environments. Notion took these actions as a precaution before confirming customer data exposure.

On November 6, Fireworks warned that attackers might also have accessed customer information. Notion immediately opened another internal investigation. Fireworks confirmed the final affected dataset on November 8, and Notion contacted impacted customers the following day.

Analysis showed that the dataset contained Notion AI chat transcripts processed for Insights. This feature helps customers understand how their teams use AI. The exposure differs from previously reported public Notion page privacy issues, which involved editor information rather than these AI conversations.

For most affected workspaces, transcripts were “unattributable.” They lacked labels identifying the author or originating workspace. They also excluded tool call results, meaning responses returned by tools were not included. However, information written inside a conversation could still identify users, companies, or sensitive activities. Notion’s statement does not say every affected transcript was unattributable.

Fireworks supported two Notion functions: classification jobs powering Insights and general inference services running open-weight models. Inference means running a model to generate responses. Notion turned off Insights during its investigation and stopped serving open-weight models through Fireworks.

Affected customers can securely retrieve and inspect exposed information inside their workspace using instructions sent by email. Notion recommends treating that information as exposed, including sensitive content pasted directly into chats.

Customers who entered passwords, API keys, or other credentials during the incident timeframe should rotate them. Notion has not disclosed a precise exposure window, transcript count, or technical entry point in its advisory.

Customers who have not received a notification are not currently identified as affected and need no action now.

Fireworks said that it believes the hackers are financially motivated and that their goals are focused on selling tokens from breaching access to large language models, or LLMs. These schemes rely on selling to unauthorized third parties, who can burn through AI model usage without using their own resources.

This is the second breach affecting Fireworks in as many months, following a security incident in October.

The breach at Fireworks AI shows how tech companies are reliant on other tech providers for their own offerings, and how cyberattacks can, as a result, spill over into other customers.

Sources

This article was drafted with AI assistance and checked against the sources above. Company claims are reported as claims. Cover image is AI-generated.