The Online Safety Act When AI Generates the Content

The Online Safety Act 2023 does not have an AI chapter, and asking “does the Act apply to AI content” produces no answer. The question that produces an answer is whether the service is a regulated user-to-user or search service, because every duty hangs off that.

The scope question comes first

The Online Safety Act 2023 received Royal Assent on 26 October 2023 and regulates two kinds of service with links to the United Kingdom: user-to-user services, where content generated, uploaded or shared by a user may be encountered by another user, and search services. Part 5 separately regulates services publishing pornographic content. The Act is on legislation.gov.uk.

The classification that decides everything is whether output is user-generated content or the provider’s own content. A standalone assistant that answers one user privately, with no sharing mechanism, looks like provider content delivered to a single user, and there is no second user to encounter it. The picture changes as soon as any of the following is true: users can share generated outputs on the service; users can create characters, personas or bots that other users interact with; or the generated material is posted into a feed, group or comment thread. In each of those the material is encountered by other users, and the service starts to look user-to-user.

Ofcom’s position on chatbots and GenAI

Ofcom, the regulator for the Act, addressed this directly in an open letter to UK online service providers published on 8 November 2024, setting out how the regime applies to generative AI and chatbots. Its stated positions were, in substance: a service allowing users to share GenAI-created content with other users is a user-to-user service; a service allowing users to create or interact with user-created chatbots may be in scope on the same reasoning; a GenAI tool that generates search results across multiple websites may be a search service; and a GenAI tool that generates pornographic content falls within the Part 5 duties. Ofcom’s online safety material is published on its site.

The letter is guidance rather than a determination, and Ofcom said as much. It matters less as authority than as notice: a service that treated its character-chat feature as outside the regime after November 2024 cannot easily say it had no reason to consider the question.

Which duties then apply

For an in-scope user-to-user service the core duties are illegal content duties and, where the service is likely to be accessed by children, children’s safety duties. Each begins with a risk assessment against Ofcom’s register of risks, and each requires proportionate systems and processes to mitigate the risks identified, content reporting and complaints procedures, and terms of service that are clear and consistently applied.

The timeline is the part to get right. Ofcom’s illegal harms codes of practice and risk assessment guidance completed the parliamentary process such that the illegal content risk assessment deadline fell in March 2025, with the safety duties and codes taking effect from 17 March 2025. The children’s access assessment and the children’s safety duties followed, with the protection of children codes taking effect on 25 July 2025 alongside the highly effective age assurance requirements for pornographic content. Categorised services face additional duties, including transparency reporting, on a later timetable driven by the register of categorised services.

Two features of generative systems interact badly with these duties and are worth planning for. First, illegal content duties are about systems, not perfection: you are asked to design proportionate measures, so the question at enforcement will be whether your generation controls, classifiers and reporting routes were reasonable, not whether anything bad was ever produced. Second, the Act’s duties apply to content encountered on the service, which means a generation feature that can produce priority illegal content on request is a risk factor in your assessment even if the output is never shared — because the sharing is a click away and your assessment must consider the design.

Intimate images and the criminal provisions

The Act also amended the criminal law. It inserted new intimate image offences into the Sexual Offences Act 2003, replacing the earlier “revenge pornography” offence with a base offence of sharing an intimate photograph or film without consent and more serious offences where the intent is to cause alarm, distress or humiliation, or for sexual gratification. The definitions are drafted to include images that appear to be photographs or films, which reaches synthetic and altered material rather than only genuine recordings.

Creating such an image, as opposed to sharing it, was a gap in the 2023 Act. Legislation to criminalise the creation of purported intimate images was taken forward in 2025 through the Data (Use and Access) Act 2025, which received Royal Assent on 19 June 2025. Commencement of individual provisions is by regulations, so whether the creation offence is in force on any given date is a question about the commencement order, not about the Act — check it before stating that it is.

Separately, proposals to criminalise the possession and supply of AI models optimised to produce child sexual abuse material were brought forward in a Crime and Policing Bill introduced in 2025. A bill is not a law; do not describe those provisions as being in force unless you have confirmed Royal Assent and commencement.

Where these offences bite for a platform is through the illegal content duties: an offence of this kind is illegal content for the Act’s purposes, and priority offences carry proactive duties, so a criminal provision expanding to cover synthetic material expands the platform duty at the same time.

What remains unresolved

Three questions are genuinely open. Whether a purely one-to-one assistant with any social feature at all crosses into user-to-user territory has not been tested, and Ofcom’s letter uses “may” rather than “is”. Whether a provider is responsible under the illegal content duties for material its own model generated, as distinct from material a user uploaded, sits awkwardly with an Act built around user content and provider mitigation. And the relationship between the Act’s duties and the labelling expectations arriving elsewhere — see the EU deepfake labelling duty and the DSA position — is not harmonised, so a service operating in both places will satisfy two different sets of expectations rather than one.