TikTok bans some forms of AI-generated content

TikTok has updated its community guidelines with a major clampdown on content generated, or altered, using AI.

It’s labelled “synthetic and manipulated media” in TikTok’s rules, and while AI posts are not banned outright, significant restrictions are to be implemented.

Has TikTok banned AI?

TikTok says “we do not allow synthetic media that contains the likeness of any real private figure.” Deepfakes of your friends, or frenemies, are not permitted.

However, those celebrity deepfakes are not off the cards yet. They will go beyond the limits of TikTok guidelines only “if the content is used for endorsements or violates any other policy.” Significant politicians, government officials and business leaders all end up in this public figure bucket, as long as they are aged 18 or older.

These terms mean people will, in theory, be able to get AI content removed if it uses their likeness. Unless they are Tom Cruise.

AI-generated material that depicts “realistic scenes” will also need to be tagged appropriately, with a word such as “synthetic”, “fake”, “altered” or “not real”.

TikTok’s approach here clearly aims to avoid some of the potential harm of AI, without banning the kind of fun content that has already gone viral on the platform.

However, there’s still a question of how effective this system will be in practice.

How TikTok moderates posts

TikTok says content is “initially reviewed by our automated moderation technology, which aims to identify content that violates our Community Guidelines”.

“If a potential violation is found, the automated moderation system will either remove it automatically or pass it on to our safety teams for further review,” a TikTok spokesperson told us.

If TikTok’s ability to identify AI-generated or modified videos was limited — and it surely must be considering it will need to be able to separate someone posting a vlog with a face filter and an AI video of someone else — some of the onus will rest on TikTok users.

Users can report violations of the platform’s guidelines, as they can already for other rule-breaking posts.

TikTok has set out a clearer policy on AI content than Facebook or Instagram, where such content is typically covered by other rules. For example, Facebook’s guidelines forbid use of the platform to “bully, impersonate or harass anyone,” which would cover some more problematic AI posts.

TikTok controversy

TikTok’s new community guidelines also introduce “tribe” as a protected term. The aim is to “strengthen our approach to counter hate on the platform, especially in parts of the world where affiliation with a particular tribe may put someone at risk of being a victim of hate speech or hateful behaviour,” a TikTok spokesperson said.

These community guidelines changes are sensible, but how effective they will be beyond giving people a clear policy to point to when someone creates AI content of them online remains to be seen.

The timing of the changes also makes this appear a strategic move. Reports of the US considering a ban of the platform continue to mount. Meanwhile, New Zealand has banned the app from government devices and the BBC has just advised staff to delete the TikTok app over privacy fears.

Clearer policies offer better ways to combat the abuse of AI, but do not cover concerns surrounding the potential influence of the Chinese government over the platform and its stored data.