Before, During and Afterwards
Before: ‘Warn’ the users in advance
Let users know at an early stage what content moderation actions might occur following conduct or content that is prohibited by ToS or by law. This means: Include information about your moderation strategies in the binding agreement with the users, i.e. in the ToS (chapter 1).This also helps to ensure that you begin to build trust with individuals before or at the time of entering into the relationship with the user and that you give a binding and unambiguous assurance that problematic and illegal content will not be tolerated on the platform and that you, as a HSP, want to protect users from such content.
During (or shortly after the action): Notify the users whose content has been blocked or moderated and offer the option to appeal the decision
Article 11 of the TCO Regulation obliges HSPs firstly, to inform the user who provided (i.e. authored and/or uploaded) the prohibited (terrorist) content that it has been blocked by the platform, and, secondly, to provide information to the user about the background or the removal order at the user’s request. Notification can be withheld from content providers temporarily for a period of no longer than six weeks, if the competent authority considers that there is a particular danger involved in communicating the removal to the content provider. When non-disclosure still remains important and appropriate, the competent authority may extend the period by further six weeks (TCO Regulation, Art. 11.3). The TCO Regulation (Art. 10) provides specific mechanisms for handling such situations. It is advisable to set up a standardised automatic process by which content providers are notified of deletions and are given an opportunity to file a complaint against them. This notification and objection mechanism should also be deployed in cases that do not fall under the TCO Regulation, but have taken place through (pro-)active measures on the part of HSPs
Afterwards: Regularly communicate key data on HSP-wide content moderation measures publicly
Transparency about content moderation decisions and outcomes is important to establish trust and accountability between HSPs and their users, and is increasingly mandated by different types of online regulations. Transparency reports capture information on platforms’ moderation decisions, including not just the number and type of violations identified but also exactly how such violations were handled. (More details on transparency reports and TCO transparency requirements can be found in chapter 6).
It is important for HSPs to have a clear process for content moderation that can be customised to specific business services and needs. It is crucial to first identify prohibited content and to be able to assess its illegality, as discussed in more detail in chapter 2. As a reminder, these are the key steps:
|
In this section we are concerned with step 5. In chapter 2, we named this step ‘Decision on how to deal with the content’. Having received a removal order from a competent authority, HSPs must remove the content and have the possibility to appeal the order. HSPs can also decide to identify and remove terrorist or prohibited content proactively (i.e. absent removal orders).
This guide distinguishes five different types of moderation: pre-, post-, reactive, distributed and automated. Brief explanations as well as their advantages and disadvantages can be found below.
| Type of moderation | Explanation |
|---|---|
| Pre-Moderation |
Moderators review the content before it is published.
+
Pros: High degree of safety and content compliance with legal standards and platform-specific regulations
–
Cons: High burden of effort, personnel and (eventual) financial costs |
| Post-Moderation |
Moderators review the content immediately after publication.
+
Pros: Enabling quick user interactions with the content (due to immediate publication)
–
Cons: Significant amount of time, personnel (and eventual) financial costs; in the case of prohibited/harmful content, users are exposed to the content |
| Reactive moderation |
Moderators review the content after users have reported it.
+
Pros: Less resource-consuming HSP effort; Building trust with users (through the possibility of reporting)
–
Cons: Responsibility of the users; in the case of prohibited/harmful content, users are still exposed to the content; potential for false alarms requiring additional effort on the part of the HSP |
| Distributed moderation |
The user community acts as moderators, often through an up- and down-vote mechanism which measures the trustworthiness of the content and is ultimately used to determine the content’s public reach (i.e. content with many up-votes ranks better and gets a higher reach).
+
Pros: Less resource-consuming on the part of the HSP; Encourage user engagement; Self-regulation
–
Cons: Responsibility of the users; in the case of prohibited/harmful content, users are exposed to the content; Susceptibility to coordinated manipulative behavior on the part of malign users |
| Automated moderation |
Models based on artificial intelligence (e.g. filters, algorithms) function as moderators.
+
Pros: Less resource-intensive once it is set-up; early, fast, highly scalable detection of potentially malicious content
–
Cons: Susceptibility to errors, especially with regards to erroneous content deletion (problematic in the context of freedom of expression, thus human assessment important); continuous maintenance and adaptation to new developments necessary |
(Based on Grimes-Viort, 2010)1Grimes-Viort, B. (2010, December 7). 6 types of content moderation you need to know about. Social Media Today. https://www.socialmediatoday.com/content/6-types-content-moderation-you-need-know-about
It is recommended to combine several approaches to content moderation. Reactive moderation, for example, can be easily combined with pre- or post-moderation. In most cases, combinations are very useful, and in some cases can even be necessary. Automated moderation decisions must always be reviewed or at least informed by human review to avoid systematic breaches of users’ fundamental rights and risks to freedom of expression.
Be aware that extremist and terrorist actors can use tactics to avoid known content moderation methods, operate under the radar, and thus circumvent moderation, particularly automated content moderation mechanisms. Popular tactics include the use of URL shortening to evade filters or websites blocks, account and content mirroring which involves posting or creating identical content/accounts multiple times to overwhelm moderators and have back-ups available in case of deletion, or deliberately choosing (incorrect) spellings of a word to slip past automated word filters. The range of evasive techniques emphasise the importance of human moderation.
Your involvement with this guide, or your (certified) participation in the online course which is also offered as part of the TATE project, show that you are interested in these trends. You can find more examples of content moderation evasion as well as feasible and effective responses in Tech Against Terrorism’s Knowledge Sharing Platform, which is available here.