The use of AI in social services can perpetuate discrimination (including indirect and intersectional) due to biases embedded in societal data, such as racial, gender, or socioeconomic biases.[1] This may lead to unfair denial of services or benefits, disproportionately affecting marginalised groups and undermining equal access to these services.

Predictive analytics, error or fraud detection and resource allocation systems are more likely to be affected by bias, as they rely on historical data and are prone to exacerbating structural discrimination and stereotypes. For instance, a fraud detector system trained on data that disproportionately reflects the experiences of certain groups is likely to develop risk profiles and create links based on bias, such as lower socio-economic status or an immigration background. This may lead to biased recommendations and eventually the violation of the right to not be discriminated against of not just individuals but whole populations perceived by the system as homogeneous. Safeguards are required, including human oversight, ensuring the critical evaluation of AI outputs and thus neutralising the risk of discriminatory effects.[2]

Where discrimination is alleged, state authorities should take all reasonable measures to determine whether the outcome was discriminatory. This should include an effective and independent investigation.[3]

 

[1] See The Netherlands – Opinion on the Legal Protection of Citizens, CDL-AD(2021)031, Venice Commission, 2021, §§ 96-98

[2] It must however be noted that human involvement is not enough by itself in neutralising discrimination risks; in the Dutch childcare benefits scandal, for example, civil servants were responsible for manually reviewing the highest risk score applications. Without being given any information as to why the system had given a particular application a high-risk score to a specific application, .civil servants have been observed to be prone to apply generalisations to the behaviour of individuals of the same race or ethnicity perceiving them stereotypically as fraudulent or deviant.

[3] Basu v. Germany, No. 215/19, 18 October 2022, § 38.