Persons in a situation of vulnerability, such as older persons, people with disabilities, or those with limited digital literacy or access to modern technology, may be ill-equipped to interact with AI systems. These groups may face difficulties in accessing AI-based services, from simple application platforms online to chatbots and virtual assistants. This could result in exclusion from social services and consequently exacerbate existing inequalities.

On the other end of social services delivery, reliance on AI systems raises quality-related questions. Such systems are, in most cases, designed to support decisions by human professionals and should not replace human judgment. Nevertheless, as evident from domestic caselaw, there may be cases where professionals lack the time, the resources or are simply prone to automation bias and reluctant to use their professional expertise to reach a different decision than the one recommended by the system. AI systems are however not error-proof,[1] and errors in welfare can be fatal for some of the most vulnerable members of the society. In addition, there is concern that “digital-by-design” social services and over-relying on AI would lead to the erosion of social workers’ skills, thus undermining the quality of service, especially in complex, sensitive cases.

 

[1] For instance, in the United Kingdom, the Johnson and others v SSWP judgment (EWCA Civ 778, Judgement, Secretary of State for Work and Pensions v Johnson et al., Case Nos: CO/1643/2018 CO/1552/2018) raised important issues arising from the implementation of an AI system making benefit and welfare decisions for the then newly introduced system of Universal Credit (a single welfare payment comprising a basing personal amount also reflecting childcare, housing, and other prescribed needs). The claimants argued that the automated assessment system used to calculate the amount of universal credit payable to each claimant was unlawful and could create income insecurity, whereas the State acknowledged that the method was “unfortunate” and “arbitrary” but redesigning the system “from scratch” to accommodate adjustments would be too onerous. This defence was rejected and the challenge succeeded, on the ground that the effects, in these instances, were judged to run counter to the policy and objectives of the UC’s underlying regulations and thus “irrational”.