2.2 The impact of AI technology on society

Discussion

As you have no doubt realised by now, the concerns raised in Gebru’s talk and in the CDT’s summary about the use of AI technology for the ICE’s visa vetting tool, are central to this course. As yet, however, we don’t have criteria whereby we can make such concerns explicit enough that we can work towards addressing them. The aim of the activities throughout this unit is to bring to your attention key dilemmas currently faced by the AI sector.

To the target question for this activity: the CDT post makes clear that the complex set of decisions that ICE were attempting to automate through the ‘Extreme Vetting Initiative’ would be difficult enough for a human to carry out, particularly given the range of subjective judgments involved in the criteria against which visa applications were being considered such as ‘positively contributing member of society’. Automating such poorly understood decision-making tasks would be dangerous, and on this basis, it would seem proper that the ICE should not proceed with creating such technology.

It is important to be aware of the long history of activism in this area, and that groups such as the #TechWontBuildIt movement, seek a whole range of changes within AI in order to address the kinds of concerns we are examining in this course. A common issue expressed by many such groups is that sustainable change is only likely if the larger companies themselves, such as Google, Microsoft and IBM, remain democratic and transparent. So the news of the firing of Gebru from Google was followed with a great deal of concern from late 2020 by people across the field of AI.

Discussion

Consider the example of an Arabic-speaking Palestinian being arrested by Israeli authorities: the key fact is that Facebook tools translated a post written in Arabic into English as ‘Attack them’, when it should have been translated as ‘Good morning’. Since the kinds of tools which Facebook employs do not come equipped with the capability of understanding the seriousness of the implications of such a (mis)translation, then there was no way in which to raise the inappropriateness, and indeed dangerousness of such a translation in this context. Perhaps more importantly in this example, Gebru points out that those using these tools, in this case the Israeli authorities, were doing so unquestioningly, which Gebru points out is an example of ‘automation bias’. Further, Gebru’s main point is that automation bias coupled with tools that are in fact highly error-prone has the potential to be very dangerous to those using them and/or those affected by their use.

Reflecting on the ICE’s proposal to do automatic visa vetting, a key question that perhaps should have been asked by those aiming to develop these tools was whether in fact it was safe to do so, given that automating the decision about whether someone was a positively contributing member of society, required collecting and analysing large amounts of data that potentially included someone’s interactions with friends and close associates. Such data is certainly not the sort of thing that would seem relevant to questions about fitness of membership to society. We will see throughout the course that questions about data are central to critiquing the application of modern AI techniques.

Discussion

As Gebru points out, because they are based on large corpora of naturally occurring text (e.g. web-based text), then word embeddings encode the kinds of societal biases found in such text. This should, of course, be expected in very large amounts of usage data. For example, analogies built from these word embeddings include not only ‘man IS TO king AS woman IS TO queen’, but also ‘man IS TO doctor AS woman IS TO nurse/housewife’ or even ‘man IS TO computer programmer AS woman IS TO home-maker’. Interesting recent work (Guo & Caliskan, 2020) points to how the biases in word embeddings can in fact more seriously disadvantage people who are subject to multiple forms of marginalisation (e.g. women from ethnically marginalised groups).