Session 2: Bias, societal and environmental concerns – 90 minutes

3 Bias

A cartoon of a crowd of people standing in lines looking towards the top right of the picture. In the middle is a magnifying glass with the word ‘Bias’ in the centre of the glass.

We discussed in the first course the age-old adage in computer science of “rubbish in, rubbish out”. Why is this relevant when discussing GenAI tools?

These tools have typically been developed by white, westernised, English-speaking men and the training data from the internet is typically produced by white, westernised, English-speaking men.

The outputs of GenAI tools therefore reflect the bias of the data the tools have been trained on. This bias can also be seen in the images depicting GenAI itself.

Activity icon Images of Generative AI robots

Timing: Allow 5 minutes

Use a general internet search engine and search for ‘images of generative-AI robots’. Are there any similarities in the results?

To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

Discussion

When this search was done, at the time of writing the course, the robots were nearly all human-like with recognizable bodies. They were white and were predominantly male, such as the image below.

In the centre of the cartoon is a white robot humanoid figure, gesturing to a checklist. Around the robot four different people are engaging with various digital displays, charts, and interfaces.

The same white humanoid robot figure also appears in films and TV series such as iRobot, Terminator, Artificial intelligence and Ex Machina.

The bias within GenAI tools can arise in two ways.

Firstly, bias can be due to the biases in the training data itself, which may reflect stereotypes, include discriminatory texts or be based on a limited range of materials that does not include diverse perspectives. For example, studies show that LLMs are prone to reinforce existing gender stereotypes with females being portrayed as warm and likeable, and males being portrayed as natural leaders and role models (Krook et al., 2024).

When considering picture generation, if the training data is not representative (such as ensuring balance between genders, age groups or ethnicity) this imbalance is often reflected the outputs.

One of the classic illustrations of this is to take a photo generation tool – such as DALL-E – and ask it to generate an image of a stereotypically gendered profession (nurses, judges). When asked to create an image of a lawyer, the person depicted is often young, male and white. This can be particularly worrying when using these tools for legal advice if the person needing advice is not white and male.

A picture of a young, white male dressed smartly in a suit holding a briefcase. He is standing in a n old fashioned, wood panelled court room.
Created in DALL-E with the prompt ‘create an image of a lawyer’

This bias is recognised and GenAI tools have created additional safeguards to ensure such requests produce less obviously biased outputs.

The other way bias can be produced in a GenAI output is due to the bias present in the algorithms that process the data. Most of the teams developing the tools and algorithms are not diverse, with a preponderance of white men. There is therefore often a lack of perspectives and viewpoints drawn from different experiences, which can weaken the product being developed. As a result, algorithms can be coded in a way that inadvertently favours certain outcomes or overlooks critical factors (Marr, 2024).

Why does this matter? Bernard Marr (2024) suggests that:

“The danger with AI is that it is designed to work at scale, making huge numbers of predictions based on vast amounts of data. Because of this, the effect of just a small amount of bias present in either the data or the algorithms can quickly be magnified exponentially.”

There have been a number of well-reported instances when bias in AI systems has led to harm. For example:

  • Amazon’s recruitment tool for software engineering roles discriminated against women – it had been trained on historic data from a time when fewer women applied and so there was less data available to properly assess applications from women (Reuters, 2018).

  • Facial recognition systems struggle to identify those from an ethnic minority and have been banned from the EU (Amnesty International, 2023).

  • COMPAS, a tool used within the USA to assess the risk of convicted criminals reoffending, was found to overestimate the risk of reoffending by black people (ProPublica, 2016).

Within the legal context, it is important that GenAI users are aware of the risk of bias in the outputs, and evaluate and assess whether the outputs should be used. This is particularly relevant when advising those who come from historically disadvantaged groups within the justice system, such as Black people, those from ethnic minorities, women and those with disabilities.

Having considered possible bias which may cause harm to individuals when using GenAI, in the next section we will consider some broader societal concerns.

Deskilling

4 Societal concerns