4 The development of Generative AI
While most people only became aware of GenAI in 2022, the origins of the current artificial intelligence technologies can be found in the Cybernetics research of the 1940s, which studied the similarities between organic beings and the wires, processors, sensors and actuators of computers and machines.
Cybernetics included thinking about how to mimic human behaviours and reasoning. Work in this area was given the label artificial intelligence (AI) in 1955 by John McCarthy, and the first AI workshop took place in 1956 at Dartmouth College.
Since then, the development of AI has seen repeated bursts of research and commercial interest, as new approaches to handling data and data processing are identified and widely optimistic claims for near future capabilities were made.
For example, in 1970 one of the most famous researchers in artificial intelligence, Marvin Minsky, made the prediction:
‘In from three to eight years, we will have a machine with the general intelligence of an average human being. I mean a machine that will be able to read Shakespeare, grease a car, play office politics, tell a joke, have a fight.’
These bursts of activity were followed by periods known as AI winters, where such interest (and research funding) declined. In the main, AI research did continue throughout the AI winters – just with reduced funding and less commercial and public interest. Many of the early attempts to build AI systems focused on the ability to build rule-based systems, followed by expert systems.
Further reading
If you are interested, you can learn more about these different forms of artificial intelligence in this supplementary course content:
The history of the development of GenAI [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)] .
Attention then focused on natural language processing, where numbers are used to encode words, phrases and sentences. This allowed researchers to work out the statistical probability that a word or words would be followed by another word. This is the basis of the text prediction on your smart phone.
Predictive tools
Predictive tools can use statistical likelihood to predict the missing word(s) in: Tracy stroked the fluffy
Think of 10 words that could fit in place of the
Discussion
Here are our suggestions: telephone, crocodile, cushion, cat, car, carpet, dinosaur, cup, calculator, first-aid kit (which also happens to be 10 things I can see looking around my desk today).
For the ranking from the highest to the lowest I’ve gone with: cushion, cat, dinosaur, crocodile, telephone, cup, calculator, first-aid kit, carpet, car.
The reasons for the ranking go something like this: the first two are easily the most probable. The dinosaur and crocodile look a bit out of place – except you might see kid’s toys on the floor and they are sort of fluffy. The next four are things that could be on a sofa but aren’t usually fluffy and the last two are usually not found on a sofa.
Now you may rank these slightly differently, and for different reasons even if you agree with us. If we shared this list with everyone taking this course and asked them to rank them, and then we collected those rankings and produced a statistical average ranking, ignoring the reasons – we’d end up with a ranking that we could use in a system.
Notice that there are no rules involved – this is purely a statistical summary of the choices made by a lot of people. This shows how we can do natural language processing with statistics.
Combining natural language processing with large quantities of training data, neural networks and transformers led to the explosion of effective GenAI tools in the 2020s, including those which produce text outputs and others which produce images, music, computer coding and a variety of other outputs.
Further reading
If you are interested, you can find out more about how the technology works in this supplementary course content – Transformer-based neural networks.
As noted above, there are different types of GenAI tools, and in the next section you learn more about the one most commonly used for legal advice and information, the large language model.
3 What is Generative AI?
