Digital humanities: humanities research in the digital age
Digital humanities: humanities research in the digital age

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

3.3 Web crawlers and metadata

For web search engines there is another, equally important stage which is the building of the index by crawling the web. Programmes known as crawlers (or spiders) retrieve information from items published online in order to create the index which users of the search engine will query.

Metadata lies at the heart of search engines and building the index. It’s a shortcut to the contents which allows the search engine to present likely matches to the users’ queries in the split-second timings that we are used to.

Web search businesses such as Google have become powerful and wealthy companies by developing techniques which appear to miraculously remove the difficulties of making sense of the vast amounts of published information we encounter. They return search results effortlessly, in a simple listing taking us straight to what we want to know. How do they do it? A lot of the ‘magic’ relies on techniques of filtering and ranking, in other words revealing and concealing some of the content, and ordering in a way which the search engine designers believe will be useful to us.

Take your learning further

Making the decision to study can be a big step, which is why you'll want a trusted University. The Open University has 50 years’ experience delivering flexible learning and 170,000 students are studying with us right now. Take a look at all Open University courses.

If you are new to University-level study, we offer two introductory routes to our qualifications. You could either choose to start with an Access module, or a module which allows you to count your previous learning towards an Open University qualification. Read our guide on Where to take your learning next for more information.

Not ready for formal University study? Then browse over 1000 free courses on OpenLearn and sign up to our newsletter to hear about new free courses as they are released.

Every year, thousands of students decide to study with The Open University. With over 120 qualifications, we’ve got the right course for you.

Request an Open University prospectus371