3.2 Evaluating open learning

Having experimented with some of the search tools available and got some results, the next step for anyone searching for relevant content is to evaluate these results in a systematic way. If you intend to use OERs for direct teaching and learning purposes, or for some repurposing prior to teaching and learning, there are several attributes that need to be considered first. Important attributes of quality OERs include:

• accuracy
• reputation of author/institution
• standard of technical production
• accessibility
• fitness for purpose
• clear rights declarations, e.g. Creative Commons.

The JISC Open Educational Resources infoKit quality considerations [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)] web page contains a range of detailed criteria for consideration. Some key criteria that you might want to adopt when evaluating your results might be grouped into attributes. For example:

• Can the content be described as follows?
  • Relevant, accurate, appropriate level of detail, objective, current and jargon-free.
  • Of good provenance (consider the reputation of the author/institution), with a list of references if appropriate.
  • Free of advertising.
• Does it fit my chosen pedagogy?
  • Learning outcomes are stated and match with learner’s needs.
  • Engaging and interactive.
  • Set at the appropriate level, with any prerequisite skills/understandings stated.
  • The time required to study is stated and equates to the importance of the learning outcomes achieved.
• How does it measure up to usability/accessibility standards?
  • Easy-to-use and well presented, with clear navigation.
  • Accessible for users with disabilities and conforms to accessibility guidance e.g. the UK Equality and Human Rights Commission’s general web accessibility guidance.
• How genuinely re-usable is it?
  • A standalone resource that can be reused in different contexts.
  • Robust and functional, and works on different browsers/platforms.
  • Rights are fully documented, e.g. does it carry a clear CreativeCommons or other rights declaration? Is it OK to re-use it? Are there any conditions?

The video below provides a record of a session previously delivered on the topic of finding and evaluating OERs as part of the activities run for SCORE fellows at The Open University. You can see from watching this that there were very diverse experiences encountered by the participants in using the search tools and locating OER that met their expectations or requirements, and that there is still a need to improve the searching experience for learners, teachers and researchers when seeking OER.

Show transcript|Hide transcript

Transcript: Finding and evaluating OERs

PROFESSOR:

There’s an awful lot in the news at the moment about copyright, intellectual property, around digital material. And there are some interesting legislative developments happening in Europe as well that we need to keep a close eye on because a lot of industries, particularly the publishing industry, are scared witless about what might happen if they start opening up and releasing some of their materials in this way. So we’re at a very interesting sort of water shed, I think, as to where this will go.

Now, many of us that are supporting students as well try and encourage them to avoid plagiarism by teaching them great information management skills and at least citation, basic citation, about where they found the material that they’re using in their assignments and their work. And what we say to them, look if it hasn’t got a licence on it then don’t touch it, basically. We’re so used to seeing these websites out there. Not many of them, actually, stamp a licence on it, on their content, on their website.

So you take that risk. And most of us, invariably, we use screenshots in our work as well. So we need to be aware of that. That’s not to say, possibly, not do it, but at least acknowledge where it’s come from, and then manage that risk associated with it if it doesn’t declare a licence on it that you can use it in a particular way.

So this morning we’re actually going to look about various ways of finding places you can find relevant, open education resources and some very basic criteria that you can begin to evaluate open education resources. The library staff here at Open University developed an open package of resources for information management skills called Safari. And so what we’ve done is adapted some of the evaluation criteria that we use in that programme to fit with open educational resources.

So this part will be a presentation introducing you to some resources. And the second part we’ll actually then it’ll be over to you so that you can start searching and finding some of these open educational resources on a topic of your own choosing. Choosing two of the specialist engines, search engines, that you’ll see this morning and using the checklist that we encourage people to use when they’re evaluating what they find once they’ve got their results.

And then, we’re finally going to end up, we might cut this short depending on what the time is like, sharing your experiences this morning of using some of these tools. It is still quite crude out there. So just be aware of that. Nobody has found the silver bullet here about searching and finding things quickly that you need. Part of it is because of the sheer volume of information and artefacts that are published out there. The other part is getting back to librarian skills here, is about describing the properties and attributes and structures of these things. So it’s down to that good old ‘m’ word, the metadata word, that people are either very enthusiastic about or glaze over all together.

But without good metadata, this really doesn’t work very effectively. And a lot of the metadata that are in things like the Jorum are very specific fields. It’s a basic standard way of describing learning objects that they’ve used. But of course, it’s just searching those basic fields. What’s coming on behind this is moves more to linking things through the use of semantic tools and technologies. But again, it’s still very, very early days. And what some of the newer technologies have been trying to do is actually match up.

So for example, if you have a researcher that’s published a lot of research work in a repository but also published a data set or whatever in a repository that’s held in the states, they will try to link those things together. So you’re actually getting much more of an association, if you like, of all the materials that that researcher has had anything to do with. It might link you to the research groups, other publications, Amazon, where you can buy the book, all these things. So that is coming. That is the future. But that’s dependent, again, on good metadata beyond just basic text finding. So watch this space. It is not a fine art at the moment. There is no one panacea with this.

OK now, just to give you an example of how you can support back in your institutions. What the library staff here have done is create an area on the intranet particularly targeting module teams here. So module teams are those that have brought together, of course, goes through a very rigorous business appraisal, marketing, all sorts of things. When it finally gets approval, they’re looking at the pedagogy. They’re looking at how the students are going to be learning, what they’re intending to deliver as part of that course. And there’ll be a mix of content and a mix of skill building.

So what we’ve done here is actually give the module team some information and some links to help them when they’re trying to think about how they might use open educational resources in the context of developing, learning packages and materials. And it’s part of a whole range of things that we offer information and advice on. It is much more targeted at the internal audience. This is not available outside. But it’s proved useful. So if people want to use the materials in workshops within the module teams, then they can do. So we found it’s a great way to disseminate information and get the debate being engaged with at least in terms of how they can use open educational resources with the students.

And we have a very specific page here. What are they? Why use them? When can I find them? How do I evaluate them and examples of good practise? And what we look to here is the external environment as much as internal use as well. We’re trying to drive up awareness of how it’s been successful, where we’ve reused and readapted things in our internal practise, as much as of finding out what everybody else has been up to.

This tends to be far more of a trend towards publishing. People create and publish rather than take down and reuse and actually say to people how they’ve reused it. So there still is that missing link, really, between the actual doing of it and reflective practise. Because the value of this is not just simply in the publication of the contents, the finding of the content, it’s about what happens to it then. How did the students engage with it? How did you build value around it? And I think that is coming right back to the heart of behaviour and practise and teaching practise and reflective practise, around how this can change what you’re doing and change the relationships that you’ve got as well with your learners.

And where can I find out more? There’s lots of information available out there on the web. And if you haven’t already seen it, this is a great place to start. So if you go to the JISC info kit it’s got lots of really practical things that you can use. Again, there’s a lot of material that you can use simply marketing back at home, and also you’ve got some myths in here.

And there’s a section on finding OERs because I think a lot of people get frightened off with these open education resources as well. They tend to think that they’re giving away all their intellectual property in their content. And we can see that actually it can work an incredible level if you look at the practise for, say, MIT, mITX. They’ve just published all their teaching content online to the world. That is an amazing step forward because they see that the value is not just simply in the content. It’s about being engaged around the content and the learning and the relationships that you have with those people that are learning with it. So take some time in your session to go and have a look at that.

And there is a particular page in the info kit just on search engines. Now again, a word of caution because you’re going to be going to this page and actually choosing two of those in the activity in a minute. Many of these things have actually been project funded. So you will find that they are in prototype form, if you like. They’ve had the money. They’ve tested the idea. Money’s dried up. That’s it. You see what you get at the end of the project.

And that is another key concern in a risk around all of this is the sustainability of it. But we need to do all of these things, if you like, because it’s about testing with the users at the end of the day and what they feel is of value. You’ve seen the power of Google. I mean, it came from nowhere and it’s in everybody’s language now. It’s in everyone’s internet searching blood.

So the crowd drove that because it answered a need. It was simple, effective or perceived to be simple and effective. So, killer application. So many of these things take it as they are at the moment. Because if the money’s dried up and they haven’t had any more sustained funding after the end of the project, it’s just been left as it is.

They’ve also now got some more information as well on the whole area of repositories. Now, repositories are literally, they are digital libraries. These often contain not just metadata, like the old fashioned library catalogue where you had just the metadata and then links into the assets or whatever, wherever they were, whether they’re available in a licenced third party host space, or whether it’s a physical book on the shelves. So these repositories often hold the artefacts. So you don’t have to go searching after it or redirecting to it. It’s there.

But you need to understand the content types and what licences there are around these. And it can be quite a challenge figuring out what all these Creative Commons licences mean and how you can use it. And I don’t know how many of you heard of the Hargreaves report. Anybody hear about Hargreaves? Well, you’ll know that government wants to sort of open up some areas, particularly to education. And a lot of the people who own the intellectual property that publish the things want to sort of hang on to it because they’re afraid of might happen or open the flood gates. So the parliament is seeking consultation at the moment over this very thing. And The Open University has been quite vocal in feeding back amongst many. And you can see a lot of the responses that are available on the web.

So interoperability and integration, these are key things for us. Anybody who’s been involved in ebooks at the moment and knows what a huge headache they are, because unless people adopt certain standards again, you can’t find them or their interfaces all look different. Some you can download, some you can’t. Only print one page at a time. It’s a nightmare. So unless we really buy into the whole global vision of these things and adopt and use those standards in creating and in describing them, which is fundamental then to people being able to find them, then we’re not going to get very far.

So there’s a huge movement, ongoing again, to adopt these standards. And so people can do, what they say, harvest the metadata. And you can find things like Core which is a repository. A big aggregate of repositories which really target and will research publications. But they are, many of them, are open, fully downloadable for reuse, swap, and exchange that metadata because they’ve bought into a certain standard for these things for describing them. So you need to know more about these repositories, and Jorum is the key one for us in the UK or Great Britain.

The OER Foundation, again, loads of information on things which are developing in the field. Also, trying to push the whole agenda for the sustainability of these because there’s been a lot of money that’s gone into building the repositories, getting academics to publish things in there and share their work, and the momentum needs to carry on.

And it’s at a critical stage now because of the financial situation that everybody is in. And instead of locking down more because of the competition, it’s the very time we all need to be getting behind this and opening it out. Because that’s the only way we’re going to democratise access to education at the moment, is through this movement. So it needs to be viral. It needs to be very proactive. And us librarians have got a great part to play in that to pushing that agenda. Otherwise, none of us will be able to afford to be educated anymore the way that things are carrying on.

Again, WikiEducator, this was, again, in that whole kind of bought into the value of it WikiEducator, sharing things globally, trying to enable best practise to be shared, and there’s a whole section, again, on finding OER and different types of repositories including specific subject focused ones. And some of the higher education academy’s as well, they’ve actually got materials in there. They’ll either publish in the Jorum or they’ll support a repository in their own subject field.

But again, I come back to this. It’s still very difficult to search across these things at the moment. You’re still havng to really go out and find these things rather than being able to use an excellent search engine to reveal some of this stuff with you. Although, some of it’s now becoming more Googleable.

Again, on WikiEducator, a whole range here of search engines that you’ll find. So you can go and use that as well. When you come to your activity you can choose which site to land on, really, and do your investigation from. And on Cloudworks itself – have you heard about Cloudworks yet?

AUDIENCE:

It was mentioned.

PROFESSOR:

It was mentioned. OK, well again, just funded and the whole idea of this cloud was being able to put an excellent tools out there, share good practise. You can see there’s a whole range of different things at the time when I just did the screen capture incorporating competencies in search into OER projects. So if you’re thinking of digital literacy development again, then it’s much easier, in fact, to create those sorts of materials around the free stuff because there’s so many issues around using some of the licenced resources.

And there’s some fabulous collections out there of digitised material. Some of the museums and galleries are actually, now, releasing some of this content and licencing it for use in an educational setting. So Cloudworks is another good place to start. And again, you can see I’ve just done a very simple search on finding OER on it. And there’s all sorts of information you can find on their projects, international projects, as well. So in particular, country settings, trends, a whole range of things to keep up to date with what’s happening in the field.

And you will be going into lab space, I think, this morning as well. I think that’s the next thing on the agenda after this session. So you’ll have an opportunity, as well, to see what’s in there, what we’ve published in there. And again, some of the librarians have been involved in sharing some of the Safari work, actually publishing in that space within the subject areas as well.

We hope to do more in this field as we start moving and broadening out from just simple information literacy into digital literacy. There’s so much information everybody needs to arm themselves with here in terms of managing risk, identity management, privacy, all these key things which now affect all of us, and so that we can become really effective digital citizens.

And here we go. Here’s the Jorum. So this particular day when I did this screen capture, they’d actually published a Harvard referencing tutorial. Now, how many of us create these things? Every single institution has probably got its version of it. So how much effort is going into that? How much better would it be if actually we bought into this one? And if it doesn’t do quite what we want, adapt and republish.

But explain and understand why it’s not doing what you want it to do because at the end of the day everyone’s just after up-skilling the students. And sometimes we become very precious about these things, and if it isn’t invented here or got that stamp on it, then people don’t feel comfortable with it. When it’s, actually, what do we need to create that’s original, and what can we reuse and repurpose which is perfectly adequate and good enough to get the students from a to b? Or in some instances, dare I say, the staff from a to b as well. I mean, we’re all in this together. These are skills everybody needs to sort of get on top of.

And in terms of open tools, as well, around managing information, you have things like Mendeley now. Now, at the moment it’s still free. We don’t know where it might go to. Often these new companies that are starting up, they will give you a free tool like that. But that’s a great tool to use again for searching and finding things and managing the results that come back. And then, you just pay for premium services.

So it might not be necessary anymore to maintain our massive database subscriptions to EndNote and RefWorks, dare I say. So this stuff can really start changing the way that we work and making us share a great practise right across all our communities. Take your time to go and have a look at Jorum.

AUDIENCE:

What’s that name, Mendely?

PROFESSOR:

Mendeley. M-E-N-D-E-L-E-Y, Mendeley. These lads are working very, very hard in the semantic technology field as well. So they’re about associations and linking, linking data, linking metadata. And you’ll hear more and more and more about this linked data, linked and open data. The government, funny enough, are very much behind publishing the data in an open way. And there’s tremendous potential for sharing this data because we can build new services on this data.

You can envision them publishing a data set on declining industry in Newcastle or whatever, understanding where the new money’s coming from, where the regeneration’s coming from and then colleges and universities and schools getting behind that and understanding how they need to change their curriculum and answer their local job market needs. So these things can mobilise change if used effectively.

OER Commons, again, you can see there’s so much out there for you to find information on this. And this was quite popular as well with our students for a long time. The 100 best open educational resources on the web. So again, these have been peer reviewed. Not all of the stuff that you’re going to find in these repositories are subject peer review. But if they have, there’s usually some annotation around them. There’s usually some commentary around them. So a bit like your Amazon. So and so liked this. You liked that. So a bit of caution really. Review. It might not be right for you. But if it’s not right for you, don’t just throw it away. Improve it, and put it back.

OK now, the once – we’re going to move into the activity in a minute, and these are the very, basic criteria that we developed. We didn’t want to go have a Masters or a PhD in finding and evaluating OER. We were about practical. Practical things that people need to apply. So the key thing, around the accuracy. So as I say, not everything is subject to peer review, and it’s important that you know where it’s come from, the authenticity, if you like, of who’s published it so that you feel very confident that you can use it.

The reputation of the offer on the institution, some people are making their reputations, their academic reputations on this. Our own Martin Weller is a prime example. So the new kind of academic reputation is just as much about sharing and publishing out there in a global network as it is about publishing the latest book. And we may well find that things like the ref changes is in order to incorporate some of this stuff. And that, again, will bring a massive challenge to the publishing industry if that was to really blow the lid off of that.

The standard of technical production is becoming increasingly important. We all have a duty to ensure that the materials we create our accessible. So if they’re not accessible again and you have skills, why not make it accessible and republish if it’s not quite as it should be?

And fitness for purpose. This is the biggest one. I think the biggest stumbling block that we find. We can find all sorts of things but when you go in there it isn’t quite at the right level or it’s not quite night for whatever reason. Rather than throw it out, build on it, publish, put it back. Get this whole transactional economy going and to make sure that there are clear rights declarations on it. You’re only safe to reuse and repurpose if it states that. Everything else got a bit of a government health warning on it.

And quality considerations, there’s a whole range of information and things that just have actually published on this. They’ve poured hundreds of thousands of pounds into these initiatives because they’ve bought into the value of them. And they want to see that moving forward and sustaining. It would be a huge shame to see all this as excellent effort actually stall now at the very time where it needed to go forward.

So maybe we can take the next 10 minutes just to share some of the frustrations. So if I could ask some of you to volunteer, OK, to – do you need a mic?

AUDIENCE:

So, rather, not the frustrations just yet. I was really surprised because what I’m looking for is how to use Wikis for assessment, and I actually used the Jorum website. And what was very useful is that it actually came up with quite a lot of case studies, and it was really clear about copyright and how it could be used. The only thing that, again, what I think seems to be the issue in my own research previous to this is it’s not sustainable. Like if you click a link and they’ll say look at this link for the project proposal, the link’s broken. Or it’s behind a login screen. That’s what I found.

Whereas, but I did actually find it was very, very useful, and it’s very user friendly. So it is something I will be using again but I think that is the issue is it’s the sustainability of these resources. And I think while it’s beneficial to do a lot of research, sometimes it’s the issue of well, do I instead just create my own resources instead of having to research because researching sometimes actually takes longer than actually creating your own. So I think that’s what seems to be coming up quite a lot, really.

PROFESSOR:

Well, you imagine everybody’s doing this because of that. And then, so we’re wasting a lot of energy. So the way to try and do this is to drive up the quality of what we’ve got all ready, and adopt good practise ourselves when we’re creating them. And eventually, eventually it will drive up the quality. But it’s a slow process I think. It’s a slow sustained process to do that. And I think because reputations are being made out there in this field right now, so the more that people are adopting these things and adopting good practise and making sure that you think about all those things that we’ve talked about this morning, when you’re creating your own material then you’re helping others find it as well. So eventually, these things will become easier.

So did anybody else use the Jorum? Would you mind sharing what you found on there. Did you find what you were looking for or not?

AUDIENCE:

Yeah, I did. I mean, it’s a little bit easier because I went in just looking for something specific around [INAUDIBLE]. I found a great video from the University of Warwick so yeah, it was really useful. In fact I’ve saved it because I’m going to use it. And just going off the checklist, it’s all there. You’ve got the accuracy, authors, everything. So yeah, I found it very useful. And there’s also comments as well, and the comments are from different universities so the resources are up to date. Two authors actually said well, I’m doing a new one. So very good.

PROFESSOR:

So how did you use the Jorum before this morning?

AUDIENCE:

I’ve dabbled but not really gone into it in any depth. But now, I did like using it. It’s a very nice interface. Obviously, as possibly there’s standard barriers at the moment so you’d expect it to be easy to use. I also used, I forgot what it is now. The glue one for gluing things together. I just signed up for an account. I think that looks really nice and quite intuitive as well. So now, I’m quite impressed.

PROFESSOR:

OK, great. Did anybody us use anything different?

AUDIENCE:

Hi, I stayed away from Jorum because I thought that would be the number one once I tried. The university learning OCU and OER one. And it turned out to be a customised Google search engine. And I’m looking at it and basically just came up with a lot of YouTube videos. So I put one on and it was very good. After the beginning it said who it was and what university they were from and I hadn’t heard of either of them, which meant that I’d have to go and look up this person to see if I actually, what his reputation was. The video looked quite nice, and he was quite interesting because he was filming on the green screens. Every now and again things came up in the background and looked very, very nice. But at the end, he had a list of reference which were all too blurry to read. And he said, I will email you these references at the end. And I said well, that wouldn’t be very useful for me, that being. But the actual explanation of some of the theories that I was looking at looked quite nice. I think it’s something I would use, but it would sort of be like, have a look at that if you’re – but I just didn’t realised that it was doing YouTube videos, which was quite interesting so that was a quite nice thing to see.

PROFESSOR:

Did you come across anything on the licencing side?

AUDIENCE:

There was nothing on licensing. Well that was the thing because it was YouTube. I actually wrote a question. I presume YouTube you can show, but I’m only assuming that because everybody does. There was no, this is licenced to be shown. It was just there. So I kind just assumed that it would be all right to view, to show it to people, just because it was on YouTube. But there’s nothing about what the licence was or whether you could or not.

PROFESSOR:

So you take the risks, actually. I mean, most people who will decide to upload to YouTube are doing that. They’ve made the choice to share it unless they’ve put it in a private area because you can keep things in private accounts as well. So that if they have made it publicly viewable then, as you say, you make an assumption and take the risk that you can do that.

So could you, if it there had been a licence very clearly declared on there that you could have reused and repurposed it, would you be tempted to improve it or not.

AUDIENCE:

Well, I don’t really think it something you can repurpose because it’s literally, it looks like it’s some sort of lecture capture with additional things behind it. So it’s as is, really. You couldn’t, really, do anything with it. You just put it up and say, what’s that? In terms of repurposing, I think there is no repurposing. That’s the resource as is. Maybe, I suppose, you could chop it down. You could take a bit out because it was two linked videos. One was nine minutes, one was seven minutes. So I didn’t watch it all. I fast forwarded it through. But in terms of repurposing there’s not a lot I would do with it, really, apart from as is.

PROFESSOR:

But you found subject content was –

AUDIENCE:

It looked quite interesting and it did it in some interesting ways that wouldn’t have. So I found that quite interesting.

PROFESSOR:

Right. So it’s given you some ideas.

AUDIENCE:

Well, I think technically he has the technologies to do it because it was obviously filmed. He had also sorts of gizmos behind him so that’s quite – I know we don’t have an eye institution so that was quite good.

PROFESSOR:

But I mean, there’s lots of free tools out there as well for video production and you can do some really quite neat things just from some of the stuff you do in iMovie or Garage Band if you’ve got an Apple machine. So there are things you can do.

AUDIENCE:

[INAUDIBLE] underneath, when you click those, like expand, the information says something like standard YouTube licence which means you can’t actually reuse it. You can show it but you can’t reuse it or reedit it. But if it says that there’s another licence there and I think there’s a bit of video editor within YouTube itself that you can then – I think I did it and then repurposed.

AUDIENCE:

There’s also technologies you can just take the Mp3 files from the YouTube videos as well.

AUDIENCE:

Sounds like dangerous territory, though.

AUDIENCE:

That’s a good way of reusing.

AUDIENCE:

You’ve heard that.

PROFESSOR:

So has anybody got anything else that they’d like to share in that exercise.

AUDIENCE:

I used expect. A very challenging search engine. I ended up with hundreds of results. Only the first page and if I’m generous, I would say the second page could be relevant somehow. And I was searching for finding keywords, something finding keywords or finding information for assignments. Initially the search was very well, with some thousands of results. So I restricted it with advanced search and I found just hundreds. But as I said, only the first two pages were worthwhile, worth checking.

So I ended up with Open University material. Obviously, it is high quality. It has learning outcomes in terms of pedagogy. It doesn’t show how much time required. In terms of my situation, my environment, although it’s pretty similar, I would say that it needs prerequisite skills to understand this bit of searching for keywords. It’s not jargon-free so I would need probably a separate glossary of something or I should explain first the terms and then use these materials to expand it. Accessible, obviously, it is which is one of my main considerations always.

So it’s clear navigation, but it’s too wordy. Although it has bullets, it’s too wordy. And people tend to have this fatigue reading or scanning online, just scanning through. So even for me, I’m reading all day through apart from the hours I’m sleeping, very few. Even for me it’s difficult to go through all the content. When I say it, I’m scared. And I say, oh dear, it’s a lot. And I think this would be a burden to adapt it as it is. And it hasn’t any interactivity. So I need to do something to engage people with it to make it more interactive. But other than that, and of course, the good thing, licence, it’s very clear stated, very clearly stated. So that’s a very good thing.

But expert is very challenging, and I assume they have strong metadata behind. Again, I’m a bit obsessive about metadata. Even for small learning objects I’m creating, I apply Metadata. And not on the same learning object but also in a different form in case my colleagues need to make them searchable. And I have applied a scheme Dublin Core with something else. So I assume Expert has metadata as well. Why is it so difficult to – what people are putting in this metadata, how they describe the source, I don’t get it really. Because I’m getting hundreds of results, and none of them are relevant.

PROFESSOR:

Well again, this is tricky, isn’t it? It’s in turn how it’s mining, basically, how it’s actually mining in matching. This is what the difference is with something like the basic Google search which is just mining for free text words or whatever. And some of the more semantic approaches which are targeting very much the link relationships between those fields and what they’re describing. We just haven’t cracked it yet because you can get a great clustering engine, but again, if can go and have a look at those things, they’re still not quite relevant or quite what you want. So the semantic technology can support all that topic clustering but the actual relevance of the results that form in the those clusters, it still might not be what you want. It’s, I suppose, the limitations of the computing against the human brain, and what’s it aspiring to.

I mean, librarians will tell you they’ve been doing reference interviews for donkey’s years. And the amount of where you travel from the beginning of the reference interview from where you actually get to what somebody is actually after because often people don’t know either. Or they have a vague idea about what they’re looking for. So try to support that kind of activity in this is fraught with all sorts of challenges, really. But at least for the basics set of Dublin Core it’s better than nothing at all, so you can still search. I mean, one of the things that the Jorum does as well, it does support browsing. So most of you, I guess today, have taken the keyword search approach. You’ve just typed in words into the search box. But you can do that as well with good repositories that have used a whole range of vocabularies, and that will also help with relevance. So, you can use standard vocabularies to structure those things and thesauri to help with that.

And a lot of the semantic technologies are actually, that’s what they’re doing. They’re trying to link the thesauri. So they’re ontologies, what they call ontologies. So that’s why they’re hoping that eventually this will bring more accurate clustering rather than just text mining, free text mining. Anybody got anything else?

AUDIENCE:

I deliberately chose things I haven’t heard about. So I first tried Discover Ed. And I was looking for curriculum redesign in higher education. And all the resources I found where about the use of OERs rather than links to OERs. So I had to give up that search engine. So I then decided to play it safe and go to OER Commons, and I changed to looking for assessment and feedback in higher education. And I have found something gut I just wanted to share the text with you if I can find it again. So I found an item called assessment strategies to provide feedback for post secondary education. I had to learn my term higher education wasn’t relevant. But the tags are blackboard, distance education, faculty development, and online teaching. So and the resource just seemed to be one page describing an exercise with no information about what the exercise was. So although it’s tagged for attribution and no strings attached, I’m not really sure how I can use this. So I have not had a very successful morning.

AUDIENCE:

Hi. I did a similar sort of thing. I thought I try something that I hadn’t used before. So I looked at the OER Commons and I was a little bit confused about what this no strings attached was. Because, obviously, you get very used to seeing the Creative Commons, and then underneath the no strings attached you see the Creative Commons so you’re kind of aware of what that is. But then, the other thing that I was finding on a lot of the different search engines, so I tried OER Glue as well. The same resources come up, and the same resources that don’t quite match my need are coming up time and time again. So I felt a little bit like I was perhaps wasting my time a little bit going through these search engines which is one of my little bug bears.

AUDIENCE:

Is it to take existing OERs and somehow stick them together?

PROFESSOR:

Yes, I think because of this problem of clustering content and being able to find, because things are in different repositories. So it’s trying to, at least, bring them together. So if you’re searching in one topic assessment or peer assessment, and you’ve got things in a whole pile of places, it is trying to link them, cluster them together and bring them to you. I think the ideal, the holy grail, I think, they’re after is a little bit very similar to RSS feeds have done for us. So rather than you going and searching for all the news, you sign up for things, and then it comes to you. And then, you use a news aggregator.

So you go into something like Netvibes and then you’ve got all those channels in one screen, if you like. So rather than you having to rush off trying to find all of these differing things, it’s coming to you.

AUDIENCE:

And you can publish into those channels.

PROFESSOR:

Yes, because RSS is very transmissional one way. They’re not interactive in that way. But it’s so hard to keep on top of this, as well. Because obviously, at the moment you’re searching in a particular time and a place. So it’s bounded by whatever you can find at that time. And the beauty of something like RSS, again, is that if a news item gets published on your topic, you then hear about it. So some of the repositories, I know are looking at doing things like that. So if somebody put something in, you get an alert. So if somebody’s put something, the latest artefact on your topic –

But we have got such a long way to go with this. It’s still very much in its infancy. And I think given the current climate, what I would encourage everybody to do is just to make Jorum as best as it possibly can be. And if you are keen on, which hopefully you are and have bought into the whole values system, really, around creating, publishing, repurposing, sharing, and very open practise, and you don’t have resources back or resources that are getting increasingly tighter backup base, use the Jorum.

It’s there. It’s a structure. It’s an infrastructure. There’s help and support there, and avoid all the costs, if you like, in the overheads of maintaining institutional repositories for these types of artefacts. And let’s just drive up Jorum and make it one of the best in the world. And that’s what I would encourage everybody to do. And to think about the metadata. Metadata, we know, it’s expensive as well in terms of its creation and the time it takes to create the metadata. Particularly, if you’re going to adopt standards, rigorous standards and maybe create a lot of metadata.

So, there’s trade-off there about how much metadata is good enough, and maybe how much standard spill description metadata we need and how much semantic link services we can develop through actually turning with the thesauri, our controlled vocabulary, all these ways we describe things, the ontologies, into machine computable ways of matching things together. So it’s a bit of both, really.

And of course, that helps with searching and discovery. But the other value of the metadata is in the long term maintenance the artefact. So just like everything, we need to clean out the cupboard now and again. It’s not just a case of building and pushing and building and pushing and building and pushing. We do need to review what’s there. We have a responsibility to clean out, decide what we want to keep which is of value longer term, or things which are more ephemeral that could be disposed of and deleted.

So it’s an active process of managing these materials, and that way we’ll avoid profligation as well, of content, which maybe of devious quality and build up more solid repositories. Certainly, core content. Things like we were talking about earlier on with Harvard referencing. We do not need millions of iterations of Harvard referencing. That’s a classic case in point, where one is good enough for everyone to use.

So I think we could be working much more effectively together. And then that releases more time to be more inventive and creative with some of this other stuff that we find and could maybe make it better. Certainly, has everybody heard of Common Craft? That’s a very good site for videos for like instructions and how to do things. Common Croft.

So if you haven’t heard of it, I’m just thinking if you want to do a rapid video or something, and it can show you how to create a video with some really basic tools. So there’s lots and lots and lots of really lovely super stuff out fact there which is still free and reusable both in terms of tools, technology, and then in terms of the content that we’ve been talking about today in these repositories.

End transcript: Finding and evaluating OERs

Finding and evaluating OERs

Lisbeth Levey highlighted the issues of finding relevant OER in a 2012 report published by the Commonwealth of Learning. Her personal account of trying to locate and use OER suitable for supporting a postgraduate university course in agriculture within an African context demonstrates:

• the richness and breadth of material available
• the considerable challenges associated with finding relevant material that is openly licensed for re-use
• the crucial need to frame searches well in order to retrieve good quality material.