From my perspective, the biggest barrier to Open Access is the symbiotic relationship between academic journal publishers, and the current academic rewards system, in which (number of) publications in high-profile or high-impact journals to a large extent determines whether researchers will get or keep jobs, and progress their careers.
Other posters here have mentioned the detrimental effects of APCs, open access costs, and so on — and these are definitely barriers to open publication of scientific research. But open access itself is not the problem. A researcher could write an article, have it go through the same process of peer review by other scientists, and publish it on an institutional or subject repository without any need for a commercial publisher to get involved, and this would be an Open Access article. It would have undergone the same peer review process currently performed by other researchers on behalf of journals for free, but would negate the need for Open Access and other fees to be paid to the journal.
However, this is unlikely to happen while the academic rewards system remains as it is, using publication in a recognised journal as a proxy for evaluating the quality of a researcher's work. Until all universities commit to judging researchers' work on its merits, rather than on its place of publication, we will be stuck with the same unsatisfactory situation regarding Open Access.