Fifteen years ago, a research group called The Fraunhofer Institute announced a new digital format for compressing movie files. This wasn't a terribly momentous invention, but it did have one interesting side-effect: Fraunhofer also had to figure out how to compress the soundtrack. The result was the Motion Picture Experts Group Format 1, Audio Layer III, a format you know and love, though only by its acronym, MP3.

The recording industry concluded this new format would be no threat, because quality mattered most. Who would listen to an MP3 when they could buy a better-sounding CD? Then Napster launched, and quickly became the fastest-growing piece of software in history. The industry sued Napster and won, and it collapsed even more suddenly than it had arisen.

If Napster had only been about free access, control of legal distribution of music would then have returned the record labels. That's not what happened. Instead, Spotify happened. ITunes happened. Amazon began selling songs in the hated MP3 format.

How did the recording industry win the battle but lose the war? They crushed Napster, but what they couldn't kill was the story Napster told.

The story the recording industry used to tell us went like this: "Hey kids, Alanis Morissette just recorded three kickin' songs! You can have them, so long as you pay for the 10 mediocrities she recorded at the same time." But Napster said: "You want just the three songs? Fine. You want every cover of Blue Suede Shoes ever made? Help yourself. You're in charge."

The people in the music industry weren't stupid, of course. They just couldn't imagine that the old way of doing things might fail. Yet things did fail, in large part because, after Napster, the industry's insistence that digital distribution be as expensive and inconvenient as a trip to the record store suddenly struck millions of people as a completely terrible idea.

Once you see this pattern – a new story rearranging people's sense of the possible, with the incumbents the last to know – you see it everywhere. First, the people running the old system don't notice the change. When they do, they assume it's minor. Then that it's a fad. And by the time they understand that the world has actually changed, they've squandered most of the time they had to adapt.

It's been interesting watching this unfold in music, books, newspapers, TV, but nothing has ever been as interesting to me as watching it happen in my own back yard. Higher education is now being disrupted; our MP3 is the massive open online course (or mooc), and our Napster is Udacity, the education startup.
We have several advantages over the recording industry, of course. We are decentralised and mostly non-profit. We employ lots of smart people. We have previous examples to learn from, and our core competence is learning from the past. And armed with these advantages, we're probably going to screw this up as badly as the music people did.

A massive open online class is usually a series of video lectures with associated written materials and self-scoring tests, open to anyone. As we learned from Wikipedia, demand for knowledge is so enormous that good, free online materials can attract extraordinary numbers of people from all over the world.
Last year, an online course in artificial intelligence from Stanford, taught by Peter Norvig and Sebastian Thrun, attracted 160,000 potential students, of whom 23,000 completed it, a scale that dwarfs anything possible on a physical campus. As Thrun put it, "Peter and I taught more students AI than all AI professors in the world combined." Seeing this, he quit and founded Udacity, an institution designed to offer moocs.
The size of Thrun and Norvig's course, and the attention attracted by Udacity (and similar organisations like Coursera, P2PU, and University of the People), have many academics worrying about the effect on higher education. In a New York Times OpEd, Mark Edmundson of the University of Virginia focused on the issue of quality, asking and answering his own question: "[C]an online education ever be education of the very best sort?"

Now you and I know what he means by "the very best sort" – the intimate college seminar, preferably conducted by tenured faculty. He's telling the story of the liberal arts education in a selective residential college and asking "Why would anyone take an online class when they can buy a better education at the University of Virginia?"

But who faces that choice? Are we to imagine an 18-year-old who can set aside $250K and 4 years, but who would have a hard time choosing between a residential college and a series of moocs? Elite high school students will not be abandoning elite colleges any time soon; the issue isn't what education of "the very best sort" looks like, but what the whole system looks like.

I was fortunate enough to get the kind of undergraduate education Edmundson praises: four years at Yale, in an incredible intellectual community, where even big lecture classes were taught by seriously brilliant people. Decades later, I can still remember my art history professor's description of the Arnolfini Wedding. But you know what? Those classes didn't create genuine intellectual community. They were just great lectures: we showed up, we listened, we took notes, and we left, ready to discuss what we'd heard in smaller sections.
And did the professors also teach our sections too? No, of course not; those were taught by graduate students. The large lecture isn't a tool for producing intellectual joy; it's a tool for reducing the expense of introductory classes.

Cheap graduate students let an institution lower the cost of teaching the sections while continuing to produce lectures as an artisanal product, from scratch, on site, real time. The minute you try to explain exactly why we do it this way, though, the setup starts to seem a little bizarre.

Every university provides access to a huge collection of potential readings, and to a tiny collection of potential lectures. We ask students to read the best works we can find, whoever produced them and where, but we only ask them to listen to the best lecture a local employee can produce that morning. Sometimes you're at a place where the best lecture your professor can give is the best in the world. But mostly not. And the only thing that kept this system from seeming strange was that we've never had a good way of publishing lectures.

The top 50 colleges on the US News and World Report list only educate something like 3% of the current US student population. The entire list, about 250 colleges, educates fewer than 25%. The very things the US News list of top colleges prizes – low average class size, ratio of staff to students – mean that any institution that tries to create a cost-effective education will move down the list. This is why most of the early work on moocs is coming out of Stanford and Harvard and the Massachusetts Institute of Technology. As the academic and games designer Ian Bogost has said, moocs are marketing for elite schools.
Outside the elite institutions, though, the other 75% of students in the US —over 13 million of them—are enrolled in the 4,000 institutions you haven't heard of. When we talk about college education in the US, these institutions are usually left out of the conversation, but Clayton State educates as many undergraduates as Harvard. Saint Leo educates twice as many. City College of San Francisco enrols as many as the entire Ivy League combined. These are where most students are, and their experience is what college education is mostly like.

The fight over moocs isn't about the value of college; thousands of institutions you haven't heard of provide an expensive but mediocre education. For-profit schools like Kaplan's and the University of Phoenix enrol around one student in eight, but account for nearly half of all loan defaults, and the vast majority of their enrollees fail to get a degree even after six years. Reading the academic press, you wouldn't think that these statistics represented a more serious defection from our mission than helping people learn something about artificial intelligence for free.

The fight over moocs isn't even about the value of online education. Hundreds of institutions already offer online classes for credit, and half a million students are already enrolled in them. If critics of online education were consistent, they would believe that the University of Virginia's Bachelor of interdisciplinary studies or Rutgers's Master of library and information science degree are abominations, or else they would have to believe that there is a credit-worthy way to do online education, one moocs could emulate. Neither argument is much in evidence.

That's because the fight over moocs is really about the story we tell ourselves about higher education: what it is, who it's for, how it's delivered, who delivers it. The most widely told story about college focuses obsessively on elite schools and answers a crazy mix of questions: How will we teach complex thinking and skills? How will we turn adolescents into well-rounded members of the middle class? Who will certify that education is taking place? How will we instil reverence for Virgil? Who will subsidise the professor's work?
Moocs simply ignore a lot of those questions. The possibility they hold out isn't replacement; anything that could replace the traditional college experience would have to work like one, and the institutions best at working like a college are already colleges. The possibility moocs hold out is that the educational parts of education can be unbundled. Moocs expand the audience for education to people ill-served or completely shut out from the current system.

In the US, an undergraduate education used to be an option, one way to get into the middle class. Now it's a hostage situation, required to avoid falling out of it. And if some of the hostages having trouble coming up with the ransom conclude that our current system is a completely terrible idea, then learning will come unbundled from the pursuit of a degree just as songs came unbundled from CDs.

If this happens, Harvard will be fine. Yale will be fine. But Bridgerland Applied Technology College? Maybe not fine. And Kaplan College, a more reliable producer of debt than education? Definitely not fine.
Udacity and its peers try to answer some new questions, questions that the traditional academy – me and my people – often don't even recognise as legitimate, like "How do we spin up 10,000 competent programmers a year, all over the world, at a cost too cheap to meter?"

Udacity may or may not survive, but as with Napster, there's no containing the story it tells: "It's possible to educate a thousand people at a time, in a single class, all around the world, for free."
Once you imagine educating a thousand people in a single class, it becomes clear that open courses, even in their nascent state, will be able to raise quality and improve certification faster than traditional institutions can lower cost or increase enrolment.

College mottos run the gamut from Bryn Mawr's Veritatem Dilexi (I Delight In The Truth) to the Laboratory Institute of Merchandising's Where Business Meets Fashion, but there's a new one that now hangs over many of them: Quae Non Possunt Non Manent. Things That Can't Last Don't. The cost of attending college is rising, while the premium for doing so shrinks. This obviously can't last, but no one on the inside has any clear idea about how to change the way our institutions work while leaving our benefits and privileges intact.
In the academy, we lecture other people every day about learning from history. Now it's our turn, and the risk is that we'll be the last to know that the world has changed, because we can't imagine – really cannot imagine – that the story we tell ourselves about ourselves could start to fail. Even when it's true. Especially when it's true.

This is an extract from a longer article. Read the original here.
• Clay Shirky is associate professor at New York University and a fellow at the Berkman Center for Internet and Society