For more than a century our politics has been dominated by three parties, thanks in no small part to a first-past-the-post system that stacks the deck against new challengers. Confronted with this almost insurmountable barrier, those who dream of remaking British politics often remain just that – dreamers. More than 400 challengers to the "big three" are registered with the electoral commission, and virtually all will sink without trace.
This is one reason why the rise of the UK Independence party is so remarkable. It is the most successful new party in a generation: the first since the Social Democratic party in the 1980s to attract double-digit support in national polling. In fact, Ukip's revolt is more impressive as the SDP's earlier challenge was orchestrated by people who already sat at the top table of British politics. Ukip has come from below; a genuine insurgency from outside the established party system. It might not yet have won a seat in Westminster, but it has attracted more than one voter in 10 and upended the agenda. This is an extraordinary achievement for a party that for much of its 20-year history has been comically disorganised, eccentric, and paralysed by infighting.
But Ukip is also remarkable because of the extent to which its support is misunderstood. Nigel Farage and his party, we are repeatedly told, are a byproduct of unresolved Conservative divisions over Europe; a second home for disgruntled Tories who are pushed into its arms by their anger at Brussels and hostility towards a Conservative prime minister who supports gay marriage and climate change. Ukip, in short, is a Tory problem.
This conventional wisdom is understandable given that the party began as a pressure group of anti-Maastricht rebels, but it is no longer accurate. In fact, Ukip raises as many questions for Labour as for the Tories.
"This is all Fleet Street," Farage said during one of our interviews at Ukip's headquarters near Bond Street. "This is their obsession and they can't get out of it. But the numbers are perfectly clear: there is now a huge class dimension to the Ukip vote." Farage was drawing on private polling and his experience on the doorstep. He might be regarded as a gadfly and bon vivant, but he has a keen understanding of his party's working-class appeal.
Farage's observations about Ukip's support closely match what we have found over the past year while probing the backgrounds, beliefs, concerns and motives of almost 6,000 of these rebel voters. Much of this directly challenges everything we thought we knew about the roots of this revolt. Forget David Cameron's unpopularity among grassroots Tories; forget the furore over EU migrants; forget single-issue concerns over the EU or the charisma of Farage.
To truly understand Ukip's appeal you need to go much deeper. The roots of this revolt can be traced back over decades. Divides in the social and economic experiences of voters have appeared, their values and priorities have been widening, and a new electorate of "left behind" voters has grown up. These voters are on the wrong side of social change, are struggling on stagnant incomes, feel threatened by the way their communities and country are changing, and are furious at an established politics that appears not to understand or even care about their concerns. And it is these left-behind voters who have finally found a voice in Farage's revolt.
Farage is no catch-all populist; his appeal is concentrated in specific groups and is utterly alien to others. Ukip has virtually no support among the financially secure and the thirty- and fortysomething university graduates who dominate politics and the media. Support is weak among women, white-collar professionals and the young. Ethnic-minority voters shun the party totally.
Make no mistake, this is a revolt dominated by white faces, blue collars and grey hair: angry, old, white working-class men who left school at the earliest opportunity and lack the qualifications to get ahead in 21st-century Britain. That Ukip's core voters are middle-class Tories animated by the single-issue of Europe is the biggest myth in Westminster. In fact, Ukip is the most working-class-dominated party since Michael Foot's Labour in 1983. They struggle financially, worry about the future, and loathe the political class, not just Cameron and the Conservatives.
Don't think of Ukip as just a party; think of them as a symptom of far deeper social and value divisions in Britain. Farage is winning over working-class, white male voters because they feel left behind by Britain's rapid economic and social transformation and left out of our political conversation; struggling people who feel like strangers in a society whose ruling elites do not talk like them or value the things which matter to them.
This should ring loud alarm bells on the left. In a time of falling incomes, rising inequality and spending cuts, such voters should be lining up behind the party that traditionally stood for social protection and redistribution. Instead, they are switching their loyalty to a right-wing party headed by a stockbroker and staffed by activists who worship Thatcher. Those who are getting hit hardest by the crisis and austerity are turning not to Labour, but to Farage for solutions.
One reason for this is that for those left behind, politics is no longer about economics. These voters are not backing Ukip because of their economic concerns; they are backing the party because they see Farage as representing an identity and set of values they cherish but do not see expressed anywhere else. These voters have been left behind not just by wider trends, but the rise to dominance of a university-educated, professional middle-class elite whose priorities and outlook now define the mainstream.
The dramatic nature of this shift is often missed because it has been accomplished over decades. Yet in only 50 years Britain has gone from a society where working-class voters with little education decided elections to one where such voters are now only spectators, and the crucial and decisive battle is fought between middle-class graduate candidates seeking middle-class graduate votes. When Harold Wilson was elected in 1964, working-class voters outnumbered professional middle-class voters two to one; by 2010 the professional middle classes had a four to three advantage. Both Tony Blair and Cameron have sought to revive their party's prospects by appealing to the rising middle classes. Neither has shown much interest in the struggling, left-behind voters, and since 1997 these voters have made their feelings about being marginalised clear: turnout from these groups has collapsed, and dissatisfaction with politics has increased. Ukip's deputy leader, Paul Nuttall, captured this sense of exclusion in a 2013 speech: "In the days of Clement Attlee, Labour MPs came from the mills, the mines and the factories. The Labour MPs today go to private school, to Oxbridge, [then] they get a job in an MP's office."
These changes have been accompanied by a major transformation in the values that dominate the country. Across Europe it is no coincidence that radical right parties similar to Ukip win support from the same working-class voters, and accomplish this by targeting the same issues: national identity; immigration; Europe; and resentment of political and social elites. This is because there is now a deep and growing divide in the values of the left-behind and the professional middle-class mainstream.
The radical right in Europe is making a similar pitch, and for the same reason: the emergence of a large section of the electorate who feel the world they grew up with and valued is fading away, that what is replacing it is alien and threatening, and that no one in the mainstream understands their desire to turn back the tide of change. You cannot just ignore these voters – you need to have a conversation.
When thinking about Ukip, those around Ed Miliband must think beyond the next 12 months to a time when Labour may be in power with a small majority, or as part of a coalition. The party will then face many of the same challenges as the current government: an ageing population; straining public services; high migration from poorer EU states; persistent inequality; and the economic and fiscal overhang of the worst crisis for 80 years. By 2015 Ukip will be a known alternative. After European, local and general elections, it will have consolidated its support and be well positioned to make inroads in Labour-dominated areas by winning votes from those who will inevitably feel disappointment with what a Labour government can achieve.
At this point Labour will be exposed to serious and sustained competition for support in its northern, working-class fortresses. The largest concentrations of core Ukip supporters are not found in Tory seats in the shires but in Labour fiefs like Miliband's Doncaster North. We identified the 10 most Ukip-friendly seats in the country, and eight are Labour. Strategists on the left need to ask themselves – are your local councillors and activists in these areas ready for the first serious challenge they have ever faced? They may be laughing now, as Ukip drive their Tory opponents to distraction, but after May 2015 the men with purple rosettes may be knocking on Labour's doors.
