Execution of Jan Hus, anonymous woodcut, 1415On a hot autumn day in Rome not long ago, I crossed the vast expanse of St Peter’s Square, paused momentarily in the shade beneath a curving flank of Bernini’s colonnade and continued a little way beyond to a Swiss Guard standing impassively at a wrought-iron gate. He examined my credentials, handed them back and saluted smartly. I hadn’t expected the gesture and almost returned the salute instinctively, but then realised it was intended for a cardinal waddling into the Vatican from behind me.

Just inside the gate, at Piazza del Sant’Uffizio 11, stands a Renaissance palazzo with a ruddy ochre-and-cream complexion. This is the headquarters of the Congregation for the Doctrine of the Faith, whose job, in the words of the Apostolic Constitution Pastor bonus, promulgated in 1988 by Pope John Paul II, is “to promote and safeguard the doctrine on faith and morals throughout the Catholic world”. Pastor bonus goes on: “For this reason, everything which in any way touches such matter falls within its competence.” It is an expansive charge. Every significant document or decision emanating from anywhere inside the Vatican must get a sign-off from the CDF. The Congregation has been around for a very long time, although until the Second Vatican Council it was called something else: the Sacred Congregation of the Holy Office. From the lips of old Vatican hands, one still hears shorthand references to “the Holy Office”, much as one hears “Whitehall”, “Foggy Bottom” or “the Kremlin”.

But before the Congregation became the Holy Office, it went by yet another name: as late as 1908, it was known as the Sacred Congregation of the Universal Inquisition. Lenny Bruce once joked that there was only one “the Church”. The Sacred Congregation of the Universal Inquisition was the headquarters of the Inquisition – the centuries-long effort by the Church to deal with its perceived enemies, within and without, by whatever means necessary, including the most brutal ones available.

The palazzo that today houses the Congregation was originally built to lodge the Inquisition when the papacy, in 1542, amid the onslaught of Protestantism and other noxious ideas, decided that the Church’s intermittent and far-flung inquisitorial investigations needed to be brought under some sort of centralised control – a spiritual Department of Homeland Security, as it were. The Inquisition had begun in the Middle Ages, to deal with Christian heresies, and been revived in Iberia, under state control, to deal with Jews and Moors. Pope Paul III considered the task of his new papal Inquisition so urgent that construction on the basilica of St. Peter’s was suspended and the labourers diverted so that work could be completed on its headquarters. At one time the palazzo held not only clerical offices but also prison cells.

The Congregation for the Doctrine of the Faith inherited more than the Inquisition’s DNA and its place on the organisational charts. It also inherited much of the paper trail. The Inquisition records are kept mainly in the palazzo itself, and for four and a half centuries that archive was closed to outsiders. Then, in 1998, to the surprise of many, the Vatican decided to make the archive available to scholars.

Any archive is a repository of what some sliver of civilisation has wrought, for good or ill. This one is no exception. The archive may owe its existence to the Inquisition, but it helps explain the world that exists today. In our imaginations, we offhandedly associate the term “inquisition” with the term “Dark Ages”. But consider what an inquisition – any inquisition – really is: a set of disciplinary procedures targeting specific groups, codified in law, organised systematically, enforced by surveillance, exemplified by severity, sustained over time, backed by institutional power and justified by a vision of the one true path. Considered that way, the Inquisition is more accurately seen not as a relic but as a harbinger.

The opening of the archive at the Vatican is one more development in what has, during the past several decades, become a golden age of Inquisition scholarship. Until the appearance of Henry Charles Lea’s magisterial History of the Inquisition of the Middle Ages, in the late 19th century, most writing about the Inquisition had consisted of bitter polemics by one side or another. In recent years, using materials newly available in repositories outside the Vatican, and now including those of the Holy See itself, historians throughout Europe and the Americas have produced hundreds of studies that, taken together, revise some traditional views of the Inquisition.

To begin with, the notion of “the Inquisition” as a monolithic force with a directed intelligence – “an eye that never slumbered”, as the historian William H Prescott once phrased it – is no longer tenable. Rather, it was an enterprise that varied in virulence and competence from place to place and era to era. “The Inquisition” remains a convenient shorthand term, but there were many inquisitions. Another finding of modern research is that, insofar as their procedures were concerned, Inquisition tribunals often proved more scrupulous and consistent than the various secular courts of the time. Of course, the bar here is low. Modern scholarship has also revised the casualty figures. Some older estimates of the number of people burned at the stake by the Inquisition range to upwards of a million; the actual number may be closer to ten thousand – perhaps two per cent of those who came before the Inquisition’s tribunals for any reason. Whatever the number killed, the Inquisition levied penalties on hundreds of thousands of people, and the fear and shame instilled by any individual case rippled outward to affect a wide social circle. Little wonder that the Inquisition has left such a lasting imprint.

But from between the lines the new scholarship has some larger lessons to offer. The Inquisition can be viewed as something greater and more insidious than an effort pursued over centuries by a single religious institution. It was enabled by the broader forces that brought the modern world into existence, and that make inquisitions of various kinds an inescapable feature of modern life. Inquisitions advance hand-in-hand with civilisation itself.

It’s a troubling conclusion but an inescapable one. Here’s the central question: why did the Inquisition come into being when it did? Intolerance, hatred and suspicion of one group by another had always existed. Throughout history, these realities had led to persecution and violence. But the ability to sustain a persecution – to give it staying power by giving it an institutional life – did not appear until the Middle Ages. Until then, the tools to stoke and manage those embers of hatred did not exist. Once the tools do exist, inquisitions become a fact of life. They are not confined to religion; they are political as well. The targets can be large or small. An inquisition impulse can quietly take root in the very systems of government and civil society that order our lives.

The tools are these: there needs to be a system of law, and the means to administer it with a certain amount of uniformity. Techniques must be developed for conducting interrogations and extracting information. Procedures must exist for record-keeping, and for retrieving information after records have been compiled and stored. An administrative mechanism – a bureaucracy – is required, along with a cadre of trained people to staff it. There must be an ability to send messages across significant distances, and also an ability to restrict the communications of others – in a word, censorship.

The Inquisition was built on all of these capabilities. The new universities brought order to canon law, defining heresy with precision and therefore defining who was “inside” and who was “outside”. The Church bureaucracy became professional; papal chanceries turned out perhaps 300 letters in 1200 and 50,000 a century later. Inquisitors learned how to organise their documents and make them searchable; a person’s testimony to one tribunal could be known to another tribunal decades later. Interrogation manuals, like the famous Practica written by Bernard Gui, were drawn up to instruct inquisitors on how to question the accused – the tricks to use, the psychology to employ. The resemblance to the modern manuals for military personnel and intelligence operatives is hard to miss. As a supplement to interrogation, torture became systematic – subject to rules, perhaps, but rules that proved elastic, as they always do.

So the tools were new. The precipitating mindset was age-old: the conviction that one is absolutely right. Cardinal Alfredo Ottaviani, the prefect of the Holy Office through most of the 1960s, adopted as his motto the Latin phrase Semper idem – “Always the same”. The inquisitors shared an outlook of moral certainty. In a world of moral certainty, the unthinkable becomes permissible. The sanctity of private conscience is no longer deemed inviolate. Techniques for ensnaring the innocent in scenarios of scripted guilt become increasingly sophisticated. A Franciscan inquisitor once confided to King Philip IV of France, in the early 14th century, that if Saints Peter and Paul had appeared before his tribunal, he had no doubt that the techniques he employed would be able to secure their convictions.

It all sounds very medieval. But it’s not merely medieval. Scholars may debate whether there truly is such a thing as a “totalitarian” state, and what its characteristics are, but the desire to control the thoughts and behaviour of others – joined to a belief that God or history will render an approving judgement – underlies much of the sad narrative of the past hundred years: the police states, the dirty wars, the ethnic cleansing, the internments, the renditions, the Red Scares, the fatwas, the special prosecutors, the electronic surveillance, the encroachments accomplished in name of national security.

One day, in the archives, I came across two polished wooden boxes, resembling old library card-catalogue drawers, with hinged wooden tops. The boxes rested one above the other on a wooden rack. Inside each box, well-worn index cards ran its length, their upper edges velvety from use. It was the catalogue of the Index of Forbidden Books, the very last one, issued in the 1940s. (After 400 years, the Index was formally discontinued in 1966.) The boxes seemed so antique; the musty smell evoked yesteryear. But was this effort any different, fundamentally, from China’s Great Firewall?

What separates an inquisition from other forms of intolerance is its staying power. It gets institutional support. It goes on and on. Today, the basic elements that can sustain inquisitorial behaviour are more prevalent and entrenched, by many orders of magnitude, than they were in the days of Bernard Gui or Tomás de Torquemada. None of them will decline in significance in the years ahead. They will only become more powerful.

Looking at the Inquisition, one sees the West crossing a threshold from one kind of world into another. Persecution acquired a modern platform – the advantages afforded by a growing web of standardised law, communications, administrative supervision and controlled mechanisms of force. It was run not merely by warriors but by an educated elite; not merely by thugs but by skilled professionals. Every subsequent outbreak of persecution, political or religious, has been abetted by these same forces. They ensure that the basic trajectory of repression will always look remarkably the same. They suggest why persecution is so difficult to stop. And they help explain why the Inquisition template has translated so easily from the religious sphere into the world of secular governments and secular ideologies. Through the lens of the Inquisition we can glimpse the world we inhabit now.

When the Inquisition’s palazzo was built, in the mid-16th century, the Pope ordered words to be carved in a marble scroll over the front door – a kind of mission statement – establishing the building as a “bulwark against heretical depravity”. The words are gone now, removed by French troops during Napoleon’s occupation. It’s easy enough to remove some words – harder to erase a legacy.

Cullen Murphy’s new book God’s Jury: The Inquisition and the Making of the Modern World will be published by Allen Lane in January.