Introduction

It is fairly common for modern society to describe itself by using a part to talk about the whole. Modern society is content, for example, to call itself an “information society”, or an “industrial society”. The feeling conveyed in both cases is that, through these self-descriptions, society grasps the essential aspect that should explain the modernity of modern society. But it does not take long to realise that what is actually being attempted is to evade the complexity of the description by means of drastic simplification. What would the “industrial” character of modern religion be? And how would a society devoid of “information” function? If these self-descriptions work well in terms of public opinion, it is because they confirm themselves through everyday life. One wonders, however, whether this evidence is enough when the aim is to make a contribution to the sociological theory of modern society.¹

The latest vogue is to describe modernity as a “search engine society”.² Several moral and legal concerns have been quickly added to this description, as follows. Search engines are opaque. Their performance is highly selective, so the image of reality they offer is inevitably distorted. Being subject of manipulation, search engines can manipulate information-and, consequently, users’ opinions. Like writing first and the printing press later, search engines are therefore looked upon with suspicion by the very society that uses them and which, at the same time, admits that it cannot do without them.

If we disregard moral concerns and take for granted the technical assumptions that allow search engines to function,³ the use of these indexing systems becomes sociologically interesting because it shows, first of all, the dependence of modern memory on archives. The second aspect that deserves attention is that search engines presuppose the mechanisation of processes that allow access to archive content. My hypothesis is that both these aspects are the result of socio-cultural evolution. If the primary function of social memory is to forget⁴ and evolution favours functions, then evolution must favour forgetting.

Archives actually allow much more to be remembered than what can be stored in individual consciousnesses because the former relieve consciousnesses of the burden of reminiscence. Indexing systems also work well when they manage to discard almost everything except for the small amount of content that is relevant to the user from time to time. No one would learn a library catalogue by heart. A metadata system therefore makes it possible to forget not only the data contained in the archive, but also the metadata itself.

From this viewpoint, archiving systems are forgetting machines.⁵ This is precisely why archives favour the use of memory to produce information. The function of social memory, after all, is precisely that of organising access to information. As Niklas Luhmann points out, the crucial problem with this function lies in the organisation and not in what actually happened.⁶ In the early-modern forms of semantic indexing, organisation was implemented by means of categories, i.e., subject headings. The relations between these categories were set up in such a way as to create a self-referentially closed system representing a certain order of knowledge.

With the increasing complexity of knowledge in modern society, order (which had previously been a solution to the problem of memory) becomes a problem. Indeed, archives are structurally oriented towards an open future. What is new, however, does not always fit into the system of cross-references that constitute the established inner order of the archives. One would have to undo the order and redo it, at an enormous expenditure of cognitive energy. Complexity thus becomes a condition for testing structural changes whose function is to reproduce complexity by means of selections. In post-modern memory,⁷ this is done through the mechanisation of archives.

Mechanising Archives

At the beginning of the 20th century-and thus before archiving systems started to be automated-, many scholars had pointed out that semantic indexing systems are problematic in several ways. First, because the rules of distinction adopted by professional indexers are not clear to users and often appear unjustified.⁸ For those unfamiliar with these rules, retrieval of documents is slow and laborious. Categorical rules of inclusion, moreover, are at the same time rules of exclusion. Consequently, if the subject headings devised by users do not coincide with the index terms used by indexers, users run the risk of not finding a relevant document that is indeed stored in the archive but has been categorized differently. Finally, the structure of semantic relations that forms the indexing system forces the repetition of pre-established associations and inhibits the search for new associations that might have a high information value for prospective users.⁹

Underlying the inability of semantic systems of indexing to cope with the complexity of ever-growing knowledge, there is, more specifically, an issue of time. John Lund and Mortimer Taube pointed out in this regard that the real obstacle was the “permanence” of library classification systems and, consequently, “the absurdity of providing for all future knowledge by subdivisions or expansions of present systems”.¹⁰

Already at the beginning of modernity, scholars argued that systematic (or methodical) orders of knowledge are very useful when it comes to remembering. Order-by definition-favours repetition and enables one to move quickly from one thing to the next without getting lost in the infinity of meaningful references that each element could reactivate. But when it comes to exploring the unknown in search for information, order becomes an impediment. That is why already in the 17th century many scholars experimented with ‘loose’ orders of knowledge to be preserved in secondary memories such as commonplace books and filing cabinets.¹¹ In the encyclopaedia, this search for loose order had led to a preference for alphabetical order of entries over methodical order. For Ephraim Chambers, the advantage of a “promiscuous” order of knowledge where “numbers of things are thrown precariously together” was exactly that “we sometimes discover relations among ’em, we should never have thought of looking for”.¹²

Underlying this possibility of knowledge storage was not simply a preference for disorder, but rather a preference for an order of a different kind. If memorable knowledge is entrusted to an external archive rather than being stored by the individual memory of those participating in the communication, it becomes possible to duplicate the order. In which order this knowledge is stored in the archive is relatively unimportant. Therefore, it is also possible to experiment with very abstract and conventional orders, such as alphabetical or numerical ones. What is important is that the archive is equipped with a retrieval system that enables the users to quickly find everything relevant to their query. This is precisely what indexing systems are for.

When archives are mechanised, indexing systems must give up the capacity to process meaning. Machines process data instead of meaning, and cannot even specify the meaning of their own processing. The semantic problem is therefore compensated for with statistics. In the first experiments carried out in the mid- 20th century, the assumptions were quite simple.¹³ For example, it was assumed that the frequency of a term within a text was an indicator of its importance. The proximity of frequent terms and their respective position in sentences were further indicators that could be used to index the document, but also to automatically extrapolate an abstract without the machine actually having to read the text.

Since the 1960s, retrieval systems have become more sophisticated. In a seminal contribution to probabilistic indexing, Melvin Maron and John Lary Kuhns focused on the calculation of closeness.¹⁴ These calculations could deal with index terms as well as retrieved documents, and could be defined both in terms of semantic relations and statistical relations.¹⁵ The machine could treat index terms and documents as points to be connected. Within these spaces of connected points, the machine could implement several heuristics, that is, rules to move in the maze.

In the index space, for example, the machine could calculate closeness and distance of several index terms, which in semantic indexing systems are usually connected through cross-indexing, i.e., with cross-references such as “see” and “see also”. Through a calculation of the numerical weights assigned to the index terms, the machine could automatically decide which index terms to see and see also. If a user entered, for example, the query terms “information” and “theory” to find out whether there is a theory of information, the machine would autonomously decide to see “mathematical theory of communication” and to see also “Shannon”, even though the user had not formulated any of these query terms. In this way, as Maron and Kuhns put it, a probabilistic “association of ideas” could be mechanised.¹⁶

User Adaptiveness

Since its inception, the mechanisation of archives has been confronted with an unprecedented and somewhat paradoxical problem: understanding users’ need for information without being able to understand users. In cybernetic terms, it was a matter of enabling the machine to deal with the enormous variety of queries without forcing users to formulate their queries in a pre-categorised and pre-classified way. The archive therefore had to be built not as an invariant machine that always gives the same answers to the same questions, but as a machine capable of adapting to the user.¹⁷ This forced a radical rethinking of indexing systems.

In modern culture, the assumption was still that knowledge could be ordered independently from the observer. Users therefore had to adapt to the (unique) established order and explore the map of knowledge in search for relevant documents. The epistemological breakthrough that took place with the mechanisation of archives consisted in overturning this perspective; it is the order of knowledge that must adapt to the users. There is no longer a one-size-fits-all map of knowledge that allows one to explore a territory that, by definition, does not change, even when one moves within the territory (just as space does not move when one moves in space). Mechanical memory is a performative memory insofar as query results depend on users and their way of exploring knowledge repositories.

In the 1960s, this paradigm shift coincided with the experimentation with new forms of data storage that renounced the rigidity of traditional cataloguing systems and opted instead for loose structures such as the database. The advantage of this looseness lay in the possibility of using the data entered by users as input from which the machine could adapt its output. The archive could thus function as a conjecturing machine.¹⁸

Underlying this transformation of memory into a conjecturing system was the implicit assumption that the real black box between user and machine is the user. Of course, all archives are black boxes for users. But users are not interested in clearing up the black box in order to infer, from its regularities, a supposed internal order. The machine, on the other hand, is interested in just that. If users are intransparent, it is still possible to discover, from the regularities emerging from their search behaviour, what their information need probably is. This is why the machine needs to train itself continuously through the data generated by user queries. For the machine, these data are clues.

The words used in the query are already a first clue, of course. The query triggers the inverted index (i.e., the first fundamental structure of the search engine) and enables a drastic selection to be made in the network of references that constitutes the internal order of the archive. To this first clue many others are then added. The aim, as it always is when it comes to conjecture, is to reduce uncertainty as much as possible and increase the probability that the machine will produce results relevant to the user.

Among the most effective clues are the user’s reactions to the results produced by the machine.¹⁹ The basic idea is relatively simple. If, from the list of results the machine produces as output, the user clicks on the third result instead of the first or second, the machine learns that, for the user who made the query, the third result is probably more relevant than the first two. This information is used by the machine to adjust its ranking and offer more relevant future lists of output.

The feedback is obviously circular. Just as users train the machine with their search behaviour so that it will provide relevant results, likewise the machine trains the users with its output so that they will formulate their queries in a relevant manner. The machine, in fact, learns not only from the successes but also from the failures. This occurs through a constant improvement of the indexing system by addition of metadata that the machine produces through the interaction with the data produced by the users. It is useful for the machine to consider not only correct data (when users formulates their query in a relevant manner, so that there is an instant match between query terms and index terms), but also incorrect data. By adding users’ query formulation errors to its metadata, the machine can not only anticipate users’ real information needs (in the typical form “Maybe you were looking for...”) but can also learn to give the right answer to the wrong query.²⁰ The indexing system-the search engine- thus becomes not only dynamic, but actually capable of learning from the results of the interaction between user and machine.²¹ This interaction takes the form of a true cybernetic system.

The Relevance of Relevance

A common characteristic of cybernetic systems is the deviation-counteracting function.²² To define deviation, one must first establish a difference (such as the temperature of the room in the well-known case of the thermostat) against which a difference can be observed as deviation. In the case of archives, the assumption is that this difference is relevance.

Since the first mechanisation projects of archives in the mid- 20th century, computer scientists stated that the main goal of automation should be “to save a prospective reader time and effort in finding information in a given article or report”.²³ This goal could only be achieved if the machine was designed to discriminate relevant documents from irrelevant ones. But for machines incapable of understanding the meaning of their own operations and exposed to the almost unlimited vagueness of queries from unknown users, achieving this goal appeared to be an almost impossible task.

This is why it is perhaps no coincidence that the issue of relevance arose at the beginning of the second half of the 20th century.²⁴ Of course, relevance had always been an underlying problem in the relationship between user and archive, but only when the processes of accessing information started to be automated did we realise that the real problem is how to discard everything that is not relevant to respective queries. The machine produces, in this regard, problems for which only the machine could provide a solution.

A first advance in not only a technical but also an epistemological sense occurs when, as we have seen, the relationship between user and machine is conceived of as a form of functional coupling;²⁵ that is, user behaviour is taken by the machine as a clue to make conjectures and restructure the indexing system. The machine thus does not immediately react to user queries, as in a stimulus/response relationship. Rather, each reaction to user queries goes through a reaction of the machine to itself. Therefore, adaptation is preserved in this cybernetic relationship if structures can be changed, akin to evolutionary processes.

The problem of relevance has not been approached with sufficient consideration for the specificity of this cybernetic relationship. In retrospect, it can be said that the concept of relevance was not “well understood” and its definition “remains problematic, if not impossible”.²⁶ This is mainly because, as with information, it was thought to be an intrinsic property of documents stored in archives and that it could be measured as such. The two classical measures of recall and precision correspond to this idea.

The concept of recall refers to the ratio of relevant documents retrieved by the machine in response to a given query to the total number of relevant documents available. The concept of precision, on the other hand, refers to the ratio of relevant documents retrieved to the total number of documents available in the archive.²⁷ In the first case, it entails an implicit calculation on how many relevant documents the machine has overlooked. In the second case, what is calculated is the machine’s error in attributing relevance to documents that do not have it. In both cases, however, relevance is assumed to be an objective property of documents independent of the respective reader.

Like the science of information retrieval, the calculation of relevance was also based on Claude Shannon’s mathematical theory of information. In this regard, Maron and Kuhns pointed out that “the problem of explicating the notion of relevance (which is the basic concept in a theory of information retrieval) is similar to that of explicating the notion of amount of information (which is the basic concept of information theory)”.²⁸ In both cases, the observer (i.e., the user) is not taken into account as a precondition of the measurement, with the consequence that the measured object is somehow ontologised.

A notable exception to this approach was that of Don Swanson who already in the 1970s suggested replacing the notion of information retrieval with that of document retrieval.²⁹ For Swanson, this meant that a judgement on the relevance or irrelevance of a retrieved document could only be made afterwards, from the user standpoint. The notion of relevance lost, from this perspective, much of its supposed pertinence. According to Swanson, relevance could only be a measure of the satisfaction of individual information needs. And as such, it could only be detected in the circularity of the interplay between user and machine, and not a priori from the documents stored in the archive.

Conclusion

One of the most intriguing consequences of the automation of archives is the abandonment of the one-ranking-fits-all principle. The idea that the universe of knowledge can be organised from outside is replaced by the idea of self-organisation that includes the observer in the observed reality. This also means that the right ranking does not exist.³⁰ As soon as the ordering is oriented towards information instead of memory retrieval, any project of permanent ordering of knowledge becomes obsolete.

Underlying the distinction between static ranking and dynamic ranking³¹ is the idea that without first exploring the state of the observer, it is not possible to rank relevant documents. However, the fact that one renounces a pre-established order of knowledge does not mean that one renounces bringing order to the web of knowledge. It does, however, change the meaning of such an exploration.

In the rhetorical warehouse (thesaurus), the rhetorician roamed the rooms in search of the acting images that would reactivate the memory of what had been forgotten. The purpose of this search was also, in a certain sense, its end: once the image had been met, uncertainty could only lie in the semiotic process that served to implement the memory. In the archives, on the other hand, the search takes the form of a conjecture that is in principle unlimited. It proceeds, like all conjecture, by trial and error³² and can treat each result as a stimulus for further research. An ultimate result, for this very reason, does not exist. Users stop not when they have found what they were looking for, but when they feel that their need for information is provisionally satisfied.

Underlying post-modern memory is ultimately a profound restructuring of the relationship between past and future. If this form of social memory serves to produce information and information does not last, then post-modern memory is contingent by definition.³³ Information is, in fact, always based on the difference between known and unknown. Only based on what it already knows can the observing system be surprised by something it did not know previously. However, every piece of information changes the observer’s state and restructures the possibility of producing further information. As Luhmann says, “every new situation renews, with the then newly organised information, the difference of past and future as well”.³⁴ Therefore, not only is the past always different, but the future that can be expected from that past is also always different.

An archive-based memory is a memory centred on the future precisely because it is confronted with the unstoppable growth of knowledge and the need to adapt indexing systems to this growth.³⁵ Corresponding to this adaptation is the counter-intuitive idea that the best order is a lack of order. If what the user is looking for is information-and much information lies in the relationships between elementary units of knowledge (and not in the elements as such)-, then what the archive must preserve is a network of relationships that can be explored at will starting from the information provided by the user. In this way, every interaction between user and machine becomes an opportunity to produce complexity through selection.

In evolutionary terms, search engines are thus not simply the result of increasing complexity. Rather, search engines are a technical device with which society can manage its social memory by increasing reducible complexity. From this standpoint, the problem should therefore not be seen in the possibility of search engines manipulating our perception of reality. Rather, the problem is how search engines can maintain their ability to discriminate between relevant and not-relevant information under conditions of increasing complexity. In order to remember more, archives must also be able to forget more. The question is therefore not which past should be remembered, but how the ability to forget can be preserved in the future. The problem is certainly technical, but the consequences are social. The latter should thus be subject of a theory of the evolution of social memory.