Doing research and evaluation. A plea for a qualitative approach to academic excellence
Centre for Applied Linguistics, Warwick, Coventry
CEMS, EHESS, Paris
Principal Investigator “The Discursive Construction of Academic Excellence”
Nowadays, researchers are subject to a regime of evaluation, which reflects a change from statist toward entrepreneurial modes of academic governance. While academic decision-making is expected to recognise quality and performance, the standards and criteria according to which research is assessed are subject to controversy (Slaughter and Leslie 1997; Münch 2011).
Against this background, I will argue that there is no one way of doing research and that evaluation needs to account for the many different practices which constitute what we call research. While the evaluation of academic excellence is sometimes equated with abstract numeric representations of research performance (think of the ISI citation indices, the Shanghai rankings, research assessment exercises etc.), it is also something researchers do and achieve in their everyday academic life.
This is why we need qualitative evaluation, which reflects on the very practices that it claims to evaluate. For qualitative evaluation, research quality is the product of complex social dynamics in which research quality is constructed by researchers as classifying experts in their specific settings. Yet qualitative evaluation comes with a price – the time and resources good evaluation needs. If decision-makers want to have a durable and long-term impact in the scientific communities, they need to understand what drives researchers in their struggles for academic excellence.
What is evaluation?
Generally speaking, we can speak of evaluation when researchers are classified in terms of quality, success or achievement, individually or collectively. Evaluation can be done before research takes place, e.g. for research projects or in the German excellence initiative, or after research has taken place, e.g. in job recruitments or the UK research assessment exercises. Evaluation is often understood as a more or less formal, systematic and transparent ranking procedure. In this sense, evaluation is part and parcel of decision-making arrangements through which positions are filled and resources are distributed.
Yet we can also see evaluation as something that researchers do in their everyday academic life (Latour 1987; Camic, Lamont and Gross 2011; Bourdieu 1988), when they communicate with each other through publications or talk at conferences for instance (Flowerdew 2001; Hyland 2009). As a result of these on-going discursive practices, some researchers come to be perceived as especially important over time. Let’s think of “great classics” such as the Newtons and Kants, the Keynes and Bourdieus, who have become leading figures in their scientific communities. We can also think of the many subtle distinctions drawn among the peers of a community, some of which come to be considered to be more important, central, legitimate than others. These perceptions are the more or less unintended products of classifying processes in large academic populations. Unlike formal evaluation procedures in institutions, these practices generally do not follow any given recipe or replicable standard; they do not follow explicit and transparent rules and procedures; they are not coordinated by any decision-making instance; they do not lead to unambiguous, numeric results that can be easily used for decision making. While institutional evaluation usually tries to assess research on the basis of common standards, everyday academic classifications testify to a great deal of heterogeneity with which researchers need to cope practically in their social lives.
Research and evaluation as classifying practices
As practices through which researchers are classified, everyday research and institutional evaluation do not necessarily follow the same logics even though both can have far-reaching consequences for the careers pursued by researchers. Everyday classifying practices have engendered a few canonical representatives within global scientific communities. Yet while strong symbolic hierarchies often separate the “classics” of the field from the rest, the living peers of a community often cannot be assessed in such a clear-cut way. The distinctions researchers produce of and among themselves are indexed on the specific web of relationships in which they try to carve out their niche. As members of certain scientific communities, they therefore do not always deal with each other in terms of higher-lower distinctions. They are usually more obsessed with questions of relevance, presence and legitimacy, which are highly indexical of their specific fields and domains.
Institutional evaluation, by contrast, categorises researchers within institutions or organisational units for decision-making purposes (Clark 1983; Musselin 2005). Institutional decision-making points to more or less formal procedures of selecting researchers for hierarchical positions in the higher education system and its institutions, most importantly through job recruitments but also through a wealth of other decision-making practices which distribute coveted resources among the researchers such as status (such as institutional job security), money (e.g. for research projects), time (vis-à-vis other activities such as teaching), help (assistants), rights (e.g. the participation in panels and the direction of research units). Decision-making is an integral part of research since researchers need not only to become legitimate members of specialised scientific communities (through “spontaneous” classifications) but they also need to be placed in institutional status hierarchies (through formal selection procedures). Thus academic research careers can hardly succeed without researchers getting involved in both worlds of specialised knowledge and organised power at the same time.
“Spontaneous” classifications of everyday research and the institutional classifications in institutions mutually support each other. The researcher usually needs a symbolic place in the scientific communities to be recruited in a higher education institution. At the same time, she or he needs some sort of institutional place to gain attention in the scientific community. Researchers failing in one or the other world are likely to be perceived as deficient, i.e. as a “mere” manager without recognition in the community or as a “mere” man of letters without institutional backing. Both the symbolic and the institutional classifications are inextricably interlinked as the former would be fuzzy and ambivalent without the latter whereas the latter would turn out artificial and arbitrary without the former. No researcher can avoid participating in this positioning game in which researchers are involved in from the very beginning – the doctoral student who aims at getting formal recognition through his PhD dissertation as well as the established professor who needs to prove his presence in the field through his publications. In the world of research, therefore, there can be no specialised knowledge production without organised power just as well as there can be no institutional decision-making without specialised knowledge. In this sense research can be considered as a discursive practice dealing with different, even contradictory social logics in the world of specialised knowledge and of organised power.
As a formal classifying practice, evaluation is part and parcel of what research is about in terms of power/knowledge. Indeed, it would be a mistake to abstract the practice of research from what researchers achieve socially. Research is not only about abstract ideas and theories but also about how researchers relate to each other. Research is indeed driven by researchers whose fundamental challenge is to be and exist for and with others in the worlds of knowledge and power, e.g. as somebody with a certain recognised expertise (an archaeologist of the late Roman period, a sociologist of early childhood or a philosopher with a background in the Frankfurt School of Critical Theory) or as somebody with a certain place in academic networks (a disciple of X or a collaborator of Y, as well connected with this journal and distant from that school), as somebody with a certain role in higher education institutions (as a teacher of an undergraduate class in early modern history and a member of the scientific council of a research organisation, with a high or low status), as well as somebody who acts in a number of non-academic contexts (such as a spouse in the family, an expert in the media or a councillor in politics). And with all these activities going on at the same time over their entire biographies the researchers need to prove their practical “excellence” as researchers by achieving a certain coherence in the way they are placed and positioned in the social world of research.
From the point of view of the researchers, academic excellence, therefore, can be considered as the success in achieving one’s symbolic-institutional place vis-à-vis others. It is the implicit or explicit social struggles about how researchers are classed as unique and coherent somebodies that pushes them to elaborate ever new ideas and that gives certain ideas their value as more “interesting”, “scientific” or “excellent” than others. This is what drives researchers in their everyday academic life. This is why they invest time and energy in certain problems and questions but not in others. This is what makes them value certain colleagues more than others. This is why certain researchers are considered to be more successful than others.
The positioning imperative in everyday academic life: the DISCONEX project
How can we account for academic excellence as the unintended effect of the practices among many researchers? We need to go down to the ground level of academic everyday life, take the perspectives of researchers and have a look at the real academic practices, which is what we do in our DISCONEX project (“The Discursive Construction of Academic Excellence. Classifying Researchers in Text-Processing Practices”, 2013-2018, Warwick/EHESS). In this discourse analytical project, we investigate how researchers succeed in finding their institutional and non-institutional positions through the classifying practices of academic discourse, such as publishing articles and books, exchanging with people at conferences, participating in committees, writing reports and reviews for journals and research organizations, setting up research networks, supervising students’ research activities, applying for funding and projects.
The question is why the researchers engage in all these frenzied activities most of which do not seem to pay off, at least not in a direct economic sense. Indeed, researchers do not always calculate; they are practical experts in building up and defending their symbolic-institutional subject position in academic discourse (see Meyer 1980). This is why professors spend their weekends on writing reports and reviews for this journal or that research organisation, why postdocs are willing to teach for little or even no money at all. It is the promise to be paid back in reputation or to get an institutional label that pushes them to do all these things against their economic interests. While they cope with their “daily flow of craziness”, researchers try to carve out and consolidate their place among countless other researchers who already have their place in their local departments, in their national peer networks and in the global communities. Indeed, for the researchers there is no more existential pressure than to build up and improve their positions and thus to become somebody for others. And this is what makes them so active and creative as researchers.
Let me dissipate some misunderstandings that may emerge. If I say that research is driven by the positioning imperative, I am not saying that researchers are just involved in rhetorical games of labelling each other without caring about “real” research, such as exciting new ideas and scientific truths. On the contrary, researchers are in pursuit of real research and the positioning imperatives is precisely what makes them want to produce excellent research. Therefore the excellence of research is inextricably linked with the capacity of researchers to establish and improve their symbolic and institutional positions in academic discourse. That’s why research is such a difficult practical task for which the researchers mobilise all their resources and their imagination. Academic excellence therefore is not so much about being faster, better or higher than others but about achieving more or less coherence in a messy world. Research is not like an athletic contest where somebody comes in first; it is more like a cooking jam session where you need to creatively improvise a tasty dish out of heterogeneous ingredients which happen to be at hand.
Indigenous excellence and the positioning imperative
The problem researchers need to respond to is that there is no overarching rationality which they or evaluators can appeal to. Researchers are always busy dealing with conflicting norms and rationalities, which is why they are under considerable stress and see so many “illogical” behaviours happening. Nor is there justice in research. Only few manage to write their symbolic and institutional positions on the map of the scientific communities while many remain invisible. Nor are knowledge producers better suited than managers to run universities. Most researchers act in both roles anyway and both knowledge production and decision-making are an integral part of what the very practice of research is about.
It is the positioning imperative that makes research a social world through and through. It needs to be emphasised that the world of really existing research testifies to a great deal of inequality (the 10 canonical figures who everybody knows versus the many who are never cited at all), exploitation (everybody contributes to the positioning dynamics in research but only few attract broad visibility and recognition), injustice (extra-academic resources such as economic and cultural capital often make a crucial difference) and contingence (the same strategy may work in one case but not in the other).
Yet at the same time there is no more powerful guarantee for doing excellent research than the positioning imperative. It is the need to become somebody, to construct a coherent academic life against many different, even contradictory constraints in the various situations and at various points of their careers that obliges them to respect the models of academic excellence in their communities.
Today most of research funding still goes into fields which are not auto-regulated by the positioning imperative – and Horizon 2020 is no exception if one thinks of the very significant amounts of money given to do research for profit-driven enterprises (the parts of the budget called “innovation”). Equally significant amounts of money are given to research for policy-makers (such as most of the collaborative projects directly funded by the European Commission). There may be reasons to give out much money for product development, political consulting and social engineering. Governments may prefer fungible technicians and versatile administrators over professional researchers with their recognized expertise and creative capacity. Yet, one needs to be aware that the impact of the enormous resources spent on policy-driven research has been negligible in global scientific communities and that’s why the status of this more applied type of knowledge remains fragile. In the long run, applied knowledges tend to be less legitimate than basic research because they are not maintained by the practice of researchers who realise their knowledge socially with their unique academic careers as it were.
The researcher-centred culture of excellence in the social sciences and humanities
This is certainly not the place where I can account for disciplines outside the social sciences and humanities. Nor can I account for the large variety of disciplinary cultures within the social sciences and humanities. However, the positioning imperative is probably nowhere else as generally accepted as in the social sciences and humanities. These disciplines testify to a specific culture of academic excellence, a culture which is centred on the researcher. Unlike scientific cultures characterised by laboratories or large experimental arrangements, the division of labour among researchers in the social sciences and humanities is usually low. Conversely, their knowledge tends to be rather holistic and multi-dimensional as they need to set up and run through every aspect of their research projects whose problems tends to be defined and solved by one person.
We can cite a number of characteristic features which seem to testify to a culture of excellence in the sciences and humanities centred around the researcher as a person of knowledge: theoretical culture and intellectual breadth, authentic research projects pursued over very long periods, research activities which seamlessly tie in with many different facets of their everyday existence, the important role of teaching for the formation of critical and autonomous spirits.
What makes the social sciences and humanities a model for all disciplines is the special emphasis on the excellence of researchers as singular, unique and autonomous knowledge creators with a sense for the many different dimensions of complex human problems. And that is what they can do best: to enable real people to develop their potentials and achieve their unique place with others. Research in the social sciences and humanities is centred on researcher-oriented models of academic excellence. These disciplines are therefore highly instrumental as instigators of social innovations for they practice knowledge production as the production of knowledge producers.
From researcher-centred excellence to researcher-centred research policy
Let me conclude with three policy recommendations which may apply to the social sciences and humanities just as to many other research fields where excellence is based on the positioning imperative.
Researcher-centred excellence needs people-centred funding.
Firstly, How can funding schemes contribute to academic excellence? To make a difference, they need to tie in with the long-term positioning practices in the communities. Funded research projects need to make sense for the large project of the researcher, which is to achieve a coherent academic career. Projects need to be designed in a way so that they can contribute to the researchers’ long-term positioning strategies. Research projects can be detrimental to the making of excellent research if they force researchers into artificial interdisciplinary consortiums or if they require them to prove their impact in contexts which are not theirs. Excellent research needs people-centred actions such as Marie Curie.
Short-term funding has little impact on global communities.
Secondly, How can research organizations leave lasting traces in global scientific communities? Considerable short-term money is spent on activities with little or no long-term effects on the communities. For the communities, short-term funding is soft currency, which will lead to many skilfully written research proposals but not necessarily to research activities which make a difference in the long term. If new research fields are to emerge, researchers need a market of lifetime positions. New communities are built on lifetime positions, which constitute the identity of a field and are the hard currency for researchers. There are many examples of fashionable new fields which immediately stop existing once short-term money goes elsewhere. Therefore, if research organisations want to increase their impact, they need to think of how to help create lifetime positions.
Good evaluation needs resources for real experts.
Thirdly, decision-making is as complex, contradictory and important a practice as any other practice in the world of research. Decision-makers usually need to negotiate many different criteria and expectations which call for their practical sense. You may want to recruit somebody because you think she did good work in the past or because you think he will do good work in the future. You may want to privilege research proposals which respond to a demand within or without the scientific community, which require more or fewer participants. Numbers and indicators cannot solve such decision-making dilemmas. They may have even adverse effects if you think of applying journals’ impact factors which systematically disqualify new but small communities. Evaluation is cheap but useless to the degree that it wants to represent academic excellence by isolated numbers. What is needed is good evaluation which can account for the coherence of the research project and the unique contribution of the researcher with her or his specific background. Since research is embedded in complex discursive practices, good evaluation needs considerable resources and time to understand where new places are being articulated and where positions are bubbling up which deserve funding.
The European Research Area as a heterogeneous institutional space
As a multileveled and heterogeneous set of discursive practices, research needs an institutional space where academic standards and practices are in healthy competition. It would therefore be adverse to apply one single model of excellence to the whole area and to monopolise evaluation in few institutions in a top-down manner. The best strategy for European policy makers should be to strengthen bottom-up approaches. The European Research Council (ERC) has taken exceptional efforts in evaluating every research proposal individually by real experts from the communities. Unlike policy-driven programmes, the ERC has indeed made a real difference in the world of research and has become an example for many research organisations in the world.
- Bourdieu, Pierre (1988): Homo Academicus. Cambridge: Polity Press.
- Camic, Charles, Michèle Lamont and Neil Gross (eds.) (2011): Social Knowledge in the Making. Chicago: University of Chicago Press.
- Clark, Burton R. (1983): The Higher Education System: Academic Organization in Cross-National Perspective. Berkeley: University of California Press.
- Flowerdew, John (ed.) (2001): Academic Discourse. Harlow: Pearson.
- Hyland, Ken (2009): Academic Discourse: English in a Global Context. London, New York: Continuum.
- Latour, Bruno (1987): Science in Action. Milton Keynes: Open University Press.
- Meyer, John W. (1980): “The Effects of Education as an Institution.” American Journal of Sociology 83: 55-77.
- Münch, Richard (2011): Akademischer Kapitalismus. Über die politische Ökonomie der Hochschulreform. Frankfurt am Main: Suhrkamp.
- Musselin, Christine (2005): Le marché des universitaires. France, Allemagne, Etats-Unis. Paris: Sciences Po.
- Slaughter, Sheila and Larry L. Leslie (1997): Academic Capitalism. Politics, Policies, and the Entrepreneurial University. Baltimore, London: The Johns Hopkins University Press.
About the author:
Education: PhD in discourse analysis and sociology, Paris 12, France / Magdeburg, Germany, 2003
Prior Work: Professor (W1) in “Sociology of Higher Education” at the Department of Sociology, Johannes-Gutenberg University Mainz, Germany, 2009-2012
Assistant Professor/ Lecturer (“Wissenschaftlicher Assistent”) at the Department of Sociology, Otto-von-Guericke University Magdeburg, Germany, 1999-2009
Postdoc researcher at Université de Picardie, Amiens and the Center for the study of discourses, images, texts, writings, communication, 2004-2006
Current Employment/Responsibilities: Leader of Research Group “The Discursive Construction of Academic Excellence” at the University of Warwick and Ecole des Hautes Etudes en Sciences Sociales (EHESS)
Research Professor of Discourse at the Centre for Applied Linguistics, University of Warwick, Coventry, UK
Areas of Research Interest:
Discourse, language, society
Higher education, science, intellectuals
Political communication, public sphere, mass media/culture
Comparative cultural studies (e.g. France, USA, UK, Germany, Russia)