Access, engagement, then impact: factors affecting decision-makers’ use of research

In this piece, Pauline Zardo uses an analysis of The Conversation Annual Survey to consider what influences policy decision-makers' use of research. Increasing access to research is the key. Pauline's piece was first published on the LSE Impact Blog. You can find her original post here.

Studies have shown that while many factors affect decision-makers’ use of research, evidence of what particular engagement actions actually work is very limited. Indeed, increasing access to research appears to be the only intervention to reliably increase use. The Conversation is a free-to-access research communication platform designed to be accessible to general audiences. Pauline Zardo reports on The Conversation Annual Survey, which offered a rare opportunity to examine whether and how readers, including decision-makers, were using research.

Academics are increasingly encouraged to engage end-users in research, based on the assumption that increased engagement will lead to increased research impact. Universities have a tendency to think about engagement in relation to securing research investment and partnerships by engaging industry, government, or donors. In Australia, this is reinforced by the way research performance is assessed. The engagement metrics applied in Australia’s first Engagement and Impact Assessment are based on industry, government, and other commercial investments in research.

Systematic reviews of the factors that affect decision-makers use of research support this, with collaboration and face-to-face relationships identified as particularly important. However, when we dig deeper into the evidence base a more complex picture emerges. A systematic review of interventions aimed at increasing decision-makers use of research showed that, more often than not, when we test these factors through well-designed interventions, even evidence-based assumptions of “what works” fail in practice.

The only two intervention types that showed reliable evidence of increasing research use were actually related to the ability to access research evidence. These were:

• “Facilitating access to research evidence, for example through communication strategies and evidence repositories”
• “Building decision-makers’ skills to access and make sense of evidence…such as critical appraisal training programmes”.

These results were dependent on the intervention design also simultaneously building decision-makers’ opportunity, capability, and motivation to use research.

These findings show that there is significant capacity for motivated, capable decision-makers to independently use research, outside of research partnerships, if they can access relevant research when the opportunity arises. Reflecting the evidence base, the 2021 UK Research Excellence Framework will only include papers that have been made open access. That means UK university performance will be based only on research that can be accessed by industry, government, and the community. In Australia, we are yet to follow suit. The percentage of “free-to-access” papers is counted, but has no impact on the assessment of a university’s performance.

The inability to access research while working in government was the light-bulb moment that led to my PhD – what’s the use of having the ability, skill, and will to use research to inform decision-making if I can’t access it? I recently attended a seminar with academic colleagues I hadn’t seen since moving back to government earlier this year. Some were quick to state their assumption that government decision-makers are uninterested in using research. I was quick to respond that over half of all research that could potentially inform policy is locked behind a paywall. I shared with them how, in my first few weeks in government without university-paid-for research access, there were several occasions when I couldn’t access a paper relevant to urgent work and therefore that paper did not inform decision-making.

These experiences are why, when I was offered the opportunity to have input on The Conversation Annual Survey in 2016, I jumped at it. The Conversation – a free-to-access research communication platform designed to be accessible to general audiences and related to current issues reported in an everyday news cycle – addresses many factors reported to affect research use. It has 3.8 million monthly readers, with a further 35 million reached through republication.

The Conversation Annual Survey presented a rare “natural experiment” opportunity to examine whether engagement actions and any other factors predict research use. The survey asks readers about engagement actions they take after reading an article, their reasons for reading The Conversation, etc., plus a number of demographic questions. I added a question asking readers how they used the articles, focusing on four different types of use (three work-related and one personal). Using the question on use as an outcome measure, my colleagues and I ran logistic regression and classification tree analyses to identify factors that predicted readers’ use of the academic expertise and research reported in The Conversation.

We found that several factors, including engagement actions, predicted use. Interestingly, our study showed that different engagement actions and factors predicted different types of use. The most interesting and important findings identified across both the regression and classification tree analysis are shared below. Our recent research paper details the predictors, methods, and all other findings.

Working as “politician, policy officer, or government employee” predicted direct, applied use

Working as a “politician, policy officer, or government employee” predicted instrumental use of a The Conversation article. Instrumental use refers to direct, applied use of research to inform development of a strategy, policy, programme, practice, presentation, etc. Most surprisingly “politician, policy officer, or government employee” was the only one of the eight specified role types asked about in the survey that predicted this type of direct, applied use.

Being highly engaged in seeking out research also predicted direct, applied use

“Undertaking further research” also predicted this type of use. The classification tree analysis showed readers who worked in one of the other seven (non-government) role types and also indicated that they used a The Conversation article to inform a report and that their main reason for reading The Conversation was work-related, showed four times the overall probability (which was 15%) of using an article in a direct, applied way. This suggests readers actively engaged in seeking out research for work are more likely to use it to inform work-related decision-making.

Government and other senior decision-makers use research in a tactical way

As well as “politician, policy officer, or government employee”, working in senior decision-making roles including “general manager, department head, senior executive, manager” and “chairperson, director, CEO/CFO, COO, owner, partner” predicted use of a The Conversation article to support existing strategy, policy, and other work-related decisions. This type of use is defined as tactical or symbolic use.

Many academics assume this is the only type of research use that decision-makers are interested in. However, this and other studies have shown it isn’t necessarily the only way research is used, and that direct, applied use can be just as likely.

“Discussed with friends and colleagues” predicted tactical and personal use

“Discuss with friends and colleagues” was an engagement action that also predicted tactical use. Interestingly, “discuss with friends and colleagues” was the only engagement action that predicted use of The Conversation to inform personal attitude or behaviour change. These findings suggest that this is a very active type of engagement, where one may be seeking to gather evidence to inform personal decision-making or to influence the opinions or decision-making of others.

Other factors that predicted use to inform personal attitude or behaviour change were to gain “expert opinion and facts” and to “explain the news”.

Conceptual use – the use of research to inform work-related discussion and debate

The only predictor of this type of use, which is to inform work-related discussion and debate without that necessarily, informing or supporting specific decision-making, was a reader indicating they read The Conversation to assist in their work or research. This highlights that readers using The Conversation to assist their work are finding information relevant to their needs. Relevance to decision-makers’ needs is one of the factors most consistently identified as critical to increasing research use.

We need to further test assumptions and expand thinking on pathways to impact

Overall, our study has demonstrated the critical role that access and accessibility play in enabling research use and impact. Policy, programme, and other senior decision-makers are seeking out and using research evidence and expertise that is freely available in an accessible format, in multiple ways. This suggests that impact can be achieved by increasing access to and accessibility of research. Strategic, innovative approaches to research translation and communication can enable use and increase the likelihood of impact from existing research resources, as well as fostering future research partnerships through increased community awareness and understanding of research.

This blog post is based on the author’s co-written article, “Does engagement predict research use? An analysis of The Conversation Annual Survey 2016”, published in PLoS ONE (DOI: 10.1371/journal.pone.0192290).

Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

About the author

Pauline Zardo currently works in research and evaluation in state government in Queensland, Australia and is also a Visiting Fellow in the School of Public Health & Social Work, Queensland University of Technology. Pauline has a PhD in translation of research to policy and practice. She has worked in strategic policy and programmes in government, as a research academic, a research professional and higher education performance strategy consultant. Her key areas of research interest are research engagement, translation and impact, open access, and research performance assessment. Pauline is interested in enabling and supporting research and researchers to have a positive impact on health and community outcomes. Her ORCID iD is: 0000-0002-6027-9010.