Who shapes the questions we ask? Holding rigour and reflexivity together

In today’s post, Dr. Maedeh Aboutalebi Karkavandi Senior Research and Evaluation Analyst at Good Shepherd Australia and New Zealand (GSANZ) reflects on the first day of the Australian Evaluation Society conference in Darwin, where a keynote from Dr Robyn Ober challenged her to think more deeply about where questions come from and whose perspectives they reflect.

The conference logo was created by artist Cyan Sue-Lee who is a proud Larrakia woman, with Wardaman, Karajarri connections. She is from a prominent artistic Larrakia family, the Lee/Cubillos. She draws inspiration from her family and the natural landscapes she calls home.

Our work relies on asking thoughtful questions. Yet the questions we ask are not neutral. They reflect what we value, what we expect to learn and the assumptions we bring to the work.

What does an evaluation question sound like?

On the first day of the conference, one story from Ober’s keynote stayed with me.

Reflecting on her PhD research, Ober described sharing her interview questions with an Indigenous community member, who responded: “Where is my sister? Where is you as a Black person in these questions? These seem to be written by a white person.”

Rather than continuing with the interview as planned, they had a yarn. Through this experience, Ober developed what she calls the Kapati Method, an approach to collecting and analysing participants’ stories over a cup of tea.

The story drew my attention to where reflexivity begins. In evaluation, we often examine how our position and experiences shape the interpretation of what people tell us. But Ober’s example showed that these influences are present before analysis begins.

Influences are already present when we decide what to ask.

Culture is present in what we consider important enough to measure, how we structure an interview and what we recognise as a useful answer.

Ober also spoke about something as simple as silence.

In some First Nations contexts, a pause in conversation can create space for reflection and understanding. In a more conventional semi-structured interview, an interviewer might read the same pause as a sign that the participant has finished speaking. They may move to the next question or ask, “Is there anything else you would like to add?”

Neither response is inherently right or wrong. But each reflects assumptions about how a conversation should unfold and how knowledge is shared.

This prompted me to reflect on my own practice.

I come from a culturally diverse background and have spent much of my professional life in Western academic environments. I value evidence, sound methods and methodological rigour. At the same time, I have become increasingly aware that many of the practices we describe as “good research” come from particular academic traditions and ways of knowing.

The challenge is to work in ways that respect both rigour and culture

This might mean being more deliberate about who helps shape an evaluation’s questions. It might mean making room for different ways of communicating and understanding. It might also mean accepting that a credible process will not always look exactly like the research methods we were trained to use.

It asks us to examine what we mean by rigour, whose standards have informed it and whether our methods are fit for the people and contexts involved.

The AES26 theme is “Making Space, Valuing Place”, and Ober’s keynote brought that theme to life for me.

I left the session thinking that reflexivity needs to happen further upstream. It is not only a matter of asking, how am I interpreting this?. We also need to ask, what assumptions sit behind my question?’ and even, what assumptions sit inside the silence?. Whose traditions and ways of knowing have shaped what I think is worth measuring, how I believe evidence should be gathered and what I recognise as a valid response?

For me, this is part of bringing my whole self to evaluation while continuing to value multiple forms of evidence, each with its own standards of rigour.

Ober, R. (2017). Kapati time: Storytelling as a data collection method in Indigenous research. Learning Communities: International Journal of Learning in Social Contexts, 22(1), 8-15.

Moderator: Dr. Jennifer Hester