Stepping Stones and Monitoring and Evaluation (M&E)

A recent AWID[1]report, based on analysis of  37 European women’s rights organizations receiving funds from the Dutch government, has offered 13 suggestions to organizations working on women’s rights. We think that these suggestions sound very relevant to us in relation to Stepping Stones also. Here is a summary of these findings.

1. Make M&E a key ingredient in our learning and accountability. This means to focus on M&E as a key part of our learning process rather than just as a requirement for reporting and accountability to donors.
2. Develop M&E capacity. We need to assess the kinds of staff capacities, time, and other resources we require, and to develop a mix of tools and methods that are feasible within the constraints we face.
3. One size does not fit all. AWID’s research shows clearly that no single M&E framework can capture all aspects of the change, impact, or results of a women’s rights / empowerment project or strategy. In fact, 51% of the women’s organizations in AWID’s study use more than four M&E approaches and tools, or elements from several, to document their progress and impact. AWID explains that is quite logical, given that the nature of gender and social power relations is complex and that organizations operate in different social, cultural, economic and political contexts. “So while one M&E instrument cannot tell us the whole story, strategic combinations can bring us much closer to a more comprehensive understanding of the change process, including its strengths and limitations.”
4. Track reversals or “holding the line” We know that most interventions that advance women’s strategic interests, and even many that address their practical needs, tend to create reactions from the status quo. So it is important that we are using M&E methods that can track negative outcomes for women and that can also recognize that even maintaining no change in women’s position can actually be an achievement. AWID, specifically mentions micro-finance programmes for women, which, whilst often considered a good idea, can also lead to “further exploitation, increased domestic violence because of their growing economic power, or exhaustion because their workloads increase without any let up in their domestic or caretaking duties—or, really poor women cannot participate in microcredit schemes at all.” There are also examples of where increased reports of domestic violence incidents, for example, are actually a reflection of greater awareness that domestic violence is unacceptable. Yet a simplistic review might just assume that an intervention is having no effect and that domestic violence is increasing.
5. Balance quantitative and qualitative assessment The most complete picture of change—whether it is positive, or includes backlash, reversals or just successfully holding the line—emerges when both quantitative and qualitative tools of assessment are used. AWID suggests that we could combine surveys which generate quantitative data on changes in women’s political participation, mobility, income, awareness of rights, literacy, health-seeking behaviour, and changes in male attitudes, with qualitative methods like narratives of individual and collective struggles, stories of change, and focus group discussions that describe how change happened.
6. Prioritize approaches that assess our contribution to change, not those that demand attribution Here the AWID report highlights the limitations of some M&E frameworks, such as the logical or Results-Based management frameworks, suggesting that these are more valuable for tracking project implementation activities, rather than how any changes took place. As AWID points out, no one intervention on its own is likely to change behaviour. Instead, it is more realistic for us to be more modest in our aspirations, and seek to understand how our intervention can contribute to the changes we seek. “Outcome Mapping”, “Most Significant Change” and “Theory of Change” approaches are cited as three useful examples of understanding how changes can take place.
7. Less is more Here the AWID report highlights the importance of the quality of the information we gather over the quantity of information gathered. It suggests that the “SMART” framework (where indicators we select are Specific, Measurable, Achievable, Realistic and Time-bound) is useful. However, we have two additional thoughts to add here. One is that we should also seek to make indicators of change “SPICED”. This means that the indicators should be meaningful to – and ideally created and identified by those most affected by an issue. “SPICED” indicators was a concept created by Chris Roche[2]and colleagues of OXFAM. “SPICED” stands for “Subjective; Participatory; Interpreted and Communicable; Cross Checked and Compared; Empowering; Diverse and Disaggregated”.
8. Flexibility and adaptability of the M&E framework AWID also recommend that if external factors change, we must make sure that we re-examine and adapt our M&E framework accordingly and that we do not stick to a rigid plan devised at the outset. This is just to reflect the reality of social change – and how none of us can predict the future, no matter how carefully we plan.
9. M&E systems must be appropriate to organizational architecture Here AWID highlights the many different layers and levels of organizations and networks seeking to effect social change these days. It explains how, for instance, measuring the numbers of women reached across a network might be a useful approach at one level – but that this figure alone gives us no idea what the quality of that reach has been. We know, for example, that it is important for participants to go through a whole Stepping Stones training together, wherever possible with 2-3 hours of contact time per session with well-trained facilitators, over around 18 sessions. But this quality of contact time cannot be reflected in a simple question “how many were reached”. This then is another call for a “combination evaluation” approach.
10. Negotiate M&E systems with donors Here AWID recommends: “Understanding and negotiating M&E expectations at the outset of a project or funding cycle” as a useful strategy to avoid tensions and misunderstandings at a later stage. Indeed, sharing the AWID resources with our potential or existing donors can help them to be aware of a wider range of materials on this topic.
11. Tailor indicators and results to time frames Furthermore, it is important for us to think about when an evaluation takes place, bearing in mind all that is known about seasonal variations and their effects on a community – including, for example on food insecurity and its effects on household quarrels and violence. For example, an evaluation should ideally take place at least one whole year after a programme has finished, in order that an annual cycle of events has taken place since its ending. It may well be also that what we can seek to achieve in a three year project is only just starting to float an idea about the need for reducing gender-based violence in a community, for instance. Yet how often does the funding cycle allow this? We need to be mindful therefore of not trying to achieve too much in a limited period. There is an excellent paper about this which explains this further[3].
12. Create baselines AWID highlights the important need for the collection of baseline data with a community, so that we are clear what the status quo is, and from which any change that does take place can be measured. See, for example “Implementing Stepping Stones”, for suggestions about how baseline data can be gathered. The “Gambia Evaluation”[4] and the “Fiji Stepping Stones Evaluation” also explain different methods for baseline data collection. NB it seems important to us to ensure that participants themselves be involved, as far as possible, in assessing their own situation at the start of a programme and any changes that take place in the community along the way. This “critical literacy” as it is known can often act as a reinforcement of a sense of pride in what positive changes have already been achieved in and by a community, and as an incentive to support further change. One simple way in which these sort of data can be gathered is illustrated from another review report from the Gambia[5]. The tables in this article were constructed by participants during the first session of the Stepping Stones programme there, as a means of asking participants to consider what hopes and fears they had for the programme. Other tables are shown which reflect what changes the participants considered that they had experienced, one year after the programme. By asking each peer group of participants separately and simultaneously for their views, this programme was able to build up independent but cross-checkable information from the community concerned about the changes they perceived to have taken place. The changes reported by the participants one year on are of marked increase in qualitative depth than the baseline hopes and expectations. This in itself suggests that there was a certain “critical literacy” that had developed amongst those involved in the programme.
13. M&E that works for us will work for others Lastly, the AWID report recommends that we “seize the initiative in our own hands, rather than wait for something to be demanded or imposed by others. Our capacity to successfully negotiate for our own M&E approach is greatly enhanced when it has been developed through a sustained and committed organizational process, and is consequently both robust and convincing.”

Click here for more on evaluation of Stepping Stones.

[1]Batliwala Srilatha (2011) Strengthening Monitoring and Evaluation for Women’s Rights: Thirteen Insights for Women’s Organizations  Association for Women’s Rights in Development (AWID)

[2]ROCHE, Chris (1999) Impact Assessment for Development Agencies. Oxford: Oxfam/NOVIB, viii:308pp.

[3]Woolcock, Michael (2009) Toward a plurality of methods in project evaluation: a contextualised approach to understanding impact trajectories and efficacy, Journal of Development Effectiveness, 1:1,1—14

[4]Paine Katie et al “Before we were sleeping, now we are awake”: preliminary evaluation of theStepping Stones sexual health programme in The Gambia. African Journal of AIDS Research, 2002, 1:41–52.

[5]Jarjue Mama Sirreh et al  Participatory Review of changes after a Stepping Stones workshop in an Islamic context, the Gambia, February 2000