By Luís Costa, Holger Schmid, Marie-Madeleine Gomez, Verónica Troya
As our funding came to an end, MAVA convened our partners to evaluate the progress and impact of the Outcome Action Plans. This blog provides a glimpse behind the scenes of how MAVA managed the challenge of measuring impact, gathering partners’ perspectives on the results of their work, and what the future of the work looks like.
During our normal activity, we are used to evaluating the grants we give and the impact of our activity. The closure of MAVA brings a new challenge: how to evaluate the results and the impact of the Foundation after 28 years of activity?
We decided to do it at three levels. First, by evaluating the strategies of our final phase, when we developed 24 Outcome Action Plans (OAP) within the four programmes (Mediterranean, Sustainable Economy, Switzerland and West Africa) between 2016 and 2022.
Adding to the 24 OAP evaluations, we worked as well on a final review of each programme for the full lifespan of MAVA since 1994, adding the Impact and Sustainability Unit to the 4 programmes above. And finally, we have performed a global final evaluation of the foundation’s impact.
Developing a simple and inclusive methodology
For defining the methodology for this evaluation, we wanted to do something that would not create unnecessary work but would rather create added value for all – in the same mindset as the collaborations of each OAP, trusting the partnership and having them self-assessing their impact.
The methodology aimed to step back from the individual project and look at the bigger picture, to show outcomes and impacts in a succinct and objective report. We chose to do this instead of individual project reports, to show in a fast and effective way the collective work done during the last 7 years.
One workshop was organized for each OAP, gathering all direct partners that collectively have been running the projects towards a common conservation goal by using the Open Standards for Conservation. As everyone was using monitoring scorecards for regularly assessing progress and effectiveness of their work, we analysed all indicators in those scorecards as a basis for discussing results and impact.
A collective exercise of writing allowed reports to be completed by the end of each workshop, in an inclusive and equitable manner that gave all partners the power of the pen. In the end, each report includes:
- A general introduction to the OAP and its Theory of Change.
- A chapter on Progress to describe how activities were carried out and how innovative the approach was, and to respond to the question ‘did we do what we were meant to do?’
- A chapter on Effectiveness and Impact, responding to the question ‘did we achieve the impact we were meant to deliver?’ to show clear results and impacts of each OAP.
- And finally, notes on Sustainability and Next Steps, showing the new challenges for the future and what will happen to the partnerships and their new projects.
- In addition, some case studies were included to illustrate good results from projects that represent the collective impact of the OAP.
The final results
All workshops took place between September and December 2022. We brought together our large community of partners that worked hard during the last 7 years to look back and identify the highlights of the journey. The results of this great collective work can be found here.
In general, we are amazed by the great work and the huge impact that this exercise of final evaluation was able to demonstrate. More than analysing individual projects within each OAP, this exercise of compilation of all projects into a single report allowed all partners and us to assess and realise the value of how well partners delivered on the commonly-defined strategy.
Evaluating the evaluation
The collective exercise of writing the reports was fun and apparently welcomed by all participants. It came with some questioning in the beginning about how feasible it could be to write a full report agreed by all partners in two or three days, but in the end all workshops delivered. The main method was to form groups of people to write each chapter of the report; after a first round of writing, groups rotated to review and complete the previous group’s work; and another rotation allowed the group to agree on a final version of each chapter before reviewing and agreeing on the global text in the end.
This exercise was the core of the workshop, but more than that, workshops provided a tremendous opportunity for partners to sit together after a couple of years of remote work, and to celebrate their successes in nice events and activities. A group photo identifies partners in each report and shows who is behind all this work.
In the end, the feedback from participants and facilitators was positive:
- Everybody felt involved in this final evaluation process and played various roles that included a valuable help from some of them to facilitate the preparation work and facilitate the workshops. In general, people were surprised how productive it was, from the quality of discussions to the final product.
- The collective writing exercise allowed everyone to write and removed the bias that usually comes when one person leads on the writing and may have a tendency to put more focus on her/his own ideas. The exercise led to a product with a high degree of buy-in and ownership, and this methodology is recommended not only for evaluation purposes, but also for strategy and programme development because it creates joint ownership and clarity.
- This process allowed partners to know more about each other’s work and how projects could contribute altogether to a common goal and objectives. The final result shows a clear collective output of each full partnership.
- Normally each project must write and submit a comprehensive final technical report. Replacing these by a single short and collective report was much better; it saved partners’ time to write their reports, but also allowed the MAVA team to analyse reports based on collective results and impact, rather than one piece at a time. Incidentally, it also made an efficient use of MAVA staff time, which we appreciated as well. Each project still had to submit final financial reports which were reviewed separately.
- By meeting together, partners had a chance to celebrate great work. In addition to some emotional goodbyes, it was especially a great and crucial opportunity to discuss how to continue collaborating and delivering on OAP goals in the future and an occasion to reinforce bonds among partners.
- The report shows collective work and impact, therefore it may be an important resource for fundraising so they can continue their work together.
The complete list of final reports is available on our website, and will remain there for the next 10 years.