Experts share insights on the mixed method framework on impact assessment

“We cannot overemphasize the importance of authentic assessment wherein we really have to make sure that we do not only focus on the numbers; rather, it would be more informative if we are also informed about the stories behind the numbers.”

This is what Dr. Aileen Lapitan said in her synthesis of the webinar on mixed method approaches to impact assessment (IA) on June 21. Dr. Lapitan is the director of the Center for Strategic Planning and Policy Studies of the College of Public Affairs and Development (CPAf) and is one of the study leaders of the research program titled “The Mixed Method Approaches to Impact Assessment of Agriculture for Development (AR4D) Interventions.”

The said research program, which aims to develop and fine tune a mixed method approach to IA for AR4D and non-AR4D interventions, was funded by DOST-Philippine Council for Agriculture, Aquatic, and Natural Resources Research and Development (PCAARRD) and the Australian Centre for International Agricultural Research (ACIAR). It was implemented by UPLB and Visayas State University (VSU), in collaboration with the Commonwealth Scientific and Industrial Research Organisation (CSIRO).

In her welcome remarks during the opening program, Vice Chancellor Merdelyn C. Lit noted that mixed method approaches have been gaining importance in economics research and interventions and is becoming a welcome model in the field.

Chancellor Jose V. Camacho, Jr., who serves as the program leader, shared to the audience that the research is one of the pioneering projects on IA in the Philippines. He said that coming up with the framework has been a fruitful journey for all the members of the research team and for the institutional partners.

The Mixed Method Framework on Impact Assessment

The mixed method framework focuses on four key areas: 1) Project Design Review, 2) Theory of Change and Impact Pathway Validation; 3) Context Mapping and Impact Scoping; and 4) Data Collection and Analysis.

As a starting point, under the project design review, the framework looks into the outcomes of the research program being assessed, the counterfactual, the impacts, and the program beneficiaries and the benefits that they have gained.

Theory of Change and Impact Pathway Validation investigates the long-term goals and intended impacts of the research project, the changes undertaken to achieve them, and the actors responsible. It also looks into why long-term goals and intended impacts were not achieved and the external factors that played a role in it.

The third key area in the framework, Context Mapping and Impact Scoping, involves determining the economic development path, the natural capital or physical resources available, the enabling policy and market environment, social relationships and community customs, as well as institutions and norms. This stage also defines the scope of assessment in terms of levels of analysis and dimensions.

The fourth key area of the framework is Data Collection and Analysis. This stage is guided by the dimensions of the identified impacts, the research questions, impact pathways, and availability of information and resources. At this stage, impact assessors determine the data to be collected and how they will be analyzed. This involves the weighing and the mixing of quantitative and qualitative strategies.

The framework also reiterated the importance of communicating the results of the assessment. According to Jeanarah Gapas, the program’s research analyst, what is reported and how it is communicated determine what the users think they are being told, and thereby the decisions that they make. For this, adequate information, methods undertaken, analysis, and limitations of the assessment should all be taken into consideration.

The Mixed Method Framework in the spotlight

The organizers invited experts from various fields to weigh in on and share their insights regarding the framework. Dr. Roderick Rejesus, professor and extension specialist at the Department of Agricultural and Resource Economics of the North Carolina State University, commended the project for emphasizing the presence of an impact assessment plan at the start. He said, “In a lot of research for development projects, the economists and impact assessors come at the very last minute so it is good that you have emphasized the presence of these experts at the start.” Dr. Rejesus also added that the framework could have also taken into account the design of the interventions.

Julieta A. Delos Reyes, assistant professor and chair of the Department of Agricultural and Applied Economics of the College of Economics and Management, UPLB, agreed with Dr. Rejesus that looking at the design of the interventions is also important. She added that it is good practice to form a team with experts representing various fields and that researchers should be immersed in the community.

Dr. Liana Williams of CSIRO mentioned that while the program was able to articulate a different approach to IA, the framework may not always align with the needs of funding agencies and research institutes. They may have other goals in mind. There is also a bit of a friction between the traditional needs and methods of IA and broadening out to mixed method approaches that have an emphasis on learning and learning what could be done better.

However, Dr. Williams said that this is just a starting point and that it is in evaluation and impact assessment “where the excitement really is because of the real momentum between doing things differently.”

Lastly, Dr. Thelma Paris, a professor at UP Open University, shared the importance of qualitative approaches in impact assessments and the value of ‘authentic assessments.’ She said that although it is not easy to publish a paper with qualitative approaches, it is better to substantiate quantitative results with qualitative stories.

Following the discussion was an open forum led by Dr. Moises Neil V. Seriño, faculty member at the Visayas State University who served as one of the study leaders of the project. Various questions from the participants were addressed by the research team. They were also joined by Princess Alma B. Ani and Ms. Mia Barbara DV. Aranas of the Socio Economics Research Division of DOST PCAARRD.

The webinar, which had more than 220 participants, was supported by the Australian Centre for International Agricultural Research (ACIAR), through its Launch Funding Program. Hosted by the UPLB College of Economics and Management, the online event kicked off a series of learning events that will be conducted by the program. The research team will also be releasing a book detailing their experience in applying the framework during the upcoming anniversary of CEM.