The International Educational Development Program at UPenn

Grace*, a 15-year-old, is happy with the kind of joy that comes from finishing a big exam. She attends a `community school’ in Amuru district in North Uganda – a school that the local community set up for their children because the nearest Government primary school is over 40 km away. The community school is still not exactly next door – Grace wakes up at 5 AM to reach school by 8 AM. The teachers here have completed their O-levels and A-levels (10 years and 12 years of schooling, respectively) and are not formally trained. But they are what the local community can access.  Grace’s aunt has to pay 15,000 UGX per term (a little over $4 – a lot of money in this part of the world) for her to attend the school – but she can’t always make the payment.

In a different part of the same district, Amuru, Angel* talks about the loan she received through a Village Saving and Loan Association (VSLA) she is a part of. Her children also attend a community school – she took the loan and invested it in her farm, but she also used it to pay the fees for the community school.

This slideshow requires JavaScript.

These community schools were set up with Government support under Uganda’s Universal Primary Education (UPE) policy. Geneva Global-Uganda (GGU), where I have been interning for the last month, has been supporting five of these schools since 2016. Over the past month, I have been leading a formative evaluation of the community school program as a part of my internship here at GGU.

Maybe because this is a new context and I have spent a lot of time re-learning the basics here, I have been thinking about the basics of evaluations – the how and why of their design. I made a lot of decisions in the course of the evaluation – mostly between being rigorous and being practical – and I have been wondering if I made the right call. In this blog, I wanted to discuss how I approached these questions in my evaluation of community schools (maybe get some feedback :)). But before I dive any further into nerd-land, here are some fun pictures of non-work things (if you make it to the end, you can meet my new friend in Gulu):

This slideshow requires JavaScript.

Ok, moving on to my fun work life – Formative evaluation of a program that supports ‘community schools’ in Northern Uganda:

(1) Why formative evaluations? Formative evaluation is “an ongoing process that allows for feedback to be implemented during a program cycle” (source). In other words, formative evaluations are meant to provide implementable feedback to program managers.

I think of formative evaluations as trying to optimize an aircraft’s performance when its already in the air. I was given two months to understand, collect data and provide actionable recommendations on the community school program – with about 6 months left for the current funding cycle (and sticking a good landing is one of the hardest parts of a pilot’s job). It was hence important to design the evaluation well; which brings us to:

(2) Designing a formative evaluation: We know what formative evaluations are meant to be, but there are a lot of opinions in the field about how they should be implemented:

What was the question again? The “Answer to the Ultimate Question of Life, the Universe, and Everything” is 42 (Google agrees); but if you don’t understand the answer, it’s because we don’t know what the Ultimate Question is.

Formative evaluations are not an end in themselves – they are trying to add value by answering important questions about the program, questions program managers need answers for. And yet it’s incredibly easy to lose track of that end goal while in the middle of playing with data (for me, at least). There is so much data that can be collected, so many potential “cool” findings that can be unearthed – that sometimes you (I) forget to ask – what question is this analysis answering? Can the program manager do something with this answer?

One of my ex-bosses had this habit of looking at research findings and asking – “so what?” – he was asking what action he could take based on the findings in front of him. It’s embarrassing – but most of my analysis wouldn’t stand up to the “so what?” test.

On this internship, I discussed my proposed goal and findings with the program managers, and subjected it to the “so what” test till we settled on the research questions (for now):

• Understanding if the quality of education at the schools improved – this can cover input measures like infrastructure available at school, teachers present at school; as well as outputs like improved teacher instruction (in this case, defined by measures of classroom management and learner-centered instruction) and improved learning outcomes
• Understanding if the gains would outlast the program i.e. improved parents and teacher capacity to manage the school, and improved financial sustainability (defined by measures like design and implementation of school development plan, sources of income for schools, management processes like recording teacher and student attendance, etc.)
• We also defined what is not a goal e.g. a process audit was not the goal of this project.

How much data is enough data? Big research projects and evaluations, with more resources, have the luxury of collecting a lot of data – and then figuring out which questions are important to focus on (my fellow IEDP-er, Susan, talks about this in her blog). Now, let’s remember that I am here on a 2-month internship. Time and program resources are precious; asking for unnecessary data has costs for me as well as the program team here. Imagine if a flight was getting ready to land, and I barged into the cockpit to ask the pilot to come with me to do a passenger headcount – an important metric but one that doesn’t help prepare a pilot for landing, and hence, is not a useful data point at that moment (and the pilot probably has other, more important things to do for the landing).

I needed to be strategic with my data collection design. My approach here was to:

• Use program data and documents shared by the implementing team (instead of collecting my own data)
• Rely more on key stakeholder interviews and discussions, over focus group discussions with the whole community (which takes more time to set up) or longitudinal quantitative data (which wasn’t available)
• Validate, validate, validate: Asking different stakeholders the same questions (and sometimes asking the same stakeholders the same questions) until a common story or theme emerges. Being an outsider also gives me the advantage of asking “stupid” questions, or asking the same questions over and over, and checking for consistency in answers

Hence, my data on enrollment in school, teacher attendance and learning outcomes came from the schools themselves. I verified the data in my visits to schools. I also tried to be respectful of the time of the program managers implementing the community school program – I asked them to let me know if they were not collecting some data I would have liked to have, and worked under these constraints. Of course, there are glaring flaws in the approach –

• The data stands to be heavily contaminated: because it wasn’t independently collected. We don’t even know how it was collected.
• The data isn’t “complete”: because, for instance, schools might not have full records of School Management Committee (SMC) meetings because their records were destroyed in a storm. Or, because the field visit happened at a time where parents could not visit the school to meet the research team (me).

But, given that this is meant to inform the implementers, and I (and the team) went with the approach of painting the best picture possible under the circumstances:

• Using resources efficiently: If I tried to conduct standardized tests like EGRA or EGMA in these schools, it would take most of my field visit time. I also did not have access to a baseline standardized assessment. In other words, if I conducted my own learning assessment, it wouldn’t tell me how students’ performance changed over time – at best, it would confirm (or dispute) the learning outcomes from the tests conducted by the school. In this case, I decided at, on the whole, it was better to not conduct a standardized test and to use the learning outcomes data shared by the school.
• Finding the strengths and gaps in the current M&E systems: I cross-validated the preliminary findings with all the stakeholders over and over — which led to very productive conversations about the schools’ and GG’s data management systems. For instance, I looked at the teacher attendance registers and checked if they match the teachers present in school on the day
• Exploring the factors at play in the community schools: This approach of repeated conversations on data artifacts also led to interesting findings about the community schools themselves. For instance, I discussed the ​​’school development plan’ (image below) developed at one of the community schools with the local stakeholders (head teacher, parents) and the program staff, to understand what different stakeholders thought a good plan was.

Section of School Development Plan (from one of the community schools)

(3) Doing formative evaluations well (alternatively, sleeping well at night): What is good evaluation anyway? How do you sleep well at night, knowing you have done good research and it has also been helpful to the program?

It’s hard to decide exactly how long a good program evaluation takes, but most researchers would say two months is probably too short a time period to do “good research”. Most program managers would say it is too long. Also, if I spent all 2 months on the research, there would be no time left to validate the research with local stakeholders and implementers or to work with them to interpret and plan implementation the findings.

Welcome to my world of formative evaluations, folks.

My report will definitely be rooted in data. However, in the last month, I haven’t attempted a Randomized Control Trial (RCT) or even data collection for a cross-sectional regression. My final report will probably lean towards a case-study method of reporting (which I am hoping will be effective in painting a picture of key background factors that lead to program results). It will hopefully provide information that is useful for the program managers here.

Is it enough? Maybe you can tell me, dear reader.

My next blog will talk about the second project I have been working on in my internship – rebooting the M&E system! Stay tuned 🙂 If you made it here, here is a picture of my new friend!

“To make friends with a cat is to make friends with scratches” – wise words from the wise @simplyujj

*name changed