Piloting the Pilot: Reflections on the EPI Process in Atlantis

Posted by on Dec 20, 2013 in News | No Comments

For the last five months, I’ve had the opportunity to work closely with our South African partners at the Institute for Justice and Reconciliation (IJR) to pilot the EPI process in Atlantis, a community known as the “forgotten town” that is located approximately 40km outside of Cape Town in South Africa’s Western Cape province.

Atlantis, a predominantly coloured community, was established in the 1970s as an industrial center. With many of its factories and businesses now closed, Atlantis’ major challenges today include high rates of employment, lack of housing, crime and gang-related violence.

Flats within Atlantis

Flats within Atlantis

Step One: Preliminary Community Visit
We began our engagement in Atlantis in late-August 2013, when IJR staff visited the community on a preliminary site visit. We used this visit as an opportunity to better understand the community dynamics and to identify potential mobilizers and key stakeholders who should be involved in the project’s focus groups and surveys.

Step Two: Initial Three Focus Groups with Men, Women and Youth
Following the initial community visit, we held our first three focus groups in September and October 2013 with men, women and youth from Atlantis. From these first discussions, in which we asked participating community members how they recognize peace and safety in their everyday lives, we were able to compile a list of the everyday peace indicators that emerged. Some of the indicators, such as “petty crime” and “gangsters in the street” were the same for two or more the groups, whereas other indicators, like “cars with bullet holes” from the women’s group and “foreigners in the neighborhood” from the youth group were unique to that demographic.

Kenneth and Stan from IJR facilitate the youth focus group in Atlantis

Kenneth and Stan from IJR facilitate the youth focus group in Atlantis

Step Three: Verification Focus Group
As a next step, we invited the groups to nominate five people from each group to attend a verification focus group that was held in November 2013. In the verification focus group, the men, women and youth group representatives were first asked to reflect on the indicators they noted during their first focus groups, and then to rank those indicators into a top ten list for their group. After each group had done that separately, we convened a plenary session, where each group had five minutes to present and explain their top ten indicators. There was some time for Q&A, but in hindsight, we wished we had allocated time for more. A lively discussion between groups ensued, with different groups questioning and even challenging the others’ top ten lists.

Representatives from the women's group rank their indicators during the verification focus group in Atlantis

Representatives from the women’s group rank their indicators during the verification focus group in Atlantis

After each group had the opportunity to present their shortlists, we taped their lists side-by-side on a wall in the church hall where we were meeting, and then asked participants to rank their personal top ten indicators using stickers. Each person was given ten stickers and could only vote for an indicator once. The groups took turns going up and allowing their members to cast individual votes.

Men's group representatives cast their final votes in the Atlantis verification focus group

Men’s group representatives cast their final votes in the Atlantis verification focus group

After everyone had the opportunity to vote, we tallied up the stickers, and I read aloud the top ten indicators that had amassed the highest number of votes. We then had a short discussion on this top ten list, and participants agreed upon a slightly modified final top ten list to be turned into questions for the community survey. We, the EPI team, also elected to include three additional indicators in the survey that did not quite make the community’s top ten list but were discussed in the focus groups and we felt were particularly interesting or relevant. An example of this is an indicator pertaining to monitoring Atlantis community websites to ascertain levels of peace and security. The final thirteen-question survey would be administered every six months over a year and a half to track how the indicators change over time.

Step Four: Preparing for the Community Meeting and First Survey
Preparing for the next step in our process, holding a community meeting, was a step in and of itself. In Atlantis, due to the prevalence of cell phones and network coverage, we elected to pursue a cutting-edge surveying method called integrated voice response, or IVR. In our envisioned IVR surveys, community members would be called every six months and asked to listen to the pre-recorded survey questions and push a number between 1 and 5 on their handsets (mobile phones) to record their answers for each question. For example, if the question asked, “In Atlantis, are there gangsters in the street?” they would have the option to press 1. Never, 2. Rarely, 3. Sometimes, 4. Often or 5. Always. While we imagined this would work in theory, our challenge was figuring out how to introduce the survey to the broader community, elicit their buy-in, prepare them to answer the IVR surveys and collect their phone numbers. We decided to hold a community meeting to accomplish these goals.

In planning for the community meeting, we had to undertake many steps, oftentimes figuring them out as we went. For instance, we turned the top indicators into survey questions like the one above and developed a script for the voice recordings. We then translated this script and recorded it in English and Afrikaans. We developed participant enrolment questions on topics such as age, gender and education level, and input those questions into English and Afrikaans survey forms on the project’s mobile phones using the Mobenzi mobile phone application. (We are using this application and mobile phones during the community meeting to enroll community members to participate in survey.)

At the community level, we identified local community mobilizers and spread the word through radio adverts and flyers. We also acknowledged the potential practical concerns of community members who would be attending the meeting on a weekday evening, and arranged for dinner to be catered and for there to be transportation home for everyone. We identified a meeting space in a local government building that was supposed to be in a safer part of town. Lastly, we designed and printed small laminated cards to give out to people who enrolled, so that in the months ahead when they received phone calls to respond to the EPI survey, they could reference a typed answer scale and phone numbers of the IJR team should they have any problems completing the survey.

During the community meeting in Atlantis, we handed out small laminated cards like this one that community members could refer to when completing the IVR surveys at a later date

During the community meeting in Atlantis, we handed out small laminated cards like this one that community members could refer to when completing the IVR surveys on their own at a later date

Step Five: Community Meeting
After several weeks of preparation, we held our first community meeting in Atlantis in late November. The IJR facilitation team for that meeting comprised of five staff and four interns. The seven of us who were to enroll Atlantis participants in the survey using the mobile phones were trained on how to use the phones on the afternoon of the community meeting.

Leila prior to the community meeting in Atlantis

The IJR team practices using the mobile phone handsets to enroll participants in the Atlantis EPI survey

We scheduled the community meeting to begin at 6pm and last approximately 1.5 hours. By 6:15pm, only 30 of the expected 200 people had arrived. After delaying the start time to allow for others to make their way to the venue, members of the IJR team explained the EPI project and survey process in a short presentation and facilitated a Q&A with community members in attendance. Much of that discussion centered on why more people had not attended and how we could bring others into the process. We learned that there had been a shooting in the area the week before, in which a gangster boss’ son was killed by a rival gang, and that the community feared retaliatory attacks. It was unclear when or where those attacks would happen, so rather than risk venturing out in the evening to attend our meeting, most people elected to stay home. Furthermore, we learned that, while we had made attempts to get the word out through mobilizers and media, perhaps we had not used the right channels or been persistent enough. Many people might simply have not heard we were holding the meeting.

After this discussion, we split into groups, and the IJR team, using the mobile phone handsets and the Mobenzi application, began enrolling people for the survey by asking them a series of short demographic questions, such as “How old are you?” and “How long have you lived in Atlantis?” At that time, we also collected their phone numbers and mobile phone carriers. Unfortunately, there was a glitch in the survey application on the phones, and it would not capture certain phone numbers. We had not caught this problem when we tried inputting our own numbers into the form earlier in the day. We had to think on our feet and input alternative numbers that could be corrected manually at a later date.

At the end of the meeting, we had collected approximately 65 out of the target 200 phone numbers. The IJR team invited those who were interested in helping us to collect additional phone numbers to meet briefly immediately after the meeting to develop a strategy for the coming days. Over the next three to four days, these community members used paper forms to enroll others from Atlantis in the survey, which we then entered into the online database. In the end, we had a pool of 163 phone numbers who were later called in the first EPI IVR survey.

Step Six: IVR Survey #1
We initially scheduled our first IVR survey in Atlantis to take place on December 4th, eight days after the community meeting. Ideally, the first call would have taken place sooner (2-3 days after the community meeting), but we needed the additional time to finalize the IVR call process. We agreed that Mobenzi would schedule the calls to take place in batches of approximately ten numbers every ten minutes from 10am to 12pm. Those who did not successfully complete the call at that time for whatever reason were called again immediately after the first attempt, and if they still were not successful, again between 6pm and 8pm later in the day. Unfortunately, we faced some challenges preparing the recordings for the IVR calls, so we had to delay this first survey by six days and it successfully took place on December 10th. Fifty-five people completed the survey and were sent approximately $1.80 of mobile phone credit (airtime) for their time.

Here is a snapshot of our initial results from Atlantis IVR Survey #1 from questions three and nine:

EPI Atlantis Survey #1 Question 3 ResultEPI Atlantis Survey #1 Question 9 Result

With time, we will post our results on the website to track how the responses change over time. The next IVR survey in Atlantis is scheduled to take place in approximately six months in June 2014.

Reflections on Our Process in Atlantis
Although I have tried to include reflections on our Atlantis process throughout this post, in summary, it was very rewarding, albeit frustrating at times, to witness the EPI process come to life through the Atlantis pilot. As we expected, we faced some challenges along the way, especially in later stages when preparing for the first community meeting and IVR survey, but it was a joy to work with IJR, Mobenzi and our local partners in Atlantis to troubleshoot and come up with alternative approaches and processes, when our initial ideas fell short.

Moving forward, we are incorporating our learnings from this pilot into our other cases. In early 2014, we hope to pilot a handset-based methodology in two rural Ugandan communities (meaning we won’t be doing IVR surveys in those communities largely because of limited mobile phone coverage), and to conduct additional IVR and handset surveys in our other partner communities in South Africa, Zimbabwe, Uganda and South Sudan. Because no two communities are the same, I am sure we will hit some snags in these other surveys along the way, but I am confident that our emphasis on reflection and action-based learning will help us to identify best practices and overcome challenges.

Leave a Reply