Development as Freedom: An Alternative Framework for ICT Program Evaluation (Part 1 of 2)

People across the world are experimenting with ways to leverage technology for economic development and social change. These projects, often referred to as Information and Communication Technologies and Development (ICTD) strategies, tackle pressing and difficult questions: what are best practices for communities seeking to use technology to identify and work towards development goals? How do we ensure that underserved communities are connected and able to participate actively in civil and political dialogue, much of which is occurring on digital platforms? Given that the benefits allowed by access to information are difficult to quantify, traditional ways of measuring development are often inadequate for evaluating the meaningful use of ICTs in people’s lives. To effectively evaluate these development projects, we must shift away from conventional technical indicators of success and examine alternative frameworks.

In early June, the World Bank requested feedback on a draft document titled “Valuing Information: A framework for evaluating the impact of ICT programs.” This handbook, written by senior governance specialist Bjorn-Soren Gigler, draws from Amartya Sen’s “capabilities framework” for evaluating ICT programs. The framework asks ICT teams to think beyond simple macroeconomic development indicators as the desired outcomes of ICTD interventions. Instead, Gigler talks about Sen’s idea of development as freedom: the freedom of individuals and communities to determine what they value, to be able to work towards attaining their goals, and to access the resources, infrastructure, and information necessary to participate in local and global dialogues. Further, the handbook points out that in order to successfully expand local capabilities and achieve development goals, communities also need trusted intermediaries - for example, established local partner organizations - who can provide ongoing technical and social support.

In light of the capabilities framework, the draft handbook proposes that ICTD projects should be evaluated by a more sophisticated and inclusive framework focused on the impact of ICTs in people’s lives, not on the technologies themselves. This represents a shift for the World Bank and other large development agencies, which up until now have been known more for massive infrastructure projects evaluated by metrics such as GDP or other macro-level indicators. Unlike these conventional methods of evaluation, which judge success based on "access, expenditure, and the establishment of infrastructure,” this handbook emphasizes the importance of providing “intermediary assistance, literary skills, and relevant content” in addition to access to a technology.

We find that Gigler’s application of the capabilities framework is an effective and meaningful way to evaluate ICT projects beyond technical and large-scale economic development. However, the paper does not describe the way a program could use the framework to plan an evaluation in concrete terms; the case studies provided only show general applicability. Specific examples of how an agency could tailor the framework for evaluating a given program along with sample indicators or indices would be helpful, since the ideas presented in the paper are mostly conceptual. For example, how does one measure the construct of “ICT Capacity Building” in practical terms? Do programs need to include all of the constructs presented in the framework or can they select a subset of ideas? Practical guidance for how to use the framework is an important next step to enable programs to start using the framework on the ground.

As the external evaluator and documentor for Broadband Technology Opportunities Program (BTOP) projects in Philadelphia and Detroit, OTI has developed an approach similar to Sen’s “capabilities framework” in designing our program evaluations. Thinking beyond the bare-bones requirements set by the National Telecommunications and Information Administration (NTIA) (traditional access indicators such as expenditures and number of home broadband subscriptions or visitors to computer centers), OTI is helping its partners document the impact of public computing centers and digital literacy trainings on peoples’ lives. We have also invited policymakers and researchers to rethink the applicability of traditional metrics for a goal as complex as broadband adoption, and will soon be co-publishing a toolkit of innovative frameworks for understanding and measuring “meaningful” broadband adoption in a forthcoming issue of the International Journal of Communication.

We recognize that while programs start with some stated goals for their participants (increased digital literacy skills, increased job opportunities, etc.), individual participants have their own rationales for participating in these programs. Using open-ended qualitative data collection and survey questions (“Did you learn what you wanted to learn?”), we are able to explore and acknowledge participants’ personal values and goals, as described in the “development as freedom” concept.

This is just one example of the specific indicators and concrete methods we are using to operationalize a capability-building approach to evaluating ICT interventions. For more, please see our set of instruments and tools for evaluation of the Philadelphia Freedom Rings BTOP Project.

As governments and other institutions increasingly invest in technology infrastructure and community networks, it is important to utilize people-centered frameworks that reflect the social complexity of these initiatives. In order to plan and evaluate the success of programs like BTOP, we need concrete and defensible methodologies. Without this on-the-ground perspective, the positive outcomes of ICT projects aimed toward community empowerment and self-actualization are only aspirational, not achieved and proven.