NEW YORK, April 4 — An explosion of data has already changed how we market products and politicians. Now a similar innovation is beginning to change how we combat poverty around the world.

Consider an unlikely problem: Finding the poor. Even in a world riddled with poverty, nearly every government, non-profit and aid agency struggles with this issue. Where in Kigali should the Rwandan health ministry place a new health clinic? Which rural districts in India should receive rice at subsidised prices? All these decisions require not just knowing poverty exists, but pinpointing areas of greatest need.

But until very recently, the data commonly used to answer these questions came almost exclusively from countrywide surveys, which are expensive and logistically challenging. It is very difficult to randomly sample people in the rural areas of Bihar in India or in a slum like Kibera in Nairobi, Kenya, where even just mapping the streets is its own project.

These challenges make new kinds of data — information that can be gathered indirectly using algorithms and novel sources — particularly valuable. Google searches and Twitter and Facebook posts, which are useful in the United States, are unlikely to help in Kibera or Dhaka, Bangladesh. But the core idea behind these sources of data — measuring without asking people directly — can be enormously helpful.

Advertisement

Take the case of measuring the most basic of economic variables: Gross domestic product. The numbers can be unreliable in countries where the statistical infrastructure is weak, the informal businesses do not want to be tracked and the numbers may be manipulated. Morten Jerven, an economist at Simon Fraser University, argues in his book, Poor Numbers, that for many African countries the lack of quality data impedes development.

To see how questionable official data can be, consider that in 2015 North Korea released a budgetary report claiming its economy had grown by roughly 225 per cent. To verify this dubious economic miracle, researchers can turn to NASA, which has night time satellite images. One image shows an ocean of lights in South Korea and China.

It also shows a vast darkness between them, depicting the grim reality of North Korea, where night lighting is a rare luxury. If North Korea is experiencing an economic miracle, it is a purely daytime affair.

Advertisement

Night time luminosity tells us not just about electrification but also about economic activity more broadly, and statistical work shows it reliably correlates with economic performance. North Korea is not the only country where satellite photos tell a story that differs from the one told by official data. A seminal paper in the American Economic Review found estimates of income growth in many places that differed from official data by as much as 3 percentage points annually.

Satellite photos provide a level of geographic specificity that national accounts do not. Another set of researchers used visual algorithms (related to those that recognise your face on Facebook or help navigate cars) to analyse these images pixel by pixel, and they were able to quantify poverty in each square kilometre of Uganda.

Satellite photos provide other useful information. In rural areas, researchers can see crops in the ground, allowing them to estimate harvest size — even before the actual harvest. This data offers a direct window into an essential part of the economic lives of many of the world’s rural poor. The information can be used to build early warning systems for crop failure, to create crop insurance or target other forms of assistance.

There are many other important, unconventional sources of data. Consider cellphones. For most of the world’s poor, each call and text has a very noticeable and real monetary cost.

The economist Joshua Blumenstock at the University of Washington uses cellphone metadata (who calls whom, when and for how long) to measure wealth. For instance, people who make calls at certain times of the day are wealthier, and people who make lots of short calls tend to be poorer than people who make fewer, longer ones. In a paper in Science, using data from Rwanda, he quantifies poverty at very high levels of resolution, focusing not just on individual villages but on individual people.

Of course, all these data sources — satellites, cellphones and many others — work even better in concert. Flowminder is a non-profit organisation that has taken on this challenge of combining data.

Researchers are beginning to see tangible benefits. For example, GiveDirectly, a non-profit group that gives cash to the poor, now uses satellite imagery to identify villages where thatched roofs signal that they may need help. “Remote sensing data can be powerful, especially when combined with cheap classification tools like crowdsourcing or machine learning,” says Paul Niehaus, co-founder of the organisation. “They’ve let us strip cost and time out of the process.”

The World Bank recently held its Big Data Innovation Challenge. Many of the winners used novel data sets to improve measurement and decision making, and their titles paint vivid, if wonky, pictures: “Improved Real Time Decision Making of Infrastructure Investments for the Philippines by Linking Geo-Spatial Road Network Data With Rich Geo-Tagged Social Data Collected Through Mobile Phones” and “Combining Taxi GPS Data and Open-Source Software for Evidence-Based Traffic Management and Planning.”

Some of these projects are about changing how old sources of data are used. Typically, surveys are used to determine which variables are most correlated with poverty, such as having a thatched roof or a dirt floor or not having a toilet. These variables are then added up to form a score that ranks households. Those with the highest rank may be deemed eligible for help, whether it is a microloan, a food subsidy or a cash transfer.

But this kind of simple score card does not take advantage of the latest technologies. For example, in the United States, modern credit scoring agencies do not simply add a few variables. They use predictive algorithms to scour data for complex relationships most associated with default, which are then combined into a tailored prediction of credit risk. Why should the financial services industry, where mere dollars are at stake, be using more advanced technologies than the aid industry, where human life is at stake?

A winning World Bank Innovation Challenge project, with which I collaborated, aims to close this gap. Melissa Adelman, an economist at the World Bank and one of the lead researchers in our project, says: “Existing surveys may contain much more information for poverty measurement than is actually used. We’re exploring how many more of the poor could be identified and targeted for services by using machine-learning tools. It’s exciting because those better algorithms are essentially free to use, so the benefit-to-cost ratio of any gain in targeting will be very large.”

All of these efforts improve efficiency. It doesn’t sound very romantic: Costs go down and accuracy goes up. Typically, efficiency gains like these excite only technocrats. But when these improvements truly help the neediest, all of us should be excited. — The New York Times

* Sendhil Mullainathan is the Robert C. Waggoner Professor of Economics at Harvard.