March 2020. GrowthPolicy’s Devjani Roy interviewed Dan Levy, Senior Lecturer in Public Policy at Harvard Kennedy School and Faculty Director of Harvard Kennedy School’s Public Leadership Credential, on data-driven policy making, machine learning and human judgment, and the future of higher education in the age of technology. | Read more interviews like this one.
Related Links: Professor Levy’s Harvard faculty page | Research | Harvard Kennedy School Public Leadership Credential | Transparency for Development Project | Evidence for Policy Design (EPoD) | Building Capacity to Use Research Evidence (BCURE) | Teachly
GrowthPolicy.org: You have done considerable research and teaching on applying insights from data to make policy decisions. What are some of your new projects and insights you can share with our readers apropos of applying data to policy making.
Dan Levy: Broadly speaking, I think of three types of uses of data to inform decisions: descriptive, causal, and predictive. I will give an example of each of these to answer your questions. On the descriptive front, we have had for many years areas of policy (such as performance management, program monitoring and evaluation, etc.) that leverage data in this manner. One development in this area that I find interesting and exciting is the use of data that are updated live for decision-makers. Whereas in the past, typical data dashboards were looking backward for weeks, months, or even years, nowadays we increasingly see the use of dashboards (especially at the level of city governments) that are updated live and allow decisionmakers to make time-sensitive decisions in a more effective manner. On the causal front, economists and other social scientists have been obsessed for many years on trying to assess causal effects of programs and policies. It is not easy to isolate causal effects because so many things other than the policy or program in question affects the outcome of interest. But in my mind, there have been three interesting developments in this area: First, the advent of increasingly sophisticated statistical methods to assess causal effects. Second, the increased use—and more recently, recognition with the 2019 Nobel Prize in Economics given to Banerjee, Duflo, and Kremer—of randomized-controlled trials as a method to assess causal effects of programs in a wide range of policy areas (education, health, governance, microfinance, poverty, etc.). And third, the use of novel sources of data to measure (and sometimes validate) important indicators in a timelier manner than the traditionally collected data. For example, night-light data being used to measure economic activity or mobile-phone data to estimate poverty or health. On the predictive front, there has been an explosion of uses of data in this area that I find very exciting. Some of the excitement in academia comes from seeing the use of sophisticated machine-learning methods and the emergence and popularization of disciplines like data science. But from a policy perspective, the really exciting development in my view is the number of areas where using data to predict has allowed better decisions and improved social welfare. One typical situation where this happens is when governments need to prioritize because of lack of time or resources, and predictive data helps them do so. As a simple example, take restaurant safety inspections. Many city governments don’t have resources to visit restaurants very frequently, which means that some restaurants might violate safety regulations for months or even years and the only way we would find out is if there is a tragedy. But nowadays, armed with data and statistical methods, restaurants can be ranked in terms of safety risk, and the inspectors can go to the higher-risk restaurants first. There are countless examples like this in many other areas of policy. Having said all of this, I don’t want to end without acknowledging some of the potential downsides of some of the algorithms in reinforcing discrimination or other injustices in society.
GrowthPolicy.org: You are one of the co-authors of a widely used HBS Case Study: “Improving Worker Safety in the Era of Machine Learning.” My question is on the topic of decision making, which typically involves two components: the ability to predict accurately and the ability to apply human judgement accurately. The strides made in machine learning have reduced the cost of the first component—i.e., prediction—thanks to the growth of big data and the advanced algorithms that big data have made possible. In such an environment, what according to you, is the importance of human judgement? How may human judgement compete, if at all, with machine learning?
Dan Levy: I think this is a great question. In my view, the distinction you make between the ability to predict accurately and the ability to apply human judgement accurately is a very important one. I think machines will increasingly take over activities related to accurate prediction because in many areas they can do a better job than human beings can. So professions that are based on the ability to predict (such as weather forecasters, radiologists, and others) will become less well remunerated and popular because the machines will act as substitutes for these professions. But I think human judgment related to what decisions to make or actions to take based on these predictions will become more valuable because it is a complement to, not a substitute of, the work of machines. So radiologists making judgments about whether a person has a disease will have a harder time in the future than doctors who need to take a probabilistic prediction (this patient has an X% chance of having the disease) and a host of other factors into account to decide what is the best course of action for the patient. If you are interested in the use of economics to understand the impact of these prediction algorithms on labor market and other societal outcomes, there is a wonderful book that illuminates these issues very well (Prediction Machines). I highly recommend it. In the world of policy, the optimal decision is often not driven only by what is technically correct (which a prediction algorithm will help you figure out), but also by a myriad of other factors (administrative feasibility, political supportability, legal standing, etc.), so if anything, I think the advent of prediction algorithms will make the job of policymakers more important.
GrowthPolicy.org: You are known to be a polymath who has made impactful contributions to the scholarship of teaching and learning. What are some ways in which today’s brick-and-mortar universities might restructure themselves, often in the face of hierarchical structures and internal bureaucracies, to serve student needs more effectively?
Dan Levy: First, I feel honored you think I am a polymath. In terms of your question, it is becoming increasingly clear that the traditional model of brick-and-mortar universities is being disrupted. As an educator who is very passionate about teaching and learning, I still think that teaching is fundamentally a social process that benefits greatly from the relationship between instructor and students and the relationships among students. I think there is something magic that can happen in a classroom and more broadly in a university campus that is hard to replicate online. But I also think that technology is substituting some of what happens on campuses nowadays. And for some education-related activities (such as learning that adapts to the skills, preferences, and pace of each individual learner), technology has a comparative advantage over traditional residential education. So I think successful universities will be the ones that leverage the comparative advantage of the physical presence and work hard to create magic in their classrooms and campuses, while at the same time leveraging technology to complement what they do residentially.
GrowthPolicy.org: You are deeply involved with multiple projects that use research-based evidence and policy interventions to improve healthcare delivery and access to health services in the developing world. What are some insights from your research that health systems in the developed world might learn from?
Dan Levy: I participated recently in a large multi-year project called Transparency for Development, which consisted in designing and evaluating a transparency and accountability intervention aimed at improving health outcomes and empowerment in rural villages in Indonesia and Tanzania. Two broad lessons from the project for me: First, it is very challenging to go from communities taking actions to generating impacts on health outcomes (see here). Second, you can learn much more from a study if you bring and combine the strengths of various disciplines and methods to shed light on the issues you are examining. In our case, our team use mixed methods that involved the use of randomized-controlled trials, qualitative interviews, focused groups, ethnographic studies, and other methods that allowed us to understand what happened in a much richer way than we could have if we had used relied on any of these methods alone.
GrowthPolicy.org: You are the faculty director of the Harvard Kennedy School Public Leadership Credential (PLC), a new non-degree online learning program for mid-career professionals. As I understand it, the PLC uses the Kennedy School’s three signature pedagogies—simulations, group work, and case studies—in an online learning environment. What does the success of the PLC teach us about the possibilities for radical disruption and the future of innovation within higher education?
Dan Levy: At the Kennedy School, we are very excited about our Public Leadership Credential for mainly two reasons: First, it is helping us expand our reach by allowing people who might not be able to come to the Kennedy School for personal, professional, or financial reasons, to benefit from our ideas and our teaching so they can help advance the public good all over the world. Second, we took the opportunity to think about how we could bring some of the best ideas and teaching of the Kennedy School into online education. This made us rethink the relative comparative advantages of residential and online education, and we are proud of what we have come up with so far. Through the careful work of faculty leads, learning designers, and other professionals, we put learning (rather than information delivery) at the center of the enterprise, and are offering courses where the learners have to do real work in the real world that will equip them to advance the public good.
GrowthPolicy.org: Where will the jobs of the future come from?
Dan Levy: I don’t think I have anything to say here that others haven’t said, but I think the jobs of the future are increasingly likely to be in the service sector rather than in agriculture or manufacturing, and in general they are likely to be ones in which humans have a comparative advantage over machines, i.e., jobs that require judgment, social skills, empathy, and many of the qualities that make us human. And experts argue that many of the jobs of the future don’t even exist today. But these are just predictions, and we as human beings don’t have a great track record when making predictions. So I will humbly call these speculations about the future rather than predictions.