Economics

Overview

Born of the Active Learning Initiative (ALI) in 2017, the economic education research group at Cornell focuses on developing standard assessments of undergraduate learning and analyzing the copious data generated by the ALI to learn how students learn economics and how we can best teach the subject. We also work closely with education researchers in other disciplines to study big issues like gender gaps in performance, the determinants of long-term retention of knowledge, and peer-to-peer learning.

Affilliated Researchers

Faculty

Douglas McKee

Research Scientists

George Orlov

Undergraduate Researchers

Richard Amaro
Vidya Balaji
Jessica Chen
Tuan Dang
Sophia Giarrusso
Raymond Wang
Finn Ye
Steven Zhu

Learning Goals

Working closely with our instructors, we have documented the learning goals for every required course in our undergraduate major. While they are certainly not appropriate for courses taught at every institution, we share them here as they can be a good starting point for other instructors looking to document the goals of their courses.

When we document learning goals (LGs) for a course, we are being explicit about what we want students to know and be able to do by the end of the term. These goals are written for three distinct audiences:

Course instructors: The LGs will form the basis of the lectures we design and the exams we give.
Instructors who teach more advanced courses: The LGs explain what incoming students know (or should know) when they walk in the door.
Students in the class: The LGs provide a clear exposition of what they will learn from the course.

The devil is in the details with learning goals—There’s a big difference between a topic you might see on a syllabus like "Instrumental Variables" and the following learning goal that describes a particular set of skills students should acquire:

Judge situations where Instrumental Variables (IV) can and cannot be applied to obtain an unbiased coefficient estimate.

Explain why IV estimation (using two stage least squares) yields unbiased estimates.
Evaluate whether the instrumental variable is correlated with the endogenous variable and assess its strength.
Evaluate whether the instrumental variable is correlated with the error term.

The topic is often meaningless to incoming students while the learning goals are much easier to turn into a lecture or assess in an exam.

Standard Assessments

Standard assessments of student learning have been vital to quantifying the effectiveness of different teaching methods (Freeman et al. 2014). In addition to allowing instructors to evaluate whether learning improves in response to teaching changes, an objective formal assessment given at the beginning of the term allows instructors of advanced courses to evaluate student preparedness and tailor their teaching accordingly. Physics, biology, and chemistry have well over 100 publicly available assessments that cover a wide range of courses and topics. Until now, economics has had only one high quality standard assessment that is appropriate for undergraduate students: the Test of Understanding College Economics (TUCE).

In 2017 we started developing a comprehensive suite of assessments appropriate for a wide range of undergraduate economics courses. Our development process is in large part based on the procedure described in Adams and Wieman (2011). Our first step is to document the learning goals we want to test and draft a set of corresponding multiple-choice questions. We then recruit faculty here and outside Cornell to provide us with feedback on whether the assessment evaluates expert-level thinking. Following revisions, we conduct interviews with undergraduate students who have previously taken the course where they verbalize their thought process as they answer the questions. Based on these interviews, we remove ambiguity in questions and add options that correspond to common mistakes.

At the end of the process, our assessments are piloted internally and externally in real classrooms. This allows us to compute standard measures of test quality and report national mean performance overall and by learning goal.

In 2022 we launched econ-assessments.org which hosts nine standard assessments developed at Cornell for commonly taught economics courses and allows instructors worldwide to set up any of these assessments for their students. Instructors get a link to their chosen assessment that can be shared with students, and students take the timed assessment when they follow the link. At the end of the test-taking period, instructors receive a report that summarizes performance in aggregate and by learning goal.

Classroom Activities

Through the Active Learning Initiative, we have worked with our instructors to build many classroom activities. We share several here that we have used successfully in our own classes.

An invention activity is a teaching technique that involves giving students a difficult substantive problem that cannot be readily solved with any methods they have already learned. Research suggests that such activities prepare students to learn the “expert's solution” better than starting with a lecture on that solution.The following six activities are described in detail in our paper Using Invention Activities to Teach Econometrics, and we have fielded them in several semesters of our own Applied Econometrics course.

Bivariate Regression: Students carefully examine several scatterplots and "invent" procedures for fitting a line through them. We expect some to suggest Ordinary Least Squares, but have seen some very creative solutions.
Categorical Independent Variables: Students construct a model of customer demand for coffee at a local shop. They need to come up with a good way of controlling for season, a discrete variable with four categories, before they have been told that they should create a set of dummy variables for it or decide on a reference category.
Heterogeneous Effects: Students are asked to generalize a linear model to allow the effect of one variable to depend on the values of another variable. This is followed by a lecture on including and interpreting interaction terms in models.
Difference-in-Differences: Students are asked to estimate the effect of a policy intervention. They have data for the years before and after it was implemented in the area and a in a nearby area that was not affected by the policy.
Regression Discontinuity: Students consider scholarship program that gives small awards to high school students who exceed a particular test score threshold if they go on to attend a public 4-year college. In this exercise students carefully examine six scatter plots that show the relationship between high school test scores and college enrollment. They must suggest an unbiased procedure for estimating the scholarship's effect.
Fixed Effects: Students are asked to tranform a fixed effect model into one that can be estimated with ordinary least squares. This is a prelude to a short lecture on estimation of fixed effect models that includes first differences.

For our Intermediate Microeconomic Theory course, we have developed 10 small group activities, some of which would also be appropriate for an introductory microeconomics course. Development of these activities was paid for through an NSF-sponsored Team-Based Learning project, and they are currently hosted on the Starting Point: Teaching and Learning Economics site:

Price Elasticity of Demand: Teams are given a list of goods and asked to identify what they think are the most elastic and least elastic.
The Value of Car Insurance: Teams identify the factors that affect the decision to purchase car insurance.
Food Trucks in the Short Run and the Long Run: Teams forecast the profitability of a food truck business in the long run, ideally taking into account the roles of their cost structure, potential market entry, and regulation.
Welfare Loss from Monopoly: Students are presented with several potential markets and asked to consider the welfare consequences of monopoly power in each.
Nonlinear Budget Constraints: Teams graph both linear and nonlinear budget constraints and identify optimal choices for consumers with different preferences.
Big Players in a Small Market: In the context of an oligopolistic entry game, students will translate a word problem into game theoretic notation and will then examine how changes to the payoff structure changes the Nash equilibria in a game. They will also solve for and discuss the Nash equilibrium in mixed strategies.
Location Choice for a Small Business: Students will discuss the location choice of a small business and how this choice determines the market structures in which the business would have to operate. Students will discuss how a profit-maximizing firm would behave in each of the three locations and what would be the resulting outcome in terms of revenue and profits. The exercise aims at helping students recognize different market structures and connect simplified "real world" descriptions to theoretical models.
Keeping Each Other in Check: Students will examine the behavior of two large oligopolists, deciding on which factors serve as the biggest deterrents to the firms' collusion. The main factor preventing collusion is the temptation to deviate from the agreement for both firms, which leads the Nash equilibrium in which both firms do not cooperate.
Shutdown Decisions: Students will discuss full and partial shutdown decisions in a context of a small business (bagel shop) which, with the advent of cold weather, considers closing its patio, keeping it open and putting up heating lamps, or closing altogether for the months of November through March. The exercise starts with a conceptual-level discussion and continues with a (relatively simple) calculation based on fixed and variable costs.
Smoking and Nicotine in New York: Students will examine the welfare effects of a tax increase in a market with negative externalities in the context of the New York City tobacco and e-cigarette market. The exercise offers a more challenging setting than traditional single-market-focused examples by having the students consider a related market for a substitute (e-cigarettes) and two groups of consumers (adult and adolescent smokers).

Research

"Econ-assessments.org: Automated Assessment of Economics Skills" (McKee, Zhu and George Orlov) Eastern Economic Journal, 2023.

Standard assessments can give instructors reliable objective measures of student skills at the beginning and end of a term. We describe seven standard assessments that we have developed for commonly taught economics courses, and introduce a new website, econ-assessments.org, that allows instructors worldwide to set up any of our assessments for their students. Instructors get a link to their chosen assessment that can be shared with students, and students take the timed assessment when they follow the link. At the end of the test-taking period, instructors receive a report that summarizes performance in aggregate and by learning goal.

"Racial and Gender Achievement Gaps in an Economics Classroom" (Bottan, McKee, Orlov, and McDougall) International Review of Economic Education, 2022.

In this paper, we document gender and race/ethnic achievement gaps over four semesters of an intermediate-level economics course. We find that male under- represented minority (URM) students earned lower final exam scores than male non-URM students, but this gap disappears when we control for differences in prior preparation. In contrast, female URM students performed significantly worse than female non-URM students, even after controlling for prior preparation. We analyze scores on low-stakes assessments and surveys about study behavior and find that the theory of stereotype threat most consistently explains our results. As these issues are unlikely to be unique to our classroom, we offer several potential pedagogical solutions to address differences in prior preparation and stereotype threat that underlie observed achievement gaps.

"Learning during the COVID-19 pandemic: It's not who you teach, but how you teach" (Orlov, McKee, Berry, Boyle, DiCiccio, Ransom, Rees-Jones, and Stoye) Economics Letters, 2021.

We use unique data from seven intermediate economics courses taught at four R1 institutions to examine the effects of the COVID-19 pandemic on student learning. Because the same assessments of course knowledge mastery were administered across semesters, we can cleanly infer the impact of the unanticipated switch to remote teaching in Spring 2020. During the pandemic, total assessment scores declined by 0.2 standard deviations on average. However, we find substantial heterogeneity in learning outcomes across courses. Course instructors were surveyed about their pedagogy practices and our analysis suggests that prior online teaching experience and teaching methods that encouraged active engagement, such as the use of small group activities and projects, played an important role in mitigating this negative effect. In contrast, we find that student characteristics, including gender, race, and first-generation status, had no significant association with the decline in student performance in the pandemic semester.

"Identifying Students at Risk with a New Math Skills Assessment" (Orlov, McKee, Foster, Bottan, and Thomas) AEA Papers and Proceedings, 2021.

Math skills are critical for success in economics courses. Introductory courses require students to be comfortable with arithmetic, algebra, and working with graphs. Intermediate level courses add calculus to this list. Students with weak math skills usually perform poorly or drop their economics courses midway through the semester. If students-at-risk can be identified at the beginning of the term, they can be given the support they need to succeed. It may involve encouraging them to attend office hours regularly, use a department tutoring service, or enroll in a supplemental course. In this paper we present two new math assessments that economics instructors can use to evaluate students early in the semester. We then show how our assessments, given at the beginning of the term, predicted performance of 900 students in an introductory microeconomics class and 200 students in an intermediate microeconomic theory class. We analyze course completion and class grades in both courses and show the use of a variety of techniques that use math test scores and easy to collect covariates to identify students for remedial policy implementation.

"Using Invention Activities to Teach Econometrics" (McKee and Orlov) Journal of Economics Teaching, 2021.

An invention activity is a teaching technique that involves giving students a difficult substantive problem that cannot be readily solved with any methods they have already learned. The work of Dan Schwartz and colleagues (Schwartz & Bransford, 1998; Schwartz & Martin, 2004), suggests that such activities prepare students to learn the “expert's solution” better than starting with a lecture on that solution. In this paper we present six new invention activities appropriate for a college econometrics course. We describe how we introduce each activity, guide students as they work, and wrap up the activity with a short lecture.

"Teaching Students to Read Journal Articles Critically" (Orlov) Journal of Economic Education, 2021.

In this paper, the author describes the use of primary literature readings in an upper-division undergraduate field course. One of the two main learning goals of the course was to teach students how to read academic articles in economics with a critical eye. This was accomplished through providing students with a structured framework for summarizing the main methods and results of each paper and feedback provided on short paper reports and during in-class discussion activities. Based on his experiences in this course, the author offers observations and suggestions to instructors wishing to integrate non-textbook academic readings in their teaching.

"Teaching Economic Evaluation with Population Health Cases" (Green, Bolbocian, Busken, Gonzalez, McKee, and Xu) Journal of Health Administration Education, 2017.

Economic evaluation is one of the largest competency gaps for public health practitioners, and researchers recommend increased training of the public health workforce in economic evaluation. This paper contributes to the literatures on case teaching in economic theory, health economics, and other fields of economics. Authors describe a technology-enhanced, case-based economic evaluation course in a school of public health at a private Midwestern university.

"Explaining Heterogeneity Across Departments in Diversity of Economics Students" (McDougall, McKee, and Orlov) (under revision)

The field of economics is severely lacking in diversity, lagging behind other STEM fields that have drastically improved on this front in the past decades. However, there is little consensus on the underlying causes of or most effective solutions to this problem. In this paper, we combine data from the Integrated Postsecondary Education Data System (IPEDS) with data from our own survey of economics departments to identify characteristics of institutions and departments that are associated with observed variation in diversity across departments. We explore four avenues that the existing literature suggests departments could pursue to improve the gender and racial diversity of their undergraduate students: student support, role modeling, course content, and the use of active learning pedagogy in the classroom. We find little to no association of variables linked to the first two approaches and either gender or racial diversity. On the other hand, we find a positive association between course content (economics courses with feminist theory) and gender diversity and between the use active learning pedagogy and gender diversity. Unfortunately, we find no such associations for racial diversity, leaving open the question of whether other avenues for increasing the uptake of the economics major by underrepresented minority students need to be explored.

"The Economic Statistics Skills Assessment (ESSA)" (McKee and Orlov) (under revision)

Measurements of student knowledge and skills are highly useful both upon the entry of students into a course, so that the gaps in prerequisite knowledge can be addressed, as well as upon course completion, so that the impact of any interventions and changes to the course can be evaluated. Final examinations often do not provide the desired coverage and are difficult to compare across terms and institutions. Within most STEM fields, this problem is solved by the use of concept inventories, which are designed as low-stakes standardized assessments of students’ core knowledge. With the exception of the Test of Understanding College Economics (TUCE), which tests introductory economics knowledge, economics as a field did not have such assessments. In this paper, we document the design, development, and validation of a 20-question Economic Statistics Skills Assessment (ESSA) that we created to test the student knowledge and understanding of probability and statistics concepts. The assessment was reviewed by economics faculty across multiple public and private institutions, validated via think- aloud interviews with students, and taken by students at multiple institutions at the conclusion of their statistics for economics courses or the start of their econometrics courses. We demonstrate, using statistical analysis, that the items in ESSA capture whether students have developed the understanding of specific probability and statistics concepts.

"Total Recall? Short- and Long-term Retention of Statistics and Econometrics Skills" (McKee and Orlov) (working paper)

While written exams are commonly used to measure student learning during or at the end of a course, research on the amount of material students retain over time is very limited. In this paper, we use unique data to study the retention of statistics and econometrics skills after course completion. We measure student skills using low- stakes assessments given regularly at our institution prior to the final examinations. The Economic Statistics Skills Assessment (ESSA) is administered at the end of the introductory statistics course, and at the start of the econometrics course, as a test of pre-existing skills. The difference in ESSA scores for students who took it after their winter and summer breaks, 1.5 and 4 months later respectively, is used as a measure of statistics skills retention. The Applied Econometrics Skills Assessment (AESA) is administered at the end of the econometrics course. To measure knowledge retention in econometrics, we induced a large proportion of non-graduating Fall 2017 and Spring 2018 students to take AESA a second time, a year after they completed the course. We find that a longer interval between the pre- and post-tests is associated with a worse performance on the post-test. Taking courses that apply statistics and econometrics skills are associated with better learning retention; however, courses on statistics and econometrics methods introducing new concepts are associated with worse performance on the post-test. We further find that female and underrepresented minority students have worse learning retention, on average. On the other hand, first generation students perform better on the post-test.

"The Effects of Peers on Effort and Achievement: Evidence from Random Group Assignments in Multiple Classrooms" (Orlov and McKee) (working paper)

While existence of peer effects in a variety of educational settings is well-established in the literature, the channels through which peer effects arise are less explored. However, understanding the root causes of peer effects is essential for effective education policy design. In this paper, we identify causal effects of peers on effort and achievement using data on 382 students randomly assigned to 107 groups within three large undergraduate economics courses that regularly incorporate small-group classroom activities and discussions. To quantify the degree of in-class exposure students have to one another, we use unique attendance data based on in-class polling questions. In-course study effort is measured using student reports of weekly and dedicated pre- exam study hours submitted as a part of each midterm and the final exam. Final exam scores and performance on low-stakes standard assessments serve as measures of student learning. Student data collected via end-of-semester surveys is further enhanced by administrative data, including start-of-semester GPA and demographic characteristics. We find that female peers both decrease female students’ study hours and improve their performance on final exams. Further, we see the importance of heterogeneity in peer effects. High-GPA students assigned to groups with higher proportions of low- GPA peers, see a reduction in weekly effort levels and large gains in test performance, explained through learning-by-teaching. This effect is not symmetric: Low-GPA students in groups with more high-GPA peers are also less likely to exhibit high weekly effort, but they experience no change in assessment performance. Finally, high-GPA grouped together seem to improve the learning experience for each other, performing better on low-stakes assessments and final exams, at least partially through increasing each other’s study effort.

"Improving Math Skills Using Concurrent Support Courses for Introductory Microeconomics" (Orlov, McKee, and McDougall) (working paper)

We use experimental data from an R1 PhD-granting university to estimate the impact of two types of one-credit supplemental support courses on student performance in Introductory Microeconomics. One version of the course focused on improving math skills, while the other helped students with more general economic problem solving. We employ difference-and-differences and fuzzy regression discontinuity designs to obtain estimates of treatment-on-the-treated effects and local average treatment effects respectively. We find that the math-focused supplemental course improved students’ mathematics skills for all attending students, and, to a much greater extent, for students who received lower scores on the math assessment at the beginning of the term. The latter effect is smaller but still present for the economics-focused supplemental course. Furthermore, our results suggest that the courses improved student performance on a standard introductory economics end-of-semester assessment. We find no statistically significant effects of either supplemental course on midterm or final exam scores.

"Student Stress and Its Impact on Learning During the COVID-19 Pandemic" (Orlov, McKee, and McDougall) (working paper)

In this project we use unique data collected in Spring 2020, Fall 2020, and Spring 2021 to determine how the learning of economics students was affected by perceived stress and how different groups of students experienced different kinds of stress. We find that stress related to living conditions and access to internet and technology was negatively associated with performance on end-of-term standardized assessments. We further find that female students, those not living in on-campus housing, and those receiving financial aid reported marginally greater stress related to living conditions, and first-generation students reported significantly higher amounts of this kind of stress. Underrepresented minority women, first-generation students, students receiving financial aid, and those not living in on-campus housing reported more stress related to internet and technology access.