# Economics

## Overview

Born of the Active Learning Initiative (ALI) in 2017, the economic education research group at Cornell focuses on developing standard assessments of undergraduate learning and analyzing the copious data generated by the ALI to learn how students learn economics and how we can best teach the subject. We also work closely with education researchers in other disciplines to study big issues like gender gaps in performance and the determinants of long-term retention of knowledge.

## Affilliated Researchers

**Faculty**

**Research Scientists**

**Undergraduate Researchers**

- Anna McDougall
- Steven Zhu

## Learning Goals

Working closely with our instructors, we have documented the learning goals for six of our core undergraduate courses, and by 2021 we will have drafts for every required course in our major. While they are certainly not appropriate for courses taught at every institution, we share them here as they can be a good starting point for other instructors looking to document the goals of their courses.

- Introduction to Microeconomics
- Introduction to Macroeconomics (coming 2021)
- Intermediate Microeconomic Theory
- Intermediate Macroeconomic Theory (coming 2021)
- Introduction to Economic Statistics
- Applied Econometrics
- Introduction to Probability and Statistics (Math-Intensive)
- Theory-based Econometrics

When we document learning goals (LGs) for a course, we are being explicit about what we want students to know and be able to do by the end of the term. These goals are written for three distinct audiences:

*Course instructors:*The LGs will form the basis of the lectures we design and the exams we give.-
*Instructors who teach more advanced courses:*The LGs explain what incoming students know (or should know) when they walk in the door. *Students in the class:*The LGs provide a clear exposition of what they will learn from the course.

The devil is in the details with learning goals—There’s a big difference between a topic you might see on a syllabus like "Instrumental Variables" and the following learning goal that describes a particular set of skills students should acquire:

- Judge situations where Instrumental Variables (IV) can and cannot be applied to obtain an unbiased coefficient estimate.

- Explain why IV estimation (using two stage least squares) yields unbiased estimates.
- Evaluate whether the instrumental variable is correlated with the endogenous variable and assess its strength.
- Evaluate whether the instrumental variable is correlated with the error term.

The topic is often meaningless to incoming students while the learning goals are much easier to turn into a lecture or assess in an exam.

## Standard Assessments

Standard assessments of student learning have been vital to quantifying the effectiveness of different teaching methods (Freeman et al. 2014). In addition to allowing instructors to evaluate whether learning improves in response to teaching changes, an objective formal assessment given at the beginning of the term allows instructors of advanced courses to evaluate student preparedness and tailor their teaching accordingly. Physics, biology, and chemistry have well over 100 publicly available assessments that cover a wide range of courses and topics. Until now, economics has had only one high quality standard assessment that is appropriate for undergraduate students: the Test of Understanding College Economics (TUCE).

We are developing a comprehensive suite of assessments appropriate for a wide range of undergraduate economics courses. Our development process is in large part based on the procedure described in Adams and Wieman (2011). We start by documenting the learning goals we want to test and draft a set of corresponding multiple-choice questions. We then recruit faculty here and outside Cornell to provide us with feedback on whether the assessment evaluates expert-level thinking. Following revisions, we conduct interviews with undergraduate students who have previously taken the course where they verbalize their thought process as they answer the questions. Based on these interviews, we remove ambiguity in questions and add options that correspond to common mistakes.

At the end of the process, our assessments are piloted internally and externally in real classrooms. This allows us to compute standard measures of test quality and report national mean performance overall and by learning goal.

To maintain the integrity of the assessments, we do not post complete questionnaires for any of our assessments here. Please contact us directly at econ-assessments@cornell.edu if you are interested in using one of them in a course you are teaching.

Economic Statistics Skills Assessment (ESSA)

ESSA is designed to test the learning goals of an introductory probability and statistics course that is geared toward economics students. It is composed of 20 multiple choice questions, and students may refer to a provided formula sheet during the test. ESSA has been piloted inside and outside Cornell.

Applied Econometrics Skills Assessment (AESA)

AESA is designed to test the learning goals of an applied econometrics course that follows an introductory course in probability and statistics. It is composed of 24 multiple choice questions about linear regression models and more advanced methods including binary outcome models, instrumental variables, and fixed effects. The subset of 16 questions on linear regression are designated AESA-Core, while the complete assessment is called AESA-Advanced. Both versions of AESA have been piloted inside and outside Cornell.

Intermediate Economics Skills Assessment - Micro (IESA-Micro)

IESA-Micro is designed to test the learning goals of a calculus-based course in intermediate microeconomic theory. It is composed of 31 multiple choice questions that test both conceptual understanding and the ability to use economic theory to make predictions. IESA-Micro has been piloted at Cornell.

Math-Intensive Economic Statistics Skills Assessment (MI-ESSA)

MI-ESSA is designed to test the learning goals of a math-intensive introductory probability and statistics course that is geared toward economics students. It is composed of 31 multiple choice questions, and students may refer to a provided formula sheet during the test. MI-ESSA has been piloted at Cornell.

Theory-based Econometrics Skills Assessment (TESA)

TESA is designed to test the learning goals of a more theory-based econometrics course that follows a rigorous introductory course in probability and statistics. It is composed of 21 multiple choice questions about linear regression models and more advanced methods including binary outcome models and instrumental variables. TESA has been piloted at Cornell.

Math for Economics Skills Assessment (MESA)

MESA is a set of questions that can be organized into assessments of the math skills that are required for success in both introductory and intermediate level economics courses. MESA-Foundations contains questions that assess skills in arithmetic, geometry and graphs, and algebra. MESA-Intermediate removes the arithmetic questions and adds questions on more advanced algebra and calculus. The tests help instructors teach students based on the skills they have instead of the skills we hope they have. In addition, the test can identify early in the semester those students that will need extra support during the term. MESA is currently under development in collaboration with Irene Foster at George Washington University.

Principles of Economics Skills Assessment - Micro (PESA-Micro)

PESA-Micro is an alternative to the TUCE that is appropriate for courses that are more explicitly theory-based.

## Classroom Activities

Through the Active Learning Initiative, we have worked with our instructors to build many classroom activities. We share several here that we have used successfully in our own classes.

An invention activity is a teaching technique that involves giving students a difficult substantive problem that cannot be readily solved with any methods they have already learned. Research suggests that such activities prepare students to learn the “expert's solution” better than starting with a lecture on that solution.The following six activities are described in detail in our paper Using Invention Activities to Teach Econometrics, and we have fielded them in several semesters of our own Applied Econometrics course.

**Bivariate Regression:**Students carefully examine several scatterplots and "invent" procedures for fitting a line through them. We expect some to suggest Ordinary Least Squares, but have seen some very creative solutions.**Categorical Independent Variables:**Students construct a model of customer demand for coffee at a local shop. They need to come up with a good way of controlling for season, a discrete variable with four categories, before they have been told that they should create a set of dummy variables for it or decide on a reference category.**Heterogeneous Effects:**Students are asked to generalize a linear model to allow the effect of one variable to depend on the values of another variable. This is followed by a lecture on including and interpreting interaction terms in models.**Difference-in-Differences:**Students are asked to estimate the effect of a policy intervention. They have data for the years before and after it was implemented in the area and a in a nearby area that was not affected by the policy.**Regression Discontinuity:**Students consider scholarship program that gives small awards to high school students who exceed a particular test score threshold if they go on to attend a public 4-year college. In this exercise students carefully examine six scatter plots that show the relationship between high school test scores and college enrollment. They must suggest an unbiased procedure for estimating the scholarship's effect.**Fixed Effects:**Students are asked to tranform a fixed effect model into one that can be estimated with ordinary least squares. This is a prelude to a short lecture on estimation of fixed effect models that includes first differences.

For our Intermediate Microeconomic Theory course, we have developed 10 small group activities, some of which would also be appropriate for an introductory microeconomics course. Development of these activities was paid for through an NSF-sponsored Team-Based Learning project, and they are currently hosted on the Starting Point: Teaching and Learning Economics site:

**Price Elasticity of Demand:**Teams are given a list of goods and asked to identify what they think are the most elastic and least elastic.**The Value of Car Insurance:**Teams identify the factors that affect the decision to purchase car insurance.**Food Trucks in the Short Run and the Long Run:**Teams forecast the profitability of a food truck business in the long run, ideally taking into account the roles of their cost structure, potential market entry, and regulation.**Welfare Loss from Monopoly:**Students are presented with several potential markets and asked to consider the welfare consequences of monopoly power in each.**Nonlinear Budget Constraints:**Teams graph both linear and nonlinear budget constraints and identify optimal choices for consumers with different preferences.**Big Players in a Small Market:**In the context of an oligopolistic entry game, students will translate a word problem into game theoretic notation and will then examine how changes to the payoff structure changes the Nash equilibria in a game. They will also solve for and discuss the Nash equilibrium in mixed strategies.**Location Choice for a Small Business:**Students will discuss the location choice of a small business and how this choice determines the market structures in which the business would have to operate. Students will discuss how a profit-maximizing firm would behave in each of the three locations and what would be the resulting outcome in terms of revenue and profits. The exercise aims at helping students recognize different market structures and connect simplified "real world" descriptions to theoretical models.**Keeping Each Other in Check:**Students will examine the behavior of two large oligopolists, deciding on which factors serve as the biggest deterrents to the firms' collusion. The main factor preventing collusion is the temptation to deviate from the agreement for both firms, which leads the Nash equilibrium in which both firms do not cooperate.**Shutdown Decisions:**Students will discuss full and partial shutdown decisions in a context of a small business (bagel shop) which, with the advent of cold weather, considers closing its patio, keeping it open and putting up heating lamps, or closing altogether for the months of November through March. The exercise starts with a conceptual-level discussion and continues with a (relatively simple) calculation based on fixed and variable costs.**Smoking and Nicotine in New York:**Students will examine the welfare effects of a tax increase in a market with negative externalities in the context of the New York City tobacco and e-cigarette market. The exercise offers a more challenging setting than traditional single-market-focused examples by having the students consider a related market for a substitute (e-cigarettes) and two groups of consumers (adult and adolescent smokers).

## Links

## Research

**"Learning during the COVID-19 pandemic: It's not who you teach, but how you teach"** (Orlov, McKee, Berry, Boyle, DiCiccio, Ransom, Rees-Jones, and Stoye) Economics Letters, 2021.

We use unique data from seven intermediate economics courses taught at four R1 institutions to examine the effects of the COVID-19 pandemic on student learning. Because the same assessments of course knowledge mastery were administered across semesters, we can cleanly infer the impact of the unanticipated switch to remote teaching in Spring 2020. During the pandemic, total assessment scores declined by 0.2 standard deviations on average. However, we find substantial heterogeneity in learning outcomes across courses. Course instructors were surveyed about their pedagogy practices and our analysis suggests that prior online teaching experience and teaching methods that encouraged active engagement, such as the use of small group activities and projects, played an important role in mitigating this negative effect. In contrast, we find that student characteristics, including gender, race, and first-generation status, had no significant association with the decline in student performance in the pandemic semester.

**"Identifying Students at Risk with a New Math Skills Assessment"** (Orlov, McKee, Foster, Bottan, and Thomas) AEA Papers and Proceedings, 2021.

Math skills are critical for success in economics courses. Introductory courses require students to be comfortable with arithmetic, algebra, and working with graphs. Intermediate level courses add calculus to this list. Students with weak math skills usually perform poorly or drop their economics courses midway through the semester. If students-at-risk can be identified at the beginning of the term, they can be given the support they need to succeed. It may involve encouraging them to attend office hours regularly, use a department tutoring service, or enroll in a supplemental course. In this paper we present two new math assessments that economics instructors can use to evaluate students early in the semester. We then show how our assessments, given at the beginning of the term, predicted performance of 900 students in an introductory microeconomics class and 200 students in an intermediate microeconomic theory class. We analyze course completion and class grades in both courses and show the use of a variety of techniques that use math test scores and easy to collect covariates to identify students for remedial policy implementation.

**"Using Invention Activities to Teach Econometrics"** (McKee and Orlov) Journal of Economics Teaching, 2021.

An invention activity is a teaching technique that involves giving students a difficult substantive problem that cannot be readily solved with any methods they have already learned. The work of Dan Schwartz and colleagues (Schwartz & Bransford, 1998; Schwartz & Martin, 2004), suggests that such activities prepare students to learn the “expert's solution” better than starting with a lecture on that solution. In this paper we present six new invention activities appropriate for a college econometrics course. We describe how we introduce each activity, guide students as they work, and wrap up the activity with a short lecture.

**"Teaching Students to Read Journal Articles Critically"** (Orlov) Journal of Economic Education, forthcoming.

In this paper, the author describes the use of primary literature readings in an upper-division undergraduate field course. One of the two main learning goals of the course was to teach students how to read academic articles in economics with a critical eye. This was accomplished through providing students with a structured framework for summarizing the main methods and results of each paper and feedback provided on short paper reports and during in-class discussion activities. Based on his experiences in this course, the author offers observations and suggestions to instructors wishing to integrate non-textbook academic readings in their teaching.

**"Teaching Economic Evaluation with Population Health Cases"** (Green, Bolbocian, Busken, Gonzalez, McKee, and Xu) Journal of Health Administration Education, 2017.

Economic evaluation is one of the largest competency gaps for public health practitioners, and researchers recommend increased training of the public health workforce in economic evaluation. This paper contributes to the literatures on case teaching in economic theory, health economics, and other fields of economics. Authors describe a technology-enhanced, case-based economic evaluation course in a school of public health at a private Midwestern university.

**"Racial and Gender Achievement Gaps in an Economics Classroom"** (Bottan, McKee, Orlov, and McDougall) (under review)

In this paper, we document gender and race/ethnic achievement gaps over four semesters of an intermediate-level economics course. We find that male under- represented minority (URM) students earned lower final exam scores than male non-URM students, but this gap disappears when we control for differences in prior preparation. In contrast, female URM students performed significantly worse than female non-URM students, even after controlling for prior preparation. We analyze scores on low-stakes assessments and surveys about study behavior and find that the theory of stereotype threat most consistently explains our results. As these issues are unlikely to be unique to our classroom, we offer several potential pedagogical solutions to address differences in prior preparation and stereotype threat that underlie observed achievement gaps.

**"Explaining Heterogeneity Across Departments in Diversity of Economics Students"** (McDougall, McKee, and Orlov) (working paper)

The field of economics is severely lacking in diversity, lagging behind other STEM fields that have drastically improved on this front in the past decades. However, there is little consensus on the underlying causes of or most effective solutions to this problem. In this paper, we combine data from the Integrated Postsecondary Education Data System (IPEDS) with data from our own survey of economics departments to identify characteristics of institutions and departments that are associated with observed variation in diversity across departments. We explore four avenues that the existing literature suggests departments could pursue to improve the gender and racial diversity of their undergraduate students: student support, role modeling, course content, and the use of active learning pedagogy in the classroom. We find little to no association of variables linked to the first two approaches and either gender or racial diversity. On the other hand, we find a positive association between course content (economics courses with feminist theory) and gender diversity and between the use active learning pedagogy and gender diversity. Unfortunately, we find no such associations for racial diversity, leaving open the question of whether other avenues for increasing the uptake of the economics major by underrepresented minority students need to be explored.

**"Total Recall? Short- and Long-term Retention of Statistics and Econometrics Skills"** (McKee and Orlov) (working paper)

While written exams are commonly used to measure student learning during or at the end of a course, research on the amount of material students retain over time is very limited. In this paper, we use unique data to study the retention of statistics and econometrics skills after course completion. We measure student skills using low- stakes assessments given regularly at our institution prior to the final examinations. The Economic Statistics Skills Assessment (ESSA) is administered at the end of the introductory statistics course, and at the start of the econometrics course, as a test of pre-existing skills. The difference in ESSA scores for students who took it after their winter and summer breaks, 1.5 and 4 months later respectively, is used as a measure of statistics skills retention. The Applied Econometrics Skills Assessment (AESA) is administered at the end of the econometrics course. To measure knowledge retention in econometrics, we induced a large proportion of non-graduating Fall 2017 and Spring 2018 students to take AESA a second time, a year after they completed the course. We find that a longer interval between the pre- and post-tests is associated with a worse performance on the post-test. Taking courses that apply statistics and econometrics skills are associated with better learning retention; however, courses on statistics and econometrics methods introducing new concepts are associated with worse performance on the post-test. We further find that female and underrepresented minority students have worse learning retention, on average. On the other hand, first generation students perform better on the post-test.

**"The Economic Statistics Skills Assessment (ESSA)"** (McKee and Orlov) (under review)

Measurements of student knowledge and skills are highly useful both upon the entry of students into a course, so that the gaps in prerequisite knowledge can be addressed, as well as upon course completion, so that the impact of any interventions and changes to the course can be evaluated. Final examinations often do not provide the desired coverage and are difficult to compare across terms and institutions. Within most STEM fields, this problem is solved by the use of concept inventories, which are designed as low-stakes standardized assessments of students’ core knowledge. With the exception of the Test of Understanding College Economics (TUCE), which tests introductory economics knowledge, economics as a field did not have such assessments. In this paper, we document the design, development, and validation of a 20-question Economic Statistics Skills Assessment (ESSA) that we created to test the student knowledge and understanding of probability and statistics concepts. The assessment was reviewed by economics faculty across multiple public and private institutions, validated via think- aloud interviews with students, and taken by students at multiple institutions at the conclusion of their statistics for economics courses or the start of their econometrics courses. We demonstrate, using statistical analysis, that the items in ESSA capture whether students have developed the understanding of specific probability and statistics concepts.

**"The Applied Econometrics Skills Assessment (AESA)"** (Orlov and McKee) (working paper)

Final exam scores and final course grades are not always reflective of student learning in courses. Unlike most STEM fields, Economics currently lacks high quality standardized assessments of learning outcomes, with the exception of the Test of Understanding of College Economics (Walstad, Watts, and Rebeck 2007), which targets introductory economics courses. The use of econometric methods is a crucial skill that all economics majors should develop and, hence, the ability to evaluate whether learning improves when the instructor changes her teaching approach is highly valuable. Further, as student self-reports of pre-existing skills are unreliable, an objective formal assessment allows instructors of advanced courses to evaluate the student preparedness and tailor the teaching accordingly. The Applied Econometrics Skills Assessment (AESA) is designed to serve both purposes.

**"Who Comes to Office Hours?"** (Bottan, McKee, and Orlov) (working paper)

Professors spend a substantial fraction of their teaching time and effort providing support to students in office hours. This paper uses attendance and survey data from two introductory economics courses and two more advanced courses to identify what characteristics of students predict attendance at office hours and why some students do not attend. We also shed light on what strategies might be effective for encouraging students to attend office hours. We find attendance rates vary substantially across courses, but female students consistently attend more than male students. Holding gender and race constant, students with very low or very high GPAs attend more often than students in the middle range. The two most common reasons given for non-attendance are a lack of perceived need for help and scheduled office hours conflicting with other classes. We find that those students who state they do not need the extra support do tend to perform better on final exams. Finally, we show that encouraging students to attend office hours through an email message had little impact, but providing small extra credit for attendance significantly increases student participation.