Last week Mike Kofoed talked with my labor economics about his recent work related to the impact of mandatory online education. During the 2020-2021 school year, Penn State joined a host of other universities that taught the majority of their students remotely using a platform like Zoom or Teams. Some of the remote courses were asynchronous, and a lot of my colleagues tried to manage a hybrid workload where they attempted to teach to students who attended in the physical classroom at the same times as they taught those on a remote feed. I was very fortunate that I had large enough classes that my courses were delivered entirely over Zoom.
For the most part, I really thought my remote-only version of the course went fairly well. There were some hiccups along the way, but I thought the course turned out as best as it could given the circumstances. We had some technical issues like poor internet connections, but there weren’t ever major disruptions during the semester.
I worked really hard that year to come up with ways to make sure the Zoom experience was as close to the “real-life” version of the course a lot of my former students loved. I was able to convert some of my in-class demonstrations to a Zoom version and I always made sure to have some sort of fun virtual background. While I missed being in person, I was happy with how the remote version turned out.
This academic year, however, everything was supposed to “return to normal,” but instead I have ended up teaching in a hybrid manner for the past two semesters. There was so much uncertainty around both the Fall and Spring semesters that I didn’t feel comfortable asking 700 students to cram into an auditorium for an hour when we had the technology to offer other modalities. While my courses were officially listed as “in-person” courses, I offered students the option to attend remotely if they felt more comfortable doing that.
Attendance in the physical classroom has dropped to about half of what I was used to and a lot of students opted to be online. I still find it so strange given all the articles I saw about the dissatisfaction with Zoom classes. Regardless, my students are adults and are free to make their own decisions about how to attend a class given the options I originally opted to provide. I have struggled a lot since the start of the Fall semester because I can see the grades for those who self-select into the Zoom version compared to the residential version. There is a big gap for students who have opted into an online feed of the classroom.
The problem with what I see in the grade book is that I’ve given my students the option to self-select into the online version of the course. It’s not necessarily the case that the Zoom feed is causing their grades to be worse. It’s a similar empirical issue when people want to measure the impact of college education, moving across the country, or joining a union. It’s easy to see who made those decisions and observe their payoffs relative to those who didn’t make the same choice. The issue is measuring the causal impact of those choices rather than just observing the correlation.
It’s difficult for economists hard to run randomized control trials, particularly when it comes to such important life decisions as going to college or moving to a new city. Researchers attempt to infer causality by looking at natural experiments. This was actually the underlying focus for the 2021 Economics Nobel Prize winners. Unlike the Nobel winners, Mike Kofoed and his coauthors were actually allowed to complete a randomized control trial in the Fall of 2020 on their students because of their unique educational arrangement. Mike Kofoed and his coauthors are professors at West Point, the US’s military academy.
In order to conduct a randomized control trial measuring the impact of enrolling in synchronous online courses, students must be randomly assigned to different types of courses. Just as medical researchers test the impact of a new drug by giving some people the actual medicine and others the placebo, the ability to measure the causal impact rests on the notion that the recipients cannot choose which one they receive.
Unlike Mike’s students, mine currently get to choose whether they want to attend each day in the classroom or on the Zoom feed. The West Point students don’t get to choose because they are training to be Army officers and following orders is an important component of serving in the military. West Point students do get to pick their major, but they’re usually randomly assigned dorms, roommates, classes, class times, and instructors. During the Fall 2021 semester, West Point students who were selected to take principles of economics were randomly assigned to either a Zoom version of the class or a physical classroom.
There were a lot of other things the researchers controlled for in their experiment. Each instructor taught a portion of their classes in person and on Zoom which meant the researchers could control for the instructor’s impact on grades. Each timeslot during the day contained both in-person and Zoom options so the researchers could control for the difference between morning and afternoon classes. The research team also had a lot of demographic information about the cadets so that they could control for some level of prior ability. All of the students used the same online homework system and took the same exams so there was no difference in the difficulty of the assignments.
So, what did they find? Being randomly assigned to a Zoom class reduced a student’s grade (on average) by about 1.5 percentage points, but the impact was largest for the C/B students and academically at-risk students. Students who received the Zoom treatment also felt less connected to their peers and their instructors, even after controlling for instructor quality.
Here’s a look at the effects of online instruction on the different types of graded assignments. Lines that are completely below the horizontal axis are statistically significant between Zoom students and in-person students. The cadets were required to complete daily homework that was graded on completion, but online students couldn’t keep up with the in-person students. The impact is on standard deviations, but a 0.2 standard deviation is about 1.5 percentage points:
While the impact may not seem large, it’s important to remember what type of students enroll at West Point and the reputation of the university as a whole. The university is considered one of the top liberal arts colleges in the United States and classroom attendance is required. All of the students who are accepted have been nominated by their Congressional Representative or Senator or they have previously enlisted in the military. The Zoom impact is likely a “lower bound” of the impact of Zoom on university students, which means the impact on others may be much larger.
Mike and his team are in the process of measuring the impact on future economics courses and are hopeful that this paper will be published in the near future. It raises an interesting policy question, not only for the future of higher education but also on the importance of well-designed online education. The pandemic introduced a lot of faculty to online teaching for the first time and many of them have received little to no training. The social cost of this pivot could be much larger than we currently realize and may take some time to fully pan out.
From February 17, 2020, to April 6, 2020, Zoom usage grew 552% [Tech Republic]
West Point enrolls approximately 4,536 undergraduate students [US News and World Report]
West Point graduates must serve a minimum of 8 years after graduation in a combination of Active Duty and Reserve Component Service [West Point]
On average, a college degree earns a person $30,000 more in salary each year than a person with a high school degree [Federal Reserve Bank of New York]
This is particularly fascinating considering the fear many academics had prior to Covid that higher ed was going to become one large MOOC. It doesn't appear that this is a concern any longer.