Stanford officials apologize for course enrollment crash, pledge to fix
As a growing number of issues plague an aging campus infrastructure, Stanford officials from various academic and IT departments sought to allay faculty concerns about problematic course registration systems at the first meeting. of the Faculty Senate of the Year on Thursday.
At the same meeting, the senators were also briefed on the University’s response to the increasingly frequent power outages on the main campus. In late June, a three-day power outage caused by a nearby wildfire affected most of the main campus.
Students faced a nearly three-hour delay to register for fall term classes when Axess, the legacy course registration system powered by decades-old software, crashed due to excessive load.
“You are almost certain to have a very stressful and frustrating night in which you don’t sleep well” on class registration night, a sophomore wrote to university administrators, as the ‘ shared Chief Information Officer Steve Gallagher in the Senate.
According to Gallagher, the root cause of the botched course registration was an overload of the student information system inherited from Axess. The overload triggered a bug that was mistakenly introduced when the University revamped Axess’ student-facing web portal component in August.
As students began logging into Axess in the minutes before registration opened at 9:00 p.m. PT, Gallagher said, the bug was triggered and overwhelmed the underlying legacy system. The bug went undetected during load tests that the university’s IT team conducted before unveiling the new system, he added.
While plans to replace the legacy system are being discussed, Gallagher said the current range of systems available are unable to support Stanford’s complex technology infrastructure. In the future, he added, he hopes the entire course management system will be hosted on a modern cloud environment.
Gallagher and the university’s registrar, Johanna Metzgar, apologized for the accident and the resulting stress on the students. The September 1 blackout was not an isolated event, however. Over the years, students have regularly reported system slowdowns during peak enrollment times.
Gallagher and Metzgar said multi-pronged efforts are underway to prevent the issue from happening again in the winter quarter. A team of external consultants will be deployed to fix identified bugs in Axess, improve peak recording load simulation and reconcile load balancing with legacy system bottlenecks, they said.
“These types of issues are completely inconsistent with the levels of service you should expect,” Gallagher said. “The number one priority is to get it right for the upcoming December 1 taping.”
Metzgar also said that technical issues students are having enrolling in classes have caused some students to lose faith in the University’s academic services.
“We are really committed to stabilizing the technical environment,” Metzgar said. “Once that’s done, I think the work of restoring trust between students falls into place.”
Lack of trust, she added, has even led some students to create bots to try to maximize their chances of accessing Axess during peak times. In an effort to rebuild trust, the Registrar highlighted a multi-year project called the ConnectED Program to modernize the enrollment infrastructure by implementing a new course scheduling system and credential auditing system that allows students to track progress in their degrees.
Power failure issues
The University not only had to contend with course enrollment issues, but also increasingly frequent power outages on the main campus due to downed power lines caused by wildfires.
The largest of these outages lasted for three days, beginning June 21. A wildfire near the Edgewood substation affected the campus substation’s main power line, the Jefferson Line. Although power to the main campus was not restored for more than three days, limited emergency power was supplied to the campus via a branch line, the Cooley Landing. According to Jack Cleary, associate vice provost for lands, buildings and real estate, nearly all of the secondary power was used to power Stanford Hospital’s chilled water system.
Cleary said that although the University’s emergency operations centers were well coordinated during the outage, communication with research staff “was unstructured.” Improving this line of communication is an important learning lesson, he added.
There have also been shorter power cuts this year to shut down the main campus. For nearly 8 hours on July 12, the Green Library and surrounding buildings were affected due to a faulty cable. On September 25, another issue affecting the Jefferson Line caused an almost three-hour outage for the entire campus.
Cleary said a new committee, the Faculty Committee on Electrical Resilience in the Research Enterprise, has been created by the provost to advise and recommend measures to reduce the risks to research posed by power outages. Additionally, the University will conduct a study to modernize its electrical infrastructure, Cleary said.