Rogers outage impacted emergency services and exposed gaps
From the start of the outage in the early morning of July 8, frontline and behind-the-scenes crews from the Southlake and Central York Fire Departments worked hard to reconnect through various means.
Just a few weeks ago, staff and leaders at Southlake Regional Health Center held a mock Code Grey, an exercise to walk through the critical steps needed to keep things running smoothly in the event of a major infrastructure failure. .
The practice came in handy at the hospital on Friday when, along with most of the rest of the country on the Rogers telecom network, they were left offline during a major network outage.
On July 8, customers across the country were largely disconnected following what Rogers described as a “network system outage following a maintenance update.”
The blackout not only left people unable to use their phones, televisions and the internet, but also without a way to use their bank cards at businesses, access their cash at ATMs or call emergency services.
“We know how much our customers rely on our networks and I sincerely apologize,” Rogers Communications President and CEO Tony Staffieri said in a statement. “We are particularly troubled by the fact that some customers have been unable to reach emergency services and we are treating the issue as an urgent priority.”
It’s an urgent priority across the country, and York Region is no exception.
From the start of the outage early Friday morning, frontline and behind-the-scenes crews from the Southlake and Central York Fire Departments worked hard to reconnect through various means.
“Although we have some electronic mapping in the truck systems, we got around all the issues we had and were able to provide all the services requested that day,” says Central York Fire Chief Ian Laing. Services. “It’s a bit of a challenge. Any delay in notifying an emergency usually makes the situation worse.
Impacts were also felt at York Paramedic Services.
“York Region Paramedic Services proactively opened the Incident Management Support Center on the morning of Friday, July 8, 2022, to provide additional support and ensure continuity of timely and critical care. to residents,” Linda Gonsalves, general manager, paramedical and senior services, said in an email. “Essential personnel reported to York Region Paramedic Services Headquarters to ensure service delivery continued during the outage. Radio ambulance operations and York Region’s 911 infrastructure were not affected. York Region Paramedic Services experienced outages on a few administrative systems, but these were easily resolved using our backup systems.
“York Region relies on multiple internet and cellular providers. Many regional facilities also rely on connectivity provided by YorkNet fiber and as such, services including landline telephone and internet communications have been relatively unaffected. York Region Paramedic Services is not aware of any service delays and has not received any service complaints to date as of Friday, July 8, 2022.”
Responding was a bit more complex for Southlake, as not only were they temporarily unable to log on to important outside information, but shift changes were also left in abeyance as staff not already on site were not available. was unable to call.
“We knew something was happening quite early in the morning when things weren’t necessarily connecting as expected,” says Robert Bull, vice president of facilities, IT and chief financial officer of Southlake Regional Health Center. “Like all Code Grey, the priority is to determine what works normally and what could be impacted. What are all these basic systems? We were able to establish fairly quickly [connection] to the main health information system that we use. Things inside the building generally worked as expected. Our internal networks were all working. It was when we needed things to connect to the outside world that it was intermittent depending on what the system was.
“With a Code Grey, we were able to identify early on the major systems that weren’t working and understand the implications of those…. We asked people to return to downtime procedures for systems that are not working. There were well-established procedures that people had in case these things happened…”
One of the most difficult elements to disentangle was the level of staffing. Rogers network staff who were sick or otherwise unable to report were unable to call the staffing office and in turn reaching staff on the same network to fill was also a challenge.
“The morning shift change is usually at 7:30 a.m., so we knew what the implications were and it was too early for major impacts by then,” says Bull. “The lack of information we had was later in the day and was going to be for the night shift, which is usually 7:30 p.m. We didn’t have great visibility on how many people had tried to call someone sick or report an absence and could not reach us.
“When we were trying to call, we weren’t having a good success rate connecting to people. We had plans in place and our command center is [planned so] people designated to certain roles…and we had planned a 24-hour command center throughout the weekend. One of the things we needed to do before evening as we headed into the 7:30 shift change was to meet with all of our leaders, managers and directors to go back to all of their units, confirm that they were fit for the night, that we as rulers [went to] staff to give them heads up as we obviously had to make sure we had enough staff throughout the night.
Eventually, Southlake was able to find alternatives to Rogers that gave them “flexibility in terms of being able to communicate with each other for people outside of the building and our offsite locations.”
“We’ll probably see a little diversification of those communication channels so that we can adapt if something like this were to happen again,” Bull says.
Brock Weir is a federally funded Local Journalism Initiative reporter at The Auroran