• HeyJoe@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      3 days ago

      “An issue with a routine firewall upgrade resulted in some callers from the Northern Territory, South Australia, Western Australia and New South Wales not being able to make emergency calls. Approximately 600 calls were identified to have failed during the outage, and at least four people who attempted to call Triple Zero during the outage have been confirmed to have died.”

      Wow, as someone who works in this exact role of upgrading firewalls i can tell you i have an incredible amount of anxiety when we need to do them because it is a high risk device that has caused issues in the past. Thankfully, our problems would never lead to the deaths of others, and I really can’t imagine how anyone who had a hand in this upgrade feels after hearing that. Plus, with how much attention it has, someone, if not multiple people, are going to lose their jobs. Sadly, after it upgraded, it probably worked fine except for this one or a few policies that controlled this one specific route. So nobody would have been able to tell until complaints rolled in. I dont even work in a huge company, and ours have hundreds of individual policies, so they easily could have a thousand or thousands of policies on 1 firewall. I really wonder what vendor they use.

      • Onno (VK6FLAB)@lemmy.radio
        link
        fedilink
        English
        arrow-up
        2
        ·
        3 days ago

        From memory they use Cisco.

        In 2023, apparently Optus attempted to blame them for the default values on the hardware (IIRC the size of the routing table), which several people pointed out was blatantly absurd. Apparently Optus also didn’t follow up with a welfare check of each of the failed calls in their network. Apparently they had to physically visit each affected router across the entire network. They were fined. I’m not aware of any other penalties.

        Now, in 2025, it appears that Optus didn’t follow their own processes, ignored several early customer reports about emergency calls not working, didn’t inform the communications minister, under reported the impact, didn’t inform customers and didn’t make any announcements until it was fixed.

        In 2023 I couldn’t help but wonder if the staff at Optus had ever heard of testing. Today it seems obvious … to me … that they don’t.

        Disclaimer: Note that whilst I’m an ICT professional, I don’t have any direct knowledge of the internals of these incidents and I’m relying on memory of reports and commentary and Wikipedia, I have also never played with Cisco routers, so YMMV. I also note that I haven’t been an Optus customer for about a decade, and my own experience with their ICT systems as a customer over fifteen years or so has been … let’s call it “suboptimal”.