John Howard was still the prime minister, the first iPhone was months away, and Facebook had only just become available to the world. When Telstra’s mobile network fell over on Wednesday morning, it was because a software defect was telling the network it was back in November 2006.

Telstra’s chief financial officer, Michael Ackland, initially explained that the software defect hit the company’s GPS node, which tells the organisation’s systems what the most accurate time is to the nanosecond.

The reset changed the time and synchronisation, which was then passed on to the rest of the Telstra network. On Thursday, Ackland confirmed that the update pushed out to the rest of the network dialled back the clock internally to November 2006.

Why did this cause the outage – and how were people affected?

How a time-telling error brought down network

Unlike a grandfather clock, or an old-school alarm clock, people generally don’t need to adjust their smartphones when daylight savings kicks in.

That is in part because of the network time protocol (NTP). A system can send an NTP request to receive accurate time information from a server dedicated to telling the time.

Phones then use this accurate time to convert to local time based on location.

On Wednesday, the NTP within Telstra’s GPS node was suddenly telling the rest of the network it was nearly 20 years ago.

Networks rely on accurate time information for synchronising and authenticating, Ackland explained, “so that one part of the network knows what the other part is doing and the time is an important part of that”.

“Otherwise you could have messages or instructions that are from a different time period being authenticated on the network,” he said.

Khurram Shahzad, a senior analyst at the research and advisory firm Gartner, said asynchronous time could have a wide range of effects on mobile networks.

“When the root clock fails, the entire digital domino chain falls within minutes,” he said.

Telstra has yet to detail how the dominoes fell, but Shahzad said network nodes from cell towers authenticate their identity with the central data hub using security certificates that have strict validity windows that are checked against the current time to ensure they haven’t expired or been compromised.

“If Telstra’s central time servers began broadcasting the wrong time – or stopped broadcasting entirely – core network switches suddenly saw valid security certificates as expired or invalid,” Shahzad said.

“The system assumed it was under a security threat and automatically terminated the connections, locking devices out of the network and throwing phones into ‘SOS only’ mode.”

He said 4G and 5G also need time measures to maximise speeds.

Ackland rejected comparisons to the Y2K bug, caused by systems not being able to deal with the date format as the year clicked over to 2000, instead describing it as a software glitch. He said Telstra would do a root cause analysis, working with vendors, partners and external support to work out what exactly happened.

Triple zero

While Telstra’s network was down, some customers who attempted to dial triple zero could not be connected, despite failsafe systems that require a customer to be connected to emergency services through one of the two other mobile network providers in the event of a network failure.

Telstra has yet to explain why this handover did not take place, but on Thursday said it had resolved the issue.

A second issue identified on Wednesday night, which Ackland said the company believed was driven by the same time-keeping issue, caused some customers to receive an error message when attempting to call other numbers or triple zero. This had been resolved by Thursday morning.

Telstra said as of 1pm AEST on Thursday, there had been 639 welfare checks completed on people who had made unsuccessful calls to triple zero. Of those, 230 advised they didn’t need assistance. The company said 402 cases required a follow-up call, with 170 of those referred to police for a welfare check, and seven referred to emergency service organisations.

Why did some trains stay offline longer?

The Australia Rail Track Corporation was forced to suspend services on Wednesday after the Telstra outage disrupted the national train communications system that allows drivers to communicate directly with the network control centre over Telstra’s 4G.

ARTC operates rail services across the country, including the Ghan between Adelaide and Darwin, XPT services between Sydney and Canberra, V/Line in Victoria, services in Queensland, and the Indian Pacific.

V/Line indicated Telstra’s 4G network had interfered with the satellite backup system used by train drivers to communicate with the network control centre.

“Overnight, Telstra were able to undertake some repair programming to stabilise the network, and that involved making sure that the 4G train radio system was talking correctly and integrated correctly with the satellite phones, which are our backup system in the train,” the chief executive of V/Line, William Tieppo, said on Thursday morning.

Ackland said Telstra was working with all affected businesses.

“While the broader outage was resolved yesterday, we continue to work closely with a small number of enterprise customers on residual flow-on impacts in their services,” he said.

“We will keep working with our customers until every remaining issue is resolved and as they complete all restoration and verification processes that they need to complete.”

Given one glitch had taken out the mobile network, Ackland was asked if that showed Telstra’s network was fragile.

Ackland said it was not. “I would say it is a complex system. I think it is robust,” he said.

“There is a lot of redundancy, but this was an unfortunate incident.”