Telecom fixes major network glitch

Telecom says it has now fixed the hardware fault that caused chaos on its network today.

The Mayoral Drive exchange network switch failure interrupted an upgrade to Telecom’s prepay mobile service, which was rendered inaccessible. The upgrade will now be completed tonight.

Telecom has also confirmed that as well as hitting its own website, call centres, email and other services, plus some corporate customers, the fault also affected some residential broadband customers.

Chorus, which has systems housed with Telecom, says affected services are being progressively restored.

“Some of the internal systems are now restored, with other internal systems expected to be restored progressively over the next few hours,” spokesman Ian Bonnar says.

Most of the phone services used by retail service internet service provider (RSP) customers to contact Chorus have been regained.

 

UPDATE 3.30pm: Network operator and wholesaler Chorus says it is currently experiencing some system outages as a result of the major network switch fault that’s hit Telecom’s giant Mayoral Drive exchange.

Earlier today, Telecom detailed how the fault had taken out its internal email, call centres and website, and stopped some corporate customers from web browsing.

Consumers were blocked from services such as MyTelecom and pre-pay mobile top-ups.

Chorus has some IT systems within the Telecom environment, the company says. It is suffering the following issues:

Loss of some internal systems that are used by retail ISPs to order new connections or lodge faults
Loss of some inbound phone lines used by our customers
Chorus.co.nz is not working
Loss of external email
Where possible, Chorus has put in place workarounds to minimise the impact on its customers, spokesman Ian Bonnar says.

Chorus has also put a special process in place to ensure medical or emergency escalations are dealt with urgently and are not impacted by the Telecom issues.

End in sight
Kate Woodruff, a spokeswoman for Telecom’s Gen-i services division, tells NBR the company is going to issue another update around 5pm.

While there is not official time to recovery, NBR understands there is cautious optimism there could be an all-clear at that time. Telecom’s home page is already back on line.

Corporate customers back online
Ms Woodruff says a minority of Gen-i’s corporate customers were hit.

The key problems were loss of web browsing and inability to access Telecom’s corporate helpdesk.

Web browsing was restored with a domain name server (DNS) re-routing workaround, the details of which were published to Telecom’s Facebook page.

 

Most corporate clients were back to full service this morning.

“The greatest impact was on our own business,” Ms Woodruff says.

With Telecom’s own call centre, email system and website down, staff have used Gmail and social media to contact customers, as well as making proactive phone calls by mobile to check in on major clients.

Switch glitch hits Telecom website and call centres, corporate customers

9.30am: Telecom says many of its communication systems are down this morning due to a technical problem.

A spokeswoman told NBR, “We believe it’s a problem with a network switch in Mayoral Drive,” — the large exchange in the Auckland CBD that is operated by Telecom (Telecom retained control of several large metro exchanges post the Chorus spin-off).

The problem is affecting call centres, online services (including Telecom’s website, MyTelecom and Live Chat) and internal Telecom networks.

Corporate clients are also affected by this issue, Telecom says, but it can’t put a number on how many.

There is a workaround for those clients, which is being published by Telecom (a technical challenge, as NBR writes, given the problems have taken down Telecom’s website and call centres).

All other Telecom customers can still call, text and use data, and 111 services are not affected .

There is no estimated time of recover at this stage.

Telecom says this morning’s problem is unrelated to a systems’ upgrade that took place over the weekend.

More at National Business Review