Phuket Live Wire: Anatomy of a land line meltdown

PHUKET: As I continued my pursuit of a decent internet connection in Phuket for the house I’m moving into, a funny thing happened. That’s funny as in strange, not funny as in hilarious.

Depending on where you lived in Thailand, at about 2am on Friday morning, February 15, if you had internet service from TOT your line went down. Again, depending on where you lived, the line came back up about 11am to 2pm. That’s more-or-less 12 hours with absolutely no internet connection, anywhere in Thailand. And, as best I can tell, it happened to every single TOT customer – Phuket, Bangkok, Chiang Mai, Isaan.

What about hot swapping? What about redundant switches? What about alternate backup sites? I mean, this is the internet, for heaven’s sake. It was designed and built to take a nuclear attack and keep on working. Sure, you get occasional outages for an hour if some idiot is asleep at the wheel and the system is set for manual routing (which should never be the case, but it happens). City blocks can get shut off if somebody digs with a backhoe and takes out the trunk line to the block. Servers can go down if they’re taken out by viruses, or if the power goes out and the generator doesn’t work – but any organization worth its salt has alternate sites that can be fired up within minutes.

But an entire country going offline for 12 hours? That’s the kind of thing you expect if Iran’s government pulls the plug, or if Syria suddenly decides its citizens shouldn’t be able to communicate. Indeed, it happened last November in Syria, from November 29 to December 1. In this case, though, the cause doesn’t appear to be politically motivated. It’s simply incompetence. I think.

The fact is that nobody knows why the internet went out. I’ve seen two plausible explanations, neither of which sounds encouraging. The first reason, reported by a local newspaper, was that TOT’s “main server in Bangkok crashed.” The second reason, reported on ThaiVisa, was that two fiber optic cables in southern Thailand were severed concurrently – “one was cut by nibbling squirrels (or possibly mice), the other by someone planting palm oil trees.”

The first explanation doesn’t ring true, at least to me. TOT’s main switch for domestic use, known as AS 9737, has a huge 120 Gbps connection to the TOT Local Network Internet Exchange switch identified as AS 37930 – yes, all of these big switches have numbers that identify them uniquely. But AS 9737 also has a backup connection at 22 Gbps to the CAT exchange AS4652 in Bangrak. CAT didn’t go down, so there should have been backup internet access immediately available to TOT. How a failure in a “server” – most likely a switch, and probably AS 37930 – could’ve taken out all domestic internet baffles me.

As for the second proffered reason – remember, neither of these are official explanations – I just don’t buy it. If all of TOT’s internet traffic, domestic and international, relies on two fiber optic cables in southern Thailand, the people who set up the network topology for the TOT system should hang up their white lab coats. And if the people in charge of watching the lines saw one of them go down, they should’ve had an army out fixing it immediately. I guess it’s possible that 1) the TOT routing system was totally borked and 2) a squirrel chewed through a line and 3) a palm oil tree digger cut a second line, all simultaneously. Somehow, I seriously doubt it.

Here’s what I know for sure: On the Phuket Gazette website, reader “Annoyed” said, “I have not had an internet connection for 24 hours; this morning I was finally able to get back on line. The TOT customer service lady was impatient (obviously stressed out with so many callers having similar problems), her English was slurred, and she was at times not even polite. Having been unable to conduct business because of the TOT problem we really didn’t need to add TOT rudeness.”

Reader “Agogohome” said, “Did I report it? Nope! It’s not worth the phone call or visit, just to receive the usual surprised response, and their worthless promises.”

And there’s the rub.

First, it’s hard for me to accept that the largest internet service provider (ISP) in the country just went dead for more-or-less 12 (or 24) hours. It shouldn’t even be possible.

Second, there’s no notification about system-wide outages. When internet services in other parts of the world go down, there’s a web site people can go to, to see up-to-the-minute information about what caused the problem, and how long it’ll take to fix it.

Third, even the employees don’t know what’s happening. I see that happening over and over again in Phuket: there’s a widespread internet problem, a customer calls the hotline, and they’re told that they’re the first ones to report it. If that doesn’t fly because the customer’s too savvy, the employee on the phone says that (choose your favorite nonsensical excuse) happened, and they have no idea when it’ll be fixed. They may even offer to send out a technician, knowing full well that a hundred other people have reported the same problem.

It’s all about communication. Painfully, our ISPs don’t seem to understand that their customers would appreciate some frank explanations. Rabid customers would also stop calling the support lines incessantly if they could just find some information they can trust.

I’m not picking on TOT – they’re not the only offenders. Since February 11, I’ve had widespread reports of outages on the True DOCSIS 20 Mbps system. (DOCSIS is the technology that relies on coax (similar to a cable TV cable), that’s only available in select parts of the island. True calls it “Ultra.”) While the “fast” DOCSIS 20 Mbps line was down, it looks like the “slow” DOCSIS 10 Mbps line was up and running fine. We’ve had complaints of bad lines at phuketinternetspeed.com, and I’ve seen similar complaints on the ThaiVisa.com forums, but there’s been nothing official.

So what really happened? It’s likely that we’ll never know. I’ve seen exactly zero reports of the TOT outage in the mainstream Thai press – a phenomenal lack of concern.

Perhaps the outage originated as a failure of the ICT Ministry’s website blocking system, which according to the Washington Post allows the government to shut down all internet in Thailand in a few minutes. Cuba, Ethiopia, Lybia, North Korea, Syria, Tunisia, Turkmenistan, Yemen, and a few African countries have similar systems. I just don’t know.

Unless the government announces an open inquiry, the outage will probably just be ignored. I won’t hold my breath.

Moral for you: the Second Law of Landlines in Phuket, as was espoused last week in this column.

Never get one landline internet connection. Get two, on two separate lines, from two different ISPs. That won’t protect you if the whole country goes internet-dark. But it should handle anything short of a catastrophic failure.

Our regular weekly computer clinic roundtables continue every Sunday morning, 10:00 at Sandwich Shoppe Chalong (in English) and 11:00 at Sandwich Shoppe Patong (in German). If you have a Windows problem that needs to be solved, or a question about internet service in Phuket, drop by and ask one of the assembled gurus. It’s always free. Sponsored by the Phuket Gazette and Khun Woody’s Sandwich Shoppes.

Live Wire is Woody Leonhard’s weekly snapshot of all things Internet in Phuket.