Transterrestrial Musings

Comments

The surest sign of poor network design is when an outfit had both primary and secondary DNS servers located on the same subnet.

Posted by Mike Puckett at December 29, 2004 07:10 PM

Rand, my experiences are very similar to yours. I'm in the Silicon Valley on a Comcast Cable cable modem. It sucks so bad I'd take a 9600 baud modem connection in a heartbeat.

I've been dealing with tech support for over a month. Recently a senior technician came to my house and **admitted** their network is overloaded. So then, I know everybody on my network (perhaps one thousand subscribers) have the same exact problem.

Prior to the admission, Comcast gave standard excuses like "the problem is in your home network" or "your computer has a virus." To which I have no doubt 99% of Comcast's customers don't question in the least and assume it to be true.

Keep in mind this is the Silicon Valley, where the population on average is more geeky than anywhere else.

My blog contains my cable modem saga.

Posted by Eric at December 29, 2004 10:32 PM

Even with Bellsouth's business DSL service, these sorts of problems exist, along with seriously over-committed lines. At my previous employer, they pay about 3 times the average home DSL cost, but the network slows to a crawl after about 3 PM most days (I suppose when the teenagers get home from school).

For me, every dealing with their technical department has been as frustrating as yours. When I first subscribed to DSL in Broward County, FL, I used Telocity (which of course was running on Bellsouth's physical network); it was mostly OK, but on a couple occasions I forgot to pay (cough) and had my service turned off. I would then have to jump through multiple hoops with Bellsouth, because apparently the service would be switched off right at Bellsouth's network, and the Telocity admins did not have the authority to switch it back on. The first time this happened, it literally took me a full day of telephone calls, working my way up the tech food chain, until finally I found someone who turned my service on in 30 seconds.

I now use Speakeasy, since Telocity closed up, and have had nary a problem in any of these respects. They have managed to set up a system where they have the authority to do everything at their end, so Bellsouth personnel need never get involved. And, they don't overcommit their network. Even though Speakeasy costs a little more, it was well worth it for me. Average tech support call wait is about 3 minutes, and their techs actually seem to know their business. And, they don't tell you that Linux or FreeBSD operating systems are "not supported". (I used to have to lie and pretend I was using a Mac to get help).

If you are stuck with Bellsouth, though, I recommend you point your network setup at an external DNS server. There are plenty out there.

Posted by rycamor at December 29, 2004 10:53 PM

Many of the your front line call center reps read from a script. I work for a company where we have several large international and domestic call centers. Quite often to get anywhere with your issue you have to get through the first line of support and try to get to someone that can do something. It is posible that the admins don't have clue that problems is occurring with their system. Although I'm not a sysadmin, I am a systems programmer, and we have had many times that a misconfiguration caused an application to break. And we just didn't know, after all we didn't use the application. For instance, I suspect the bell south employees don't use the DNS servers that provide service for their DSL customers. The chore is getting to the person who knows who to call about the situation, and get it resolved.

Hopefully you can get your DSL situation improved. My DSL currently is very stable. (I host my web server on it) But I fought terrible battles in the past in the early days of DSL.

Posted by Mark Smith at December 29, 2004 11:02 PM

Rand,

I have BellSouth DSL, and have been having exactly the same problem for about 5 weeks now. So you havea a name / number I can call to help turn up the heat?

Posted by Steve at December 30, 2004 05:43 AM

Rand,

I have BellSouth DSL, and have been having exactly the same problem for about 5 weeks now. So you havea a name / number I can call to help turn up the heat?

Posted by Steve at December 30, 2004 05:43 AM

Quote: "TCP/IP (is there anything more infuriating than dealing with a supposed tech-support person who is clearly clueless, and much less knowledgable on the subject than you?)"

I'm a 2nd level helpdesk technician for a well known semi conductor manufacturer in Texas. Here what you do:

1. Call once and just for grins and giggles go through the scripted walk through with the 1st line tech support person. Don't let the call go for more than 30 minutes. Most likely they will be trying to get you off the phone in 15 minutes anyways as their call metrics will be setup to encourage analysts to answer as many phone calls as possible. At the conclusion of the call Tell the technician that you want them to research/escalate the issue and that you will call back later. You must, I say MUST, get a ticket number. Every helpdesk uses a call tracking software of some type where they can type in the details of the call. That number is your ticket to escalation. Go ahead and hang up with that technician and go to lunch.

2. Feel full? Feel relaxed and ready to sit on the phone for a possibly a few hours? Good. Call back introduce yourself and tell the technician that you already have an open ticket and give them the ticket number. Whether the ticket is actually open or not doesn't matter it just servers as living proof that you've already jumped through all the circus hoops like a good caller. They will start to go through their troubleshooting flowchart again anyways. Tell them that you have already gone through these steps as can be seen from my previous ticket and that you need to have your issue escalated, be persistent. For me its easier can I always say that I work on a helpdesk and know exactly who I need to talk to. I usually throw out big helpdesk words like, 'escalate', 'action request', or 'second level queue'. They will then tell you to hold and will put you in their 2nd level support queue. Be prepared to wait for several minutes as this is the queue to techs that actually know what they are doing and are likely to be on calls lasting an hour or longer. These will also be the techs that have the authority to escalate a ticket on up to 3rd level administrative support or can schedule on-site service.

3. Be polite, patient, and courteous to get the best possible service. I understand this can be difficult in this day and age of talking to a service rep in some overseas location in India. This will only serve to fluster the service representative and diminish their analytical ability. I've taken helpdesk courses that have drill sergeant type instructors that do roleplaying calls as pretend customers screaming in your ear while you have to lookup knowledge base articles and articulate calm and clear instructions. So, we expect it but it certainly doesn't help with our jobs or your issues. Fortunately, we have mute buttons on our phones that let us make fun of someone whenever they start yelling or are being stupid with us.

Posted by Josh "Hefty" Reiter at December 30, 2004 07:39 AM

I don't know if this will help you or not, Rand, but I was having problems exactly like this with charter.net in South Carolina this summer after I installed a wireless router so I could share my cable modem. The problem went completely away after I modified the connections on the PCs by enering the DNS server IPs instead of leaving the "Obtain DNS Server address automatically" radio button checked.

Posted by Rick C at December 30, 2004 08:16 AM

No, that doesn't help, Rick. That's been my setup all along--I don't run DHCP on my home network, and all the machines (including my Debian router/firewall) already have the DNS servers hardwired. There is no doubt at all that Bellsouth's servers are flaking out (probably due to overload), because since switching over to 4.2.2.2, things have been much improved. I just think that they haven't bothered to increase server capacity because they haven't gotten enough explicit complaints about it to perceive it as a customer relations issue.

Posted by Rand Simberg at December 30, 2004 08:22 AM

I have Time Warner digital cable. This past weekend, the system routinely went down in the evening and did not return until about 8AM in the morning. I called everyday.

The first day, I just hoped they would reset the modem, which they do remotely and sadly occurs about every other month. When it didn't work, they noted that there was a known outage in my area.

The second day, no known outage. We'll look at the problem and get back to you. They did at 3pm, about 7 hours after service returned. No mention of an outage.

The third day: By now, I'm trying to coach them into admitting that they are servicing the system, and intend for it to be down during that time (it makes some sense to do major work during the holiday's when customers are distracted... and don't represent a business...). Nope, they'll send a tech out to my house 3 days later (this Friday) to look at my problem.

Since the third day, it seems whatever service work has been completed, and no down time has occurred. So what do I say to the tech that arrives tomorrow? I'm thinking, "You know, if your customer service department actually admitted reality, they could have saved you a trip."

Now, I have been considering there new VOIP phone service. But since they do not call it that. No one at tech services seems to know the term VOIP. And they also their cable service went down for three days without explanation... I think I'll forget that idea. That's significant, because the switch was primarily to unify my communication services to one bill. I guess I'll switch to Sprint as a provider for those services.

I have an even longer story on SBC DSL services. It always fun to have a collection agent call about $21 you owe them, when they haven't paid you the $250 they owe you (seriously, it happened).

Posted by Leland at December 30, 2004 09:24 AM

Dude, set up your own caching only DNS and use that. And avoid theirs alltogether. You have a linux box... and I think Windows can do this... then you'll never have to depend on them for DNS again... pless there'll be less to change when you change providers. (Note that is 'WHEN').

Posted by at December 30, 2004 01:59 PM

When someone calls and reports an issue a ticket is created, every call gets a ticket. Even though the frontline techs maybe handing out PR propaganda answers. There are network engineers analyzing the call reports based upon categorized datagram and can see trends of increased call volume due to poor performance or inadequate equipment. Also through a variety of network diagnostic tools alarms in the network operations center will notify them of performance issues along the infrastructure. Its most likely a upper management decision that needs to give the go ahead to upgrade the network to meet demand. Its also probably related to a pre planned technology road map that dictates when their customer base reaches X number of subscribers then that will cover the cost of service improvement. So, its assumed that the existing customer base will put up with inadequate service to cover the cost of the improved service of future customers.

Posted by Josh "Hefty" Reiter at December 30, 2004 02:16 PM

Dude, set up your own caching only DNS and use that.

How much disk space would that require? Could I do it on a machine other than my firewall (e.g., my Linux file server)?

Posted by Rand Simberg at December 30, 2004 04:54 PM

Alternative solution is to run your own DNS caching server. I have one setup in one of my own machines as a fall back server in case my cable company's DNS is hosed.

Posted by BigFire at December 31, 2004 06:10 AM

Wow, I feel for you guys in the USA. I live in Japan and have 26MBPS DSL (called ASDL here) for $30 per month including VOIP calls to the USA for about 7 cents for 3 minutes. If you have optical fiber you can get 100 MBPS to 1GBPS. No problems whatsoever. In 2 years I've never needed to call tech support. Had a little problem with the motherboard software in a new computer I had built last year, and got the problem solved by the guy who answered the phone - he knew his stuff. I'm wondering if I really want to move back to America after 20 years after hearing all the horror stories!

Posted by Kurt at December 31, 2004 06:13 AM

Haven't hosted a debian caching DNS, but have done it on Win2K. Low overhead process, with low storage requirements.

Here's something that might do the job for you -
"dproxy is a small caching name server (DNS) for use between Local Area Networks and the internet.

"It features:

* small size: less than 3000 lines of code, ~25KB binary
* modest resource requirements (runs great on an old 486)
* discovers and serves names from /etc/hosts and dhcp lease files
* human readable/editable ASCII text cache file
* drops root priveledges early after startup"

More info - http://dproxy.sourceforge.net/

Appears to be quite a bit simpler than BIND. I would host it on the firewall, but that isn't required. I like the fact that it runs as USER.

Posted by Doug Zimmerman at December 31, 2004 06:13 AM

Re: Rand

In my Linux setup, the entire DNS setup is about 350KB, binaries included (executable binaries should already be present in most Linux package).

Posted by BigFire at December 31, 2004 06:15 AM

I am a Bellsouth dsl user; I've been noticing the (intermittently) hosed up domain name resolution for weeks. I finally called tech support, and the 1st line guy wanted me to do the usual: shut everything down, restart. I politely declined, and he eventually promised to have someone call me back.

The guy who called back clearly knew about the problem. He admitted that Bellsouth was indeed have DNS problems, though he did not give me any details.

He did suggest that I explicitly specify the ip addresses of the Bellsouth domain servers rather than obtain them automatically. This seems to have helped. The addresses were: 205.152.64.20 and 205.152.0.5

Posted by edpi at December 31, 2004 06:31 AM

I began noticing this a few weeks ago on my home net, especially on my Linux workstation right after I installed Firefox 1.0. Since I hadn't experienced it so much with the pre-release versions, and I wasn't using other internet apps so much, I thought it was FF. In fact, on their support forums, there was a long thread and an actual configuration tweak to make the "problem" better.

That didn't help. Then I noticed Thunderbird, gFtp and other net apps were experiencing the same issues. Once the lookup was complete, the load of the site or host was very fast (I have Bellsouth's 3 MB DSL service). I even tested my connection on broadbandreports.com, and found consistent 2.3 to 2.7 MB downloads, so I know it's not a DSL bandwidth issue.

I'm just glad to know it isn't just me.

I have also tried the hardwired DNS address fix, but it eventually breaks. The suggestions to set up a caching DNS server on your local net is a good one, though...I used to have such a setup when I used a Linux box as a firewall (prior to the router appliance days). I'm in the process of setting it up now on my server.

Posted by Joe at December 31, 2004 06:47 AM

Kurt:
Just to let you know that I live in Australia, and that I hate you.
But seriously folks... we pay far more for a lot less here. OTOH it's reliable.
e.g. I'm on ADSL, cost is $25/month US for the ISP (limit 1.3 Gb/month), plus another $30 US a month to the cable company. Total including 20 channels of TV and phone service is about $80 US/month.
Still beats the hell out of the system 5 years ago, or even 2.

Posted by Alan E Brain at December 31, 2004 06:49 AM

I've had DNS problems with Bellsouth (FastAccess in NE Florida) from the first time I signed up (3 years ago). I even had to call and get them to update their DNS servers as they weren't accepting a DNS change that had taken place 5 days prior and ever other DNS server I used showed it to be working.

Posted by Andrew Connell at December 31, 2004 06:54 AM

I have had the same problem, Their tech told me they were making changes to the system and a time frame when it would be fixed.They know of the problem , question is will they or can they fix it.Loss of customers will speed up their efforts.

Posted by L P Malmquist at December 31, 2004 07:40 AM

Rand - Is the problem continuing? Or have you gotten resolution?

I'm with BellSouth.net, and I can take this issue directly to the DNS systems owner; he sits about 30 feet from me.

Posted by Matthew at December 31, 2004 07:41 AM

Or have you gotten resolution?

I'll assume that wasn't an intentional pun...

The problem is continuing in the sense that I'm still getting resolution delays. It's now tolerable because I'm no longer relying on Bellsouth's DNS--I'm using a public server (4.2.2.2, at the suggestion of the tech I talked to the other day), which only gives a slight delay (probably because it's got a heavy load). I suspect that if I removed it from my nameserver list in resolv.conf on my firewall/gateway, I'd go back to frequent timeouts.

In the absence of some reason to think that Bellsouth has done something about it, I'm not in a mood to do continued experiments, but if you're aware of some change in the status, or reason to think that you've both found the problem and resolved it, it please let me (and Glenn) know. If you get it fixed, I'll put up a new post to that effect.

I should note that the tech to whom I spoke to was quite adamant that there was nothing wrong at their end, as far as he knew. He was chastened only after we demonstrated that it worked with the public server, and didn't with Bellsouth's.

Thanks for the offer.

Posted by Rand Simberg at December 31, 2004 08:07 AM

I'm not sure it's relevant, but I'm on BellSouth dial-up and have been having the same DNS problems for about the same period of time (slightly more than a month). I went the round-robin with Indian tech support until one night my repeated calls got me kicked to a higher-level tech support back in America.

He said that if you call multiple times (three seemed to be the magic number, though he didn't give a time frame), you get automatically sent to a second-level tech. But this guy seemed self-confidently sure that the problem was on my computer. He gave me a text string to add to "extra settings" on dial-up Properties that did nothing.

I'm glad to have run across this discussion, as I was about to "upgrade" to BellSouth DSL!

Posted by mike hollihan at December 31, 2004 08:12 AM

This whole thread is terrific. Thank you very much.

My BellSouth DSL was troublefree and excellent up to about a month ago. I then got exactly the symptom Rand describes--webpages either very slow to load or not loading at all. But my e-mail program and my Agent newsgroup reader continued to work about 95% of the time. I suspected a DNS problem, but didn't know for sure.

The good news is that in the last week or so, my service returned to its usual excellent level. But the bad news is, I guess, when the holidays are over the problem could well reappear.

If Rand or someone else can post a solution that does not involve BellSouth--changing the DNS setup as Rand suggests?--and that applies to BellSouth's DSL modem, and that can be followed by people who are only semi-literate in tech stuff, I, for one, would be most grateful.

Rand and the other posters, thanks again for a great public service.

Posted by Craig Newmark at December 31, 2004 08:15 AM

Rand & Matthew:

I am in Louisiana (Baton Rouge), and have been having the same problems since late October / early November with our BellSouth DSL - glacial-speed page loads, constant timeouts, and failure to load pages (which will, about 75% of the time, correctly load when the 'refresh' button is hit). We have run anti-virus, redone our routers, etc. Problem persists.

Matthew, I sure would appreciate it if you'd let the Owner know. Before October/November, we've not had any real issues with BellSouth.

.....CLIFF

Posted by cliff at December 31, 2004 08:18 AM

I find this all very odd. Having been responsible for monitoring name resolution performance at several companies, I can tell you that it is *very* simple for BellSouth to test.

I suspect that they already know of this problem. As to why they haven't fixed it, one can only speculate.

Posted by Doug Zimmerman at December 31, 2004 08:31 AM

I have had intermittant probems with Bellsouth, specifically, slow loading email pages. Tech support was useless but an email to the webmaster seems to have worked. E-mail: webmaster@bellsouth.com (yes, com).

Posted by Charles Kanige at December 31, 2004 08:37 AM

Lots of complaints about BellSouth DNS performance over on DSLReports.com.

BellSouth NOC told customer problems were fixed.

Posted by Doug Zimmerman at December 31, 2004 09:09 AM

I'm on BellSouth DSL and I've been seeing it, too. It was really bad just a couple of days ago -- pretty much unusable. It seems to be that it "comes and goes". Right now everything seems to be working Ok. Tomorrow, who knows?

Posted by Gene Britton at December 31, 2004 09:18 AM

This is very interesting. I have Bellsouth DSL and have been experiencing DNS timeouts too. My problems started in mid-November. I finally fixed the problem myself by Googling for Bellsouth's name servers, doing a DNS lookup on those servers, and then manually entering every nameserver IP address I found.

My Name Server Addr. list now looks like this:

205.152.244.252
205.152.0.5
205.152.32.20
205.152.0.20
205.152.16.20

Name resolution still seems slow to me, but I no longer get timeouts.

I've not spoken to Bellsouth tech support. Several previous encounters have convinced me that contacting them should be a method of last resort.

Posted by Bob at December 31, 2004 09:20 AM

I can't count the number of times I've been frustrated by flaky DNS. The suggestion of setting up your own DNS is a good one, but BIND is notoriously tricky to configure. You might be better off with djbdns.

Posted by David Gillies at December 31, 2004 09:29 AM

Here's an update to my earlier post.

I suggested to Rand that he think of setting up an internal caching/forwarding name server.

I decided to reset mine up to see if I had an improvement. I have a small server on the internal network running Slackware Linux. I installed the Bind/named package, set up DNS and pointed my workstation to it. I have named using two bellsouth.net nameservers for forwarding (205.152.244.252 and 205.152.37.254), but internally caching.

In some brief testing, the results are remarkable. I've been experiencing up to 20 second delays in lookups before a page would load, but that seems to have pretty much been eradicated.

Naturally, I can't tell if this is a fix or a placebo, but it sure seems to have helped. I can explain how to do this on a Linux system generically, so if anyone is interested, mail me.

Posted by Joe at December 31, 2004 01:32 PM

I have been experiencing the same problems as you note in your post. I called support and was told it must be my computer as bellsouth had not received complaints about this. They had better get it fixed or I am switching to another provider.

Posted by Michael Bryant, Ft Lauderdale, Fl at January 1, 2005 07:19 AM

Update on my stuff (if anyone interested): Supposedly my signal strength was too strong and made stronger by the cooler weather. Not really strong on RF, so I'll accept the excuse for now. However, it will take another tech on another trip to actually fix the problem.

Posted by Leland at January 1, 2005 12:10 PM

The company I work for experienced something similar with PacBell service. One of the theories I found is that a recent switch of some top level name servers to support IPV6 caused problems for a number of networks. I guess the severity of the problem depended on what hardware/software they were running.

Since we are most a Macintosh using company, we (and a lot of other people) assumed it was an Apple problem, but the IPV6 switch seemed to fit the facts the best.

Not that that suggests anything useful you can do.

Posted by AndrewS at January 4, 2005 10:18 AM

I have the same dns issue with Bellsouth , I started using the public dns servers of opennic and my problems stopped temporarily , recently the entire dsl connection has started failing and BS is telling me now I'm using too much bandwidth with Xbox 360 accessing Xbox Live. Whew! Those outsourced techs will say anything !

Posted by Dee at June 30, 2006 09:56 AM

Bellsouth DNS Bullshit

First, if you haven't noticed through your conversations with bellsouth support - they really couldn't give a shit. it's sad, but it's true. the indian folks get paid by how much time they keep you on the phone and they couldn't breath unless doing so is in the script. it was said earlier that the call centers incentives to quickly process issues is contrary to how they get paid. But, don't feel slighted, Linksys/Cisco is the same thing.

Secondly, did anyone notice that the DNS problems began about the same time they got to work for the NSA et al. Since I'm writing this in October 2006 and this thread started in December 2004, I assume they've had plenty of time and complaints to have long ago solved this issue if they had any intention of doing so.

Just so everyone knows, the DNS problem is still there. I live in southeast GA, and there is a minimum five full second (5.0s) responses to DNS queries. Contrast that with my Comcast DNS response times of (0.05s). So my 256KB/256KB Comcast connection is 100x faster at responding to DNS queries than my 6MB/384KB Bellsouth connection.

This thread is two years old, and this problem persists. Maybe everything is actually working but the NSA has to approve your DNS request first :) There is no valid technical reason for this level of a problem for this length of time. And it doesn't matter what time of day it is, so the DNS workload defense doesn't hold up.

PS: Bellsouth did eventually deny participation, but as far as I know for certain, there were only two companies that actually refused the unconstitutional demands and bellsouth wasn't one of them, but Google was !! Too bad google won't just give us all free DNS, imagine the statistics they could derive from that. Oh well, PEACE netizens.

Posted by James at October 28, 2006 01:33 PM

Well since everyone is complaining about their bellsouth experiences here, I'll share mine. So today, November 20, 2006, around 7:00, I went to my Tae Kwon Do dojo and my master asks says to me that he had just gotten his box refromated. but he says that his internet was exceptionally slow today. I looked and sure enough, google.com took like 5mins to load. I couldn't figure out what the problem was... until I got home.

Once I got home, a friend of mine started freaking out saying that her ie wasn't working and that all the BELLSOUTH friends she talked to said the same thing. I asked her some questions and relized it was the same problem as my masters, i got interested and started asking some others and test out my own ie. Mine too took like 3mins to load a single page. Myspace is a joke. Some pages sometimes loads up and then sometimes shows a "page cannot be shown" error. I went to pcpitstop to see if it was downloading issues. My downloading speed was just the same as before: ~1200 kbps, the stupid 1.5mb "ultra speed" that I subscribed to which they wouldn't even update for me even after all these years as their "valued customer". anyways. Since the downloading was the same, I figure it was the problem with the dns. So I looked it up on google, found this page, and here I am writing this 2 page essay and wasting time I could use for my school homework.

I've had many problems with bellsouth before and was meaning to transfer to comcast... but im just too lazy to pick up that damn phone and make the stupid call. When I get fustrated enough, I call bellsouth tech support to tell them to do something about it even though I know they would be no help. All I get from them is "Have you checked if the green like labeled 'internet' on your black modem you got when you purchased your fast access dsl service is on and solid?" or "Have you tried pushing the little gray reset button on the back of your modem?" I mean HELLO! I think I can manage to press a freaking button or manually change it factory defaults and im not blind that I can't see a stupid green like on my modem! The most help I've ever got from them was, "Ok hold on for only about 1 hour while you try to get our higher tech support to assist you because I really have NO idea what is wrong with your internet service." even those "higher-level techies" weren't any help. I ended up finding the problem and fixing it on my own.

The sucky thing is, this time, its a DNS problem and there is really nothing I can do about it but just to complain to them to fix it or something, which they would totally ignore.

Posted by Jason YIm at November 20, 2006 07:53 PM

The sucky thing is, this time, its a DNS problem and there is really nothing I can do about it but just to complain to them to fix it or something, which they would totally ignore.

Posted by Jason YIm at November 20, 2006 07:55 PM

Post a comment