sudo with askpass

Normally I connect to a remote server with ssh then at the command prompt execute a command using sudo if I require the additional privileges. Just today, I had occasion to run sudo over ssh (something broken with the login prompt). Without having preconfigured anything, I got this to work:

ssh (server) SUDO_ASKPASS=/usr/lib/ssh/ssh-askpass sudo -A (command)

In my case, I had an appropriate command at ‘/usr/lib/ssh/ssh-askpass’. Your system might be different.

Telstra Bigpond pays up for router to make up for crappy Netgear cable modem!

After a frustrating series of unsuccessful attempts to get Telstra Bigpond to replace their crappy Netgear Cable modem with something that would actually work, and resolving the issue by supplying my own router, I made a formal complaint to Telstra documenting all of the trouble shooting that I’ve done, the dates that I had rang technical support, etc, etc.

The crux of my complaint was this:

I have provided enough technical analysis of the problem to unequivocally identify that the Netgear CGD24N cable modem is incapable of performing the required functionality for the “Telstra Complete Home Saver 200GB Bundle” service.

I have given Telstra Bigpond a fair opportunity to resolve the issue and it has not been resolved.

I have taken further steps and resolved the issue myself by disabling a significant portion of the Netgear CGD24N’s functionality and installing an additional wireless router (Linksys model E2000) at my own expense.

I therefore make the following demands:

1. I demand that the cost of this Linksys E2000 router ($115.00) be reimbursed to me as a credit to my Bigpond account. (Original invoice attached.)
2. I demand acknowledgement from Telstra that no suitable customer premises equipment has been delivered and subsequently the conditions for the contract for service have not been met. I will then continue to use Bigpond as my Internet service provider just as if a contract period had expired naturally, subject to no cancellation clauses, and continuing on a rolling monthly basis.
3. I demand a written response to this complaint with 14 days. 

And today, I received a phone call from their complaints resolution people, offering me a refund for the $115.00 which I happily accepted and a guarantee that I’ll be released from any contract-period cancellation charges should I cancel my service. The guy was polite and gave me a reference number, his Telstra employee number and the phone number for his office (call centre I suppose). I’ll take what I can get.

A refund of $115.00 certainly costs Telstra less than a complaint escalated to the TIO, is less than what they’ve already spent on technical support that couldn’t deal with the issue, and is worth way less than the amount of time I’ve spent on this issue, but I’ll take it as a win.

I just hope they’ve really learnt something from this and can give Netgear a boot up the proverbial for making such second rate equipment (and themselves for supplying it).

Happy days.

Telstra Bigpond Cable crappy Netgear Router part 7

In my last post, I wrote that after too many unsuccessful calls to Bigpond technical support to replace my crappy Netgear CGD24N cable modem, I spoke with someone in sales who said they would send me a replacement modem and that it would be a Thomson cable modem. Well the modem arrived in the mail today, and guess what? They sent me another Netgear CGD24N. Fools.

Enough is enough. I went to my local computer store and bought a Linksys wireless router, disabled NAT on the CGD24N (effectively disabling all of its router functions and using it just as a modem), and connected the Linksys router. Boom. Fast, snappy perfect internet.

It just goes to show that the router part of the CGD24N is total rubbish.

I’ve been on the phone to Telstra Bigpond about a dozen times now, have had a field technician check the line 3 times (it’s fine, there’s nothing wrong with the line) and replace the modem with the same model 3 times (it’s still rubbish) and have no solution. Except for me to spend more money on another piece of gear to do the job of the router that Telstra provided.

Now that I have a perfectly working connection, my motivation to speak to Telstra call centres that can’t do anything about this problem is massively reduced.

However, I think other potential customers should know about these issues. And likewise, so does the TIO.

Telstra Bigpond Cable crappy Netgear Router part 6

So I got home from work and my wife was downloading something onto her computer, I plugged in my laptop and boom… Web pages timing out. Massive SYN_SENT in my netstat output. No problem connecting to the router, I could see that the router had a good signal strength, and the line definitely hadn’t dropped out because the other computer was downloading. I could connect to my nas and do a backup there.

The only piece of the network that wasn’t working correctly was the Netgear router, with the symptoms described by others when the NAT (Network Address Translation) table is full and the router simply cannot handle any more connections.

So, on the phone to Telstra Bigpond technical support again to see what can be done about it. Nothing, it turns out, because all they can do is send out a Field Technician to check the line (which has happened 3 times now), or replace the modem with the same model. Not good enough.

They referred me to sales. Odd, but at least I’m going somewhere else because the people on the tech support line clearly don’t have any capability to deal with my issue. Logically this makes some sense – the sales team have provisioned substandard equipment, what can support do about it?

So I got on the phone to sales to continue complaining about this. After explaining the situation yet again – I need to make a recording to play down the line to them – they decided to send me a replacement modem. I insisted that they needed to provide a “better” modem, because another Netgear would simply result in the same problems (we’ve already been down that path).

The girl tells me that she’s going to send me a Thomson modem. I’d only heard of Thomson making ADSL modems, and she *couldn’t* tell me the model number which is quite concerning. I’ve since looked up Thomson (bloody hard to find anything on Technicolor’s crappy website) and they do indeed make cable modems.

So fingers crossed, this replacement is a cable modem and that it’s router works properly.

Failing that, I can feel a letter to the TIO coming on.

Oh, and if anyone reading this has used a Thomson Cable modem, especially if provisioned by Telstra, I’d love to hear how it went!

When rate limiting your server more than doubles your server output…

At work, we’ve had a few customers mentioning to us that they’ve experienced slow downloads of data from our servers. When I’ve tested it at home, I’ve experienced the same thing, albeit not quite as bad. The best data rate I could get was about 30% of our server’s bandwidth.

In the last few days I’ve had several conversations with the network engineers at our ISP trying to identify exactly what the problem is. (Thank goodness we’re not with Telstra, if we had to wait for 3 times for a field technician to check if it was plugged in ok, we’d lose our business!)

After having the ISP’s network engineer change a few settings on their equipment, and doing some speed tests to a mini speed test site on their servers, we were still only able to utilise about 30% of our output bandwidth. Crapola.

He explained to me that our rate limiting was done by traffic policing at the switch on the other end of our link. After some reading about what traffic policing was, I’m led to understand that when your data rate is exceeded, packets are dropped. Shouldn’t be too much of a drama, TCP is designed to recover from packet loss, and it does a great job of it, right?. But, what does this packet loss mean to our actual throughput rates?

After making numerous other changes, none of which helped our bandwidth problem, I decided to try something else: rate limiting our server.

Our web files are served by apache running on a Mac, and luckily the Mac OS includes rate limiting controls in its built in firewall. (Great little tutorial here).

So with the `ipfw` command at the ready, I limited outgoing traffic on port 80 (http) to 80% of our bandwidth. And viola! Download rates rose more than double from 30% to 80% of our output limit!!

I never expected that rate limiting our server would cause our outgoing data rate to increase! Especially, more than double!

I’m sure there is a time and place for dropping packets (traffic policing), but it appears to be not working well for us. If anyone has more input on where this is appropriate or for suggestions of other alternatives, please let me know!

Telstra Bigpond Cable crappy Netgear Router part 5

When I previously rang Telstra, continuing my hunt for a replacement of Telstra Bigpond’s crappy Netgear CGD24N cable modem, I was told a field technician was required to replace the modem THREE TIMES before they would escalate the issue beyond the call centre to someone higher up in Telstra. Well guess what? The guy never showed up.

I guess this is some ploy of theirs to never get to the THREE TIMES so that they never have to escalate the issue. Bad form.

Last night I rang again, and at least this time was escalated to a senior person in the call centre. This guy, Ed, was pretty bright. But he still didn’t know anything about Network Address Translation as insisted that noise on the line was causing dropouts because of dynamic IP configuration. Yeah, right.

So now I have an appointment for another technician, this will be visit #2 on Friday morning. I hope this guy turns up.

Telstra Bigpond Cable crappy Netgear Router part 4

Today, I rang Bigpond support again about my issue with the Netgear CGD24N router being slow and intermittently having computers time out when accessing web pages. I’m not alone in this issue, simply do a search on the Whirlpool forum to see more people have the same problem.

The girl I spoke to was, as usual, in a foreign call centre. She wanted to check my wireless settings, signal status, etc. I politely obliged her for a while before insisting that she reference the previous information and escalate the issue to a higher level department.

ONLY NOW, I find out that there policy is only to escalate the issue if it persists after a field technician has been to my house THREE TIMES. How much of my time do I have to sacrifice from my work to convince them that there’s something wrong in the ROUTER.

Telstra Bigpond Cable crappy Netgear Router part 3

So another ten days have gone by, with me turning the router on and off to reset it when its NAT table fills up, which causes slow connections and site dropouts – despite some connections continuing to work.

After doing more reading and testing, I was thoroughly convinced that this is a problem with the router. Whether its a limitation of firmware design, memory, whatever, I don’t know. It’s a limitation, and it makes our internet connection painful to use.

So I gathered a bunch of data, video recording of the lights on the modem, printout of modem’s signal status page, dumps of the `netstat` command in various states of working, slow and ridiculously slow, and sent them all in to big pond for a technical support request. The main part of the request is this:

If I bought a device as crappy as this from a retail shop, like Harvey Normal or Office Works, I’d be well within my rights to return it as faulty, get a refund and buy a different model. However, since Telstra Bigpond provide the equipment, this process is *far* more difficult.

It appears that there are two solutions:

1. Disable NAT in the CGD24N, effectively using it only as a modem, and buy my own wireless router; or
2. Exchange the CGD24N for a Motorola single port modem, and buy my own wireless router.

They both suck, because of the obvious step of “buy my own wireless router” which is at my cost.

With that being the punch line, I sent all of that into to BigPond support. They asked me to do some line checks, which all passed, of course, because the problem is in the router, not the modem or the line. But to eliminate the possibility of a faulty unit, or faulty line, I had to get a technician out to check the line. Which happens during business hours, costing my time and more money!

Here comes the Technician

So, along comes the technician. A friendly fellow. Checks the line status. All good, no surprises. He marks it as a faulty router and replaces it. Let’s see how it goes.

Telstra Bigpond Cable crappy Netgear Router part 2

This Netgear CGD24N router has really been giving me a headache. We have several network devices, more than some, but not what I call extreme. There are three people living in our house, and we have:

• 2 x laptop computers
• 2 x desktop computers
• 2 x iPhones
• 1 x iPad
• 1 x AppleTV

There’s a few other devices, like a network printer, but they aren’t heavy on internet traffic.

After doing quite a bit of reading on the Whirlpool forum, it’s clear that there is a known problem with the Netgear CGD24N router with its Network Address Translation (NAT), which is a really important function of a router to share an internet connection with multiple devices (in layman’s terms).

Another guy “Extreame” has started his own forum in competition to Whirlpool (it seems, not as good though… too much flashy useless stuff) and has some good information too. He seems to be quite the expert on cable modems and recommends turning NAT off and buying your own wireless router.

The Symptoms

After using the router for a while, certain web pages will completely time out. It looks like a line drop out at first, but then you discover that some pages continue to work while others don’t. When accessing a web site, the browser tends to keep a connection to the server open for a subsequent request (perhaps some javascript, images, etc). These pages continue to work. Other pages don’t.

Logging in to the router, I can see that the connection is still good. (Also proven by the web pages that do work) and that you can ping the outside world from the router. Frustrating.

Further Testing

The `netstat` command on Mac (and similar on Linux / Windows) show the status of your network connections. Typically, when everything really starts to slow right down, I see a lot of this in my netstat output:

Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0      0  192.168.0.19.60951     118.215.36.244.443     SYN_SENT
tcp4       0      0  192.168.0.19.60950     118.215.36.244.443     SYN_SENT
tcp4       0      0  192.168.0.19.60949     118.215.36.244.443     SYN_SENT
tcp4       0      0  192.168.0.19.60948     118.215.36.244.443     SYN_SENT
tcp4       0      0  192.168.0.19.60947     118.215.36.244.443     SYN_SENT
tcp4       0      0  192.168.0.19.60942     118.215.36.245.443     SYN_SENT
tcp4       0      0  192.168.0.19.60938     118.215.36.244.443     SYN_SENT
tcp4       0      0  192.168.0.19.60937     184.72.233.6.80        SYN_SENT
tcp4       0      0  192.168.0.19.60936     174.129.225.194.80     SYN_SENT
tcp4       0      0  192.168.0.19.60935     174.129.205.91.80      SYN_SENT
tcp4       0      0  192.168.0.19.60934     174.129.205.86.80      SYN_SENT
tcp4       0      0  192.168.0.19.60933     174.129.205.84.80      SYN_SENT
tcp4       0      0  192.168.0.19.60932     64.237.104.20.80       SYN_SENT
tcp4       0      0  192.168.0.19.60931     205.196.123.122.80     SYN_SENT
tcp4       0      0  192.168.0.19.60930     75.101.155.80.80       SYN_SENT
tcp4       0      0  192.168.0.19.60929     75.101.152.248.80      SYN_SENT
tcp4       0      0  192.168.0.19.60928     204.11.109.23.80       SYN_SENT
tcp4       0      0  192.168.0.19.60927     50.17.249.53.80        SYN_SENT
tcp4       0      0  192.168.0.19.60926     50.17.248.226.80       SYN_SENT
tcp4       0      0  192.168.0.19.60925     204.11.109.22.80       SYN_SENT
tcp4       0      0  192.168.0.19.60924     50.17.237.29.80        SYN_SENT
tcp4       0      0  192.168.0.19.60923     204.11.109.21.80       SYN_SENT
tcp4       0      0  192.168.0.19.60922     69.194.143.60.80       SYN_SENT
tcp4       0      0  192.168.0.19.60921     69.194.143.60.80       SYN_SENT
tcp4       0      0  192.168.0.19.60920     204.11.109.24.80       SYN_SENT
tcp4       0      0  192.168.0.19.60919     202.79.210.121.80      ESTABLISHED
tcp4       0      0  192.168.0.19.60918     69.194.143.60.80       ESTABLISHED
tcp4       0      0  192.168.0.19.60917     69.194.143.60.80       SYN_SENT
tcp4       0      0  192.168.0.19.60916     69.194.244.14.80       ESTABLISHED
tcp4       0      0  192.168.0.19.60915     208.96.4.68.80         CLOSE_WAIT

What’s all this SYN_SENT business? I haven’t seen that before. After looking into it, this is what happens when your computer is trying to make a connection to a remote server but no response has been received from the server. This is consistent with the NAT table being full, such that the router cannot return the packets from the server back to the computer. This is also consistent with why some sites continue to work while others time out.

According to the Whirlpool status page, the CGD24N has a NAT routing table size of 1024. So that’s 1024 connections, should be enough for about 8 devices right? Let’s think about it. Each mail account has probably 2 connections (send via smtp and receive via imap/pop). Each web site you go to typically has about 5 connections from modern browsers like Firefox and Safari. Other applications, such as iTunes, that access the internet all make their own connections too.

The result is, that you don’t need to have too many windows or tabs open at once to run into this limit. So is 1024 normal? small?

I found one web page that lists some routers used on Verizon DSL connections. They have substantially more than 1024! Even the smallest is 7500 and they get into the hundreds of thousands!

Proving the point

I managed to get an ssh connection to a computer at my work, and ran a SOCKS proxy over the connection. After adding the SOCKS proxy to my network configuration, all of a sudden, my connections were all working. A few more hops to the internet, and doubly using my work’s internet connection, which isn’t an acceptable solution, but it proves that bypassing the NAT in the router and making all requests through a proxy (which are sharing one established connection) gives no network problems at all.

Out of curiosity, the `netstat` output looks like this:

Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)    
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54335        ESTABLISHED
tcp4       0      0  127.0.0.1.54335        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54334        ESTABLISHED
tcp4       0      0  127.0.0.1.54334        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54333        ESTABLISHED
tcp4       0      0  127.0.0.1.54333        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54332        ESTABLISHED
tcp4       0      0  127.0.0.1.54332        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54331        ESTABLISHED
tcp4       0      0  127.0.0.1.54331        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54330        ESTABLISHED
tcp4       0      0  127.0.0.1.54330        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54329        ESTABLISHED
tcp4       0      0  127.0.0.1.54329        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54328        ESTABLISHED
tcp4       0      0  127.0.0.1.54328        127.0.0.1.1085         ESTABLISHED
--snip--
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54249        ESTABLISHED
tcp4       0      0  127.0.0.1.54249        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54248        ESTABLISHED
tcp4       0      0  127.0.0.1.54248        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54247        ESTABLISHED
tcp4       0      0  127.0.0.1.54247        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54246        CLOSE_WAIT 
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54241        CLOSE_WAIT 
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54239        ESTABLISHED
tcp4       0      0  127.0.0.1.54239        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54237        ESTABLISHED
tcp4       0      0  127.0.0.1.54237        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54224        ESTABLISHED
tcp4       0      0  127.0.0.1.54224        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54223        ESTABLISHED
tcp4       0      0  127.0.0.1.54223        127.0.0.1.1085         ESTABLISHED
tcp4       0      0  127.0.0.1.1085         127.0.0.1.54222        CLOSE_WAIT 
tcp4       0      0  192.168.0.4.53657      17.149.35.59.5223      ESTABLISHED
tcp4       0      0  192.168.0.4.53643      17.250.248.83.5223     ESTABLISHED
tcp4       0      0  192.168.0.4.53632      17.172.62.121.443      ESTABLISHED
tcp4       0      0  192.168.0.4.53617      69.89.31.155.22        ESTABLISHED
tcp4       0      0  192.168.0.4.52488      199.47.217.147.80      ESTABLISHED

Not a single SYN_SENT. And minimal connections to the outside world via our local router and it’s ridiculously small NAT table.

What now?

Great, so Telstra Bigpond provide a cable modem and router where the router stinks. Now, if I bought this from a shop, I’d return it, and buy a different model… but what are the choices with Telstra??

Telstra Bigpond Cable crappy Netgear Router part 1

We just signed up for a Telstra Bigpond cable connection. I was previously with Internode, who I was very happy with, for many years, and it was a big decision to leave. Basically, we changed because of more data, less price, and most importantly faster downloads. Previously, I’d had good experience with ADSL2, but our house we moved into a year and a half ago has a crappy phone line where we could only get around 4 Mb/s, which didn’t cut it for our increasing internet video usage. (Apple TV and Youtube more and more these days).

So, along comes with installer with a Netgear CGD24N cable modem and router. It’s got a two nice features, our old modem didn’t have:

1. Wireless N
2. Guest network

Great. BUT, there’s always a BUT. These are the issues I discovered straight off:

• admin password doesn’t always work, meaning sometimes you cannot access the router config
• no available manual
• some really slow connection times… more on this later.

I spent time on the phone to Netgear chasing the manual. They were very unhelpful, claiming it was a model specific to Telstra (despite being on their website with technical specifications). So my first support call to Bigpond was to get the manual. At least they provided it to me. Thanks.

So far, the internet connection is largely very good. Downloads are smashing fast. Speedtest tops about 28Mb/sec from a wired connection to the router. And HD videos on the Apple TV stream nearly instantly.

In the main, I’m happy with the connection, but I have a feeling this little router is going to annoy the crap out of me.