How to murder a Blog

Me.

During a server upgrade a few years ago I ran a script to automatically update the code on all of my sites. It just so happened that the script was based on my new site install script which included the above robots.txt line to prevent staging sites from getting indexed before their launch.

So, you guessed it, it took me just long enough to notice my FUBAR that Inert Ramblings had effectively disappeared off the Net over the course of two weeks. Unfortunately, recovery wasn’t as simple as just restoring the original robots.txt and the site instantly reappearing; it still took months for Google and other search engines to reindex the entire site and for traffic to pick back up.

Is there a moral to the story (other than pointing and laughing at me)? Sure. Be absolutely certain that none of your maintenance scripts make unexpected changes to robots.txt or any other config file. Luckily only one site was effected in the long-term but it could have been much worse had it taken longer for me to notice.

Just in case you’ve been keeping track, this is the seventh time a Spammer has forged one of the domains under my control as the return address of their Spam run. The first time it happened I felt shocked, angered, and betrayed. After I lost count of the incidents on my left hand and had to move on to my right hand to keep track it now just feels old-hat. Sure, it still pisses me off, but I now just chalk it up to being the owner of many high profile domain names.

Previously I noticed Spam runs immediately because the mail server started choking on the influx of bounces. This time the domain in question had a Barracuda Spam Firewall 200 sitting on the front lines in the DMZ; I didn’t notice the barrage until the midnight reports ran and I received the pretty pie charts showing a 600% increase in e-mail (and that was just the first day).

Days 2 and 3 jumped up to a 800% to 900% increase in e-mail before starting to plateau and drop off on Day 4. All in all around 350,000 bounces were blocked which is pretty routine for a forged run but the Barracuda performed without so much as a hiccup.

And, for the more visual among you, I’ve of course attached the obligatory pretty graph to show what a Spam forgery looks like. Green is legitimate e-mail, red is blocked Spam, magenta is bad recipient, and blue is rate control (too many connections).

Plesk comes configured with a limit of 4 connections per IP address and a limit of 40 connections total. Modern IMAP clients such as Mozilla Thunderbird use mailbox caching to open up multiple connections to increase performance. In the case of Thunderbird, it opens up 5 connections by default which is already 1 connection more than Courier-IMAP’s default restriction. Add another few family or corporate computers behind a firewall and those additional users won’t be able to connect at all since a single Thunderbird client is already utilizing all 4 connections.

To increase this restriction, modify the /etc/courier-imap/imapd configuration file and change MAXDAEMONS and MAXPERIP to a more sane number. In the case of my configuration, I changed MAXDAEMONS from 40 to 80 and MAXPERIP from 4 to 40. This allows all the machines behind my home firewall to connect to multiple accounts on the e-mail server with mailbox caching enabled.

But even those numbers may be too low for a corporate colocated server that services an entire company. Tweak those numbers based on your employee base; if 50 employees are connecting to the e-mail server from behind the same firewall then MAXPERIP could need to go as high as 250 (50 employees times 5 cached mailbox connections). Add e-mail clients of people working from home and MAXDAEMONS could go as high as 300 or 400.

Obviously, the connection limits are to prevent the Courier-IMAP server from using too many memory and CPU resources on the machine. Tweak the numbers based on the memory footprint of each daemon process and how much memory you have.

When the latest RHEL RPMs were rolled out the /etc/named.conf symlink which should point at Plesk’s /var/named/run-root/etc/named.conf config file was changed to point at RedHat’s default /var/named/chroot/etc/named.conf file. Post-install scripts then munged the location of the Plesk config files. This resulted in the following error when starting or restarting the nameserver:

none:0: open: /etc/named.conf: file not found

A thread was started in the Plesk forums reporting the problem and a Rackspace employee replied in an earlier thread with a fix.

Basically, rpm -e bind-chroot to remove the RedHat RPM that conflicts with Plesk and then re-symlink /etc/named.conf to point back at the proper /var/named/run-root/etc/named.conf config file. This drops Plesk’s config files back into place.

Original E-mail:

Save those old phone bills!

• NY Times Article
• USA Today Article

If you can’t access those articles for some reason, you can Google phone tax Spanish and get several articles that tell about the decision to end the 108-year-old tax that had been implemented to help pay for the Spanish-American war.

Bottom line – you may be able to get refunds on taxes paid. 3% – may be worth it to you. I don’t know how much hassle will be involved.

According to the USA Today article, you can file for refunds back to 3/1/03 (2003, not 1903).

Response #1:

Crap! I just shredded all of those.

Response #2:

Call up NSA – they should have the records [VBG]

Werd. I love the Internet.

So, for any of you receiving the following errors, double-check your crontab and make sure MRTG is only running once!

Rateup WARNING: /usr/bin/rateup could not read the primary log file for colo1.pixoul.com_2
Rateup WARNING: /usr/bin/rateup Can't remove foo.old updating log file
Rateup WARNING: /usr/bin/rateup Can't rename foo.tmp to foo.log updating log file

To solve the problem go to Preferences, Advanced, and use INBOX as your IMAP Path Prefix. Problem solved!

Many thanks go out to MacFixIt and SpamapS.

Update (03/29/2007): Please note that this fix is not a magic bullet and is specifically for Courier IMAP mail servers running on Linux managed by a Plesk control panel environment because that’s the only place I’ve tested it. There are literally hundreds of different IMAP servers out there and each one has its own quirks and configuration options.

Looks like my IP address at home changed. Would that irritate the SonicWall? My new IP is XXX.XXX.XXX.XXX, can you feed that to the SonicWall? I’ll sacrifice a chicken or two over here, maybe together we can get it to work.

I updated his IP address on the firewall and got the best response evar:

OMG it works now!!!11

kthxbye

Ha. I love my co-workers.

Since it was a few hours before I realized what had happened when the incoming Spam getting by SpamAssassin increased tenfold, it was too late to run sa-learn over the old Spams because they had already been truly deleted.

I had to wipe my Bayesian database and start over. No big deal; just inconvenient.

I suck.