A badly configured script to update the DNS configurations for the .se domain name. The problem was fixed relatively quickly but due to the prevelance of DNS caching the problem persisted much longer after the fix. According to the article most of the major Swedish ISP’s quickly dropped their DNS cache so that the new settings could be set and working, but foreign ISP’s may not have known about the failure and haven’t allowed their cache to get the new data.
So what does this mean for testing? There are obviously redundant DNS servers in place for changes like this, but what about the test servers. As the article mentions the .se domain name has about 900k domain names. Imaging the repercussions if the same thing would have happened to the .com domain (about 80M names). Why wasn’t the script thrown at a few test DNS servers? and if it was, why weren’t those tests able to find the problem. You’d think that an unreachable error testing a DNS entry would be a fail point.
I’m going to go out on a big limb here and assume that the test servers weren’t configured the same way (I don’t want to assume that no testing was done). As much as a company/organization/department can they should setup their test environment to be as identical as possible to the production environment. In the case of web application testing your QA equipment should be configured exactly the same as the production equipment from an OS standpoint. For installed software you should be testing it on machines that your userbase will be on. And for server settings, update, and maintenance testing you should have a box with the same exact everything as a production model. SAME EXACT! Model Numbers, suppliers, OS Install base, and settings. Otherwise, how do you know that the update you just successfully tested on your QA side didn’t just break the internet… literally?
See Also:
