When it rains it pours

Well, in the spirit of sharing information freely I´ll let you in on what happened tonight although it´s almost too embarrasing to reveal. When debugging the network issues last week (the dos attack), we did several things, including accidentally introducing a routing loop. The gateway router in front of the cluster also has an STM-1 and a satelite connection to Iceland and the loop went undetected until tonight when there was traffic to the subnet in question and the traffic volume started to build up. We quickly discovered the error and corrected it but not before it had deteriorated performance of the cluster connection for a couple of hours. Man, I hope this is the last incident in at least a month or two..