|
|
|
server Sorcerer's Apprentice
Tuesday, January 3 2017
At a little after 8:00pm, when my colleague in China finally got online, we decided to try putting some code I'd written live. It was code related to enabling more postfix instances on the email server, and I'd thought I'd tested it pretty well. So we had Da push it live using git, and it was immediately apparent that bad things were happening. "Roll it back!" I shouted via Slack, and a minute or two, Da rolled it back. But soon it was clear that his roll-back hadn't actually rolled anything back, because bad things continued to happen. Massive numbers of emails were being added to the outgoing queue, and though I'd clear that by restarting mysqld (the queue was in a memory table, which disappears with every mysqld restart), it soon inflated back to an unimaginable size. For me, it was like being trapped in a real-live Sorcerer's Apprentice. I eventually copied known good files to the server, bypassing the evidently-failing git system. After that, most of the problem with the overstuffed queue slackened, though for some reason one of the instances maintained a permanently-inflated queue count of about 50,000. It should've been something closer to zero. There were other problems as well: a script designed to turn the emailing process on and off kept producing multiple copies of itself, something I eventually had to reboot the server to fix. By around 10:00pm, the worst of it was over, and though it was growing smaller, the queue remained inflated for instance 1. I was terribly stress-out from this experience; the stress was actually so bad that I could feel it everywhere in my body and it seemed a little dangerous. I decided to take a hot bath in an effort to relax. Had it been a night for drinking booze, I would have. But it wasn't, and my alcohol rules are a very important limit on my behavior.
But I didn't bathe for long. Eventually I got out of the tub and returned to my computer and resumed working, completely naked. I needed to get to the bottom of the inflated queue size in instance 1. I soon discovered that my new code had nulled-out some columns holding maximum IDs, and without those numbers in place, the algorithm had gone back to the bottom of a table and begun resending emails for multiple lists. What a nightmare! My stress returned, manifesting as an awful pressure inside my head. I could also feel myself sweating, though I was not hot.
I quickly put large values in all the columns that had been nulled-out in an effort to stop the mailings, and then restarted mysqld yet again. This seemed to work, and the queue went back to being more of a trickle.
But that wasn't good either; it meant a lot of emails that were supposed to go out were not going out. Fortunately, there was an emaillog table I could query to find out what those values should've been. By about 1:30am, I'd managed to restore everything to its proper state, though of course there was nothing I could do about all the emails that had gone out. This made me feel physically weak and psychologically self-critical. How could I have failed so badly? And who would be yelling at me about this tomorrow. I'd seen in my logs that one of the email resends had gone to the president of The Organization, and I've heard from him before (though always indirectly) when unexpected things have happened in the email system.
For linking purposes this article's URL is:
http://asecular.com/blog.php?170103 feedback
previous | next |
