[Most Recent Entries] [Calendar View] [Friends]
Below are the 2 most recent journal entries recorded in
GRAVE STATUS UPDATES AND CEMETERY NEWS' InsaneJournal:
| Saturday, August 13th, 2022 | |||||||||
| 8:22 pm | AWS downtime followup (bold is my anger)
So, if left to AWS, DJ would have been down completely from Sunday until early Thursday morning because someone couldn't figure out how to go flip a switch. If they weren't expiring their old network method that's been this way SINCE THE BEGINNING OF TIME, none of this would have happened. Clearly their playbook team didn't actually consider ancient VMs running... which is going to bite them badly on the 15th (tomorrow).  Anyway, I had moved everything where it needs to be on Thursday/Friday and finally got around to this nice little post-mortem. You can follow us on the bird shitpost site: Follow @deadjournal | ||||||||
| Tuesday, August 9th, 2022 | |||||||||
| 3:56 pm | Welcome back You might have noticed a recent downtime. Or, you may have not. Regardless, for those of you not following @deadjournal on the bird site, here is a boring update: We had been receiving notifications from Amazon AWS that this was happening, and I scheduled time out of my life to deal with this for the past weekend: We are contacting to remind you that on August 15, 2022 we will retire the EC2-Classic environment and you will need to migrate your AWS resources from EC2-Classic to Amazon VPC before this date. So, I educated myself and spent all day Saturday configuring a VPC that would work for DJ. It was convoluted and annoying because I couldn't use their automation tools to convert it over, so it had to be manually. After bringing up some test instances of DJ and hooking it to the VPC, I worked out the kinks(!) and bugs. So, I brought down DJ and created a new instance and used the VPC. All was looking great until I tried to move over the IP to it...  TLDR; we're obviously up on a different IP while they figure their shit out. TL: A long time ago, I had to have DJ's IP permitted to send email through AWS (which by default doesn't permit inbound, because we get abuse, and support updates, as well as some people use email to reply to journal comments), so they tagged the IP as allow. We also needed the IP to have reverse DNS correct so that the majority of email exchangers (like gmail, outlook, hotmail(!), etc) wouldn't just either tag it as spam or outright refuse the connection. They made the changes (this was before they added the option to the AWS Console UI to do it yourself). So the interface was burping that I couldn't move the IP to the VPC because it was... locked. So, I was ready and couldn't get the IP over to the new instance on the VPC. I popped a support ticket to them (OH YOU THINK DEALING WITH CABLEMODEM SUPPORT IS BAD?). They even had a little pull-down about an issue with the EC2->VPC as the subject... how convenient.  I get someone telling me to read pages of documentation and what not, when I specifically pointed out that the problem was the UI and that the IP was locked and that someone needed to do their magic and move it to the VPC scope in a chat with the rep. I get a reply that they have, as a courtesy to me, moved it to premium support, since we only have basic support and that I would have to wait for them to reply. Chat over, and now I'm stuck with doing updates to the ticket through the support site. TIME PASSES. NOTHING HAPPENS. Then I get a response that I need to remove the reverse DNS which will take a couple days by filling out a form and waiting for an email. I replied with that this was unacceptable, and that the problem is on their side and some wizard needs to fix it... to which I was met with crickets. I updated Twitter, and the typical AWS support account that searches the troves of bird shitposts responds to the message asking for the case #, to which I DM'd them. I get a beautiful response that they have looked at the ticket and that due to the nature of Twitter that they can't discuss it and that they have notified support to look at it. Much screaming later... I decided to go ahead and start some much-needed maintenance on the database holding all of your precious diatribes and tribulations that was going to take at least a day of downtime anyway. At the same time, I started changing everything over to a new IP, because who knows how long this is going to take them? I get a support response that they are forwarding it to other people to deal with it. Meanwhile, the database fixing finished, and the DNS IP change I made to the domain finished propagating, so I just brought DJ back up on it. AND WE ARE STILL WAITING FOR A RESOLUTION. When that happens, I can switch the IP back, and all will be back to normal. In the meantime, any journal entry reply notices, etc may end up in your SPAM folder because of the RDNS and lack of reputation this new IP has with spam checkers. So, once again, AWS is causing time-consuming headaches. Stay tuned. Or don't. I'm not your parent. | ||||||||
