By day 4, my attention span and body were starting to give out. I really needed to hang in there, though, because this was the night of re:Play, the big party. I always enjoy going to this party to see Jen Lasher play. She is super energetic, talented with mixing beats, and really fun to watch. Her enthusiasm is infectious.
She did not disappoint at all.
I knocked out one session of notes - the other sessions I attended did not garner anything interesting to write down. Read on for the notes on architecting serverless apps at scale.
Day 4 of re:Invent 2019 and my suspicions were correct: This show was being run much better than before. The busses were punctual. Hell, they actually had too many busses. There was no waiting. The conference staff was jubilant. The mood wasn’t stressed at all. Everyone was having a great time.
I only took notes for one session on this day. It was a session about serverless networking, which focused primarily on what happens when you run Lambda “inside” your VPC. Spoiler alert: it’s not running inside your VPC at all. This actually solved a mystery we were having and it was worth the visit for that knowledge alone.
On the evening of this day, I had the honor of attending the Aerosmith concert at the Park MGM. It was wonderfully awesome. Steven Tyler is 71 years old and still going strong like he’s 20. I hope I’m still ticking like that at 71.
Read on for the session notes.
On day 2 of Re:Invent 2019, it was pretty clear that Amazon Web Services had done a lot of work to scale this conference out. Last year, it was 55,000 attendees and the logistics were terrible. The app crashed often and it was damn near impossible to reserve a seat for any sessions. Getting up and down the strip was a nightmare. AWS had promised a shuttle system, but the shuttles were not well thought out and it actually made it even more difficult to get from venue to venue. They didn’t have enough busses. They didn’t have enough personnel. They didn’t have enough of anything. Ironically, it was like they couldn’t scale out the conference like they can scale their compute services.
There were definitely more people this year. How much more? At least 65,000. But later I learned that the final count was somewhere closer to 80,000… and they were handling them all very well. I was happy to see this because I had pretty much sworn that I wouldn’t be attending anymore re:Invent conferences because it was such an awful experience. This conference was already turning into a complete 180.
I skipped Andy Jassey’s keynote. Andy is a competent guy and runs his business very well. He is not, however, a very good presenter. His keynotes run 2-2.5 hours and they’re full of marketing and momentum-bursting interruptions with musical acts. It’s like he’s trying to be Apple, but not sure how to do it. He needs to get some better coaching. I watched his keynote from the comfort of the certification lounge the last few years. This year, I decided to just skip the keynote altogether and read the summaries on the net later. They had already announced so many new services and enhancements that I had no idea what they might want to introduce in the actual keynote. The keynotes were reserved seating only and I didn’t realize that until it was much too late.
I spent the day chasing sessions. It was here that my strategy formulated for this re:Invent and the years to come: favor chalk talks, builders sessions and workshops over sessions. Sessions are recorded for YouTube. The others are not. They’re also highly interactive and much more involved. That’s just my opinion anyway.
Read on for session notes.
I’m at Amazon Web Services’ Re:Invent 2019. I’ve gone to every re:invent except for one. It’s always an interesting mixed bag of experiences, but I can say for sure that I think they’ve just about got this thing down. This conference is run much better than it was last year. There’s enough session repeats, enough venues, enough shuttles, and just generally enough of everything to keep you moving. There’s almost too much to do on the social side of things. There’s no way you can hit every social event. It’s even hard to find them because there’s so many. (Seems strange, but true).
Over the next few days, I intend to keep a log of some of the more important notes I took during sessions and other things. There’s really too much data at this point. AWS is moving too fast and you can definitely make a lifetime career out of specializing in AWS.
Let’s start with my notes from day one.
I’ve spent a number of years working in a few DevOps/DevSecOps roles to transform organizations into new ways of doing business. I love automation and cloud, and I particularly love infrastructure as code. DevSecOps transformation is not only about the tech, but it’s also about the people. Maybe even more so.
TLS 1.3 is here and ready to go. But a couple of jerks thought breaking it on purpose would be a prudent thing to do. Apparently, they even tried to get NIST to stifle TLS 1.3 to let this broken standard gain traction.
I guess we’ll never learn.
Last week, I sat for the AWS DevOps Professional certification. I took the 2018 version of the test, because it’s still in rotation until sometime in February. It was an interesting, grueling slog of a test… as AWS tests usually are. It wasn’t as difficult as the SA Pro, though.
You may have figured out that the site has changed once again.
I did this once in 2016. I tried to move the site to Hugo. But I was frustrated with the need to run an EC2 instance.
No longer.
I crafted a Terraform module that creates a full AWS publishing CodePipeline for a Hugo site to an S3 bucket.
Clever laziness.
Bye bye, Wordpress.
I just completed some work on a little project with some unique requirements. It’s a project that uses Terraform to provision infrastructure within AWS. That’s not too terribly hard. We’re trying to make the platform, infrastructure and code as reusable as possible while maintaining customer-specific privacy and security requirements.
Did you create a multi-tier Elastic Beanstalk deployment? Did you tie it to CodePipeline to deploy out of Github? Has it been working well until just recently?
…did you accidentally leave RDS attached to your worker tier?
This post is for you.
I built an Elastic Beanstalk for a customer with those characteristics. It’s been working great for about a year, until suddenly… the developer of the application reports that he’s no longer able to deploy his code changes. It keeps failing and rolling back all of the changes to the last known good state, which includes older versions of his code. This was bad news for everyone because we had a Monday-morning deadline to demo code changes to a new customer.
Sunday morning offered me a chance to sit and focus on this. I’ve been trying to understand this problem for a few days. It looks like I was finally able to understand the issue after some focus and coffee.
First, let’s cover what was actually happening. When the developer pushed his code updates through CodePipeline, Elastic Beanstalk was working through its “magic” (cough) to update the config to its “known good state” (which was wrong) and failed to apply the changes because of CloudFormation problems. This triggered a rollback on CloudFormation, CodePipeline, and Elastic Beanstalk config changes. Hence the failure.
How did it all get out of whack?
There were several mistakes committed, most of them on my part. Some of them are just problems with Elastic Beanstalk itself. But I’ll make the no-no list:
Don’t let Elastic Beanstalk manage your RDS instance. Remove all references to RDS in all tiers before you build your RDS instance. Even AWS tells you to not to do this. I missed the one in the worker tier.
If you proceed forward with RDS tied to your EB, do NOT use the RDS console to make any changes to the RDS instance. EB won’t know about the changes and will get really angry when they don’t match. In our case, we did some performance testing and modified the RDS instance size from db.t2.micro to db.m4.large. We also changed the storage setting from 20gb to 100gb. We made those changes in the RDS console and not the EB console. Don’t do that.
You should change one setting in the RDS console. Turn off automatic version upgrade. In our case, RDS was upgrading the minor version of the database and once again, EB got angry. Worse yet, you can’t change the minor version in EB’s console. It’s locked. That’s EB’s fault. But whatever.
Those three items led to a huge bag of fail whenever our developer pushed changes. Elastic Beanstalk would initiate changes, but see that RDS’ configuration was out of whack from its understanding. It would fail and roll everything back.
But wait - there’s more!
Elastic Beanstalk was also using some very old CloudFormation to make changes to the RDS instance. It was still using DBSecurityGroups, which apparently is illegal to use now… at least for our case. We were using postgres and minor version 9.6.6. It looks like the RDS team has moved on from DBSecurityGroups and now enforces the use of VPC Security Groups. Therefore, any changes to RDS would completely fail with the error:
Updating RDS database named: <instance> failed Reason: DB Security Groups can no longer be associated with this DB Instance. Use VPC Security Groups instead.
Ouch.
How do you fix all of this mess?
Let’s go over how Elastic Beanstalk actually works. I’ll be describing some of the simple concepts that are covered in documentation on the AWS site. Bookmark it and keep it handy.
First thing’s first. You need to understand that Elastic Beanstalk is really driven by a simple YAML file. This YAML file is specific to the “environment”, which is a child of the “Application” in Elastic Beanstalk. This always confuses me because I think of an “Environment” as being a place to put an “Application,” but in Elastic Beanstalk it’s backwards of how I think. AWS has a pretty good document on how you can look at this YAML file and see what’s going on.
In this case, I was able to save the configuration as described in the AWS document. I then visited the S3 bucket and was able to see a few things that was making my life difficult. There was also a clue left in this document about how EB was driving changes to the RDS instance via CloudFormation. I knew this was happening. If you’re using Elastic Beanstalk, take a few minutes to go look at your CloudFormation console. You’ll see a template in there - one for each EB “environment” you have deployed. The top of your EB environment dashboard has an “environment ID” displayed in a very small font. This environment ID corresponds to the CloudFormation template ID in the CloudFormation console. You can see the nitty-gritty of what it’s trying to do in there.
But Elastic Beanstalk is coughing up some invalid CloudFormation. How do I know? That security group error that was coming up is actually coming out of CloudFormation. I can see the error event in there. CloudFormation is the service that actually triggers the rollback. CloudFormation and RDS is enforcing the change away from DBSecurityGroups to VPCSecurityGroups. But when Elastic Beanstalk creates the CloudFormation template to initiate the change, it uses DBSecurityGroups.
I used one troubleshooting session to manually fix the CloudFormation JSON that Elastic Beanstalk is spitting out. I pushed it through by hand and it worked. I made the changes to the security groups in the way that CloudFormation and RDS expect - however, if I initiated a change through Elastic Beanstalk or the developer pushed a code update, it would fail with invalid CloudFormation once again.
I’ll take a quick break to break down what’s happening here. When you make a change in Elastic Beanstalk, my new understanding is that this happens:
Elastic Beanstalk console writes new YAML config file to S3 –> Elastic Beanstalk parses the config file and decides what changes should be made –> Elastic Beanstalk generates a CloudFormation JSON template –> Elastic Beanstalk saves the CloudFormation JSON to S3 –> Elastic Beanstalk pokes CloudFormation and asks it to update –> CloudFormation updates… if a failure is encountered, it rolls back and tells Elastic Beanstalk that everything is hosed –> Elastic Beanstalk rolls back the version of code that was deployed to a known good state.
Now I understand the root cause here. RDS made a change to enforce the security group update. Elastic Beanstalk can’t seem to figure that out.
Here’s how to resolve this.
Look at the AWS documentation on Elastic Beanstalk’s config above. Follow their steps to save the configuration file from the console. Then, get your favorite code application out. Download the file and manipulate it by hand.
I changed the RDS properties to reflect reality. EB still thought it was postgres, version 9.6.2, on a db.t2.micro with 20gb of storage. I updated these properties to reality.
Then, I saw it. At the bottom of the file, there is a block of YAML that tells Elastic Beanstalk where to pick up the CloudFormation JSON and feed parameters. The default value was:
Extensions:
RDS.EBConsoleSnippet:
Order: null
SourceLocation: https://s3.amazonaws.com/elasticbeanstalk-env-resources-us-east-1/eb_snippets/rds/rds.json
Take a look at that URL. Go ahead. I’ll wait.
See it?
It’s the bad CloudFormation template.
How did I resolve this? Well, I took that template and downloaded it. I modified it in my code editor to change the DBSecurityGroup resources into VPC Security Group resources. I had to manually add the SecurityGroupIngress information too, but because I speak CloudFormation this wasn’t too hard. It’s cheating a little bit, but not a big deal.
I created a new S3 bucket and uploaded my new CloudFormation JSON template into that bucket. Then, I revisited this YAML config and changed the URL to point to my new private copy of the CloudFormation template.
Go back to the Elastic Beanstalk console and “load” the configuration template and wham, it worked. Everything was fine.
Now I know how Elastic Beanstalk really works, and I figured out some super advanced ways to manipulate it to my bidding.
I hope this helps you understand Elastic Beanstalk a little more - it certainly helped me. Now I know how to trick Elastic Beanstalk into working if it hoses up again.
Since it’s working, turn off minor version upgrade in RDS to prevent this from happening, then use your AWS support plan to tell them that Elastic Beanstalk has a bug with CloudFormation and RDS security groups :)
Happy cloud days.