WARNING: Unfortunately negative post.
This has been an interesting year in the Appleverse. iOS 13, iPadOS and macOS Catalina were all dropped on us. This new software “regime” has been quite the challenge for me.
iOS 13 and iPadOS haven’t been that troubling. They generally work and do what they promised to do. I did find it curious that iOS 13.1, 13.1.1 and 13.1.2 all dropped in pretty rapid succession. That’s usually a bad sign that things weren’t up to standards and had to go through some quick resolution. There were either fixes or outright removals to get things out the door. I don’t like it when that happens, but I get it. I’m glad they stay on top of things well enough.
Catalina and Apple TV on the other hand… have been a complete shitshow.
Update: I resolved the Google Chrome issue. If you’re just interested in the resolution, please go here.
I’m at the NAB show in Las Vegas, NV this week. I’m here to represent my project for my main customer, which is a federal agency that does space stuff. You can guess.
This is the first trip where I’ve had to accept and travel under the new DFARS requirements. I’m here to tell you, this shit is for the birds. I’m carrying two laptops. Two chargers. Two sets of cables. All because the feds thinks this somehow makes it all more secure.
The backpack is heavy. There’s maximum effort to “be secure” and minimal gain. As a matter of fact, I would warrant that this renders my setup even less secure. Now I have to keep up with two laptops - and what happens if I leave one somewhere? What happens if I leave my ID card somewhere? What happens if somehow, someone was able to take the federal laptop and my PIV card and get into the VPN?
They’d have full access to the enterprise, that’s what.
You people that think this is more secure - you’re really nuts. You’re not thinking clearly at all. On top of it, you’re making IT workers’ already-difficult lives even more difficult.
Did you create a multi-tier Elastic Beanstalk deployment? Did you tie it to CodePipeline to deploy out of Github? Has it been working well until just recently?
…did you accidentally leave RDS attached to your worker tier?
This post is for you.
I built an Elastic Beanstalk for a customer with those characteristics. It’s been working great for about a year, until suddenly… the developer of the application reports that he’s no longer able to deploy his code changes. It keeps failing and rolling back all of the changes to the last known good state, which includes older versions of his code. This was bad news for everyone because we had a Monday-morning deadline to demo code changes to a new customer.
Sunday morning offered me a chance to sit and focus on this. I’ve been trying to understand this problem for a few days. It looks like I was finally able to understand the issue after some focus and coffee.
First, let’s cover what was actually happening. When the developer pushed his code updates through CodePipeline, Elastic Beanstalk was working through its “magic” (cough) to update the config to its “known good state” (which was wrong) and failed to apply the changes because of CloudFormation problems. This triggered a rollback on CloudFormation, CodePipeline, and Elastic Beanstalk config changes. Hence the failure.
How did it all get out of whack?
There were several mistakes committed, most of them on my part. Some of them are just problems with Elastic Beanstalk itself. But I’ll make the no-no list:
Don’t let Elastic Beanstalk manage your RDS instance. Remove all references to RDS in all tiers before you build your RDS instance. Even AWS tells you to not to do this. I missed the one in the worker tier.
If you proceed forward with RDS tied to your EB, do NOT use the RDS console to make any changes to the RDS instance. EB won’t know about the changes and will get really angry when they don’t match. In our case, we did some performance testing and modified the RDS instance size from db.t2.micro to db.m4.large. We also changed the storage setting from 20gb to 100gb. We made those changes in the RDS console and not the EB console. Don’t do that.
You should change one setting in the RDS console. Turn off automatic version upgrade. In our case, RDS was upgrading the minor version of the database and once again, EB got angry. Worse yet, you can’t change the minor version in EB’s console. It’s locked. That’s EB’s fault. But whatever.
Those three items led to a huge bag of fail whenever our developer pushed changes. Elastic Beanstalk would initiate changes, but see that RDS’ configuration was out of whack from its understanding. It would fail and roll everything back.
But wait - there’s more!
Elastic Beanstalk was also using some very old CloudFormation to make changes to the RDS instance. It was still using DBSecurityGroups, which apparently is illegal to use now… at least for our case. We were using postgres and minor version 9.6.6. It looks like the RDS team has moved on from DBSecurityGroups and now enforces the use of VPC Security Groups. Therefore, any changes to RDS would completely fail with the error:
Updating RDS database named: <instance> failed Reason: DB Security Groups can no longer be associated with this DB Instance. Use VPC Security Groups instead.
Ouch.
How do you fix all of this mess?
Let’s go over how Elastic Beanstalk actually works. I’ll be describing some of the simple concepts that are covered in documentation on the AWS site. Bookmark it and keep it handy.
First thing’s first. You need to understand that Elastic Beanstalk is really driven by a simple YAML file. This YAML file is specific to the “environment”, which is a child of the “Application” in Elastic Beanstalk. This always confuses me because I think of an “Environment” as being a place to put an “Application,” but in Elastic Beanstalk it’s backwards of how I think. AWS has a pretty good document on how you can look at this YAML file and see what’s going on.
In this case, I was able to save the configuration as described in the AWS document. I then visited the S3 bucket and was able to see a few things that was making my life difficult. There was also a clue left in this document about how EB was driving changes to the RDS instance via CloudFormation. I knew this was happening. If you’re using Elastic Beanstalk, take a few minutes to go look at your CloudFormation console. You’ll see a template in there - one for each EB “environment” you have deployed. The top of your EB environment dashboard has an “environment ID” displayed in a very small font. This environment ID corresponds to the CloudFormation template ID in the CloudFormation console. You can see the nitty-gritty of what it’s trying to do in there.
But Elastic Beanstalk is coughing up some invalid CloudFormation. How do I know? That security group error that was coming up is actually coming out of CloudFormation. I can see the error event in there. CloudFormation is the service that actually triggers the rollback. CloudFormation and RDS is enforcing the change away from DBSecurityGroups to VPCSecurityGroups. But when Elastic Beanstalk creates the CloudFormation template to initiate the change, it uses DBSecurityGroups.
I used one troubleshooting session to manually fix the CloudFormation JSON that Elastic Beanstalk is spitting out. I pushed it through by hand and it worked. I made the changes to the security groups in the way that CloudFormation and RDS expect - however, if I initiated a change through Elastic Beanstalk or the developer pushed a code update, it would fail with invalid CloudFormation once again.
I’ll take a quick break to break down what’s happening here. When you make a change in Elastic Beanstalk, my new understanding is that this happens:
Elastic Beanstalk console writes new YAML config file to S3 –> Elastic Beanstalk parses the config file and decides what changes should be made –> Elastic Beanstalk generates a CloudFormation JSON template –> Elastic Beanstalk saves the CloudFormation JSON to S3 –> Elastic Beanstalk pokes CloudFormation and asks it to update –> CloudFormation updates… if a failure is encountered, it rolls back and tells Elastic Beanstalk that everything is hosed –> Elastic Beanstalk rolls back the version of code that was deployed to a known good state.
Now I understand the root cause here. RDS made a change to enforce the security group update. Elastic Beanstalk can’t seem to figure that out.
Here’s how to resolve this.
Look at the AWS documentation on Elastic Beanstalk’s config above. Follow their steps to save the configuration file from the console. Then, get your favorite code application out. Download the file and manipulate it by hand.
I changed the RDS properties to reflect reality. EB still thought it was postgres, version 9.6.2, on a db.t2.micro with 20gb of storage. I updated these properties to reality.
Then, I saw it. At the bottom of the file, there is a block of YAML that tells Elastic Beanstalk where to pick up the CloudFormation JSON and feed parameters. The default value was:
Extensions:
RDS.EBConsoleSnippet:
Order: null
SourceLocation: https://s3.amazonaws.com/elasticbeanstalk-env-resources-us-east-1/eb_snippets/rds/rds.json
Take a look at that URL. Go ahead. I’ll wait.
See it?
It’s the bad CloudFormation template.
How did I resolve this? Well, I took that template and downloaded it. I modified it in my code editor to change the DBSecurityGroup resources into VPC Security Group resources. I had to manually add the SecurityGroupIngress information too, but because I speak CloudFormation this wasn’t too hard. It’s cheating a little bit, but not a big deal.
I created a new S3 bucket and uploaded my new CloudFormation JSON template into that bucket. Then, I revisited this YAML config and changed the URL to point to my new private copy of the CloudFormation template.
Go back to the Elastic Beanstalk console and “load” the configuration template and wham, it worked. Everything was fine.
Now I know how Elastic Beanstalk really works, and I figured out some super advanced ways to manipulate it to my bidding.
I hope this helps you understand Elastic Beanstalk a little more - it certainly helped me. Now I know how to trick Elastic Beanstalk into working if it hoses up again.
Since it’s working, turn off minor version upgrade in RDS to prevent this from happening, then use your AWS support plan to tell them that Elastic Beanstalk has a bug with CloudFormation and RDS security groups :)
Happy cloud days.
DFARS.
If you don’t know what that is, look it up. I’m not going to go into it in this article. I only want to discuss the ramifications of DFARS and how it’s being interpreted/implemented.
Every federal contractor company I’ve worked for has a “matrixed” business model. This means in order to save money, they will employ you on a single federal contract - but “leverage your expertise” on other federal contracts. The end result of this is that you’ll end up working on multiple projects across multiple agencies. Because federal agencies refuse to get along and agree on standards, this means you get to go through multiple clearances and obtain multiple credentials (i.e. CAC or PIV cards and usernames/passwords).
This is a little disingenius on the part of the contractor company. It’s been my experience that they will tie you to a single contract and then matrix you to others. But if the funding lapses on the primary contract, they’ll show you the door. Valuable employees are kept but others that are lower level (but still matrixed!) will be laid off.
That’s another issue that is between you and your company.
Anyway… DFARS. The way companies and agencies are interpreting DFARS is the subject of this article. Basically, if you’re a matrixed employee, the end result is that you will end up with one laptop and one mobile device per project.
That’s right.
If you’re matrixed across three different projects, you will end up with three laptops and three different mobile devices. None of these devices will be allowed to communicate with the other agency. Your company will likely issue a company-specific laptop and mobile device as well. In my case, this could result in four separate devices to do your work.
That sounds reasonable, but it’s woefully ignorant of how a matrixed employee does business. Every agency expects the employee to be devoted to their contract, even if they are on record as having only a slice of time. The agency/customer expects that employee to be available at any time… not just during certain hours of the day.
The end result is that the matrixed employee is expected to manage multiple meeting requests across multiple devices without a single integrated view of meeting and work conflicts. This means the employee will miss meetings, emails and lort knows what else.
I predict this will be rolled back within a few years.
It’s untenable.
Me? I’m going to set “out of office” replies that notify senders that I only check my email and calendar during certain parts of the day. They’ll receive that autoreply every time they email me. Sure, I can set it to reply once a day.
I wouldn’t want to like… be annoying, or something.
I must really be out of the loop. I had no idea Microsoft bought SwiftKey. Anyway, they are killing the Windows Phone keyboard for IOS and focusing exclusively on SwiftKey.
When Microsoft does things that makes sense, I’m always surprised. When they do things that do not make sense (like beefing Skype for the iPhone) I am rarely surprised.
Microsoft’s Windows Phone keyboard for the iPhone is dead - The Verge
Microsoft will be killing off “Microsoft Paint” in the next release of Windows 10 (the so-called “Fall Creator’s Update”).
This article on the Verge points out the various things that are being shed. Microsoft Paint seems to be the most significant user-facing thing, but I can imagine some enterprises will have difficulty with other changes.
Be careful when you’re working with CloudFormation in the AWS GovCloud region. Almost every code snippet available on the Internet refers to the public regions of AWS. If you’re making resources in GovCloud with a Cloudformation templates, there are subtle differences.
For instance, referring to an S3 bucket in a code snippet is:
Yesterday I sat for the most difficult IT certification test I’ve ever attempted – the AWS Solutions Architect Professional test.
I passed it… by the skin of my teeth.
I’ve essentially studied for this test for two years or more. I took the Solutions Architect Associate test two years ago and I’ve been involved with AWS projects ever since. Actually I was involved in AWS projects since before that test.
If you’re plagued by the Apple Software Update popping up every day to remind you that iCloud for Windows 5.1 is available to install… even when you already have it installed… you need to go registry-diving.
This community post on the Apple site worked for me. Whew. Finally. That was super annoying.
Hope you’re all well. Reach out to me and let me know how your life is going.
It looks like Satya Nadella’s strategies are paying off. Microsoft reported some gains in Windows 10 and its cloud business.
That’s good news for overall competition. I’ve also witnessed Microsoft become more and more “Internet friendly,” by way of embracing multi-platform applications and developers. This is what needed to happen.
Just think if Ballmer was still around… maybe Microsoft would be folding by now.
They do need to do something about the Windows handset business, but I’m betting that Satya has a plan for that.