An Interest In:
Web News this Week
- March 20, 2024
- March 19, 2024
- March 18, 2024
- March 17, 2024
- March 16, 2024
- March 15, 2024
- March 14, 2024
Some of Our Sources
- Engadget
- Web Designer Wall
- Team Treehouse
- Spoon Graphics
- Abduzeedo
- Noupe
- Web Design Ledger
- Freelance Switch
- Web Resource Source
- Android Dissected
Help Webnuz
Referal links:
Upgrading my AWS CDK stacks to AWS CDKv2
The CDK has been released in two major versions, v1 and v2. This is the Developer Guide for AWS CDK v2.
AWS CDK v1 entered maintenance on June 1, 2022. As part of some recent talks and demos I was doing, I wanted to update the CDK stacks for Managed Workflows for Apache Airflow (MWAA) and thought I would outline what I learned along the way.
Prepping for CDKv2
In AWS CDKv1, the different CDK constructs were imported as discrete Python libraries (for example aws-cdk.aws-s3-deployment, aws-cdk.aws-mwaa, aws_cdk.aws-kms etc) that were imported in the requirements.txt file. AWS CDKv2 has a different approach, so I needed to clean up all the CDK construct modules that I had imported for the CDKv1 stacks. I ended up using the following command to do this.
You can read more about the differences in this post, Migrating to AWS CDK v2
pip list |grep cdk | xargs pip uninstall -yI updated my requirements.txt file as follows:
aws-cdk-lib==2.2.0constructs>=10.0.0,<11.0.0Finally, I had to update the cdk.json file as some of the values have changed.
{ "app": "python app.py", "context": { "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": false, "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": false, "@aws-cdk/aws-rds:lowercaseDbIdentifier": false, "@aws-cdk/core:stackRelativeExports": false }}Installing AWS CDKv2
To install CDKv2, I did the following:
npm install -g aws-cdkYou can check that it has updated by running:
cdk --version2.28.1 (build d035432)It should display 2.x - if it displays 1.x, then check your alias commands, alternatives or try and uninstall and then reinstall.
If you need to keep both CDKv1 and v2, then consider using alias commands to help you.
alias cdk1="npx [email protected]"alias cdk="npx [email protected]"
Re-bootstraping AWS CDK
One of the first things once I fixed/updated my CDK stacks was to re-bootstrap CDK. I did not know I had to do that, but the first time I did a "cdk deploy" I was prompted with this message first:
current credentials could not be used to assume 'arn:aws:iam::704533066374:role/cdk-hnb659fds-lookup-role-704533066374-eu-central-1', but are for the right account. Proceeding anyway.followed quickly by an error message
mwaa-hybrid-backend failed: Error: mwaa-hybrid-backend: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html) at CloudFormationDeployments.validateBootstrapStackVersion (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:482:13) at processTicksAndRejections (internal/process/task_queues.js:95:5) at CloudFormationDeployments.publishStackAssets (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:457:7) at CloudFormationDeployments.deployStack (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:339:7) at CdkToolkit.deploy (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24) at initCommandLine (/Users/ricsue/.nvm/versions/node/v14.18.2/lib/node_modules/aws-cdk/lib/cli.ts:341:12)mwaa-hybrid-backend: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)I was able to fix this quickly by running the CDK Bootstrap command:
cdk bootstrap aws://704533066374/eu-central-1!Note I was doing this with an IAM user with AdminAccess.
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.In the past when I have talked with builders who do not have Admin access but are using either an IAM user or a Role, they have to following this doc and modify the bootstrap template with the role and permissions they have. (--cloudformation-execution-policies) also see this issue
Changes to imports and other niggles
The first thing I needed to do was get my head around the changes to the imports and libraries within my stacks. For example, I had to change from
from aws_cdk import corefrom aws_cdk.core import Tagsimport aws_cdk.aws_ec2 as ec2import aws_cdk.aws_s3 as s3import aws_cdk.aws_s3_deployment as s3deployimport aws_cdk.aws_mwaa as mwaaimport aws_cdk.aws_iam as iamimport aws_cdk.aws_kms as kmsto this
from aws_cdk import ( aws_iam as iam, aws_ec2 as ec2, aws_s3 as s3, aws_s3_deployment as s3deploy, aws_mwaa as mwaa, aws_kms as kms, Stack, CfnOutput, Tags)from constructs import ConstructWhich is actually nicer and cleaner I think. I had to update references to the classes within my code, but that was pretty straight forward and just took some time to go through to make sure I had not missed anything.
Fixing changes to the VPC private subnet constructs
In some of the constructs, it looks like some of the values have changed between v1 and v2. In the VPC construct for example, creating a Private subnet used to be done via:
self.vpc = ec2.Vpc( self, id="MWAA-Hybrid-ApacheAirflow-VPC", cidr="10.192.0.0/16", max_azs=2, nat_gateways=1, subnet_configuration=[ ec2.SubnetConfiguration( name="public", cidr_mask=24, reserved=False, subnet_type=ec2.SubnetType.PUBLIC), ec2.SubnetConfiguration( name="private", cidr_mask=24, reserved=False, subnet_type=ec2.SubnetType.PRIVATE) ], enable_dns_hostnames=True, enable_dns_support=True )and this had to be updated to
self.vpc = ec2.Vpc( self, id="MWAA-Hybrid-ApacheAirflow-VPC", cidr="10.192.0.0/16", max_azs=2, nat_gateways=1, subnet_configuration=[ ec2.SubnetConfiguration( name="public", cidr_mask=24, reserved=False, subnet_type=ec2.SubnetType.PUBLIC), ec2.SubnetConfiguration( name="private", cidr_mask=24, reserved=False, subnet_type=ec2.SubnetType.PRIVATE_WITH_NAT) ], enable_dns_hostnames=True, enable_dns_support=True )This was the only one (PRIVATE to PRIVATE_WITH_NAT) that I needed to adjust, but you may have others.
Fixing SSM Errors
Once the CDK stacks were validated and clear for deployment, I encountered a "feature" when generating IAM policies and roles. It was whilst fixing a different CDK stack (Orchestrating Hybrid Workflows with Apache Airflow, GitHub repo blogpost-airflow-hybrid that I found that even though the stack had synthesised ok, during the deployment, I got the following error:
Invalid principal in policy: "SERVICE":"ssm.eu-west-1.amazonaws.com" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 5763050d-c789-49ef-b7d7-9e80a62f0e9e; Proxy: null)This error appeared in both the command prompt where I ran the "cdk deploy", but also appeared in the CloudFormation console which is a good place to go when you are trying to figure out what is going wrong.
Upon looking at the synthesised CloudFormation template, you could see that IAM Principal was regionalising the 'ssm.amazonaws.com' (so it was creating this as 'ssm.eu-west-1.amazonaws.com' in my case) which was a problem. In my stack I defined the following role:
external_task_def_policy_document_role = iam.Role( self, "ExternalECSAnywhereRole", role_name=f"{props['ecsclustername']}-ExternalECSAnywhereRole", assumed_by=iam.ServicePrincipal("ssm.amazonaws.com") )and this was failing (it worked fine in CDKv1). In the end, after reading GitHub issues 17646 and 16188, I was able to fix this issue within my stack by adding this code:
ssmfix = external_task_def_policy_document_role.node.default_child ssmfix.add_property_override( "AssumeRolePolicyDocument.Statement.0.Principal.Service", "ssm.amazonaws.com" )Deploying the updated stacks
Once I had made these changes, I was then able to deploy the MWAA stack, first the backend (VPC and Networking) and then the environment.
You can find the code for this in the GitHub repository, blogpost-cdk-mwaa
Deployment of the MWAA network (backend)
Here is the output that you will see (more or less)
Synthesis time: 7.09smwaa-hybrid-backend: deploying...[0%] start: Publishing 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1[100%] success: Published 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1mwaa-hybrid-backend: creating CloudFormation changeset... mwaa-hybrid-backend Deployment time: 172.13sOutputs:mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPC677B092EF6F2F587 = vpc-0bbdeee3652ef21ffmwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet1Subnet2A6995DF7F8D3134 = subnet-01e48db64381efc7fmwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet2SubnetA28659530C36370A = subnet-0321530b8154f9bd2mwaa-hybrid-backend.VPCId = vpc-0bbdeee3652ef21ffStack ARN:arn:aws:cloudformation:eu-central-1:704533066374:stack/mwaa-hybrid-backend/b05897d0-f087-11ec-b5f3-02db3f47a5ca Total time: 179.22sDeployment of the MWAA environment
Here is the output that you will see (more or less)
Including dependency stacks: mwaa-hybrid-backend[Warning at /mwaa-hybrid-environment/mwaa-sg] Ignoring Egress rule since 'allowAllOutbound' is set to true; To add customize rules, set allowAllOutbound=false on the SecurityGroup Synthesis time: 12.37smwaa-hybrid-backendmwaa-hybrid-backend: deploying...[0%] start: Publishing 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1[100%] success: Published 2695cb7a9f601cf94a4151c65c9069787d9ec312084346f2f4359e3f55ff2310:704533066374-eu-central-1 mwaa-hybrid-backend (no changes) Deployment time: 1.97sOutputs:mwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPC677B092EF6F2F587 = vpc-0bbdeee3652ef21ffmwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet1Subnet2A6995DF7F8D3134 = subnet-01e48db64381efc7fmwaa-hybrid-backend.ExportsOutputRefMWAAHybridApacheAirflowVPCprivateSubnet2SubnetA28659530C36370A = subnet-0321530b8154f9bd2mwaa-hybrid-backend.VPCId = vpc-0bbdeee3652ef21ffStack ARN:arn:aws:cloudformation:eu-central-1:704533066374:stack/mwaa-hybrid-backend/b05897d0-f087-11ec-b5f3-02db3f47a5ca Total time: 14.35smwaa-hybrid-environmentThis deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).Please confirm you intend to make the following modifications:IAM Statement Changes Resource Effect Action Principal Condition + ${Custom::CDKBucketDeploymen Allow sts:AssumeRole Service:lambda.amazonaws.com t8693BB64968944B69AAFB0CC9EB 8756C/ServiceRole.Arn} + ${mwaa-dags.Arn} Deny s3:ListAllMyBuckets AWS:${mwaa-service-role} ${mwaa-dags.Arn}/* + ${mwaa-dags.Arn} Allow s3:* AWS:${mwaa-service-role} ${mwaa-dags.Arn}/* + ${mwaa-dags.Arn} Allow s3:Abort* AWS:${Custom::CDKBucketDeplo ${mwaa-dags.Arn}/* s3:DeleteObject* yment8693BB64968944B69AAFB0C s3:GetBucket* C9EB8756C/ServiceRole} s3:GetObject* s3:List* s3:PutObject + ${mwaa-hybrid-demoKey.Arn} Allow kms:CancelKeyDeletion AWS:arn:${AWS::Partition}:ia kms:Create* m::704533066374:root kms:Decrypt* kms:Delete* kms:Describe* kms:Disable* kms:Enable* kms:GenerateDataKey* kms:Get* kms:List* kms:Put* kms:Revoke* kms:ScheduleKeyDeletion kms:Update* + ${mwaa-hybrid-demoKey.Arn} Allow kms:Decrypt* Service:logs.eu-central-1.am "ArnLike": { kms:Describe* azonaws.com "kms:EncryptionContext:aws:lo kms:Encrypt* gs:arn": "arn:aws:logs:eu-centr kms:GenerateDataKey* al-1:704533066374:*" kms:PutKeyPolicy } kms:ReEncrypt* + ${mwaa-service-role.Arn} Allow sts:AssumeRole Service:airflow-env.amazonaw s.com Service:airflow.amazonaws.co m Service:ecs-tasks.amazonaws. com + * Allow logs:DescribeLogGroups AWS:${mwaa-service-role} + * Allow ecs:DescribeTaskDefinition AWS:${mwaa-service-role} ecs:DescribeTasks ecs:ListTasks ecs:RegisterTaskDefinition ecs:RunTask + * Allow iam:PassRole AWS:${mwaa-service-role} "StringLike": { "iam:PassedToService": "ecs-t asks.amazonaws.com" } + * Allow kms:Decrypt AWS:${mwaa-service-role} "StringEquals": { kms:DescribeKey "kms:ViaService": [ kms:Encrypt "sqs.eu-central-1.amazonaws kms:GenerateDataKey* .com", kms:PutKeyPolicy "s3.eu-central-1.amazonaws. com" ] } + arn:${AWS::Partition}:s3:::c Allow s3:GetBucket* AWS:${Custom::CDKBucketDeplo dk-hnb659fds-assets-70453306 s3:GetObject* yment8693BB64968944B69AAFB0C 6374-eu-central-1 s3:List* C9EB8756C/ServiceRole} arn:${AWS::Partition}:s3:::c dk-hnb659fds-assets-70453306 6374-eu-central-1/* + arn:aws:airflow:eu-central-1 Allow airflow:PublishMetrics AWS:${mwaa-service-role} :704533066374:environment/mw aa-hybrid-demo + arn:aws:logs:eu-central-1:70 Allow logs:CreateLogGroup AWS:${mwaa-service-role} 4533066374:log-group:airflow logs:CreateLogStream -mwaa-hybrid-demo-* logs:DescribeLogGroups logs:GetLogEvents logs:GetLogGroupFields logs:GetLogRecord logs:GetQueryResults logs:PutLogEvents + arn:aws:sqs:eu-central-1:*:a Allow sqs:ChangeMessageVisibility AWS:${mwaa-service-role} irflow-celery-* sqs:DeleteMessage sqs:GetQueueAttributes sqs:GetQueueUrl sqs:ReceiveMessage sqs:SendMessage IAM Policy Changes Resource Managed Policy ARN + ${Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C/Ser arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasic viceRole} ExecutionRole Security Group Changes Group Dir Protocol Peer + ${mwaa-sg.GroupId} In Everything ${mwaa-sg.GroupId} + ${mwaa-sg.GroupId} Out Everything Everyone (IPv4) (NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)Do you wish to deploy these changes (y/n)? ymwaa-hybrid-environment: deploying...[0%] start: Publishing e9882ab123687399f934da0d45effe675ecc8ce13b40cb946f3e1d6141fe8d68:704533066374-eu-central-1[0%] start: Publishing 983c442a2fe823a8b4ebb18d241a5150ae15103dacbf3f038c7c6343e565aa4c:704533066374-eu-central-1[0%] start: Publishing 91ab667f7c88c3b87cf958b7ef4158ef85fb9ba8bd198e5e0e901bb7f904d560:704533066374-eu-central-1[0%] start: Publishing f2a926ee3d8ca4bd02b0cf073eb2bbb682e94c021925bf971a9730045ef4fb02:704533066374-eu-central-1[25%] success: Published 983c442a2fe823a8b4ebb18d241a5150ae15103dacbf3f038c7c6343e565aa4c:704533066374-eu-central-1[50%] success: Published 91ab667f7c88c3b87cf958b7ef4158ef85fb9ba8bd198e5e0e901bb7f904d560:704533066374-eu-central-1[75%] success: Published f2a926ee3d8ca4bd02b0cf073eb2bbb682e94c021925bf971a9730045ef4fb02:704533066374-eu-central-1[100%] success: Published e9882ab123687399f934da0d45effe675ecc8ce13b40cb946f3e1d6141fe8d68:704533066374-eu-central-1mwaa-hybrid-environment: creating CloudFormation changeset... mwaa-hybrid-environment Deployment time: 1412.35sOutputs:mwaa-hybrid-environment.MWAASecurityGroup = sg-0ea83e01caded2bb3Stack ARN:arn:aws:cloudformation:eu-central-1:704533066374:stack/mwaa-hybrid-environment/450337a0-f088-11ec-a169-06ba63bfdfb2 Total time: 1424.72sConclusion
It was pretty straight forward in the end to update my stacks, but it did take me longer thanks to this being the first time and also encountering the SSM issue along the way. As I have updated subsequent stacks, it has definitely become much easier and quicker, so I recommend that you begin on a simple stack to gain confidence and then apply that to your other stacks.
If you encounter any issues running my stacks on CDKv2, please raise an issue on the GitHub rep and I will take a look and fix.
Original Link: https://dev.to/aws/upgrading-my-aws-cdk-stacks-to-aws-cdkv2-2fa3
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To