Your Web News in One Place

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

Sign up for GreenGeeks web hosting
December 14, 2021 12:35 pm GMT

Setting up MWAA to use a KMS key

Introduction

In a previous post, I shared how you can using AWS CDK to provision your Apache Airflow environments using the Managed Workflows for Apache Airflow service (MWAA).

I was contacted this week by Michael Grabenstein, who flagged an issue with the code in that post. The post used code that configured a kms key for the MWAA environment, but when trying to deploy the app it would fail with the following error:

14:59:15 | CREATE_FAILED        | AWS::MWAA::Environment         | airflowtestenvironmentResource handler returned message: "Creation failed" (RequestToken: 5b5b5a02-8cee-e48a-66ea-a07a8b1ad41a, HandlerErrorCode: NotStabilized)

This issue was that the CDK stack did not have everything needed to properly configure a customer KMS key, so I wanted to address that in this post, how to configure MWAA so that you can use your own KMS key to properly ensure that everything is encrypted.

I want to thank Michael for flagging the issue and his PR to the project and help getting this tested.

Configuring KMS for MWAA

In order to get KMS configured, we need to do a number of things:

  • create new IAM policies to configure access to our KMS keys
  • create our KMS key
  • configure MWAA to use that KMS key

Once we have done that, we can then deploy our stack and create our MWAA environment. Lets get started.

Creating new IAM policies to configure access to our KMS keys

In the original CDK stack, we created IAM policies for the MWAA execution role which covered what was defined in the MWAA docs. In the stack, this was as follows:

{            "Effect": "Allow",            "Action": [                "kms:Decrypt",                "kms:DescribeKey",                "kms:GenerateDataKey*",                "kms:Encrypt"            ],            "Resource": "arn:aws:kms:{your-region}:{your-account-id}:key/{your-kms-cmk-id}",            "Condition": {                "StringLike": {                    "kms:ViaService": [                        "sqs.{your-region}.amazonaws.com",                        "s3.{your-region}.amazonaws.com"                    ]                }            }

As this stands, this should be ok for what we need. What we needed to do however, is create a new policy that will be used and assigned to our KMS key. This policy will provide the right permissions for MWAA and the workers to be able to access the keys to decrypt/encrypt messages and logs.

Within the MWAA CDK stack, we create a new policy as follows (and thanks to Michael for his help with this):

        kms_mwaa_policy_document = iam.PolicyDocument(            statements=[                iam.PolicyStatement(                    actions=[                         "kms:Create*",                         "kms:Describe*",                         "kms:Enable*",                         "kms:List*",                         "kms:Put*",                         "kms:Decrypt*"                    ],                    principals=[                        iam.AccountRootPrincipal(),                        # Optional:                        # iam.ArnPrincipal(f"arn:aws:sts::{self.account}:assumed-role/AWSReservedSSO_rest_of_SSO_account"),                    ],                    resources=["*"]),                iam.PolicyStatement(                    actions=[                        "kms:Decrypt*",                        "kms:Describe*",                        "kms:GenerateDataKey*",                        "kms:Encrypt*",                        "kms:ReEncrypt*",                        "kms:PutKeyPolicy"                    ],                    effect=iam.Effect.ALLOW,                    resources=["*"],                    principals=[iam.ServicePrincipal("logs.amazonaws.com",region=f"{self.region}")],                    conditions={"ArnLike": { "kms:EncryptionContext:aws:logs:arn": f"arn:aws:logs:{self.region}:{self.account}:*" }},                ),            ]        )

We will use this policy in the next step, when we create the key.

Note! When I originally put this together, I did not have the "principals=[iam.AccountRootPrincipal(),..." statement, and it did work for me. However, Michael saw the following error:

Resource handler returned message: "The new key policy will not allow you to update the key policy in the future. (Service: Kms, Status Code: 400

Which was resolved by adding that additional statement

Creating our KMS key

Now that we have our policy, we can create the key. The previous code is modified slightly to incorporate this new policy. (policy=kms_mwaa_policy_document)

        key = kms.Key(            self,            f"{mwaa_props['mwaa_env']}Key",            enable_key_rotation=True,            policy=kms_mwaa_policy_document        )        key.add_alias(f"alias/{mwaa_props['mwaa_env']}")

When the key is created, it will incorporate the policy we created which you can see from the KMS console.

Configuring MWAA to use that KMS key

Now that we have that done, we can now configure MWAA to use this. In the original CDK code, I incorrectly used key_id instead of key_arn. The correct CDK code is as follows:

        managed_airflow = mwaa.CfnEnvironment(            scope=self,            id='airflow-test-environment',            name=f"{mwaa_props['mwaa_env']}",            airflow_configuration_options={'core.default_timezone': 'utc'},            #airflow_version='1.10.12',            airflow_version='2.0.2',            dag_s3_path="dags",            environment_class='mw1.small',            execution_role_arn=mwaa_service_role.role_arn,            kms_key=key.key_arn,            logging_configuration=logging_configuration,            max_workers=5,            network_configuration=network_configuration,            #plugins_s3_object_version=None,            #plugins_s3_path=None,            #requirements_s3_object_version=None,            #requirements_s3_path=None,            source_bucket_arn=dags_bucket_arn,            webserver_access_mode='PUBLIC_ONLY',            #weekly_maintenance_window_start=None        )

You can view the complete code in the GitHub repo, blogpost-cdk-mwaa and the changeset here

Deploying MWAA

You can now deploy and create your MWAA environment in the same way, using the following CDK command, and responding Y when prompted.

cdk deploy MWAA-Backendcdk deploy MWAA-Environment

I have upgraded to v2. of CDK, so this was tested on 2.0.0 (build 4b6ce31)

Once deployed (it will take 25-30 minutes typically) you should now be able to access your MWAA environment as per usual. The only difference is that now all the interactions between MWAA and the worker nodes, the messages in the scheduler, the log files written to CloudWatch are all encrypted using that key.

Other observations and possible improvements

Whilst troubleshooting this, I used CloudWatch Logs and CloudWatch Trail to narrow down the source of the problem. CloudWatch Logs show med that there were some AccessDenied errors, and looking within CloudWatch Trail, I could dive deeper into those errors:

...    "eventTime": "2021-12-14T10:36:43Z",    "eventSource": "kms.amazonaws.com",    "eventName": "Decrypt",    "awsRegion": "eu-central-1",    "sourceIPAddress": "airflow.amazonaws.com",    "userAgent": "airflow.amazonaws.com",    "errorCode": "AccessDenied",    "errorMessage": "User: arn:aws:sts::704533066xxx:assumed-role/MWAA-Environment-V2-mwaaservicerole26CF19B9-1VOIDCIZRBxxx/AmazonMWAA-airflow is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-central-1:704533066374:key/cf448xxx-7b60-4db6-9b52-b8b00047xxx because no resource-based policy allows the kms:Decrypt action",    "requestParameters": null,...

One final note, I could improve this setup by encrypting the S3 bucket that is configured for MWAA using the same kms key. When I have some time, I will come back and revisit that.


Original Link: https://dev.to/aws/setting-up-mwaa-to-use-a-kms-key-4p6b

Share this article:    Share on Facebook
View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To