An Interest In:
Web News this Week
- April 24, 2024
- April 23, 2024
- April 22, 2024
- April 21, 2024
- April 20, 2024
- April 19, 2024
- April 18, 2024
Some of Our Sources
- BoingBoing
- Technology Review
- Simplebits
- Just Creative
- Smashing Magazine
- Smashing Apps
- TutsPlus - Design
- Fuel Your Creativity
- Web Designer Depot
- Fudge Graphics
Help Webnuz
Referal links:
Getting mwaa-local-runner up on AWS Cloud9
Here is a quick recipe if you are looking to get mwaa-local-runner up and running on your Cloud9 developer setup. This might not be the most optimised way, so I am very happy to received suggestions on how to improve this. What I will cover here is how to deploy mwaa-local-runner onto a standard Cloud9 IDE, deployed in a default VPC.
Updating my AWS Cloud9 environment
The first thing I needed to do was to increase the size of my local disk as Cloud9 only provides 10gb of storage. This is fine for typical use cases, but we are going to be building container images, so need to set this higher.
pip3 install --user --upgrade boto3export instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)python -c "import boto3import osfrom botocore.exceptions import ClientError ec2 = boto3.client('ec2')volume_info = ec2.describe_volumes( Filters=[ { 'Name': 'attachment.instance-id', 'Values': [ os.getenv('instance_id') ] } ])volume_id = volume_info['Volumes'][0]['VolumeId']try: resize = ec2.modify_volume( VolumeId=volume_id, Size=30 ) print(resize)except ClientError as e: if e.response['Error']['Code'] == 'InvalidParameterValue': print('ERROR MESSAGE: {}'.format(e))"if [ $? -eq 0 ]; then sudo rebootfiInstalling some missing tools
The next thing I need to do is install "docker-compose" as whilst Docker is installed, docker-compose is not. This is the script I use
wget https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)sudo mv docker-compose-$(uname -s)-$(uname -m) /usr/bin/docker-composechmod +x /usr/bin/docker-composeAnd if this is successful, you should now have it available via the command line
ops:~/environment $ docker-compose versionDocker Compose version v2.17.2Grabbing my AWS Cloud9 public IP address
This should be everything you need to get started. However, we do need one piece of information before we proceed, and that is the public IP address for your Cloud9 instance. We need this as we are going to update the URL configuration parameter in mwaa-local-runner. (If you do not do this, you will get 400 Gateway errors when accessing Airflow via the UI). Run the following command to find yours
curl http://169.254.169.254/latest/meta-data/public-ipv43.253.9.131Configuring and building mwaa-local-runner
You can now check out the mwaa-local-runner project into your Cloud9 environment
git clone https://github.com/aws/aws-mwaa-local-runner.gitWe now need to change a couple of things.
The first is to update the airflow.cfg (docker/config/airflow.cfg) so that we can update the base_url parameter (line 406) with the public IP of your Cloud9 instance.
#base_url = http://localhost:8080base_url = https://3.253.9.131:8080The next change, still in the airflow.cfg, to update how quickly it picks up DAGs in the DAGs folder. Line 757 is changed
#dag_dir_list_interval = 300dag_dir_list_interval = 5You can now save this file, and you are ready to build your mwaa-local-runner image. You do this using the following command (and this works against any version of Apache Airflow that mwaa supports)
mwaa-local-env build-imageThis will now take around 10 minutes to complete the build process. Go grab your favourite drink and then come back. Hopefully, you should see something like:
[+] Building 543.8s (27/27) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 43B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/amazonlinux:2 1.6s => [auth] library/amazonlinux:pull token for registry-1.docker.io 0.0s => CACHED [ 1/21] FROM docker.io/library/amazonlinux:2@sha256:ad9b9135cd4eaa9f5a7b903b1cd72fef23027260c9aa84a743d090b64ce3609e 0.0s => [internal] load build context 0.0s => => transferring context: 5.46kB 0.0s => [ 2/21] COPY script/bootstrap.sh /bootstrap.sh 0.1s => [ 3/21] COPY script/systemlibs.sh /systemlibs.sh 0.0s => [ 4/21] COPY script/generate_key.sh /generate_key.sh 0.0s => [ 5/21] COPY script/run-startup.sh /run-startup.sh 0.0s => [ 6/21] COPY script/shell-launch-script.sh /shell-launch-script.sh 0.0s => [ 7/21] COPY script/verification.sh /verification.sh 0.0s => [ 8/21] COPY config/constraints.txt /constraints.txt 0.0s => [ 9/21] COPY config/mwaa-base-providers-requirements.txt /mwaa-base-providers-requirements.txt 0.0s => [10/21] RUN chmod u+x /systemlibs.sh && /systemlibs.sh 133.1s => [11/21] RUN chmod u+x /bootstrap.sh && /bootstrap.sh 343.3s => [12/21] RUN chmod u+x /generate_key.sh && /generate_key.sh 0.6s => [13/21] RUN chmod u+x /run-startup.sh 0.3s => [14/21] RUN chmod u+x /shell-launch-script.sh 0.3s => [15/21] RUN chmod u+x /verification.sh 0.5s => [16/21] COPY script/entrypoint.sh /entrypoint.sh 0.0s => [17/21] COPY config/airflow.cfg /usr/local/airflow/airflow.cfg 0.0s => [18/21] COPY config/webserver_config.py /usr/local/airflow/webserver_config.py 0.0s => [19/21] RUN chown -R airflow: /usr/local/airflow 31.4s => [20/21] RUN chmod +x /entrypoint.sh 0.4s => [21/21] WORKDIR /usr/local/airflow 0.1s => exporting to image 31.7s => => exporting layers 31.7s => => writing image sha256:54d466e5db131a71b694018f94ab69653794548551c37b05d172f5070f4deda8 0.0s => => naming to docker.io/amazon/mwaa-local:2_4 0.0sUse 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix themCongratulations, the hard part has been done.
Note If you see the following error:
16 188.4 + sudo -u airflow pip3 install --no-use-pep517 --constraint /constraints.txt poetry16 188.8 16 188.8 Usage: 16 188.8 pip3 install [options] <requirement specifier> [package-index-options] ...16 188.8 pip3 install [options] -r <requirements file> [package-index-options] ...16 188.8 pip3 install [options] [-e] <vcs project url> ...16 188.8 pip3 install [options] [-e] <local project path> ...16 188.8 pip3 install [options] <archive url/path> ...16 188.8 16 188.8 --no-use-pep517 error: It is not possible to use --no-use-pep51716 188.8 without setuptools and wheel installed.executor failed running [/bin/sh -c chmod u+x /bootstrap.sh && /bootstrap.sh]: exit code: 2You will need to update the "bootstrap.sh" script, and update line 43 so that it looks like:
pip3 install $PIP_OPTION --upgrade 'pip<23'This is to resolve an issue with pip version 23.1
To start mwaa-local-runner, all you need to do now is:
mwaa-local-env startAnd Apache Airflow will start to boot. It will take 2-3 minutes, but once ready you will see something like
aws-mwaa-local-runner-2_4-local-runner-1 | [2023-04-17 10:09:29 +0000] [202] [INFO] Starting gunicorn 20.1.0aws-mwaa-local-runner-2_4-local-runner-1 | [2023-04-17 10:09:29 +0000] [202] [INFO] Listening at: http://0.0.0.0:8080 (202)And you can now open a browser, pointing to your public Ip address and access Apache Airflow on port 8080.
Conclusion
This short post showed you how you can get mwaa-local-runner up and running on your AWS Cloud9 instances. I have been experimenting using this, and really like that I can decouple this from my local setup. I am still working on tweaking the setup so that I can locadown public access to the Cloud9 instance to just my own IP address, something I would recommend you do to.
If you have found this blog post helpful, please give me some feedback by completing this very short survey here
Original Link: https://dev.to/aws/getting-mwaa-local-runner-up-on-aws-cloud9-1nhd
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To