Setup KOPS on AWS with GruCloud

The aim of this tutorial is to automatically create and destroy the AWS resources required by kops, a tool to create a Kubernetes cluster.

The section 'setup your environment' from the official kops documentation will be automated with GruCloud

Below is the diagram generated from the target code, it illustrates the resources and their association with each other:

Regarding this DNS scenario, the case of a subdomain where a top-level hosted zone already exists is implemented.

TD;DR

1. Get this example code and install the dependencies.
2. Edit the configuration file and set the domain name, the subdomain name, the region, and the zone.
3. gc apply

All the AWS resources required by kops should have been created. The environment file kops.env containing the necessary information should have been generated too.

You are now ready to create a cluster with kops,

Here are a few npm scripts wrapper: npm run kops:create, npm run kops:update and npm run kops:validate.

Steps

Here is a description of the steps that are automated:

IAM

• create a kops group, attach 5 IAM policies.
• create a kops user, attach the user to the kops group.
• create access and secret key for the kops user.

Route53

• create a hosted zone for a subdomain.
• create a DNS record of type NS in the top-level hosted zone with the DNS servers as values from the subdomain hosted zone.

S3

• create an S3 bucket with encryption and versioning.

kops.env file

• create a file containing the environment variable for kops

You will be free from performing all these commands manually. The same applies to the destruction of all these resources.

Requirements

• Access to the AWS console
• AWS CLI configured
• A domain name registered on Route53.
• Node.js
• GruCloud CLI

Install

Clone this code, change to the kops folder, install the npm dependencies:

git clone https://github.com/grucloud/grucloudcd grucloud/examples/aws/kopsnpm install

Configuration

Edit config.js and set the domainName, the subDomainName, the zone and the region:

Double check your configuration with gc info:

  - provider:      name: aws      type: aws    stage: dev    config:      projectName: @grucloud/create-aws-kops      kops:        domainName: grucloud.org        subDomainName: kops.example.grucloud.org        groupName: kops        userName: kops      stage: dev      zone: us-east-1a      accountId: 4444454555555      region: us-east-1

The domain name must be registered with Route53 for the current AWS user.
Let's also verify that a top level hosted zone already exists.
You could use the gc list command with the Route53Domain and HostedZone type filter:

gc list  -t Route53Domain -t HostedZone

[...Truncated] aws                                                                                          Route53Domain       grucloud.org                                                            HostedZone          grucloud.org.                                                          

iac.js

For your information, the architecture is described in iac.js.
In this use, the cloud provider is AWS, so we'll use the GruCloud AWS Provider to create the resources.

Target Graph

Another way to explore the iac.js is to generate a diagram of the target resources:

gc graph

Deploying

Ready to deploy the user, group, s3 bucket, route53 hosted zone and record ?

gc apply

The AWS resources should have been deployed.
Let's find out our live resources as well as a diagram showing the dependencies between these resources:

gc list --graph --our

 aws                                                                                          IamPolicy           AmazonEC2FullAccess                                                                         AmazonRoute53FullAccess                                                                     AmazonS3FullAccess                                                                          IAMFullAccess                                                                               AmazonVPCFullAccess                                                     IamGroup            kops                                                                    S3Bucket            kops.example.grucloud.org                                               IamUser             kops                                                                    HostedZone          kops.example.grucloud.org.                                                                  grucloud.org.                                                           Route53Record       kops.example.grucloud.org-ns                                           11 resources, 15 types, 1 provider

Envirornment variables

At the end of the deployment, the environment file kops.env is generated with the variables required by kops:

# kops.envexport AWS_ACCESS_KEY_ID=XXXXXXNBM2ZQEPXXXXXexport AWS_SECRET_ACCESS_KEY=XXXXXiXmSB3aZTK/AxOOvSPcGby3XXXXXXexport NAME=kops.example.grucloud.orgexport KOPS_STATE_STORE=s3://kops.example.grucloud.orgexport REGION=eu-west-2export ZONE=eu-west-2a

Source with variables with:

source kops.env

When the deploment is destroyed with gc destroy, kops.env is removed.

The file hook.js is the place where this logic is implemented.

NPM kops scripts

The following npm scripts manage the kops commands, the environment variables are sourced from kops.env.

npm run kops:createnpm run kops:updatenpm run kops:validate

List Resources

Let's fetch all the live resources, we'll see that kops creates many resources such as autoscaling groups, ec2 instances, subnets, vpc, internet gateway, volumes, key pair and so on:

gc list --graph --all --default-exclude --types-exclude Certificate --types-exclude Route53Domain --types-exclude NetworkInterface

You could inspect and generate a diagram of any existing AWS infrastruture for the most used resources

Destroy

To destroy the resources created by GruCloud, use the destroy command.

Ensure the cluster is destroyed before.

npm run kops:destroygc destroy

Alternatively, gc could also destroy all the resources created by kops, use the all flag:

gc destroy --all

Further Step

Congratulations, you know how to create and destroy a Kubernetes cluster with kops.
What about a load balancer, DNS records, SSL certificates ? Grucloud provides some ready made modules distributed with npm, the node package manager.

Have a look at:

• @grucloud/module-aws-certificate: Create a certificate and a Route53 record for validation.
• @grucloud/module-aws-load-balancer: Manage a load balancer, target groups, listeners and rules. A leaner alternative the AWS Load Balancer Controller which runs on the cluster.
• @grucloud/module-aws-vpc: Contains the base resources required to create a Kubernetes cluster.

On the Kubernetes side, be aware of the GruCloud Kubernetes Provider. In a nutshell, instead of writing YAML manifest, Javascript is used instead to define the manifests, no more templating engine, enjoy a real programming language instead.

Would you like to deploy a full stack application on EKS ? Choose the flavour depending on who is reponsible to create the load balancer, target groups, listener and rules:

• Load balancer resources created inside the cluster with the AWS Load Balancer Controller: eks-lbc.

• A leaner solution where the load balancer resources are created by GruCLoud outside the cluster: eks-lean.

Links

• GitHub
• Documentation
• Website
• Twitter