AWS State Machines Decouple and Orchestrate Lambda Functions

I recently wrote an article about AWS Lambda. My article served as a quick introduction to the service and its functionalities. In this article, we explore another great asset in your AWS serverless arsenal: Step Functions, and specifically State Machines.

AWS decided to give us access to their CloudWatch Events API in July 2019 with the rebranding of CloudWatch events to EventBridge that is their serverless event bus solution. In conjunction with Step Functions, it is pretty transformational.

Step Functions allow you to orchestrate and decouple Lambda functions with State Machines. The flow of sequential or parallel workloads can be defined in States to build event-driven decoupled architectures for your processes. Suitable workloads for State Machines are microservices, IT/DevOps automation, ETL, business processes, data streaming, and more.

In State Machines, every step of your workflow is represented by a State. Every State has an input and an output. The input of a State is the output of the previous State. You can specify a custom input to the first State when you start a new execution. Inputs and Outputs must be JSON payloads.

A great benefit of State Machines is Observability. With Workflow Studio, you get to visually design your processes by dragging Actions blocks for your States and Flow blocks.

Example of Actions blocks:

• Invoke Lambda Function
• Public to an SNS topic
• Run a task in ECS
• Run a Job in Glue
• Integration with many more AWS Services

Example of Flow blocks:

• Choice block for if-then-else logic
• Parallel block for concurrency
• Map block for for-each logic and dynamic parallelism
• Success block to handle success
• Fail block to handle errors

Of course, actions and flows can also be handled traditionally with code and be executed in your workloads with Lambda invocations.

As a simple example, lets build a State Machine that finds all running EC2 instances in the first three AWS Regions (ordered by the describe_regions API Response).

To decouple and parallelize our workload we can split the logic in multiple steps and two Lambda functions:

• The first Lambda function will retrieve and return the first three AWS Regions. The AWS Regions will be the first States output Payload.
• Next, we will get all running instances per region by invoking a lambda function for each of the first three AWS Regions. The execution will be parallelized as soon as the first Lambda function is done executing, retrieving the first three regions. For every Lambda invocation, we will filter the previous states output payload by region accordingly.All of my Lambda functions will be in Python. Note that we could easily use a Map flow block, in this case, to dynamically iterate over the regions, but for the sake of visualizing what is happening, we will use three parallel branches that we will manually instantiate.

Lets build the first Lambda function: retrieve_regions.

import jsonimport boto3def describe_regions():    regions = []ec2_client = boto3.client(        "ec2",        "us-east-1"        )    response = ec2_client.describe_regions()for r in response['Regions'][:3]:        regions.append(r['RegionName'])    return regionsdef lambda_handler(event, context):    regions = describe_regions()    response = {}    i = 1    for r in regions:        index = "Region" + str(i)        response[index] = {}        response[index]['Region'] = r        i+=1    return response

This function will return a nested dictionary with the first three regions of AWS:

{  "Region1": {    "Region": "eu-north-1"  },  "Region2": {    "Region": "ap-south-1"  },  "Region3": {    "Region": "eu-west-3"  }}

Next, lets create our State Machine and map the first State to Invoke our retrieve_regions Lambda function in the Workflow Studio:

When we execute the State Machine, we can visualize the first States input and output. As mentioned before, the output payload of a State will be the input of the next State:

The output payload can be accessible by another lambda function by using the event parameter in the lambda handler. You can control which portion of the payload dictionary gets passed to the next state by using the InputPath filter in the next States configuration (more on that later). Note that you have a hard-limit service quota on the size of the asynchronous payload in Lambda.

Now, lets create the second regional Lambda function get_running_instances:

import jsonimport boto3def get_running_instances(region):    ec2_client = boto3.client(        "ec2",        region        )    instances = ec2_client.describe_instances()    ids = []    for instance in instances['Reservations']:        for i in instance['Instances']:            ids.append(i['InstanceId'])    return idsdef lambda_handler(event, context):    region = event["Region"]    instances = get_running_instances(region)    return instances

Modifying our State Machine:

Modifying the input payload for region 1 by setting InputPath to $.Region1 to filter the first region:

Do the same for region 2, and region 3. Set InputPath to $.Region2 for GET RUNNING INSTANCE REGION 2 and InputPath to $.Region3 for GET RUNNING INSTANCES REGION 3.

When adding a Lambda invocation to your State Machine, you should also modify the execution role and add the new Lambda function to that role.

For the purpose of this exercise, I spun up an EC2 Instance in each of the first 3 regions. Testing the execution of our State Machine:

I hope that this example served as a good introduction to building decoupled workflows in State Machines and that you can apply what was showcased to design your own mission-critical business processes or other applications!