DynamoDB Streams

While working on a backend service, I came across this problem where I had to retrieve a set of data of a specific type, but that type wasn't having any index and the data was also updated frequently.

Since the data was constantly changing (add/update), first thing that came to my mind was to have a trigger but then as I looked at the documentation of how to add triggers to Dynamo in AWS, I decided to pursue a different solution :p

So, there were three easy solutions that were present before me without thinking much -

Complete scan of the table and filter the results
This solution was screaming "slow" right from the moment it entered my mind.

Add an Index for that type

There were couple of reasons I didn't go along with this idea

• That wasn't a very frequently queried data
• Already had so many indexes so didn't want to add one more just yet before trying other solutions

Run a cron-job
A cronjob that will scan the table and store the result in a cache or in another table.

Great man once said, If you can't think of anything else, run a cron-job.

Back in my mind, I wasn't quite convinced with any of the above three solutions and so decided to give dynamodb triggers try. I was aware of DynamoDb streams but never tried them.

DynamoDB Streams are basically triggers like that we have in Relational DBs but the only difference is that it generates stream of events when there is a data change and you can easily pipe that stream into Kinesis or a Lambda (In my case it was lambda).

AWS made the things look difficult but the process was not complex at all.

All I had to do was the following -

• Setup a stream on the dynamodb table which was just a one click action.
• Create a new lambda function and attach it to the dynamodb stream which was also fairly easy
• Rest was basic stuff of listening the data then processing it and finally saving it to the new table

DynamoDB streams are much more than what I have covered in my use case. You can read more about it here : -https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/