Visualizing Amazon S3 Storage Usage with Grafana and AWS SDK for Python (Boto3)

Visualizing Amazon S3 Storage Usage with Grafana and AWS SDK for Python (Boto3)

Monitoring storage usage in Amazon S3 buckets is essential for managing costs and optimizing storage. This blog post will guide you through the process of visualizing S3 storage usage with Grafana using AWS SDK for Python (Boto3).

Prerequisites:

1. An Amazon Web Services (AWS) account with access to Amazon S3
2. A Grafana instance with the Simple JSON datasource plugin installed
3. Python 3.x and pip installed
4. Familiarity with AWS SDK for Python (Boto3)

Steps to Visualize S3 Storage Usage with Grafana:

1. Install AWS SDK for Python (Boto3) and Dependencies

First, install Boto3 and its dependencies using pip.

pip install boto3

2. Create a Python Script to Retrieve S3 Storage Usage Data

Next, create a Python script that uses Boto3 to retrieve S3 storage usage data and outputs it in a format compatible with Grafana's Simple JSON datasource plugin.

Example using Python and Boto3:

import boto3import jsonimport datetime# Replace 'your-bucket-name' with the name of your S3 bucketBUCKET_NAME = 'your-bucket-name'# Initialize the Boto3 S3 clients3_client = boto3.client('s3')# Get the S3 bucket size and object countresponse = s3_client.list_objects_v2(Bucket=BUCKET_NAME)bucket_size = sum([obj['Size'] for obj in response.get('Contents', [])])object_count = len(response.get('Contents', []))# Format the data for Grafana's Simple JSON datasource plugindata = {    "time": int(datetime.datetime.now().timestamp()),    "tags": {"bucket": BUCKET_NAME},    "values": [        {"name": "bucket_size", "value": bucket_size},        {"name": "object_count", "value": object_count}    ]}# Print the data in JSON formatprint(json.dumps(data))

3. Configure Grafana to Use the Python Script as a Datasource

In Grafana, add a new datasource using the Simple JSON plugin. Configure the datasource to run the Python script and fetch the S3 storage usage data.

Example using Grafana:

a. Add a new datasource:

• Click the Grafana logo in the top left corner
• Select "Configuration" from the side menu
• Click "Data sources"
• Click the "Add data source" button
• Select "Simple JSON" from the list of plugins

b. Configure the datasource:

• Name: S3 Storage Usage
• Type: Simple JSON
• URL: leave empty
• Access: Server
• JSON Data: leave empty
• HTTP Method: POST
• Basic Auth: leave empty
• With Credentials: leave unchecked
• Time Field: time
• Time Format: unix
• Timezone: UTC
• Fields:
  • Name: bucket_sizeType: Gauge
  • Name: object_countType: Gauge

c. Set up script execution:

• Enable "Include time in JSON output"
• Command: python /path/to/your/script.py
• Command options: leave empty
• Timeout: 30s
• Interval: 60s

1. Create a Grafana Dashboard to Visualize S3 Storage Usage

Finally, create a new Grafana dashboard to visualize the S3 storage usage data. Add panels for the bucket size and object count, and configure the visualizations as desired.

Example using Grafana:

a. Create a new dashboard:

• Click the "Create" button in the top menu
• Select "Dashboard"

b. Add a panel for the bucket size:

• Click the "Add panel" button
• Select "Graph"
• Configure the panel:
  • Title: Bucket Size
  • Query:
  • Metric: bucket_size
  • Stat: Current
  • Format: Bytes

c. Add a panel for the object count:

• Click the "Add panel" button
• Select "Graph"
• Configure the panel:
  • Title: Object Count
  • Query:
  • Metric: object_count
  • Stat: Current
  • Format: Short

Conclusion:

Visualizing Amazon S3 storage usage with Grafana is an effective way to monitor and manage storage costs. By following the steps outlined in this blog post, you can create a Python script to retrieve S3 storage usage data using Boto3, configure Grafana to use the script as a datasource, and create a dashboard to visualize the data. This will enable you to easily monitor and optimize your S3 storage usage.