AWS S3

5 min

Write events directly to an Amberflo supplied and secured AWS S3 bucket

Amberflo provisions an AWS S3 bucket for your account with the necessary access rights and permissions to write meter data.

Meter files are automatically picked up for processing as soon as they arrive in the bucket.

If processing fails, error reports are written back into the same bucket for review.

There are several ways to write data to S3, including:

AWS S3 SDK

AWS Glue

Logstash

Fluentd

And other ingestion tools

To get your S3 bucket provisioned, please contact us.

Format

The meter records you send to the S3 bucket should be of the same standardized format as accepted by the ingest meter API Here are some examples:

[{
     "customerId": "customer-123",
     "meterApiName": "ComputeHours",
     "meterValue": 5,
     "meterTimeInMillis": 1619445706909,
     "dimensions": {
        "region": "us-west-2",
        "az": "az1"
     }
}]

We also support NDJSON format (JSON separated by a newline)

{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 5, "meterTimeInMillis": 1619445706909 }
{ "customerId": "customer-321", "meterApiName": "ComputeHours", "meterValue": 4, "meterTimeInMillis": 1619445712341 }
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 1, "meterTimeInMillis": 1619445783456 }

Code Example

The S3 object key should have the date to allow easier troubleshooting (e.g.: /ingest/amberdata/06-07-2022/0000a2e4-e6ad-11ec-8293-6a8da1c4f9f0);

import json
from uuid import uuid1
from datetime import date

import boto3

records_to_send = [{
     'customerId': 'customer-123',
     'meterApiName': 'ComputeHours',
     'meterValue': 5,
     'meterTimeInMillis': 1619445706909,
     'dimensions': {
        'region': 'us-west-2',
        'az': 'az1'
     }
}]

bucket_name = '183-amberflo'

date = datetime.now().strftime(r'%m-%d-%Y')

object_key = 'ingest/amberdata/' + date + '/' + str(uuid1())

s3 = boto3.resource('s3')

s3.Object(bucket_name, object_key).put(Body=json.dumps(records_to_send), ACL='bucket-owner-full-control')

Troubleshooting

If you encounter issues while ingesting data through S3, Amberflo will generate a failure report in the following S3 path:

s3://<bucket_name>/failed-requests/<date>/<original uri>.reason.txt

This file contains the reason for the failure and can help you diagnose and resolve the issue.

Compression (gzip support)

Amberflo supports ingesting gzip-compressed files.

Simply upload files with the .gz extension

Amberflo will automatically decompress these files during ingestion