Write events directly to an Amberflo supplied and secured AWS S3 bucket

Amberflo provides you with an AWS S3 bucket with access rights and controls to write meters.

It automatically picks up the meters for processing as they arrive into the bucket. Failure reports are written back into the bucket.

There are many ways to write to S3. You can use AWS S3 SDK, AWS Glue, Logstash, Fluentd and other tools.

Please contact us to get the S3 bucket provisioned for your account.

Format

The meter records you send to the S3 bucket should be of the same standardized format as accepted by the ingest API. Here are some examples:

[{
     "customerId": "customer-123",
     "meterApiName": "ComputeHours",
     "meterValue": 5,
     "meterTimeInMillis": 1619445706909,
     "dimensions": {
        "region": "us-west-2",
        "az": "az1"
     }
}]

We also support NDJSON format (JSON separated by a newline)

{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 5, "meterTimeInMillis": 1619445706909 }
{ "customerId": "customer-321", "meterApiName": "ComputeHours", "meterValue": 4, "meterTimeInMillis": 1619445712341 }
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 1, "meterTimeInMillis": 1619445783456 }

Code Example

The S3 object key should have the date to allow easier troubleshooting (e.g.: /ingest/amberdata/06-07-2022/0000a2e4-e6ad-11ec-8293-6a8da1c4f9f0);

import json
from uuid import uuid1
from datetime import date

import boto3

records_to_send = [{
     'customerId': 'customer-123',
     'meterApiName': 'ComputeHours',
     'meterValue': 5,
     'meterTimeInMillis': 1619445706909,
     'dimensions': {
        'region': 'us-west-2',
        'az': 'az1'
     }
}]

bucket_name = '183-amberflo'

date = datetime.now().strftime(r'%m-%d-%Y')

object_key = 'ingest/amberdata/' + date + '/' + str(uuid1())

s3 = boto3.resource('s3')

s3.Object(bucket_name, object_key).put(Body=json.dumps(records_to_send), ACL='bucket-owner-full-control')

Troubleshooting

For troubleshooting issues with ingesting through S3, Amberflo will generate an S3 file with the failure reason in the following location:

s3://<bucket_name>/failed-requests/<date>/<original uri>.reason.txt

Compression (gzip support)

You can write compressed files in gzip format with .gz extensions. Amberflo will decompress while ingesting if the file extension is ".gz" .