AWS S3

5min
Write events directly to an Amberflo supplied and secured AWS S3 bucket
Amberflo provides you with an AWS S3 bucket with access rights and controls to write meters.
It automatically picks up the meters for processing as they arrive into the bucket. Failure reports are written back into the bucket.
There are many ways to write to S3. You can use AWS S3 SDK, AWS Glue, Logstash, Fluentd, and other tools.
Please contact us to get the S3 bucket provisioned for your account.
Format
The meter records you send to the S3 bucket should be of the same standardized format as accepted by the ingest API﻿. Here are some examples:
JSON
[{
     "customerId": "customer-123",
     "meterApiName": "ComputeHours",
     "meterValue": 5,
     "meterTimeInMillis": 1619445706909,
     "dimensions": {
        "region": "us-west-2",
        "az": "az1"
     }
}]
[{
     "customerId": "customer-123",
     "meterApiName": "ComputeHours",
     "meterValue": 5,
     "meterTimeInMillis": 1619445706909,
     "dimensions": {
        "region": "us-west-2",
        "az": "az1"
     }
}]
﻿
We also support NDJSON format (JSON separated by a newline)
JSON
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 5, "meterTimeInMillis": 1619445706909 }
{ "customerId": "customer-321", "meterApiName": "ComputeHours", "meterValue": 4, "meterTimeInMillis": 1619445712341 }
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 1, "meterTimeInMillis": 1619445783456 }
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 5, "meterTimeInMillis": 1619445706909 }
{ "customerId": "customer-321", "meterApiName": "ComputeHours", "meterValue": 4, "meterTimeInMillis": 1619445712341 }
{ "customerId": "customer-123", "meterApiName": "ComputeHours", "meterValue": 1, "meterTimeInMillis": 1619445783456 }
﻿
Code Example
The S3 object key should have the date to allow easier troubleshooting (e.g.: /ingest/amberdata/06-07-2022/0000a2e4-e6ad-11ec-8293-6a8da1c4f9f0);
Python
import json
from uuid import uuid1
from datetime import date

import boto3

records_to_send = [{
     'customerId': 'customer-123',
     'meterApiName': 'ComputeHours',
     'meterValue': 5,
     'meterTimeInMillis': 1619445706909,
     'dimensions': {
        'region': 'us-west-2',
        'az': 'az1'
     }
}]

bucket_name = '183-amberflo'

date = datetime.now().strftime(r'%m-%d-%Y')

object_key = 'ingest/amberdata/' + date + '/' + str(uuid1())

s3 = boto3.resource('s3')

s3.Object(bucket_name, object_key).put(Body=json.dumps(records_to_send), ACL='bucket-owner-full-control')
import json
from uuid import uuid1
from datetime import date

import boto3

records_to_send = [{
     'customerId': 'customer-123',
     'meterApiName': 'ComputeHours',
     'meterValue': 5,
     'meterTimeInMillis': 1619445706909,
     'dimensions': {
        'region': 'us-west-2',
        'az': 'az1'
     }
}]

bucket_name = '183-amberflo'

date = datetime.now().strftime(r'%m-%d-%Y')

object_key = 'ingest/amberdata/' + date + '/' + str(uuid1())

s3 = boto3.resource('s3')

s3.Object(bucket_name, object_key).put(Body=json.dumps(records_to_send), ACL='bucket-owner-full-control')
﻿
Troubleshooting
For troubleshooting issues with ingesting through S3, Amberflo will generate an S3 file with the failure reason in the following location:
s3://<bucket_name>/failed-requests/<date>/<original uri>.reason.txt
Compression (gzip support)
You can write compressed files in gzip format with .gz extensions. Amberflo will decompress while ingesting if the file extension is ".gz" .
Updated 05 Nov 2024
Did this page help you?
Ingestion Options
AWS CloudWatch