Metering Design Principles

One of the core design principles of a Cloud Metering Service Platform is guaranteed accuracy.

The metering system needs to be built from the ground up with the core design principles of accuracy, idempotency, and data deduplication.

1. Accuracy

The data must be completely accurate to allow the metering system to be the single source of truth for metering and consumption data. This means other systems can depend on the accuracy and completeness of the metering data in Amberflo for mission-critical functions like invoicing and billing.

2. Idempotency

A record is processed once, and once only.

That is, it cannot go unprocessed, it must be processed. It cannot be processed twice (or more than once). It must be processed once, and once only.

Example: If a valid record is ingested and fails mid-stream through processing, it still needs to be accounted for correctly (counted and processed once, and once only. No drops and no double counting). Amberflo Cloud Metering Service Platform provides you this guarantee out of the box.

3. Data deduplication

Data deduplication is a technique for eliminating duplicate copies of repeated data. This can be a common occurrence, depending on the source generating the meters, and it is challenging to guarantee deduplication if the processing system (Cloud Metering Service Platform) itself is a highly-distributed stateless system. In a distributed system, any part of the system or node can fail at any time, and if it fails mid-stream as data is undergoing transformation or processing, it makes it very difficult to "guarantee" data de-duplication.

In cloud metering, it is common for the source (client generating the meters) to send duplicate meters. Because the cloud metering service serves as the system of record and the single source of truth for usage and consumption data, duplicate records must be deduped.

Data deduplication is a core platform tenet to deliver accurate usage data to downstream systems such as pricing, billing, and others.

Amberflo Cloud Metering Service Platform provides you with an out-of-the-box data deduplication guarantee. Here are some different ways how you can enforce data deduplication using Amberflo:

The Amberflo.io platform will not store duplicate data for the same meter record. If you call the ingest API with the same record, the meter repository will only hold the first record and discard any subsequent records. The deduplication key is based on all the meter attributes. That means if one of the meter record attributes is different (either the record time or any dimension value) we will consider it as a different record.

Unique ID: if the source system generated a unique ID for this records we can use that for the dedup key

dimensions_with_unique_id.append({'Name': 'unique_id', 'Value': str(uuid4())}) 
metering.meter(options.tenant, options.meter_name,int(options.meter_value), dimensions=dimensions_with_unique_id)

Unique timestamp: if we want to make sure we use one meter for the current hour/minute/second, we can set the timestamp of the request

metering.meter(options.tenant, options.meter_name,int(options.meter_value),  dimensions=dimensions,timestamp=str(int(round(time.time() * 1000))))