MongoDB

ingest meters from mongodb amberflo supports metering data ingestion from mongodb , allowing you to set up metering alongside your current system without disruption if your system already uses mongodb, you can configure amberflo to ingest data without impacting existing functionality recommended methods for mongodb integration we recommend the following approaches to ingest data from mongodb into amberflo 1 using logstash logstash is an open source data collection engine with real time data pipelining capabilities it can unify data from different sources , normalize it, and forward it to your target systems by using logstash’s input, output, filtering, and buffering capabilities, you can reliably move data from mongodb to amberflo plugin setup use either the mongodb input plugin or the jdbc input plugin configure the s3 output plugin to write to the amberflo provided s3 bucket once data is in the s3 bucket, amberflo will automatically pick it up for ingestion https //www elastic co/guide/en/logstash/current/plugins inputs jdbc html https //github com/phutchins/logstash input mongodb 2 query mongodb using code you can also query mongodb programmatically using its sdk and then use the amberflo sdk to ingest the retrieved data example (python) use pymongo to query data from mongodb use the amberflo python sdk to ingest the results as meter records this approach gives you full control over filtering and transforming data before ingestion handling new data only , by persisting the last processed timestamp or id amberflo automatically performs deduplication , so repeated ingestion of the same data will not create duplicates 3 use mongodb's change data stream with the amberflo sdk, you can use the mongodb real time data stream to act upon every insert https //developer mongodb com/quickstart/python change streams/ https //docs amberflo io/docs/python library set up virtual environment and install sdk \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# shut down ingest api client ingest api shutdown() \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# shut down ingest api client ingest api shutdown() \# setup usage api client and get a usage report from metering usage import (aggregationtype, take, timegroupinginterval, timerange, usageapiclient, create usage query) usage api = usageapiclient(os environ get("api key")) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# shut down ingest api client ingest api shutdown() \# setup usage api client and get a usage report from metering usage import (aggregationtype, take, timegroupinginterval, timerange, usageapiclient, create usage query) usage api = usageapiclient(os environ get("api key")) since two days ago = timerange(int(time()) 60 60 24 2) query = create usage query( meter api name="api calls", aggregation=aggregationtype sum, time grouping interval=timegroupinginterval day, time range=since two days ago, group by=\["customerid"], usage filter={"customerid" \["customer 123"]}, take=take(limit=10, is ascending=false), ) report = usage api get(query)