Ingest Meters from MongoDB

You can use MongoDB as the metering data source. If your system is currently using MongoDB, you can set up Amberflo side-by-side to get metering data into Amberflo without impacting the current system.

We recommend the following alternative methods to ingest data from MongoDB.

Using Logstash

Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice.
To this end, we can take advantage of buffering, inputting, outputting, and filtering abilities from Logstash by adding a MongoDB input and S3 output plugin.

You can use either the JDBC input plugin or the MongoDB input plugin and write to the S3 folder provisioned by Amberflo.
https://amberflo.readme.io/docs/logstash
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html
https://github.com/phutchins/logstash-input-mongodb

Query MongoDB using code

You can query MongoDB with its SDK and use the Amberflo SDK to ingest the data. For example, in Python you can use PyMongo to query MongoDB and ingest the result using the Amberflo Python SDK.
For getting only new data, you will persist the last timestamp/ID used. Amberflo handles deduplication.

Use MongoDB's Change data stream

With the Amberflo SDK, you can use the MongoDB real-time data stream to act upon every insert:
https://developer.mongodb.com/quickstart/python-change-streams/