Prevent Data Feed Ingestion Errors with IFIF

Intelligent Feed Ingestion Framework uses machine learning to avoid long delays and reduce the time to get data into critical business systems.

Get Information

Intelligent Feed Integrator

The Intelligent Feed Integrator is configured to be aware of existing data feeds, their type and structure. Feed types and structures are stored in a feed dictionary maintained in the Metadata repository. Feeds are defined by their source and name, format (CSV, JSON, XML, Etc.) and their structure (schema, column layout, etc.).The IFI is responsible for reading feed content as it arrives and requesting evaluation by the Intelligent Feed Evaluator.

Contact us

Intelligent Feed Machine Learning Module 

The Intelligent Feed Machine Learning Module evaluates feeds to establish thresholds, ranges, and content features for each of the data elements that comprise a feed. The IFMLM is trained with representative good data for each feed to establish the baseline criteria against which future instances of the same feed will be evaluated. The IFMLM can be trained with as little as a single feed instance and supports an iterative workflow that allows subsequent valid instances to be introduced as new training data to the IFMLM. This allows for a rapid deployment and usage of the system, while simultaneously ensuring that the system leverages all available information to improve over time 

Intelligent Feed Evaluator

The Intelligent Feed Evaluator is configured to accept feed evaluation requests from the IFI. In response to a request, the IFE inspects the data for the requested feed applying thresholds, ranges and models to each property contained in the feed. Upon completion of feed inspection, results are written to the Evaluation Results repository and can be reviewed and remediated by data managers and other interested users.

 

Intelligent Feed User Interface

The Intelligent Feed User Interface provides a smooth and intuitive user experience for accomplishing the following workflows:

  • Identifying and Reviewing feeds that contain content that violates a learned threshold, exists outside a learned range, or fails to conform to a discovered model.
  • Promoting passing feeds or remediated feeds to training data status for reincorporating into the learning algorithm that drives the dynamic thresholds, ranges, and content features.
  • Curating learned models, thresholds and ranges where domain expertise needs to override discovered knowledge

Evaluation Result and Metadata Repository

This repository maintains several key data sets that are required for platform operation:

  • Feed dictionary
  • Feed metadata (types and structures)
  • Learned thresholds
  • Learned ranges
  • Learned content features (NLP)
  • Evaluation results and status for remediation, etc.

Learn about Deployment Options