13 July 2019

Lucid Pipeline

Most AI solutions can be built as pipelined implementations with various sources to sinks from a set of generalizable models. Invariably, knowledge graph will act as a key layer for evolvable feature engineering that can be translated into ontological vectors and fed into AI models. Split the pipeline as a lucid funnel, lucid reactor, lucid ventshaft, and lucid refinery using a loose analogy of a distillation process. The following components highlight the key abstractions:

AI/DS Engine Layers:

  • Disc (frontends - discovery/visualization layer)
  • Docs (live specs via swagger, etc - documentation layer)
  • APIs (proxy/gateway services connected with elasticsearch or solr - application layer)
  • DS (models and semantics - AI layer)
  • Eval (benchmarks, workbench and metrics - evaluation layer)
  • Human (optional human in the loop - human/annotation layer)
  • Tests (load, unit, uat, service, etc - testing layer)
  • Funnel (ingestion, pre-process, post-process layer using brokers like Kafka/Kinesis)
  • Reactor (reactive processes - workflow/transformational layer - via Spark, Beam, Flink, Dask, etc)
  • Ventshaft (filter, match, fuzzy, distance, probabilistic, relational - functional layer)
  • Refinery  (context types, objects, attributes and methods as blueprints - entity/object layer)
  • Datapiles (indexed data sources as services for document/column/graph stores - data access layer)
  • Conf (environment configurations for nginx, etc - configuration layer)
  • Cloud (connected services for AWS/GCP orchestration - infrastructure/platform layer)