Most AI solutions can be built as pipelined implementations with various sources to sinks from a set of generalizable models. Invariably, knowledge graph will act as a key layer for evolvable feature engineering that can be translated into ontological vectors and fed into AI models. Split the pipeline as a lucid funnel, lucid reactor, lucid ventshaft, and lucid refinery using a loose analogy of a distillation process. The following components highlight the key abstractions:
AI/DS Engine Layers:
- Disc (frontends - discovery/visualization layer)
- Docs (live specs via swagger, etc - documentation layer)
- APIs (proxy/gateway services connected with elasticsearch or solr - application layer)
- DS (models and semantics - AI layer)
- Eval (benchmarks, workbench and metrics - evaluation layer)
- Human (optional human in the loop - human/annotation layer)
- Tests (load, unit, uat, service, etc - testing layer)
- Admin (control for access management, operations workloads, and automation - administration layer)
- Funnel (ingestion, pre-process, post-process layer using brokers like Kafka/Kinesis)
- Reactor (reactive processes - workflow/transformational layer - via Spark, Beam, Flink, Dask, etc)
- Ventshaft (fuzzy matches, distance matches, probabilistic filters, relational matches, clusters, fake filters, fake matches, feature selection filters, component factors, informed searches, uninformed searches, string matches, projection filters, samplings, tree searches, validations, verifications - functional/utility layer)
- Refinery (context types, objects, attributes and methods as blueprints - entity/object layer)
- Datapiles (indexed data sources as services for document/column/graph stores - data access layer)
- Conf (environment configurations for nginx, etc - configuration layer)
- Cloud (connected services for AWS/GCP orchestration - infrastructure/platform layer)
