DataFlow is an enterprise-grade data integration platform. Design complex pipelines visually, execute them with sub-millisecond precision, and monitor everything in real time.
DataFlow is built around a high-performance execution engine, a visual flow designer, and a comprehensive monitoring system — all wired together through a unified API gateway.
DAG-based pipeline executor with wave-parallel processing, conditional branching, sub-flows, data lineage tracking, and cron scheduling. Built on Tokio async runtime for maximum throughput.
Drag-and-drop React canvas powered by React Flow. Auto-generated config forms from module JSON Schemas. Version history, diff viewer, one-click rollback, and credential management.
Real-time execution waterfall, structured log viewer, system health indicators, and cancel actions. OpenTelemetry traces, Prometheus metrics, and pre-built Grafana dashboards.
Full REST API with RBAC enforcement, OIDC/SAML/local auth, JWT middleware, and an MCP server for AI assistant integration. All endpoints are typed, audited, and tested.
OIDC, SAML 2.0, and local auth with 16-permission RBAC. Ed25519-signed offline licenses — no call-home required, works in air-gapped networks.
First-class Claude AI Assistant module for data analysis, document generation, and natural language flow design. Plus an AI Agent Gateway supporting OpenAI, Gemini, AWS Bedrock, Azure OpenAI, and self-hosted models — all with unified prompt templating, structured output, and agentic tool-use within flows.
Every module implements the same FlowModule trait — chainable, configurable, and testable. The ecosystem spans file transfer, cloud storage, data processing, B2B/EDI, ERP/CRM, e-commerce, security, AI/LLM, and more. Build custom modules in Python via the module SDK, or distribute and install them through the module marketplace.
| Module | Type ID | Description |
|---|---|---|
| SFTP Transfer | sftp-transfer | Upload/download via SFTP with key/password auth, glob patterns, post-download actions |
| AS2 Transfer | as2-transfer | Send/receive files via AS2 (EDI over HTTP) with MDN acknowledgment support |
| FTP Transfer | ftp-transfer | Upload/download via FTP/FTPS (explicit and implicit TLS), active/passive modes, glob patterns |
| AS4 Transfer | as4-transfer | ebMS 3.0 / AS4 for EU B2B (Peppol, e-SENS), WS-Security signing/encryption, receipt handling |
| OFTP Transfer | oftp-transfer | OFTP2 (RFC 5024) for automotive/manufacturing supply chains, TLS, EERP/NERP receipts, restart/recovery |
| Module | Type ID | Description |
|---|---|---|
| AWS S3 | cloud-s3 | Upload/download/list/delete S3 objects, multipart uploads, server-side encryption, presigned URLs, SQS/SNS triggers |
| Google Cloud Storage | cloud-gcs | Upload/download/list/delete GCS objects, IAM or HMAC key auth, lifecycle policies, Pub/Sub event trigger |
| Azure Blob Storage | cloud-azure-blob | Upload/download/list/delete blobs, Shared Key or SAS token auth, hot/cool/archive tiers, Event Grid trigger |
| Cloudflare R2 | cloud-r2 | S3-compatible API for Cloudflare R2 with zero egress fees, Workers integration |
| Module | Type ID | Description |
|---|---|---|
| File Transform | file-transform | CSV, JSON, XML, and fixed-width format conversions |
| File Router | file-router | Route data to different branches based on content rules or metadata |
| File Compression | file-compress | ZIP, GZIP, and TAR compression and decompression |
| CSV File | csv-file | Parse, process rows, and generate CSV with per-row filtering and routing |
| XML Processor | xml-process | Parse, transform (XSLT), validate (XSD), XPath extraction, generate. Streaming for large documents |
| JSON Processor | json-process | Parse, transform (JSONPath/JMESPath), validate (JSON Schema), merge, split, flatten/unflatten |
| Document Processor | doc-process | Read/write DOCX (template fill, mail merge), XLSX (formulas, named ranges), PDF (text extraction, form fill) |
| Module | Type ID | Description |
|---|---|---|
| PGP Encrypt/Decrypt | pgp-crypto | Encrypt, decrypt, sign, verify with rPGP 0.14. ASCII armor and binary output |
| AES Encrypt/Decrypt | aes-crypto | AES-128/192/256 in CBC, CTR, GCM, CCM modes. PKCS7 padding, configurable IV, Base64 or binary output |
| 3DES Encrypt/Decrypt | 3des-crypto | Triple DES in CBC and ECB modes for legacy system interoperability (banking, POS, FIPS 46-3) |
| Hash & Checksum | hash-digest | MD5, SHA-1/256/384/512, SHA3, CRC32, Blake2b, Blake3. hash_file, verify_hash, hash_manifest operations |
| HMAC | hmac-auth | HMAC-SHA256/384/512/SHA3 for webhook signature verification (Shopify, Stripe, QuickBooks) and B2B integrity |
| S/MIME Encrypt/Sign | smime-crypto | S/MIME (RFC 8551) encryption and signing with X.509 certs. Integrates with platform certificate manager |
| Module | Type ID | Description |
|---|---|---|
| Web API Call | http-request | HTTP requests with Bearer, Basic, ApiKey, OAuth2, and mTLS auth types; template substitution |
| Email Notification | email-notify | Send SMTP email notifications on flow events or pipeline results |
| Database Query | db-query | Execute queries against PostgreSQL, MySQL, MSSQL, Oracle with per-row processing |
| ODBC Database | odbc-connect | ODBC driver-based connectivity for any database with an ODBC driver (DB2, Informix, Teradata, Snowflake, SAP HANA, etc.). Parameterized queries, bulk insert, connection pooling |
| Module | Type ID | Description |
|---|---|---|
| EDI Translator | edi-translate | Parse/generate X12 EDI with 14 transaction sets (830, 832, 850, 855, 856, 810, 860, 879, 888, 889, 997, 852, 940, 945) with retailer-specific profiles |
| Module | Type ID | Description |
|---|---|---|
| QuickBooks Connector | quickbooks-connect | Read/write QuickBooks Online (OAuth2 REST) and Desktop (QBXML) — invoices, POs, items, customers |
| SAP Connector | sap-connect | SAP ERP and S/4HANA via RFC/BAPI or OData APIs. IDocs inbound/outbound, sales orders, material master |
| Sage Connector | sage-connect | Sage 50/200/Intacct REST API — invoices, purchase orders, GL entries, inventory, customer/vendor records |
| Salesforce Connector | salesforce-connect | Salesforce REST/Bulk API with OAuth2 — SOQL queries, sObject CRUD, bulk import/export, Platform Events |
| Module | Type ID | Description |
|---|---|---|
| Shopify Connector | shopify-connect | Shopify Admin REST/GraphQL — orders, products, inventory, fulfillments. Webhook-triggered flows |
| Instacart Connector | instacart-connect | Instacart Connect API — catalog sync, order ingestion, availability updates, delivery status tracking |
| DoorDash Connector | doordash-connect | DoorDash Drive API — delivery creation, status tracking, order integration, menu/catalog sync |
| Module | Type ID | Description |
|---|---|---|
| Spreadsheet Connector | spreadsheet-connect | Read/write Excel (.xlsx/.xls) and Google Sheets with column mapping and header auto-detection |
| Module | Type ID | Description |
|---|---|---|
| Script Runner | script-runner | Execute sandboxed Python or shell scripts within a flow |
| File Watcher | file-watcher | Trigger flows based on file system events |
| Module | Type ID | Description |
|---|---|---|
| Claude AI Assistant | claude-ai | Anthropic Claude integration for data analysis, document generation, content classification, field extraction, and natural language flow design assistance |
| AI Agent Gateway | ai-agent | Multi-provider AI module supporting OpenAI, Google Gemini, AWS Bedrock, Azure OpenAI, and self-hosted models (Ollama, vLLM). Text generation, structured output, embeddings, and agentic tool-use |
| Module | Type ID | Description |
|---|---|---|
| BigQuery Connector | bigquery-connect | Google BigQuery read/write. SQL queries, streaming inserts, batch load from GCS, table/dataset management, schema inspection. OAuth2 or service account auth |
| Snowflake Connector | snowflake-connect | Snowflake read/write. SQL queries, COPY INTO for bulk load/unload, warehouse/database/schema management, stage operations. Key-pair or OAuth auth |
| Redshift Connector | redshift-connect | Amazon Redshift read/write. SQL queries, COPY/UNLOAD for S3-based bulk operations, Redshift Serverless support. IAM or username/password auth |
| Azure Synapse Connector | synapse-connect | Azure Synapse Analytics read/write. SQL queries, COPY INTO for ADLS-based bulk operations, dedicated and serverless pools. Azure AD or SQL auth |
| Databricks Connector | databricks-connect | Databricks SQL Warehouse and Delta Lake read/write. SQL queries, Unity Catalog integration, Delta table operations. PAT or OAuth auth |
| Module | Type ID | Description |
|---|---|---|
| Google Pub/Sub | gcp-pubsub | Publish and subscribe to Google Cloud Pub/Sub topics. Batch publish, pull/streaming-pull subscriptions, dead-letter handling, ordering keys, message filtering |
| AWS SNS/SQS | aws-messaging | Publish to SNS topics, send/receive SQS messages. FIFO and standard queues, dead-letter queues, message attributes, batch operations, long polling |
| Azure Service Bus | azure-servicebus | Send/receive Azure Service Bus messages. Queues and topics/subscriptions, sessions, dead-letter, scheduled messages, peek-lock/receive-and-delete modes |
| Apache Kafka | kafka-connect | Produce and consume Apache Kafka messages. Topic management, consumer groups, offset management, schema registry integration (Avro/JSON Schema), exactly-once semantics |
| Module | Type ID | Description |
|---|---|---|
| MQTT Client | mqtt-connect | Publish and subscribe to MQTT brokers (v3.1.1 and v5.0). QoS levels 0/1/2, retained messages, last will, topic wildcards, TLS/mTLS, WebSocket transport |
DataFlow ships with pre-built flow templates for common B2B scenarios — demonstrating how the engine's modules chain together in production. Retail EDI is one powerful example: connecting suppliers to Walmart, Amazon, Target, and Costco over AS2 with full X12 support. The same pipeline engine handles any industry, any protocol, any data shape.
The DataFlow engine is built for production. Every execution detail — waves, conditions, sub-flows, lineage — is tracked, stored, and visible.
Independent steps are grouped into waves via DAG topological analysis. Each wave executes concurrently up to a configurable max_parallel limit using Tokio JoinSet.
10 condition variants: Expression, Equals, NotEquals, GreaterThan, LessThan, Exists, All, Any, Not, Always. Template variable resolution from previous step outputs.
Flows can invoke other flows as sub-flows with full parent chain tracking, depth limits, and cycle detection. Enables modular pipeline design at scale.
Every DataProduced, DataConsumed, DataTransformed, SubFlowLink, ExternalSource, and ExternalSink event is tracked. Upstream and downstream traces via recursive CTE queries.
Configurable exponential backoff per step. Fail-fast propagation stops downstream steps on error. Cancellation checks between waves for clean shutdown.
Cron-based flow triggering with timezone support. Preview next run times via API. Full schedule lifecycle management through the UI or API.
DataFlow is available on AWS and Google Cloud Marketplace with one-click deployment and integrated billing — or self-host on your own infrastructure with full control. Pricing is based on the number of active data flows: Core ($2,000/yr · 5 flows), Standard ($3,750/yr · 15 flows), Professional ($9,500/yr · 40 flows), Enterprise (from $15,000/yr · unlimited). No per-row or per-execution charges. See full licensing details →
Deploy via a pre-built AMI, EKS Helm chart, or CloudFormation quick-start template. Integrated billing through your AWS account with SaaS subscriptions and annual commit options.
One-click GKE Autopilot deployment with Cloud SQL, load balancing, and managed certificates. Billing through your Google Cloud account via the Procurement API.
Production Helm chart with HPA, PDB, NetworkPolicy, ServiceMonitor, and Ingress with TLS. Deploy to any K8s cluster.
Terraform modules for EKS, GKE, and AKS — each paired with a managed PostgreSQL instance. One command to provision a full production environment.
Interactive and unattended systemd installer. Minimal footprint, no external dependencies.
NSSM-based Windows service installer. Runs alongside existing Windows infrastructure.
DataFlow uses Ed25519-signed offline licenses — no call-home, no license server, and full support for air-gapped deployments. Four license tiers gate module access at design time and runtime. Native HA deployment modes support Production/DR pairs and Active-Active clusters.
All license verification is done locally using Ed25519 signature verification against an embedded public key. Works in air-gapped networks and classified environments without any external connectivity.
Core → Standard → Professional → Enterprise. Each module declares its minimum tier via min_license_level(). The engine enforces tier checks at flow save, execution time, and in the module listing UI.
License keys encode HA role: Standalone, Production (active primary), Disaster Recovery (standby, reduced pricing), or Active-Active (full price per node). The engine enforces execution restrictions for DR nodes until promoted.