What is in the suite

Design on a canvas, run with the same engine, deploy as a signed file, watch the runs in one place.

Design

A canvas with lanes

Each lane is a pipeline. Lanes run in parallel and pass rows to each other through bridges. There are 45 node types, listed below by role.

Real data in every preview

Every node previews your data. “Explain this value” shows step by step how one output value was computed. Field lineage shows, for any field, where it comes from and what it feeds, down to the source and target columns.

Mistakes show up while you design

Missing inputs, unknown fields and expressions that do not compile are reported as you work. Undo and redo, history with named checkpoints, and environments (dev, test, prod) with profiles are built in.

Input

  • Database
  • File
  • HTTP API
  • FTP / SFTP
  • LDAP
  • GitHub
  • Folder watcher
  • ActiveMQ
  • MQTT
  • Webhook
  • Watchdog

Transform

  • Transform
  • Filter
  • Join
  • TMap
  • Aggregate
  • Window
  • Pivot / Unpivot
  • Explode
  • Union
  • JSON / XML parse and serialize
  • Script
  • Data Quality
  • Materialize
  • Report Generator

Output

  • Database
  • File
  • ActiveMQ
  • MQTT
  • FTP / SFTP
  • Mail
  • Webhook responder

Flow and DevOps

  • Bridges
  • Stop
  • Error Handler
  • Shell
  • SSH

The Node Reference manual documents all 45 node types.

FPEL, one expression language

The same language writes computed fields, conditions, data-quality rules and scripts. It has 107 built-in functions, user-defined functions with examples that act as tests, shareable libraries and decode tables.

Null handling is explicit: three-valued logic as in SQL, with is empty and is not empty.

Data in and out

Databases
PostgreSQL, MySQL/MariaDB and SQLite with built-in drivers. Oracle, Informix and any ODBC source through ODBC, using the drivers of your system.
Files
CSV/TSV, Excel (xlsx, xlsm, xlsb, xls, ods), JSON, JSON Lines, XML, text and binary. Local files, glob patterns, or one file per incoming row.
Messaging and services
ActiveMQ (STOMP), MQTT 3.1.1 with TLS, HTTP APIs, webhooks and mail (SMTP).
Filter pushdown
A Filter after a database input becomes a WHERE clause automatically. Results never change; only less data travels.

Reliability

  • Transactions. Native and XA (two-phase commit across databases), with exact, documented rules for commit and rollback.
  • Error routing. Reject ports, catch ports and a per-lane Error Handler with rules, critical nodes and error_out.
  • Flows that wait. A flow can wait for new files in a folder, HTTP calls (with HMAC signatures, an IP allowlist and de-duplication) or queue messages. A watchdog waits for another system to be ready.
  • Clean stop. Ctrl+C or systemctl stop ends a service in an orderly way.

Deployment and security

  • Signed artifacts. Ed25519 signatures. The runner refuses anything not signed by a trusted key, and checks it before a single node runs.
  • Configuration without rebuilding. Variables marked configurable can be set per machine; everything else is frozen in the signed file.
  • Secrets never in project files. They are provided at run time from the environment or a config file.
  • A policy per machine. An administrator can, for example, forbid Shell and SSH nodes whatever the artifact says.
  • Transport and access. TLS for databases, SSH host-key checking, and monitor tokens split into ingest and read.

Operations

  • A single executable. No Java or other runtime. Plain exit codes: 0 success, 1 failure, 3 refused, 130 interrupted.
  • Stilton Monitor. Every run from every machine in one browser page: outcome, timeline, rows and errors. If the monitor is unreachable, runs are kept in an offline fallback file.
  • Charts in the studio. Memory and timing, per node and over time.

Performance, with the conditions

All figures were measured on a small virtual machine with 2 vCPU (Xeon 2.1 GHz). Real servers are faster; what matters is the order of magnitude and the memory profile.

Throughput and memory
BenchmarkRowsThroughputPeak memory
CSV (35 MB), Transform with 3 expressions, Filter, CSV1,000,000about 380,000 rows/s (2.6 s)33 MB
TMap: 500,000 orders against 9,000 customers, lookup join, 2 outputs500,000about 273,000 rows/snot measured
Start-up
StepTime
Admission check: signature, policy, configuration3 ms
Complete minimal run, process start to exit8–17 ms

Compared with a typical JVM-based ETL runtime

  • Milliseconds to start, against seconds.
  • Tens of MB of memory held flat, against hundreds of MB of heap.
  • No garbage-collection pauses.
  • One executable, against a JVM on every machine.

Do not expect a speed-up on I/O-bound jobs: there the database, the network or the disk set the pace, for any engine. The Performance & Best Practices manual explains how to read these figures.