PipelineNeural architecture search to no-code deployment — Pipeline dissects what moved, what shipped, and what's worth your attention before the standup. For engineers who build with data for a living.
The headline numbers look close. AutoKeras 2.1 clocks in at 2.3s median inference on the Kaggle tabular benchmark suite; Optuna-tuned XGBoost lands at 2.1s. But the distribution tells a different story — AutoKeras' 95th-percentile latency spikes to 11.7s, nearly 3× Optuna's 4.2s. If you're serving real-time predictions, that tail is your SLA breach waiting to happen.
| # | Framework | F1 Score ↓ | P50 Latency | Deploy Complexity | Adoption % |
|---|---|---|---|---|---|
| 01 | H2O AutoML | 0.924 | 1.8s | Medium | 68% |
| 02 | AutoKeras 2.1 | 0.918 | 2.3s | Low | 74% |
| 03 | FLAML | 0.912 | 1.4s | High | 51% |
| 04 | AutoGluon 1.2 | 0.907 | 3.1s | Low | 62% |
| 05 | Optuna 3.4 | 0.899 | 2.1s | High | 83% |
We've been running FLAML in staging for 6 weeks. The good: setup is genuinely 15 minutes. The bad: it silently ignores feature interaction constraints you set in the config...
The new multimodal pipeline is impressive on paper. Tested on an A10G and it fits, but only if you disable the ensemble stacking layer. Anyone found a workaround that...
After the DARTS v3 paper last month, I'm not sure where the line is anymore. The search space overlaps significantly with HPO when you account for...
Pipeline tracks the metrics that move roadmaps — not the press-release numbers, but the ones that show up in benchmark repos and internal postmortems.
Kaggle tabular classification suite · 15 datasets · median of 5 runs each. Click any column header to re-rank. Updated weekly.
| # | Framework | F1 Score↓ | P50 Lat. | P95 Lat. | Memory × | Setup | Deploy | Adoption | Verdict |
|---|---|---|---|---|---|---|---|---|---|
| 01 | H2O AutoMLv3.46 | 0.924 | 1.8s | 5.2s | 1.4× | 8 min | Medium | 68% | Best accuracy |
| 02 | AutoKerasv2.1 | 0.918 | 2.3s | 11.7s | 2.3× | 12 min | Low | 74% | Watch P95 |
| 03 | FLAMLv2.3 | 0.912 | 1.4s | 3.9s | 1.1× | 15 min | High | 51% | Fastest |
| 04 | AutoGluonv1.2 | 0.907 | 3.1s | 8.4s | 3.1× | 6 min | Low | 62% | Best multimodal |
| 05 | Optunav3.4 | 0.899 | 2.1s | 4.2s | 0.9× | 45 min | High | 83% | Most adopted |
| 06 | TPOTv0.12 | 0.891 | 4.7s | 14.1s | 1.8× | 20 min | Medium | 39% | Declining |
| 07 | PyCaretv3.3 | 0.884 | 2.8s | 6.3s | 1.6× | 10 min | Low | 71% | Best DX |
Every issue goes past the announcement. We run the code, check the math, and tell you what the benchmark paper didn't.
DARTS v3 landed quietly last month. No blog post, no Twitter thread — just a commit and a paper. We ran it against EfficientNetV2-S on ImageNet and the story is more nuanced than the abstract suggests.
Three platforms promise one-click deployment. We deployed the same XGBoost model to all three and measured what "one click" actually costs — in latency, in money, and in the debugging hours you'll never get back.
We shipped it. Here's what broke, what held, and the one config flag that makes or breaks inference performance at scale. Spoiler: it's not the one in the docs.
Forget the benchmarks for a second. When you're buying for 200 data scientists, the question isn't F1 score — it's support SLAs, SSO, audit logging, and whether the vendor picks up the phone.
Issue #214 is live. AutoKeras 2.1 latency breakdown, the FLAML production report, and three NAS papers worth your time. No paywall, no signup.
Read This Week's IssueNot a general data newsletter. A specific one — for the engineers who know the difference between AutoML and automated ML, and care about which one it is.
“Pipeline is the only newsletter I read the same morning it lands. The benchmark methodology is rigorous enough that I've cited it in internal RFC docs without embarrassment.”
Join 28,400 ML engineers and data science leads. One email, every Tuesday. Unsubscribe in one click, no questions asked.