Model Benchmarks

Foundation model · 130M synthetic datasets

A transformer pre-trained on 130 million synthetic tabular datasets, learning the statistical patterns common across all tabular data. At inference it performs in-context learning: you show it a handful of labeled examples plus a test row, and it predicts — without any training on your specific dataset. The tabular equivalent of GPT's few-shot prompting. Wins statistically significantly at small scale (<1K rows); gradient boosters catch up as data grows. Runs on an A10G GPU, ~50s latency.

Amazon · stacked ensemble AutoML

Not a single model but a system. Trains LightGBM + CatBoost + XGBoost + Random Forest + neural nets + FT-Transformer in parallel, then stacks them with a meta-learner. The tradeoff for its accuracy is 10-minute fold times and a harder-to-deploy artifact.

Yandex · native categorical handling

Gradient boosting that builds decision trees sequentially — each new tree corrects the previous tree's errors. CatBoost's edge is native support for categorical variables via ordered target statistics, which matters enormously here: we have 257 districts, 235 wards, and 974 streets that other frameworks either one-hot-encode (explosion) or ordinal-encode (lossy). Deployed in production at depth=10, autoresearch-tuned.

Microsoft · leaf-wise, histogram-based

Microsoft's speed-optimized gradient booster. Uses histogram-based feature binning and leaf-wise tree growth (vs XGBoost's level-wise), which converges faster on most tabular problems. Trained here with Huber loss to resist the long tail of outlier listings, and handles NaN values natively.

The original boosting framework

Popularized gradient boosting on tabular data and still a strong baseline. Slightly behind CatBoost and LightGBM on this dataset because it needs ordinal encoding for high-cardinality categoricals (its native categorical mode fails on district-level granularity).

Bagged decision trees

An ensemble of decision trees trained in parallel on bootstrapped samples, with predictions averaged across trees. Interpretable and robust, but lacks the sequential error-correction mechanism of boosting — which is why it plateaus around RMSE 0.56 here while boosters reach 0.34.

k-nearest neighbors · non-parametric

Predicts by averaging the 10 nearest listings in feature space, distance-weighted. No training phase; pays the cost at inference time. Struggles as categorical cardinality and feature dimensionality grow.

L2-regularized linear regression

Linear regression with L2 (squared coefficient) penalty — shrinks all coefficients toward zero. Useful as a sanity-check baseline: quantifies how much of the variance is linearly explained before we invoke non-linear models.

L1-regularized · sparse

Linear regression with L1 (absolute coefficient) penalty. Drives unimportant coefficients to exactly zero, giving implicit feature selection.

L1 + L2 blend

Blends Lasso (L1) and Ridge (L2) penalties. Handles correlated features more gracefully than pure Lasso (which arbitrarily picks one of a correlated pair).

ε-insensitive loss

Support vector regression with a linear kernel. Optimizes an ε-tube (ignores errors below ε, penalizes errors outside) rather than squared error — different loss geometry, comparable expressiveness to Ridge.

Plain OLS · no regularization

Classical ordinary least squares. The null hypothesis against which every more complex model must justify its complexity.