NVIDIA has released Kumo Tabular, an open-source transformer model designed for classification and regression on tabular data. Available in sizes up to 215 million parameters, it performs predictions in a single forward pass without requiring task-specific training or manual feature engineering, utilizing in-context learning to relate labeled examples to new query rows.
In-context learning for structured data
Kumo Tabular uses a transformer architecture designed to process numerical and categorical data through three distinct attention mechanisms. The system applies cell embeddings using Fourier features to handle various data types and missing values without imputation. It then alternates between column attention, which learns value distributions, and row attention to understand feature interactions.
Unlike traditional gradient-boosted trees that require manual hyperparameter tuning for every new dataset, this model uses in-context learning. By reading a set of labeled rows as context, it predicts labels for new rows directly. The model is released under the OpenMDW-1.1 license, allowing for commercial applications in tasks like demand forecasting or churn prediction.
Technical constraints and evaluation
The current implementation natively supports up to 10 classes in a single pass, though an associated library can extend this through output coding. While it handles numerical and categorical columns, other formats like text or timestamps require pre-processing into features. The model was pretrained exclusively on synthetic data.
Performance may decrease if table values fall far outside training ranges or if query rows differ significantly from the provided context distribution. NVIDIA notes that accuracy and calibration must be validated on specific user data before deployment, as results depend on the quality of the context rows provided during inference.
Original source
This report summarises the source below. Analysis is labelled separately; product and research claims remain attributed to their source.
Read the original at Hugging Face