Gators Logo

Gators is a lightning-fast data preprocessing and feature engineering library built on top of Polars, designed to streamline your entire ML workflow from raw data to production-ready models — benchmarked faster than scikit-learn and feature-engine across common preprocessing tasks (see Benchmarks below).

Built by the PSP Data Team at PayPal, Gators makes data preprocessing and feature engineering both faster and simpler.

Key Features#

  • 🚀 Lightning Fast: Benchmarked faster than scikit-learn and feature-engine on common preprocessing tasks

  • 🔄 Unified API: Consistent sklearn-style .fit() and .transform() interface

  • 📦 Production Ready: Deploy the same Python code from notebook to production

  • 🎯 Comprehensive: 108 preprocessing transformers across 11 categories

  • 🔗 Pipeline Support: Chain transformers seamlessly with the Pipeline class

  • 📤 ONNX Export: Export fitted pipelines to ONNX for low-latency inference (most transformers supported; a handful of feature_generation_str transformers can’t convert due to ONNX’s limited string-tensor op support)

  • 🎓 Easy to Learn: If you know sklearn, you already know Gators

Benchmarks#

Gators transformers are timed head-to-head against their closest scikit-learn and feature-engine equivalents (same algorithm, fit + transform, best-of-3 runs) on a 500,000-row synthetic dataset:

Transformer

gators (s)

scikit-learn (s)

feature-engine (s)

speedup vs sklearn

speedup vs feature-engine

NumericImputer (mean)

0.005

0.025

0.012

5.2x

2.4x

StandardScaler

0.002

0.009

n/a

3.8x

n/a

QuantileClipper

0.005

n/a

0.056

n/a

12.3x

EqualSizeDiscretizer (5 bins)

0.025

0.112

0.182

4.5x

7.3x

OneHotEncoder

0.045

0.283

0.320

6.2x

7.0x

OrdinalEncoder

0.029

0.267

0.127

9.1x

4.3x

TargetEncoder

0.029

0.442

0.163

15.0x

5.5x

WOEEncoder

0.028

n/a

0.181

n/a

6.5x

n/a means no equivalent implementation exists in that library. Measured on an Apple M3 Max; see Benchmarking for full methodology, environment details, and reproduction instructions.

Quick Start#

import polars as pl
from gators.data_cleaning import DropHighNaNRatio, VarianceFilter
from gators.encoders import OneHotEncoder
from gators.imputers import NumericImputer
from gators.scalers import StandardScaler
from gators.pipeline import Pipeline

# Load your data
X =  pl.read_csv("data.csv")

# Build a preprocessing pipeline
pipeline = Pipeline(steps=[
    ('drop_nan', DropHighNaNRatio(max_ratio=0.5)),  # drop columns with >50% missing values
    ('impute', NumericImputer(strategy='median')),  # impute missing values with median
    ('variance', VarianceFilter(min_var=0.01)),     # remove numerical columns with a variance < 0.01
    ('encode', OneHotEncoder()),  # one-hot encode categorical variables
    ('scale', StandardScaler())  # standardize numerical features
])

# Fit and transform
X_processed = pipeline.fit_transform(X)

# Deploy the same pipeline in production!

What Can Gators Do?#

108 transformers across 11 categories:

  • 🧹 Data Cleaning - Quality filters, deduplication, rounding, and type casting (16)

  • ✂️ Clippers - Custom, Gaussian, IQR, MAD, and Quantile outlier clipping (5)

  • 🧩 Encoders - OneHot, Target, WOE, CatBoost, Binary, Hash, and more (10)

  • 🎯 Numeric Features - Polynomial, ratio, Fourier, aggregation, rule-based (21)

  • 📝 String Features - Length, patterns, n-grams, TF-IDF, regex extraction (19)

  • 📅 DateTime Features - Cyclical encoding, holidays, business hours, time windows (8)

  • 🔄 Imputation - Numeric, string, boolean, KNN, iterative, and group-based strategies (6)

  • 📊 Discretization - Equal-width, quantile, tree-based binning, and more (7)

  • ⚖️ Scalers - Standard, min-max, robust, Box-Cox, Yeo-Johnson, and more (9)

  • ✨ Feature Selection - Correlation, stability, IV, mutual information, permutation (6)

  • 🔗 Pipeline - Chain transformers seamlessly (1)

  • 📤 ONNX Export - Export most fitted pipelines to ONNX for language-agnostic inference (a handful of feature_generation_str transformers aren’t convertible)

Credits#

Developed by the PSP Data Team at PayPal.

⚡ Built by data scientists, for data scientists

Standing on the Shoulders of Giants#

“If I have seen further, it is by standing on the shoulders of giants.” — Isaac Newton

Gators builds upon the incredible work of the open-source community. We are deeply grateful to:

Gators continues this tradition with Polars-powered performance.

Indices and tables#