House Price Prediction with Gators

This notebook demonstrates how to use the gators library for feature engineering in a regression problem. We’ll predict house prices from the Kaggle House Prices dataset using various gators transformers.

Table of Contents

1. Import Libraries

2. Load and Preprocess Data

3. Build Feature Engineering Pipeline

4. Feature Selection with Stability Index

5. Train Model and Evaluate

6. Generate Predictions

7. ONNX Export

8. Summary

Key Features Demonstrated:

• Data cleaning and type casting

• Missing value imputation (simple and group-based)

• Mathematical feature engineering (ratios, statistics)

• Feature encoding (one-hot and count encoding)

• Feature stability analysis for robust feature selection

• Regression modeling with XGBoost

• ONNX export for portable, runtime-independent inference

Dataset: Kaggle House Prices - Advanced Regression Techniques

1. Import Libraries

Import the necessary libraries including gators transformers for feature engineering.

import polars as pl
import numpy as np
from IPython.display import display

from gators.encoders import OneHotEncoder, CountEncoder
from gators.data_cleaning import DropColumns, CastColumns
from gators.feature_generation import MathFeatures, RatioFeatures
from gators.imputers import StringImputer, NumericImputer, GroupByImputer
from gators.pipeline import Pipeline

from sklearn.model_selection import cross_val_score
from sklearn.metrics import root_mean_squared_error, make_scorer

from xgboost import XGBRegressor
from gators.feature_selection.feature_stability_index import feature_stability_index

2. Load and Preprocess Data

Load the house prices dataset and perform initial data cleaning:

• Handle ‘NA’ string values as null

• Apply log transformation to the target (SalePrice) to normalize distribution

• Clean inconsistent category values

• Fix problematic year values in GarageYrBlt

# Load data with 'NA' strings interpreted as null values
X_train = pl.read_csv('../../../../../Documents/kaggle/house/train.csv', null_values=['NA'])
X_test = pl.read_csv('../../../../../Documents/kaggle/house/test.csv', null_values=['NA'])

# Apply log transformation to target variable for better distribution
X_train = X_train.with_columns(pl.col('SalePrice').log().alias('target'))

# Drop ID column and constant features
y_train = X_train["target"]
X_train = X_train.drop(['Id', 'Utilities', "target", "SalePrice"])
X_test = X_test.drop(['Utilities'])

def clean_data(X):
    """Clean inconsistent values and fix data quality issues"""
    return X.with_columns(
        # Fix Exterior2nd inconsistent naming
        pl.when(pl.col('Exterior2nd') == 'Brk Cmn')
        .then(pl.lit('BrkComm'))
        .otherwise(pl.col('Exterior2nd'))
        .alias('Exterior2nd'),

        # Fix future GarageYrBlt values (replace with YearBuilt if > 2010)
        pl.when(pl.col('GarageYrBlt') <= 2010)
        .then(pl.col('GarageYrBlt'))
        .otherwise(pl.col('YearBuilt'))
        .alias('GarageYrBlt')
    )

X_train = clean_data(X_train)
X_test = clean_data(X_test)

print(f"Training data shape: {X_train.shape}")
print(f"Test data shape: {X_test.shape}")

Training data shape: (1460, 78)
Test data shape: (1459, 79)

3. Build Feature Engineering Pipeline

The gators Pipeline orchestrates multiple feature transformers:

Data Type Management:

• CastColumns: Convert boolean and numeric codes to strings for categorical encoding

Missing Value Imputation:

• StringImputer: Fill missing categorical values with constants or most frequent values

• NumericImputer: Fill missing numeric values with constants (0 for counts/areas)

• GroupByImputer: Fill missing values using group-based statistics (e.g., median by neighborhood)

Feature Engineering:

• RatioFeatures: Create ratio features (e.g., living area / lot area)

• MathFeatures: Generate statistical features from related columns (sum, std, range, min, max)

Encoding:

• OneHotEncoder: Create binary indicator features for categories

• CountEncoder: Replace categories with their frequency counts

# Identify columns by data type for appropriate transformations
dtypes = dict(zip(X_train.columns, X_train.dtypes))
boolean_columns = [col for col, dtype in dtypes.items() if dtype == pl.Boolean]
int8_columns = [col for col, dtype in dtypes.items() if dtype == pl.UInt8]
columns_to_encode = [col for col, dtype in dtypes.items() if dtype == pl.String] + boolean_columns

# Numeric columns that should be treated as categories
num_columns_to_cast = ['MSSubClass', 'OverallCond', 'YrSold', 'MoSold']

# Build the pipeline
steps = [
    # Cast data types for proper handling
    ('cast_boolean_to_string', CastColumns(
        subset=boolean_columns,
        dtype=pl.String,
        inplace=True
    )),
    ('cast_num_to_string', CastColumns(
        subset=num_columns_to_cast,
        dtype=pl.String,
        inplace=True
    )),
    ('cast_int_to_float', CastColumns(
        subset=int8_columns,
        dtype=pl.Float64,
        inplace=True
    )),

    # Impute missing string values
    ('string_imputer_missing', StringImputer(
        strategy='constant',
        value='MISSING',
        subset=['PoolQC', 'MiscFeature', 'Alley', 'Fence', 'FireplaceQu',
                 'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond',
                 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1',
                 'BsmtFinType2', 'MasVnrType'],
        inplace=True
    )),
    ('string_imputer_const', StringImputer(
        strategy='constant',
        value='Typ',
        subset=['Functional'],
        inplace=True
    )),
    ('string_imputer_most_freq', StringImputer(
        strategy='most_frequent',
        subset=['MSZoning', 'Electrical', 'KitchenQual', 'Exterior1st', 'SaleType'] + num_columns_to_cast,
        inplace=True
    )),

    # Impute missing numeric values
    ('numerical_imputer_const', NumericImputer(
        strategy='constant',
        value=0,
        subset=['GarageYrBlt', 'GarageArea', 'GarageCars', 'BsmtFinSF1',
                 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'BsmtFullBath',
                 'BsmtHalfBath', 'MasVnrArea'],
        inplace=True
    )),

    # Group-based imputation: use neighborhood median for lot frontage
    ('group_imputer', GroupByImputer(
        group_by_column='Neighborhood',
        strategy='median',
        subset=['LotFrontage'],
        inplace=True
    )),

    # Create ratio features
    ('ratio_features', RatioFeatures(
        numerator_columns=['GrLivArea'],
        denominator_columns=['LotArea'],
        new_column_names=['LivLotArea']
    )),

    # Generate statistical features from related area measurements
    ('stat', MathFeatures(
        groups=[['TotalBsmtSF', '1stFlrSF', '2ndFlrSF']],
        func=['sum', 'std', 'range', 'min', 'max']
    )),

    # Encode categorical variables
    ('onehot', OneHotEncoder(
        subset=columns_to_encode,
        drop_columns=False
    )),
    ('count_encoder', CountEncoder(
        subset=columns_to_encode,
        drop_columns=True,
        inplace=True
    )),
]

pipe = Pipeline(steps=steps)
X_train = pipe.fit_transform(X_train)
X_test = pipe.transform(X_test)

# print(f"\nOriginal features: {X_train.shape[1]}")
# print(f"Engineered features: {X_test.shape[1]}")

4. Feature Selection with Stability Index

Use the feature_stability_index to identify robust features that consistently contribute to model performance across multiple training runs. This helps prevent overfitting by selecting only stable, reliable features.

# Define conservative model parameters to prevent overfitting
conservative_params = {
    'max_depth': 4,
    'min_child_weight': 3,
    'learning_rate': 0.05,
    'n_estimators': 500,
    'gamma': 0.1,
    'reg_alpha': 0.5,
    'reg_lambda': 1.0,
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'objective': 'reg:squarederror',
    'eval_metric': 'rmse',
    'tree_method': 'hist',
    'random_state': 42,
    'n_jobs': -1
}

# Calculate feature stability index
estimator = XGBRegressor(**conservative_params)

from sklearn.model_selection import KFold
skf = KFold(n_splits=5, shuffle=True, random_state=0)
fsi_results = feature_stability_index(estimator, skf=skf, X=X_train, y=y_train)

# Filter to stable features with non-zero importance
fsi_results = fsi_results.filter(
    (pl.col("importance") != 0) & (pl.col("fsi") != 0)
)
fsi_results = fsi_results.sort(by="importance", descending=True)

# Select stable features
selected_features = fsi_results["feature"].to_list()
print(f"Number of selected stable features: {len(selected_features)}")
print(f"\nTop 10 features by importance:")
display(fsi_results.head(10))

Number of selected stable features: 147

Top 10 features by importance:

shape: (10, 3)

feature	fsi	importance
str	f64	f32
"ExterQual"	1.0	0.144757
"OverallQual"	1.0	0.129108
"TotalBsmtSF_1stFlrSF_2ndFlrSF_…	1.0	0.070859
"GarageCars"	1.0	0.028582
"TotalBsmtSF"	1.0	0.024367
"GrLivArea"	1.0	0.024277
"CentralAir"	1.0	0.022926
"CentralAir__Y"	1.0	0.018369
"GarageCond__TA"	0.8	0.017281
"YearBuilt"	1.0	0.015775

5. Train Model and Evaluate

Train an XGBoost regression model using only the stable features and evaluate performance with cross-validation.

"target" in X_train.columns

False

# Prepare training data with selected features
X_train = X_train.select(selected_features).with_columns(pl.all().cast(pl.Float64))

# Update model parameters for final training
final_params = conservative_params.copy()
final_params['max_depth'] = 4  # Slightly deeper trees for better performance

# Create RMSE scorer (lower is better)
rmse_scorer = make_scorer(root_mean_squared_error, greater_is_better=False)

# Train and evaluate with cross-validation
estimator = XGBRegressor(**final_params)
cv_scores = cross_val_score(estimator, X_train, y_train, scoring=rmse_scorer, cv=5)

print(f"Cross-Validation RMSE: {-cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")

# Train final model on all training data
estimator.fit(X_train, y_train)
print("\nFinal model trained successfully!")

Cross-Validation RMSE: 0.1305 (+/- 0.0100)

Final model trained successfully!

6. Generate Predictions

Apply the trained model to generate predictions for the test set and create a submission file.

# Prepare test data
id_test = X_test["Id"]
X_test =  X_test.select(selected_features).with_columns(pl.all().cast(pl.Float64))

# Generate predictions (remember to reverse log transformation)
log_predictions = estimator.predict(X_test)
predictions = np.exp(log_predictions)  # Reverse log transformation

# Create submission file
submission = pl.DataFrame({
    "Id": id_test,
    "SalePrice": predictions
})
submission.write_csv("house_price_submission.csv")

print("Submission file created successfully!")
print(f"\nPredictions summary:")
print(f"Mean: ${predictions.mean():,.2f}")
print(f"Median: ${np.median(predictions):,.2f}")
print(f"Min: ${predictions.min():,.2f}")
print(f"Max: ${predictions.max():,.2f}")

Submission file created successfully!

Predictions summary:
Mean: $177,317.23
Median: $156,689.72
Min: $47,496.06
Max: $486,809.31

7. ONNX Export

Convert the fitted pipeline and XGBoost regressor to ONNX for portable, runtime-independent inference.

Function	Purpose
check_pipeline_onnx_compatibility	Audit which steps have native ONNX converters
pipeline_to_onnx	Export preprocessing pipeline to a self-contained ONNX graph
pipeline_to_scoring_onnx	Chain preprocessing graph with an ML model ONNX graph
create_session	Create an ORT_ENABLE_ALL-optimised InferenceSession
run_session	Run session on a Polars DataFrame (supports batching)

from gators.onnx_converters import (
    check_pipeline_onnx_compatibility,
    pipeline_to_onnx,
    pipeline_to_scoring_onnx,
    create_session,
    run_session,
)
import onnx
import os

ONNX_DIR = "onnx_models"
os.makedirs(ONNX_DIR, exist_ok=True)

# ── 7.1  Audit ONNX compatibility ───────────────────────────────────
compat = check_pipeline_onnx_compatibility(pipe)
supported     = sorted(name for name, ok in compat.items() if ok)
not_supported = sorted(name for name, ok in compat.items() if not ok)
print(f"Native ONNX converters  ({len(supported)}): {supported}")
print(f"Identity pass-throughs  ({len(not_supported)}): {not_supported}")

# ── 7.2  Export preprocessing pipeline ───────────────────────────────────
preprocessing_onnx = pipeline_to_onnx(pipe, errors="coerce")
onnx.save(preprocessing_onnx, f"{ONNX_DIR}/house_price_preprocessing.onnx")

print(f"\nPreprocessing graph saved.")
print(f"  Nodes  : {len(preprocessing_onnx.graph.node)}")

print(f"  Inputs : {len(preprocessing_onnx.graph.input)}")
print(f"  Outputs: {len(preprocessing_onnx.graph.output)}")

Native ONNX converters  (12): ['cast_boolean_to_string', 'cast_int_to_float', 'cast_num_to_string', 'count_encoder', 'group_imputer', 'numerical_imputer_const', 'onehot', 'ratio_features', 'stat', 'string_imputer_const', 'string_imputer_missing', 'string_imputer_most_freq']
Identity pass-throughs  (0): []

Preprocessing graph saved.
  Nodes  : 1558
  Inputs : 78
  Outputs: 349

from onnxmltools import convert_xgboost
from onnxmltools.convert.common.data_types import FloatTensorType

feature_columns = selected_features
n_features = len(feature_columns)

# ── 7.3  Export XGBoost regressor to ONNX ────────────────────────────────────
# estimator was fit() on a named Polars DataFrame, so its booster stores real column
# names (e.g. "OverallQual") instead of the "f0","f1",... pattern onnxmltools expects
# when converting a raw Booster - clear them so the dump uses generic feature indices.
booster = estimator.get_booster()
booster.feature_names = None
xgb_onnx = convert_xgboost(
    booster,
    initial_types=[("input", FloatTensorType([None, n_features]))],
)
onnx.save(xgb_onnx, f"{ONNX_DIR}/house_price_xgb.onnx")
print(f"XGBoost model exported  ({n_features} input features)")

# ── 7.4  Build end-to-end scoring model: raw data → predicted log(SalePrice) ──
scoring_onnx = pipeline_to_scoring_onnx(
    pipe,
    xgb_onnx,
    feature_columns=feature_columns,
    errors="coerce",
)
onnx.save(scoring_onnx, f"{ONNX_DIR}/house_price_scoring.onnx")
print(f"\nEnd-to-end scoring model saved.")
print(f"  Inputs : {len(scoring_onnx.graph.input)}")
print(f"  Outputs: {[o.name for o in scoring_onnx.graph.output]}")

XGBoost model exported  (147 input features)

End-to-end scoring model saved.
  Inputs : 78
  Outputs: ['variable']

import numpy as np

# ── 7.5  Create ORT sessions and run inference ────────────────────────────────
pre_session     = create_session(preprocessing_onnx)
scoring_session = create_session(scoring_onnx, optimized_model_path=f"{ONNX_DIR}/house_price_scoring_opt.onnx")

# Reload + clean the raw test data (X_test above was overwritten by feature selection)
X_test_raw = pl.read_csv('../../../../../Documents/kaggle/house/test.csv', null_values=['NA'])
X_test_raw = clean_data(X_test_raw.drop(['Utilities']))
X_test_raw = X_test_raw.select(pipe._input_columns)

X_test_onnx = run_session(pre_session, X_test_raw)
print(f"ONNX preprocessing output: {X_test_onnx.shape}")

# End-to-end: raw data → predicted log(SalePrice)
onnx_preds = run_session(scoring_session, X_test_raw)
onnx_output_name = scoring_onnx.graph.output[0].name
onnx_log_predictions = onnx_preds[onnx_output_name].to_numpy().flatten()
onnx_predictions = np.exp(onnx_log_predictions)

# ── 7.6  Compare xgb_onnx vs scoring_onnx ────────────────────────────────────
# xgb_onnx expects a [N, F] float32 matrix — feed the same selected/cast features used to fit()
xgb_session      = create_session(xgb_onnx)
xgb_input_name   = xgb_session.get_inputs()[0].name
X_xgb            = X_train.to_numpy().astype("float32")
xgb_log_preds    = xgb_session.run(None, {xgb_input_name: X_xgb})[0].flatten()
native_log_preds = estimator.predict(X_train)
max_diff         = float(abs(xgb_log_preds - native_log_preds).max())

print(f"\n{'Path':<45} {'log(SalePrice) first 5 rows'}")
print("=" * 75)
print(f"{'xgb_onnx  (gators features → xgb)':<45} {xgb_log_preds[:5].round(6)}")
print(f"{'scoring_onnx (raw data → pipeline → xgb)':<45} {onnx_log_predictions[:5].round(6)}")
print(f"\nMax |xgb_onnx - native|: {max_diff:.2e}")
assert max_diff < 1e-4, f"Models diverge: {max_diff:.2e}"
print("ONNX predictions match native predictions ✓")

ONNX preprocessing output: (1459, 349)

Path                                          log(SalePrice) first 5 rows
===========================================================================
xgb_onnx  (gators features → xgb)             [12.239504 12.090002 12.271276 11.971485 12.584524]
scoring_onnx (raw data → pipeline → xgb)      [11.725914 11.985086 12.135552 12.136435 12.146822]

Max |xgb_onnx - native|: 1.24e-05
ONNX predictions match native predictions ✓

2026-08-24 10:19:50.934 Python[54107:319616] 2026-08-24 10:19:50.934237 [W:onnxruntime:, inference_session.cc:2670 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED and the NchwcTransformer enabled. The generated model may contain hardware specific optimizations, and should only be used in the same environment the model was optimized in.

Summary

This notebook demonstrates the power of the gators library for feature engineering in regression tasks:

Key Takeaways:

1. Comprehensive Imputation: Multiple strategies for handling missing values (constant, most frequent, group-based)

2. Smart Type Handling: Automatic casting and conversion of data types for proper feature engineering

3. Mathematical Features: Create ratio and statistical features from related columns

4. Feature Stability: Use FSI (Feature Stability Index) for robust feature selection

5. Flexible Encoding: Support for both one-hot and count encoding strategies

6. Pipeline Integration: Seamlessly integrates with scikit-learn Pipeline and XGBoost

The gators library simplified complex feature engineering workflows while maintaining code clarity and reproducibility, particularly excelling at handling missing data through the innovative GroupByImputer transformer.