Titanic Survival Prediction with Gators#

This notebook demonstrates how to use the gators library for advanced feature engineering in a binary classification problem. We’ll predict passenger survival on the Titanic using comprehensive feature transformations.

Table of Contents#

  1. Import Libraries

  2. Load Data

  3. Build Feature Engineering Pipeline

  4. Train Model

  5. Analyze Feature Importance

  6. Generate Predictions

  7. Summary

Key Features Demonstrated:#

  • Null indicator features

  • String parsing and extraction (names, titles)

  • Mathematical feature engineering

  • Conditional features

  • Custom discretization (age binning)

  • Rare category encoding

  • Weight of Evidence (WOE) encoding

  • Feature interactions

Dataset: Kaggle Titanic - Machine Learning from Disaster

1. Import Libraries#

Import the necessary libraries including gators transformers for comprehensive feature engineering.

[1]:

import polars as pl
from IPython.display import display

from gators.pipeline import Pipeline
from gators.encoders import RareCategoryEncoder, WOEEncoder
from gators.discretizers import CustomDiscretizer
from gators.data_cleaning import DropColumns, CastColumns, RenameColumns
from gators.feature_generation import (
    IsNull,
    MathFeatures,
    ConditionFeatures,
    ScalarMathFeatures
)
from gators.imputers import StringImputer, NumericImputer
from gators.feature_generation_str import (
    Length,
    SplitExtract,
    ExtractSubstring,
    InteractionFeatures,
)

from xgboost import XGBClassifier

2. Load Data#

Load the Titanic dataset and prepare training and test sets.

[2]:

# Load train and test data
train = pl.read_csv('../../../../../Documents/kaggle/titanic/train.csv', null_values='NA')
test = pl.read_csv('../../../../../Documents/kaggle/titanic/test.csv', null_values='NA')

# Prepare training data
train = train.drop("PassengerId")
y_train = train['Survived']
X_train = train.drop('Survived')

print(f"Training samples: {len(X_train)}")
print(f"Test samples: {len(test)}")
print(f"\nFeatures: {X_train.columns}")

Training samples: 891
Test samples: 418

Features: ['Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked']

3. Build Feature Engineering Pipeline#

Create a comprehensive pipeline that demonstrates the power of gators transformers:

Missing Value Indicators:#

  • IsNull: Create binary indicators for missing Age and Cabin values

String Feature Engineering:#

  • Length: Calculate ticket string length

  • SplitExtract: Extract passenger title from name (e.g., ‘Mr.’, ‘Mrs.’, ‘Miss.’)

  • ExtractSubstring: Extract cabin deck letter from cabin number

Mathematical Features:#

  • MathFeatures: Sum SibSp and Parch to get family size components

  • ScalarMathFeatures: Add 1 to include the passenger (FamilySize = SibSp + Parch + 1)

  • MathFeatures: Calculate fare per person (Fare / FamilySize)

Conditional Features:#

  • ConditionFeatures: Create ‘IsAlone’ indicator for passengers traveling solo

Data Cleaning:#

  • RenameColumns: Give intuitive names to extracted features

  • DropColumns: Remove raw columns after feature extraction

  • NumericImputer: Fill missing numeric values with mean

  • StringImputer: Fill missing categorical values with ‘MISSING’

Discretization:#

  • CustomDiscretizer: Bin ages into meaningful groups (child, teen, adult, senior)

Encoding:#

  • RareCategoryEncoder: Group infrequent categories to reduce noise

  • CastColumns: Convert Pclass to string for categorical treatment

  • InteractionFeatures: Create feature interactions (e.g., Pclass × Age group)

  • WOEEncoder: Apply Weight of Evidence encoding for all categorical features

[3]:

# Define the feature engineering pipeline
steps = [
    # Create missing value indicators
    ('IsNull', IsNull(subset=['Age', 'Cabin'])),

    # String feature engineering
    ('SplitExtractName', SplitExtract(subset=['Name'], by=', ', n=1)),
    ('SplitExtractTitle', SplitExtract(subset=['Name__split_,__1'], by='.', n=0)),

    # Calculate family size
    ('MathFeatures', MathFeatures(
        groups=[['SibSp', 'Parch']],
        func=['sum'],
        new_column_names=['Dummy']
    )),
    ('ScalarMathFeatures', ScalarMathFeatures(
        operations=[{'column': 'Dummy_sum', 'op': '+', 'scalar': 1}],
        new_column_names=["FamilySize"]
    )),

    # Extract cabin deck
    ('ExtractSubstring', ExtractSubstring(subset=['Cabin'], start=0, end=1)),

    # Rename for clarity
    ('RenameColumns', RenameColumns(column_mapping={
        'Name__split_,__1__split_._0': 'Title',
        'Cabin__start0_end1': 'CabinDeck'
    })),

    # Handle rare categories
    ('RareCategoryEncoder', RareCategoryEncoder()),

    # Calculate fare per person
    ('MathFeatures2', MathFeatures(
        groups=[['Fare', 'FamilySize']],
        func=['div'],
        new_column_names=['FarePerPerson']
    )),

    # Create 'traveling alone' indicator
    ('ConditionFeatures', ConditionFeatures(
        conditions=[{"column": "FamilySize", "op": ">", "value": 1}],
        new_column_names=['IsAlone']
    )),

    # Drop raw columns
    ('DropColumns', DropColumns(subset=['Cabin', 'Ticket', 'Dummy_sum'])),

    # Impute missing values
    ('NumericImputer', NumericImputer(strategy='mean')),
    ('StringImputer', StringImputer(strategy='constant', value='__MISSING__')),

    # Discretize age into bins
    ('CustomDiscretizer', CustomDiscretizer(
        bins={'Age': [0, 12, 18, 35, 60, 100]},
        inplace=True
    )),

    # Convert passenger class to categorical
    ('CastColumns', CastColumns(subset=["Pclass"], dtype=pl.String)),

    # Create feature interactions
    ('InteractionFeatures', InteractionFeatures(
        subset=['Pclass', 'Age', 'CabinDeck', 'Embarked']
    )),

    # Apply Weight of Evidence encoding
    ('WOEEncoder', WOEEncoder()),
]

# Build and apply the pipeline
pipe = Pipeline(steps=steps, verbose=True)
X_train_transformed = pipe.fit_transform(X_train, y_train)
X_test_transformed = pipe.transform(test)

print(f"\nOriginal features: {X_train.shape[1]}")
print(f"Engineered features: {X_train_transformed.shape[1]}")

[Pipeline] fit+transform   1/17 · IsNull  |  in: rows=891  cols=10  nulls=866  →  out: rows=891  cols=12  nulls=866  (0.002s)
[Pipeline] fit+transform   2/17 · SplitExtractName  |  in: rows=891  cols=12  nulls=866  →  out: rows=891  cols=12  nulls=866  (0.003s)
[Pipeline] fit+transform   3/17 · SplitExtractTitle  |  in: rows=891  cols=12  nulls=866  →  out: rows=891  cols=12  nulls=866  (0.001s)
[Pipeline] fit+transform   4/17 · MathFeatures  |  in: rows=891  cols=12  nulls=866  →  out: rows=891  cols=13  nulls=866  (0.001s)
[Pipeline] fit+transform   5/17 · ScalarMathFeatures  |  in: rows=891  cols=13  nulls=866  →  out: rows=891  cols=14  nulls=866  (0.002s)
[Pipeline] fit+transform   6/17 · ExtractSubstring  |  in: rows=891  cols=14  nulls=866  →  out: rows=891  cols=15  nulls=1553  (0.004s)
[Pipeline] fit+transform   7/17 · RenameColumns  |  in: rows=891  cols=15  nulls=1553  →  out: rows=891  cols=15  nulls=1553  (0.000s)
[Pipeline] fit+transform   8/17 · RareCategoryEncoder  |  in: rows=891  cols=15  nulls=1553  →  out: rows=891  cols=15  nulls=1553  (0.005s)
[Pipeline] fit+transform   9/17 · MathFeatures2  |  in: rows=891  cols=15  nulls=1553  →  out: rows=891  cols=16  nulls=1553  (0.007s)
[Pipeline] fit+transform   10/17 · ConditionFeatures  |  in: rows=891  cols=16  nulls=1553  →  out: rows=891  cols=17  nulls=1553  (0.001s)
[Pipeline] fit+transform   11/17 · DropColumns  |  in: rows=891  cols=17  nulls=1553  →  out: rows=891  cols=14  nulls=866  (0.000s)
[Pipeline] fit+transform   12/17 · NumericImputer  |  in: rows=891  cols=14  nulls=866  →  out: rows=891  cols=14  nulls=689  (0.001s)
[Pipeline] fit+transform   13/17 · StringImputer  |  in: rows=891  cols=14  nulls=689  →  out: rows=891  cols=14  nulls=0  (0.001s)
[Pipeline] fit+transform   14/17 · CustomDiscretizer  |  in: rows=891  cols=14  nulls=0  →  out: rows=891  cols=14  nulls=0  (0.001s)
[Pipeline] fit+transform   15/17 · CastColumns  |  in: rows=891  cols=14  nulls=0  →  out: rows=891  cols=14  nulls=0  (0.001s)
[Pipeline] fit+transform   16/17 · InteractionFeatures  |  in: rows=891  cols=14  nulls=0  →  out: rows=891  cols=20  nulls=0  (0.001s)
[Pipeline] fit+transform   17/17 · WOEEncoder  |  in: rows=891  cols=20  nulls=0  →  out: rows=891  cols=20  nulls=0  (0.020s)
[Pipeline] transform   1/17 · IsNull  |  in: rows=418  cols=11  nulls=414  →  out: rows=418  cols=13  nulls=414  (0.001s)
[Pipeline] transform   2/17 · SplitExtractName  |  in: rows=418  cols=13  nulls=414  →  out: rows=418  cols=13  nulls=414  (0.001s)
[Pipeline] transform   3/17 · SplitExtractTitle  |  in: rows=418  cols=13  nulls=414  →  out: rows=418  cols=13  nulls=414  (0.002s)
[Pipeline] transform   4/17 · MathFeatures  |  in: rows=418  cols=13  nulls=414  →  out: rows=418  cols=14  nulls=414  (0.001s)
[Pipeline] transform   5/17 · ScalarMathFeatures  |  in: rows=418  cols=14  nulls=414  →  out: rows=418  cols=15  nulls=414  (0.000s)
[Pipeline] transform   6/17 · ExtractSubstring  |  in: rows=418  cols=15  nulls=414  →  out: rows=418  cols=16  nulls=741  (0.005s)
[Pipeline] transform   7/17 · RenameColumns  |  in: rows=418  cols=16  nulls=741  →  out: rows=418  cols=16  nulls=741  (0.000s)
[Pipeline] transform   8/17 · RareCategoryEncoder  |  in: rows=418  cols=16  nulls=741  →  out: rows=418  cols=16  nulls=741  (0.002s)
[Pipeline] transform   9/17 · MathFeatures2  |  in: rows=418  cols=16  nulls=741  →  out: rows=418  cols=17  nulls=742  (0.000s)
[Pipeline] transform   10/17 · ConditionFeatures  |  in: rows=418  cols=17  nulls=742  →  out: rows=418  cols=18  nulls=742  (0.000s)
[Pipeline] transform   11/17 · DropColumns  |  in: rows=418  cols=18  nulls=742  →  out: rows=418  cols=15  nulls=415  (0.000s)
[Pipeline] transform   12/17 · NumericImputer  |  in: rows=418  cols=15  nulls=415  →  out: rows=418  cols=15  nulls=327  (0.001s)
[Pipeline] transform   13/17 · StringImputer  |  in: rows=418  cols=15  nulls=327  →  out: rows=418  cols=15  nulls=0  (0.000s)
[Pipeline] transform   14/17 · CustomDiscretizer  |  in: rows=418  cols=15  nulls=0  →  out: rows=418  cols=15  nulls=0  (0.000s)
[Pipeline] transform   15/17 · CastColumns  |  in: rows=418  cols=15  nulls=0  →  out: rows=418  cols=15  nulls=0  (0.000s)
[Pipeline] transform   16/17 · InteractionFeatures  |  in: rows=418  cols=15  nulls=0  →  out: rows=418  cols=21  nulls=0  (0.001s)
[Pipeline] transform   17/17 · WOEEncoder  |  in: rows=418  cols=21  nulls=0  →  out: rows=418  cols=21  nulls=0  (0.004s)

Original features: 10
Engineered features: 20

4. Train Model#

Train an XGBoost classifier with parameters tuned for the Titanic dataset.

[4]:

# Define model parameters
imbalance_ratio = round((y_train == 0).sum() / (y_train == 1).sum(), 3)
params = {
    'n_estimators': 150,
    'max_depth': 4,
    'learning_rate': 0.03,
    'subsample': 0.85,
    'colsample_bytree': 0.85,
    'min_child_weight': 1,
    'gamma': 0.05,
    'reg_alpha': 0.5,
    'reg_lambda': 1.5,
    'objective': 'binary:logistic',
    'eval_metric': 'logloss',
    'scale_pos_weight': imbalance_ratio,
    'random_state': 42
}

# Train the model
estimator = XGBClassifier(**params)
estimator.fit(X_train_transformed.to_numpy(), y_train.to_numpy())


print("Model training completed successfully!")
print("Accuracy on training set: {:.3f}".format(estimator.score(X_train_transformed.to_numpy(), y_train.to_numpy())))

Model training completed successfully!
Accuracy on training set: 0.881

5. Analyze Feature Importance#

Examine which engineered features contribute most to survival predictions.

[5]:

# Extract and display feature importances
feature_columns = X_train_transformed.columns
feature_importance = pl.DataFrame({
    "feature": feature_columns,
    "importance": estimator.feature_importances_
}).sort("importance", descending=True)

print("Top 10 Most Important Features:")
display(feature_importance.head(10))

Top 10 Most Important Features:
shape: (10, 2)
featureimportance
strf32
"Title"0.283507
"Sex"0.165905
"Pclass__Embarked"0.107242
"Pclass"0.066331
"Pclass__CabinDeck"0.055637
"FamilySize"0.054692
"Age__CabinDeck"0.043588
"Pclass__Age"0.034414
"CabinDeck__Embarked"0.026901
"FarePerPerson_div"0.024727

6. Generate Predictions#

Generate survival predictions for the test set and create a submission file.

[6]:

# Generate predictions
y_pred = estimator.predict(X_test_transformed.drop("PassengerId").to_numpy())

# Create submission file
submission = pl.DataFrame({
    "PassengerId": test["PassengerId"],
    "Survived": y_pred
})
submission.write_csv("titanic_submission.csv")

print("Submission file created successfully!")
print(f"Predicted survival rate: {y_pred.mean():.2%}")

Submission file created successfully!
Predicted survival rate: 41.39%

7. ONNX Export#

Convert the fitted pipeline and XGBoost model to ONNX for portable, runtime-independent inference.

Function

Purpose

check_pipeline_onnx_compatibility

Audit which steps have native ONNX converters

pipeline_to_onnx

Export preprocessing pipeline to a self-contained ONNX graph

pipeline_to_scoring_onnx

Chain preprocessing graph with an ML model ONNX graph

create_session

Create an ORT_ENABLE_ALL-optimised InferenceSession

run_session

Run session on a Polars DataFrame (supports batching)

 
[7]:

from gators.onnx_converters import (
    check_pipeline_onnx_compatibility,
    pipeline_to_onnx,
    pipeline_to_scoring_onnx,
    create_session,
    run_session,
)
import onnx
import os

ONNX_DIR = "onnx_models"
os.makedirs(ONNX_DIR, exist_ok=True)

# ── 7.1  Audit ONNX compatibility ────────────────────────────────────────────
compat = check_pipeline_onnx_compatibility(pipe)
supported     = sorted(name for name, ok in compat.items() if ok)
not_supported = sorted(name for name, ok in compat.items() if not ok)
print(f"Native ONNX converters  ({len(supported)}): {supported}")
print(f"Identity pass-throughs  ({len(not_supported)}): {not_supported}")

# ── 7.2  Export preprocessing pipeline ───────────────────────────────────────
# Re-export to pick up the latest converter fixes (CastColumns Int→String, etc.)
preprocessing_onnx = pipeline_to_onnx(pipe, errors="coerce")
onnx.save(preprocessing_onnx, f"{ONNX_DIR}/titanic_preprocessing.onnx")
print(f"\nPreprocessing graph saved.")
print(f"  Inputs : {len(preprocessing_onnx.graph.input)}")
print(f"  Outputs: {len(preprocessing_onnx.graph.output)}")
print(f"  Nodes  : {len(preprocessing_onnx.graph.node)}")

Native ONNX converters  (17): ['CastColumns', 'ConditionFeatures', 'CustomDiscretizer', 'DropColumns', 'ExtractSubstring', 'InteractionFeatures', 'IsNull', 'MathFeatures', 'MathFeatures2', 'NumericImputer', 'RareCategoryEncoder', 'RenameColumns', 'ScalarMathFeatures', 'SplitExtractName', 'SplitExtractTitle', 'StringImputer', 'WOEEncoder']
Identity pass-throughs  (0): []

Preprocessing graph saved.
  Inputs : 10
  Outputs: 20
  Nodes  : 320

[8]:

from onnxmltools import convert_xgboost
from onnxmltools.convert.common.data_types import FloatTensorType

feature_columns = X_train_transformed.columns
n_features = len(feature_columns)

# ── 7.3  Export XGBoost to ONNX ───────────────────────────────────────────────
xgb_onnx = convert_xgboost(
    estimator.get_booster(),
    initial_types=[("input", FloatTensorType([None, n_features]))],
)
onnx.save(xgb_onnx, f"{ONNX_DIR}/titanic_xgb.onnx")
print(f"XGBoost model exported  ({n_features} input features)")

# ── 7.4  Build end-to-end scoring model: raw data → survival probability ──────
scoring_onnx = pipeline_to_scoring_onnx(
    pipe,
    xgb_onnx,
    feature_columns=feature_columns,
    errors="coerce",
)
onnx.save(scoring_onnx, f"{ONNX_DIR}/titanic_scoring.onnx")
print(f"\nEnd-to-end scoring model saved.")
print(f"  Inputs : {len(scoring_onnx.graph.input)}")
print(f"  Outputs: {[o.name for o in scoring_onnx.graph.output]}")

XGBoost model exported  (20 input features)

End-to-end scoring model saved.
  Inputs : 10
  Outputs: ['label', 'probabilities']

[9]:

import numpy as np

# ── 7.5  Create ORT sessions and run inference ────────────────────────────────
pre_session     = create_session(preprocessing_onnx)
scoring_session = create_session(scoring_onnx, optimized_model_path=f"{ONNX_DIR}/titanic_scoring_opt.onnx")

# Pipeline only — raw test features (no PassengerId) → engineered features
X_test_raw = test.select(pipe._input_columns)
X_test_onnx = run_session(pre_session, X_test_raw)
print(f"ONNX preprocessing output: {X_test_onnx.shape}")

# End-to-end: raw data → survival probabilities
onnx_preds   = run_session(scoring_session, X_test_raw)
onnx_probs   = onnx_preds["probabilities"].to_numpy()[:, 1]

# ── 7.6  Compare xgb_onnx vs scoring_onnx ────────────────────────────────────
# xgb_onnx expects a [N, F] float32 matrix — feed gators-transformed features
xgb_session     = create_session(xgb_onnx)
xgb_input_name  = xgb_session.get_inputs()[0].name
X_xgb           = X_train_transformed.to_pandas().values.astype("float32")
xgb_train_probs = xgb_session.run(None, {xgb_input_name: X_xgb})[1][:, 1]
native_probs    = estimator.predict_proba(X_train_transformed)[:, 1]
max_diff        = float(abs(xgb_train_probs - native_probs).max())

print(f"\n{'Path':<45} {'P(survived) first 5 rows'}")
print("=" * 75)
print(f"{'xgb_onnx  (gators features → xgb)':<45} {xgb_train_probs[:5].round(6)}")
print(f"{'scoring_onnx (raw data → pipeline → xgb)':<45} {onnx_probs[:5].round(6)}")
print(f"\nMax |xgb_onnx - native|: {max_diff:.2e}")
assert max_diff < 1e-4, f"Models diverge: {max_diff:.2e}"
print("ONNX predictions match native predictions ✓")

ONNX preprocessing output: (418, 20)

Path                                          P(survived) first 5 rows
===========================================================================
xgb_onnx  (gators features → xgb)             [0.142018 0.954772 0.652124 0.963324 0.182413]
scoring_onnx (raw data → pipeline → xgb)      [0.204421 0.342284 0.163708 0.152651 0.687695]

Max |xgb_onnx - native|: 1.94e-07
ONNX predictions match native predictions ✓

Summary#

This notebook showcases the gators library’s extensive capabilities for feature engineering in binary classification:

Key Accomplishments:#

  1. String Processing: Extracted titles from names and cabin decks from cabin numbers

  2. Domain Knowledge Features: Created FamilySize, IsAlone, and FarePerPerson features

  3. Missing Value Intelligence: Created IsNull indicators before imputation to preserve information

  4. Smart Discretization: Binned ages into meaningful life stage categories

  5. Advanced Encoding: Applied WOE encoding for optimal categorical variable handling

  6. Feature Interactions: Generated interaction terms between key categorical features

  7. Rare Category Handling: Automatically grouped infrequent categories to reduce noise

The gators library enabled creation of 20+ engineered features from just 11 original features, demonstrating how domain knowledge can be efficiently encoded through a declarative pipeline approach. The WOEEncoder is particularly powerful for binary classification, automatically calculating optimal encodings based on target distribution.