Rule Cross-Validation#

Functions#

validate_rules_cv#

iguanas.rule_cv.validate_rules_cv(X: polars.DataFrame, y: polars.Series, rules: list[str], n_folds: int = 5, cv_metrics: list[str] | None = None, weight_column: str | None = None, shuffle: bool = True, random_state: int | None = None) → polars.DataFrame[source]#

Evaluate rule stability across K folds.

Validates already-generated rules on K held-out folds without re-generating them. For each fold the rules are evaluated on the validation split, and the mean, standard deviation, and minimum of each requested metric across folds are returned. Rules with a high {metric}_cv_std or a low {metric}_cv_min are likely over-fitted to the training data.

Parameters:

  • X (pl.DataFrame) – Feature DataFrame. Must contain all columns referenced in rules.

  • y (pl.Series) – Target series (boolean or binary).

  • rules (list[str]) – Rule expressions to validate. Typically obtained from apply_filter_and_deduplicate_rules() or similar.

  • n_folds (int, default=5) – Number of CV folds. The data is split into n_folds contiguous blocks (after optional shuffling).

  • cv_metrics (list[str] | None, default=None) – Metric names to compute CV statistics for. When None, defaults to ["precision", "recall", "f1"].

  • weight_column (str | None, default=None) – Name of a column in X to use as sample weights when computing metrics. If None, all samples are weighted equally.

  • shuffle (bool, default=True) – Whether to shuffle the row indices before splitting into folds.

  • random_state (int | None, default=None) – Random seed for reproducibility when shuffle=True.

Returns:

One row per rule with columns:

  • rule

  • {metric}_cv_mean — mean of the metric across folds

  • {metric}_cv_std — standard deviation across folds

  • {metric}_cv_min — worst-fold value (lowest)

Sorted by rule name.

Return type:

pl.DataFrame

Examples

>>> import polars as pl
>>> X = pl.DataFrame({"age": [25, 30, 35, 40, 45, 50, 55, 60, 65, 70]})
>>> y = pl.Series([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
>>> rules = ['(X["age"] >= 50)']
>>> validate_rules_cv(X, y, rules, n_folds=2, random_state=0)

Notes

The CV stability scores carry an optimism bias: because rules are generated from the full dataset before validate_rules_cv is called, the held-out folds were already seen during rule extraction. The reported {metric}_cv_min and {metric}_cv_std are therefore a lower bound on overfitting, not a true out-of-sample estimate. Use them to flag unstable rules rather than to estimate deployment performance.

See also

apply_filter_and_deduplicate_rules

Complete evaluation pipeline that produces the rules input for this function.

identify_unstable_rules#

iguanas.rule_cv.identify_unstable_rules(cv_result: polars.DataFrame, metric: str = 'f1', max_std: float = 0.05, min_mean: float | None = None) → polars.DataFrame[source]#

Return rules whose cross-fold metric is unstable or consistently poor.

Filters the output of validate_rules_cv() to surface rules that are likely over-fitted (high variance across folds) or simply weak (low mean metric).

Parameters:

  • cv_result (pl.DataFrame) – Output of validate_rules_cv(). Must contain columns {metric}_cv_std and (if min_mean is set) {metric}_cv_mean.

  • metric (str, default="f1") – Metric prefix to inspect (must match one used in validate_rules_cv()).

  • max_std (float, default=0.05) – Rules whose {metric}_cv_std exceeds this threshold are flagged as unstable.

  • min_mean (float | None, default=None) – If provided, also flag rules whose {metric}_cv_mean is below this threshold (consistently weak rules).

Returns:

Subset of cv_result containing only flagged rules, sorted by {metric}_cv_std descending (most unstable first).

Return type:

pl.DataFrame

Raises:

ValueError – If required columns are absent from cv_result.

Examples

>>> cv = validate_rules_cv(X, y, rules, n_folds=5)
>>> identify_unstable_rules(cv, metric="f1", max_std=0.05, min_mean=0.3)