Beyond the Credit Score: Gradient Boosting, Transformers and Sequential Models for Credit Risk Prediction

Traditional credit scoring looks at a snapshot: registration data, income, indebtedness, bureau information and a history condensed into a few variables. That snapshot works and is still the basis of many credit decisions. But the digitization of payments added another layer of information: the trajectory of financial behavior — how credit-limit utilization evolves month by month, how payments are spread out, how income flows in and out of the account.

The temptation is to conclude that more sophisticated models, able to “read” these trajectories, will automatically predict better. The right question is a different one: which approach delivers the best discrimination, calibration, temporal stability, explainability and economic value for a specific problem and dataset? This article walks through tabular models, tabular deep learning, sequential models, hybrid architectures and survival analysis, focusing on how to compare them fairly.

The problem: what exactly are we predicting#

Before choosing an algorithm, the target has to be defined precisely. In credit risk, the most common target is the PD (Probability of Default): the probability that a customer or loan defaults within a defined horizon.

Four definitions must be written down before any modeling:

• Observation (or decision) date: the moment the model is applied — loan origination or the date of a monitoring cycle.
• Observation window: the period before the decision date from which features are built.
• Prediction horizon: the period after the decision date in which the event is observed — for example, 12 months.
• Default event: the objective rule that defines the label, such as arrears beyond a days-past-due threshold within the horizon. The exact rule depends on the institution’s policy and must be documented.

flowchart LR
A["Observation window<br/>history used for features"] --> B(("Decision<br/>date"))
B --> C["Performance window<br/>12-month horizon"]
C --> D{"Default within<br/>horizon?"}

It also helps to separate two uses that are often mixed up:

Transactional and sequential data tend to be much richer in behavioral scoring, because the customer already has a history with the institution. In application scoring, the available trajectory may be short or nonexistent.

Tabular models#

In the tabular format, each row represents a customer or loan at a reference date, and each column is a feature computed up to that date. Examples: income, debt-to-income ratio, limit utilization, relationship tenure, accumulated arrears and moving averages of transactions.

The algorithms#

• Logistic regression: the interpretable baseline. Each coefficient has a direct reading, and the model is well understood by risk, audit and regulators. It usually requires careful handling of non-linearities (for example, with binning and WoE).
• Random Forest: an ensemble of trees trained on samples and feature subsets. Robust and not very sensitive to hyperparameters, but it tends to produce less calibrated probabilities.
• Gradient boosting (XGBoost, LightGBM, CatBoost): the most common choice for tabular credit data. It builds trees sequentially, each one correcting the errors of the previous ones.

The idea behind gradient boosting fits in one formula: the model is a sum of trees, and each new tree is fitted to the negative gradient of the loss with respect to the current prediction.

In practice, a few details matter as much as the algorithm:

• Categorical variables: CatBoost has native handling based on ordered target statistics; LightGBM also accepts categoricals directly. Encoding categories with the target by hand requires care not to leak the label.
• Missing values: all three libraries learn a default direction for missing values at each split. Still, it is worth understanding why data is missing — in credit, absence is often informative.
• Regularization: maximum depth, minimum samples per leaf, row and column subsampling and L1/L2 penalties control overfitting.
• Monotonic constraints: when the business requires that risk never decreases as a variable increases (such as the number of arrears), the libraries allow enforcing that monotonicity. This improves the model’s coherence and explainability.
• Tuning: hyperparameter search with temporal validation, never with a random split when the model will be used in the future.

Temporal feature engineering#

This is where the tabular model becomes competitive. Even without a sequential model, the customer’s trajectory can be summarized into features:

• Window aggregations: count, sum, mean, standard deviation and maximum over 7, 30, 90 and 180 days.
• Recency: days since the last arrear, the last payment or the last transaction.
• Frequency: number of events of each type per window.
• Trend: slope of limit utilization or balance over recent months.
• Volatility: variation of credited income or spending.
• Ratios and deltas: 30-day utilization divided by 180-day utilization, change in average balance between windows.

A sketch of how to compute aggregations that respect the decision date:

import pandas as pd

def point_in_time_features(tx: pd.DataFrame, ref: pd.DataFrame, windows=(7, 30, 90, 180)) -> pd.DataFrame:
    """tx: transactions (client_id, ts, amount); ref: (client_id, decision_date).
    Uses only events strictly before the decision date."""
    df = ref.merge(tx, on="client_id", how="left")
    df = df[df["ts"] < df["decision_date"]]
    age_days = (df["decision_date"] - df["ts"]).dt.days
    keys = [df["client_id"], df["decision_date"]]

    features = []
    for w in windows:
        in_window = df[age_days <= w]
        agg = in_window.groupby(["client_id", "decision_date"])["amount"].agg(["count", "sum", "mean", "std"])
        agg.columns = [f"tx_{stat}_{w}d" for stat in agg.columns]
        features.append(agg)

    recency = age_days.groupby(keys).min().rename("days_since_last_tx").to_frame()
    out = ref.set_index(["client_id", "decision_date"]).join(features + [recency])
    return out.reset_index()

Customers without events in a window end up with missing values; counts can be filled with zero, while means and standard deviations should stay missing for the model to handle.

Tabular deep learning#

Neural networks for tabular data have evolved considerably. The most cited families:

• MLP and tabular ResNet: dense networks, with or without residual connections. Simple and often underestimated as a baseline.
• FT-Transformer: turns each feature into a token (numeric or categorical, via embeddings) and applies attention across features. It was proposed by Gorishniy et al. (2021), together with a ResNet baseline.
• TabNet: uses sequential attention to select features at each decision step.
• TabPFN: a tabular foundation model pre-trained on synthetic data that makes predictions in a single forward pass, without traditional training on the target dataset. The Nature paper reports results for datasets of up to 10,000 samples (Hollmann et al., 2025), which limits its direct use on large portfolios.

The strengths of these networks are rich embeddings for high-cardinality categories, attention across features and the possibility of joining the tabular part with sequential encoders in a single end-to-end model. The costs show up in training time, sensitivity to hyperparameters, the need for normalization and, above all, on small or medium datasets.

The literature does not support the idea that Transformers always beat trees. Grinsztajn, Oyallon & Varoquaux (2022) show that tree-based models remain state of the art on medium-sized tabular datasets and attribute this to inductive biases: neural networks handle uninformative features poorly and struggle to learn irregular functions. Gorishniy et al. themselves conclude that there is no universally superior approach. In credit specifically, Schmitt (2022) finds gradient boosting generally ahead of deep learning on the datasets evaluated, with faster training, while stressing that the best model depends on the data.

Sequential models#

In the sequential format, each customer has an ordered sequence of events or snapshots: payments, arrears, transactions, limit utilization, balances and income inflows. Instead of summarizing that sequence into hand-picked aggregates, the model learns the representation directly.

The most common architectures:

• LSTM and GRU: recurrent networks with gating mechanisms that keep a state along the sequence. The GRU is simpler and usually trains faster.
• TCN (Temporal Convolutional Networks): causal, dilated convolutions that capture local and long-range patterns with parallel training.
• Temporal Transformers: attention across the events of the sequence, with positional or time encoding. Recent work also combines LSTMs with attention mechanisms adapted to credit risk (Wang & Bellotti, 2026).

Real financial data brings difficulties that tutorials skip:

• Irregular sequences: events do not arrive at fixed intervals. Adding the interval between events (Δt) as a feature or using continuous time encoding helps the model tell “three payments in one week” from “three payments in three months”.
• Different lengths: padding and masks ensure empty positions do not influence the result.
• Truncation: long-standing customers have long histories; you have to decide how many events or months to keep.
• Missingness: the absence of events in a period is also information and can be encoded explicitly.
• Observation window: the sequence ends at the decision date, under the same point-in-time rules as the tabular features.

A simple trajectory example: one customer’s limit utilization over six months.

xychart-beta
x-axis [Jan, Feb, Mar, Apr, May, Jun]
y-axis "Limit utilization (%)" 0 --> 100
line [25, 35, 48, 67, 84, 96]

Formally, the difference between the two approaches lies in what the function receives. In the tabular model, the estimated PD for customer $i$ depends on a feature vector; in the sequential one, it also depends on the event sequence $s_{i1}, \dots, s_{iT}$:

The gain from sequences must be measured against a tabular baseline that already has good temporal features. Comparing a sequential model with a tabular one without temporal aggregations measures the lack of feature engineering, not the value of the architecture.

Hybrid architectures#

In practice, the question is rarely “tabular or sequential”. A hybrid architecture combines both paths:

flowchart TD
A[Sources: registration, bureau, transactions and payments] --> B[Point-in-time dataset]
B --> C[Tabular features and aggregations]
B --> D[Temporal sequences]
C --> E[LightGBM / CatBoost]
D --> F[GRU / Transformer]
E --> G[Leakage-free fusion / stacking]
F --> G
G --> H[PD calibration]
H --> I[Credit policy and monitoring]

A sketch of stacking with out-of-fold predictions that respect time order:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import TimeSeriesSplit

def oof_predictions(model, X, y, n_splits=5):
    """Out-of-fold predictions; X and y sorted by decision date."""
    oof = np.full(len(X), np.nan)
    for train_idx, valid_idx in TimeSeriesSplit(n_splits=n_splits).split(X):
        model.fit(X.iloc[train_idx], y.iloc[train_idx])
        oof[valid_idx] = model.predict_proba(X.iloc[valid_idx])[:, 1]
    return oof

oof_tab = oof_predictions(tabular_model, X_tab, y)
oof_seq = oof_predictions(sequence_model, X_seq, y)  # wrapper with a scikit-learn API

# The first temporal fold gets no prediction, so it stays out of the meta-model.
mask = ~np.isnan(oof_tab) & ~np.isnan(oof_seq)
meta = LogisticRegression()
meta.fit(np.column_stack([oof_tab, oof_seq])[mask], y.to_numpy()[mask])

TimeSeriesSplit splits by row order; if the same customer appears at several dates, you also need to make sure their records do not sit on both sides of a split in a way that leaks future information.

Finally, the hybrid architecture has a cost: two data pipelines, two models to monitor and more latency. The evaluation should compare the incremental gain with that cost — and if the gain is small or unstable across vintages, the simpler model wins.

Survival analysis and multiple horizons#

Binary “default within 12 months” classification answers a single question and throws information away: a customer who defaults in month two gets the same label as one who defaults in month eleven. And loans with fewer than 12 months of observation have no reliable label.

Survival analysis models the time to the event. In its discrete-time version, it estimates the hazard: the probability of default in period $t$, given that the customer reached it without defaulting. The cumulative PD up to horizon $t$ comes from the product of the probabilities of “surviving” each period:

This formulation brings three advantages:

• Censoring: loans observed for less time than the horizon still contribute the periods in which they were observed.
• Coherent multiple horizons: since each term $(1 - h_k)$ is between 0 and 1, the cumulative PD never decreases with the horizon. The consistency $PD_{3m} \leq PD_{6m} \leq PD_{12m}$ is guaranteed by construction, something three independent binary classifiers do not guarantee.
• Competing risks: prepayment or early settlement ends the loan without default. Treating them as ordinary censoring distorts the estimates; competing-risk models estimate a cause-specific hazard for each outcome.

In practice, the discrete-time model can be trained with any binary classifier — including gradient boosting — on a dataset expanded into person-period format:

import pandas as pd

def to_person_period(loans: pd.DataFrame, horizon: int = 12) -> pd.DataFrame:
    """loans: one row per loan, with months_observed (until default or censoring) and defaulted (0/1).
    Produces one row per loan and month at risk; the target is 1 only in the month of default."""
    rows = []
    for loan in loans.itertuples():
        last = min(loan.months_observed, horizon)
        for t in range(1, last + 1):
            event = int(loan.defaulted and t == loan.months_observed)
            rows.append({"loan_id": loan.loan_id, "t": t, "event": event})
    return pd.DataFrame(rows)

The classifier receives the loan’s features and period $t$ and estimates $h_t$; the cumulative PD for 3, 6 and 12 months follows from the formula above.

Validation and metrics#

The way you validate determines whether the comparison between models can be trusted. In credit, some precautions are mandatory:

• Temporal split: training, validation and test in successive periods, with an out-of-time holdout that simulates real use.
• Leakage across records: the same customer may appear at several reference dates; the split must prevent their future information from contaminating training.
• Post-decision variables: any feature computed after the decision date must be removed.

Discrimination and calibration metrics#

Two formulas help separate ranking from calibration. The Brier score measures the squared error of the probabilities; ECE groups predictions into bins and compares the average predicted PD with the observed default rate in each bin:

A model can have an excellent AUC and poorly calibrated PDs — and, in credit, the PD drives pricing, limits and provisioning. That is why calibration (for example, with isotonic regression or Platt scaling, fitted on data outside training) is part of the model, not a detail.

Economic value, stability and explainability#

• Economic metrics: expected cost, approval rate, expected loss and the risk/return trade-off at each cutoff. Every economic metric depends on assumptions (the cost of a false negative, margin, LGD) that must be made explicit.
• Stability: population drift (for example, with PSI), performance by vintage or cohort and by segment. A model that is better on average but unstable across vintages is a risk.
• Explainability: SHAP, where appropriate, to understand global and individual contributions — and to check whether the model uses the signals the business expects.

In production, the model is only one part of the system: feature and model versioning, an audit trail of decisions, drift and performance monitoring, and controlled retraining with approval and comparison against the current model. Recent discussions also point out how temporal leakage, distribution shift and explanation instability combine in high-stakes tabular applications such as credit (Rahman & Tabassum, 2026, preprint).

A reproducible experimental plan#

To answer whether sequences and hybrid models pay off on a specific dataset, a staged plan isolates the contribution of each component:

Experiment C only makes sense if the dataset has sequences with adequate granularity. For a fair comparison:

• the same labels, population, horizon and the same temporal splits in every experiment;
• an ablation table showing the effect of each feature group and each path of the hybrid;
• inference cost, latency and memory alongside the performance metrics;
• temporal robustness: performance by vintage in the out-of-time period;
• confidence intervals (for example, via bootstrap) whenever feasible, so differences within the noise are not treated as gains.

Study dataset#

A good starting point to reproduce this plan is the Home Credit — Credit Risk Model Stability competition on Kaggle. It provides multiple related tables and history useful for feature engineering, and its name points to the emphasis on model stability over time.

Research hypotheses#

The experimental plan exists to test hypotheses — not to confirm conclusions chosen in advance.

Conclusion#

The practical recommendation is incremental. Start with a robust tabular baseline — logistic regression and gradient boosting — fed by good temporal features and validated out-of-time. Only then add sequential encoders and fusion, and keep those layers only if the gain is consistent across vintages and pays for the operational cost. For those who need PDs over several horizons, discrete-time survival analysis offers coherence across horizons and correct handling of censoring.

Above all, in credit real performance is not a single AUC. Calibration, governance, fairness, stability and explainability are part of the result — and they decide whether a model can actually go to production. For a broader view of the literature, the systematic review by Ayari, Guetari & Kraïem (2025) is a good starting point.

References#

1. Grinsztajn, L., Oyallon, E. & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS 2022, Datasets and Benchmarks Track. Link
2. Gorishniy, Y., Rubachev, I., Khrulkov, V. & Babenko, A. (2021). Revisiting Deep Learning Models for Tabular Data. NeurIPS 2021. Link
3. Schmitt, M. (2022). Deep Learning vs. Gradient Boosting: Benchmarking state-of-the-art machine learning algorithms for credit scoring. arXiv:2205.10535. Link
4. Wang, H. & Bellotti, A. (2026). Long short-term memory network with adapted attention mechanism for credit risk modeling. International Journal of Forecasting. DOI
5. Ayari, H., Guetari, R. & Kraïem, N. (2025). Machine learning powered financial credit scoring: a systematic literature review. Artificial Intelligence Review, 59(1), 13. DOI
6. Ehrhardt, A., Biernacki, C., Vandewalle, V., Heinrich, P. & Beben, S. (2021). Reject inference methods in credit scoring. Journal of Applied Statistics. Link
7. Hollmann, N., Müller, S., Purucker, L. et al. (2025). Accurate predictions on small data with a tabular foundation model. Nature, 637, 319–326. DOI
8. Home Credit — Credit Risk Model Stability. Kaggle competition. Link
9. Rahman, N. & Tabassum, N. (2026). Temporal Leakage, Distribution Shift, and Explainability Instability in High-Stakes Tabular Machine Learning. Preprint, SSRN. DOI