← DemosIn your browser

Drift with PSI

Shift the production distribution and watch the PSI and the alert that triggers retraining.

mlops-platform

Production scenario

Pick a case or tune it by hand.

Training: 4,000 customers, mean 35, sd 11.

00.10.250.5
PSI0,000
Stable

Nothing to do: the model sees customers like the ones it was trained on.

Customer age: training × production

In training every bin holds 10% of customers (the dashed line). Bars are production: the warmer the colour, the more the bin weighs in the PSI.

TrainingProduction

Week by week in production

If the audience drifts gradually toward the chosen scenario, this is how monitoring sees the PSI climb.

First warning
Retrain opened
PSI in week 12
See the table with each bin’s numbers

The PSI is the sum of the right-hand column.

Age binAgeTrainingProductionProd.ContributionContrib.
PSI

How PSI works

  1. 1

    Slice the training data

    Training customer ages are split into 10 bins holding 10% of customers each (deciles).

  2. 2

    Count production

    Customers arriving in production are placed into those same bins.

  3. 3

    Add up the gap

    Each bin adds (production − training) × ln(production ÷ training). An unchanged bin adds zero.

PSI = Σ (pprod − ptrain) · ln(pprod / ptrain)

Rule of thumb used in industry and in mlops-platform< 0.1 stable0.1 to 0.25 investigate> 0.25 retrain

In mlops-platform this runs for every feature of the credit model. Above 0.25 it opens a retrain, and the new model only replaces the champion if it wins on quality and fairness criteria.