Anopsy

uv that's the tweet

Every time you say "garbage in, garbage out" an ML model dies.

Post-deployment data science. That's the tweet.

Univariate data drift doesn't show the full picture. Take a look at this demo created by my colleague @Anopsy There are two univariate distributions (top and right) which remain almost unchanged during the whole process. We can say that both look like normal distributions. If we run univariate drift detection methods such as Jensen-Shannon, Kolmogorov-Smirnov or any other, we wouldn't be alerted by those minimal changes. But if we look at their joint distribution, the story is different. The distributions of X and Z didn't change independently, but we do see sudden changes in the relationship between them. This is called Multivariate Data Drift, and no traditional univariate drift detection method can detect it. In this example, we only have two variables, so it’s easy to see the change. But what if we have a model with 10, 50, or 100 features? For that, we need algorithms capable of learning the relationships between many features and detecting when they change substantially. We came up with a method for this. It’s called Data Reconstruction with PCA. This method uses PCA to compress the feature space into a latent space. The algorithm then decompresses the latent space data and reconstructs it with some error. This error is called the reconstruction error. We can then use the learned compressor/decompressor to transform any production data and measure its reconstruction error. If the reconstruction error is bigger than an threshold, we say that the structure learned by PCA no longer represents the underlying structure of the initial data. Indicating that there is MULTIVARIATE data drift in the production data.

Data Science is my passion.