Research Exposé · Agentic Systems Lab

Contamination Detection for Time-Series Models under Continual Learning

Ronald Domi July 2026

Motivation

Benchmark contamination happens when evaluation data leaks into the training. When this happens the reported model capabilities are inflated because of memorization rather than generalization.

Pipeline diagram
Figure 1 - Contamination can enter at the pre-training stage or during continual learning.

Contamination isn't limited to benchmark leaks - it can also mean a user's own data being used to train a model without their knowledge. This motivates a contamination detector that can run locally. A user could test whether any of their own data was used to train the model. Contamination during CL can happen due to errors in the CL pipeline like missed bugs or missing logic independent of the original pretraining stage.

01 - ContributionAdapting CoDec to time-series models

CoDec is a very reliable contamination detection method but only used for text-based models. Until now it hasn't been applied to time-series models. Since TSLMs predict continuous values rather than discrete tokens, CoDec needs to be adapted to this architecture.

In the image shown, we present a simplified showcase of CoDec on a pretrained GPT-2 fine-tuned on target passages. The point of this demo is to be a small proxy to confirm the workings of CoDec before attempting the harder numeric adaptation.

CoDec baseline on GPT-2
Figure 2 - CoDec score distribution on GPT-2 fine-tuned on target passages. Contaminated samples (orange) show elevated scores versus the clean baseline (blue).

This is a similar example, but here an initial adaptation was made to use TinyTS, showing an early proof of concept that CoDec can be adapted. Notice that the contaminated CoDec score becomes high after fine-tuning on the target dataset.

CoDec-TS baseline on TinyTS
Figure 3 - CoDec-TS score on TinyTS after fine-tuning on contaminated time-series data. The contaminated score rises clearly above the control.

02 - ContributionDoes the signal survive continual learning?

Here we want to understand what happens to the signal CoDec provides across successive CL cycles following contamination. Either the signal stays consistent across cycles, or it degrades. If it degrades, we study the resulting reliability window: the number of CL cycles within which a leak remains detectable. A model updated many times since a suspected leak may report "clean" simply because the check has gone stale, not because the leak never happened.

Reliability window diagram
Figure 4 - The reliability window: the interval following contamination within which detection remains possible.

Figures 5a and 5b test this empirically across two domains. Each has two panels: the left traces the CoDec score across checkpoints for the contaminated model (orange) and a matched control trained identically but on unrelated content (grey); the right shows the LMC barrier at the same steps. CoDec can no longer distinguish a contaminated model from a clean one.

Figure 5a (GPT-2 proxy) shows the gap close by steps 3–4. Figure 5b (CoDec-TS on ETTh1) shows the same collapse more sharply (initial training randomly had a high CoDec score). At that point, CoDec's answer is effectively the same whether contamination happened or not.

CL trajectory (text proxy)
Figure 5a - CoDec score across CL cycles on the GPT-2 proxy.
CL trajectory (TinyTS)
Figure 5b - CoDec-TS score across CL cycles on TinyTS.

Side noteDoes LMC extend the window?

As an exploratory, secondary task, we test whether Linear Mode Connectivity (LMC) stays informative once CoDec-TS can no longer discriminate. Unlike CoDec, LMC requires at least two saved checkpoints to compare.

By the third CL cycle in our numeric experiment, CoDec's score cannot distinguish between contaminated and control while LMC at the same point separates them (0.4 vs 0.1). This separation also stays during the fourth cycle. This small experiment may suggest that LMC extends the window past what CoDec offers.

LMC barrier extends detection beyond CoDec window
Figure 5.1 - The LMC barrier remains separated between contaminated and control after CoDec-TS loses discriminative power, extending the detectable window.

Scope

All figures above come from proxy setups - a pretrained GPT-2 and a small local TinyTS - both run under limited compute. This thesis will apply the same methodology to OpenTSLM, the actual target model. Compute budget, choice of benchmark datasets, and the scale of experiments will be expanded to draw more robust conclusions.

Deliverables