+0.009 locally, −0.01 on the server
One ensemble improved on holdout and got worse on the server. Comparing subtasks and the runtime environment.
That is how macro F1 for content type changed for one ensemble of two logistic regressions in the Mediascope experiments. Better on holdout, worse after submission to the server. We rolled that variant back. Other configurations were checked separately, so this does not generalise to all ensembles.
The local code had three metrics: F2 for video intent, macro F1 for content type and token F1 for the title, combined with weights 0.35 / 0.30 / 0.35. I look at the parts separately, because a gain in one field can hide an error in another.
For example, “watch films tonight” is about video but has no title. An empty field is more useful than a random film. That is a separate task for HasTitle.
The local 20% of data was used both for evaluation and for tuning thresholds, so that holdout gradually takes part in choosing the solution. An independent final test needs a different set.
I also check the environment: whether original row indexes come back, whether libraries are available, whether there is network access. An external LLM did not help in a sandbox without internet, even when it gave a gain locally.