How much of the training and test sets in TREC or MS Marco overlap? Can we evaluate on different splits of the data to isolate the extrapolation performance?
In this episode of Neural Information Retrieval Talks, Andrew Yates and Sergi Castella i Sapé discuss the paper "Evaluating Extrapolation Performance of Dense Retrieval" byJingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma.
📄 Paper: https://arxiv.org/abs/2204.11447
❓ About MS Marco: https://microsoft.github.io/msmarco/
❓About TREC: https://trec.nist.gov/
🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ
Timestamps:
00:00 Introduction
01:08 Evaluation in Information Retrieval, why is it exciting
07:40 Extrapolation Performance in Dense Retrieval
10:30 Learning in High Dimension Always Amounts to Extrapolation
11:40 3 Research questions
16:18 Defining Train-Test label overlap: entity and query intent overlap
21:00 Train-test Overlap in existing benchmarks TREC
23:29 Resampling evaluation methods: constructing distinct train-test sets
25:37 Baselines and results: ColBERT, SPLADE
29:36 Table 6: interpolation vs. extrapolation performance in TREC
33:06 Table 7: interplation vs. extrapolation in MS Marco
35:55 Table 8: Comparing different DR training approaches
40:00 Research Question 1 resolved: cross encoders are more robust than dense retrieval in extrapolation
42:00 Extrapolation and Domain Transfer: BEIR benchmark.
44:46 Figure 2: correlation between extrapolation performance and domain transfer performance
48:35 Broad strokes takeaways from this work
52:30 Is there any intuition behind the results where Dense Retrieval generalizes worse than Cross Encoders?
56:14 Will this have an impact on the IR benchmarking culture?
57:40 Outro
Contact: castella@zeta-alpha.com