Overview

This post is the summary of the following methods; they rank top on the CivilComments-WILDS benchmark:

Rank	Method	Paper
1	FISH	[2104.09937] Gradient Matching for Domain Generalization (Shi et al., ICLR 2022
2, 3	IRMX	[2206.07766] Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalization (Chen et al., ICLR 2023)
4	LISA	[2201.00299] Improving Out-of-Distribution Robustness via Selective Augmentation (Yao et al., ICML 2022)
5	DFR	[2204.02937] Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (Kirichenko et al., ICLR 2023)
6, 8	Group DRO
7, 12	Reweighting	[1901.05555] Class-Balanced Loss Based on Effective Number of Samples (Cui et al., CVPR 2019) is one example that uses this method; the reweighting method could date back to much earlier works.

Reweighting, IRM, and CORAL

IRM [2] and CORAL [3] are two extensions of the basic reweighting method by adding an additional penalty term on top of the reweighting loss; this term is based on some measures of the data representations from different domains to encourage the data distribution of different domains to be similar.

Reference

1. [2012.07421] WILDS: A Benchmark of in-the-Wild Distribution Shifts
2. [1907.02893] Invariant Risk Minimization (Arjovsky et al.)
3. [2007.01434] In Search of Lost Domain Generalization (Gulrajani and Lopez-Paz)