Transparency in Fairness

Oliver Thomas - ot44 at sussex.ac.uk

Predictive Analytics Lab (PAL), University of Sussex, UK

Algorithmic fairness definitions

## Mutual exclusivity by Bayes' rule

<small>
        $$
        \underbrace{\text{Pr}(Y=1|\hat{Y}=1)}_{\text{Positive Predicted Value (PPV)}} = \frac{\text{Pr}(\hat{Y}=1|Y=1)\overbrace{\text{Pr}(Y=1)}^{\text{Base Rate (BR)}}}{\underbrace{\text{Pr}(\hat{Y}=1|Y=1)}_{\text{True Positive Rate (TPR)}}\text{Pr}(Y=1) + \underbrace{\text{Pr}(\hat{Y}=1|Y=-1)}_{\text{False Positive Rate (FPR)}}(1-\text{Pr}(Y=1))}
        $$
        </small>

<section>
            Suppose we have FPR<sub>S=1</sub> = FPR<sub>S=0</sub> and TPR<sub>S=1</sub> = TPR<sub>S=0</sub> (<span class="highlight">equalised odds</span>), can we have PPV<sub>S=1</sub> = PPV<sub>S=0</sub> (<span class="highlight">predictive parity</span>)?
        </section>
        <section>
            YES! But only if we have a <span class="highlight">perfect dataset</span> (i.e. BR<sub>S=1</sub> = BR<sub>S=0</sub>) or a <span class="highlight">perfect predictor</span> (i.e. FPR=0 and TPR=1 for S=1 and S=0)
            <span class="citeme">Kehrenberg, Chen, NQ: Tuning fairness by marginalizing latent target labels, Oct 2018</span>
            <div style="height:20px;font-size:1px;"> </div>
            <span class="citeme">Roth, Impossibility results in fairness as Bayesian inference, Feb 2019</span>
        </section>

## Mutual exclusivity by Bayes' rule
        <small>
        $$
        \overbrace{\text{Pr}(\hat{Y}=1)}^{\text{Acceptance/Positive Rate (AR or PR)}} = \underbrace{\text{Pr}(\hat{Y}=1|Y=1)}_{\text{TPR}}\underbrace{\text{Pr}(Y=1)}_{\text{Base Rate (BR)}} + \underbrace{\text{Pr}(\hat{Y}=1|Y=-1)}_{\text{FPR}}(1-\text{Pr}(Y=1))
        $$
        </small>

<section>
        YES! But only if we have a <span class="highlight">perfect dataset</span> (i.e. BR<sub>S=1</sub> = BR<sub>S=0</sub>)

<span class="citeme">Kehrenberg, Chen, NQ: Tuning fairness by marginalizing latent target labels, Oct 2018</span>
        <div style="height:20px;font-size:1px;"> </div>
        <span class="citeme">Roth, Impossibility results in fairness as Bayesian inference, Feb 2019</span>
        </section>

Algorithmic fairness methods

Transparency in fairness

Results on the adult income dataset

- Analysis on the relationship feature

- Feature values of the minority group are transformed to match the majority group

- Here, the wife value is translated to husband

Results on the adult income dataset

Interpretable can be fair!

	original $X$		fair & interpretable $X$		latent embedding $Z$
	Accuracy $\uparrow$	Eq. Opp $\downarrow$	Accuracy $\uparrow$	Eq. Opp $\downarrow$	Accuracy $\uparrow$	Eq. Opp $\downarrow$
LR	$85.1\pm0.2$	$\mathbf{9.2\pm2.3}$	$84.2\pm0.3$	$\mathbf{5.6\pm2.5}$	$81.8\pm2.1$	$\mathbf{5.9\pm4.6}$
SVM	$85.1\pm0.2$	$\mathbf{8.2\pm2.3}$	$84.2\pm0.3$	$\mathbf{4.9\pm2.8}$	$81.9\pm2.0$	$\mathbf{6.7\pm4.7}$
Fair Reduction LR	$85.1\pm0.2$	$\mathbf{14.9\pm1.3}$	$84.1\pm0.3$	$\mathbf{6.5\pm3.2}$	$81.8\pm2.1$	$\mathbf{5.6\pm4.8}$
Fair Reduction SVM	$85.1\pm0.2$	$\mathbf{8.2\pm2.3}$	$84.2\pm0.3$	$\mathbf{4.9\pm2.8}$	$81.9\pm2.0$	$\mathbf{6.7\pm4.7}$
Kamiran & Calders LR	$84.4\pm0.2$	$\mathbf{14.9\pm1.3}$	$84.1\pm0.3$	$\mathbf{1.7\pm1.3}$	$81.8\pm2.1$	$\mathbf{4.9\pm3.3}$
Kamiran & Calders SVM	$85.1\pm0.2$	$\mathbf{8.2\pm2.3}$	$84.2\pm0.3$	$\mathbf{4.9\pm2.8}$	$81.9\pm2.0$	$\mathbf{6.7\pm4.7}$
Zafar et al.	$85.0\pm0.3$	$\mathbf{1.8\pm0.9}$	---	---	---	---

## Results on the CelebA dataset

- <span class="highlight">Left</span>: Examples of the spurious residual and non-spurious translation
- <span class="highlight">Right</span>: In the semantic attribute domain ... same conclusion (<span class="highlight">changes in the eyes and lips regions</span>)
<center>
  <table style="border-collapse: collapse; border: none;">
        <tbody ><tr style="border: none;"><td width="40%" colspan="4" style="border: none;"></td><td width="1%" style="border: none;"></td><td width="40%" style="border: none;"></td></tr>
        <tr style="border: none;">
            <td width="10%" style="border: none;">Non-spurious</td>
            <td width="20%" style="border: none;"><img src="images/celeba_res/006126.jpg" width=70% title="Im1"/></td>
            <td width="20%" style="border: none;"><img src="images/celeba_res/015365.jpg" width=70% title="Im1"/></td>
            <td width="20%" style="border: none;"><img src="images/celeba_res/015505.jpg" width=70% title="Im1"/></td>
            <td width="1%" rowspan="2" style="border: none;"><hr width="2" size="250"></td>
            <td width="30%" rowspan="2" style="border: none;"><img src="images/features.png" width=100% title="features"/></td>
        </tr>
        <tr style="border: none;">
            <td width="10%" style="border: none;">Spurious</td>
            <td width="20%" style="border: none;"><img src="images/celeba_res/006126res.jpg" width=60% title="RIm1"/></td>
            <td width="20%" style="border: none;"><img src="images/celeba_res/015365res.jpg" width=60% title="RIm1"/></td>
            <td width="20%" style="border: none;"><img src="images/celeba_res/015505res.jpg" width=60% title="RIm1"/></td>
        </tr></tbody>
    </table></center>

Problems?

Spurious and non-spurious visualisations are not exciting!

Residual unfairness (transferability)

## Contrastive on Adult Income dataset
- Connections to <span class="highlight">counterfactuals</span> based on the Nabi \& Shpitser's (2018) causal graph
 - To remove bias towards females, the direct effect of gender on income, as well as, the <span class="highlight">effect of gender on income through marital status have to be suppressed</span>
- Experimental results
<small>
  <table>
        <tbody><tr>
            <td></td>
            <td width="10%">Accuracy $\uparrow$</td>
            <td width="10%">TPR Diff. $\downarrow$</td>
            <td width="10%">FPR Diff. $\downarrow$</td>
        </tr>
        <tr>
            <td width="10%">LR (Real)</td>
            <td width="10%">$85.16\pm0.14$ </td>
            <td width="10%">$7.98\pm1.52$ </td>
            <td width="10%">$7.23\pm0.41$</td>
        </tr>
        <tr>
            <td width="10%">Kamiran & Calders LR (Real)</td>
            <td width="10%">$84.37\pm0.28$ </td>
            <td width="10%">$14.3\pm1.16$ </td>
            <td width="10%">$1.17\pm0.29$ </td>
        </tr>
        <tr>
            <td width="20%">LR (Real and NN contrastive)</td>
            <td width="10%">$85.01\pm0.25$ </td>
            <td width="10%">$14.80\pm1.90$ </td>
            <td width="10%">$8.20\pm0.51$ </td>
        </tr>
        <tr>
            <td width="20%">LR (Real and GAN contrastive)</td>
            <td width="10%">$82.48\pm0.44$ </td>
            <td width="10%">$4.95\pm3.67$ </td>
            <td width="10%">$3.94\pm1.33$ </td>
        </tr>
			</tbody>
    </table>
    </small>