003.010 The Fairness of Open Events

Previously we stipulated that a championship event is "gender-fair" if men and women are given the same opportunity to compete in that event.  We quantified fairness using the likelihood of making the cut: in fair events, men and women have similar likelihoods of making their cuts.


While this approach to quantifying fairness works well for the age-limited events in age group championships, where athletes in an age group are of similar abilities, it may not accurately measure the fairness of open events.  Open events are prima facie open to athletes of all ages.  However, open events with qualifying times are more open to older athletes, who are more likely to achieve the qualifying time.  The statistical relation between athlete age and speed is different for men and women, as is their participation, which further complicates the issue.


In this post, we’ll first explain why our previous methodology does not work for open championship events.  Next we’ll present a refined methodology that does. A previous post in this series, entitled “Introduction to Time Standards”, provides required background.

Open Championship Events are Age-Limited.

Let’s consider an example from the 2019 Speedo Junior National Championships, an elite USA-S championship event targeted towards the top 18/Under athletes in the USA.  The meet accepts conforming LCM times and non-conforming SCY times. The qualifying times for the 100m Breastroke are 1:13.29L or 1:02.89Y for women and 1:06.09L or 55.79Y for men.


Running those qualifying times through our data results in the following graph.  Each bar reports the likelihood that a USA-S athlete of a given age and gender would qualify for the 2019 JNAT 100 Breast.  


The plot reveals that this open championship event has de facto age restrictions.  According to USA-S national age group records, no 10/Under athlete ever made those cuts.  And in our data, only one 12 year old athlete and twenty three 13 year old athletes made the cuts, out of over one million athletes.  Thus our chosen event is de facto age-limited to 14/Overs for all practical purposes, with older athletes having an exponentially greater likelihood of qualifying.

Infeasible Ages Distort Acceptance Likelihoods.

If the event is age-limited, we may well wonder what that limit is and how it affects the event’s gender fairness.  To address those questions, we’ll graph the overall event acceptance likelihood as a function of a hypothetical minimum eligibility age.  The leftmost values, labeled with age 6, depict the event’s acceptance likelihoods if the minimum age was 6. The rightmost values, labeled with age 18, depict the acceptance likelihoods if the minimum age was 18.


This graph shows that the choice of minimum age determines the apparent gender-fairness of the open event.  With the actual minimum age of 6, men are slightly more likely to qualify than women, but with the de facto minimum age of 14, women are slightly more likely to qualify than men.  Is the event fair or not? It’s hard to tell. Below we argue that this discrepancy between the actual and de facto minimum age implies that the gender-fairness of open events can’t be determined by comparing men’s and women’s overall qualifying likelihoods.

Infeasible Ages Distort Gender Fairness.

The relation between the event’s gender-fairness and it’s minimum age is further revealed by this log-ratio plot, which shows that the event is only gender-fair when the minimum age is 13.  It is unfair to men if the minimum age is 14 or 15, and otherwise unfair to women.



Thus, open events with qualifying times pose a challenge to our previous analysis, because overall 18/Under qualifying likelihoods do not capture the qualifying likelihood of athletes in the event’s target age range.   JNAT qualifying times implicitly exclude 12/Unders, and therefore it would be misleading to include 12/Unders in our fairness assessment. Any assignment of a de facto minimum age to an open event will be ad hoc and arbitrary.  Without a principled way to choose an open event’s de facto age range, we can’t credibly measure an event’s gender fairness.  For these reasons, we’ll need to develop a more sophisticated technique for analyzing the gender-fairness of open events with qualifying times.

Acceptance-Weighted Gender Fairness.

The crux of our improved technique is to weight the contribution of a given age by that age’s overall acceptance likelihood.  Ages that are more likely to make the qualifying times will contribute more to the overall gender-fairness of the event. Ages that have no possibility of making the cut will have no effect on the event’s gender fairness.


Let’s work through the analysis step-by-step for the 2019 JNATs 100 Breast.  The following table records for each athlete age: the percent women in our data (“Eligible”), together with the number that qualified (“Qualified”) and the number that would qualify in a gender-fair event (“Fair”).  The table starts at age 12, which is the youngest age in our data to qualify for the event.



Eligible

Qualified

Qualified

Qualified

Fair

Fair

Fair

Fair

Age

%Women

Women

Men

%Women

Women

Men

%Women

LogRatio

12

59

1


100

0.6

0.4

59


13

59

21

2

91

13.5

9.5

59

-2.0072

14

57

67

12

85

44.7

34.3

57

-1.4563

15

55

172

55

76

124.5

102.5

55

-0.9466

16

51

265

157

63

217.0

205.0

51

-0.4669

17

48

295

328

47

300.4

322.6

48

0.0348

18

44

221

346

39

249.6

317.4

44

0.2076

Total

56

1042

900

54

950.1

991.9

49

-0.1896


At age 12, one athlete qualified from a population that is 59% female; therefore the fair allocation of age 12 athletes is 0.59 women and 0.41 men.   At age 13, 23 athletes qualified from a population that is 59% women; therefore the fair allocation of age 13 athletes is 13.5 women and 9.5 men. At age 14, 79 athletes qualified from a population that is 57% women; therefore, the fair allocation of age 14 athletes is 44.7 women and 34.3 men.  Continuing in this manner, the final row of our table (marked “Total”) shows that 1042 women and 900 men qualified for the event (54% women) but the sum of the gender-fair allocations is 950.1 women and 991.9 men (49% women). Since more women qualified than would be gender-fair, the overall event is unfair to men.


The rightmost column in the table (“Fair LogRatio”) reports the natural logarithm of the ratio of men’s effective acceptance likelihood to women’s.  This quantity can be calculated from the gender fractions of two populations, without knowing the acceptance likelihoods. Let t_f be the fraction of women in the fair allocation and a_f be the fraction of women in the actual allocation.  Then the ratio of men’s to women’s acceptance likelihoods is [ t_f * (1 - a_f) / (1 - t_f) * a_f ]. The log-ratio is the natural logarithm of the ratio. Negative log-ratio values indicate that it’s easier for women to qualify, while positive values indicate that it’s easier for men. The last row in the table (“Total”) reports that men have exp(-0.1896) = 1/1.21 times the effective likelihood of qualifying for this event than women, or, equivalently, women have 1.21 times the likelihood of men.


Our example shows that no stipulated minimum age can capture the effective fairness of the event.  Above we calculated the effective acceptance log-ratio for the 200 Breast at -0.1896. Using a minimum age of 15 would create an event with the minimum acceptance log ratio (-0.0425), which is still well above the effective acceptance log ratio of -0.1896.


To choose fair qualifying times for an open championship event, we need to know the joint gender and age distribution of athletes, and the fraction of athletes that meet the qualifying times by age and gender.  These distributions are not publically available and so it is difficult to see how meet organizers would be able to reliably choose gender-fair qualifying times.

The Fundamental Challenge of Fair Open Events

We’ve seen that it’s difficult for the untrained eye to assess the gender-fairness of open events, because doing so requires knowledge of multiple hidden distributions.  Let’s consider another example from the 2019 JNATs: the 200 Freestyle, which is the fairest event in the meet. The qualifying times for the 200m Freestyle are 2:04.29L or 1:47.39Y for women and 1:54.29L or 1:38.39Y for men.


Running those qualifying times through our data results in the following graph.  Each bar reports the likelihood that a USA-S athlete of a given age and gender would qualify for the 200 Freestyle in the 2019 JNATs.  

Note that, like the 100 Breast, the 200 Free admits more 17/Over men and more 16/Under women.  The principle difference is that the fair event accepts relatively fewer 16/Under women than the unfair event, resulting in an effective acceptance log ratio of -0.0089.


Eligible

Qualified

Qualified

Qualified

Fair

Fair

Fair

Fair

Age

%Women

Women

Men

%Women

Women

Men

%Women

LogRatio

12

59

2


100

1.2

0.8

59


13

59

3


100

1.8

1.2

59


14

57

46

6

88

29.4

22.6

57

-1.7734

15

55

109

42

72

82.8

68.2

55

-0.7601

16

51

177

148

54

167.1

157.9

51

-0.1223

17

48

239

314

43

266.7

286.3

48

0.2017

18

44

219

332

40

242.5

308.5

44

0.1754

Total

56

795

842

49

791.4

845.6

48

-0.0089


Here is the same information in graphical form.  Note that columns sum to 100%. This shows that the larger number of 16/Under women combines with a higher acceptance likelihood to result in relatively more 16/Under women qualifying than men.


When the unweighted acceptance likelihoods are rolled up, JNAT 2019 appears to favor men because 18/Under men have exp(0.31) = 1.37 times the likelihood of being accepted to the meet than 18/Under women.  Yet the weighted acceptances show that the meet is fair overall, with men and women capable of making the cuts having nearly identical effective likelihoods of being accepted to the meet.



The discrepancy between the two metrics arises because USA-S has many more 14/Under women than men who have no chance of making the cut. Including these athletes in the unweighted metric lowers the women’s acceptance likelihood more than the men’s, which creates an illusion of inequity.  Weighting athletes by their acceptance likelihood reveals that among athletes who make the cuts, men and women are equally likely to make the cut.


We’ve refined our fairness metric to accommodate open championship events.  Does our revised metric change our prior analysis of age group championships, which showed that most age group championships were unfair?  Let’s revisit our previous analysis with our new metric.

Junior Olympics are Still Unfair.

This plots shows that  the new weighted acceptance ratios are nearly identical to the original unweighted acceptance ratios for the 2018 JOs in our data.  Both metrics show that LE and NT were the most fair JOs while MI and FL were the least fair.

Converting log-ratios to ratios shows that the original unweighted acceptance ratios for 2018 JOs are less than 1% greater than the new weighted ratios.  The largest increase is for CA2018L (1.1%). The smallest increase is for NI2018L (0.2%).


The new metric diverges from the old metric for events that accept a wide range of athlete ages, with differing acceptance likelihoods and gender distributions.  The JO events that we analyzed had relatively homogeneous age groups (9-10, 11-12, 13-14), so the new weighted acceptance ratio is nearly identical to the original unweighted acceptance ratio on these events.

Zone Age Group Championships are Still Unfair.

The new weighted acceptance ratios are also very similar to the original unweighted acceptance ratios for the 2018 Zones in our data.  Both metrics show that CZ was the most fair while SZ and WZ were the least fair.

For SZ2018L, we calculated the weighted acceptance ratio using actual 14/Under attendance.



Eligible

Qualified

Qualified

Qualified

Fair

Fair

Fair

Fair

Age

%Women

Women

Men

%Women

Women

Men

%Women

LogRatio

11

59

12

17

41

17.2

11.8

59

0.3147

12

59

76

70

52

85.7

60.3

59

0.1175

13

59

41

21

66

36.3

25.7

59

-0.3249

14

57

53

75

41

72.4

55.6

57

0.6106

Total

58

182

183

50

211.6

153.4

58

0.3272


Converting log-ratios to ratios shows that the original unweighted acceptance ratios for 2018 Zones are less than 1.03 times greater than the new weighted ratios.  The largest increase is for SZ2018L (2.8%). The smallest increase is for EZ2018Y (1.4%).

Again, the two metrics are nearly identical because we restricted our analysis to 14/Under athletes, who have relatively homogeneous age groups. Bottom line: age group championships are mostly unfair to women under both metrics.

Conclusion.

We’ve identified a limitation in using acceptance likelihoods to analyze the gender-fairness of open events.  To remedy the limitation, we’ve proposed a new fairness metric that weights the fairness of each age band by its acceptance likelihood.  The resulting metric is better able to quantify the gender-fairness of open events. Applying our new metric to the 2018 age group championships shows that our prior conclusions hold - all but a handful are unfair to women.