Auditing for Bias in Resume Search Engines

02 Nov 2020 - Arun Balajiee

Talk Speaker: Christo Wilson

Talk Date: 11/02/2020

In this talk Dr. Wilson talked about understanding the intrinsic bias of different resume search engines on job portals that recruiters use to find potential candidates for vacant positions. To understand this better, Dr. Wilson et al. shortlist 3 popular search engines and understand the behaviour of the search engines over large number of query search results ( > 500 k) through data collection and case control studies. Their work is presented in a CHI’ 18 paper. They define two types of fairness measurements – individual fairness ( equal opportunity for all people with similar qualifications) and group fairness ( disparate impact on a community of people with similar qualifications). Further, they specifically investigate the effects of gender based discriminations. They also try to understand if the search engines parse the information from the resume that is not explicitly asked as a question when candidate fill their job experience and qualifications profiles and use this inherent information to cause bias towards ranking of certain candidates higher than the others. They used a mixed random effects model to identify if there was individual fairness in the search results of the resume engines and found results that men were potentially ranked a few ranks higher (but not significantly higher) than women. Further, they run a Mann-Whitney U test with false positive rate correction methods to find that consistently men were ranked higher than women in the search results. This could mean an impact if the recruiter were to only see the first page of the search results, and women being ranked lower could be listed on a different page in search result that appears later. Finally, they didn’t significant levels of information extraction and bias based on those factors such as job qualifications, university where the candidate got their degree, etc. while ranking the search results.

In all, it was a great talk and I inferred that there was still a lot of possiblity for the resume search engines to improve their search algorithms and mitigate bias. However, there are the questions for discussions on how much information shared by the candidates should be used by the search engines that need to be considered. More importantly, it is important to know if these corporate companies running the resume search engines would be willing to share more information regarding the internal workings of their algorithms. It is a tight balance between fairness, privacy and equal opportunity for all people looking to prosper.