Bloomberg's experiment shows thatOpenAI GPT3.5 There is a clear racial bias in the use of fictitious names in resume sorting. This experiment randomly assigned equally qualified resumes to names that were associated with a particular race or ethnicity at least 90% from voter and census data.
When sorting these resumes 1,000 times, GPT3.5 tended to favor names of certain ethnic groups more often, violating a benchmark for assessing job discrimination against protected groups.advancedSoftware Engineer, Retail Manager, and Financial Analyst), names associated with Black Americans were least likely to be rated by GPT in the Financial Analyst and Software Engineer roles.OptimalShow candidate.
The experiment also showed that GPT had different gender and racial preferences in different positions. Although GPT did not consistently favor a particular group, it would select winners and losers in different situations. In addition, when testing with the less used GPT-4, obvious bias was also found.
In response to Bloomberg's detailed questions, OpenAI said that the results of using the GPT model "out of the box" may not reflect the way its customers use the model. Companies usually take further measures to mitigate bias when using its technology, including fine-tuning the software's responses, managing system messages, etc.
While generative AI techniques have attracted widespread attention for their use in the HR field, this experiment highlights the serious risk of automated discrimination when using these techniques for recruiting and hiring. Adjusting AI models for bias remains a major challenge for AI companies and researchers, and automated recruiting systems could further complicate corporate diversity efforts.