Benchmarking AI models from an ethical angle involves ensuring that the evaluation processes promote fairness, transparency, and accountability. Fairness is crucial, requiring benchmarks to evaluate AI models across diverse demographics to prevent discrimination and bias. This includes using fairness metrics to detect and mitigate biases in data and algorithms, ensuring equitable treatment for all groups. This is one aspect of the talk. The other aspect is how leakage of benchmarking data happens allowing models to train on this data and thus produce higher scores that what is realistic. There are many safeguards taken to prevent this from happening and we will discuss the ethical aspects of this. Finally, we will cover a brief survey of AI benchmarks for ethics and where the industry is trending.

About the Speaker: Rajeswaran Viswanathan is head of AI CoE in Capgemini (www.capgemini.com) and also part of the AI Futures lab. His work involves working with multiple customers on GenAI implementation, including ethical aspects of GenAI solutions.