AI: the principle of fairness of data processing and the issue of data bias

As we know, Artificial Intelligence systems have the ability to analyze huge amounts of data and they require them during the learning and testing phases. In relation to machine learning techniques, it is often said that the more training data that can be put into a model, the better results it will produce.

However, the quality of learning data can in many cases be substantially more important than quantity. When training a given algorithm, it is essential that the selection of training data is representative and contextualized with respect to the task the model will have to solve in the future.

To understand this, let's analyze a practical example.

An American hospital in the mid-90s undertook an experiment with the aim of creating an accurate mathematical model to classify the risk of complications or death for patients suffering from pneumonia using machine learning software. [1] The aim was to develop a tool to be used so that high-risk patients could be admitted to hospital while low-risk patients were treated as outpatients. To the great surprise of doctors, the results showed that the algorithm assigned patients suffering from both asthma and pneumonia a low risk category. Although these patients were at greater risk of complications, they also had higher survival rates. What the model failed to recognize was that the apparently lower risk stemmed from the increased health care and more intensive treatment that these patients received. [2]

This example makes us understand the importance of the accuracy of the data as well as knowledge of the sector to which it belongs. This concept is often expressed with the maximum garbage in, garbage out: algorithms trained on the basis of unbalanced datasets will produce a compromised output and this is in express violation of the principle of correctness of treatment ex art. 5 letter a) GDPR, as well as being contrary to the right not to be discriminated against.

It may happen, in fact, that AI systems learn from data that reflect the characteristics of a particular group of individuals and that this results in a substantial imbalance between the groups represented, such that some are represented more or less than others. As observed in the field of criminal justice, this imbalance may result in discriminatory results based on sex, race, age, health or other characteristics under analysis.

Reference is made, for example, to the now famous Supreme Court of Wisconsin case, State of Wisconsin v. Eric L. Loomis. Mr Loomis was sentenced to six years' imprisonment on the basis, in part, of the scoring result (relating to the defendant's likelihood of recidivism) provided by software called COMPAS, which was found to be profoundly discriminatory against African-Americans.

The risks seem to be very high also in the case of artificial intelligence applications in healthcare.

In October 2019 the journal Science published an article[3] in which it stated that a commercial algorithm used in almost every major U.S. healthcare system, as well as other institutions such as insurance companies, seemed to discriminate against AfroAmerican patients. This system in particular deals with the identification of patients with complex healthcare needs (so-called "high-risk care managament"): through this algorithm, patients with special healthcare needs are automatically flagged and, once selected, can receive more money.

It is therefore possible to predict which patients will benefit most from receiving extra care and this allows for efficient allocation of resources. In order to make this prediction, the algorithm is based on data on how much it costs for a health care provider to treat a patient: due to the disparity in access to health care, however, the dataset used shows that African-American patients systematically spend much less on their health care than equally sick white patients.

As a result, the number of black patients who are reported for additional treatment is reduced by more than half: "Less money is spent on Black patients who have the same level of need, and the algorithm thus falsely concludes that Black patients are healthier than equally sick White patients".

The UK Information Commissioner's Office (ICO) in the Guidance on the AI Auditing Framework identified two main reasons why a Machine Learning system used to classify or make predictions about individuals could lead to discrimination:

1. Training data are not balanced: this is the case when, for example, there is an over-representation of the male population in the data. in this case the model will pay more attention to statistical reports concerning male subjects and less to those concerning female subjects. If a software that performs facial recognition activities has been trained with a much higher number of white male faces, it will be led to better recognize white male faces than other ethnic groups and genders;
2. The training data reflects past discrimination

According to the ICO, data controllers who decide to undertake such an activity must implement a series of controls based on three distinct phases, of which we report the most interesting steps:

• Adopt a data governance system that describes how the personal data used for training, testing or evaluation of the based system is as correct, accurate, relevant, representative and up-to-date as possible;
• Ensure that software developers have received appropriate training;
• Include in the DPIA an in-depth assessment of the risk of discrimination as well as the measures envisaged to mitigate/control it

(b) Identification: regularly monitor the impartiality of the treatment performed by the algorithm;

(c) Correction:

• Add or remove data on underrepresented / overrepresented groups, including thorough analysis / justification.
• Re-train the model with impartiality constraints.

In conclusion, it is pointed out that once trained, some machine learning systems can be considered real "black boxes" (we also hear about black box medicine) whose methods are accurate but difficult to interpret. The Owner may not be able to explain how some results were generated or what particular characteristics of a subject were important to reach a final decision.

This is expressly contrary to the principle of transparency of processing, as well as to the articles of the GDPR dedicated to the rights of data subjects: this will be the subject of the next article in this section.

Sources:

[1] Caruana R, You Y, Gehrke J, Koch P, Sturm M, Elhadad N. 2015 Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721–173.

[2] Datatilsynet – The Norwegian Data Protection Authority, AI & Privacy Report, January 2018.

[3] Ziad Obermeyer, Brian Powers, Christine Vogeli , Sendhil Mullainathan, Dissecting racial bias in an algorithm used to manage the health of populations, Science  25 Oct 2019: Vol. 366, Issue 6464, pp. 447-453.