AI Bias in Litigation: What a Technical Expert Actually Tests

Introduction

Claims involving algorithmic bias are arising more often in technology disputes. They span a range of contexts: employment decisions made by automated screening tools, credit and insurance assessments, predictive risk-scoring systems used in public sector settings, and content moderation or recommendation systems that are alleged to have treated users differently on the basis of protected characteristics.

Each of these contexts raises technical questions that a technology expert may be asked to address. The legal framing will vary, depending on whether the claim sounds in discrimination, breach of contract, breach of statutory duty, or regulatory non-compliance. The technical questions, however, share a common structure: what the system does, how it produces its outputs, and whether its behaviour systematically disadvantages particular groups.

The sections below set out how bias claims are assessed from a technical perspective and what solicitors should be aware of when managing these cases.

What “bias” means technically

The word “bias” is used differently in legal, statistical, and engineering contexts, and this distinction matters when instructing an expert or reading an opposing report.

In statistical terms, bias refers to a systematic deviation between the outputs of a model and some reference outcome. A model is biased in the technical sense if its predictions are consistently wrong in a particular direction for a particular group. This is distinct from random error, which affects all groups equally and does not produce systematic disadvantage.

In the context of discrimination claims, the legally relevant question is often whether the system produces disparate impact: whether members of a protected group are treated less favourably by the system’s outputs, regardless of the developer’s intent. This is materially different from intentional discrimination, which requires evidence of purpose. A system can produce disparate impact without any discriminatory intent on the part of its designers, because the patterns present in training data, or the choice of features used to train the model, can encode historical inequalities and reproduce them at scale.

The distinction between disparate impact and intentional discrimination is important at a technical level because it shapes the analysis. Establishing disparate impact requires measuring the system’s outputs across protected groups and assessing whether the differential is statistically meaningful. Establishing intentional discrimination requires examining design choices, the selection of training data, and (in some cases) internal documentation about the development process. The two analyses are not mutually exclusive, but they call for different methods and different evidence.

It is also worth noting that algorithmic systems can produce differential outcomes through what are sometimes called proxy variables: features that are not themselves protected characteristics but that correlate with them. Postcode is a common example in credit scoring. A model trained on postcode data may not explicitly encode race or ethnicity, but if residential patterns reflect historical discrimination, postcode can function as a proxy. Identifying proxy variables and assessing their role in the model’s outputs is a central part of any bias audit.

Fairness metrics and why they matter

There is no single agreed definition of algorithmic fairness. Researchers and practitioners use a range of statistical metrics, and these metrics can conflict with one another in ways that are not immediately obvious.

Statistical parity (sometimes called demographic parity) requires that the model’s positive outcome rate is equal across protected groups. A credit model satisfies statistical parity if it approves loans at the same rate for applicants from different groups.

Equalised odds requires that the model’s true positive rate and false positive rate are equal across groups. This means the model is equally accurate for all groups, in the sense of catching genuine positives and avoiding false positives at equal rates. This is a stricter requirement than statistical parity.

Calibration requires that, among individuals to whom the model assigns the same predicted probability, that probability is equally accurate across groups. A given risk score should correspond to an actual outcome rate at a similar level for all groups, not for some groups and not for others.

These three metrics cannot all be satisfied simultaneously unless the base rates of the outcome are equal across groups. Where base rates differ (as they do in many real-world settings, often as a result of the same historical factors that give rise to discrimination claims), a system designer must choose which fairness criterion to optimise for. That choice is a technical decision, but it has normative implications, and those implications may be legally relevant.

An expert report can proceed on the basis of a single fairness metric without addressing the others. A report that concludes a system is “fair” because it satisfies statistical parity should be scrutinised to determine whether it addresses calibration and equalised odds as well, and whether the choice of metric was explained and justified.

How a bias audit works in practice

A bias audit begins with understanding what the system does and how it does it. This requires access to technical documentation describing the system’s architecture, the data used to train it, the features used as inputs, and the outputs it produces. In many commercial settings, this documentation is incomplete, and the expert may need to examine the system’s code directly or conduct empirical testing to understand its behaviour.

The data requirements for a bias audit are significant. To assess whether a system produces differential outcomes across protected groups, the expert needs access to the system’s outputs alongside data identifying the protected characteristics of the individuals affected. This data may not be held by the system operator, either because the operator does not collect it directly or because of data minimisation principles under data protection law. Where direct data is unavailable, it may be possible to use indirect estimation methods, but these introduce additional uncertainty that the expert must acknowledge.

Once the data is available, the analysis typically involves the following steps. First, the expert examines the distribution of outputs across protected groups, identifying any differential in outcomes. Second, the expert assesses whether the differential is statistically significant, applying appropriate tests that account for sample size and variance. Third, the expert investigates whether the differential can be explained by legitimate non-protected factors, or whether it persists after controlling for those factors. Fourth, where a differential does persist, the expert examines the model’s inputs and training data to identify potential sources of the bias.

This last step is particularly important and is often where the more technically demanding analysis sits. Modern machine learning models, including deep learning and ensemble methods, do not produce outputs through a transparent rule-based process. The relationship between inputs and outputs can be complex and difficult to interrogate. Techniques such as feature importance analysis, partial dependence plots, and counterfactual explanation methods can provide insight into how the model uses its inputs, but each technique has limitations and the results should be interpreted with care.

Where the system in dispute is a commercial off-the-shelf product rather than a bespoke development, the expert may face additional constraints. The vendor may decline to provide access to training data or model weights on grounds of confidentiality or intellectual property. In those circumstances, the expert’s analysis may be limited to empirical testing of the system’s observable behaviour, and the report must be transparent about what cannot be established from the available evidence.

The regulatory landscape

The regulatory framework governing algorithmic decision-making is developing rapidly, and the applicable rules depend on the system’s context of use and the jurisdiction in which it operates.

In the European Union, the EU AI Act (which entered into force in 2024) imposes risk-based requirements on AI systems, with the most stringent obligations applying to “high-risk” systems as defined in Annex III of the Act. High-risk categories include systems used in employment and worker management, credit scoring, education, and access to essential services. Providers and deployers of high-risk systems are required to conduct conformity assessments, maintain technical documentation, and implement bias testing as part of their quality management obligations. Where a dispute involves a high-risk system within scope of the EU AI Act, the conformity assessment and technical documentation are likely to be important evidential materials.

The United Kingdom has taken a different regulatory approach. Rather than a cross-sector AI statute, UK policy favours sector-specific regulation applied through existing regulators. The Information Commissioner’s Office, the Financial Conduct Authority, and the Equality and Human Rights Commission have each issued guidance on algorithmic decision-making within their respective remits. This fragmented approach means that the applicable regulatory standards in a UK dispute depend on the sector and the type of decision involved.

The Equality Act 2010 provides the principal legal basis for discrimination claims in England and Wales. Where an automated system is alleged to have produced outcomes that are discriminatory on the basis of a protected characteristic, the relevant provisions are those governing indirect discrimination: a provision, criterion, or practice that is facially neutral but that puts persons sharing a protected characteristic at a particular disadvantage. The technical evidence in such a claim must address whether the system’s outputs constitute a provision, criterion or practice, whether a group disadvantage exists, and whether any disadvantage is a proportionate means of achieving a legitimate aim. These are ultimately legal questions, but each has a technical dimension that the expert will need to address.

It is worth noting that the Equality Act does not require proof of intent. A system can produce discriminatory outputs without any discriminatory purpose, and this is precisely the scenario that algorithmic bias claims most commonly present.

Limitations of bias testing

Bias testing, however thorough, has inherent limitations that the expert must communicate clearly and that solicitors should understand before placing weight on a bias audit’s conclusions.

The most significant limitation is that bias testing can establish what a system does; it cannot in every case establish why. Where a model produces differential outcomes, the audit may identify candidate explanations (proxy variables, underrepresentation of certain groups in the training data, label noise in historical outcomes) but confirming the mechanism of bias with certainty requires access to the training data and development process in a level of detail that is not available in every case.

A related limitation concerns the training data itself. Machine learning models learn from patterns in historical data, and if that data reflects historical discrimination or structural inequality, the model may reproduce those patterns without any design flaw in the conventional sense. The bias, in this context, is in the world that the training data describes, not in the model’s processing of it. Whether a system should be treated as biased when it accurately reflects a biased historical baseline is a normative question that the expert can inform but cannot resolve.

Proxy variables present a particular challenge. Identifying that a feature functions as a proxy for a protected characteristic does not automatically establish that its use was unreasonable. Some features that correlate with protected characteristics also have genuine predictive value for the outcome the model is designed to predict. The expert’s role is to assess the extent to which the feature’s predictive value is independent of its proxy relationship, and to assess whether removing or adjusting it would produce a more equitable outcome without material loss of accuracy. This analysis is fact-specific and the conclusions will depend on the system’s purpose and the data available.

Finally, bias audits assess the system as it exists at the time of testing. They do not establish what the system was doing at the time of the events in dispute. Where a system has been updated or retrained since the relevant period, the expert may need to address whether the version tested is materially different from the version in use at the relevant time, and what evidence is available about the earlier version’s behaviour.

Practical guidance for solicitors

AI bias disputes present distinctive evidence preservation and case management challenges that are worth addressing at an early stage.

The system itself is a form of evidence. Unlike a document that is created at a point in time and does not change, an algorithmic system may be continuously retrained, updated, or replaced. Where litigation is anticipated or underway, the system operator should be asked to preserve the version of the system in use at the relevant time, including the model weights, training data, feature definitions, and output logs. This preservation obligation should be treated as analogous to document preservation, and the consequences of failing to preserve an updated or replaced system should be considered at the outset.

Output logs are often more obtainable than model internals and can be highly informative. Where the system in dispute has produced outputs for a large population, those outputs, combined with data on the characteristics of the population, may be sufficient to conduct a meaningful disparate impact analysis even without access to the model itself. Solicitors should request the preservation and disclosure of output logs as a matter of course in bias-related disputes.

When instructing a technology expert, it is worth being specific about the questions the expert is asked to address. Algorithmic bias is a broad topic, and an expert report that attempts to address all aspects without a clear framework is unlikely to be useful in proceedings. The key questions will typically include whether the system produces differential outcomes across relevant groups, whether those differences are statistically significant, what the potential mechanisms are, and (where the regulatory framework requires it) whether the system’s design and operation were consistent with applicable standards. Each of these questions calls for a different type of analysis, and the expert should be instructed with that structure in mind.

It is also worth considering at an early stage whether a single joint expert is appropriate or whether the issues in dispute require separate party experts. Bias assessments in complex systems can involve genuine methodological disagreements, particularly regarding the choice of fairness metric and the appropriate statistical threshold for significance. Where those disagreements are likely to be central to the dispute, independent expert evidence may provide a clearer basis for the court to adjudicate the issues than a single joint report that obscures the methodological choices made.

Early engagement with a technology expert can also assist with scoping disclosure requests. Knowing in advance what data and documentation a bias audit requires, including model documentation, training data, feature definitions, version histories, and output logs, allows disclosure obligations to be framed precisely and reduces the risk of relevant material being inadvertently deleted or overwritten before it can be examined.