When the claim for judicial review was first lodged, reliance was placed in the Statement of Facts and Grounds on the first witness statement of Dr Anil Jain, dated 30 September 2018. Dr Jain is a professor in the Department of Computer Science and Engineering at Michigan State University in the United States. In his first witness statement he explains that the performance of AFR technology is affected by a number of variables, including “training datasets”: see para. 38(a). What this means is that the particular software which is used is “trained”. At para. 47, Dr Jain says that the accuracy of an AFR system depends to a considerable extent on the training dataset. He goes on to say, at para. 48, that AFR systems can suffer from training “bias”. At para. 49, he says that one cause of such training bias can be any imbalance in the demographic of subjects in the training datasets, resulting in the AFR system having a high false alarm rate or a high false reject rate for that particular demographic. At para. 51 Dr Jain states that it would appear that SWP were not aware of the dataset used to train the AFR system in this case. For that reason, he states, it would be difficult for SWP to confirm whether the technology is in fact biased. As a minimum for confirming whether an AFR system is biased, the database statistics, such as the number of males to females, and different races considered, would need to be known.
At the hearing before us Mr Beer placed particular reliance on a witness statement, filed in response to the first witness statement of Dr Jain, by Mr Paul Roberts, dated 23 November 2018. The first point to note about Mr Roberts’ statement is that it was obtained in response to the challenge brought in the present proceedings; it was not obtained proactively by SWP in order to fulfil the PSED.
Mr Roberts was employed by NEC (UK) Ltd as Head of Products and Solutions for Facial Recognition and is now employed by Northgate Public Services (UK) Ltd as
Head of Global Facial Recognition. Both companies are subsidiaries of NEC Corporation. One of the products of the business is the NeoFace Watch software with which the present case is concerned.
Mr Roberts makes the point in his witness statement that Dr Jain’s report gave a generic description of generally available AFR software in the marketplace but is not a fully accurate description of NeoFace Watch. He states that the NeoFace algorithm is trained in laboratories and, on a typically annual basis, a new version of the algorithm is released containing improvements from, amongst other things, additional training. No further training is carried out by the system in any customer environments: see para. 10 of his witness statement. At para. 20, Mr Roberts states that the precise makeup, scale and sources of the training data used are commercially sensitive and cannot be released. He was, however, able to share certain facts, which he sets out in the following paragraphs. At para. 22 he states that:
“To minimise any impact of bias as a result of gender, the NeoFace Algorithm training data set contains roughly equal quantities of male and female faces.”
At para. 24 he states that the NeoFace Algorithm training data includes a wide spectrum of different ethnicities and has been collected from sources in regions of the world to ensure a comprehensive and representative mix. He states that great care, effort and cost is incurred by NEC, as a socially responsible major corporation, to ensure that this is the case.
Dr Jain responded to Mr Roberts’ statement in a second witness statement dated 25 January 2019. He fairly acknowledges, at para. 15, that he cannot comment on whether AFR Locate has a discriminatory impact as he does not have access to the datasets on which the system is trained and therefore cannot analyse the biases in those datasets. He goes on to say, however, that bias has been found to be a feature of common AFR systems and that SWP themselves are not in a position to evaluate the discriminatory impact of AFR Locate.
At paras. 24-28, Dr Jain specifically responds to the witness statement of Mr Roberts. He expresses the opinion that what Mr Roberts says is not sufficient to be able to determine that the NeoFace algorithm is not biased towards a particular demographic group. To make this determination, he says, a thorough evaluation needs to be done of the demographic composition of the NeoFace algorithm training dataset. Dr Jain states at para. 28, without that information SWP are not able to assess whether the training dataset is biased or may be.
We acknowledge that it is not the role of this Court to adjudicate on the different points of view expressed by Mr Roberts and Dr Jain. That would not be appropriate in a claim for judicial review, still less on appeal. The fact remains, however, that SWP have never sought to satisfy themselves, either directly or by way of independent verification, that the software program in this case does not have an unacceptable bias on grounds of race or sex. There is evidence, in particular from Dr Jain, that programs for AFR can sometimes have such a bias. Dr Jain cannot comment on this particular software but that is because, for reasons of commercial confidentiality, the manufacturer is not prepared to divulge the details so that it could be tested. That may be understandable but, in our view, it does not enable a public authority to discharge its own, non-delegable, duty under section 149 .
Finally, we would note that the Divisional Court placed emphasis on the fact that SWP continue to review events against the section 149(1) criteria. It said that this is the approach required by the PSED in the context of a trial process. With respect, we do not regard that proposition to be correct in law. The PSED does not differ according to whether something is a trial process or not. If anything, it could be said that, before or during the course of a trial, it is all the more important for a public authority to acquire relevant information in order to conform to the PSED and, in particular, to avoid indirect discrimination on racial or gender grounds.
In all the circumstances, therefore, we have reached the conclusion that SWP have not done all that they reasonably could to fulfil the PSED. We would hope that, as AFR is a novel and controversial technology, all police forces that intend to use it in the future would wish to satisfy themselves that everything reasonable which could be done had been done in order to make sure that the software used does not have a racial or gender bias.
For the above reasons this appeal will be allowed on Ground 5.
Procedural matters
For the sake of completeness we mention briefly two procedural matters which were before us at the hearing of the appeal. They both relate to applications to refer to new material, which was not before the Divisional Court because it did not exist at that time. This Court has a discretion to permit such evidence under CPR 52.21(2). It is well-established that the discretion is to be exercised having regard to the criteria in Ladd v Marshall [1954] 1 WLR 1489: see Terluk v Berezovsky [2011] EWCA Civ 1534 , at [31]-[32] (Laws LJ).