Police mass AI surveillance pilot draws rebuke from privacy experts

Privacy concerns raised over Australian-first live AI police face scanning trial An Australian-first trial of live AI facial recognition technology by a police force has been criticised by privacy experts and legal advocates. The mass AI surveillance pilot has been rolled out in Western Australia and is being closely watched by law enforcement agencies across the country. Since the trial began in June, cameras linked to facial recognition software have scanned more than 130,000 people in Perth and Fremantle, comparing their faces against a police watchlist in real time. Western Australian police say the results are promising but questions have been raised about an opaque watchlist and trial site locations that some say may target First Nations people. In June, the Western Australian Police announced it would be the first police force to use live facial recognition technology on the public. Facial recognition technology has become increasingly commonplace. It includes simpler one-to-one systems that check whether a single face matches a particular identity, like those used to unlock a mobile phone. It also includes the more complex one-to-many systems like those used in Bunnings to compare faces of customers against a database of people suspected of activity including theft or abuse of staff. Since June, WA Police have been trialling that more complex type of system. Using tech company NEC's Neoface m40 product, a pilot program scanned more than 130,000 faces around Perth and Fremantle in the first week of the trial. It compared their faces against a watchlist of about 4,000 people, including people accused of serious offences, people who are missing, and people who may pose a risk to themselves or others, police said. WA Police celebrated the early results from the trial, which resulted in 33 alerts and 18 arrests. Of the faces scanned in the first week the algorithm falsely identified one person as being on the watchlist — a better result than the 2023 UK trial of the same technology. Since then, the pilot has continued for several weeks across multiple sites, but WA Police declined to release any more data about the trial saying more would come in a "co-ordinated matter". Technology has known accuracy issues, say experts Melbourne Law School associate professor Jake Goldenfein said facial recognition technologies have known accuracy issues with particular demographic groups. "Historically, these systems have performed worse for people of colour and for women, and especially worse for women of colour," he said. That is because the algorithms used in the systems are trained on a data set of predominantly white and male faces, and an under-representation of other demographics in the training set results in worse accuracy. Dr Goldenfein said the systems produced both false positives, which could mean incorrectly identifying someone as being on the watchlist, as well as false negatives, which could lead the technology to fail to identify someone being sought. When choosing the technology for its trial, WA Police pointed to the findings of a 2023 study by the UK's National Physical Laboratory which tested the NEC algorithm and found there were no statistically significant differences in false positives by gender or ethnicity when used under specific conditions. But the study found accuracy issues increased when the facial recognition system's settings were relaxed and once real-world variables crept in, including changes to picture quality, lighting, skin tone and subject age. Michael Birtwisle, an associate director of data and AI law and policy at the Ada Lovelace Institute in London, was called to give evidence at a 2023 UK parliamentary inquiry about the Metropolitan Police Service and South Wales Police's use of facial recognition technology. He said Australian police could not rely on the UK's trial data when applying the technology here, particularly given the lack of testing on First Nations faces. "It's a snapshot so it can only tell you [about] that version of that system, for that population," Mr Birtwisle told the ABC. WA Police did not respond to a question about whether NEC's technology had been tested or trained on First Nations faces prior to the trial. "WA Police is committed to transparency and is collating statistics throughout the trial," a spokesperson said. "The trial will assess demographics during the deployments and gauge whether or not there is a disproportionate impact on an ethnicity or group within the community." Legal service left in the dark Aboriginal Legal Service of Western Australia's (ALSWA) legal services director Peter Collins was on leave in June when he received an email from WA Police. It was from a police superintendent asking for "20 minutes" to discuss the new pilot. "This is the extent of the consultation: an email asking to meet two days before it is launched," Mr Collins said. Mr Collins said he thought that WA Police would want the input of the state's primary legal agency for First Nations people involved in the trial, given WA's high incarceration rates of Aboriginal people and the risks of over-policing on a vulnerable cohort. But he said the consultation was simply "an exercise in tokenism". "This was a done deal from the get go. No matter what ALSWA had to say about the pilot it would have gone ahead,"he said. Mr Collins said WA Police had not explained whether the technology was being used to identify children. A WA Police spokesperson told the ABC that the force had made "attempts" to meet with ALSWA. 'Tick a box exercise' WA Police completed an assessment of the trial's impact on privacy as part of its preparations. On a form for the privacy impact assessment released publicly by police, it declared the trial would not "primarily or especially affect Aboriginal people". Peter Collins said the trial was one of the growing ways the WA Aboriginal community was being over-surveilled by police. "It can only have been by design, not chance, that facial recognition technology has been rolled out in those suburbs and places where everyone is aware that significant numbers of Aboriginal people either live or frequent," he said. "It is easy to understand why many Aboriginal people in WA believe they live in a police state." A WA Police spokesperson said "the locations are identified ... based on policing need and policing objectives. Locations do not target any group of people or demographic." Privacy expert at Electronic Frontiers, John Pane, said the consultation was thin but the bigger problem was the law that guides it. "It grants extraordinary powers, with limited oversight, to a police agency to do incredibly invasive things,"he said. Mr Pane believes the privacy act needs to be amended urgently to keep up with the tech. No consultation on trial, says watchdog WA's privacy watchdog, the Office of the Information Commissioner (OIC), said the state police force did not invite it to take part in any "meaningful" consultation process for the trial. The watchdog launched an unusually public broadside about the dangers of such a program without appropriate guardrails in July. A WA Police spokesperson said it had briefed the OIC in April and invited the office to a demonstration of the technology. An OIC spokesperson told the ABC it turned down that invitation because it did not think it was appropriate to attend, given its role as an independent regulator and to avoid appearing as though it was endorsing the trial. Former Australian Privacy Commissioner Malcolm Crompton said the Information Commissioner's missive was a clear warning from the agency. "The police had messed up. They were caught, and I'll use the word caught,"Mr Crompton said. "And the Information Commissioner put a warning shot across their bows, very clearly." Mr Crompton, who now works as a privacy consultant, said the trial's privacy assessment was "feather-light" and should have undergone a proper consultation by an independent assessor. "[The] privacy impact assessment is the cornerstone of this particular thing," he said. "It's not an 'I'm going to wave this piece of paper in front of you and I've let you know it's happening consultation'," he said. 'It's going to catch more Aboriginal people' Associate professor Hannah McGlade is a human rights lawyer and Noongar woman based at Curtin University in Perth. She said she had deep concerns about the roll out of a program that would impact First Nations people more than other groups. "This new policing program is looking at people with outstanding warrants. We know Aboriginal people are over-represented here," Dr McGlade said. "The technology is being set up in well-known Aboriginal areas: Midland, Northbridge." The human rights law expert said WA Police needed to be more transparent about who it had chosen to put on its watchlist. "Should this sort of technology be used for people who have engaged in minor offending?" Dr McGlade said. "Now, if people are at risk to the public, there might be other arguments in favour of it, or more surveillance. But here we just don't know." Mr Birtwistle said the UK Home Office was currently considering laws that would limit inclusion on facial recognition watchlists to only those who had committed serious offences. The ABC asked WA Police for more information on the nature of the 16 arrests for outstanding warrants in the first week of the trial. It did not respond. Dr McGlade said the lack of transparency was unacceptable. "And why aren't we being told? I think we're not being told because we will see that … Aboriginal people are being picked up more and over-represented,"she said. "If they've got nothing to hide then they really should show us who's being charged." Dr Goldenfein said the WA trial could help shape how the technology was adopted elsewhere in Australia. "I hope that as Australia starts to look again at the management of AI technologies in a more holistic way, the use of these technologies by law enforcement and intelligence apparatus doesn't escape that regulatory oversight," he said. Other state police forces are paying close attention to the trial of the technology too, according to WA Police Commissioner Col Blanch. He told a press conference when the WA trial was announced: "My colleagues, other police commissioners in other jurisdictions, are watching this trial closely to see how it goes."