SC nixes rulings by NCLT, NCLAT based on fake AI citations

NEW DELHI: In the first case of its kind, Supreme Court on Thursday quashed orders passed by NCLT and NCLAT on the ground that they decided a case on the basis of AI-generated non-existent case laws and verdicts, cautioning that unchecked use of AI could distort the justice delivery system. A bench of Justices P S Narasimha and Alok Aradhe cautioned both the Bar and the Bench on the use of AI, saying the human element should remain in the driving seat and have "absolute and total control over the application and usage of AI". "For us... the production of fake, non-existent, and hallucinated material and its utilisation as precedents in law, is like the release of methyl isocyanate in the province of law and justice: invisible, insidious, and catastrophic by the time anyone notices," Justice Narasimha said. The bench said AI was not just an aid but "an alternative to our own thinking, reasoning and even decision making", making regulation essential. In the insolvency case, NCLT relied on AI-generated material and cited six non-existent SC judgments. The fake judgments also escaped scrutiny by NCLAT. Lawyers told the apex court they had not supplied the fake judgments and that the tribunal had obtained the material through its own research. "It is necessary for courts to adopt a zero-tolerance mode for producing, citing or using AI-generated precedents without verification," the bench said, adding that any decision relying even partly on fake or hallucinated AI material "is no decision in the eyes of the law" and must be set aside. The court directed the Bar Council of India to constitute a committee to deliberate on the problem of lawyers submitting fake or hallucinated AI-generated material as legal precedents. It warned that delegating thinking to AI could have serious consequences for the human capacity to distinguish truth from falsehood and justice from injustice.