Breaking AI & Tech News Analyzed

AI‑Powered Sextortion Sparks Urgent Call for Photo Removal in UK SchoolsChild‑safety specialists and the National Crime Agency (NCA) have highlighted a growing threat: criminals are exploiting generative AI to manipulate pupils’ photos into sexually explicit images and then blackmail schools for cash. The warning follows a recent incident in which a secondary school’s website was used to harvest images that were transformed into illegal content.How AI Is Used to Manipulate Pupils’ Photos for BlackmailThe Internet Watch Foundation (IWF) identified an unnamed UK secondary school that received a blackmail package containing AI‑generated child sexual abuse material (CSAM). The perpetrators scraped the school’s online galleries, ran the pictures through AI tools, and threatened to publish the fabricated images unless a payment was made. The IWF created a digital hash of the images and shared it with major platforms to block re‑uploads.Scale of the Threat: Images, Reports, and Growth Rate150 images from the school incident could be classified as CSAM under UK law.The Report Remove service logged 394 sextortion reports from under‑18s in the past year – a 34% increase on 2024.Criminal gangs operating from West Africa, particularly Nigeria, are identified as the primary perpetrators.Implications for School Safeguarding and PolicyThe Early Warning Working Group (EWWG) issued guidance urging schools to:Remove face‑on photos; use distant, blurred, or back‑of‑head shots instead.Limit identifiable information such as full names.Apply strict privacy settings on websites and social‑media accounts.Conduct regular audits of all published images.Retain consent agreements and immediately involve police if an incident occurs.Jess Phillips, minister for safeguarding, called the trend a “deeply worrying emerging threat” and signalled that legislation on AI‑generated CSAM will be updated if needed. The Confederation of School Trusts (CST) said it will “carefully consider” the guidance while balancing the desire to celebrate pupils’ achievements.Future Safeguarding Measures and AI Regulation OutlookAnalysts expect tighter controls on AI models capable of producing explicit content, potentially extending the recent ban on possessing such models. Schools are likely to adopt more restrictive image policies, invest in AI‑detection tools, and collaborate with law‑enforcement to monitor digital fingerprints. As AI‑driven sextortion gains visibility, further legislative action and industry‑wide content‑filtering standards are anticipated.

California Lawmakers Target Big Tech Over Child Abuse MaterialAssembly members Maggy Krell and Buffy Wicks announced a new legislative effort aimed at giving California a clear legal pathway to sue social‑media companies that do not adequately police child sexual abuse material (CSAM) on their services.AB 1946: New Legal Pathway for Child‑Safety LawsuitsThe amended bill, known as AB 1946, was published on 6 April 2026. Key provisions include:Biannual independent audits of platform design choices for child‑safety risks, submitted to the state attorney general.Streamlined reporting mechanisms for users who encounter CSAM.Reduction of the current 30‑day response window to 48 hours for many harmful‑content cases.Mandatory human‑moderator review of any newly detected CSAM.Penalties collected by the attorney general to fund a survivor‑support fund.If passed by the end of the legislative session in August 2026, the law would take effect on 1 January 2027.Potential Financial Exposure for PlatformsRecent verdicts in California and New Mexico have already exposed Meta and YouTube to multi‑million‑dollar judgments for design‑related harms to children. AB 1946 could amplify those costs by:Opening the door to state‑level civil actions for failure to detect or remove CSAM.Imposing audit‑related compliance fees and possible fines that could run into tens of millions per platform.Redirecting legal‑defense spending toward platform‑safety engineering, as lawmakers argue.Shifting Landscape of Platform Liability in the U.S.Federal law currently shields online services from civil liability for user‑generated content, except for sex‑trafficking violations. AB 1946 challenges that shield at the state level, echoing a broader national trend where states are seeking to hold tech firms accountable for design choices that facilitate abuse. The bill also empowers the attorney general and local prosecutors to access platform data, a move that could set a precedent for other jurisdictions.What the Next Legislative Session Could Mean for Tech GiantsAnalysts expect intense lobbying from the tech industry as the bill moves toward a vote. If enacted, the legislation could:Force platforms to redesign recommendation algorithms that target minors.Accelerate the rollout of AI‑driven CSAM detection tools.Prompt other states to draft similar statutes, potentially leading to a fragmented regulatory environment.In the longer term, the success of AB 1946 may push Congress to revisit the federal safe‑harbor provisions, reshaping the balance between free expression and child safety online.

The UK's communications regulator has launched a formal investigation into Telegram, examining whether the popular messaging platform is failing to prevent the sharing of child sexual abuse material (CSAM) under the country's Online Safety Act. This significant regulatory action follows evidence from the Canadian Centre for Child Protection indicating that harmful content is allegedly present and being shared on the platform. Key Developments Ofcom has initiated an investigation into Telegram regarding alleged child sexual abuse material (CSAM) on the platform The investigation follows evidence from the Canadian Centre for Child Protection Telegram faces potential fines of up to £18m or 10% of worldwide revenue if found in violation This is part of broader regulatory actions against platforms failing to protect users Ofcom has also opened investigations into Teen Chat and Chat Avenue regarding child protection Data & Market Impact The investigation is supported by substantial evidence of harmful content on the platform. The Guardian identified 150 Telegram channels globally where AI-generated deepfake nudes were being created and shared, including in the UK. A report by AI Forensics found 24,671 Telegram users actively sharing non-consensual intimate images in Italy and Spain alone, with content often monetized through one-time fees or monthly subscriptions. Previous enforcement actions against filesharing services like Pixeldrain and Yolovit resulted in implementation of detection algorithms and platform blocking, demonstrating Ofcom's willingness to take decisive action against non-compliant services. Why This Matters This investigation represents a critical moment in the battle against online child exploitation. For users, particularly children and vulnerable individuals, this could mean greater protection from harmful content on one of the world's most popular messaging platforms. For businesses, it signals that regulatory bodies are increasingly holding tech companies accountable for content moderation, potentially reshaping how platforms approach safety measures. The UK's actions could influence global regulatory approaches, as other countries consider similar legislation. For Telegram, which has positioned itself as a privacy-focused platform, this investigation could force a difficult balance between privacy obligations and content safety responsibilities. Expert Insight The investigation highlights a fundamental tension in modern digital regulation: the balance between privacy rights and platform responsibility. Telegram's denial and framing of the investigation as a potential "attack on freedom of speech" suggests this case could become a landmark precedent for how privacy-focused platforms handle illegal content. The presence of AI-generated deepfakes adds a new dimension to this challenge, as automated detection becomes more complex. The fact that perpetrators were predominantly "young heterosexual men" and that content was monetized through subscriptions indicates a sophisticated ecosystem that requires multi-faceted regulatory responses beyond simple content removal. What Happens Next If found in violation, Telegram could face substantial financial penalties and potentially be blocked in the UK if it fails to comply. This case may prompt other regulators globally to initiate similar investigations. We can expect increased pressure on Telegram to enhance its detection algorithms and cooperation with law enforcement. The outcome could set important precedents for how other privacy-focused platforms approach content moderation. Additionally, this investigation may accelerate the development of more sophisticated AI tools for detecting both traditional CSAM and AI-generated deepfake content, potentially leading to industry-wide standards for content safety.

The European Parliament's decision to block the extension of a law that permits big tech firms to scan for child sexual exploitation on their platforms has created a legal gap that child safety experts say will lead to crimes going undetected. The law, which was a temporary measure allowing companies to use automated detection technologies to scan messages for harms, including child sexual abuse material (CSAM), grooming, and sextortion, expired on April 3.Google, Meta, Snap, and Microsoft have expressed disappointment over the EU's decision, stating that they will continue to voluntarily scan their platforms for CSAM. In a joint statement posted on a Google blog, the companies said, 'We are disappointed by this irresponsible failure to reach an agreement to maintain established efforts to protect children online.'Child protection advocates had warned that allowing the legislation to lapse would probably trigger a steep fall in reports of child sexual abuse. They point to a similar legal gap that occurred in 2021, when reports of such material from EU-based accounts to the National Center for Missing and Exploited Children (NCMEC) fell by 58% over a period of 18 weeks.The EU's decision to prohibit scanning will have ripple effects in other regions around the world, child safety experts said. Many internet crimes are cross-border, with perpetrators sending illegal images to people or targeting children in other countries. 'The offender can be anywhere in the world, but they could have unfettered access to minors in Europe now that there's legal uncertainty around those safeguards and protections to identify when a child is being groomed,' said John Shehan, vice-president at NCMEC.In 2025, NCMEC received 21.3m reports that included more than 61.8m images, videos, and other files suspected of being related to child abuse, from around the world. About 90% of these reports are related to countries outside the US.

The UK has witnessed a significant surge in online sextortion attempts reported by children, with a record 394 cases in 2025, representing a 34% increase from the previous year. The majority of victims, 98%, were boys aged 14 to 17.Sextortion involves a person being manipulated into sending explicit selfies or videos to a predator, who then threatens to publish them online unless the victim sends money or further intimate images. This crime has been linked to the suicides of several British teenagers who have taken their own lives after receiving extortion threats.Campaigners, including the Molly Rose Foundation and the Internet Watch Foundation, are calling on tech companies to take more decisive action to disrupt blackmail attempts. They argue that social media platforms, where initial grooming of victims often occurs, should introduce nudity-detection technology to prevent the spread of explicit content.The parents of 16-year-old Murray Dowey, who took his own life after being duped by criminals online, are suing Meta, the owner of Instagram and Facebook, for allegedly failing to implement adequate safeguards. The company continues to challenge the suit.The Report Remove service, which allows children to flag intimate images or videos that have appeared or could appear online, reported a 66% increase in under-18s seeking help in 2025, with 1,175 out of 1,894 reports classified as child sexual abuse material.Tech companies, including Google and Apple, have invested in protections against sextortion, but campaigners argue that more needs to be done. Google has implemented sensitive content warnings in its messaging app, while Apple has a communication safety system that warns under-18s about potentially explicit content.

California is taking a significant step in regulating the artificial intelligence (AI) industry by introducing new standards for companies seeking to do business with the state. This move directly contradicts former President Donald Trump's stance on keeping the industry as deregulated as possible. Governor Gavin Newsom signed an executive order on March 30, giving the state four months to develop AI policies that prioritize public safety. Companies hoping to secure contracts with California will be required to demonstrate policies that prevent AI from distributing child sexual abuse material and violent pornography. They must also show how their models avoid incorporating “harmful bias” and detail policies aimed at avoiding “unlawful discrimination, detention, and surveillance”. The order also directs the state to come up with best practices for watermarking AI-generated or -manipulated images and videos. Newsom emphasized California's commitment to innovation while ensuring that companies protect people's rights and do not exploit or put them in harm's way. California's actions are part of a broader trend of state-level attempts to regulate an AI industry that has raised public safety concerns and worries about the potential for job displacement due to automation. According to the New York Times, states have passed more than 100 laws to shield children from chatbots and to block AI companies from using copyright-protected material. The White House issued a national policy framework for AI in December that discouraged states from passing such regulations, with Trump's executive order calling for minimal regulation to allow U.S. AI companies to innovate freely. In response, the Justice Department established an “AI Litigation Task Force” to challenge state AI regulations.

A New Mexico jury has ordered Meta to pay $375m in civil penalties after finding the company liable for misleading consumers about the safety of its platforms and enabling harm, including child sexual exploitation. This verdict marks the first bench trial to find Meta liable for acts committed on its platform. The lawsuit, brought by New Mexico Attorney General Raúl Torrez, claimed that Meta executives knew their products harmed children but disregarded warnings from their own employees and lied to the public about the risks. The jury found Meta liable for violating New Mexico's consumer protection laws, specifically the Unfair Practices Act. The penalty of $375m is the maximum allowed under the law of $5,000 per violation. Meta has announced its intention to appeal the ruling, accusing Torrez of making 'sensationalist, irrelevant arguments.' Internal Meta documents and testimony revealed that company employees and external child safety experts repeatedly warned about risks on Meta's platforms. Evidence presented included details of a sting investigation, 'Operation MetaPhile,' which led to the arrest of three men charged with sexually preying on children through Meta's platforms. The New Mexico court also heard about deficiencies in Meta's reporting of crimes on its platforms, including the exchange of child sexual abuse material. Meta generated high volumes of 'junk' reports by overly relying on AI to moderate its platforms, making it difficult for law enforcement to investigate crimes. In the next phase of the legal proceedings, the attorney general's office will seek additional financial penalties and court-mandated changes to Meta's platforms to offer stronger protections for children. The state is seeking design feature changes, including 'enacting effective age verification, removing predators from the platform, and protecting minors from encrypted communications.' This verdict is seen as a 'historic victory' for children and families who have been affected by Meta's actions. The case may also open the floodgates to further litigation and regulation of social media companies.

The amount of AI-generated child sexual abuse material found online surged by 14% in 2025, according to the Internet Watch Foundation (IWF). The watchdog identified 8,029 AI-made images and videos of realistic child sexual abuse material (CSAM) last year.The IWF reported a 260-fold increase in videos, with 65% of the 3,443 videos classified as category A, the most severe type under UK law. This is compared to 43% of non-AI videos. Kerry Smith, IWF chief executive, emphasized that advances in technology should not come at the expense of a child's safety and wellbeing.The IWF also found that offenders are discussing the potential of AI systems to create more realistic and violent content, including adding audio to videos or manipulating imagery of real children. The UK government has announced plans to test AI tools to prevent CSAM and introduced legislation to ban AI models designed to generate child sexual abuse material.Eight out of 10 UK adults want the government to introduce legislation ensuring AI systems are developed with safety as a priority. The IWF operates a hotline and has a global remit to monitor child sexual abuse content.