When Microsoft first announced its suite of AI-powered features, some users were rightly concerned about safety and privacy. This was mostly directed at the ‘Recall’ feature, which was capable of searching through everything that had ever been done on your PC. These objections prompted Microsoft to delay Recall for further adjustment, but a new privacy concern has now emerged tied to Microsoft’s Copilot AI.
A survey conducted by Concentric AI, published as part of the 2025 Data Risk Report, showed that Copilot accessed almost three million confidential records per organisation in the first half of 2025 alone. These figures were compiled based on data collected by Concentric AI customers and includes companies based in industries such as healthcare, financial services and government.
These sectors handle a great deal of personal data, including medical and financial records. The report shows that people who work with this type of data are also regular users of agentic AI like Copilot. The Concentric AI report indicates that organisations surveyed had an average of 3,000 interactions with Copilot AI.
Microsoft says on its website that Copilot: “complies with our existing privacy, security and compliance commitments, including General Data Protection Regulation (GDPR) and European Union (EU) Data Boundary.”
The company also says that Copilot operates with multiple protections. However, unless a company or organisation has specific guidelines and protections in place, Copilot AI could potentially have access to sensitive information, since data on interactions between the AI agent and the user are stored, including the specific prompts used and the Copilot responses.
As generative and agentic AI becomes increasingly entrenched in business operations, this could lead to Copilot becoming the vector for a catastrophic data breach unless companies and organisations develop specific countermeasures to prevent it.
Recent cyberattacks on prominent companies like Marks & Spencer and Jaguar Land Rover have highlighted the vulnerability of data held by such companies, which can include home addresses, emails, bank details and passwords. It has also shown how even massive organisations can be brought to their knees for days or weeks by such hacks.
If Copilot should prove to be an entry point for such attacks in the future, this could cause great harm to retailers, manufacturers and other services along with potentially making hundreds of thousands of people targets of fraud and identity theft.
