Embedded AI, the quiet rewrite of the deal, and whether “opt-out” still means anything
When did anyone agree to this? You buy a tool; a wiki, a ticketing system, accounting software, an office suite that does a job. You scroll past the terms, like everyone, and sign up. Then one morning the product opens with a shimmering new button inviting you to ‘ask the assistant.’ Nobody installed it. Nobody requested it. There is no checkbox that says no thank you; I’d rather not have an AI read over my shoulder. The assistant simply arrived, already running, already indexed, already finished with its first pass through your work.
The pattern repeats across the market, collaboration platforms, productivity clouds, accounting suites, and more. The assistant is allowed by default and cannot truly be turned off. An administrator may be able to block AI features module by module through a central toggle, but there is no per-user opt-out, none. It ships inside the mid- and upper-tier plans, switched on in batches across the customer base whether the people typing into those tools asked for it or not.
On its own that is a usability complaint. It becomes a privacy complaint the moment you notice what ‘ready’ requires. An AI assistant cannot summarize a page it has not read, so the content must be processed and indexed in advance. Vendor documentation typically confirms these systems store the content of connected files, along with usage data and permission settings, to power their search, chat, and agent features. Indexing is processing, and processing… not selling, not even using… is the regulated act. The EU GDPR (European Union General Data Protection Regulation) defines it about as broadly as a word can be defined: collection, structuring, storage, retrieval, consultation, use. Indexing a knowledge base to make it AI-searchable sits squarely inside that definition.
What the Assistants get right
Three concessions clear the easy objections out of the way, so the harder ones can be seen plainly.
First, permissions are respected. The better-built assistants inherit the existing permission model: if a user cannot see a restricted page, neither can the assistant when that user asks.
Second, for a long time the vendors did not train on the data. The common position was that customer data submitted to or generated by the assistant was never used to train, fine-tune, or improve any model, the vendor’s or its providers’. Prompts sent to external models travelled over encrypted connections, and those providers were contractually barred from training on them.
Third, controls exist: connector block lists, audit logs, data-residency pinning, and increasingly, an option to keep inference inside the vendor’s own cloud boundary so prompts never reach an external provider at all.
A careful reader could stop there and conclude the matter is handled. None of those three concessions, however, touches the part that actually bites.
Consent was never asked
The permission model governs who can see an answer. It says nothing about whether the people whose work is indexed and transmitted were ever offered a choice. Much of the world’s privacy law is built on the premise that the choice is the point.
Under the EU GDPR, where processing relies on consent, that consent must be free, specific, informed and unambiguous (Article 4(11)), expressed through a clear affirmative act. Recital 32 is emphatic that silence, pre-ticked boxes, and inactivity do not qualify. Article 7 adds further conditions; Article 6 requires a lawful basis in the first place. A feature that is on by default, cannot be declined by the individual it touches, and is bundled inseparably into the service is the photographic negative of “freely given, specific, unambiguous.”
India’s Digital Personal Data Protection Act, 2023 is blunter still. Section 6 demands consent that is free, specific, informed, unconditional, and unambiguous with a clear affirmative action, limited to the data necessary for the stated purpose, read by regulators as a bar on bundled consent. Section 5 requires an itemized, plain-language notice at or before the point of processing. The DPDP Rules 2025, notified on 13 November 2025, add Rule 8, which outlaws dark patterns and asymmetric “easy to switch on, hard to switch off” designs. A default-on assistant with a one-way door reads like the worked example Rule 8 was written to prohibit.
ISO/IEC 27701:2019, the privacy extension that turns ISO 27001/27002 into a Privacy Information Management System, converts these duties into auditable controls. For PII controllers, Clause 7 / Annex A requires a documented lawful basis (7.2.2), the ability to obtain and demonstrate consent where consent is the basis (7.2.4), processing tied to a specified purpose (7.2.1), transparency to data subjects (7.3.x), and a mechanism to withdraw consent (7.3.5). For a certified organization, ‘the vendor turned it on and our people have no off switch’ is not a defense; it’s a nonconformity.
One qualification keeps the argument honest. In most workplace deployments consent is not even the operative lawful basis: the customer organization is the controller and the vendor is the processor… and the organization usually relies on legitimate interests or contractual necessity to process employee content. That does not rescue the vendor, it relocates the duty. Legitimate interest still requires a balancing test, transparency, purpose limitation, and a right to object, and it places on the customer-controller the obligation to have authorized and documented the new processing in the first place. This is exactly where the next two problems live.
First Problem: It’s processed outside your walls
‘Processed outside the organization’ deserves precision, because the precise version is true and the loose version is not.
By default, many assistants route prompts and context to third-party large language models hosted by external providers. Vendor trust materials often confirm that when third-party models are used, data is sent outside the customer’s chosen region for AI processing; content leaves the vendor’s own cloud boundary. An in-boundary model option can prevent this, but it is typically off unless an administrator enables it, applies organization-wide, and is pinned to a single jurisdiction.
For a compliance program that is not a footnote but a chapter. GDPR Article 28 permits a processor to engage sub-processors only with the controller’s authorization, and requires the same obligations to flow down the chain; the controller is supposed to know who is in it. New AI sub-processors quietly added to power an assistant nobody requested strain that arrangement. GDPR Chapter V (Articles 44–49) governs international transfers: content pinned to an EU region for residency and then shipped to a model hosted elsewhere for inference is a transfer that needs a lawful mechanism. ISO/IEC 27701 Clause 8 / Annex B requires processors to record and disclose sub-processors (8.4) and to govern cross-border transfer and disclosure of PII (8.5), while ISO 27001’s change-management (A.8.32) and supplier-relationship controls (A.5.19–A.5.23) assume that a material change to how a supplier processes your data passes through your change process… not the vendor’s, unilaterally.
The sentence that survives cross-examination is this: an assistant nobody opted into causes regulated personal data to be processed by new external sub-processors, in new jurisdictions, under a lawful basis the customer was never given the chance to assess.
Second Problem: They changed the deal, and disclosed it afterward
This is where a strong argument becomes a sharp one. The market has begun living through a now-familiar arc. For a couple of years the message was that customer data is never used to train. Then the policy turned: vendors have begun announcing that, by default, they will use customer content and metadata, the substantive text people write inside the product, not merely usage signals, to train their AI offerings. Such changes can reach hundreds of thousands of customers at once. The defaults tend to tier in a revealing way: the most consequential collection switched on by default, the cleanest full opt-out reserved for the highest-priced tier, and lower tiers sometimes unable to opt out of certain categories at all. Collected data may be retained for years. And the opt-out is generally not retroactive: it halts future collection and starts deletion timelines, but does nothing about data already trained into model weights before the switch was flipped.
Consider the order of operations. The capability shipped first. The contractual permission to use the data for the new, expanded purpose was published afterward, with enforcement date months out, the most consequential setting defaulted on, and the cleanest exit reserved for those who pay the most. Vendor AI terms now commonly state that inputs and outputs remain the customer’s data and that subcontractors may not train on them, a sound protection, often published as the practice was already in motion, and partly overtaken by the very training program the older assurances had disclaimed. The body text of a Restricted page is, after all, still a page, and is in scope by default unless an administrator carves it out in time.
The relevant law here is no longer about consent boxes; it is about fairness, transparency, and purpose limitation. GDPR Article 5(1)(b) constrains repurposing: data collected to run a tool and then redirected to train a vendor’s models is processing for a new purpose that must be compatible with the original or separately justified and disclosed. Articles 13–14 require that purposes be disclosed at the time of collection, not announced a year later. Article 35 makes material new high-risk processing the trigger for a Data Protection Impact Assessment difficult to conduct before the fact when the fact has already shipped. India’s DPDP Act echoes the principle in Section 5(a), and ISO/IEC 27701 (7.2.1 and 7.2.6) treats a change of purpose as a governed event rather than a default. “Published after the fact” is not rhetoric; it is the literal sequence - feature, then index, then, later, the legal scaffolding and the expanded data-use policy that the earlier scaffolding had ruled out.
Where this lands
Default-on, no-individual-opt-out, all-or-nothing AI assistants sit in real tension with the consent, transparency, purpose-limitation, sub-processor, and international-transfer requirements of the GDPR and the DPDP Act, and with the auditable controls of ISO/IEC 27701 and 27001; and the shift to default data contribution for model training, disclosed only after the capability was already running, is close to a textbook purpose-limitation and transparency problem. The deal that was signed quietly became a different deal, disclosed once it was already true.
The remedy is governance, not outrage. Map the sub-processors. Pin or restrict residency. Enable the in-boundary model option where it exists. Scope Restricted spaces out of any training program now. Bring privacy notices, the DPIA, the ISMS scope, and the supplier register into line with what the tools actually do. And when a vendor adds an assistant nobody asked for, treat it as a material change to processing that requires the customer’s authorization… not a fait accompli with a forum thread for a complaints department.
Nobody hired the assistant. The organization is accountable for it anyway.
Citations
• GDPR (Regulation (EU) 2016/679): Art. 4(11) consent definition; Recital 32 clear affirmative act and no pre-ticked boxes; Art. 5(1)(a) lawfulness, fairness, transparency; Art. 5(1)(b) purpose limitation; Art. 5(2) accountability; Art. 6 lawful basis; Art. 7 conditions for consent; Arts. 13–14 information at collection; Art. 28 processors and sub-processors; Art. 35 DPIA; Chapter V, Arts. 44–49 international transfers.
• India — Digital Personal Data Protection Act, 2023, and DPDP Rules 2025: Section 5 notice (itemized, plain-language, at or before processing); Section 5(a) purpose limitation; Section 6 consent (free, specific, informed, unambiguous, unconditional, clear affirmative action; right to withdraw; bar on bundled consent); DPDP Rules 2025 (notified 13 Nov 2025), Rule 8 prohibition on dark patterns and asymmetric withdrawal.
• ISO/IEC 27701:2019 (PIMS, extending ISO/IEC 27001 & 27002): Clause 7 / Annex A (PII controllers): 7.2.1 purpose, 7.2.2 lawful basis, 7.2.4 obtain and record consent, 7.2.6 change of purpose, 7.3.x information to data subjects, 7.3.5 consent withdrawal. Clause 8 / Annex B (PII processors): 8.4 sub-processor disclosure, 8.5 sharing, transfer, and disclosure of PII. Annex D GDPR mapping; Annex E ISO/IEC 27018 cloud-PII mapping.
• ISO/IEC 27001 (Annex A controls): A.5.12 information classification; A.5.34 privacy and protection of PII; A.5.19–A.5.23 supplier relationships and cloud-service security; A.8.32 change management.
• Adjacent regimes: California CPRA (right to opt out of “sharing”; note B2B and employee nuances); Brazil LGPD (consent and legitimate interest, Arts. 7–10); Canada PIPEDA (meaningful consent); EU AI Act (transparency and risk-tiering obligations).
About the Author (Actors)
Peter vR Sternkopf, Vigilant Systems, LLC
This article was collaboratively developed using human expertise in information security, compliance, and AI governance, augmented by AI systems for research, analysis, and drafting. All concepts were directed, validated, and refined by a human actor (author) who takes full responsibility for the content and conclusions presented. Consequently, it’s now considered open source.