rewrite this title Ready to talk to your PC? Here are all the upgrades coming to Copilot in Windows 11

Follow ZDNET: Add us as a preferred source on Google.

ZDNET’s key takeaways

Microsoft adds multimodal Copilot features to Windows 11.Talk to your PC, share your screen, and let it act for you.Stronger security with a cautious design and built-in guardrails.

Microsoft just announced a handful of significant updates to its Copilot AI in Windows 11, making an evolutionary jump from text prompts to multi-modal AI you can speak to, show what’s on your screen, and command to take actions on your behalf.

“We’ve seen the first evolution of AI in the form of chatbots,” Yusuf Mehdi, Microsoft’s EVP, said in a recent press briefing. “We’re on the cusp of the next evolution, not just in the chatbot, but integrated AI.”

Also: This new Copilot trick will save you tons of time in Windows 11 – here’s how

Microsoft wants to employ this integrated AI across Windows so you can speak to it naturally in Copilot Voice, allow it to “see” what you’re doing with Copilot Vision, and take actions on your PC, from anything as simple as opening an app to as complex as booking a plane ticket with Copilot Actions.

Yes, these features are rife with security concerns, but Microsoft is taking a different approach after its controversial Recall feature generated enough pushback to send it back to the drawing board. Essentially, Copilot will need explicit permission to do anything, its boundaries are clearly defined (as opposed to an “always on” AI), and the user can take control or revoke access at any time.

The new features are reserved for members of the Windows Insider program for now, but will eventually roll out to all PCs on Windows 11 — regardless of whether or not it’s considered a “Copilot+ PC.” Let’s take a look at what’s coming in the upgrade.

Copilot Actions

The biggest feature rolling out in Windows 11 is Copilot Actions, an AI agent that doesn’t just respond to your questions, but actually takes action on your behalf, including opening and closing apps, typing, scrolling, and performing complex chains of actions.

Microsoft’s goal here is to transform AI agents beyond the passive “question and answer” chatbot into digital collaborators that can carry out tasks for you on the PC: updating documents, writing and sending emails, or even booking airline tickets in the browser.

Microsoft is taking a more conservative approach to the way agents work in Windows 11, relegating the feature to a separate account, on a separate desktop called the “Agent Workspace,” with strict permissions to what it can access.

Also: You can test Microsoft’s new in-house AI image generator model now – here’s how

It starts with limited permissions and will prompt the user for any changes it wants to make to the device. Additionally, access can be turned off at any time. The intention is to provide a clearly defined boundary as to what the agent has access to, along with what it’s doing at any given time.

Once you grant the agent access, it will be able to see your local files, apps, and data to gain insight into what’s on your PC, allowing the user to use natural language to make requests in existing apps, documents, or files with no context, and have the agent automatically understand what you’re referring to.

Also: I unleashed Copilot on my Microsoft and Google accounts – here’s what happened

It’s a big jump forward in Copilot’s breadth of abilities, and it doesn’t come without significant security concerns. The last time Microsoft tried launching a feature with sweeping access to user data, it didn’t go well. Microsoft Recall went back in development and was relaunched with changes to its functionality.

‘Hey, Copilot’

Microsoft really wants you to talk to your computer, and Copilot’s voice support is geared to make your Windows 11 PC the “computer you talk to.” Microsoft says its goal here is to open up new ways of interacting with your PC outside of the keyboard and mouse — an interesting concept — but, in practice, it reveals a handful of obvious concerns.

Also: Copilot Chat arrives free for Microsoft 365 users – check if you have it

Copilot Voice is designed to bridge the gap between AI “prompting” — a skill all its own — and being actually useful for tasks relevant to the average user. This includes locating keywords in documents, tracking down an email, and finding the random folder where you downloaded that file, etc., all while leveraging natural language, not search keywords.

There is something to be said about decoupling prompting skill from effectiveness, as up until now, these two things were inherently connected. But it ultimately depends on how well it actually works at delivering results. This is important, because Copilot has had not-so-stellar results at doing even the most basic tasks in Windows.

Also: How to upgrade your ‘incompatible’ Windows 10 PC to Windows 11 – 2 free options

Then there’s the security issue. In an office full of workers dealing with potentially sensitive data, do we really want everyone talking at their PCs? Luckily, during a press briefing, I received confirmation from Microsoft that Copilot Voice is not replacing text input, it’s merely an additional feature that users can choose to use — or not.

Copilot Vision

Copilot’s new approach to multi-modal AI is being called Copilot Vision, which lets the AI “see” what you’re currently doing on the PC and give you suggestions, instructions, or information either verbally or through text. Vision became available for Windows Insiders to test earlier this year, but it is included as a facet of Copilot’s core functionality with the new features.

Also: Microsoft Copilot AI can now pull information directly from Outlook, Gmail, and other apps

Vision is not “on” all the time; instead, like Voice, it has to be invoked. It works in Windows, Microsoft Edge, or on mobile, and it can’t take action on your behalf. Instead, it just analyzes what’s on the screen and responds with directions, feedback, or summaries.

The design here includes some immediate security guardrails on its usability. For one, you have to manually select which app you want Vision to have access to, and you can only select up to two apps at once. At launch, it will only be available with voice, but Microsoft has confirmed that text will be available shortly after.

Also: Microsoft lets you pick a character for your AI – with its new Copilot Portraits feature

To activate, click the glasses icon in Copilot, then tell it which app to look at. A floating toolbar will pop up, and after a short voice prompt, you can ask questions in natural language. Again, it won’t take actions, but it can show you where something is located on the screen by activating its own cursor.

Microsoft 365 integrations

In addition to these new functionalities, Copilot can also now interface directly with Microsoft 365 (FKA Microsoft Office) apps, including OneDrive, Outlook, and Google Drive. Using Voice and Vision, Copilot can not only respond to prompts about your own documents (that you’ve granted access to), but create, export, design, and write documents in those apps — all via requests in natural language.

Also: Microsoft Copilot now offers Claude models – how to try them

This feature is also still in testing and currently available for members of Copilot’s Insider program, and it was announced earlier in October.

Get the morning’s top stories in your inbox each day with our Tech Today newsletter.