Using the paid M365 Copilot ($30/mo) Chat and Researcher agent, I recently discovered an interesting limit: Copilot is technically unable to retrieve more than 24 email messages. Ever.
We can't know if the answers I got from it are reliable but it seems like the Microsoft Graph API calls it makes and the tools Copilot has are missing the option to call the next page. So, a paginated response is missing all data beyond the first page.
I tried copilot agent once, and it just claimed that it accessed a website that should have been blocked by corporate firewalls and uploaded a bunch of proprietary data. Lots of very specific information about how it clicked on specific buttons of the website etc.
We raised a high priority ticket with MS and turns out that Copilot Agent lied about the entire thing because the website was blocked. It completely made it up.
The fact that we are supposed to use Copilot Agent for open-ended "research" is mind-boggling.
Try and search for 'recycle bin' and you get zero local results in Windows 11. Unless you've gone and manually added the desktop icon into you Start menu items.
30 years ago, Win95 introduced the Recycle Bin. Maybe, just maybe, you should have made it discoverable via the Start menu by now?
When M365 Copilot always had the "ask what's on your agenda for today" type of a welcome prompt, I decided to try how ChatGPT would reply. Whereas Microsoft has my email, calendar and files, its answer is just boring facts about tasks and activities.
ChatGPT, on the other hand, had none of those connections. Yet its answer was significantly better. Because based on our daily chats, it knew what was important to me and what I should be focusing on to pursue my goals.
This made me realize what kind of a threat OpenAI is to the likes of Google and MS. They don't need to gain access to your data. You are profiling yourself to ChatGPT in a way that your calendar and email never was. By having "private" discussions with the computer.
Could have been leveraging the new Agen Mode in Excel preview released as a Labs feature. Built on Claude and not GPT-5, this is what the MS teams behind apps like Excel, PowerPoint etc. are now doubling down on.
Possibly the Office Agent can become a V2 of what M365 Copilot was originally sold as. It's not at all surprising that it takes a few years to figure out what LLMs can really be used for in Office tools. Whereas asking for businesses to fund all these experiments with a $30pupm license is not the best move to create early adoption and fans...
The recent Copilot search that was rolled out to replace the traditional M365 search is actually pretty good. When you use that search UI specifically.
If only asking from Copilot in a chat, the results are less spectacular or reliable. That just shows how a chat UI isn't the best solution for everything. Yet taking the M365 Search terms as input for the LLM to translate into a query that doesn't require the user to know the exact keyword to search - yes, that's a clear step forward.
And Graph grounding remains one of the features behind the premium license. So, only ~2% of M365 users will currently see this benefit. Unless the pay-as-you-go pricing option has huge adoption that we don't have any leaked figures on yet.
The crazy thing is how the premium M365 Copilot has only gotten worse as the free Copilot Chat has been rolled out. When the AI assistant from Microsoft can't create a calendar entry in Outlook, what are they even doing with this tech at MS?
You either get A) hallucinations of "I created the calendar entry now" (it didn't) or B) an .ics file you need to download, then import to Outlook manually.
8 million active users after almost 2 years from making the $30pupm license available. Less than 2% of Microsoft 365 paying customers choosing to pay extra for Copilot. It's not difficult to see why Microsoft themselves have stopped reporting on AI revenue, as well as not disclosing any official numbers on M365 Copilot sales. Luckily, a source leaked these figures for Ed Zitron to report in his newsletter.
Part of the reason is Microsoft's borgy/corpo-confusing service levels.
I have a paid 365 account and couldn't determine from logging in or account info screens if I was on paid or just the freemium version with my 365 plan
In testing out what I did have access to with Copilot, it was incredibly bad compared to ChatGPT or Claude, so I decided not to pay for Copilot whenever I see an ad for it.
Funny how renaming standard Office apps to "Microsoft 365 Copilot" in mobile apps, web home page for office.com etc. did NOT make people realize they should buy an additional $30 plan called "Microsoft 365 Copilot" to actually get to use all features of Microsoft 365 Copilot.
If only MS could have asked Copilot whether their naming strategy makes sense.
The product naming is completely insane. In the span of a year my corporate portal changed name from portal dot office dot com, to Microsoft 365 dot com, and now its copilot something or other dot com.
The portal completely changed in design and the former portal functionality is hidden in a tiny search icon in a knock off chatgpt interface.
The average non-technical corporate user must be so confused.
The confusing nature of product naming is I think pretty typical of MS. It’s just like with personal accounts, where there is no Outlook or Hotmail domain for logging in, it’s still live dot com, and Windows Live hasn’t been a thing for at least 13 years.
And then they have differerent versions of copilot. The consumer one at copilot.microsoft.com, the office one at really hard to remember m365.cloud.microsoft/chat etc. It's really annoying.
And it used to be better. Copilot.microsoft.com would just redirect to the best version you had access to. But they dropped that for whatever reason.
Sharepoint has been a complete mess ever since its inception. It lacked any data organisation features so unless you had really strict processes and oversight it turned into a shitheap within months. This is still really the case.
Even now it's mostly useful as a teams backend. But even that doesn't solve the shitheap problem, as any team you create in teams, or every yammer group automatically creates sharepoint sites which are not very useful as a sharepoint site and clutter up search results.
Users often don't understand that when they share something in teams there is an actual sharepoint behind it and sometimes people fiddle with permissions there and then other people don't understand why they're not allowed to do X or Y because the controls aren't fully exposed in the Teams interface. It's such a mess.
.NET was always a clusterfuck of naming, as in the early 2000s Microsoft was slapping the .NET label everywhere. .NET Passport (now known as Microsoft accounts), .NET My Services, Visual Studio .NET (which was the same as regular VS, just with support for building C# and VB.NET apps added), .NET Server 2003 (Windows Server 2003), et weary cetera.
In this respect, as in many others related to .NET, Microsoft was inspired by Java. In the late 90s, Java wasn't just a language, it was a VM, runtime environment, enterprise platform (Java EE, now Jakarta EE), smartcard technology, remoting protocol, operating system, desktop environment, floor wax, dessert topping...
So Microsoft did with .NET what they now do with Copilot, right? GitHub Copilot (AI-assisted code completion and chat), Microsoft 365 Copilot (the suite), Microsoft Copilot (the chat thingy), Windows Copilot (the chat thingy, but directly on Windows), Copilot for Azure (LLM doing RAG), Copilot for Dynamics 365 (I assume something similar)...
I don't know, does it matter? What I mean is that the product only just, in the grand scheme of things, got rolled out. MS isn't under some instant pressure to make loads of cash off this new feature. Year by year, people will just become used to having a tool, part of their OS, that they can ask questions to to get some general information on whatever topic.
It's easy to pick an arbitrary date and point out the lack of profit. But why don't we pick a date a year from now? Or three years. Or five years.
I think over time users will come to assume that their computing device has, as part of the many tools on it, a tool that they can ask about stuff and get general answers.
Is it profitable right exactly now? No. But I suspect it will become completely commonplace in a few years, and not even worthy of note or comment for the average user.
MS isnt some cash-strapped startup that needs to post instant profits right this moment.
> MS isnt some cash-strapped startup that needs to post instant profits right this moment.
Hmm no, but they are clearly under pressure. Integrating Copilot Chat in office was a pretty big move, and the constant rebranding of everything also feels like they have zero vision and are losing control and scrambling to fix it.
I wonder if Copilot Chat in office will really convince users though. As it doesn't have access to Graph it will lose some of the most valuable 'intelligence'. And thus users will try it, get convinced it's pretty useless and be more hesitant to pay for a license in the future.
I’m pretty sure those figures aren’t global, since very few people at Microsoft have access to that kind of data outside their area of direct accountability.
The author had worked at Microsoft for 12 years before starting the company that Google acquired. So, not only does he have a fair amount of experience from both sides, he's also worked at MS during its "lost decade" i.e. the Ballmer years.
The similarities between what MS was going through then and what Google is now facing is to me the most interesting aspect of the story. Whether you're printing money via OS or search monopoly, it seems like both the direction and timing of what's going to eventually happen to your business are almost inevitable.
MS today is quite different from what it was 10 years ago when the writer left. By getting acquired by Google in 2020, it's almost as he travelled back in time, into another Big Tech company at that same stage of the enterprise lifecycle.
We can't know if the answers I got from it are reliable but it seems like the Microsoft Graph API calls it makes and the tools Copilot has are missing the option to call the next page. So, a paginated response is missing all data beyond the first page.
I vibe coded this page as "documentation" since obviously no official MS docs exist for anything like this: https://vibes.jukkan.com/copilot-search-gotchas.html
reply