INGO Experiences Adopting Microsoft Copilot

The Sandbox group — part of the Natural Language Processing Community of Practice — organized a session on Wednesday, July 10th, 2024, which showed how Microsoft Copilot has been used at three organizations. The session featured Eric Couper, Digital Development Director at the International Youth Foundation, Hanna Camp, Director of MEL Technologies at Mercy Corps, and Jay Chaudhuri, Director of Systems at Médecins Sans Frontières.

Their presentations and the robust audience discussion offered useful insights for organizations who are considering integrating Microsoft Copilot into their organizational processes. You can find the full session recording here.

Using MS Copilot at the International Youth Foundation

Eric Couper of International Youth Foundation kicked off the session with an overview of the various Microsoft Copilots, including:

• “Core” Copilot
• Copilot M365
• Copilot Pro
• Copilot Studio
• Copilots for Sales, Service, and Finance

He then gave a deep dive into each Copilot’s key features. This section of the blog post will delve into the key takeaways from Eric’s session and explore the various Copilot tools available.

Core Copilot

Microsoft’s “Core” Copilot is comparable to OpenAI’s ChatGPT and is included for free as part of most Microsoft Office business licenses. In fact, like all of Microsoft’s copilots, it builds on the ChatGPT-4 language model. It acts as a chat assistant, generating (and occasionally hallucinating) responses to user questions. In addition to this, unlike ChatGPT-4, it uses Bing to draw and reference information from the internet, providing citations when it does. A significant advantage of using Copilot in a business environment is its built-in commercial data protection policy. Two downsides are that it does not retain historical conversations and works less consistently than ChatGPT-4. Key features include:

Edge Integration: In additional to being available as a standard webpage, Copilot is integrated into the Microsoft Edge browser, offering functionalities like page summarization and ChatGPT-style conversations. These tools are integrated into the sidebar of Microsoft’s browser, allowing users to summarize pages, pull excerpts, and have ChatGPT-style conversations alongside their web browsing activities.

Notebook: This feature doesn’t have a chat element and it doesn’t retain historical conversations, but it supports larger text files than Core Copilot (​​instead of 4,000 characters, it can take 18,000 characters). This is a useful feature for tasks such as rewriting content, e.g. turning a speech or presentation into a blog post.

Data Protection: In Core Copilot, queries are wiped after generating responses, ensuring that user data does not feed into training the language model. This is an especially useful feature for organizations that work with sensitive data.

Copilot for M365

Copilot for M365 is designed for organizations that are already integrated into the Microsoft ecosystem. It offers an AI-powered Microsoft Office Suite and costs an additional $30 per user per month beyond an organization’s current M365 licenses. Most users report the MS Teams features related to summarizing and interrogating transcripts of meetings along with email summarizations to be the most valuable at present.

Copilot Pro

Copilot Pro offers the features of Core Copilot to individuals and organizations not fully integrated into the Microsoft ecosystem or who have a family (rather than business) license. Copilot Pro offers access to the walled-off, paid version of Core Copilot through a monthly subscription.

Copilot Studio

Copilot Studio allows users to create highly specific and customized chatbots. It builds on existing prescriptive/”hardcoded” chatbot  software, adding layers of intelligence that enable it to understand more ambiguous requests and to support tailored response pathways for specific use cases. This flexibility makes it useful for organizations looking to develop chatbots that can handle both predictable and unpredictable queries. The software allows your to feed in your own preferred websites and documents as reference material Copilot Studio is relatively affordable. It costs  $50 per nonprofit organization (not per seat) for the basic tier.

Copilot for Sales, Service, and Finance

Microsoft has designed specific Copilot tools for sales, service, and finance sectors to simplify work processes. 

Concluding Thoughts

One of the main reasons Eric advocated for the adoption of Copilot within his organization was its ability to protect his organization’s sensitive data. Furthermore, Copilot integrates seamlessly with Microsoft 365, and its various versions, such as Copilot Studio, offer unique functionalities tailored to different needs.

Piloting Microsoft Copilot 365 at Mercy Corps

Hanna Camp, Director of MEL Technologies at Mercy Corps, shared insights into their experience with Microsoft Copilot 365, particularly within the Office suite, including Teams, Outlook, Excel, and Word. This exploration was driven by the need to find safe and effective generative AI tools for MEL (Monitoring, Evaluation, and Learning) operations.

Initial Steps and Challenges

When the hype around ChatGPT peaked, Mercy Corps initially banned its use, prompted by data sensitivity concerns. However, they recognized that this wasn’t the best response, since their staff were turning to gen-AI for a reason.

To solve this, they convened a working group to think through the use cases that 1) its standard MEL tech could address and that staff must use when dealing with program data and 2) that its standard MEL tech could not address and would benefit from use of gen-AI, but that still required sharing sensitive data.

Identified Use Cases

The working group identified several use cases for generative AI that Mery Corp’s existing MEL technologies couldn’t handle, such as:

• Drafting narrative reports from spreadsheets
• Creating slide decks from reports
• Conducting deep qualitative analysis
• Reviewing documents for compliance with best practices

To address these needs, they decided to test Microsoft Copilot 365, despite not rolling it out organization-wide due to cost and unknowns.

Testing Copilot 365

Mercy Corps purchased a handful of test licenses and created a testing matrix based on identified use cases and Copilot’s advertised features. The preliminary results showed mixed outcomes:

• Word: Copilot struggled with large program documents, requiring multiple attempts and experienced analysts to generate usable summaries. Outputs were rudimentary, lacking the holistic analysis needed.
• Qualitative Data: Copilot didn’t perform well in coding qualitative data or interpreting typical MEL data spreadsheets, which are often not clean.
• Teams and Email: Auto note-taking in Teams was a significant time-saver, and Teams Copilot excelled at surfacing content from SharePoint. Email summaries were also helpful.

Utility and Cost Concerns

The overall utility of Copilot was found to be uneven, and it required more time and effort to use effectively than initially expected. While some features were highly beneficial, the high cost ($30 per month) made it less appealing. There was consensus that at a lower price point ($5-10 per month), it might be worth the investment.

Alternative Approaches

Mercy Corps is also experimenting with Azure AI Studio for building custom chatbots, which has shown more direct relevance to their specific needs compared to Copilot. This custom approach appears to be more successful in fulfilling their targeted use cases.

Conclusion

While Microsoft Copilot 365 shows promise, its current state presents challenges in utility and cost-effectiveness. Continuous improvements and potential price adjustments might make it more viable in the future. For now, Mercy Corps continues to explore both Copilot and custom solutions to meet their AI needs in MEL operations.

Why MS CoPilot has stalled at MSF

Jay Chaudhuri is the Director of Systems at Médecins Sans Frontières (MSF). In his presentation, he discussed the complexities of integrating Microsoft Copilot into MSF’s decentralized organizational structure. MSF operates as separate entities under a common brand, with a shared IT service center in Prague hosting their cloud infrastructure on a single tenant. This centralized approach brings both advantages and challenges, particularly in deploying AI tools like Copilot.

Risks and Governance

One major concern is the amplification of risks related to data stored in OneDrive, SharePoint, and various customized repositories. The governance around the administration of Copilot is unclear, raising questions about who delegates access and sets up controls. This ambiguity in administration poses significant risks that need addressing to ensure secure and effective AI deployment.

Opportunities for Copilot

Despite the challenges, there are notable opportunities for Copilot within MSF:

• Office Productivity: Enhancing productivity in everyday office tasks, particularly in fundraising and communications.
• Humanitarian Communication: Improving workflows related to translation and speeding up the publication process for documents awaiting translation.

Internal Server and Stateless Data Management

To mitigate risks, MSF has developed a stateless internal server where data and prompts are kept local and not shared externally. This setup provides a secure environment for initial AI integration, allowing MSF to explore AI capabilities while safeguarding sensitive information.

AI Risk-Management Framework

Jay emphasized the importance of the AI risk-management framework from NIST. This framework helps MSF evaluate risks at the application layer of AI technology rather than focusing on broader, more abstract concerns. It provides a structured approach to managing AI-related risks and supports internal discussions about AI deployment.

Testing and Future Plans

In the past two weeks, MSF has rolled out Copilot licenses to a select group of users across various offices as part of a test phase. This group includes staff in New York, field offices, and European locations. The goal is to evaluate the tool’s effectiveness and identify any potential issues over the next six months.

Embracing Risk and Opportunity

Jay concluded by acknowledging the inevitability of accepting some level of risk to capitalize on the opportunities AI presents. MSF aims to manage these risks through careful planning, possibly reducing the scope of Copilot’s use or relying on their internal portal for more sensitive tasks.

Conclusion

The integration of Microsoft Copilot at MSF illustrates the potential benefits and challenges of AI deployment in a decentralized organization. By addressing governance issues, leveraging risk-management frameworks, and conducting thorough testing, MSF is navigating the complexities of AI integration to enhance productivity and communication within their operations.

Successfully Integrating Microsoft Copilot: Key Insights and Tips for Organizations

Accessing Microsoft Copilot

To utilize Microsoft Copilot, you don’t need to be tied to a specific browser like Edge. You can visit copilot.microsoft.com from any browser. However, for enhanced features like secure corporate data protection, you must log in with your organization’s Microsoft credentials. The Edge browser offers a side-by-side experience for Copilot, though shifting to Edge if you currently use a different browser (e.g.,  Chrome) can be a minor inconvenience.

Key Use Cases and Benefits

1. Language Support: Copilot has been particularly beneficial for translations in organizations that work in multiple languages. This includes helping users clean up language and understand why changes are made. Some organizations are encouraging staff to use it to support both language refinement and language learning. 

2. Handling Qualitative Data: Organizations with large volumes of qualitative data find Copilot useful for summarizing and extracting key themes, thus making data more manageable.

3. Generating Deliverables: Copilot can generate takeaways from memos, create high-level summaries for client deliverables, and produce transcript summaries, providing a solid starting point even if some refinement is needed.

4. Query Transcripts: Useful for pulling verbatim quotes and creating query transcripts, which streamlines data analysis, though there are privacy concerns with allowing it to connect directly to key informant transcripts and potentially proprietary data.

Prompt Design

Crafting effective prompts is crucial for maximizing the benefits of Copilot. More detailed and specific prompts tend to yield better results. Iteration is key; refining prompts based on initial outputs can significantly improve the quality of responses. An Excel document or “prompt library” with sample prompts helps expand users’ understanding of Copilot’s capabilities.

Privacy and Internal Policies

Navigating AI regulation is complex, with different regions having varying guidelines. Organizations like MercyCorp and MSF emphasize the importance of following existing privacy policies and being aware of additional local regulations. Guidelines often stress the need to avoid using free versions of AI tools with sensitive information.

Managing Complex Data Structures

Breaking down tasks into manageable pieces is essential when dealing with complex data. Large tasks can overwhelm the AI. Instead, processing images and tables into text before feeding them into Copilot can yield higher-quality results. Users might need to switch between different Copilot tools to handle specific tasks effectively.

Voice to Text

Using voice to text can enhance interaction with Copilot. People tend to be more descriptive and provide better context when speaking compared to writing. This approach can make prompts clearer and more effective.

Handling Historical Data

Managing historical data with AI tools poses significant challenges due to privacy concerns, especially in fields like humanitarian aid. It’s essential to carefully consider what data to feed into the AI, as de-identifying information can be complex and sensitive. It’s unclear how to handle consent for use of historical data.

Training and Adoption

Training is critical for successful adoption of Copilot. Mercy Corp uses peer demonstrations, recording tasks performed by staff members, and sharing these recordings to facilitate learning. Tech Soup offers valuable training resources, and both Microsoft and Google provide introductory AI courses. MercyCorp integrates AI training into their existing MEL technology training programs.

Usage in Communications

Copilot is often leveraged by communications teams for language refinement and generating engaging content. For example, it has been used by one organization to create a series of engaging reminders for training sessions, streamlining the content creation process.

Conclusion

Microsoft Copilot offers some powerful tools to enhance productivity and data management, but organizations are still figuring out how to integrate it successfully and manage privacy and costs. By focusing on effective prompt design, adhering to privacy policies, and leveraging training resources, organizations can maximize the benefits of this AI tool. As technology continues to evolve, staying adaptable and informed will be key to harnessing the full potential of Copilot.

Next Steps

• Be sure to join the NLP-CoP if you’d like to stay connected and receive more information about events like this.