Model Context Protocol (MCP) MSPaint App Automation

What is Model Context Protocol (MCP) MSPaint App Automation?

This project automates interactions with the legacy Windows application MSPaint using the Model Context Protocol (MCP). It leverages pywinauto to control Paint and fastmcp to define tools that an AI agent, powered by Google's Gemini model, can call via natural language instructions.

How to use Model Context Protocol (MCP) MSPaint App Automation?

Set up a Python 3.11 Conda environment, install dependencies from requirements.txt, and add your Gemini API key to a .env file. Then run the MCP client with python mcp_paint_app/mcp_client.py.

Key features of Model Context Protocol (MCP) MSPaint App Automation

• Uses pywinauto to control the MSPaint application.
• Exposes Paint automation functions as MCP tools via fastmcp.
• AI agent driven by Google Gemini model.
• Tools include open_paint, draw_rectangle, and add_text_in_paint.
• Accepts natural language instructions to trigger paint actions.

Use cases of Model Context Protocol (MCP) MSPaint App Automation

• Automate drawing shapes and adding text in MSPaint via an AI agent.
• Demonstrate how MCP can control legacy Windows applications.
• Enable generative AI to perform visual output through Paint.
• Serve as a template for integrating AI agents with desktop automation.

FAQ from Model Context Protocol (MCP) MSPaint App Automation

What is the Model Context Protocol (MCP)?

MCP is a framework that enables AI models to interact with external tools and resources in a standardized way.

What are the system requirements?

Python 3.11+, Conda (recommended), a Google Gemini API key, and packages: pywin32, pywinauto, fastmcp, python-dotenv, and google-genai.

How do I set up the Gemini API key?

Create a .env file in the project directory and add a line GEMINI_API_KEY=YOUR_API_KEY. The key is then loaded by python-dotenv.

What should I do if the coordinates are off?

Adjust the click coordinates in the code based on your screen resolution and window size. Use the provided debugging print statements to identify correct values.

What if the AI agent selects the wrong tool?

Review the system prompt and ensure the tool descriptions in the MCP server are accurate and specific enough for the agent to choose correctly.