Sail MCP Server for Spark SQL

Name: Sail MCP Server for Spark SQL
Author: lakehq

@lakehq

About Sail MCP Server for Spark SQL

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

#sail #spark #sql #data-analytics

Basic information

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "sail": {
      "command": "sail",
      "args": [
        "spark",
        "mcp-server",
        "--transport",
        "stdio"
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Sail MCP Server for Spark SQL?

Sail MCP Server for Spark SQL is a drop-in Apache Spark replacement written in Rust, providing a Spark SQL and DataFrame API server over the Spark Connect protocol. It unifies batch processing, stream processing, and compute-intensive AI workloads on a distributed engine. Based on derived TPC-H benchmarks, it is ~4× faster and 94% cheaper than Spark.

How to use Sail MCP Server for Spark SQL?

Install pysail and pyspark-client, then start the server via the command sail spark server --port 50051 or using the Python API with SparkConnectServer(port=50051).start(). Connect a PySpark client session by setting the remote address to sc://localhost:50051.

Key features of Sail MCP Server for Spark SQL

Drop-in compatibility with Spark SQL and DataFrame API.
100% Rust-native engine with no JVM overhead.
Supports Delta Lake and Apache Iceberg table formats.
Integrates with AWS Glue, Unity Catalog, Hive Metastore, and OneLake.
Lightning-fast Python UDFs with zero-copy Arrow data sharing.
Lightweight, stateless workers for elastic scaling.

Use cases of Sail MCP Server for Spark SQL

Migrating existing PySpark workloads to a faster, cheaper engine.
Running batch ETL and data analytics pipelines.
Real-time stream processing using Spark-compatible APIs.
Compute-intensive AI workloads requiring high performance.

FAQ from Sail MCP Server for Spark SQL

How does Sail compare to Apache Spark?

Sail is a drop-in replacement that is ~4× faster and requires 94% less hardware cost, with zero shuffle spill per derived TPC-H benchmarks.

What are the runtime dependencies?

Python and the PySpark client package (pyspark-client). The server itself is a standalone Rust binary with no JVM required.

Where does data live?

Data can be read from and written to AWS S3, Azure, Google Cloud Storage, HDFS, Cloudflare R2, HTTP/HTTPS, Hugging Face, and local filesystems.

Is my existing Spark code compatible?

Yes, existing PySpark code works out of the box when the client connects over Spark Connect. An experimental compatibility check script scans your codebase for supported functions.