Introduction to Apache Kyuubi

Apache Kyuubi is a multi-tenant SQL gateway for distributed data systems. It gives people and BI tools a stable JDBC or ODBC endpoint without requiring every user to manage Spark application lifecycles, cluster queues, or engine configuration.

Kyuubi separates a long-lived client endpoint from the processes that execute queries. This distinction is its main architectural contribution:

• Kyuubi servers authenticate clients, maintain protocol sessions, select engines, and forward operations.
• Kyuubi engines run SQL and consume compute resources. Spark SQL is the most common engine, while Kyuubi releases can also integrate with other query engines.
• External services still provide resource management, metadata, storage, authorization policy, and service discovery.

Kyuubi therefore does not replace Spark, YARN, Kubernetes, a Hive metastore, or object storage. It coordinates access to them.

Architecture #

A typical Spark-on-YARN deployment looks like this:

Beeline, BI tool, or application
            |
            | JDBC/ODBC using the HiveServer2 protocol
            v
   Kyuubi server pool
      |          |
      |          `----> ZooKeeper or another configured discovery service
      |
      | launch, discover, or reuse
      v
Spark SQL engine application
      |----> YARN for driver and executor resources
      |----> Hive Metastore for table metadata
      |----> HDFS or object storage for table data
      `----> Ranger or another authorization system

The Kyuubi server stays relatively lightweight because the engine performs parsing, planning, scans, joins, shuffles, and writes. Scaling server nodes increases connection and operation-handling capacity; it does not by itself add Spark executors.

The Apache Hive / Kyuubi Overview compares this design with HiveServer2 and Spark Thrift Server in more detail.

Follow a Query #

Suppose Beeline sends this statement:

SELECT region, SUM(amount)
FROM lakehouse.sales
WHERE sale_date >= DATE '2026-08-01'
GROUP BY region;

The request normally follows these stages:

1. The client opens a transport connection and performs the configured authentication exchange with a Kyuubi server.
2. Kyuubi creates a protocol session and determines the effective user, engine type, engine-sharing scope, and allowed configuration.
3. Kyuubi searches for a compatible running engine. If none exists, it submits a new engine application to YARN, Kubernetes, or the configured local environment.
4. The server waits for the engine to register and then forwards the operation.
5. The engine parses and plans the SQL, reads catalog metadata, checks authorization, requests compute resources, and accesses table files.
6. Kyuubi exposes operation state and logs while the client polls or fetches results.
7. Closing the client session releases its operation state. The engine may remain alive for reuse or stop according to sharing and timeout policy.

This explains two characteristic latency patterns. A cold session may wait for a Spark driver and executors to start; a warm session can reuse an existing engine and begin planning almost immediately.

Choose an Engine Share Level #

An engine share level defines which sessions may reuse one engine. Names and details should be checked against the deployed Kyuubi release, but the standard choices are:

USER is a common default because it balances reuse and isolation. It is not automatically the right production choice:

• CONNECTION limits cross-session interference but can flood the cluster with drivers.
• USER can strand resources when many occasional users each retain an engine.
• GROUP improves utilization but requires trustworthy group mapping and compatible authorization rules.
• Server-wide sharing places unrelated users in one engine process. Session-level SQL authorization must be strong, and administrators must prevent unsafe configuration, temporary-view, cache, and resource interactions.

Sharing is both a security boundary and a capacity decision. Do not select it only to reduce startup time.

Understand Engine Lifecycle #

The server and engine have independent lifecycles. A client can reconnect through another server node while a remotely deployed engine continues to run, provided the selected discovery and recovery behavior supports it.

An engine typically moves through these states:

requested -> submitted -> starting -> registered -> active -> idle -> stopped

Important controls include:

• Launch timeout: how long a session waits for resource allocation and engine registration.
• Idle timeout: how long an engine with no active sessions remains available.
• Session and operation timeouts: when abandoned client state is reclaimed.
• Engine lifetime: an optional upper bound that periodically replaces long-running engines.
• Failure limits and retry policy: whether Kyuubi retries launch or reports the failure to the client.

Long idle timeouts improve interactive latency but reserve driver memory and may retain executors, cached data, class loaders, and catalog state. Short timeouts save resources but make users pay the cold-start cost repeatedly. Measure arrival patterns and engine startup time before choosing a value.

Separate Configuration Layers #

A Kyuubi deployment contains several configuration scopes:

1. Server settings control listeners, authentication, discovery, server timeouts, logging, and the rules used to launch engines.
2. Engine settings control the Spark driver, executors, catalogs, extensions, classpath, resource queue, and SQL runtime.
3. Session settings affect one logical client session when the administrator allows them.
4. Statement settings are SQL commands such as SET, subject to engine and security restrictions.

Configuration can arrive from server files, engine defaults, environment-specific templates, JDBC URL parameters, and client properties. The exact precedence and allowlist behavior are version-dependent. Administrators should explicitly restrict client overrides instead of assuming every spark.* property is harmless.

For example, allowing a client to select a YARN queue may be useful:

#spark.yarn.queue=analytics-interactive

Allowing arbitrary archives, JARs, extensions, proxy users, event-log destinations, or catalog implementations can instead become a code-execution or privilege-escalation path. Treat configuration policy as part of the security model.

Connect with JDBC #

Kyuubi exposes a HiveServer2-compatible endpoint, so Beeline and many Hive JDBC tools can connect without a Kyuubi-specific wire protocol:

beeline -u \
  'jdbc:hive2://sql.example.com:10009/default;ssl=true;principal=kyuubi/sql.example.com@EXAMPLE.COM'

The URL has several distinct regions:

jdbc:hive2://host:port/database;session-properties?query-properties#engine-properties

Support and interpretation depend on the client and server versions. Keep passwords out of shell history and process arguments, use TLS when credentials or query results cross an untrusted network, and verify the server certificate and hostname.

For a complete secured example, the Kerberos Beeline-to-Kyuubi walkthrough explains this URL:

jdbc:hive2://kyuubi.hadoop.vip.mycompany.com:10010/default;ssl=true;principal=kyuubi/kyuubi.hadoop.vip.mycompany.com@PROD.MYCOMPANY.COM?kyuubi.session.cluster=analytics-prod#spark.yarn.queue=analytics-high-memory

It traces DNS and TLS, the ticket-granting ticket, service-ticket acquisition, SASL/GSSAPI, Kyuubi’s service keytab, YARN submission, and Hadoop delegation tokens.

Deploy Kyuubi #

Server Nodes #

Run multiple Kyuubi servers behind a stable endpoint or use the supported service discovery mode. Keep server nodes interchangeable:

• distribute the same Kyuubi version, engine artifacts, configuration, trust stores, and authentication material;
• give each node a unique advertised address where required;
• keep mutable query state in engines or supported shared services rather than on one server’s local disk;
• drain listeners before upgrades so existing operations can finish; and
• test mixed-version behavior before performing a rolling upgrade.

A TCP load balancer is simple for clients, while discovery-aware JDBC can expose the registered server pool directly. Neither mechanism guarantees that a failed connection’s in-flight operation is transparently replayed. Clients need bounded retries and must understand whether a statement is safe to resubmit.

Spark Engines on YARN #

For YARN deployment, Kyuubi submits an application with a driver and later requests executors. Verify:

• the submitter or proxy identity can use the selected queue;
• the engine can localize Spark, Kyuubi engine JARs, Hadoop configuration, keytabs, and catalog dependencies;
• the ApplicationMaster can register back to Kyuubi through firewalls;
• driver and executor hosts can reach HMS, storage, KMS, Ranger, and shuffle services;
• queue maximums permit the requested driver and executor shape; and
• engine logs and Spark event logs survive container termination.

A session stuck at engine creation is often a resource-allocation or callback problem, not a JDBC problem.

Engines on Kubernetes #

On Kubernetes, the corresponding controls are namespaces, service accounts, RBAC, pods, images, Secrets, ConfigMaps, persistent or object-backed logs, network policies, and resource quotas. Avoid putting long-lived credentials directly in images. Pin images by digest, control which session settings can influence pod templates, and separate trusted administrative engines from general user engines.

Integrate Catalogs and Storage #

Kyuubi does not store table metadata. A Spark engine may use Hive Metastore, Iceberg catalogs, JDBC catalogs, or other catalog plugins. The engine needs:

• matching client libraries and catalog extensions;
• network access and authentication credentials;
• compatible Hadoop filesystem implementations;
• object-store or HDFS credentials;
• encryption-key access where applicable; and
• warehouse and scratch locations with correct ownership.

Catalog libraries belong on the engine classpath, not merely on the Kyuubi server classpath. A server can accept JDBC sessions even when every query later fails because an engine lacks a catalog JAR or cannot reach HMS.

Build the Security Model #

Security spans multiple hops:

client -> Kyuubi server -> engine -> metastore/storage/authorization service

Each hop answers a different question:

• TLS protects transport and authenticates endpoints when certificates are validated.
• Kerberos, LDAP, PAM, or another configured mechanism authenticates the client to Kyuubi.
• Impersonation or proxy-user rules determine which identity launches the engine and reaches downstream Hadoop services.
• Delegation tokens and service credentials let a long-running engine access HDFS, HMS, KMS, and related services.
• SQL authorization, often through a policy plugin such as Ranger, decides which catalogs, tables, columns, rows, and operations a user may access.
• YARN queues or Kubernetes quotas constrain compute rather than data privileges.

Authentication does not imply authorization. Successfully obtaining a Kerberos service ticket proves an identity to Kyuubi; it does not grant access to a table.

Audit records should preserve at least the authenticated user, effective user, client address, server, engine application ID, session and operation IDs, normalized statement or statement hash, target objects where available, start/end time, outcome, and resource queue. Protect audit logs because SQL text may contain sensitive literals.

Operate and Observe the Service #

Monitor the gateway and engines separately.

Useful server signals include:

• open and rejected connections;
• active sessions and operations;
• authentication failures;
• engine launch requests, duration, timeout, and failure rate;
• request queue depth and RPC latency; and
• server heap, garbage collection, threads, and file descriptors.

Useful engine signals include:

• active sessions and statements per engine;
• driver memory and garbage collection;
• executor allocation and loss;
• SQL planning and execution duration;
• shuffle read/write, spill, skew, and failed stages;
• catalog and storage latency; and
• idle duration and termination reason.

Correlate Kyuubi session and operation identifiers with the Spark application ID. A single dashboard that reports only Kyuubi server CPU can look healthy while all queries are blocked on YARN capacity or object-storage throttling.

Set service-level objectives by phase. Connection establishment, cold engine startup, warm query startup, and query execution have different causes and should not be combined into one latency percentile.

Plan Capacity #

Estimate at least four independent limits:

1. Server connection capacity: concurrent transports, sessions, operations, and result-fetch traffic.
2. Driver capacity: the number and size of engines retained by the chosen sharing policy.
3. Executor capacity: peak and sustained resources required by active SQL.
4. Downstream capacity: HMS connections, object-store requests, HDFS handlers, authorization checks, and shuffle bandwidth.

For USER sharing, a rough driver-memory bound is:

concurrently retained user engines
  * (driver memory + driver overhead)

This is separate from executor demand. Hundreds of mostly idle user engines may exhaust queue application limits or driver memory before consuming substantial executor CPU. Engine pooling, carefully chosen idle timeouts, and group sharing can reduce that cost, but each changes isolation.

Protect interactive workloads with queue limits, statement timeouts, admission control, and maximum result sizes. A JDBC client that fetches millions of rows through Kyuubi is usually worse than writing the result to governed storage.

Diagnose Common Failures #

The Client Cannot Connect #

Check DNS, load balancer health, listener address, TLS trust and hostname validation, authentication configuration, and whether the selected server is registered. A TCP connection refusal occurs before engine selection.

Authentication Succeeds but the Engine Does Not Start #

Inspect the Kyuubi server log and cluster-manager application state. Common causes are queue denial, exhausted application limits, missing engine artifacts, invalid Spark configuration, unavailable container images, callback-network restrictions, or an identity that cannot submit applications.

The Engine Starts but SQL Cannot Resolve Tables #

Check the engine’s catalog configuration and classpath, HMS URI, DNS, TLS or Kerberos credentials, warehouse path, and catalog version compatibility. Changing only the Kyuubi server classpath will not repair a remote engine.

Queries Fail with Permission Errors #

Identify which layer denied the request:

• Kyuubi authentication or proxy-user policy;
• SQL authorization policy;
• HMS authorization;
• HDFS permissions or ACLs;
• object-store IAM;
• encryption-key policy; or
• YARN/Kubernetes resource policy.

Do not hide the failure by giving the Kyuubi service identity broad storage access. Verify the authenticated and effective identities end to end.

Warm Queries Become Slow #

Separate planning from execution. Look for HMS latency, stale statistics, driver garbage collection, too many sessions sharing one engine, executor starvation, skew, shuffle spill, object-store throttling, or oversized result fetching. Restarting the server may temporarily mask an unhealthy engine without identifying the cause.

Clients Lose Results During Failover #

Server discovery makes another endpoint available, but it cannot make arbitrary SQL operations replay-safe. Determine whether the operation completed, use application-level idempotency for writes, and avoid blind retries of INSERT, MERGE, or DDL.

Compare the Main Options #

Kyuubi adds operational machinery and is not automatically faster than Spark Thrift Server. Its value comes from engine isolation, lifecycle management, routing, shared security controls, and a stable service boundary.

Decide Whether Kyuubi Fits #

Kyuubi is a strong fit when:

• analysts, dashboards, notebooks, and applications need a stable SQL endpoint;
• Spark or another supported backend is already the desired execution engine;
• workloads require per-user, per-group, or per-connection isolation;
• administrators need centralized authentication, audit, configuration policy, queue routing, and engine lifecycle management; and
• cold-start latency can be balanced against the cost of retained engines.

It may be unnecessary when all workloads are scheduled jobs, one trusted team can use a single Spark Thrift Server, or another query service already provides the required latency, concurrency, governance, and ecosystem support.

The useful mental model is: Spark supplies computation; catalogs describe data; the cluster manager allocates resources; Kyuubi turns those components into a managed, multi-user SQL service.