Apache Polaris and Apache Iceberg

Apache Polaris is a catalog service for Apache Iceberg tables. Apache Iceberg is an open table format for large analytical datasets. Neither project is a general-purpose computation engine:

• Polaris identifies tables, governs access, and coordinates changes to the current Iceberg metadata.
• Iceberg defines how table state, snapshots, manifests, data files, and delete files fit together.
• Spark, Flink, Trino, and other engines plan and execute computations.
• HDFS or object storage holds the Iceberg metadata and table data files.

Keeping those responsibilities separate prevents a common category error: Polaris is not an alternative to Spark. A Spark deployment can use Polaris as its Iceberg catalog.

Place Each Component in the Stack #

A typical deployment has four distinct layers:

SQL client, notebook, scheduler, or application
                         |
                         v
Compute engine: Spark, Flink, Trino, or another Iceberg client
             |                              |
             | Iceberg REST requests        | file I/O
             v                              v
Apache Polaris                         Object storage or HDFS
             |                              |
             | current metadata pointer     | Iceberg metadata,
             `----------------------------> | manifests, and data files

The control path and data path are separate:

1. The engine asks Polaris to resolve a namespace and table.
2. Polaris returns information needed to load the current Iceberg table metadata.
3. The engine reads metadata, manifests, and data files from storage.
4. The engine performs scans, joins, aggregations, and writes.
5. For a write, the engine asks Polaris to atomically commit a new table metadata version.

Table rows do not flow through Polaris. Consequently, adding Polaris does not provide compute capacity or make a slow query faster by itself.

Understand What Iceberg Adds #

Parquet, ORC, and Avro describe individual files. They do not define which files constitute a table at a particular moment. Iceberg adds that table-level state.

An Iceberg table is represented by a metadata tree:

Catalog entry
    |
    v
Current table metadata JSON
    |
    +--> schemas, partition specifications, properties
    |
    `--> snapshots
           |
           v
      manifest list
           |
           v
        manifests
           |
           +--> data files
           `--> delete files

Each committed snapshot describes a complete, consistent table state. Readers that loaded an older snapshot can continue using it while a writer prepares and commits a newer one.

This model supplies capabilities that a directory of Parquet files does not supply on its own:

• atomic table changes;
• snapshot isolation and time travel;
• schema evolution based on stable field identifiers;
• hidden partitioning and partition-spec evolution;
• file-level statistics for scan planning;
• row-level delete and update representations in table-format versions that support them; and
• optimistic concurrency between writers.

Iceberg normally treats data and metadata files as immutable. A write produces new files and then attempts to replace the catalog’s current metadata pointer. The new files become visible together only after that commit succeeds.

Understand What Polaris Adds #

Iceberg requires a catalog to map a logical table name to its current metadata and to coordinate commits. Apache Polaris provides that catalog through the Iceberg REST Catalog API.

Polaris is responsible for concerns such as:

• catalogs, namespaces, and table identifiers;
• the authoritative pointer to current Iceberg table metadata;
• atomic commit checks and conflict detection;
• authentication and catalog authorization;
• integration with storage credentials; and
• a standard REST interface usable by multiple engines.

The REST interface matters because each compute engine can implement one Iceberg REST client instead of implementing every catalog backend separately. It plays a role similar to the Hive Metastore Thrift API, but the table and transaction models are different.

Polaris does not:

• execute SQL;
• schedule YARN applications;
• run Spark executors;
• store table rows in its service database;
• replace Parquet, ORC, or Avro; or
• turn arbitrary files into an Iceberg table automatically.

Compare Polaris with Hive Metastore #

Polaris and Hive Metastore are both catalog services, but they should not be treated as interchangeable implementations of the same metadata model.

For a Hive expert, a useful approximation is:

HMS       = catalog plus Hive-oriented table and partition metadata
Polaris   = catalog and commit coordinator for Iceberg metadata
Iceberg   = table state that HMS-style tables traditionally lack

The approximation is not an API compatibility claim. Moving a table from HMS to Polaris requires an Iceberg-aware migration or registration procedure, not changing a Thrift URI to a REST URI.

Compare Polaris with Spark #

Spark and Polaris occupy different layers:

Their normal relationship is:

Spark driver
   |---- load and commit table metadata ----> Polaris
   |
   `---- schedule tasks --------------------> Spark executors
                                                   |
                                                   `--> Iceberg files in storage

The driver uses an Iceberg client to communicate with Polaris. Executors read and write the files selected by the planned Iceberg scan or write. Polaris is not in the executor data path.

Integrate Compute Engines #

Every engine needs three independent connections:

1. a network and authentication path to the Polaris REST endpoint;
2. a Polaris catalog identifier, called a warehouse by the Iceberg REST API; and
3. a supported method for accessing the Iceberg files in object storage or HDFS.

The examples below show the important integration boundary rather than a complete production configuration. Property names and supported Iceberg features change between engine releases. Use mutually supported engine, connector, Iceberg, and Polaris versions.

Integrate Apache Doris™ #

Apache Doris 3.1 and later can configure Polaris as an Iceberg REST catalog. Doris uses OAuth 2.0 credentials to obtain a Polaris token and can request temporary object-storage credentials from Polaris:

CREATE CATALOG polaris_vended PROPERTIES (
  'type' = 'iceberg',
  'warehouse' = 'analytics_catalog',
  'iceberg.catalog.type' = 'rest',
  'iceberg.rest.uri' = 'https://polaris.example.com/api/catalog',
  'iceberg.rest.security.type' = 'oauth2',
  'iceberg.rest.oauth2.credential' = '<client-id>:<client-secret>',
  'iceberg.rest.oauth2.server-uri' =
    'https://polaris.example.com/api/catalog/v1/oauth/tokens',
  'iceberg.rest.oauth2.scope' = 'PRINCIPAL_ROLE:doris_role',
  'iceberg.rest.vended-credentials-enabled' = 'true'
);

The Doris frontend resolves Iceberg objects through Polaris, while Doris backend or compute nodes read the selected data files from storage. Grant the Polaris principal only the required catalog privileges and configure the Polaris catalog with a storage role that can vend appropriately scoped credentials.

If credential vending is disabled, Doris needs its own storage credentials. That changes the storage authorization path but not the Polaris catalog path. Test reads and writes separately because an identity may be able to load table metadata while lacking permission to access or create data files.

Integrate Apache Flink® #

Flink uses the Iceberg connector and registers Polaris as an Iceberg REST catalog. A representative Flink SQL catalog definition is:

CREATE CATALOG polaris WITH (
  'type' = 'iceberg',
  'catalog-impl' = 'org.apache.iceberg.rest.RESTCatalog',
  'uri' = 'https://polaris.example.com/api/catalog',
  'warehouse' = 'analytics_catalog',
  'credential' = '<client-id>:<client-secret>',
  'scope' = 'PRINCIPAL_ROLE:flink_role'
);

The Flink distribution must contain an Iceberg Flink runtime compatible with that exact Flink release, plus the file-I/O dependencies required by the storage backend. Catalog authentication and storage authentication are separate even when the REST response includes vended credentials.

For streaming writes, files become visible only when the Iceberg sink commits a successful checkpoint. A job can therefore have files in storage while its latest records remain absent from the current Iceberg snapshot. Monitor Flink checkpoints and Iceberg commits together rather than treating file creation as proof of table visibility.

Integrate Apache Spark™ #

Spark configures an Iceberg SparkCatalog whose backend is the REST catalog:

spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.polaris.type=rest
spark.sql.catalog.polaris.uri=https://polaris.example.com/api/catalog
spark.sql.catalog.polaris.warehouse=analytics_catalog
spark.sql.catalog.polaris.credential=<client-id>:<client-secret>
spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:spark_role

Use the catalog in multipart names such as polaris.sales.orders. The Spark driver loads and commits metadata through Polaris; executors read and write Iceberg files directly. The Spark distribution needs Iceberg runtime artifacts built for its Spark and Scala versions.

Spark’s Iceberg SQL extensions may be required for procedures and row-level commands beyond basic reads and writes. Support also depends on the table’s Iceberg format version. Polaris accepting a commit does not imply that every other engine can interpret features written by that Spark version.

Integrate Dremio® OSS #

Dremio OSS does not currently expose the Iceberg REST Catalog source needed for a supported direct connection to Polaris. Do not assume that the presence of Iceberg readers in Dremio OSS implies REST catalog support, and do not document an Enterprise-only source as an OSS integration.

Dremio Enterprise editions provide an Iceberg REST Catalog Source that can use a Polaris endpoint and vended credentials. That configuration is not portable to the OSS edition. For an all-open-source deployment, use a Polaris-compatible engine such as Spark, Flink, Trino, Doris, or a supported StarRocks release, or verify that a future Dremio OSS release has explicitly added the source before designing around it.

Integrate StarRocks #

StarRocks configures Polaris as the REST backend of an external Iceberg catalog:

CREATE EXTERNAL CATALOG polaris
PROPERTIES (
  "type" = "iceberg",
  "iceberg.catalog.type" = "rest",
  "iceberg.catalog.uri" =
    "https://polaris.example.com/api/catalog",
  "iceberg.catalog.warehouse" = "analytics_catalog",
  "iceberg.catalog.security" = "oauth2",
  "iceberg.catalog.oauth2.credential" =
    "<client-id>:<client-secret>",
  "iceberg.catalog.oauth2.scope" =
    "PRINCIPAL_ROLE:starrocks_role"
);

StarRocks frontend and backend or compute nodes must be able to reach the catalog and storage services required by their roles. Storage credential properties depend on S3, an S3-compatible service, Azure, GCS, or HDFS. Credential vending for REST catalogs is release-dependent; current StarRocks documentation identifies it as available from version 4.0.

Check the StarRocks support matrix before enabling Iceberg v2 delete files, newer format versions, writes, or maintenance operations. Being able to query a basic Parquet-backed table does not prove support for every snapshot feature another engine may commit.

Integrate Trino #

Trino uses its Iceberg connector with rest as the catalog type. A catalog properties file contains settings similar to:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://polaris.example.com/api/catalog
iceberg.rest-catalog.warehouse=analytics_catalog
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.vended-credentials-enabled=true
fs.s3.enabled=true

Add the supported OAuth credentials, Polaris realm header when the deployment requires one, and storage-specific properties through a protected secret and configuration mechanism. Do not place real client secrets directly in a world-readable catalog properties file.

Trino’s coordinator communicates with Polaris during planning and commits, while workers read and write Iceberg files. Trino also exposes Iceberg maintenance procedures, but their availability and semantics depend on the connector release. Coordinate snapshot expiration, orphan-file removal, and data-file optimization across engines so that independent maintenance jobs do not use conflicting retention policies.

Follow an Iceberg Read #

Consider a query issued through Spark SQL:

SELECT customer_id, SUM(amount)
FROM lake.sales.orders
WHERE order_date >= DATE '2026-08-01'
GROUP BY customer_id;

A simplified read proceeds as follows:

1. Spark resolves lake.sales.orders through the configured Polaris catalog.
2. Spark loads the table’s current Iceberg metadata and selects a snapshot.
3. Iceberg evaluates the query predicate against partition information and manifest statistics.
4. Spark plans tasks only for data and delete files that may match.
5. Executors read those files directly from storage.
6. Spark performs the aggregation and returns the result.

Polaris participates in name resolution and authorization. Iceberg supplies the snapshot and file-planning model. Spark supplies the SQL semantics, optimizer, scheduler, and runtime.

Follow an Iceberg Write #

An append illustrates the commit boundary:

1. The engine loads the current Iceberg table metadata from Polaris.
2. Distributed tasks write new data files to storage.
3. The engine writes new manifests, a manifest list, and table metadata.
4. The engine asks Polaris to replace the old metadata pointer with the new one, conditional on the base version still being current.
5. Polaris commits the pointer atomically or rejects the stale update.
6. On a conflict, the Iceberg client decides whether the operation can be safely retried against the new current state.

Files written before a failed commit are not part of the table merely because they exist in the table directory. They are orphan files until cleanup removes them or a later valid operation references them.

This is why applications must not write directly into an Iceberg table location and then expect the files to appear. Visibility is controlled by committed Iceberg metadata, not directory listing.

Explain the Lakehouse Label #

The term lakehouse combines two properties:

• Lake: table data remains in HDFS or object storage, commonly in open file formats and accessible by multiple engines.
• Warehouse: the table layer supplies schemas, transactions, snapshots, governance, and predictable SQL behavior.

In this stack, the label applies to the complete architecture rather than to Polaris alone:

Lake storage
    + Iceberg table semantics
    + Polaris catalog and governance
    + SQL compute engines
    = an open lakehouse architecture

Polaris contributes catalog interoperability and governance. Iceberg contributes warehouse-like table semantics over lake storage. Compute engines provide the actual analytical processing.

Separate Namespaces from Storage Paths #

A logical identifier such as sales.orders is not a filesystem path. Polaris resolves the identifier to Iceberg metadata, while that metadata ultimately references files in storage.

This separation allows administrators to:

• change logical organization without exposing storage layout to users;
• enforce access at catalog and namespace boundaries;
• use different storage locations for different catalogs or namespaces; and
• give several engines a consistent view of the same tables.

Storage permissions still matter. Catalog authorization can permit a table operation while the engine lacks permission to read the referenced objects. Conversely, direct storage access can bypass catalog policy if users receive unrestricted bucket credentials.

Design Authentication and Storage Access Together #

A secure request crosses at least two authorization boundaries:

User or workload
    |
    | authenticate and authorize catalog operation
    v
Polaris

Compute engine
    |
    | authorize object reads and writes
    v
Object storage or HDFS

Deployments can use fixed engine credentials or a supported credential-vending design. Credential vending gives an authenticated client scoped, temporary storage access instead of distributing broad, long-lived storage credentials. Its exact behavior depends on the Polaris release, storage backend, and engine integration.

Catalog policy and storage policy must agree. Test both control-plane denial and data-plane denial:

• Can the principal discover the catalog and namespace?
• Can it load or modify the table?
• Can its engine read existing metadata and data files?
• Can a writer create files and complete a catalog commit?
• Can credentials access unrelated prefixes after the operation?

TLS, secret storage, credential rotation, audit records, and clock synchronization remain deployment responsibilities.

Operate the Catalog as a Control-Plane Service #

Polaris is outside the query data path, but its availability affects table discovery and commits. Operate it as durable control-plane infrastructure:

• run the supported high-availability topology;
• protect and back up its persistent state;
• monitor request latency, error rates, authentication failures, and commit conflicts;
• retain audit records for catalog and privilege changes;
• coordinate server and client upgrades against supported Iceberg APIs; and
• test restoration without assuming that storage files reconstruct every catalog object or policy.

Monitor storage separately for failed-write debris, expired snapshots, small files, oversized manifests, and delete-file accumulation. Those are Iceberg table-maintenance concerns rather than evidence that Polaris should scan or rewrite table data.

Avoid Common Misconceptions #

Polaris Is Not a Compute Engine #

Polaris does not compete with Spark, Flink, or Trino. Those engines use Polaris.

Iceberg Is Not a File Format #

Iceberg is a table format. Its data files commonly use Parquet, ORC, or Avro.

A Catalog Is Not the Table’s Entire Metadata Tree #

Polaris maintains catalog state and the current table metadata reference. Iceberg metadata files and manifests are normally stored alongside table data in HDFS or object storage.

Object-Store Files Are Not Automatically Table Rows #

Only files referenced by a committed Iceberg snapshot belong to the table. Directory presence alone is insufficient.

REST Compatibility Does Not Guarantee Identical SQL #

Two engines can share a Polaris catalog and the same Iceberg tables while differing in SQL syntax, type handling, supported Iceberg features, write semantics, and authorization integration. Verify the exact engine, connector, Iceberg format version, and Polaris release combination.

Choose the Components Deliberately #

Polaris is useful when several Iceberg-capable engines need a shared, vendor-neutral REST catalog and a central authorization boundary. It is less useful when an environment has no Iceberg tables or when an existing catalog already satisfies interoperability, governance, and operational requirements.

Evaluate the layers independently:

No single component makes the stack a lakehouse. The architecture works when the catalog, table format, engines, and storage agree on commits, security, and supported features.

Official References #

• Apache Polaris
• Apache Iceberg documentation
• Iceberg table specification
• Iceberg REST Catalog specification
• Apache Doris and Polaris integration
• Apache Polaris and Flink guide
• Apache Polaris and Spark guide
• Dremio Iceberg REST Catalog documentation
• StarRocks Iceberg catalog documentation
• Apache Polaris and Trino guide
• Trino Iceberg connector documentation