Kerberos

Kerberos is a network authentication protocol for systems under coordinated administration. A user authenticates to a trusted Key Distribution Center (KDC), then obtains time-limited tickets for individual services. Services can verify the user’s identity without receiving the user’s password, and clients can request proof that a service possesses the expected secret.

Kerberos primarily uses symmetric cryptography. Symmetric cryptography makes the KDC a central trust anchor: compromise of a KDC or its database can compromise every principal whose long-term key the KDC controls. Kerberos therefore trades decentralized trust for centralized identity administration, efficient ticket issuance, and single-sign-on within a realm.

Trust Model and Core Objects #

A realm is an administrative authentication domain, conventionally written in uppercase, such as EXAMPLE.COM. A realm name often resembles a DNS domain, but the two namespaces are distinct. Configuration or DNS discovery maps hosts and domain names to Kerberos realms.

A principal names an identity:

alice@EXAMPLE.COM
HTTP/web.example.com@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM

Principal names contain one or more components followed by a realm. alice is a user principal. HTTP/web.example.com is a service principal whose first component denotes the service and whose second component denotes the host. Principal strings are case-sensitive even though operational conventions differ between realm, user, and host components.

Every principal has one or more long-term keys registered with the KDC. Different principal types obtain those keys differently:

• A human principal commonly derives keys from a password through an enctype-specific string-to-key function and salt.
• A service principal commonly receives random keys generated by an administrator.
• The realm’s krbtgt/REALM principal has keys used to encrypt ticket-granting tickets (TGTs).

A keytab stores principal names, key version numbers (KVNOs), encryption types, and long-term keys. Daemons use keytabs because unattended services cannot type passwords. A keytab is a credential, not a public certificate; anyone who can read a usable service key can impersonate that service and decrypt tickets issued for its principal.

The KDC provides two logical services:

1. The Authentication Service (AS) issues a TGT after initial authentication.
2. The Ticket-Granting Service (TGS) exchanges a valid TGT for a ticket to a specific service.

Both services normally run in the same KDC process and use the same realm database. Production realms deploy redundant KDCs because ticket acquisition otherwise depends on one host, while database administration and replication require additional controls.

Protocol Flow #

Suppose alice@EXAMPLE.COM wants to access HTTP/web.example.com@EXAMPLE.COM. Three exchanges establish the required credentials. Names below describe Kerberos protocol messages, not application-specific packets.

Initial Authentication: AS-REQ and AS-REP #

The client sends an AS-REQ requesting a TGT for krbtgt/EXAMPLE.COM@EXAMPLE.COM. The request includes the client principal, requested validity and flags, a nonce, supported encryption types, and, when required, preauthentication data.

Password preauthentication commonly contains a current timestamp encrypted with a key derived locally from Alice’s password. The KDC looks up Alice’s key and verifies the timestamp. Some clients first send an AS-REQ without preauthentication, receive KDC_ERR_PREAUTH_REQUIRED plus method and salt information, and then retry. That first error is protocol negotiation rather than an authentication failure.

The KDC returns an AS-REP containing two related results:

• A TGT encrypted with the realm’s krbtgt key. The client stores the ticket but cannot decrypt its protected contents.
• Reply data encrypted with Alice’s long-term key. The reply gives the client the TGT session key, ticket times and flags, the nonce response, and related metadata.

Successful decryption of the reply proves that the client knows Alice’s password-derived key. The client can discard the password after placing the TGT and its session key in a credential cache.

Ticket Acquisition: TGS-REQ and TGS-REP #

The client sends a TGS-REQ for HTTP/web.example.com@EXAMPLE.COM. The request includes:

• the opaque TGT;
• a fresh authenticator encrypted with the TGT session key;
• the requested service principal, lifetime, options, nonce, and supported encryption types.

The KDC decrypts the TGT with its krbtgt key, obtains the TGT session key, and uses that session key to verify the authenticator. Valid proof ties the request to the client named inside the TGT.

The TGS-REP again has two useful parts:

• A service ticket encrypted with the long-term key of HTTP/web.example.com. Alice cannot decrypt the protected ticket.
• Reply data encrypted with the TGT session key. Alice can decrypt that data to obtain the new client-service session key and ticket metadata.

TGT presentation does not send Alice’s password-derived key to the TGS. TGT theft can still enable impersonation until expiration or revocation controls take effect, because the cache also contains the associated session key.

Service Authentication: AP-REQ and AP-REP #

The client sends an AP-REQ to the web service. It contains:

• the service ticket encrypted for HTTP/web.example.com; and
• a new authenticator encrypted with the client-service session key.

The service finds the matching principal, KVNO, and encryption type in its keytab, decrypts the ticket, extracts the session key, and verifies the authenticator. Applications or GSS-API libraries also verify ticket times, names, checksums, and other policy-relevant fields.

An authenticator contains a client identity and a current timestamp with subsecond data. It can also carry a checksum over application binding data, a sequence number, and a client-generated subkey. Services maintain a replay cache so the same ticket-authenticator pair cannot be accepted repeatedly within the clock-skew window. A service ticket can be reused during its lifetime, but each authentication exchange needs a fresh authenticator.

When mutual authentication is requested, the service returns an AP-REP encrypted with the client-service session key. Its protected timestamp response, and potentially a subkey and sequence number, proves that the peer decrypted the service ticket. Kerberos supports mutual authentication, but an application must request and validate it rather than assume every integration does so.

Client                         KDC                         Service
  |--- AS-REQ ----------------->|                             |
  |<-- AS-REP: TGT + Kc,tgs ----|                             |
  |--- TGS-REQ: TGT + auth ---->|                             |
  |<-- TGS-REP: ticket + Kc,s ---|                             |
  |--- AP-REQ: ticket + auth ------------------------------->|
  |<-- AP-REP, when mutual authentication is requested ------|

Kc,tgs denotes the TGT session key, and Kc,s denotes the client-service session key. Both are short-lived keys generated for protocol use rather than long-term principal keys.

Why Passwords Do Not Reach Services #

Alice’s password is an input to a local string-to-key function. Salt and encryption type affect the resulting key, so the same password need not produce the same key in different realms or under different enctype settings. The KDC stores password-equivalent key material and verifies proof made with it; the web service stores only its own random service keys.

Consequently, the web service receives neither Alice’s password nor Alice’s long-term key. It receives a ticket encrypted with the service’s key and verifies an authenticator made with the ticket’s session key. A compromised service therefore does not directly expose every user’s password.

Password-based Kerberos still permits offline guessing in some circumstances. An attacker who obtains suitable encrypted protocol material can test password candidates without repeatedly contacting the KDC. Preauthentication prevents issuance of an AS-REP to an unauthenticated request, but encrypted timestamp preauthentication can itself provide material for password guessing. Strong passwords, protected KDC databases, modern encryption types, rate controls, and stronger initial authentication mechanisms such as PKINIT or FAST where supported reduce different parts of that risk.

Ticket Contents and Encryption Boundaries #

Kerberos packets expose enough outer structure for routing and algorithm selection while encrypting identity assertions and session keys for the intended reader. Conceptually:

• A ticket identifies its destination realm and service in outer fields. Its encrypted part contains the client principal, a session key, flags, authentication and expiry times, optional renewable time, transited-realm information, optional client addresses, and authorization data.
• A TGT’s encrypted part is protected by a krbtgt key, so the KDC can read it.
• A service ticket’s encrypted part is protected by the service principal’s long-term key, so the target service can read it.
• AS reply data for a password-authenticated user is protected by the user’s long-term key.
• TGS reply data is protected with a key already available through the TGT exchange, commonly the TGT session key.
• Authenticators are protected by the session key associated with the presented ticket.

Clients receive each ticket and its corresponding session key through separate encryption boundaries. A client needs the session key to construct authenticators but does not need to decrypt the ticket. Likewise, a service does not need Alice’s long-term key because its own key decrypts the service ticket.

Ticket flags alter how credentials may be used. A forwardable TGT can produce a forwarded credential for another host, while a renewable TGT can receive a later expiration time until its renewable limit. Neither flag makes a ticket immortal; realm policy and the ticket’s encoded times remain authoritative.

Operational Dependencies #

Protocol correctness depends on names, time, transport, and shared key state. Many Kerberos failures therefore occur before an application evaluates user permissions.

DNS and Name Canonicalization #

Service principals encode hostnames, so DNS aliases and canonicalization influence principal selection. A client connecting to https://reports.example.com might request HTTP/reports.example.com@EXAMPLE.COM, while a library that canonicalizes the name might request HTTP/web17.example.com@EXAMPLE.COM. Only a principal provisioned in the KDC and keytab will work.

MIT Kerberos settings such as rdns and, in versions that support it, dns_canonicalize_hostname control parts of hostname canonicalization. Setting rdns = false avoids deriving service names from reverse DNS. Disabling forward canonicalization can also make service names predictable, but every client library and application must be tested because some perform their own resolution.

Kerberos can discover KDCs through DNS SRV records such as _kerberos._udp.example.com and _kerberos._tcp.example.com. Explicit [realms] entries avoid discovery ambiguity. Forward and reverse records still need consistent hostnames for applications, TLS certificates, Hadoop _HOST expansion, and operations.

Time Synchronization and Replay Protection #

Authenticators contain current time, and KDCs and services enforce a configured clock skew. NTP or another authenticated, monitored time service is therefore an operational dependency. A clock jump can invalidate preauthentication, cause KRB_AP_ERR_SKEW, or make valid tickets appear not yet valid or expired.

Replay protection combines short authenticator acceptance windows with service-side replay caches. Disabling a replay cache to hide filesystem or permission problems removes an important protocol defense. Services instead need a writable, private, and stable replay-cache location appropriate to their process model.

UDP, TCP, and Firewalls #

Kerberos uses port 88, and current implementations support TCP while many also support UDP. Large replies can exceed practical UDP limits because group and authorization data increase ticket size. Clients can retry over TCP, but firewalls must permit the transport chosen by the implementation. MIT Kerberos administrators sometimes set udp_preference_limit = 1 to prefer TCP; compatibility and network policy should drive that decision. Administrative and password-change services also use separate ports, commonly TCP 749 for kadmind and TCP/UDP 464 for password changes.

Cross-Realm Trust #

Cross-realm trust lets a principal from realm A.EXAMPLE request a service ticket in realm B.EXAMPLE. KDCs establish the path through shared cross-realm krbtgt principals. For a direct path from A to B, both realms must possess matching key material for a principal such as:

krbtgt/B.EXAMPLE@A.EXAMPLE

The client first receives a cross-realm TGT from A, then presents it to B to obtain the service ticket. Hierarchical realm names can imply a path, while [capaths] configuration can define nonhierarchical paths. Ticket transited fields record realms in the path for policy evaluation.

Trust direction, key distribution, encryption-type compatibility, and principal mapping require explicit administration. Authentication across a trust path also does not grant application access; realm B and the target service still apply local authorization policy.

Building a Small MIT Kerberos Realm #

The following Debian or Ubuntu example creates EXAMPLE.COM with KDC kdc01.example.com. Package names, paths, service units, and defaults differ across distributions, so package documentation remains authoritative. Use dedicated hosts, backups, replication, monitoring, and restricted administration for a production KDC.

Install the Packages #

sudo apt-get update
sudo apt-get install krb5-kdc krb5-admin-server krb5-user

Package configuration may ask for a default realm and KDC. Files below make the example explicit.

Configure Clients and Realm Mapping #

Create /etc/krb5.conf on KDCs, clients, and service hosts:

[libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = false
    dns_lookup_kdc = false
    rdns = false
    dns_canonicalize_hostname = false
    ticket_lifetime = 10h
    renew_lifetime = 7d
    forwardable = false

[realms]
    EXAMPLE.COM = {
        kdc = kdc01.example.com
        admin_server = kdc01.example.com
        kpasswd_server = kdc01.example.com
    }

[domain_realm]
    .example.com = EXAMPLE.COM
    example.com = EXAMPLE.COM

forwardable = false makes least privilege the client default. Workflows that require delegation can request it explicitly. Sites using DNS KDC discovery can omit explicit KDC hosts and set discovery options after publishing and validating SRV records.

Configure the KDC #

On common Debian-derived packages, /etc/krb5kdc/kdc.conf can contain:

[kdcdefaults]
    kdc_ports = 88
    kdc_tcp_ports = 88

[realms]
    EXAMPLE.COM = {
        database_name = /var/lib/krb5kdc/principal
        admin_keytab = FILE:/etc/krb5kdc/kadm5.keytab
        acl_file = /etc/krb5kdc/kadm5.acl
        key_stash_file = /etc/krb5kdc/stash
        kadmin_port = 749
        max_life = 10h
        max_renewable_life = 7d
        default_principal_flags = +preauth
        supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal
    }

Encryption types must match the oldest required clients and services. AES-SHA2 enctypes are available in newer Kerberos stacks, while older deployed software may only interoperate with AES-SHA1. Administrators should inventory compatibility and migrate deliberately rather than enable obsolete algorithms without a documented requirement.

Grant a dedicated administration class broad rights in /etc/krb5kdc/kadm5.acl:

*/admin@EXAMPLE.COM *

Tighter ACLs should delegate only required operations to automation and operators. Protect KDC configuration, database, stash, and ACL files with root ownership and restrictive permissions.

Initialize the Database and Principals #

Create the realm database and stash the database master key:

sudo kdb5_util create -s -r EXAMPLE.COM
sudo kadmin.local -q "addprinc admin/admin@EXAMPLE.COM"
sudo kadmin.local -q "addprinc alice@EXAMPLE.COM"

kdb5_util create prompts for a database master password. addprinc prompts for each human principal’s password. kadmin.local accesses the local database directly and is appropriate for initial bootstrap; remote kadmin authenticates as an administrative principal and is governed by kadm5.acl.

Start the services after the database exists:

sudo systemctl enable --now krb5-kdc
sudo systemctl enable --now krb5-admin-server
sudo systemctl status krb5-kdc krb5-admin-server

Unit names vary by package. Verify that port 88 is reachable from clients and expose administrative ports only to management networks that need them.

Create a Service Principal and Keytab #

Assume a local service account named websvc runs the HTTP application:

sudo install -d -m 0750 -o root -g websvc /etc/security/keytabs
sudo kadmin.local \
  -q "addprinc -randkey HTTP/web.example.com@EXAMPLE.COM"
sudo kadmin.local \
  -q "ktadd -k /etc/security/keytabs/web.keytab HTTP/web.example.com@EXAMPLE.COM"
sudo chown websvc:websvc /etc/security/keytabs/web.keytab
sudo chmod 0600 /etc/security/keytabs/web.keytab
sudo klist -k -e -t /etc/security/keytabs/web.keytab

ktadd commonly creates a new key version while writing the keytab. Running it again without understanding its key-changing behavior can invalidate copies already deployed. For a replicated service, generate or extract one controlled key version according to the site’s tooling, distribute it over a protected channel, verify the KVNO everywhere, and remove transfer artifacts. Never place a keytab in source control, a container image layer, or an unrestricted configuration store.

Service account permissions should permit reading only the keytab entries needed by that process. Separate service principals and keytabs reduce the impact of one daemon or host compromise.

Using Kerberos Clients #

Obtain and inspect a TGT:

kinit alice@EXAMPLE.COM
klist -ef

klist -e displays encryption types, while -f displays ticket flags. Request a service ticket and confirm that principal selection works before testing an application:

kvno HTTP/web.example.com@EXAMPLE.COM
klist

kvno asks the KDC for the named service ticket and reports its KVNO. Success proves that the client can find the KDC, use its TGT, and locate the service principal. It does not prove that the server keytab contains the same key or that the application accepts the ticket.

Other core operations include:

# Change the principal's password through the Kerberos password service.
kpasswd alice@EXAMPLE.COM

# Obtain a 10-hour forwardable TGT renewable for up to seven days.
kinit -f -l 10h -r 7d alice@EXAMPLE.COM

# Renew a still-valid renewable TGT.
kinit -R

# Authenticate a daemon or diagnostic process from a keytab.
kinit -k -t /etc/security/keytabs/web.keytab \
  HTTP/web.example.com@EXAMPLE.COM

# Remove the active credential cache.
kdestroy

Realm policy can shorten requested lifetimes or reject requested flags. Forwardable credentials should be enabled only when a workflow needs delegation, because a compromised intermediate host can use forwarded credentials as the user until those credentials expire.

Credential Caches and Collections #

Kerberos clients store tickets and session keys in a credential cache. KRB5CCNAME can select its type and location:

export KRB5CCNAME=FILE:/run/user/$(id -u)/krb5cc_example
kinit alice@EXAMPLE.COM
klist
kdestroy

Common cache types include FILE, DIR, kernel keyring, and KCM, depending on the operating system and Kerberos build. A FILE cache holds one primary credential set. Collection-capable types can hold credentials for several principals and select a primary cache. In MIT Kerberos versions that support the corresponding operations, klist -l lists collection members, klist -A displays all caches, and kdestroy -A destroys the collection.

Scripts should not assume that a cache path is a regular file. Containers, sudo, systemd services, user namespaces, and SSH sessions can change cache visibility and ownership. Set an explicit private cache for automation, clean it up on exit, and avoid sharing one mutable cache among unrelated processes.

Integrating Applications #

Applications rarely construct AS, TGS, and AP messages directly. Common integration layers include:

• GSS-API, which provides security contexts, mutual authentication, integrity, and optional confidentiality while hiding Kerberos message details;
• SPNEGO, which negotiates a GSS-API mechanism and underlies HTTP Negotiate;
• SASL/GSSAPI, which adds Kerberos authentication to protocols such as LDAP, Kafka, and database connections.

For HTTP Negotiate, a browser or command-line client first receives a challenge:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Negotiate

A GSS-API-capable curl can then use the current credential cache:

kinit alice@EXAMPLE.COM
curl --negotiate -u : https://web.example.com/protected

curl sends an Authorization: Negotiate ... token containing the GSS-API exchange. The server loads a keytab containing HTTP/web.example.com@EXAMPLE.COM, accepts the security context, and maps the authenticated principal to an application identity. A server process commonly receives its keytab location through an implementation-specific setting such as:

KRB5_KTNAME=FILE:/etc/security/keytabs/web.keytab ./web-server

Production service managers should set that environment variable and keytab permission without exposing either to unrelated processes.

GSS-API host-based names are often expressed as HTTP@web.example.com; the Kerberos mechanism maps that form to a service principal such as HTTP/web.example.com@EXAMPLE.COM. Clients must use the hostname covered by the provisioned principal. Load balancer names and DNS aliases therefore need corresponding service-principal planning rather than reliance on accidental canonicalization.

GSS-API can protect messages when an application requests integrity or confidentiality. HTTP Negotiate commonly authenticates the connection while HTTPS protects HTTP headers and bodies. TLS remains necessary for server certificate validation, traffic confidentiality outside GSS-wrapped messages, and protection of cookies, bearer tokens, and application data.

Trace a Kerberized Beeline-to-Kyuubi Connection #

Consider a Kyuubi endpoint whose organization-specific names have been anonymized as mycompany:

jdbc:hive2://kyuubi.hadoop.vip.mycompany.com:10010/default;ssl=true;principal=kyuubi/kyuubi.hadoop.vip.mycompany.com@PROD.MYCOMPANY.COM?kyuubi.session.cluster=analytics-prod#spark.yarn.queue=analytics-high-memory

Quote the complete URL when invoking Beeline because ; and # have meanings to a shell:

kinit alice@PROD.MYCOMPANY.COM
klist
kvno kyuubi/kyuubi.hadoop.vip.mycompany.com@PROD.MYCOMPANY.COM

beeline -u \
  'jdbc:hive2://kyuubi.hadoop.vip.mycompany.com:10010/default;ssl=true;principal=kyuubi/kyuubi.hadoop.vip.mycompany.com@PROD.MYCOMPANY.COM?kyuubi.session.cluster=analytics-prod#spark.yarn.queue=analytics-high-memory'

The URL has several independent responsibilities:

• kyuubi.hadoop.vip.mycompany.com:10010/default selects the Kyuubi listener and initial database.
• ssl=true enables TLS for transport encryption and server-certificate validation. TLS is separate from Kerberos authentication.
• principal=kyuubi/kyuubi.hadoop.vip.mycompany.com@PROD.MYCOMPANY.COM names the Kerberos identity that the client expects the Kyuubi server to prove. It is not Alice’s principal and is not a password.
• kyuubi.session.cluster=analytics-prod selects a deployment-specific Kyuubi session cluster.
• spark.yarn.queue=analytics-high-memory requests a YARN queue for the session’s Spark engine.

Kyuubi JDBC drivers accept session properties after a ? or #. Older Hive JDBC URL parsers distinguish configuration and variable suffixes more narrowly. A URL using both delimiters is therefore a deployment contract: the installed Beeline driver and Kyuubi version must agree on how those properties are forwarded. Neither property changes the Kerberos service principal.

Kerberos participates in the connection as follows:

1. kinit obtains Alice’s TGT and stores it with its session key in the credential cache. Beeline does not obtain Alice’s password from the URL.
2. The JDBC driver reads the cache and requests a service ticket for kyuubi/kyuubi.hadoop.vip.mycompany.com@PROD.MYCOMPANY.COM. If the ticket is not already cached, the Kerberos library sends a TGS-REQ using Alice’s TGT.
3. The realm’s TGS returns a service ticket encrypted for the Kyuubi principal and a client-service session key readable by Alice’s Kerberos library.
4. During the JDBC transport’s SASL/GSSAPI negotiation, Beeline sends an AP-REQ containing that service ticket and a fresh authenticator.
5. Kyuubi reads its protected keytab, selects the matching principal, KVNO, and encryption type, decrypts the ticket, and verifies the authenticator. Mutual authentication lets Beeline verify that the peer possesses the expected Kyuubi service key.
6. Kyuubi now has Alice’s authenticated Kerberos identity. Authorization rules decide whether Alice may open the session, select analytics-prod, use the requested YARN queue, and access data.
7. When Kyuubi starts or reuses a Spark engine, Hadoop delegation tokens can carry Alice’s authorized HDFS and YARN access to the distributed application without copying Alice’s TGT to every executor.

The hostname appears in both the network address and service principal, but the two checks are distinct. TLS validates the certificate for kyuubi.hadoop.vip.mycompany.com; Kerberos validates possession of the key for kyuubi/kyuubi.hadoop.vip.mycompany.com@PROD.MYCOMPANY.COM. DNS canonicalization, realm mapping, the KDC database, Kyuubi’s configured principal, and its keytab must agree on that identity.

Use the command sequence to isolate common failures:

• klist without a valid TGT explains a failure before any service ticket can be requested.
• kvno reporting “server not found in Kerberos database” indicates a principal, realm, DNS, or provisioning mismatch.
• kvno succeeding while Beeline reports a server-principal or integrity error points toward Kyuubi’s keytab, KVNO, encryption types, or configured server name.
• Kerberos succeeding while TLS fails points toward the certificate chain, truststore, or certificate hostname.
• Authentication succeeding while the session or query is rejected points toward Kyuubi policy, YARN queue authorization, SQL authorization, or storage permissions.

Kerberos in the Hadoop Ecosystem #

Secure Hadoop deployments use Kerberos to establish daemon and user identities across HDFS, YARN, Hive, and related services. Representative service principals include:

nn/nn01.example.com@EXAMPLE.COM
dn/dn01.example.com@EXAMPLE.COM
rm/rm01.example.com@EXAMPLE.COM
nm/nm01.example.com@EXAMPLE.COM
hive/hive01.example.com@EXAMPLE.COM
HTTP/nn01.example.com@EXAMPLE.COM

Hadoop configuration often uses _HOST:

<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>
<property>
  <name>dfs.namenode.kerberos.principal</name>
  <value>nn/_HOST@EXAMPLE.COM</value>
</property>
<property>
  <name>dfs.namenode.keytab.file</name>
  <value>/etc/security/keytabs/nn.service.keytab</value>
</property>

At runtime, Hadoop replaces _HOST with the host’s resolved canonical name. DNS, hostname configuration, and the actual KDC principal must therefore agree. An nn/nn01.example.com keytab cannot satisfy a request for nn/namenode-alias.example.com unless both principal names are provisioned as intended.

NameNode, DataNode, ResourceManager, NodeManager, HiveServer2, and HTTP endpoints need the service principals and keytab entries prescribed by the component’s security documentation. Sharing one keytab across every daemon broadens compromise impact and makes rotation harder.

Hadoop clients first authenticate with Kerberos, then can receive delegation tokens for supported services. A delegation token is a time-limited bearer-style credential issued and verified by a Hadoop service’s secret manager. Tokens let distributed jobs access HDFS or other services without contacting the KDC for every RPC and without copying a user’s TGT to every task. Token scope, renewal, cancellation, and maximum lifetime remain service-specific, and theft permits use within those limits.

Kerberos answers which principal initiated a connection. HDFS permissions and ACLs, YARN queue policy, Hive or SQL privileges, proxy-user rules, and external authorization systems decide what that principal may do. Principal-to-local-name rules such as Hadoop’s auth_to_local mapping also require review because an unexpected mapping can collapse distinct Kerberos identities into one local account.

Security Practices #

Kerberos deployments need controls around every long-term key and every reusable credential:

• Restrict each keytab to its service account, include only required principals, and avoid copying one keytab to unrelated hosts.
• Rotate service keys through a procedure that accounts for KVNO propagation, running processes, replicated hosts, and tickets issued under the prior key. Abrupt removal of old keys can break unexpired tickets.
• Keep client, KDC, and service encryption-type support compatible. Inspect both tickets and keytabs before disabling an enctype.
• Require preauthentication for password principals unless a reviewed mechanism needs an exception. Use strong password policy and stronger preauthentication mechanisms where threat models require them.
• Protect credential caches and forwarded credentials. SSH credential delegation and application impersonation features expand the hosts able to act as a user.
• Patch and isolate KDCs, restrict administrative access, back up the realm database and master-key material securely, and test restoration.
• Keep passwords, keytabs, KDC database dumps, stash files, and credential caches out of repositories, logs, support bundles, and image layers.
• Use TLS even when Kerberos authenticates peers. Kerberos does not automatically encrypt every application’s transport.

Key rotation needs overlap planning. A KDC can retain several service keys distinguished by KVNO and enctype, and a keytab can contain several entries. Exact rollover behavior depends on KDC tooling and application keytab reload behavior, so verify new tickets, old unexpired tickets, and every replica before retiring a previous key.

Troubleshooting by Causal Layer #

Start below the application layer. A browser error or generic GSS exception often wraps a realm, DNS, clock, cache, or keytab failure.

Trace Ticket Acquisition #

MIT Kerberos can trace library decisions:

env KRB5_TRACE=/dev/stderr kinit alice@EXAMPLE.COM
env KRB5_TRACE=/dev/stderr \
  kvno HTTP/web.example.com@EXAMPLE.COM

Trace output shows configuration files, KDC selection, preauthentication negotiation, cache operations, referrals, hostname decisions, and transport retries. Traces can contain principal names and operational metadata, so sanitize them before sharing.

Inspect credentials and server keys:

klist -ef
sudo klist -k -e -t /etc/security/keytabs/web.keytab
kvno HTTP/web.example.com@EXAMPLE.COM

Compare the service principal, KVNO, and enctype reported for the acquired ticket with entries available to the daemon. Also confirm that the running process reads the keytab inspected by the operator rather than another path or stale container secret.

Verify Names, Time, and Transport #

hostname -f
getent hosts web.example.com
dig +short -x 192.0.2.20
dig +short SRV _kerberos._tcp.example.com
timedatectl status

Forward lookup, reverse lookup, configured service hostname, TLS name, and KDC principal do not need to be identical in every design, but differences must be intentional and supported by client canonicalization rules. Check port 88 over the required transports and inspect firewalls when small requests work but large replies fail.

Map Common Errors to Causes #

Error text varies by implementation. Evidence from KRB5_TRACE, KDC logs, service logs, klist, kvno, DNS tools, and synchronized timestamps provides a more reliable diagnosis than changing several configuration options at once.

Boundaries and Limitations #

Kerberos provides authentication and shared session-key establishment within configured trust relationships. It does not by itself provide:

• Authorization: applications still enforce permissions, roles, ACLs, and policy.
• Universal transport encryption: applications must request GSS confidentiality or use a transport such as TLS.
• A directory service: LDAP, Active Directory, databases, or local mappings may store profiles, groups, and attributes.
• Internet-wide public-key identity: realms require prior KDC and trust configuration; Kerberos is not a replacement for the Web PKI.
• Compromise immunity: stolen keytabs, credential caches, KDC keys, or delegation tokens remain usable credentials within their scope and lifetime.

Centralized administration explains both Kerberos’ value and its operational burden. Services avoid collecting user passwords because the KDC vouches for principals through encrypted tickets, but reliable authentication then depends on KDC security, key distribution, principal naming, DNS behavior, time synchronization, and application-level validation.

Official References #

• MIT Kerberos Documentation
• MIT Kerberos: Installing KDCs
• MIT Kerberos: krb5.conf
• MIT Kerberos: kdc.conf
• RFC 4120: The Kerberos Network Authentication Service (V5)
• RFC 4121: Kerberos Version 5 GSS-API Mechanism
• RFC 4178: SPNEGO
• RFC 4559: SPNEGO-Based Kerberos and NTLM HTTP Authentication
• Apache Hadoop: Secure Mode