Kafka 配置手册

简介

Broker 配置项

Kafka-1.1.x

• broker.id

• log.dirs

• zookeeper.connect

• advertised.host.name

• advertised.listeners

• advertised.port

• auto.create.topics.enable

• auto.leader.rebalance.enable

• background.threads

• compression.type

• delete.topic.enable

• host.name

• leader.imbalance.check.interval.seconds

• leader.imbalance.per.broker.percentage

• listeners

• log.dir

• log.flush.interval.messages

• log.flush.interval.ms

• log.flush.offset.checkpoint.interval.ms

• log.flush.scheduler.interval.ms

• log.flush.start.offset.checkpoint.interval.ms

• log.retention.bytes

• log.retention.hours

• log.retention.minutes

• log.retention.ms

• log.roll.hours

• log.roll.jitter.hours

• log.roll.jitter.ms

• log.roll.ms

• log.segment.bytes

• log.segment.delete.delay.ms

• message.max.bytes

• min.insync.replicas

• num.io.threads

• num.network.threads

• num.recovery.threads.per.data.dir

• num.replica.alter.log.dirs.threads

• num.replica.fetchers

• offset.metadata.max.bytes

• offsets.commit.required.acks

• offsets.commit.timeout.ms

• offsets.load.buffer.size

• offsets.retention.check.interval.ms

• offsets.retention.minutes

• offsets.topic.compression.codec

• offsets.topic.num.partitions

• offsets.topic.replication.factor

• offsets.topic.segment.bytes

• port

• queued.max.requests

• quota.consumer.default

• quota.producer.default

• replica.fetch.min.bytes

• replica.fetch.wait.max.ms

• replica.high.watermark.checkpoint.interval.ms

• replica.lag.time.max.ms

• replica.socket.receive.buffer.bytes

• replica.socket.timeout.ms

• request.timeout.ms

• socket.receive.buffer.bytes

• socket.request.max.bytes

• socket.send.buffer.bytes

• transaction.max.timeout.ms

• transaction.state.log.load.buffer.size

• transaction.state.log.min.isr

• transaction.state.log.num.partitions

• transaction.state.log.replication.factor

• transaction.state.log.segment.bytes

• transactional.id.expiration.ms

• unclean.leader.election.enable

• zookeeper.connection.timeout.ms

• zookeeper.max.in.flight.requests

• zookeeper.session.timeout.ms

• zookeeper.set.acl

• broker.id.generation.enable

• broker.rack

• connections.max.idle.ms

• controlled.shutdown.enable

• controlled.shutdown.max.retries

• controlled.shutdown.retry.backoff.ms

• controller.socket.timeout.ms

• default.replication.factor

• delegation.token.expiry.time.ms

• delegation.token.master.key

• delegation.token.max.lifetime.ms

• delete.records.purgatory.purge.interval.requests

• fetch.purgatory.purge.interval.requests

• group.initial.rebalance.delay.ms

• group.max.session.timeout.ms

• group.min.session.timeout.ms

• inter.broker.listener.name

• inter.broker.protocol.version

• log.cleaner.backoff.ms

• log.cleaner.dedupe.buffer.size

• log.cleaner.delete.retention.ms

• log.cleaner.enable

• log.cleaner.io.buffer.load.factor

• log.cleaner.io.buffer.size

• log.cleaner.io.max.bytes.per.second

• log.cleaner.min.cleanable.ratio

• log.cleaner.min.compaction.lag.ms

• log.cleaner.threads

• log.cleanup.policy

• log.index.interval.bytes

• log.index.size.max.bytes

• log.message.format.version

• log.message.timestamp.difference.max.ms

• log.message.timestamp.type

• log.preallocate

• log.retention.check.interval.ms

• max.connections.per.ip

• max.connections.per.ip.overrides

• max.incremental.fetch.session.cache.slots

• num.partitions

• password.encoder.old.secret

• password.encoder.secret

• principal.builder.class

• producer.purgatory.purge.interval.requests

• queued.max.request.bytes

• replica.fetch.backoff.ms

• replica.fetch.max.bytes

• replica.fetch.response.max.bytes

• reserved.broker.max.id

• sasl.enabled.mechanisms

• sasl.jaas.config

• sasl.kerberos.kinit.cmd

• sasl.kerberos.min.time.before.relogin

• sasl.kerberos.principal.to.local.rules

• sasl.kerberos.service.name

• sasl.kerberos.ticket.renew.jitter

• sasl.kerberos.ticket.renew.window.factor

• sasl.mechanism.inter.broker.protocol

• security.inter.broker.protocol

• ssl.cipher.suites

• ssl.client.auth

• ssl.enabled.protocols

• ssl.key.password

• ssl.keymanager.algorithm

• ssl.keystore.location

• ssl.keystore.password

• ssl.keystore.type

• ssl.protocol

• ssl.provider

• ssl.trustmanager.algorithm

• ssl.truststore.location

• ssl.truststore.password

• ssl.truststore.type

• alter.config.policy.class.name

• alter.log.dirs.replication.quota.window.num

• alter.log.dirs.replication.quota.window.size.seconds

• authorizer.class.name

• create.topic.policy.class.name

• delegation.token.expiry.check.interval.ms

• listener.security.protocol.map

• metric.reporters

• metrics.num.samples

• metrics.recording.level

• metrics.sample.window.ms

• password.encoder.cipher.algorithm

• password.encoder.iterations

• password.encoder.key.length

• password.encoder.keyfactory.algorithm

• quota.window.num

• quota.window.size.seconds

• replication.quota.window.num

• replication.quota.window.size.seconds

• ssl.endpoint.identification.algorithm

• ssl.secure.random.implementation

• transaction.abort.timed.out.transaction.cleanup.interval.ms

• transaction.remove.expired.transaction.cleanup.interval.ms

• zookeeper.sync.time.ms

• 备注

 从1.1开始，kafka的broker部分配置支持动态跟新，不需要重启broker；
 read-only: 需要重启broker, 存储在server.properties
 per-broker：动态更新，每个broker都要更新,存储在zookeeper
 cluster-wide： May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing，存储在zookeeper

Topic 配置

Kafka-1.1.x

• cleanup.policy

• compression.type

• delete.retention.ms

• file.delete.delay.ms

• flush.messages

• flush.ms

• follower.replication.throttled.replicas

• index.interval.bytes

• leader.replication.throttled.replicas

• max.message.bytes

• message.format.version

• message.timestamp.difference.max.ms

• message.timestamp.type

• min.cleanable.dirty.ratio

• min.compaction.lag.ms

• min.insync.replicas

• preallocate

• retention.bytes

• retention.ms

• segment.bytes

• segment.index.bytes

• segment.jitter.ms

• segment.ms

• unclean.leader.election.enable

Productor 配置

Kafka-1.1.x

• key.serializer

• value.serializer

• acks

• bootstrap.servers

• buffer.memory

• compression.type

• retries

• ssl.key.password

• ssl.keystore.location

• ssl.keystore.password

• ssl.truststore.location

• ssl.truststore.password

• batch.size

• client.id

• connections.max.idle.ms

• linger.ms

• max.block.ms

• max.request.size

• partitioner.class

• receive.buffer.bytes

• request.timeout.ms

• sasl.jaas.config

• sasl.kerberos.service.name

• sasl.mechanism

• security.protocol

• send.buffer.bytes

• ssl.enabled.protocols

• ssl.keystore.type

• ssl.protocol

• ssl.provider

• ssl.truststore.type

• enable.idempotence

• interceptor.classes

• max.in.flight.requests.per.connection

• metadata.max.age.ms

• metric.reporters

• metrics.num.samples

• metrics.recording.level

• metrics.sample.window.ms

• reconnect.backoff.max.ms

• reconnect.backoff.ms

• retry.backoff.ms

• sasl.kerberos.kinit.cmd

• sasl.kerberos.min.time.before.relogin

• sasl.kerberos.ticket.renew.jitter

• sasl.kerberos.ticket.renew.window.factor

• ssl.cipher.suites

• ssl.endpoint.identification.algorithm

• ssl.keymanager.algorithm

• ssl.secure.random.implementation

• ssl.trustmanager.algorithm

• transaction.timeout.ms

• transactional.id

Newer Consumer 配置

Kafka-1.1.x

• key.deserializer

• value.deserializer

• bootstrap.servers

• fetch.min.bytes

• group.id

• heartbeat.interval.ms

• max.partition.fetch.bytes

• session.timeout.ms

• ssl.key.password

• ssl.keystore.location

• ssl.keystore.password

• ssl.truststore.location

• ssl.truststore.password

• auto.offset.reset

• connections.max.idle.ms

• enable.auto.commit

• exclude.internal.topics

• fetch.max.bytes

• isolation.level

• max.poll.interval.ms

• max.poll.records

• partition.assignment.strategy

• receive.buffer.bytes

• request.timeout.ms

• sasl.jaas.config

• sasl.kerberos.service.name

• sasl.mechanism

• security.protocol

• send.buffer.bytes

• ssl.enabled.protocols

• ssl.keystore.type

• ssl.protocol

• ssl.provider

• ssl.truststore.type

• auto.commit.interval.ms

• check.crcs

• client.id

• fetch.max.wait.ms

• interceptor.classes

• metadata.max.age.ms

• metric.reporters

• metrics.num.samples

• metrics.recording.level

• metrics.sample.window.ms

• reconnect.backoff.max.ms

• reconnect.backoff.ms

• retry.backoff.ms

• sasl.kerberos.kinit.cmd

• sasl.kerberos.min.time.before.relogin

• sasl.kerberos.ticket.renew.jitter

• sasl.kerberos.ticket.renew.window.factor

• ssl.cipher.suites

• ssl.endpoint.identification.algorithm

• ssl.keymanager.algorithm

• ssl.secure.random.implementation

• ssl.trustmanager.algorithm

样例配置

生产服务配置

# Replication configurations
num.replica.fetchers=4
replica.fetch.max.bytes=1048576
replica.fetch.wait.max.ms=500
replica.high.watermark.checkpoint.interval.ms=5000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.lag.time.max.ms=10000

controller.socket.timeout.ms=30000
controller.message.queue.size=10

# Log configuration
num.partitions=8
message.max.bytes=1000000
auto.create.topics.enable=true
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.hours=168
log.flush.interval.ms=10000
log.flush.interval.messages=20000
log.flush.scheduler.interval.ms=2000
log.roll.hours=168
log.retention.check.interval.ms=300000
log.segment.bytes=1073741824

# ZK configuration
zookeeper.connection.timeout.ms=6000
zookeeper.sync.time.ms=2000

# Socket server configuration
num.io.threads=8
num.network.threads=8
socket.request.max.bytes=104857600
socket.receive.buffer.bytes=1048576
socket.send.buffer.bytes=1048576
queued.max.requests=16
fetch.purgatory.purge.interval.requests=100
producer.purgatory.purge.interval.requests=100