Kafka confluent安装部署

official Confluent Platform Quick Start ; Manual Install using ZIP and TAR Archives

confluent的安装部署相对比较简单,confluent为我们提供了Confluent Platform,我们即可以快速启动整个confluent平台,也可以单独启动想要的组件。接下来我们详细介绍如何操作。

confluent platform下载

首先看看如何快速启动confluent platform全家桶ZooKeeper,Kafka,Schema Registry,Control Center,Kafka Connect,Kafka REST Proxy,KSQL

快速启动platform

confluent platform分两个版本Confluent EnterpriseConfluent Open SourceConfluent Enterprise拥有更多的组件,这里测试选择Confluent Enterprise启动,因为它里面包含了Control Center方便我们测试,直观的从浏览器上看到数据信息。

  • 启动 (特别说明我们的命令执行目录都是在confluent目录下,如我的目录/Users/mo/runtime/confluent-5.0.0.2)
    1
      ./bin/confluent start
    

    看到如下信息,说明我们的confluent platform中的多个组件都启动成功。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
      Starting zookeeper
      zookeeper is [UP]
      Starting kafka
      kafka is [UP]
      Starting schema-registry
      schema-registry is [UP]
      Starting kafka-rest
      kafka-rest is [UP]
      Starting connect
      connect is [UP]
      Starting ksql-server
      ksql-server is [UP]
      Starting control-center
      control-center is [UP]
    
  • 访问测试 通过使用http://localhost:9021来访问Control Center,如图。

自定义启动

这里我们使用两台机器模拟集群192.168.226.184,192.168.226.27,’192.168.226.189’分别编排为host1,host2,host3。修改三台机器对应的hosts文件。分别添加如下配置

1
2
3
4
5
6
7
8
9
10
11
0.0.0.0 localhost  host1
192.168.226.184 localhost  host2
192.168.226.189  localhost host3

192.168.226.27  localhost host1
0.0.0.0   localhost  host2
192.168.226.189  localhost  host3

192.168.226.27  localhost  host1
192.168.226.189  localhost   host2
0.0.0.0  localhost  host3

分别为每台机器创建myid文件,没个myid保存要给唯一的数字即可,我这里三个host分别指定为1,2,3。

1
2
sudo mkdir /var/lib/zookeeper
sudo vi /var/lib/zookeeper/myid

每台机器分别指定如下配置

zookeeper配置和启动
  • vi etc/kafka/zookeeper.properties添加如下配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
      tickTime=2000
      dataDir=/var/lib/zookeeper/
      clientPort=2181
      initLimit=5
      syncLimit=2
      server.1=host1:2888:3888
      server.2=host2:2888:3888
      server.3=host3:2888:3888
      autopurge.snapRetainCount=3
      autopurge.purgeInterval=24
    
  • 启动

    1
     ./bin/zookeeper-server-start etc/kafka/zookeeper.properties
    
kafka配置和启动
  • 修改配置vi etc/kafka/server.properties
1
2
zookeeper.connect=host1:2181,host2:2181,host3:2181

  • 设置broker.id=0

    这里我们可以使用broker.id.generation.enable=true自动生成替代。

    1
    2
    3
    #broker.id=0
    broker.id.generation.enable=true
    advertised.listeners=PLAINTEXT://本机IP:9092
    
  • 启动

    1
    ./bin/kafka-server-start  etc/kafka/server.properties
    
Schema Registry配置和启动(可选)
  • 配置vi etc/schema-registry/schema-registry.properties

    kafkastore.connection.url=host1:2181,host2:2181,host3:2181

  • 启动

    1
    ./bin/schema-registry-start etc/schema-registry/schema-registry.properties
    
kafka connect配置和启动

这里我们不使用官方模式的avro序列化方式,所有不启动组件schema-registry

  • 配置 cp etc/schema-registry/connect-avro-distributed.properties etc/schema-registry/connect-distributed.properties

  • 修改vi etc/schema-registry/connect-distributed.properties

    bootstrap.servers=host1:9092,host2:9092,host3:9092

1
2
3
4
	key.converter=org.apache.kafka.connect.json.JsonConverter
	#key.converter.schema.registry.url=http://192.168.226.184:8081
	value.converter=org.apache.kafka.connect.json.JsonConverter
	#value.converter.schema.registry.url=http://192.168.226.184:8081
  • 启动
    1
    ./bin/connect-distributed etc/schema-registry/connect-distributed.properties
    
Control Center配置和启动
  • 配置vi etc/confluent-control-center/control-center-dev.properties
1
2
3
4
	bootstrap.servers=host1:9092,host2:9092,host3:9092
	zookeeper.connect=host1:2181,host2:2181:host3:2181
	#confluent.controlcenter.schema.registry.url=http://host1:8081,http://host2:8081,http://host3:8081
	confluent.controlcenter.connect.cluster=http://192.168.222.54:8083
  • 启动
    1
    ./bin/control-center-start etc/confluent-control-center/control-center-dev.properties 
    

到此为止kafka connect集群搭建成功。