Kafka confluent安装部署

Posted on

official Confluent Platform Quick Start ; Manual Install using ZIP and TAR Archives

confluent的安装部署相对比较简单，confluent为我们提供了Confluent Platform,我们即可以快速启动整个confluent平台，也可以单独启动想要的组件。接下来我们详细介绍如何操作。

confluent platform下载

  • 地址https://www.confluent.io/download/

  • 解压tar -zxvf confluent-5.0.0-2.11.tar.gz后可以看到文件列表。

    1
    2
    3
    4
    5
    6
    7
      -rw-r--r--@  1 mo  staff   871  7 29 08:19 README
  drwxr-xr-x@ 68 mo  staff  2176  7 29 07:52 bin 		//Driver scripts for starting and stopping services
  drwxr-xr-x@ 23 mo  staff   736  7 29 07:52 etc 		//Configuration files
  drwxr-xr-x@  3 mo  staff    96  7 29 07:38 lib 		//Systemd services
  drwxr-xr-x   2 mo  staff    64  8  4 21:36 logs 	//Log files
  drwxr-xr-x@  7 mo  staff   224  7 29 07:52 share 	//Jars and licenses
  drwxr-xr-x@  9 mo  staff   288  7 29 08:19 src 		//Source files that require a platform-dependent build

首先看看如何快速启动confluent platform全家桶ZooKeeper,Kafka,Schema Registry,Control Center,Kafka Connect,Kafka REST Proxy,KSQL。

快速启动platform

confluent platform分两个版本Confluent Enterprise和Confluent Open Source，Confluent Enterprise拥有更多的组件，这里测试选择Confluent Enterprise启动，因为它里面包含了Control Center方便我们测试，直观的从浏览器上看到数据信息。

  • 启动 (特别说明我们的命令执行目录都是在confluent目录下，如我的目录/Users/mo/runtime/confluent-5.0.0.2) 
    1
      ./bin/confluent start

    看到如下信息，说明我们的confluent platform中的多个组件都启动成功。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
      Starting zookeeper
  zookeeper is [UP]
  Starting kafka
  kafka is [UP]
  Starting schema-registry
  schema-registry is [UP]
  Starting kafka-rest
  kafka-rest is [UP]
  Starting connect
  connect is [UP]
  Starting ksql-server
  ksql-server is [UP]
  Starting control-center
  control-center is [UP]
  • 访问测试 通过使用http://localhost:9021来访问Control Center,如图。

自定义启动

这里我们使用两台机器模拟集群192.168.226.184,192.168.226.27,’192.168.226.189’分别编排为host1,host2,host3。修改三台机器对应的hosts文件。分别添加如下配置

1
2
3
4
5
6
7
8
9
10
11
0.0.0.0 localhost  host1
192.168.226.184 localhost  host2
192.168.226.189  localhost host3

192.168.226.27  localhost host1
0.0.0.0   localhost  host2
192.168.226.189  localhost  host3

192.168.226.27  localhost  host1
192.168.226.189  localhost   host2
0.0.0.0  localhost  host3

分别为每台机器创建myid文件,没个myid保存要给唯一的数字即可，我这里三个host分别指定为1，2，3。

1
2
sudo mkdir /var/lib/zookeeper
sudo vi /var/lib/zookeeper/myid

每台机器分别指定如下配置

zookeeper配置和启动

  • vi etc/kafka/zookeeper.properties添加如下配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
      tickTime=2000
  dataDir=/var/lib/zookeeper/
  clientPort=2181
  initLimit=5
  syncLimit=2
  server.1=host1:2888:3888
  server.2=host2:2888:3888
  server.3=host3:2888:3888
  autopurge.snapRetainCount=3
  autopurge.purgeInterval=24

  • 启动

    1
     ./bin/zookeeper-server-start etc/kafka/zookeeper.properties
kafka配置和启动
  • 修改配置vi etc/kafka/server.properties
1
2
zookeeper.connect=host1:2181,host2:2181,host3:2181

  • 设置broker.id=0

    这里我们可以使用broker.id.generation.enable=true自动生成替代。

    1
    2
    3
    #broker.id=0
broker.id.generation.enable=true
advertised.listeners=PLAINTEXT://本机IP:9092

  • 启动

    1
    ./bin/kafka-server-start  etc/kafka/server.properties
Schema Registry配置和启动(可选)

  • 配置vi etc/schema-registry/schema-registry.properties

    kafkastore.connection.url=host1:2181,host2:2181,host3:2181

  • 启动

    1
    ./bin/schema-registry-start etc/schema-registry/schema-registry.properties
kafka connect配置和启动

这里我们不使用官方模式的avro序列化方式，所有不启动组件schema-registry。

  • 配置 cp etc/schema-registry/connect-avro-distributed.properties etc/schema-registry/connect-distributed.properties

  • 修改vi etc/schema-registry/connect-distributed.properties

    bootstrap.servers=host1:9092,host2:9092,host3:9092

1
2
3
4
	key.converter=org.apache.kafka.connect.json.JsonConverter
	#key.converter.schema.registry.url=http://192.168.226.184:8081
	value.converter=org.apache.kafka.connect.json.JsonConverter
	#value.converter.schema.registry.url=http://192.168.226.184:8081
  • 启动 
    1
    ./bin/connect-distributed etc/schema-registry/connect-distributed.properties
Control Center配置和启动
  • 配置vi etc/confluent-control-center/control-center-dev.properties
1
2
3
4
	bootstrap.servers=host1:9092,host2:9092,host3:9092
	zookeeper.connect=host1:2181,host2:2181:host3:2181
	#confluent.controlcenter.schema.registry.url=http://host1:8081,http://host2:8081,http://host3:8081
	confluent.controlcenter.connect.cluster=http://192.168.222.54:8083
  • 启动 
    1
    ./bin/control-center-start etc/confluent-control-center/control-center-dev.properties

到此为止kafka connect集群搭建成功。