Canal介绍

      基于mysql数据库binlog的增量订阅&消费

QuickStart

几点说明：(mysql初始化)

a.  canal的原理是基于mysql binlog技术，所以这里一定需要开启mysql的binlog写入功能，并且配置binlog模式为row. 

1.[mysqld]
2.log-bin=mysql-bin #添加这一行就ok
3.binlog-format=ROW #选择row模式
4.server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复

b.  canal的原理是模拟自己为mysql slave，所以这里一定需要做为mysql slave的相关权限.

1.CREATE USER canal IDENTIFIED BY 'canal';
2.GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
3.-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
4.FLUSH PRIVILEGES;

针对已有的账户可通过grants查询权限：

1.show grants for 'canal'  

启动步骤：

1.  下载canal

下载部署包

1.wget http://canal4mysql.googlecode.com/files/canal.deployer-1.0.0.tar.gz  

or 

自己编译 

1.git clone git@github.com:otter-projects/canal.git
2.cd canal;
3.mvn clean install -Dmaven.test.skip -Denv=release

编译完成后，会在根目录下产生target/canal.deployer-$version.tar.gz 

2.  解压缩

1.mkdir /tmp/canal
2.tar zxvf canal.deployer-1.0.0.tar.gz  -C /tmp/canal

解压完成后，进入/tmp/canal目录，可以看到如下结构：

1.drwxr-xr-x 2 jianghang jianghang  136 2013-02-05 21:51 bin
2.drwxr-xr-x 4 jianghang jianghang  160 2013-02-05 21:51 conf
3.drwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib
4.drwxr-xr-x 2 jianghang jianghang   48 2013-02-05 21:29 logs  

3.  配置修改

公用参数：   

1.vi conf/canal.properties  

1.#################################################
2.#########               common argument         #############
3.#################################################
4.canal.id= 1
5.canal.address=
6.canal.port= 11111
7.canal.zkServers=
8.# flush data to zk
9.canal.zookeeper.flush.period = 1000
10.## memory store RingBuffer size, should be Math.pow(2,n)
11.canal.instance.memory.buffer.size = 32768
12.
13.## detecing config
14.canal.instance.detecting.enable = false
15.canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
16.canal.instance.detecting.interval.time = 3
17.canal.instance.detecting.retry.threshold = 3
18.canal.instance.detecting.heartbeatHaEnable = false
19.
20.# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
21.canal.instance.transactionn.size =  1024
22.
23.# network config
24.canal.instance.network.receiveBufferSize = 16384
25.canal.instance.network.sendBufferSize = 16384
26.canal.instance.network.soTimeout = 30
27.
28.#################################################
29.#########               destinations            #############
30.#################################################
31.canal.destinations= example
32.
33.canal.instance.global.mode = spring
34.canal.instance.global.lazy = true  ##修改为false，代表立马启动
35.#canal.instance.global.manager.address = 127.0.0.1:1099
36.canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
37.#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

应用参数：

1.vi conf/example/instance.properties  

1.#################################################
2.## mysql serverId
3.canal.instance.mysql.slaveId = 1234
4.
5.# position info
6.canal.instance.master.address = 127.0.0.1:3306 #改成自己的数据库地址
7.canal.instance.master.journal.name =
8.canal.instance.master.position =
9.canal.instance.master.timestamp =
10.
11.#canal.instance.standby.address =
12.#canal.instance.standby.journal.name =
13.#canal.instance.standby.position =
14.#canal.instance.standby.timestamp =
15.
16.# username/password
17.canal.instance.dbUsername = retl  #改成自己的数据库信息
18.canal.instance.dbPassword = retl  #改成自己的数据库信息
19.canal.instance.defaultDatabaseName =   #改成自己的数据库信息
20.canal.instance.connectionCharsetNumber = 33  #改成自己的数据库信息
21.canal.instance.connectionCharset = UTF-8  #改成自己的数据库信息
22.
23.# table regex
24.canal.instance.filter.regex = .*\\..*
25.
26.#################################################

说明：

• canal.instance.connectionCharset 代表数据库的编码方式对应到java中的编码类型，比如UTF-8，GBK , ISO-8859-1
• canal.instance.connectionCharsetNumber 代表数据库的编码方式对应mysql中的唯一id，详细的映射关系可查看：com.mysql.jdbc.CharsetMapping.INDEX_TO_CHARSET
  针对常见的编码：
  utf-8  <=>  33
  gb2312 <=> 24
  gbk <=> 28

4.   准备启动

1.sh bin/startup.sh  

5.  查看日志

1.vi logs/canal/canal.log  

1.2013-02-05 22:45:27.967 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2.2013-02-05 22:45:28.113 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
3.2013-02-05 22:45:28.210 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......

具体instance的日志：

1.vi logs/example/example.log  

1.2013-02-05 22:50:45.636 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2.2013-02-05 22:50:45.641 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
3.2013-02-05 22:50:45.803 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
4.2013-02-05 22:50:45.810 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....

6.关闭