mysql版本:Percona server 5.6.19-67.0 Percona Server (GPL), Release 67.0, Revision 618
Mysql安装参考:http://my.oschina.net/anthonyyau/blog/284092
Fabric State store安装参考:http://my.oschina.net/anthonyyau/blog/307165
半同步复制参考:http://my.oschina.net/anthonyyau/blog/269800
6、启动或禁用故障检测
如果primary故障,可能想自动提升一个次级到primary,重定向剩余的secondaries到新的primary:
# mysqlfabric group activate my_group Procedure : { uuid = 1d78324a-d39d-4098-8774-b0c2359a57ed, finished = True, success = True, return = True, activities = }
状态检测语句:
[DEBUG] 1409157170.648943 - FailureDetector(my_group) - Statement (SELECT server_uuid, server_address, mode, status, weight FROM servers WHERE group_id = %s, Params(('my_group',)).
[DEBUG] 1409157172.655084 - FailureDetector(my_group) - Statement (SELECT group_id, description, master_uuid, master_defined, status FROM groups WHERE group_id = %s, Params(('my_group',)).
测试关闭当前master,看是否选取新的master,fabric能自动进程故障切换:
查看当前组的状态:当前master为"172.17.0.51"
# mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'PRIMARY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'SECONDARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
关闭当前master实例:
# mysqladmin -h172.17.0.51 -uroot -padmin@123 shutdown 关闭172.17.0.51实例后,fabric已经标记它的状态为"FAULTY";
# mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'FAULTY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'PRIMARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
重新启动172.17.0.51:3306实例,但是状态不会恢复:
# mysql -h172.17.0.51 -uroot -padmin@123 -e "show master status" +------------------+----------+--------------+------------------+-------------------------------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+------------------+-------------------------------------------+ | mysql-bin.000003 | 191 | | | 19a37552-2d44-11e4-af5c-763d1493518d:1-10 | +------------------+----------+--------------+------------------+-------------------------------------------+ # mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'FAULTY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'PRIMARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
需要手动设置服务器的状态:
查看帮助命令:
# mysqlfabric help server set_status
server set_status server_id status [--update_only] [--synchronous]
Set a server's status.
支持的状态值为primary,secondary,spare,faulty。
能手动改变服务器的状态,改变到spare,避免写和读访问,同时保证当故障切换时不会被选取。
改变错误状态的实例(172.17.0.51)的状态为secondary,注意需要先转换成spare:
# mysqlfabric server set_status 3ecc746f-2e05-11e4-b448-560d7281695e spare Procedure : { uuid = e96802d7-5fc1-4a70-94d4-4d8d31d13960, finished = True, success = True, return = True, activities = } # mysqlfabric server set_status 3ecc746f-2e05-11e4-b448-560d7281695e secondary Procedure : { uuid = c74edd89-ac4a-45ce-8d18-22423ed73561, finished = True, success = True, return = True, activities = }
查看状态:
# mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'PRIMARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
禁用故障检测自动切换:
# mysqlfabric group deactivate my_group Procedure : { uuid = a88d2bb7-f0c7-46d8-925a-c982287d6447, finished = True, success = True, return = True, activities = }
7、执行更新和查询
应用程序通过fabric-aware连接器连接到fabric,获取服务器、组信息。
测试脚本:
# cat fabric_test1.py #!/usr/bin/env python #coding:utf-8 # Author: AnthonyYau --<q_yxian@163.com> # Purpose: fabric test scripts1. # Created: 2014/8/26 import sys import mysql.connector from mysql.connector import fabric def main(): try: conn = mysql.connector.connect( fabric = {"host":"localhost","port":32274,"username":"admin", "password":"admin"}, #指定使用xml-rpm连接fabric的用户名和密码 user = 'root',database = 'test',password = 'admin@123',autocommit = True #指定mysql实例的用户名和密码 ) except Exception,e: print e sys.exit(1) else: conn.set_property(mode=fabric.MODE_READWRITE,group="my_group") #指定访问模式,以及需要访问的HA组 cur = conn.cursor() cur.execute( "CREATE TABLE IF NOT EXISTS subscribers (" #创建表 " sub_no INT," " first_name CHAR(40)," " last_name CHAR(4)" ")" ) finally: conn.close() if __name__ == '__main__': main()
# cat fabric_test2.py #!/usr/bin/env python #coding:utf-8 # Author: AnthonyYau --<q_yxian@163.com> # Purpose: fabric test scripts2. # Created: 2014/8/26 import sys import mysql.connector from mysql.connector import fabric def add_subscriber(conn,sub_no,first_name,last_name): conn.set_property(group="my_group",mode=fabric.MODE_READWRITE) #指定访问模式,以及访问的组 cur = conn.cursor() cur.execute( "INSERT INTO subscribers VALUES (%s,%s,%s)", (sub_no,first_name,last_name) ) def main(): try: conn = mysql.connector.connect( fabric = {"host":"localhost","port":32274,"username":"admin", "password":"admin"}, #指定使用xml-rpc连接fabric的用户名和密码 user="root",database="test",password="admin@123",autocommit=True #指定访问mysql实例的用户名和密码 ) except Exception,e: print e sys.exit(1) else: #conn.set_property(group="my_group",mode=fabric.MODE_READWRITE) add_subscriber(conn, 72, "Billy", "Fish") #调用函数,插入数据 add_subscriber(conn, 500, "Billy", "Joel") add_subscriber(conn, 1500, "Arthur", "Askey") add_subscriber(conn, 5000, "Billy", "Fish") add_subscriber(conn, 15000, "Jimmy", "White") add_subscriber(conn, 17542, "Bobby", "Ball") finally: conn.close() if __name__ == '__main__':
执行测试程序:
python fabric_test1.py python fabric_test2.py
检查结果(查看两台从服务器的数据):当前slave为172.17.0.50和172.17.0.51
# mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'PRIMARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
使用脚本检查结果:
# python fabric_test3.py 72 Billy Fish 500 Billy Joel 1500 Arthur Aske 5000 Billy Fish 15000 Jimmy Whit 17542 Bobby Ball
查询脚本如下:
#!/usr/bin/env python #coding:utf-8 # Author: AnthonyYau --<q_yxian@163.com> # Purpose: fabric test scripts3. # Created: 2014/8/27 import sys import mysql.connector from mysql.connector import fabric def main(): try: conn = mysql.connector.connect( fabric = {"host":"localhost","port":32274,"username":"admin", "password":"admin"}, user="root",database="test",password="admin@123",autocommit=True ) except Exception,e: print e sys.exit(1) else: conn.set_property(group="my_group",mode=fabric.MODE_READONLY) #使用readonly访问模式(从slave获取数据),也可以使用readwrite cur = conn.cursor() cur.execute('select * from test.subscribers') results = cur.fetchall() if results: for id,f_name,l_name in results: print "%st%st%s" %(id,f_name,l_name) finally: conn.close() if __name__ == '__main__': main()
使用mysql客户端进行检查:
# mysql -uroot -padmin@123 -h172.17.0.51 -e "select * from test.subscribers" +--------+------------+-----------+ | sub_no | first_name | last_name | +--------+------------+-----------+ | 72 | Billy | Fish | | 500 | Billy | Joel | | 1500 | Arthur | Aske | | 5000 | Billy | Fish | | 15000 | Jimmy | Whit | | 17542 | Bobby | Ball | +--------+------------+-----------+ [chinastor.com-root@yau620 ~]# mysql -uroot -padmin@123 -h172.17.0.50 -e "select * from test.subscribers" +--------+------------+-----------+ | sub_no | first_name | last_name | +--------+------------+-----------+ | 72 | Billy | Fish | | 500 | Billy | Joel | | 1500 | Arthur | Aske | | 5000 | Billy | Fish | | 15000 | Jimmy | Whit | | 17542 | Bobby | Ball | +--------+------------+-----------+
三、组维护
1、找出哪些服务器在组中,查看组状态
# mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'PRIMARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '7bd52611-2d44-11e4-af5f-3ecad7c2f82a', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.47:3306'}, {'status': 'SECONDARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
2、从组中移除mysql实例 有时需要将secondaries离线进行维护,在停止服务之前,需要先从组中移除它,避免fabric错误检查触发任何动作: # mysqlfabric help group remove group remove group_id server_id [--synchronous] Remove a server from a group. 如从my_group组中移除server_uuid为7bd52611-2d44-11e4-af5f-3ecad7c2f82a的实例: # mysqlfabric group remove my_group 7bd52611-2d44-11e4-af5f-3ecad7c2f82a Procedure : { uuid = 07317ba2-497c-43f0-b8a9-e43ef9ded834, finished = True, success = True, return = True, activities = } # mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'PRIMARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
然后可以重新添加到组,将自动开启复制: # mysqlfabric group add my_group 172.17.0.47:3306 Procedure : { uuid = 39344790-3af8-4049-92a6-669bd2642a1c, finished = True, success = True, return = True, activities = }
查看复制状态: # mysql -ufabric -pfabric@456 -h172.17.0.47 -e 'show slave statusG' *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 172.17.0.50 Master_User: fabric Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000004 Read_Master_Log_Pos: 3451 Relay_Log_File: d4f404f647b0-relay-bin.000002 Relay_Log_Pos: 408 Relay_Master_Log_File: mysql-bin.000004 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 3451 Relay_Log_Space: 619 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 27 Master_UUID: 19a37552-2d44-11e4-af5c-763d1493518d Master_Info_File: /usr/local/Percona-Server-5.6.19-rel67.0-618.Linux.x86_64/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: 19a37552-2d44-11e4-af5c-763d1493518d:1-10 Auto_Position: 1
3、添加一个全新的Mysql实例(实例IP为172.17.0.51)到HA group组 创建数据库复制账号: # mysql -uroot -padmin@123 -h172.17.0.51 -e "grant replication slave on *.* to fabric@'172.17.0.%' identified by 'fabric@456'" # mysql -uroot -padmin@123 -h172.17.0.51 -e "show slave status"
添加到my_group组: # mysqlfabric group add my_group 172.17.0.51:3306 Procedure : { uuid = 45c32c66-6e60-44d8-8406-308de01b1ef9, finished = True, success = True, return = True, activities = }
查看复制状态,fabric已经开启复制: # mysql -uroot -padmin@123 -h172.17.0.51 -e "show slave statusG" *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 172.17.0.50 Master_User: fabric Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000004 Read_Master_Log_Pos: 3451 Relay_Log_File: 9355b5f279ad-relay-bin.000003 Relay_Log_Pos: 448 Relay_Master_Log_File: mysql-bin.000004 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 3451 Relay_Log_Space: 4169 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 27 Master_UUID: 19a37552-2d44-11e4-af5c-763d1493518d Master_Info_File: /usr/local/Percona-Server-5.6.19-rel67.0-618.Linux.x86_64/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 19a37552-2d44-11e4-af5c-763d1493518d:1-10 Executed_Gtid_Set: 19a37552-2d44-11e4-af5c-763d1493518d:1-10 Auto_Position: 1
4、关闭primary primary不能从组中移除,使用group demote关闭在组中的任何primary,不会重新选取一个新master,同时也不会关闭故障检测。 # mysqlfabric group demote my_group Procedure : { uuid = 448df7ff-f0b5-4868-be15-8b505ec38c19, finished = True, success = True, return = True, activities = }
查看组状态:所有mysql实例的状态都已经变成SECONDARY # mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'SECONDARY', 'server_uuid': '7bd52611-2d44-11e4-af5f-3ecad7c2f82a', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.47:3306'}, {'status': 'SECONDARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
5、手动提升一个服务组,选取一个master 当前master是172.17.0.48: # mysqlfabric group promote my_group Procedure : { uuid = d3072441-a023-42bd-8fe8-6746a620189d, finished = True, success = True, return = True, activities = } # mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'SECONDARY', 'server_uuid': '7bd52611-2d44-11e4-af5f-3ecad7c2f82a', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.47:3306'}, {'status': 'PRIMARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
重新选取一个新master:fabric已经自动选取了172.17.0.47作为master # mysqlfabric group promote my_group Procedure : { uuid = c779bfa6-5a57-4d60-9899-d64b85e9bee3, finished = True, success = True, return = True, activities = } # mysqlfabric group lookup_servers my_group Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '3ecc746f-2e05-11e4-b448-560d7281695e', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.51:3306'}, {'status': 'PRIMARY', 'server_uuid': '7bd52611-2d44-11e4-af5f-3ecad7c2f82a', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.47:3306'}, {'status': 'SECONDARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.48:3306'}] activities = }
6、移除组(使用group destroy) 如果组没有服务器,是空的,能移除组 # mysqlfabric group create my_test Procedure : { uuid = d53ba76d-1ed0-4e9d-96b3-01102b46f0bf, finished = True, success = True, return = True, activities = } # mysqlfabric group destroy my_test Procedure : { uuid = 7046fea7-6e87-4dc0-afd4-e21ea399f9c1, finished = True, success = True, return = True, activities = }
也能强制的移除非空的组(使用--force): # mysqlfabric group create my_test Procedure : { uuid = be15e7bd-e597-4e67-aa4d-4ff34e0d3af8, finished = True, success = True, return = True, activities = } # mysqlfabric group add my_test 172.17.0.52:3306 Procedure : { uuid = da981fe4-eeaf-4a3f-a770-d82d60a2ad64, finished = True, success = True, return = True, activities = } # mysqlfabric group lookup_servers my_test Command : { success = True return = [{'status': 'SECONDARY', 'server_uuid': '7bd52611-2d44-11e4-af5f-3ecad7c2f82a', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.52:3306'}] activities = } # mysqlfabric group destroy my_test Procedure : { uuid = 63b0ec92-ffa2-4f54-9c21-530b1532d21c, finished = True, success = False, return = GroupError: Group (my_test) is not empty., activities = } # mysqlfabric group destroy my_test --force Procedure : { uuid = 37c5b49b-c451-41ad-9193-c9e1f06edaa9, finished = True, success = True, return = True, activities = }
四、总结 1、使用fabric实现mysql复制高可用,以组为单位对mysql实例进行管理,可以管理多个HA组; 2、当前fabric只能实现自动配置异步复制,不支持半同步复制; 3、mysql实例进行恢复后,fabric不会自动改变实例的状态,需要手工修改; 4、能在线添加和删除mysql实例; 5、fabric state store和fabric note当前是单点,虽然fabric-aware连接器可以缓存路由信息(可以配置ttl值); 6、使用fabric不需要对slave配置vip,不像MHA和MMM等需要对vip进行管理; 7、mysql实例需要指出gtid,也是就要升级到mysql 5.6.10以上版本;
mysql版本:Percona server 5.6.19-67.0 Percona Server (GPL), Release 67.0, Revision 618 Mysql安装参考:http://my.oschina.net/anthonyyau/blog/284092 Fabric State store安装参考:http://my.oschina.net/anthonyyau/blog/307165 半同步复制参考:http://my.oschina.net/anthonyyau/blog/269800 参考: 1、http://www.clusterdb.com/mysql-fabric/mysql-fabric-adding-high-availability-to-mysql 2、官档:http://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric-quick-start-replication.html 原文链接:http://my.oschina.net/anthonyyau/blog/307355