Pointing MySQL Read-Replica to new master after failover

Here I am describing a simple scenario where we have a read replica (other than slave) which is currently pointing to master mysql instance.
We have another slave which is also pointing to same master. What we are going to see is, how to point read replica to new master after failover.

Environment:

To make you familier with environment, we have 3 instances of mysql

mysql_a -> Current master
mysql_b -> Current slave
mysql_c -> read replica pointing to mysql_a

Technically read replica is same as slave, but in this context we are saying read replica to make you understand that this is additional slave which does not take part in failover scenario.
So excercise is to failover master from mysql_a to mysql_b and point read replica mysql_c to mysql_b

I am using GTID on all 3 instances and both slaves are in sync with master.

On Master (mysql_a):

root [mysql] >show master status \G
*************************** 1. row ***************************
             File: bin_log.000007
         Position: 612
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:1-91
1 row in set (0.00 sec)

On Slave (mysql_b):

mysql> show slave status \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
               ...
               ...
              Master_Log_File: bin_log.000007
          Read_Master_Log_Pos: 612
               Relay_Log_File: relay_log.000010
                Relay_Log_Pos: 818
        Relay_Master_Log_File: bin_log.000007
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
            ...
            ...
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:1-91
            Executed_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:1-91
                Auto_Position: 1

On read-replica (mysql_c):

root [mysql] >show slave status \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
               ...
               ...
              Master_Log_File: bin_log.000007
          Read_Master_Log_Pos: 612
               Relay_Log_File: relay_log.000005
                Relay_Log_Pos: 442
        Relay_Master_Log_File: bin_log.000007
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
            ...
            ...
           Retrieved_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:91
            Executed_Gtid_Set: 6e68fbf2-f9a0-11e4-a3f7-90e2ba6ebdb0:1-7,
f7718b80-c237-11e4-baa8-a0369f370a52:1-91
                Auto_Position: 1

Lagging read-replica:

Before I failover, I want to make read replica lag by couple of transaction so that when we recover and point replica to new master we can clearly see transactions are getting applied from old and new master.
I will stop IO slave on read replica (mysql_c) so that it will stop reading binlog from master (mysql_a)

root [mysql] >stop slave IO_thread;
Query OK, 0 rows affected (0.03 sec)

root [mysql] >show slave status \G
*************************** 1. row ***************************
               Slave_IO_State:
               ...
               ...
             Slave_IO_Running: No
            Slave_SQL_Running: Yes

So any change to master will not be replicated to read replica (mysql_c)

Now, lets change few records in master and move forward its GTID

On Master (mysql_a):

root [deo] >show master status \G
*************************** 1. row ***************************
             File: bin_log.000007
         Position: 1220
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:1-93
1 row in set (0.00 sec)

I did couple of transactions on master (mysql_a) and increased GTID from 1-91 to 1-93.

I see that those trasactions are already applied to slave (mysql_b)

On Slave (mysql_b):

           Retrieved_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:1-93
            Executed_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:1-93
                Auto_Position: 1
1 row in set (0.00 sec)

But we didnt get those transaction on read replica (mysql_c) as IO thread is down

On read replica (mysql_c):

           Retrieved_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:91
            Executed_Gtid_Set: 6e68fbf2-f9a0-11e4-a3f7-90e2ba6ebdb0:1-7,
f7718b80-c237-11e4-baa8-a0369f370a52:1-91
                Auto_Position: 1

Failover:

Now, we will failover master to slave (mysql_b)

Failover is simple process. We just have to stop current master and make sure slave is not read only. Plus any other custom configuration that you have done on master, you can need to do the same on slave.

So after failover to mysql_b, my new master looks like below

mysql> show master status \G
*************************** 1. row ***************************
             File: bin_log.000003
         Position: 2495
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: f7718b80-c237-11e4-baa8-a0369f370a52:1-93
1 row in set (0.00 sec)

we can also do reset master, that will flush binlog file and reset GTID to 1. But lets not do that as that is not the objective of our excercise.

Pointing read replica to new master:

Now, we need to point our read replica (which is still pointing to old master (mysql_a)) to new master (mysql_b)
To do that we need to recover the transactions that were missing from old master and continue receving transactions from new master.

When we do new transactions on new master, we get different GTID.
Example, I did 1 transaction after failover on new master and my new master status looks like below

mysql> show master status \G
*************************** 1. row ***************************
             File: bin_log.000003
         Position: 2802
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: c21f28d2-c243-11e4-baf5-2c600c20dba4:1,
f7718b80-c237-11e4-baa8-a0369f370a52:1-93
1 row in set (0.00 sec)

c21f28d2-c243-11e4-baf5-2c600c20dba4:1 is the new GTID

If we just start slave on read replica it will not be able to do fetch any transaction as old master is down. So we need to first point read replica to new master and then start slave which will fetch all old and new transactions

root [mysql] >stop slave;
Query OK, 0 rows affected (0.03 sec)

root [mysql] >change master to master_host='mysql_b.example.com', master_port=3306, master_user='replicate', MASTER_PASSWORD = 'welcome', MASTER_AUTO_POSITION = 1;
Query OK, 0 rows affected, 2 warnings (0.07 sec)

root [mysql] >start slave;
Query OK, 0 rows affected (0.03 sec)

root [mysql] >show slave status \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: mysql_b.example.com
                  ...
                  ...
           Retrieved_Gtid_Set: c21f28d2-c243-11e4-baf5-2c600c20dba4:1,
f7718b80-c237-11e4-baa8-a0369f370a52:92-93
            Executed_Gtid_Set: 6e68fbf2-f9a0-11e4-a3f7-90e2ba6ebdb0:1-7,
c21f28d2-c243-11e4-baf5-2c600c20dba4:1,
f7718b80-c237-11e4-baa8-a0369f370a52:1-93
                Auto_Position: 1

In above output of slave status, we can check Retrieved_Gtid_Set and we can clearly see it has retrieved previous master’s 2 transactions f7718b80-c237-11e4-baa8-a0369f370a52:92-93 as well as new transaction that I did after failover – c21f28d2-c243-11e4-baf5-2c600c20dba4:1
In Executed_Gtid_Set we can see it has executed all these transactions – c21f28d2-c243-11e4-baf5-2c600c20dba4:1 and f7718b80-c237-11e4-baa8-a0369f370a52:1-93

Hope this helps !!