#==== EXAMPLE FOR THE codfw -> eqiad SWITCH ==== Executing commands ('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s1\' ORDER BY ts DESC LIMIT 1"',) on 1 hosts: db2048.codfw.wmnet ----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' ----- 2018-09-06T16:42:37.001790 ================ PASS: |####################################################################################################| 100% (1/1) [00:00<00:00, 3.06hosts/s] FAIL: | | 0% (0/1) [00:00= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'. 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. Executing commands ('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s1\' ORDER BY ts DESC LIMIT 1"',) on 1 hosts: db1067.eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' ----- 2018-09-06T16:42:50.001530 ================ PASS: |####################################################################################################| 100% (1/1) [00:00<00:00, 1.43hosts/s] FAIL: | | 0% (0/1) [00:00= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'. 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. And the later check passed without retries. This because we do first gather all the positions from dc_from and then check dc_to, so some time passes between one and the other.