Alerts first fired on the morning of Friday April 8th, following their re-image being kicked off late on the 7th (see T299609 and T294949): Apr 8 04:25:12 alert1001 icinga: HOST ALERT: ml-serve1005;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 08:17:14 alert1001 icinga: HOST ALERT: elastic1089;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 08:22:32 alert1001 icinga: HOST ALERT: elastic1091;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 08:22:48 alert1001 icinga: HOST ALERT: elastic1092;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 08:23:14 alert1001 icinga: HOST ALERT: elastic1090;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 08:27:14 alert1001 icinga: HOST ALERT: elastic1094;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 08:40:23 alert1001 icinga: HOST ALERT: elastic1095;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 08:47:37 alert1001 icinga: HOST ALERT: elastic1096;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 09:01:18 alert1001 icinga: HOST ALERT: elastic1101;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 09:01:30 alert1001 icinga: HOST ALERT: elastic1100;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% Apr 8 09:02:26 alert1001 icinga: HOST ALERT: elastic1102;DOWN;HARD;2;PING CRITICAL - Packet loss = 100% They remained in this state until the mac-ip table was cleared on associated top-of-rack switches on April 19th: Apr 19 07:32:31 alert1001 icinga: HOST ALERT: elastic1089;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.52 ms Apr 19 07:32:59 alert1001 icinga: HOST ALERT: elastic1092;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.32 ms Apr 19 07:32:59 alert1001 icinga: HOST ALERT: elastic1091;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.28 ms Apr 19 07:33:05 alert1001 icinga: HOST ALERT: elastic1090;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.27 ms Apr 19 07:33:09 alert1001 icinga: HOST ALERT: elastic1094;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.23 ms Apr 19 07:33:09 alert1001 icinga: HOST ALERT: ml-serve1005;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.29 ms Apr 19 07:33:37 alert1001 icinga: HOST ALERT: elastic1096;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.47 ms Apr 19 07:34:05 alert1001 icinga: HOST ALERT: elastic1095;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.42 ms Apr 19 07:34:27 alert1001 icinga: HOST ALERT: elastic1101;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 3.31 ms Apr 19 07:34:57 alert1001 icinga: HOST ALERT: elastic1102;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.91 ms Apr 19 07:34:57 alert1001 icinga: HOST ALERT: elastic1100;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.27 ms