[12:13:58] FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:16:36] PROBLEM - MariaDB sustained replica lag on s1 on db2212 is CRITICAL: 67.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2212&var-port=9104 [12:17:36] RECOVERY - MariaDB sustained replica lag on s1 on db2212 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2212&var-port=9104 [12:27:12] PROBLEM - MariaDB sustained replica lag on s1 on db1195 is CRITICAL: 31 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1195&var-port=9104 [12:30:12] RECOVERY - MariaDB sustained replica lag on s1 on db1195 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1195&var-port=9104 [12:55:04] PROBLEM - MariaDB sustained replica lag on s1 on db1169 is CRITICAL: 12 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1169&var-port=9104 [12:55:41] Anything in particular going on in s1? [12:56:04] RECOVERY - MariaDB sustained replica lag on s1 on db1169 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1169&var-port=9104 [12:56:34] may have been some spikes, so it looks good now [12:59:30] PROBLEM - MariaDB sustained replica lag on s1 on db1235 is CRITICAL: 163.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1235&var-port=9104 [13:00:30] RECOVERY - MariaDB sustained replica lag on s1 on db1235 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1235&var-port=9104 [13:59:21] marostegui: federico: s[2-5] codfw are now doing rolling restarts, each run slightly offset in time from the previous one [14:58:56] elukey: do you happen to know if there is a reason for the linear backoff with a increment of 3s is set for spicereack.icinga.IcingaHosts.wait_for_optimal? For the DB, this frequently reaches 14/15 or even exceeds the retry window after a reboot [14:59:41] cezmunsta: I think it was a plausible default, but IIRC it can be tuned [15:01:09] The decorator on the "def check" doesn't look to accept adjustments via the call [15:02:16] The default on the wmflib one, which is ultimately called uses exponential as the default [15:06:14] I'll create a ticket, 3 out of 4 processed that I kicked off aborted on this [15:07:14] It could perhaps just be the "MariaDB sustained replica lag" check, which is an Icinga check using Grafana data [15:36:35] T433266 for ref - I have left it as a question of best option for now [15:36:36] T433266: spicerack.icinga.IcingaHosts.wait_for_optimal: allow user-defined interval and backoff - https://phabricator.wikimedia.org/T433266 [15:56:36] marostegui: any idea if these timeouts been happening for a while, or recently appeared? [16:14:27] FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:14:27] FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed