[00:00:08] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:15] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:03:26] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1118.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1132.eqiad.wm [00:03:26] ikube-worker1330.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1069.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1147.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-worker1 [00:03:26] d.wmnet, wikikube-worker1142.eqiad.wmnet, wikikube-worker1057.eqiad.wmnet, wikikube-worker1131.eqiad.wmnet, wikikube-worker1076.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:03:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [00:06:25] FIRING: SystemdUnitFailed: sync-puppet-volatile.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:08:30] PROBLEM - Check unit status of sync-puppet-volatile on puppetserver2002 is CRITICAL: CRITICAL: Status of the systemd unit sync-puppet-volatile https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [00:09:19] (03PS1) 10RLazarus: Revert "kubernetes.yaml: Disable mwdebug-next httpbb checks" [puppet] - 10https://gerrit.wikimedia.org/r/1358895 [00:10:24] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 609.14 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:11:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:14:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1130.eqiad.wm [00:14:14] ikube-worker1044.eqiad.wmnet, wikikube-worker1164.eqiad.wmnet, wikikube-worker1051.eqiad.wmnet, wikikube-worker1342.eqiad.wmnet, wikikube-worker1140.eqiad.wmnet, wikikube-worker1373.eqiad.wmnet, wikikube-worker1074.eqiad.wmnet, wikikube-worker1120.eqiad.wmnet, wikikube-worker1117.eqiad.wmnet, wikikube-worker1271.eqiad.wmnet, wikikube-worker1346.eqiad.wmnet, wikikube-worker1316.eqiad.wmnet, wikikube-worker1348.eqiad.wmnet, wikikube-worker1 [00:14:14] d.wmnet, wikikube-worker1113.eqiad.wmnet, wikikube-worker1332.eqiad.wmnet, wikikube-worker1054.eqiad.wmnet, wikikube-worker1311.eqiad.wmnet, wikikube-worker1326.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:14:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:18:30] RECOVERY - Check unit status of sync-puppet-volatile on puppetserver2002 is OK: OK: Status of the systemd unit sync-puppet-volatile https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [00:19:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1322.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1288.eqiad.wm [00:19:24] ikube-worker1160.eqiad.wmnet, wikikube-worker1261.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1164.eqiad.wmnet, wikikube-worker1131.eqiad.wmnet, wikikube-worker1125.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1154.eqiad.wmnet, wikikube-worker1293.eqiad.wmnet, wikikube-worker1370.eqiad.wmnet, wikikube-worker1240.eqiad.wmnet, wikikube-worker1067.eqiad.wmnet, wikikube-worker1303.eqiad.wmnet, wikikube-worker1 [00:19:24] d.wmnet, wikikube-worker1296.eqiad.wmnet, wikikube-worker1257.eqiad.wmnet, wikikube-worker1375.eqiad.wmnet, wikikube-worker1080.eqiad.wmnet, wikikube-worker1276.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:21:25] RESOLVED: SystemdUnitFailed: sync-puppet-volatile.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:25:22] FIRING: [3x] SwitchCoreInterfaceDown: Switch core interface down - ssw1-a1-eqiad:lag1 (Core: asw2-a-eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [00:25:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [00:27:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:27:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:30:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1123.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1116.eqiad.wm [00:30:14] ikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1 [00:30:14] d.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1261.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:30:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-worker1335.eqiad.wm [00:30:24] ikube-worker1286.eqiad.wmnet, wikikube-worker1081.eqiad.wmnet, wikikube-worker1267.eqiad.wmnet, wikikube-worker1253.eqiad.wmnet, wikikube-worker1035.eqiad.wmnet, wikikube-worker1241.eqiad.wmnet, wikikube-worker1342.eqiad.wmnet, wikikube-worker1140.eqiad.wmnet, wikikube-worker1066.eqiad.wmnet, wikikube-worker1150.eqiad.wmnet, wikikube-worker1357.eqiad.wmnet, wikikube-worker1266.eqiad.wmnet, wikikube-worker1065.eqiad.wmnet, wikikube-worker1 [00:30:24] d.wmnet, wikikube-worker1296.eqiad.wmnet, wikikube-worker1382.eqiad.wmnet, wikikube-worker1367.eqiad.wmnet, wikikube-worker1122.eqiad.wmnet, wikikube-worker1073.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:32:26] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:33:55] FIRING: [2x] SystemdUnitFailed: rsyslog-imfile-remedy.service on aux-k8s-worker1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:35:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1291.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1263.eqiad.wm [00:35:24] ikube-worker1244.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1087.eqiad.wmnet, wikikube-worker1154.eqiad.wmnet, wikikube-worker1373.eqiad.wmnet, wikikube-worker1354.eqiad.wmnet, wikikube-worker1074.eqiad.wmnet, wikikube-worker1138.eqiad.wmnet, wikikube-worker1 [00:35:24] d.wmnet, wikikube-worker1381.eqiad.wmnet, wikikube-worker1370.eqiad.wmnet, wikikube-worker1240.eqiad.wmnet, wikikube-worker1292.eqiad.wmnet, wikikube-worker1120.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:35:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [00:38:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:38:24] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.05 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:38:26] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:38:51] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1358897 [00:38:51] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1358897 (owner: 10TrainBranchBot) [00:41:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1291.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1371.eqiad.wm [00:41:14] ikube-worker1016.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1 [00:41:14] d.wmnet, wikikube-worker1305.eqiad.wmnet, wikikube-worker1369.eqiad.wmnet, wikikube-worker1164.eqiad.wmnet, wikikube-worker1035.eqiad.wmnet, wikikube-worker1299.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:41:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wm [00:41:24] ikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1 [00:41:24] d.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:45:21] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1358897 (owner: 10TrainBranchBot) [00:46:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:46:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:49:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1042.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1344.eqiad.wm [00:49:14] ikube-worker1162.eqiad.wmnet, wikikube-worker1147.eqiad.wmnet, wikikube-worker1336.eqiad.wmnet, wikikube-worker1272.eqiad.wmnet, wikikube-worker1044.eqiad.wmnet, wikikube-worker1286.eqiad.wmnet, wikikube-worker1357.eqiad.wmnet, wikikube-worker1065.eqiad.wmnet, wikikube-worker1269.eqiad.wmnet, wikikube-worker1292.eqiad.wmnet, wikikube-worker1165.eqiad.wmnet, wikikube-worker1294.eqiad.wmnet, wikikube-worker1353.eqiad.wmnet, wikikube-worker1 [00:49:14] d.wmnet, wikikube-worker1276.eqiad.wmnet, wikikube-worker1052.eqiad.wmnet, wikikube-worker1349.eqiad.wmnet, wikikube-worker1246.eqiad.wmnet, wikikube-worker1374.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:49:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1121.eqiad.wm [00:49:24] ikube-worker1050.eqiad.wmnet, wikikube-worker1007.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1 [00:49:24] d.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [00:50:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [00:55:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [01:08:21] FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [01:08:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:08:37] !incidents [01:08:37] 8653 (ACKED, 73h 18m old) Manual (paged) by SBassett (sbassett@wikimedia.org): Need to confirm envoy and nginx config are protective in Wikimedia production [01:08:37] 8659 (ACKED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet eqiad) [01:08:38] 8658 (RESOLVED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet codfw) [01:09:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [01:11:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1281.eqiad.wm [01:11:24] ikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1 [01:11:24] d.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1296.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [01:19:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:19:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:19:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [01:21:14] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [01:22:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wm [01:22:14] ikube-worker1384.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1 [01:22:14] d.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [01:22:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1121.eqiad.wm [01:22:24] ikube-worker1050.eqiad.wmnet, wikikube-worker1274.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1 [01:22:24] d.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [01:23:21] FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [01:23:36] FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [01:24:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:24:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:27:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [01:27:44] Deployment eventstreams-production in eventstreams at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=eventstreams&var-deployment=eventstreams-production - ... [01:27:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [01:28:09] !incidents [01:28:10] 8653 (ACKED, 73h 37m old) Manual (paged) by SBassett (sbassett@wikimedia.org): Need to confirm envoy and nginx config are protective in Wikimedia production [01:28:10] 8659 (ACKED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet eqiad) [01:28:10] 8658 (RESOLVED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet codfw) [01:28:21] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqiad&var-cluster=text&var-origin=eventstreams.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [01:35:11] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [02:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0200) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0200) [02:08:00] (03CR) 10Ottomata: [C:03+1] revision-feature-counts lac-lambda: Add test wikis [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358864 (https://phabricator.wikimedia.org/T442448) (owner: 10AKhatun) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1291.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1049.eqiad.wm [02:14:14] ikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1 [02:14:14] d.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:15:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1347.eqiad.wm [02:15:24] ikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1007.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1 [02:15:24] d.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1296.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:16:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:16:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:24:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wm [02:24:14] ikube-worker1007.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1359.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1 [02:24:14] d.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:25:22] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [02:25:26] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wm [02:25:26] ikube-worker1334.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1 [02:25:26] d.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1353.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:26:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:26:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:35:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1371.eqiad.wm [02:35:14] ikube-worker1079.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1 [02:35:14] d.wmnet, wikikube-worker1066.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1135.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1147.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:35:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1347.eqiad.wm [02:35:24] ikube-worker1116.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1 [02:35:24] d.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:36:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:36:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:44:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1274.eqiad.wm [02:44:14] ikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1 [02:44:14] d.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:44:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wm [02:44:24] ikube-worker1132.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1 [02:44:24] d.wmnet, wikikube-worker1261.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1130.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:46:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:47:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:56:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1148.eqiad.wm [02:56:14] ikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1 [02:56:14] d.wmnet, wikikube-worker1021.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1336.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-worker1272.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:56:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1161.eqiad.wm [02:56:24] ikube-worker1157.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1256.eqiad.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1142.eqiad.wmnet, wikikube-worker1 [02:56:24] d.wmnet, wikikube-worker1369.eqiad.wmnet, wikikube-worker1125.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1373.eqiad.wmnet, wikikube-worker1357.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [02:57:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:57:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0300) [03:02:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [03:02:44] Deployment eventstreams-production in eventstreams at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=eventstreams&var-deployment=eventstreams-production - ... [03:02:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [03:06:01] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/1357757 (owner: 10Hnowlan) [03:11:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:20:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1380.eqiad.wm [03:20:14] ikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1 [03:20:14] d.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [03:20:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1116.eqiad.wm [03:20:24] ikube-worker1281.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1 [03:20:25] d.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [03:21:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [03:23:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:23:26] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:26:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [03:28:55] RESOLVED: SystemdUnitFailed: rsyslog-imfile-remedy.service on aux-k8s-worker1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:30:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1320.eqiad.wm [03:30:14] ikube-worker1315.eqiad.wmnet, wikikube-worker1074.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1296.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1 [03:30:14] d.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1289.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [03:30:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wm [03:30:24] ikube-worker1050.eqiad.wmnet, wikikube-worker1274.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1 [03:30:24] d.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1352.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [03:33:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:33:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:41:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1345.eqiad.wm [03:41:14] ikube-worker1121.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1359.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1 [03:41:14] d.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [03:41:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1259.eqiad.wm [03:41:24] ikube-worker1155.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1 [03:41:24] d.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [03:44:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:44:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:52:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [03:52:44] Deployment eventstreams-production in eventstreams at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=eventstreams&var-deployment=eventstreams-production - ... [03:52:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [04:00:04] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0400) [04:25:22] FIRING: [3x] SwitchCoreInterfaceDown: Switch core interface down - ssw1-a1-eqiad:lag1 (Core: asw2-a-eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [05:11:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [05:16:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [05:17:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [05:17:44] Deployment eventstreams-production in eventstreams at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=eventstreams&var-deployment=eventstreams-production - ... [05:17:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [05:19:20] !log tchin@deploy1003 helmfile [eqiad] START helmfile.d/services/eventstreams: apply [05:19:30] !log tchin@deploy1003 helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply [05:28:39] (03PS1) 10Nvdtn19: Enable NewUserMessage for viwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) [05:32:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=drmrs&var-cluster=text&var-origin=eventstreams.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [05:35:11] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [05:42:11] FIRING: ProbeDown: Service phab1005:443 has failed probes (http_phabricator_wikimedia_org_collab_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#phab1005:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:42:16] Phabricator seems very flaky for me (MySQL server has gone away), is this known? [05:42:36] chlod: yeah we are aware [05:42:43] gotcha, thanks! [05:47:11] RESOLVED: ProbeDown: Service phab1005:443 has failed probes (http_phabricator_wikimedia_org_collab_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#phab1005:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:47:27] ^ i don't think that's really resolve [05:47:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [05:47:44] Deployment eventstreams-production in eventstreams at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=eventstreams&var-deployment=eventstreams-production - ... [05:47:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [05:52:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=drmrs&var-cluster=text&var-origin=eventstreams.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [05:53:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [05:58:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0600) [06:00:04] marostegui, cezmunsta, and federico3: #bothumor My software never has bugs. It just develops random features. Rise for Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0600). [06:00:22] (03CR) 10Ayounsi: [C:03+1] "yep that lgtm!" [puppet] - 10https://gerrit.wikimedia.org/r/1358850 (https://phabricator.wikimedia.org/T442063) (owner: 10JHathaway) [06:00:37] (03CR) 10Tryvix1509: "`wgNewUserMinorEdit` isn't set to `true` for viwikivoyage." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [06:01:13] wiki_willy [06:01:16] oops [06:07:13] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12476066 (10ayounsi) I agree that we shouldn't have more than one cable connected at a time. Better to troubleshot the 10G nic than run with both. [06:12:37] (03CR) 10Nvdtn19: "`wgNewUserMinorEdit` is already `'default' => true`. I don't think it's necessarily to add this for viwikivoyage, unless viwikivoyage want" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [06:17:59] !log silence ATSBackendErrorsHigh eventstreams.discovery.wmnet drmrs page for 12h - T442462 [06:18:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:18:04] T442462: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462 [06:24:59] (03PS1) 10Hashar: Disable banner about emergency maintenance [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359144 [06:25:22] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:26:55] (03PS1) 10Ayounsi: network/data.yaml: Removed unused and WMCS v6 ranges from network::external [puppet] - 10https://gerrit.wikimedia.org/r/1359145 (https://phabricator.wikimedia.org/T442497) [06:27:21] jouncebot: nextandnow [06:27:26] jouncebot: nowandnext [06:27:26] For the next 0 hour(s) and 32 minute(s): MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0600) [06:27:26] For the next 0 hour(s) and 2 minute(s): Primary database switchover (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0600) [06:27:26] In 0 hour(s) and 32 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0700) [06:27:27] :b [06:27:38] oh yeah DST madness again pff [06:27:52] anyway I am going to update a javascript plugin on Gerrit (no restart etc) [06:28:09] (03CR) 10Hashar: [C:03+2] Disable banner about emergency maintenance [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359144 (owner: 10Hashar) [06:28:38] (03Merged) 10jenkins-bot: Disable banner about emergency maintenance [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359144 (owner: 10Hashar) [06:29:14] !log hashar@deploy1003 Started deploy [gerrit/gerrit@38dd63c]: Disable banner about emergency maintenance [06:29:29] !log hashar@deploy1003 Finished deploy [gerrit/gerrit@38dd63c]: Disable banner about emergency maintenance (duration: 00m 14s) [06:35:48] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4 [06:37:18] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4 [06:49:42] !log add 5xx responses as a % of total traffic to https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-3c-3e-origin-servers-overview - T400675 [06:49:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:49:47] T400675: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675 [06:50:04] (03PS1) 10Marostegui: clouddb1032: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1359149 [06:51:03] (03CR) 10Marostegui: [C:03+2] clouddb1032: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1359149 (owner: 10Marostegui) [07:00:05] Amir1, urbanecm, and awight: How many deployers does it take to do UTC morning backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T0700). [07:00:05] Neriah: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:07:53] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6 [07:09:00] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6 [07:09:17] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 [07:10:13] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 [07:11:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:17:02] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6 [07:18:30] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6 [07:23:32] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 [07:23:33] !log marostegui@cumin1004 END (FAIL) - Cookbook sre.switchdc.databases.prepare (exit_code=93) for the switch from eqiad to codfw for section s5 [07:23:44] !log slyngshede@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp [07:23:53] !log slyngshede@cumin1004 END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp [07:24:12] !log slyngshede@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp [07:24:35] !log slyngshede@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp2056*} OR P{cp2058*} and A:cp [07:24:56] hi [07:24:57] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 [07:25:37] Is there a deployer available? [07:26:04] ping Amir1 and urbanecm [07:26:17] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5 [07:32:47] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet [07:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:34:25] (03CR) 10Arthur taylor: [C:03+2] wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358692 (https://phabricator.wikimedia.org/T441994) (owner: 10Arthur taylor) [07:35:20] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet [07:36:42] (03Merged) 10jenkins-bot: wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358692 (https://phabricator.wikimedia.org/T441994) (owner: 10Arthur taylor) [07:37:24] !log arthurtaylor@deploy1003 helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [07:37:53] !log arthurtaylor@deploy1003 helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [07:38:01] !log arthurtaylor@deploy1003 helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [07:38:23] !log arthurtaylor@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply [07:38:30] !log arthurtaylor@deploy1003 helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [07:38:52] !log arthurtaylor@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [07:43:10] hmmm [07:43:12] Is there a deployer available? [07:44:20] the deployment window is about to end [07:44:30] hashar: any releng folks about ^ [07:45:52] Dreamy_Jazz: fyi too ^ [07:50:22] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2 [07:52:09] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2 [07:57:19] Neriah: it seems not, you may have to reschedule your patch for later in the day [08:02:09] Ugh, this is frustrating. I'm not sure I'll have time later today... [08:02:29] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3 [08:03:56] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3 [08:04:19] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7 [08:05:51] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie [08:05:57] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7 [08:05:59] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (push config only) [08:06:03] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (push config only) [08:06:27] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8 [08:07:51] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8 [08:08:04] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4 [08:08:31] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet [08:09:03] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet [08:09:32] !log elukey@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2006'] [08:09:32] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4 [08:09:38] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sretest2006'] [08:10:29] !log elukey@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2006'] [08:11:34] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet [08:12:37] jmm@cumin2003 drain-node (PID 1102110) is awaiting input [08:13:20] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet [08:13:24] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1 [08:13:35] 06SRE, 10SRE-Access-Requests: Requesting access to codesearch-roots for addshore - https://phabricator.wikimedia.org/T442219#12476341 (10LSobanski) [08:13:53] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet [08:13:54] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2056*} OR P{cp2058*} and A:cp [08:15:00] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1 [08:15:33] !log Enabled database circular replication https://phabricator.wikimedia.org/T439063 [08:15:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:16:16] jmm@cumin2003 drain-node (PID 1102219) is awaiting input [08:19:23] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet [08:19:55] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet [08:20:13] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet [08:20:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet [08:24:44] jmm@cumin2003 drain-node (PID 1103421) is awaiting input [08:25:22] FIRING: [3x] SwitchCoreInterfaceDown: Switch core interface down - ssw1-a1-eqiad:lag1 (Core: asw2-a-eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [08:25:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 27 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) (owner: 10Neriah) [08:25:56] elukey@cumin1004 upgrade-firmware (PID 3335107) is awaiting input [08:26:28] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet [08:26:35] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet [08:28:41] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2026.codfw.wmnet [08:28:46] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet [08:29:43] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (remove config only) [08:29:47] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (remove config only) [08:29:47] !log elukey@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie [08:32:43] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS trixie [08:33:39] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs4010.ulsfo.wmnet (push config only) [08:33:44] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4010.ulsfo.wmnet (push config only) [08:34:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet [08:35:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet [08:35:29] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet [08:35:39] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet [08:35:53] !log elukey@cumin1004 START - Cookbook sre.hosts.powercycle for host ml-serve1016 [08:37:55] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet [08:39:50] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet [08:40:24] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2026.codfw.wmnet [08:40:43] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2041.codfw.wmnet [08:40:57] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host ml-serve1016 [08:42:42] Neriah: RhinosF1: yeah sorry I was around but did not pay attention to the notifications [08:43:52] !log elukey@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2006'] [08:44:21] jouncebot: nowandnext [08:44:22] No deployments scheduled for the next 1 hour(s) and 15 minute(s) [08:44:22] In 1 hour(s) and 15 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1000) [08:45:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet [08:45:49] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet [08:46:09] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2041.codfw.wmnet [08:46:27] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs4010.ulsfo.wmnet (remove config only) [08:46:34] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4010.ulsfo.wmnet (remove config only) [08:48:01] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet [08:48:09] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:48:16] (checked with SREs got a +1) [08:48:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by hashar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) (owner: 10Neriah) [08:48:29] hashar: has something changed with the new node26 CI images in relation to cypress? https://integration.wikimedia.org/ci/job/generic-node26-browser/12/console [08:48:46] Dreamy_Jazz: I am rolling a mediawiki-config change you +1ed earlier https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1358880 "Allow those with CheckUser right to view private filters on WMF projects" [08:48:48] This job is failing, and nothing has really changed about the app or its (cypress-related) dependencies [08:49:09] codders: I'll check after the deployment :) [08:49:13] (03Merged) 10jenkins-bot: Allow those with CheckUser right to view private filters on WMF projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) (owner: 10Neriah) [08:49:15] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage [08:49:19] yup - no stress! thanks :) [08:49:56] !log hashar@deploy1003 Started scap sync-world: Backport for [[gerrit:1358880|Allow those with CheckUser right to view private filters on WMF projects (T442464)]] [08:50:01] T442464: Allow those with CheckUser right to view private filters on WMF projects - https://phabricator.wikimedia.org/T442464 [08:50:04] !log elukey@cumin1004 START - Cookbook sre.hosts.powercycle for host ml-serve1016 [08:50:27] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet [08:51:29] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2041.codfw.wmnet [08:51:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2041.codfw.wmnet [08:53:39] jmm@cumin2003 drain-node (PID 1108834) is awaiting input [08:53:48] thank you hashar! [08:54:20] !log hashar@deploy1003 neriah, hashar: Backport for [[gerrit:1358880|Allow those with CheckUser right to view private filters on WMF projects (T442464)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:54:24] Neriah: we had a busy week last week and the daylight saving time has shifted the backport window one hour earlier than it used to :) [08:54:27] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage [08:54:33] plus honestly, I was defnitely NOT paying attention :-] [08:54:47] np [08:55:07] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host ml-serve1016 [08:55:08] Neriah: do you have time to validate on the test servers with the debug extension? [08:55:12] it is there already [08:55:24] then I am not sure we need to verify. The change is quite straightforward [08:56:01] yes [08:56:01] I'm testing [08:58:06] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:59:04] hashar: looks good, you can continue [08:59:18] !log hashar@deploy1003 neriah, hashar: Continuing with deployment [08:59:23] Neriah: you are awesome thank you! [08:59:25] !log fabfur@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_esams [09:02:28] !log imported routinator 0.15.3 to thirdparty/routinator [09:02:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:02:50] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2042.codfw.wmnet [09:02:57] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet [09:04:46] !log upgrade rpki2003 to routinator 0.15.3 [09:04:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:05:00] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie [09:05:08] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only) [09:05:08] !log hashar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1358880|Allow those with CheckUser right to view private filters on WMF projects (T442464)]] (duration: 15m 12s) [09:05:12] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only) [09:05:15] T442464: Allow those with CheckUser right to view private filters on WMF projects - https://phabricator.wikimedia.org/T442464 [09:05:21] Neriah: your patch has been fully deployed! [09:06:25] yay [09:06:27] thanks for the deploy! [09:07:36] !log bump space for prometheus k8s-(dse|aux|ml-serve) in eqiad [09:07:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:08:17] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet [09:08:24] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet [09:09:39] jmm@cumin2003 drain-node (PID 1112892) is awaiting input [09:11:24] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet [09:12:17] (03PS2) 10Fabfur: cache::haproxy: set low prio QoS to all files from upload [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) [09:12:40] (03CR) 10Fabfur: cache::haproxy: set low prio QoS to all files from upload (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:13:40] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12476541 (10cmooney) >>! In T438161#12472660, @elukey wrote: > @Jclark-ctr is there a specific reason for it? Is the 10G NIC not recognized in the boot list? I can try to tak... [09:15:02] (03CR) 10Fabfur: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:15:15] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie [09:15:16] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS trixie [09:16:43] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie [09:16:52] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only) [09:16:56] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only) [09:17:16] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12476562 (10cmooney) [09:17:24] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie [09:17:32] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (push config only) [09:17:36] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (push config only) [09:17:59] (03CR) 10Cathal Mooney: [C:03+1] LbertyGlobal: disable flowspec AF [homer/public] - 10https://gerrit.wikimedia.org/r/1358695 (https://phabricator.wikimedia.org/T441325) (owner: 10Ayounsi) [09:18:36] (03CR) 10Cathal Mooney: [C:03+1] "Good shout." [homer/public] - 10https://gerrit.wikimedia.org/r/1358757 (https://phabricator.wikimedia.org/T441325) (owner: 10Ayounsi) [09:18:37] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2042.codfw.wmnet [09:18:47] (03PS7) 101Veertje: Add rotating Commons background to the Gerrit login page [puppet] - 10https://gerrit.wikimedia.org/r/1345634 (https://phabricator.wikimedia.org/T442505) [09:18:51] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet [09:18:59] (03PS8) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (https://phabricator.wikimedia.org/T442505) [09:20:30] PROBLEM - Host aux-k8s-etcd2003 is DOWN: PING CRITICAL - Packet loss = 100% [09:20:33] (03PS9) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (https://phabricator.wikimedia.org/T442505) [09:20:51] (03CR) 10Brouberol: [C:03+1] "Looks good!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [09:21:17] (03PS7) 101Veertje: Move Gerrit login styling into GerritSite.css [puppet] - 10https://gerrit.wikimedia.org/r/1345606 (https://phabricator.wikimedia.org/T442505) [09:21:58] jmm@cumin2003 drain-node (PID 1114549) is awaiting input [09:22:26] (03PS1) 10Majavah: Revert "Temporarily disable public openstack APIS" [puppet] - 10https://gerrit.wikimedia.org/r/1359207 [09:22:49] (03CR) 10CI reject: [V:04-1] Revert "Temporarily disable public openstack APIS" [puppet] - 10https://gerrit.wikimedia.org/r/1359207 (owner: 10Majavah) [09:23:43] (03CR) 10Ayounsi: [C:03+2] Disable LLDP on OOB interfaces [homer/public] - 10https://gerrit.wikimedia.org/r/1358757 (https://phabricator.wikimedia.org/T441325) (owner: 10Ayounsi) [09:23:57] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2042.codfw.wmnet [09:24:16] (03CR) 10Ayounsi: [C:03+2] LbertyGlobal: disable flowspec AF [homer/public] - 10https://gerrit.wikimedia.org/r/1358695 (https://phabricator.wikimedia.org/T441325) (owner: 10Ayounsi) [09:24:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2042.codfw.wmnet [09:24:34] !log installing ca-certificates updates for trixie [09:24:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:24:44] (03Merged) 10jenkins-bot: Disable LLDP on OOB interfaces [homer/public] - 10https://gerrit.wikimedia.org/r/1358757 (https://phabricator.wikimedia.org/T441325) (owner: 10Ayounsi) [09:25:03] (03PS2) 10Majavah: Restore OpenStack APIs [puppet] - 10https://gerrit.wikimedia.org/r/1359207 [09:25:03] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2043.codfw.wmnet [09:25:24] (03Merged) 10jenkins-bot: LbertyGlobal: disable flowspec AF [homer/public] - 10https://gerrit.wikimedia.org/r/1358695 (https://phabricator.wikimedia.org/T441325) (owner: 10Ayounsi) [09:25:32] RECOVERY - Host aux-k8s-etcd2003 is UP: PING OK - Packet loss = 0%, RTA = 30.61 ms [09:27:08] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie [09:27:14] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9560/co" [puppet] - 10https://gerrit.wikimedia.org/r/1359207 (owner: 10Majavah) [09:28:10] jmm@cumin2003 drain-node (PID 1118499) is awaiting input [09:29:18] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet [09:29:22] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet [09:31:02] (03CR) 10Slyngshede: [C:03+1] cache::haproxy: set low prio QoS to all files from upload [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:33:57] (03CR) 10Vgutierrez: cache::haproxy: set low prio QoS to all files from upload (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:34:50] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet [09:35:11] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:35:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet [09:36:01] jmm@cumin2003 drain-node (PID 1118499) is awaiting input [09:36:34] (03PS1) 10Vgutierrez: hieradata: Depool lvs4009 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359213 (https://phabricator.wikimedia.org/T401832) [09:36:42] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (remove config only) [09:36:46] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (remove config only) [09:36:46] !log elukey@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie [09:36:50] (03PS1) 10Ayounsi: Apply LLDP disable to mr roles too [homer/public] - 10https://gerrit.wikimedia.org/r/1359214 [09:37:39] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie [09:37:46] (03CR) 10Cathal Mooney: [C:03+1] Apply LLDP disable to mr roles too [homer/public] - 10https://gerrit.wikimedia.org/r/1359214 (owner: 10Ayounsi) [09:37:47] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only) [09:37:51] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only) [09:38:20] (03CR) 10Ayounsi: [C:03+2] Apply LLDP disable to mr roles too [homer/public] - 10https://gerrit.wikimedia.org/r/1359214 (owner: 10Ayounsi) [09:39:18] (03Merged) 10jenkins-bot: Apply LLDP disable to mr roles too [homer/public] - 10https://gerrit.wikimedia.org/r/1359214 (owner: 10Ayounsi) [09:39:35] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2043.codfw.wmnet [09:39:43] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet [09:39:56] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie [09:41:15] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359213 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:41:34] PROBLEM - Host dse-k8s-etcd2003 is DOWN: PING CRITICAL - Packet loss = 100% [09:41:37] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie [09:41:45] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only) [09:41:50] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only) [09:42:14] (03PS3) 10Fabfur: cache::haproxy: set low prio QoS to all files from upload [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) [09:42:32] (03CR) 10Fabfur: cache::haproxy: set low prio QoS to all files from upload (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:42:49] jmm@cumin2003 drain-node (PID 1121270) is awaiting input [09:44:52] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs4009 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359213 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:45:17] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2043.codfw.wmnet [09:45:35] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2043.codfw.wmnet [09:46:02] RECOVERY - Host dse-k8s-etcd2003 is UP: PING OK - Packet loss = 0%, RTA = 31.98 ms [09:46:05] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2044.codfw.wmnet [09:46:05] (03CR) 10Vgutierrez: cache::haproxy: set low prio QoS to all files from upload (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:46:09] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1048.eqiad.wmnet [09:46:47] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs4009 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359213 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:46:49] (03CR) 10Fabfur: [C:03+1] hieradata: Depool lvs4009 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359213 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:46:57] (03CR) 10Hnowlan: [C:03+2] Revert "logstash: increase mediawiki retention" [puppet] - 10https://gerrit.wikimedia.org/r/1357757 (owner: 10Hnowlan) [09:47:25] Thanks hashar for deploying the change [09:48:00] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica [09:48:08] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica [09:48:22] PROBLEM - Host dse-k8s-etcd1003 is DOWN: PING CRITICAL - Packet loss = 100% [09:48:30] PROBLEM - Host ml-etcd1002 is DOWN: PING CRITICAL - Packet loss = 100% [09:49:17] !log extend silence ATSBackendErrorsHigh eventstreams.discovery.wmnet to all sites for another ~8h - T442462 [09:49:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:49:22] T442462: 2026-10-26 stream.wikimedia.org 5xx errors - https://phabricator.wikimedia.org/T442462 [09:49:40] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica [09:49:44] Neriah: are you still around and do you still need a patch deployed? I'm not usually awake at that of the day :/ I should really switch to another window. [09:49:59] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica [09:50:52] RECOVERY - Host dse-k8s-etcd1003 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [09:51:02] RECOVERY - Host ml-etcd1002 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [09:51:29] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1048.eqiad.wmnet [09:51:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1048.eqiad.wmnet [09:52:05] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS trixie [09:52:46] (03PS4) 10Fabfur: cache::haproxy: set low prio QoS to all files from upload [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) [09:52:52] jmm@cumin2003 drain-node (PID 1121930) is awaiting input [09:53:01] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet [09:53:59] (03CR) 10Vgutierrez: [C:03+1] cache::haproxy: set low prio QoS to all files from upload [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:55:01] (03CR) 10Brouberol: [C:03+2] etcd: add restart/reboot cookbook (034 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1358763 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [09:55:07] (03CR) 10Fabfur: cache::haproxy: set low prio QoS to all files from upload (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:55:37] (03CR) 10Fabfur: [C:03+2] cache::haproxy: set low prio QoS to all files from upload [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:55:48] (03CR) 10Cathal Mooney: [C:03+1] "Intent here looks good to me, the subtleties of the syntax I leave to the traffic team thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [09:56:52] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2044.codfw.wmnet [09:56:56] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1058.eqiad.wmnet [09:56:58] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs4009.ulsfo.wmnet with OS trixie [09:57:50] (03PS1) 10Jelto: cdn: exclude miscweb from ATSBackendErrorsHigh [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) [09:57:55] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs4009.ulsfo.wmnet (push config only) [09:58:01] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4009.ulsfo.wmnet (push config only) [09:58:32] PROBLEM - Host dse-k8s-ctrl2002 is DOWN: PING CRITICAL - Packet loss = 100% [09:58:48] PROBLEM - Host kubestagemaster2003 is DOWN: PING CRITICAL - Packet loss = 100% [09:59:24] PROBLEM - Host ml-etcd2003 is DOWN: PING CRITICAL - Packet loss = 100% [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1000) [10:00:07] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12476699 (10Fabfur) 05Open→03Resolved This should be ok now [10:00:09] (03PS3) 10Majavah: Restore OpenStack APIs [puppet] - 10https://gerrit.wikimedia.org/r/1359207 (https://phabricator.wikimedia.org/T442042) [10:00:52] RECOVERY - Host ml-etcd2003 is UP: PING OK - Packet loss = 0%, RTA = 32.05 ms [10:01:00] RECOVERY - Host dse-k8s-ctrl2002 is UP: PING OK - Packet loss = 0%, RTA = 30.73 ms [10:01:00] RECOVERY - Host kubestagemaster2003 is UP: PING OK - Packet loss = 0%, RTA = 30.58 ms [10:02:13] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2044.codfw.wmnet [10:02:19] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2044.codfw.wmnet [10:02:32] (03PS1) 10Brouberol: cumin/aliases: define aliases to address the main/core etcds [puppet] - 10https://gerrit.wikimedia.org/r/1359219 (https://phabricator.wikimedia.org/T420588) [10:03:14] jmm@cumin2003 drain-node (PID 1138960) is awaiting input [10:03:26] (03PS2) 10Brouberol: cumin/aliases: define aliases to address the main/core etcds [puppet] - 10https://gerrit.wikimedia.org/r/1359219 (https://phabricator.wikimedia.org/T420588) [10:03:39] (03CR) 10Neriah: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [10:03:57] (03CR) 10Clément Goubert: [C:03+1] services_proxy:linkrecommendation: Increase timeout to 31s [puppet] - 10https://gerrit.wikimedia.org/r/1358890 (https://phabricator.wikimedia.org/T442468) (owner: 10Urbanecm) [10:04:02] (03CR) 10Clément Goubert: [C:03+1] GrowthExperiments: Set link recommendation service timeout to 32s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358891 (https://phabricator.wikimedia.org/T442468) (owner: 10Urbanecm) [10:04:41] FIRING: ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [10:04:43] (03CR) 10Clément Goubert: [C:03+1] Revert "kubernetes.yaml: Disable mwdebug-next httpbb checks" [puppet] - 10https://gerrit.wikimedia.org/r/1358895 (owner: 10RLazarus) [10:06:57] !log fabfur@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_esams [10:07:27] (03CR) 10Muehlenhoff: cumin/aliases: define aliases to address the main/core etcds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359219 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [10:08:17] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet [10:08:40] FIRING: KubernetesRsyslogDown: rsyslog on dse-k8s-worker2007:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker2007 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [10:09:41] FIRING: [7x] ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [10:10:24] (03PS1) 10Brouberol: etcd: address post-merge reviews from last patch [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [10:10:56] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs4009.ulsfo.wmnet (remove config only) [10:11:04] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4009.ulsfo.wmnet (remove config only) [10:11:15] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet [10:13:50] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4009.ulsfo.wmnet with reason: host reimage [10:14:06] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (remove config only) [10:14:11] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (remove config only) [10:14:41] FIRING: [13x] ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [10:15:41] (03CR) 10Brouberol: cumin/aliases: define aliases to address the main/core etcds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359219 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [10:16:10] (03Abandoned) 10Brouberol: cumin/aliases: define aliases to address the main/core etcds [puppet] - 10https://gerrit.wikimedia.org/r/1359219 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [10:16:23] (03PS1) 10Fabfur: cache:haproxy: fix typo in haproxy conf [puppet] - 10https://gerrit.wikimedia.org/r/1359222 (https://phabricator.wikimedia.org/T440990) [10:16:44] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet [10:16:46] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet [10:16:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1058.eqiad.wmnet [10:17:38] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4009.ulsfo.wmnet with reason: host reimage [10:18:40] RESOLVED: KubernetesRsyslogDown: rsyslog on dse-k8s-worker2007:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker2007 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [10:19:11] (03CR) 10Vgutierrez: [C:03+1] cache:haproxy: fix typo in haproxy conf [puppet] - 10https://gerrit.wikimedia.org/r/1359222 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [10:19:41] FIRING: [21x] ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [10:19:41] (03CR) 10Filippo Giunchedi: [C:03+1] Restore OpenStack APIs [puppet] - 10https://gerrit.wikimedia.org/r/1359207 (https://phabricator.wikimedia.org/T442042) (owner: 10Majavah) [10:20:02] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [10:20:12] (03CR) 10Fabfur: [C:03+2] cache:haproxy: fix typo in haproxy conf [puppet] - 10https://gerrit.wikimedia.org/r/1359222 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [10:20:20] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock2001.codfw.wmnet [10:20:36] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host rdb-lock2001.codfw.wmnet [10:21:29] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock2001.codfw.wmnet [10:21:33] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host rdb-lock2001.codfw.wmnet [10:21:39] (03CR) 10Vgutierrez: [C:03+1] "https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1357479 needs to be merged & deployed first" [puppet] - 10https://gerrit.wikimedia.org/r/1357480 (https://phabricator.wikimedia.org/T441912) (owner: 10Dpogorzelski) [10:21:50] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [10:22:03] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock2001.codfw.wmnet [10:22:05] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [10:22:57] jouncebot nowandnext [10:22:58] For the next 0 hour(s) and 37 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1000) [10:22:58] In 1 hour(s) and 37 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1200) [10:23:32] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb1014.eqiad.wmnet [10:24:41] FIRING: [28x] ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [10:25:22] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:25:52] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock2001.codfw.wmnet [10:27:38] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock2002.codfw.wmnet [10:28:00] (03PS2) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [10:28:50] (03PS3) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [10:29:41] FIRING: [32x] ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [10:29:45] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1014.eqiad.wmnet [10:30:02] (03PS1) 10JavierMonton: spiderpig: relative-trending deployment [puppet] - 10https://gerrit.wikimedia.org/r/1359226 (https://phabricator.wikimedia.org/T442341) [10:31:15] (03CR) 10Filippo Giunchedi: [C:03+1] "Ok thank you for the explanation, +Taavi though LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) (owner: 10Dzahn) [10:31:38] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock2002.codfw.wmnet [10:31:47] (03CR) 10CI reject: [V:04-1] migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [10:31:59] (03CR) 10Majavah: [C:03+2] Restore OpenStack APIs [puppet] - 10https://gerrit.wikimedia.org/r/1359207 (https://phabricator.wikimedia.org/T442042) (owner: 10Majavah) [10:32:10] (03PS4) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [10:32:41] (03PS5) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [10:32:53] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet [10:33:10] (03PS6) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [10:34:18] (03CR) 10Majavah: [C:04-1] puppet-diffs: give cloudinfra-cloudvps-puppetserver its own facts realm (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) (owner: 10Dzahn) [10:34:27] jouncebot: nowandnext [10:34:27] For the next 0 hour(s) and 25 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1000) [10:34:28] In 1 hour(s) and 25 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1200) [10:34:31] (03CR) 10Filippo Giunchedi: "Specifically for this process-only check though it can go now, correct?" [puppet] - 10https://gerrit.wikimedia.org/r/1344231 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [10:34:41] RESOLVED: [32x] ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [10:36:26] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4009.ulsfo.wmnet with OS trixie [10:37:02] (03CR) 10Jelto: "looks mostly fine to me but some questions in line (mostly process-related)" [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [10:38:42] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock2003.codfw.wmnet [10:38:44] (03PS1) 10Brouberol: dumps/distribution: set a more informative brownout info URL [puppet] - 10https://gerrit.wikimedia.org/r/1359232 (https://phabricator.wikimedia.org/T437461) [10:38:58] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb1016.eqiad.wmnet [10:41:28] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12476836 (10CWilliams-WMF) @Volans having cookbooks fail at random points due to 5xx or Phabricator down issues mak... [10:42:31] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock2003.codfw.wmnet [10:44:27] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1016.eqiad.wmnet [10:45:40] (03CR) 10Dpogorzelski: [C:03+2] rest-gateway: route hummingbird.wikimedia.org to liftwing-studio [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357479 (https://phabricator.wikimedia.org/T441912) (owner: 10Dpogorzelski) [10:45:51] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12476842 (10CWilliams-WMF) [10:45:52] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb2012.codfw.wmnet [10:46:19] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp2059.codfw.wmnet [10:46:20] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp2049*} OR P{cp2051*} OR P{cp2053*} OR P{cp2055*} OR P{cp2057*} OR P{cp2059*} and A:cp [10:46:46] !log dpogorzelski@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [10:46:51] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs4009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359233 [10:47:19] !log dpogorzelski@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [10:47:30] (03PS2) 10CWilliams: Gracefully handle Phabricator being inaccessible during Spicerack calls [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359228 (https://phabricator.wikimedia.org/T442374) [10:48:07] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock1001.eqiad.wmnet [10:50:20] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs4009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359233 (https://phabricator.wikimedia.org/T401832) [10:50:28] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359233 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:51:10] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2012.codfw.wmnet [10:51:48] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb2014.codfw.wmnet [10:52:08] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock1001.eqiad.wmnet [10:54:00] (03CR) 10Guilherme Gonçalves: [V:03+1 C:03+1] dumps/distribution: set a more informative brownout info URL [puppet] - 10https://gerrit.wikimedia.org/r/1359232 (https://phabricator.wikimedia.org/T437461) (owner: 10Brouberol) [10:54:04] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock1002.eqiad.wmnet [10:54:12] !log dpogorzelski@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [10:54:15] (03CR) 10Brouberol: [C:03+2] dumps/distribution: set a more informative brownout info URL [puppet] - 10https://gerrit.wikimedia.org/r/1359232 (https://phabricator.wikimedia.org/T437461) (owner: 10Brouberol) [10:54:37] !log dpogorzelski@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [10:54:42] (03PS1) 10Elukey: sre.hosts.dhcp: add retry when contacting the BMC [cookbooks] - 10https://gerrit.wikimedia.org/r/1359235 [10:55:57] !log dpogorzelski@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [10:55:58] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS trixie [10:56:05] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (push config only) [10:56:10] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (push config only) [10:56:26] !log dpogorzelski@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [10:56:32] (03CR) 10Marostegui: "Adding Luca in case he's got time before he leaves XD" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359228 (https://phabricator.wikimedia.org/T442374) (owner: 10CWilliams) [10:57:02] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs4009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359233 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:57:06] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2014.codfw.wmnet [10:57:54] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock1002.eqiad.wmnet [10:58:35] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet [10:59:10] (03CR) 10Dpogorzelski: "curl -sv --connect-to hummingbird.wikimedia.org:443:rest-gateway.discovery.wmnet:4113 https://hummingbird.wikimedia.org/ -o /dev/null" [puppet] - 10https://gerrit.wikimedia.org/r/1357480 (https://phabricator.wikimedia.org/T441912) (owner: 10Dpogorzelski) [10:59:35] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb-lock1003.eqiad.wmnet [10:59:39] (03CR) 10Dpogorzelski: [C:03+2] cache: send hummingbird.wikimedia.org through the rest-gateway [puppet] - 10https://gerrit.wikimedia.org/r/1357480 (https://phabricator.wikimedia.org/T441912) (owner: 10Dpogorzelski) [10:59:50] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica [11:00:08] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009.ulsfo.wmnet} and A:liberica [11:00:55] (03PS7) 10Brouberol: migrate the sre.etcd.roll-restart-reboot cookbook under sre.k8s [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) [11:03:08] (03PS1) 10Vgutierrez: hieradata: Depool lvs4008 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359238 (https://phabricator.wikimedia.org/T401832) [11:03:14] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb-lock1003.eqiad.wmnet [11:05:37] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359238 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:05:59] (03PS2) 10Meno25: Update redirected links [puppet] - 10https://gerrit.wikimedia.org/r/1359236 [11:05:59] (03CR) 10Meno25: "Minor change: Updated redirected links. Please review. Thanks" [puppet] - 10https://gerrit.wikimedia.org/r/1359236 (owner: 10Meno25) [11:07:08] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2006.codfw.wmnet (remove config only) [11:07:13] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2006.codfw.wmnet (remove config only) [11:09:24] !log elukey@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2006.codfw.wmnet with reason: host reimage [11:11:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:12:59] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb1013.eqiad.wmnet [11:13:07] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2006.codfw.wmnet with reason: host reimage [11:13:12] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs4008 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359238 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:14:20] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb2011.codfw.wmnet [11:14:29] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12476918 (10elukey) Summary of my tests so far: * Confirmed that Debian Install doesn't see any nvme drives, but I clearly recall it working and T438161#12352303 supports it... [11:14:53] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet [11:15:57] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica [11:16:15] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica [11:17:14] (03PS1) 10Hnowlan: mediawiki: remove nrpe_check_opcache [puppet] - 10https://gerrit.wikimedia.org/r/1359244 (https://phabricator.wikimedia.org/T407331) [11:17:34] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs4008 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359238 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:18:20] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Depool lvs4009 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359233 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:19:10] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1013.eqiad.wmnet [11:19:28] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2011.codfw.wmnet [11:20:01] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [11:22:11] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS trixie [11:22:16] (03CR) 10Clément Goubert: [C:03+1] mediawiki: remove nrpe_check_opcache [puppet] - 10https://gerrit.wikimedia.org/r/1359244 (https://phabricator.wikimedia.org/T407331) (owner: 10Hnowlan) [11:22:22] (03CR) 10Kosta Harlan: [C:03+1] WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358893 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [11:23:06] (03CR) 10Hnowlan: [C:03+2] mediawiki: remove nrpe_check_opcache [puppet] - 10https://gerrit.wikimedia.org/r/1359244 (https://phabricator.wikimedia.org/T407331) (owner: 10Hnowlan) [11:23:07] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs4008.ulsfo.wmnet (push config only) [11:23:12] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4008.ulsfo.wmnet (push config only) [11:26:58] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet [11:28:08] !log upload purged 0.25+deb13u2 to trixie-wikimedia (apt.wm.o) - T442513 [11:28:08] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb1015.eqiad.wmnet [11:28:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:28:14] T442513: Use go1.27 to compile purged - https://phabricator.wikimedia.org/T442513 [11:30:14] (03CR) 10Marostegui: [C:03+1] cdn: exclude miscweb from ATSBackendErrorsHigh (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) (owner: 10Jelto) [11:30:24] going to do a backport for T442468 in just a few minutes [11:30:24] T442468: linkrecommendation envoy timeout is shorter than app timeout - https://phabricator.wikimedia.org/T442468 [11:30:27] !log testing purged 0.25+deb13u2 in cp7010 and cp7016 - T442513 [11:30:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:30:33] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2006.codfw.wmnet with OS trixie [11:31:25] (03CR) 10Blake: [C:03+2] services_proxy:linkrecommendation: Increase timeout to 31s [puppet] - 10https://gerrit.wikimedia.org/r/1358890 (https://phabricator.wikimedia.org/T442468) (owner: 10Urbanecm) [11:32:30] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet [11:33:28] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1015.eqiad.wmnet [11:34:25] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo02_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:36:55] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs4008.ulsfo.wmnet (remove config only) [11:37:01] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs4008.ulsfo.wmnet (remove config only) [11:37:48] !log cgoubert@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw [11:37:53] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet [11:37:55] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet [11:38:53] (03CR) 10Hnowlan: [C:03+2] team-sre: add exporter-based host_down check [alerts] - 10https://gerrit.wikimedia.org/r/1351431 (https://phabricator.wikimedia.org/T395448) (owner: 10Hnowlan) [11:39:49] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage [11:40:19] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet [11:41:45] (03Merged) 10jenkins-bot: team-sre: add exporter-based host_down check [alerts] - 10https://gerrit.wikimedia.org/r/1351431 (https://phabricator.wikimedia.org/T395448) (owner: 10Hnowlan) [11:42:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by blake@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358891 (https://phabricator.wikimedia.org/T442468) (owner: 10Urbanecm) [11:42:43] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage [11:43:13] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet [11:43:17] (03Merged) 10jenkins-bot: GrowthExperiments: Set link recommendation service timeout to 32s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358891 (https://phabricator.wikimedia.org/T442468) (owner: 10Urbanecm) [11:43:41] !log blake@deploy1003 Started scap sync-world: Backport for [[gerrit:1358891|GrowthExperiments: Set link recommendation service timeout to 32s (T442468)]] [11:43:46] T442468: linkrecommendation envoy timeout is shorter than app timeout - https://phabricator.wikimedia.org/T442468 [11:44:49] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1358844 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [11:45:35] (03CR) 10Filippo Giunchedi: "To clarify what I meant in the "test it on the instance": the easiest I found is:" [puppet] - 10https://gerrit.wikimedia.org/r/1238335 (https://phabricator.wikimedia.org/T439324) (owner: 10Physikerwelt) [11:47:16] jmm@cumin2003 drain-node (PID 1158126) is awaiting input [11:47:20] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet [11:47:21] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet [11:47:27] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet [11:47:29] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet [11:47:50] !log blake@deploy1003 urbanecm, blake: Backport for [[gerrit:1358891|GrowthExperiments: Set link recommendation service timeout to 32s (T442468)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:48:19] !log blake@deploy1003 urbanecm, blake: Continuing with deployment [11:49:25] RESOLVED: SystemdUnitFailed: netbox_ganeti_ulsfo02_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:51:02] 10SRE-tools, 06Infrastructure-Foundations, 06SRE Observability (FY2025/2026-Q1): More frequent Puppet runs on the alert hosts? - https://phabricator.wikimedia.org/T398444#12477149 (10hnowlan) 05Open→03Resolved a:03hnowlan Puppet runs on alert hosts every 5 minutes - additional concerns around reima... [11:54:00] !log blake@deploy1003 Finished scap sync-world: Backport for [[gerrit:1358891|GrowthExperiments: Set link recommendation service timeout to 32s (T442468)]] (duration: 10m 18s) [11:54:05] T442468: linkrecommendation envoy timeout is shorter than app timeout - https://phabricator.wikimedia.org/T442468 [11:54:20] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [11:54:34] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [11:54:35] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [11:54:39] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [11:54:50] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet [11:54:52] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet [11:54:58] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet [11:54:59] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet [11:57:02] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1200) [12:00:07] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:14] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:18] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:26] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:40] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet [12:00:45] (03PS1) 10Clément Goubert: rest-gateway: linkrecommendation: Increase timeout to 31s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359254 (https://phabricator.wikimedia.org/T442468) [12:01:28] (03CR) 10Blake: [C:03+1] rest-gateway: linkrecommendation: Increase timeout to 31s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359254 (https://phabricator.wikimedia.org/T442468) (owner: 10Clément Goubert) [12:02:02] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet [12:02:04] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet [12:02:10] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet [12:02:11] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet [12:02:31] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS trixie [12:03:00] (03CR) 10Clément Goubert: [C:03+2] rest-gateway: linkrecommendation: Increase timeout to 31s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359254 (https://phabricator.wikimedia.org/T442468) (owner: 10Clément Goubert) [12:04:27] (03Merged) 10jenkins-bot: rest-gateway: linkrecommendation: Increase timeout to 31s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359254 (https://phabricator.wikimedia.org/T442468) (owner: 10Clément Goubert) [12:05:05] bjensen: you can go ahead and deploy the rest-gateway [12:05:26] ack, proceeding [12:05:39] !log blake@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [12:05:59] !log cgoubert@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [12:06:12] !log blake@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [12:06:16] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [12:06:39] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [12:06:40] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [12:06:41] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs4008 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359255 [12:06:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet [12:06:50] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet [12:06:55] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [12:07:03] (03CR) 10Dbrant: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358870 (owner: 10PipelineBot) [12:07:10] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs4008 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359255 (https://phabricator.wikimedia.org/T401832) [12:07:14] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359255 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:08:08] !log cgoubert@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [12:08:29] !log cgoubert@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [12:08:32] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet [12:08:34] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet [12:08:35] 06SRE, 06Infrastructure-Foundations, 06SRE Observability: librenms-syslog leaks memory - https://phabricator.wikimedia.org/T397427#12477297 (10hnowlan) The original issue is mitigated here but altas exporter is persistently a bit daft: {F111003718} [12:08:40] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet [12:08:42] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet [12:08:58] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358870 (owner: 10PipelineBot) [12:09:30] !log cgoubert@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [12:10:15] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [12:10:31] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [12:11:40] !log cgoubert@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [12:12:23] !log cgoubert@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [12:12:33] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [12:12:58] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [12:13:06] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [12:13:11] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet [12:13:35] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [12:13:35] !log cgoubert@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [12:14:36] !log cgoubert@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [12:14:56] !log cgoubert@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. [12:15:06] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet [12:15:07] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet [12:15:14] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2006.codfw.wmnet [12:15:16] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2006.codfw.wmnet [12:15:17] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org [12:17:13] !log cgoubert@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. [12:17:59] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet [12:19:02] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3005.esams.wmnet [12:20:07] !log herron@cumin1004 END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling reboot on A:logstash-eqiad [12:20:23] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti3005.esams.wmnet [12:21:12] !log cgoubert@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [12:21:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org [12:21:36] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2006.codfw.wmnet [12:21:37] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2006.codfw.wmnet [12:21:38] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw [12:22:19] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet [12:22:26] !log cgoubert@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [12:24:02] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet [12:24:11] !log cgoubert@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [12:24:32] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet [12:24:46] !log cgoubert@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [12:25:11] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [12:25:22] FIRING: [3x] SwitchCoreInterfaceDown: Switch core interface down - ssw1-a1-eqiad:lag1 (Core: asw2-a-eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [12:25:31] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet [12:26:50] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [12:27:07] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [12:27:37] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs4008 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359255 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:27:39] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [12:28:22] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3005.esams.wmnet [12:28:28] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3005.esams.wmnet [12:29:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet [12:30:32] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica [12:30:50] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4008.ulsfo.wmnet} and A:liberica [12:31:41] (03PS1) 10Hnowlan: prometheus: don't scrape gerrit when using in pontoon [puppet] - 10https://gerrit.wikimedia.org/r/1359258 (https://phabricator.wikimedia.org/T441498) [12:35:42] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Depool lvs4008 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359255 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:38:47] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet [12:38:47] !log fabfur@cumin1004 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_esams [12:42:30] 06SRE, 10SRE-Access-Requests: Requesting access to codesearch-roots for addshore - https://phabricator.wikimedia.org/T442219#12477449 (10LSobanski) [12:43:08] 06SRE, 10SRE-Access-Requests: Requesting access to codesearch-roots for addshore - https://phabricator.wikimedia.org/T442219#12477450 (10LSobanski) Approved. [12:43:25] 06SRE, 10SRE-Access-Requests: Requesting access to codesearch-roots for addshore - https://phabricator.wikimedia.org/T442219#12477452 (10LSobanski) [12:45:37] (03PS1) 10Dpogorzelski: rest-gateway: skip JWT validation for hummingbird [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359259 (https://phabricator.wikimedia.org/T441912) [12:47:59] (03CR) 10Dpogorzelski: "This was not triggered after the deployment of the previous 2 changes that's why it flew under the radar intially" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359259 (https://phabricator.wikimedia.org/T441912) (owner: 10Dpogorzelski) [12:53:59] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet [12:54:46] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3006.esams.wmnet [12:58:46] jmm@cumin2003 drain-node (PID 1178381) is awaiting input [12:59:35] jmm@cumin2003 drain-node (PID 1178477) is awaiting input [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) UTC afternoon backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1300). [13:00:05] Thiemo_WMDE: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:01:32] (03PS1) 10Dreamy Jazz: AbuseReview: Allow promoting vandalism out of alpha test [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359261 (https://phabricator.wikimedia.org/T442379) [13:01:40] (03PS1) 10Atsuko: airflow-fr-tech: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359262 (https://phabricator.wikimedia.org/T435479) [13:01:45] (03PS1) 10Dreamy Jazz: AbuseReview: Allow promoting vandalism out of alpha test [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359263 (https://phabricator.wikimedia.org/T442379) [13:02:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 27 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359263 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:02:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 27 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359261 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:03:14] o/ [13:04:11] I can deploy if Thiemo_WMDE is around [13:04:19] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet [13:04:53] (03CR) 10Xcollazo: [C:03+1] "LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1359236 (owner: 10Meno25) [13:05:01] in the meantime I guess Dreamy_Jazz can deploy, though that’s going to take ages due to the i18n changes… maybe worth just inlining that message in the backport? (it’s not getting translations in this train branch anyway ^^) [13:05:18] Yeah, happy for others to go first [13:05:39] Also happy to wait for Thiemo_WMDE to get online for a few minutes while I'm adding some other items to the board [13:05:41] (03CR) 10Hashar: "I did this change as the first of my series (https://gerrit.wikimedia.org/r/q/bug:T441920 ) to simplify the Puppet manifest and because th" [puppet] - 10https://gerrit.wikimedia.org/r/1357838 (https://phabricator.wikimedia.org/T441920) (owner: 10Hashar) [13:05:48] (03CR) 10Atsuko: [C:03+2] airflow-fr-tech: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359262 (https://phabricator.wikimedia.org/T435479) (owner: 10Atsuko) [13:06:21] I’ll ping Thiemo [13:07:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 27 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358893 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:07:26] Thanks [13:07:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet [13:07:33] (03Merged) 10jenkins-bot: airflow-fr-tech: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359262 (https://phabricator.wikimedia.org/T435479) (owner: 10Atsuko) [13:07:41] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti3006.esams.wmnet [13:07:50] (My config patch needs to go with my wmf backports, so will do them as one) [13:07:55] ack [13:08:42] If no objection I'll start at :10 past the hour if Thiemo isn't here yet [13:09:24] PROBLEM - Host kubestagemaster1003 is DOWN: PING CRITICAL - Packet loss = 100% [13:09:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:09:30] sure [13:10:19] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:11:10] just in the nick of time :D hi Thiemo_WMDE! [13:11:13] Dreamy_Jazz: ^ fyi ^^ [13:11:24] fwiw I’m still grepping for tmpEnableMobileEditingUIBetaFeature to check if it’s actually unused [13:11:27] Hey, late for the backport but maybe early enough? [13:11:30] Literally just in time, spiderpig was asking me if I wanted to go :D [13:11:48] (I've paused and will wait for you to finish yours) [13:12:07] Thiemo_WMDE: the grep hasn’t finished yet but it looks like the code still exists in wmf.24, which is currently still deployed in group1 [13:12:15] are you sure it’s already okay to deploy this change? [13:13:19] (03CR) 10Lucas Werkmeister (WMDE): "No longer used in 1.48.0-wmf.1, but still used in 1.47.0-wmf.24, which is currently still deployed:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355485 (https://phabricator.wikimedia.org/T437054) (owner: 10Thiemo Kreuz (WMDE)) [13:13:28] left a comment with the grep output ^ as well (it finished running now) [13:13:51] Dang, I didn't expect that. Then it's better to wait for group1 to catch up. [13:13:54] I’m wondering if this should wait for another day or so because php2shell broke the train state [13:13:55] ok [13:14:03] Dreamy_Jazz: go ahead then [13:14:22] Thanks, proceeding with mine [13:14:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359263 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:14:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359261 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:14:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358893 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:14:47] s/wednesday/train/ https://blog.luitjes.it/posts/i-want-my-wednesday-back/ [13:14:52] (/hj) [13:15:38] (03Merged) 10jenkins-bot: WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358893 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:15:39] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet [13:15:41] (03Merged) 10jenkins-bot: AbuseReview: Allow promoting vandalism out of alpha test [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359263 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:15:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet [13:15:50] FIRING: [3x] ProbeDown: Service ganeti1038:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:15:52] RECOVERY - Host kubestagemaster1003 is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms [13:15:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3006.esams.wmnet [13:15:58] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3006.esams.wmnet [13:16:04] FIRING: KubernetesCalicoDown: kubestagemaster1003.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-staging&var-instance=kubestagemaster1003.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:16:07] (03Merged) 10jenkins-bot: AbuseReview: Allow promoting vandalism out of alpha test [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359261 (https://phabricator.wikimedia.org/T442379) (owner: 10Dreamy Jazz) [13:16:08] (03CR) 10Lucas Werkmeister (WMDE): "(Per discussion in #wikimedia-operations, this should wait until the wmf.1 train has reached group1.)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355485 (https://phabricator.wikimedia.org/T437054) (owner: 10Thiemo Kreuz (WMDE)) [13:16:34] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) ([[gerrit:1359263|1.47.0-wmf.24]], [[gerrit:1359261|1.48.0-wmf.1]]), [[gerrit:1358893|WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379)]] [13:16:39] T442379: AbuseReview: Switch vandalism queue on enwiki to use rollback, remove the banner from the alpha test - https://phabricator.wikimedia.org/T442379 [13:18:19] RESOLVED: KubernetesCalicoDown: kubestagemaster1003.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-staging&var-instance=kubestagemaster1003.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:18:28] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355485 (https://phabricator.wikimedia.org/T437054) (owner: 10Thiemo Kreuz (WMDE)) [13:20:50] RESOLVED: [4x] ProbeDown: Service ganeti1038:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:21:02] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet [13:21:07] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3007.esams.wmnet [13:22:13] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12477707 (10Volans) I think that in general is the responsibility of the cookbook authors decide which dependencies... [13:25:32] (03CR) 10Ladsgroup: "gentle ping on this." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357426 (owner: 10Ladsgroup-claude) [13:26:55] (03CR) 10Volans: "See my comments on the related task for my general opinion on this topic" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359228 (https://phabricator.wikimedia.org/T442374) (owner: 10CWilliams) [13:27:10] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti3007.esams.wmnet [13:27:11] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet [13:29:31] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12477766 (10Volans) Of course that can also be improved to either do a more specific check on the failure or perfor... [13:32:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet [13:32:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet [13:33:59] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet [13:35:11] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:35:20] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3007.esams.wmnet [13:35:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3007.esams.wmnet [13:36:56] (03PS2) 10Jelto: cdn: exclude miscweb from ATSBackendErrorsHigh [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) [13:37:04] jmm@cumin2003 drain-node (PID 1195075) is awaiting input [13:37:16] (03CR) 10Jelto: cdn: exclude miscweb from ATSBackendErrorsHigh (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) (owner: 10Jelto) [13:38:02] (03CR) 10AKhatun: [C:03+2] revision-feature-counts lac-lambda: Add test wikis [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358864 (https://phabricator.wikimedia.org/T442448) (owner: 10AKhatun) [13:38:25] !log dreamyjazz@deploy1003 dreamyjazz: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) ([[gerrit:1359263|1.47.0-wmf.24]], [[gerrit:1359261|1.48.0-wmf.1]]), [[gerrit:1358893|WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:38:31] T442379: AbuseReview: Switch vandalism queue on enwiki to use rollback, remove the banner from the alpha test - https://phabricator.wikimedia.org/T442379 [13:39:29] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [13:40:16] (03Merged) 10jenkins-bot: revision-feature-counts lac-lambda: Add test wikis [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358864 (https://phabricator.wikimedia.org/T442448) (owner: 10AKhatun) [13:40:33] (03CR) 10Bking: [C:03+2] matomo: Use an initContainer to copy static assets, customize httpd-fcgi config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [13:42:37] (03CR) 10Jgiannelos: Actually send the PoolCounter RELEASE command (033 comments) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357426 (owner: 10Ladsgroup-claude) [13:45:46] !log akhatun@deploy1003 helmfile [staging] START helmfile.d/services/lac-lambdas: apply [13:45:54] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet [13:47:04] !log akhatun@deploy1003 helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply [13:48:17] (03PS1) 10Brouberol: test-kitchen: drop the phabricator_api_token secret [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359276 (https://phabricator.wikimedia.org/T442521) [13:49:46] !log akhatun@deploy1003 helmfile [codfw] START helmfile.d/services/lac-lambdas: apply [13:49:59] (03PS1) 10Dreamy Jazz: AbuseReview: Enable queues on several wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359277 (https://phabricator.wikimedia.org/T442476) [13:51:07] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for AbuseReview: Allow promoting vandalism out of alpha test (T442379) ([[gerrit:1359263|1.47.0-wmf.24]], [[gerrit:1359261|1.48.0-wmf.1]]), [[gerrit:1358893|WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki (T442379)]] (duration: 34m 33s) [13:51:13] T442379: AbuseReview: Switch vandalism queue on enwiki to use rollback, remove the banner from the alpha test - https://phabricator.wikimedia.org/T442379 [13:51:14] !log akhatun@deploy1003 helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply [13:51:26] !log akhatun@deploy1003 helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply [13:52:25] !log akhatun@deploy1003 helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply [13:52:29] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [13:52:57] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [13:53:25] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [13:53:53] (03CR) 10Jgiannelos: [C:03+1] "Ideally this patch should have a unit test. Code-wise it looks OK. This is a valid memory leak issue and that fixes it locally." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357426 (owner: 10Ladsgroup-claude) [13:55:07] Since my deployment more requests have been failing with request timeout exceptions [13:55:41] It appears unrelated though [13:55:48] (03CR) 10Hashar: "Moved to https://gitlab.wikimedia.org/repos/sre/docker-pkg/-/merge_requests/4" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 (owner: 10Hashar) [13:56:21] !log failover Ganeti master in esams to ganeti3005 [13:56:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:57:01] (03CR) 10Marostegui: [C:03+1] "thanks" [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) (owner: 10Jelto) [13:57:48] (03CR) 10Hashar: "Moved to https://gitlab.wikimedia.org/repos/sre/docker-pkg/-/merge_requests/5" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1166856 (owner: 10Hashar) [13:58:58] PROBLEM - ganeti-wconfd running on ganeti3008 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [13:58:59] (03CR) 10Hashar: "Moved to https://gitlab.wikimedia.org/repos/sre/docker-pkg/-/merge_requests/6" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337562 (owner: 10Hashar) [13:59:05] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [13:59:18] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [13:59:18] (03PS5) 10Ladsgroup: Actually send the PoolCounter RELEASE command [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357426 (owner: 10Ladsgroup-claude) [14:00:04] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400) [14:00:04] slyngs: Time to snap out of that daydream and deploy Datacenter Switchover - Full switchover to CODFW. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400). [14:00:04] slyngs: That opportune time for a Deployment server switchover deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400). [14:00:09] (03CR) 10Hashar: "Moved to https://gitlab.wikimedia.org/repos/sre/docker-pkg/-/merge_requests/7" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337559 (owner: 10Hashar) [14:00:28] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [14:00:41] (03CR) 10Elukey: [C:03+1] Netbox: add report on old devices with still a planned status [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1358192 (owner: 10Ayounsi) [14:00:58] jmm@cumin2003 drain-node (PID 1195075) is awaiting input [14:01:21] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie [14:02:00] !log tchin@deploy1003 helmfile [staging] START helmfile.d/services/eventstreams: apply [14:02:22] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) [14:02:36] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) [14:03:00] (03CR) 10Ladsgroup: "I decided against unit test for it. Felt like too much but I can write some in a follow up." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357426 (owner: 10Ladsgroup-claude) [14:03:35] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) [14:03:52] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) [14:03:53] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie [14:04:59] !log tchin@deploy1003 helmfile [staging] DONE helmfile.d/services/eventstreams: apply [14:06:18] 06SRE, 06Infrastructure-Foundations, 10netops, 07SecTeam-Processed, and 2 others: Routinator: CVE-2026-72752 CVE-2026-72753 CVE-2026-72754 CVE-2026-72755 CVE-2026-72756 CVE-2026-72757 CVE-2026-72758 - https://phabricator.wikimedia.org/T442502#12477982 (10sbassett) [14:06:26] 06SRE, 06Infrastructure-Foundations, 10netops, 07SecTeam-Processed, and 2 others: Routinator: CVE-2026-72752 CVE-2026-72753 CVE-2026-72754 CVE-2026-72755 CVE-2026-72756 CVE-2026-72757 CVE-2026-72758 - https://phabricator.wikimedia.org/T442502#12477984 (10sbassett) [14:06:38] (03PS4) 10Krinkle: MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) [14:06:42] FIRING: JobUnavailable: Reduced availability for job pdu_sentry4 in ops@eqsin - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:07:16] PROBLEM - Recursive DNS on 103.102.166.36 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:08:08] (03CR) 10Kosta Harlan: [C:03+1] AbuseReview: Enable queues on several wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359277 (https://phabricator.wikimedia.org/T442476) (owner: 10Dreamy Jazz) [14:08:31] jouncebot: nowandnext [14:08:31] For the next 0 hour(s) and 21 minute(s): Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400) [14:08:31] For the next 2 hour(s) and 51 minute(s): Datacenter Switchover - Full switchover to CODFW (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400) [14:08:31] For the next 1 hour(s) and 51 minute(s): Deployment server switchover (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400) [14:08:31] In 0 hour(s) and 21 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1430) [14:08:47] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359277 (https://phabricator.wikimedia.org/T442476) (owner: 10Dreamy Jazz) [14:09:14] RECOVERY - Recursive DNS on 103.102.166.36 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [14:09:26] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie [14:09:36] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) [14:09:38] (03Merged) 10jenkins-bot: AbuseReview: Enable queues on several wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359277 (https://phabricator.wikimedia.org/T442476) (owner: 10Dreamy Jazz) [14:09:49] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) [14:10:07] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1359277|AbuseReview: Enable queues on several wikis (T442476 T442204)]] [14:10:13] T442476: Enable Special:AbuseReview vandalism queue for Hebrew Wikipedia (limited access) - https://phabricator.wikimedia.org/T442476 [14:10:13] T442204: Enable AbuseReview queues on additional wikis with limited access - https://phabricator.wikimedia.org/T442204 [14:10:42] * Krinkle sits next to Dreamy_Jazz to await his unscheduled turn [14:10:49] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) [14:10:51] !log tchin@deploy1003 helmfile [staging] START helmfile.d/services/eventstreams: apply [14:10:55] !log tchin@deploy1003 helmfile [staging] DONE helmfile.d/services/eventstreams: apply [14:10:58] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:11:03] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) [14:11:04] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie [14:11:05] :D [14:11:09] I'll ping you when done [14:13:34] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet [14:14:21] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1359277|AbuseReview: Enable queues on several wikis (T442476 T442204)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:14:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:14:50] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.powercycle for host lvs5006 [14:15:58] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:18:21] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12478065 (10CWilliams-WMF) > I think that in general is the responsibility of the cookbook authors decide which dep... [14:18:42] (03CR) 10Ladsgroup: [C:03+2] "pushing to production!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357426 (owner: 10Ladsgroup-claude) [14:19:03] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet [14:19:27] !log vgutierrez@cumin1004 END (ERROR) - Cookbook sre.hosts.powercycle (exit_code=97) for host lvs5006 [14:19:29] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet [14:19:42] (03CR) 10Hashar: "Moved to https://gitlab.wikimedia.org/repos/sre/docker-pkg/-/merge_requests/8" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1178068 (https://phabricator.wikimedia.org/T401733) (owner: 10Hashar) [14:19:44] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [14:19:53] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie [14:20:03] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) [14:20:14] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) [14:20:40] (03Merged) 10jenkins-bot: Actually send the PoolCounter RELEASE command [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357426 (owner: 10Ladsgroup-claude) [14:20:55] (03CR) 10CWilliams: "That could equally be a nested class so that it is confined to the Spicerack class though, not everything needs to be a module and that do" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359228 (https://phabricator.wikimedia.org/T442374) (owner: 10CWilliams) [14:25:22] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:27:20] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1359277|AbuseReview: Enable queues on several wikis (T442476 T442204)]] (duration: 17m 13s) [14:27:27] T442476: Enable Special:AbuseReview vandalism queue for Hebrew Wikipedia (limited access) - https://phabricator.wikimedia.org/T442476 [14:27:27] T442204: Enable AbuseReview queues on additional wikis with limited access - https://phabricator.wikimedia.org/T442204 [14:28:04] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet [14:28:25] (03PS1) 10Ottomata: Prefetch the LAC revision_feature_counts artifact on group1 wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359282 (https://phabricator.wikimedia.org/T432604) [14:30:04] slyngs: How many deployers does it take to do Datacenter Switchover - Full switchover to CODFW deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400). [14:30:05] slyngs: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Deployment server switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400). [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1430) [14:31:39] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) [14:31:57] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) [14:31:58] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie [14:33:20] PROBLEM - NTP peers and stratum check on dns5003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/NTP [14:33:24] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet [14:34:58] !log installing Django security updates [14:35:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:35:44] (03CR) 10Ayounsi: [C:03+2] Netbox: add report on old devices with still a planned status [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1358192 (owner: 10Ayounsi) [14:36:43] Dreamy_Jazz: ready? [14:36:50] Oh sorry [14:36:52] Yeah [14:36:56] I forgot to ping :( [14:36:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358370 (https://phabricator.wikimedia.org/T288223) (owner: 10Krinkle) [14:37:02] np [14:37:20] (03Merged) 10jenkins-bot: Netbox: add report on old devices with still a planned status [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1358192 (owner: 10Ayounsi) [14:37:42] (03Merged) 10jenkins-bot: xff.log: Fix jobrunner filter and log whether TrustedXFF was used [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358370 (https://phabricator.wikimedia.org/T288223) (owner: 10Krinkle) [14:38:05] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1358370|xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223)]] [14:38:10] T288223: Move Wikimedia's XFF log to MediaWiki core - https://phabricator.wikimedia.org/T288223 [14:38:21] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12478226 (10Volans) >>! In T442374#12478065, @CWilliams-WMF wrote: > The healthcheck would need to be done every ca... [14:39:33] jmm@cumin2003 drain-node (PID 1206266) is awaiting input [14:40:11] (03PS1) 10Ladsgroup: thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359285 (https://phabricator.wikimedia.org/T431768) [14:40:33] (03PS1) 10Hashar: ci: stop including firewall rules on WMCS instances [puppet] - 10https://gerrit.wikimedia.org/r/1359286 [14:40:47] (03PS1) 10Elukey: Add a systemd timer to periodically drop the old MediaWiki images [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) [14:41:05] (03CR) 10Hashar: "Indeed that class is no more of any use. I am removing it with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359286 and rebasing t" [puppet] - 10https://gerrit.wikimedia.org/r/1320916 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [14:41:28] !log ayounsi@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [14:41:49] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [14:42:17] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1358370|xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:42:48] (03PS4) 10Hashar: ci: allow inbound ssh from controller on prod agent [2] [puppet] - 10https://gerrit.wikimedia.org/r/1320916 (https://phabricator.wikimedia.org/T418521) [14:42:54] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: power-on failure for druid-internal1004.eqiad.wmnet - https://phabricator.wikimedia.org/T441866#12478269 (10VRiley-WMF) Just got this as a response from supermicro Hi Valerie, Just to let you know, the SYS-111E-WR with serial number XXXXX that you have requ... [14:42:56] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie [14:43:03] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ping1004.eqiad.wmnet [14:43:07] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) [14:43:08] (03CR) 10Ladsgroup: [C:03+2] thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359285 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup) [14:43:10] RECOVERY - NTP peers and stratum check on dns5003 is OK: NTP OK: Offset 0.007683876 secs, stratum=2 https://wikitech.wikimedia.org/wiki/NTP [14:43:15] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) [14:43:42] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet [14:44:11] !log ayounsi@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [14:44:37] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [14:45:01] (03Merged) 10jenkins-bot: thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359285 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup) [14:45:14] (03CR) 10Volans: "replies inline" [cookbooks] - 10https://gerrit.wikimedia.org/r/1358763 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [14:45:28] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.8 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [14:45:50] !log krinkle@deploy1003 krinkle: Continuing with deployment [14:46:14] (03CR) 10Ahmon Dancy: [C:03+1] Revert "kubernetes.yaml: Disable mwdebug-next httpbb checks" [puppet] - 10https://gerrit.wikimedia.org/r/1358895 (owner: 10RLazarus) [14:46:27] jouncebot: nowandnext [14:46:27] For the next 2 hour(s) and 13 minute(s): Datacenter Switchover - Full switchover to CODFW (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400) [14:46:27] For the next 1 hour(s) and 13 minute(s): Deployment server switchover (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400) [14:46:27] For the next 0 hour(s) and 13 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1430) [14:46:27] In 0 hour(s) and 13 minute(s): Datacenter Switchover - Services and Traffic (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1500) [14:46:27] In 0 hour(s) and 13 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1500) [14:46:31] (03PS1) 10Andrew Bogott: Openstack api policies: permit 'secgroupadmin' to modify security groups [puppet] - 10https://gerrit.wikimedia.org/r/1359288 (https://phabricator.wikimedia.org/T442554) [14:46:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1004.eqiad.wmnet [14:46:58] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:47:05] !log Restarting Gerrit [14:47:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:47:14] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1206 - https://phabricator.wikimedia.org/T442039#12478295 (10VRiley-WMF) Spoke briefly with Ben on Slack. He is currently off but did seem optimistic about updating the firmware on this unit with no problems. Will proceed with it to correct the issue. [14:47:15] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [14:47:26] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [14:47:28] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [14:48:46] FIRING: GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [14:49:04] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet [14:49:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet [14:49:46] FIRING: GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [14:50:27] ^ gerrit restarted [14:50:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:51:21] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [14:51:21] (03CR) 10Hashar: "recheck I was restarting Gerrit" [puppet] - 10https://gerrit.wikimedia.org/r/1359288 (https://phabricator.wikimedia.org/T442554) (owner: 10Andrew Bogott) [14:51:26] (03PS2) 10Nvdtn19: Enable NewUserMessages for viwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) [14:51:27] (03CR) 10David Caro: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1359288 (https://phabricator.wikimedia.org/T442554) (owner: 10Andrew Bogott) [14:51:29] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1358370|xff.log: Fix jobrunner filter and log whether TrustedXFF was used (T288223)]] (duration: 13m 23s) [14:51:33] T288223: Move Wikimedia's XFF log to MediaWiki core - https://phabricator.wikimedia.org/T288223 [14:51:58] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:52:16] (03PS3) 10Nvdtn19: Enable NewUserMessage for viwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) [14:52:46] FIRING: GerritHigh5xxRatio: High rate of HTTP 5xx responses on gerrit1003 - TODO - https://grafana.wikimedia.org/d/nX8li17Sk/gerrit-overview-sre-collab - https://alerts.wikimedia.org/?q=alertname%3DGerritHigh5xxRatio [14:53:30] (03CR) 10Volans: "Nice, much simpler! I'll leave it to ServiceOps for the review." [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [14:53:46] RESOLVED: [3x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [14:54:22] (03PS1) 10Snwachukwu: Deploy new edit/v4/per_editor endpoint to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359309 (https://phabricator.wikimedia.org/T434136) [14:54:46] RESOLVED: GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [14:54:49] (03PS9) 10Jelto: sre.k8s: add upgrade-kubelets-inplace cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) [14:54:49] (03CR) 10Jelto: "thanks for the review, comment answers in-line" [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [14:55:00] (03CR) 10Andrew Bogott: [C:03+2] Openstack api policies: permit 'secgroupadmin' to modify security groups [puppet] - 10https://gerrit.wikimedia.org/r/1359288 (https://phabricator.wikimedia.org/T442554) (owner: 10Andrew Bogott) [14:55:14] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [14:55:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:55:32] PROBLEM - Host an-worker1206 is DOWN: PING CRITICAL - Packet loss = 100% [14:55:50] (03CR) 10Ayounsi: [C:03+1] sre.hosts.dhcp: add retry when contacting the BMC [cookbooks] - 10https://gerrit.wikimedia.org/r/1359235 (owner: 10Elukey) [14:56:04] (03CR) 10Elukey: [C:03+2] sre.hosts.dhcp: add retry when contacting the BMC [cookbooks] - 10https://gerrit.wikimedia.org/r/1359235 (owner: 10Elukey) [14:57:23] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet [14:57:37] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:57:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:57:41] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3008.esams.wmnet [14:57:46] RESOLVED: GerritHigh5xxRatio: High rate of HTTP 5xx responses on gerrit1003 - TODO - https://grafana.wikimedia.org/d/nX8li17Sk/gerrit-overview-sre-collab - https://alerts.wikimedia.org/?q=alertname%3DGerritHigh5xxRatio [14:57:56] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1206 - https://phabricator.wikimedia.org/T442039#12478453 (10VRiley-WMF) 05Open→03Resolved Installed the new version of BIOS and it seemingly has corrected the issue. Will be closing this ticket. [14:59:11] (03CR) 10Brouberol: [C:03+2] etcd: add restart/reboot cookbook (033 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1358763 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [14:59:34] (03CR) 10Filippo Giunchedi: prometheus: don't scrape gerrit when using in pontoon (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359258 (https://phabricator.wikimedia.org/T441498) (owner: 10Hnowlan) [15:00:04] PROBLEM - NTP peers and stratum check on dns5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/NTP [15:00:04] slyngs: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Datacenter Switchover - Full switchover to CODFW. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400). [15:00:04] slyngs: #bothumor My software never has bugs. It just develops random features. Rise for Deployment server switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400). [15:00:05] slyngs: May I have your attention please! Datacenter Switchover - Services and Traffic. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1500) [15:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor I � Unicode. All rise for SRE Collaboration Services office hours deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1500). [15:00:12] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12478486 (10elukey) @cmooney after re-reading T438161#12350717 I had a chat with John, and IIUC the 10g ports are available and "bootable" but they are copper based, not SFP.... [15:00:29] jmm@cumin2003 drain-node (PID 1211486) is awaiting input [15:02:11] (03PS1) 10CWilliams: backup: adding initial offsite manifest [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) [15:03:04] (03PS2) 10CWilliams: backup: adding initial offsite manifest [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) [15:03:16] (03CR) 10CWilliams: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [15:03:27] We will be starting the depooling of eqiad as part of the DC switch over. This will take a little longer than normal due to an additional step. [15:03:35] See: https://phabricator.wikimedia.org/T439675 [15:03:45] break a leg! [15:04:40] jmm@cumin2003 drain-node (PID 1211497) is awaiting input [15:04:54] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [15:05:37] \o/ [15:05:39] (03CR) 10Dzahn: "did you also want to delete the included file (later)?" [puppet] - 10https://gerrit.wikimedia.org/r/1359286 (owner: 10Hashar) [15:06:05] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [15:06:56] (03CR) 10Bking: [C:03+1] wdqs: exclude public main from DC switchover [puppet] - 10https://gerrit.wikimedia.org/r/1344974 (https://phabricator.wikimedia.org/T435443) (owner: 10Ryan Kemper) [15:07:09] (03CR) 10Muehlenhoff: backup: adding initial offsite manifest (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [15:07:47] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet [15:08:35] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) [15:08:47] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) [15:08:48] !log vgutierrez@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie [15:08:57] !log slyngshede@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] [15:09:04] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] [15:11:24] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:11:28] (03CR) 10Ryan Kemper: [C:03+2] wdqs: exclude public main from DC switchover [puppet] - 10https://gerrit.wikimedia.org/r/1344974 (https://phabricator.wikimedia.org/T435443) (owner: 10Ryan Kemper) [15:11:33] !log slyngshede@cumin1004 START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T439675 - DC Switchover [15:11:39] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [15:12:38] (03PS1) 10Ayounsi: Netbox: add alerting for coherence.DevicesCoherence report [puppet] - 10https://gerrit.wikimedia.org/r/1359315 [15:12:54] (03PS1) 10Clément Goubert: Revert "trafficserver: Remove mwdebug-next XWD" [puppet] - 10https://gerrit.wikimedia.org/r/1359316 [15:13:10] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie [15:13:21] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) [15:13:29] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) [15:13:37] (03CR) 10Ayounsi: "New check is working, this change is to run it periodically and alert about failures." [puppet] - 10https://gerrit.wikimedia.org/r/1359315 (owner: 10Ayounsi) [15:13:43] (03CR) 10Tiziano Fogli: "LGTM, though I think unit tests for the rules would be welcome." [alerts] - 10https://gerrit.wikimedia.org/r/1341852 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [15:14:30] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: Device does not support ifTable - try without -I option https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [15:14:58] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:15:28] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [15:15:45] (03CR) 10Tiziano Fogli: [C:03+1] logstash: add service ordering to unit [puppet] - 10https://gerrit.wikimedia.org/r/1341921 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [15:16:37] !log slyngshede@cumin1004 END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-api-ext-ro in eqiad: T439675 - DC Switchover [15:16:45] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [15:17:01] (03CR) 10Dzahn: [C:03+1] "can it be merged now?" [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [15:19:54] RECOVERY - NTP peers and stratum check on dns5004 is OK: NTP OK: Offset 0.000270924 secs, stratum=2 https://wikitech.wikimedia.org/wiki/NTP [15:19:58] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:20:50] 10ops-codfw, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568 (10phaultfinder) 03NEW [15:21:06] PROBLEM - Auth DNS on dns5004 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [15:21:10] !log cgoubert@cumin2003 START - Cookbook sre.discovery.service-route check mw-api-ext-ro: maintenance [15:21:10] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-ext-ro: maintenance [15:22:04] RECOVERY - Auth DNS on dns5004 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [15:22:16] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12478688 (10CWilliams-WMF) Ah, I see, makes sense +1 [15:30:06] PROBLEM - Recursive DNS on 103.102.166.10 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [15:30:44] ^dns5003 [15:31:04] RECOVERY - Recursive DNS on 103.102.166.10 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [15:31:22] (03CR) 10BCornwall: "Sadly, I don't think it can. We rely on those healthcheck endpoints being green. There's the `haproxy_alive` underneath but (for reasons t" [puppet] - 10https://gerrit.wikimedia.org/r/1344231 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [15:32:36] !log slyngshede@cumin1004 START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: T439675 - DC Switchover [15:32:43] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [15:33:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [15:33:19] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) [15:33:30] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) [15:33:31] !log vgutierrez@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie [15:33:58] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:37:41] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie [15:37:43] !log slyngshede@cumin1004 END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: T439675 - DC Switchover [15:37:53] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) [15:37:55] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [15:38:01] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [15:38:01] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) [15:38:58] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:40:53] 10ops-codfw, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574 (10phaultfinder) 03NEW [15:42:10] (03CR) 10CWilliams: backup: adding initial offsite manifest (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [15:42:43] (03CR) 10Hashar: "Ia2a1a19645096db594fa9ea432f191cc02af087f is the change removing `profile::ci::firewall::jenkinsagent`. And you mentioned at" [puppet] - 10https://gerrit.wikimedia.org/r/1359286 (owner: 10Hashar) [15:46:23] !log slyngshede@cumin1004 START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T439675 [15:46:31] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [15:47:32] (03CR) 10Jelto: [C:03+1] "lgtm and yes that should fix the race condition which I mentioned in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1358693/comments" [puppet] - 10https://gerrit.wikimedia.org/r/1359226 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [15:49:30] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3074.esams.wmnet [15:54:00] (03CR) 10JavierMonton: "oh, sorry, I missed that comment! Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1359226 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [15:55:46] 10ops-codfw, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577 (10phaultfinder) 03NEW [16:00:05] slyngs: gettimeofday() says it's time for Datacenter Switchover - Full switchover to CODFW. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1400) [16:00:05] slyngs: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Datacenter Switchover - Services and Traffic deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1500). [16:00:05] jhathaway and rzl: Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1600). Please do the needful. [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:01:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:05:29] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: dc=eqsin,cluster=dnsbox [16:06:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:06:30] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.3 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:08:26] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:08:45] (03CR) 10Herron: "rebase conflicts will need to be sorted out first" [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [16:10:44] !log vgutierrez@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5006.eqsin.wmnet with OS trixie [16:11:42] FIRING: [3x] JobUnavailable: Reduced availability for job pdu_sentry4 in ops@eqsin - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:22] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [16:12:41] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.reimage for host lvs5006.eqsin.wmnet with OS trixie [16:12:52] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (push config only) [16:12:57] (03PS1) 10Ayounsi: eqsin: drain HE transport to eqiad [homer/public] - 10https://gerrit.wikimedia.org/r/1359323 [16:13:01] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (push config only) [16:13:19] (03CR) 10Cathal Mooney: [C:03+1] "let's merge it in a few weeks time though :)" [homer/public] - 10https://gerrit.wikimedia.org/r/1358545 (owner: 10Ayounsi) [16:13:30] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.3 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:14:01] (03CR) 10Cathal Mooney: [C:03+1] eqsin: drain HE transport to eqiad [homer/public] - 10https://gerrit.wikimedia.org/r/1359323 (owner: 10Ayounsi) [16:15:04] (03CR) 10Ayounsi: [C:03+2] eqsin: drain HE transport to eqiad [homer/public] - 10https://gerrit.wikimedia.org/r/1359323 (owner: 10Ayounsi) [16:16:21] (03Merged) 10jenkins-bot: eqsin: drain HE transport to eqiad [homer/public] - 10https://gerrit.wikimedia.org/r/1359323 (owner: 10Ayounsi) [16:16:28] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.7 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:16:42] FIRING: [3x] JobUnavailable: Reduced availability for job pdu_sentry4 in ops@eqsin - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:22] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:19:04] (03CR) 10Cathal Mooney: [C:03+2] anycast_prefixes: use ns2 /48 range instead of /128 [homer/public] - 10https://gerrit.wikimedia.org/r/1358545 (owner: 10Ayounsi) [16:19:36] slyngshede@cumin1004 datacenter (PID 3413620) is awaiting input [16:20:22] (03Merged) 10jenkins-bot: anycast_prefixes: use ns2 /48 range instead of /128 [homer/public] - 10https://gerrit.wikimedia.org/r/1358545 (owner: 10Ayounsi) [16:20:42] (03PS2) 10Hnowlan: prometheus: don't scrape gerrit when using in pontoon [puppet] - 10https://gerrit.wikimedia.org/r/1359258 (https://phabricator.wikimedia.org/T441498) [16:20:48] 10ops-codfw, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12479155 (10phaultfinder) [16:21:42] RESOLVED: [3x] JobUnavailable: Reduced availability for job pdu_sentry4 in ops@eqsin - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:55] (03PS4) 10CWilliams: backup: adding initial offsite manifest [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) [16:22:31] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [16:25:06] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - T439675 [16:25:14] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [16:26:24] RESOLVED: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:26:54] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: dc=eqsin,cluster=dnsbox [16:27:06] !log sukhe@dns1004 START - running authdns-update [16:27:25] 10SRE-Access-Requests: Add yubikey backed SSH keys for joal - https://phabricator.wikimedia.org/T442587 (10JAllemandou) 03NEW [16:28:05] (03CR) 10Hnowlan: prometheus: don't scrape gerrit when using in pontoon (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359258 (https://phabricator.wikimedia.org/T441498) (owner: 10Hnowlan) [16:29:05] !log slyngshede@cumin1004 START - Cookbook sre.discovery.service-route pool eventstreams in codfw: T439675 - DC Switchover [16:29:32] !log sukhe@dns1004 END - running authdns-update [16:30:09] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM! nice" [puppet] - 10https://gerrit.wikimedia.org/r/1359258 (https://phabricator.wikimedia.org/T441498) (owner: 10Hnowlan) [16:30:11] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:30:49] !log fabfur@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet [16:30:49] !log fabfur@cumin1004 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_esams [16:34:10] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool eventstreams in codfw: T439675 - DC Switchover [16:34:16] (03CR) 10Dzahn: "Of course! Sorry, I was distracted when I said that. All makes sense to me." [puppet] - 10https://gerrit.wikimedia.org/r/1359286 (owner: 10Hashar) [16:34:17] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [16:34:31] (03CR) 10Dzahn: [C:03+2] ci: stop including firewall rules on WMCS instances [puppet] - 10https://gerrit.wikimedia.org/r/1359286 (owner: 10Hashar) [16:35:11] RESOLVED: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:36:20] (03CR) 10Dzahn: [C:03+2] ci: allow inbound ssh from controller on prod agent [2] [puppet] - 10https://gerrit.wikimedia.org/r/1320916 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [16:36:26] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1015 - https://phabricator.wikimedia.org/T442385#12479425 (10VRiley-WMF) →14Duplicate dup:03T442340 [16:36:29] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12479427 (10VRiley-WMF) [16:36:31] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ping2004.codfw.wmnet [16:37:11] !log slyngshede@cumin1004 START - Cookbook sre.discovery.service-route depool eventstreams in eqiad: T439675 - DC Switchover [16:37:37] (03CR) 10Dzahn: [C:03+2] "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1320916 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [16:39:44] FIRING: [4x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [16:40:21] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2004.codfw.wmnet [16:40:33] (03CR) 10Phuedx: [C:03+1] kafka-jumbo: declare Experiment Platform topics [puppet] - 10https://gerrit.wikimedia.org/r/1355381 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [16:40:56] (03CR) 10Phuedx: [C:03+1] "Thanks. That makes sense." [puppet] - 10https://gerrit.wikimedia.org/r/1355381 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [16:41:16] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [16:42:15] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool eventstreams in eqiad: T439675 - DC Switchover [16:42:22] T439675: Day 1 - Source datacenter depooling (eqiad - Tue. Oct. 27nd - 2026) - https://phabricator.wikimedia.org/T439675 [16:42:32] (03PS1) 10Dzahn: admin: addtionally add addshore to codesearch-roots [puppet] - 10https://gerrit.wikimedia.org/r/1359327 (https://phabricator.wikimedia.org/T442219) [16:43:11] (03PS2) 10Dzahn: admin: additionally add addshore to codesearch-roots [puppet] - 10https://gerrit.wikimedia.org/r/1359327 (https://phabricator.wikimedia.org/T442219) [16:43:50] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5006.eqsin.wmnet (remove config only) [16:44:02] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5006.eqsin.wmnet (remove config only) [16:44:33] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti3008.esams.wmnet [16:44:51] eqiad depooling is done. We'll pick up next step of the switchover tomorrow. Thank you everyone :-) [16:44:56] (03CR) 10Dzahn: [C:03+2] admin: additionally add addshore to codesearch-roots [puppet] - 10https://gerrit.wikimedia.org/r/1359327 (https://phabricator.wikimedia.org/T442219) (owner: 10Dzahn) [16:46:35] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [16:46:52] (03CR) 10RLazarus: [C:03+2] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1358844 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [16:47:10] slyngs: \o/ [16:47:14] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet [16:47:50] 10SRE-swift-storage, 10Cloud-VPS (Debian Bullseye Deprecation): Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12479549 (10Ladsgroup) It looks like it should be closed. I leave it to you. [16:48:18] (03CR) 10Filippo Giunchedi: "I'm clearly missing context here on what the goal/relationship really is between alerts/haproxy/cookbooks, at any rate thank you for the e" [puppet] - 10https://gerrit.wikimedia.org/r/1344231 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [16:48:43] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12479581 (10ayounsi) I asked Claude to have a look at this task, as the issue was alerting on AlertManager and the problem well defined. Its investiga... [16:48:52] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5006.eqsin.wmnet with reason: host reimage [16:49:44] FIRING: [4x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [16:49:47] PROBLEM - Host ml-etcd1003 is DOWN: PING CRITICAL - Packet loss = 100% [16:51:17] RECOVERY - Host ml-etcd1003 is UP: PING OK - Packet loss = 0%, RTA = 0.68 ms [16:52:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet [16:52:46] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet [16:53:00] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3008.esams.wmnet [16:53:06] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3008.esams.wmnet [16:53:39] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5006.eqsin.wmnet with reason: host reimage [16:54:16] (03CR) 10Gmodena: [C:03+1] kafka-jumbo: declare Wikidata Platform topics (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1355380 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [16:54:46] 06SRE, 06serviceops-deprecated, 10Thumbor: Thumbor units failing / service general slowness - https://phabricator.wikimedia.org/T312722#12479662 (10Ladsgroup) 05Open→03Resolved We don't use thumbor on baremetal, on top of that, we don't install firejail on thumbor images (understandably). So this bug... [16:54:58] (03PS1) 10Dzahn: codesearch: only search the git clones in delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1359330 (https://phabricator.wikimedia.org/T441991) [16:55:43] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for addshore - https://phabricator.wikimedia.org/T442219#12479677 (10Dzahn) 05Open→03Resolved a:03Dzahn ` [codesearch1001:~] $ id addshore uid=2178(addshore) gid=500(wikidev) groups=500(wikidev),770(codesearch-r... [16:57:53] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12479708 (10cmooney) Are we sure the issue is DAD? If it is I don't see an issue with the sysctl changes, as you say we statically configure the addr... [16:58:11] (03CR) 10RLazarus: [C:03+2] Revert "kubernetes.yaml: Disable mwdebug-next httpbb checks" [puppet] - 10https://gerrit.wikimedia.org/r/1358895 (owner: 10RLazarus) [16:58:38] (03PS2) 10Dzahn: codesearch: only search the git clones in delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1359330 (https://phabricator.wikimedia.org/T441991) [16:59:30] 10SRE-swift-storage, 10Thumbor: Thumbnail generation fails (HTTP 503) for some sizes of File:Montreal_Skyline_from_Mont_Royal_raw.png - https://phabricator.wikimedia.org/T253368#12479763 (10Ladsgroup) https://thumb.wikimedia.org/wikipedia/commons/thumb/3/37/Montreal_Skyline_from_Mont_Royal_raw.png/3840px-Montr... [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1700) [17:02:25] (03PS1) 10GergesShamon: [arwiki] Remove uploader user group and merge it with autoreview [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359334 (https://phabricator.wikimedia.org/T442578) [17:03:46] (03PS1) 10Jgiannelos: Make exiv calls thread safe [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359335 (https://phabricator.wikimedia.org/T442547) [17:04:10] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 27 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359334 (https://phabricator.wikimedia.org/T442578) (owner: 10GergesShamon) [17:05:19] (03CR) 10Jgiannelos: "This came up while testing async loading of the thumbor engine. There are some race conditions where exif metadata is empty because of con" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359335 (https://phabricator.wikimedia.org/T442547) (owner: 10Jgiannelos) [17:05:28] 06SRE, 10observability, 10Observability-Alerting, 13Patch-For-Review: MD RAID: remove mdadm daily check - https://phabricator.wikimedia.org/T169564#12479859 (10hnowlan) 05Open→03Resolved a:03hnowlan This was moved to a timer in 2022 (20cc904f7f016334c4cfe3ecf7e6780952fc811c). [17:05:39] (03CR) 10Dzahn: [C:03+2] codesearch: only search the git clones in delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1359330 (https://phabricator.wikimedia.org/T441991) (owner: 10Dzahn) [17:05:44] 06SRE, 10observability, 07LDAP: prometheus-openldap-exporter: Request.write called on a request after Request.finish was called - https://phabricator.wikimedia.org/T217758#12479866 (10hnowlan) 05Open→03Declined Exporter no longer in puppet. [17:05:52] 06SRE, 10Cloud-VPS, 06tools-infrastructure-team, 07LDAP: investigate slapd memory leak - https://phabricator.wikimedia.org/T130593#12479870 (10hnowlan) [17:07:03] 06SRE, 10MediaWiki-Core-JobQueue, 10observability: Redis monitoring needs to be improved - https://phabricator.wikimedia.org/T133179#12479889 (10hnowlan) 05Open→03Resolved a:03hnowlan [17:08:07] 06SRE, 10[Deprecated] service-template-node, 10observability, 10RESTBase: [Discussion] Consider validating JSON schemas when running x-ample tests? - https://phabricator.wikimedia.org/T110240#12479909 (10hnowlan) 05Open→03Declined x-ample no longer in use. [17:08:46] 06SRE, 10observability: thanos-store OOMing on titan eqiad - https://phabricator.wikimedia.org/T411343#12479923 (10hnowlan) 05Open→03Resolved a:03hnowlan [17:09:22] !log confctl: depooling releases.wikimedia.org in codfw - it is wrongly marked active/active but in reality it is not. only releases1003 is behind the releases.discovery.wmnet name - adjusting config to reality. [17:09:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:11:23] !log dzahn@cumin1004 conftool action : set/pooled=false; selector: dnsdisc=releases,name=codfw [17:12:07] (03CR) 10Dzahn: [C:03+1] "17:09 < mutante> !log confctl: depooling releases.wikimedia.org in codfw - it is wrongly marked active/active but in reality it is not. on" [puppet] - 10https://gerrit.wikimedia.org/r/1353476 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [17:17:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:17:21] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5006.eqsin.wmnet with OS trixie [17:17:44] (03CR) 10Dzahn: [C:03+2] service: mark releases as active/passive [puppet] - 10https://gerrit.wikimedia.org/r/1353476 (https://phabricator.wikimedia.org/T440068) (owner: 10Dzahn) [17:19:36] 10ops-codfw, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614 (10phaultfinder) 03NEW [17:19:44] (03PS1) 10Jforrester: wikifunctions: Expand check-wf-services.py to allow specifying a cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359336 [17:21:49] (03PS1) 10Jforrester: deployment_server::deploy_service::service_catalog: Add Wikifunctions [puppet] - 10https://gerrit.wikimedia.org/r/1359337 [17:21:49] (03PS1) 10Jforrester: deployment_server::deploy_service::service_catalog: Run Wikifunctions checks [puppet] - 10https://gerrit.wikimedia.org/r/1359338 [17:22:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:22:38] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12480117 (10ayounsi) Not sure, but I think that's the workaround worth testing, and if it solves that issue, worth rolling out infra wide to shave som... [17:23:04] (03CR) 10CI reject: [V:04-1] deployment_server::deploy_service::service_catalog: Run Wikifunctions checks [puppet] - 10https://gerrit.wikimedia.org/r/1359338 (owner: 10Jforrester) [17:24:28] (03PS4) 10Dzahn: puppet-diffs: give cloudinfra-cloudvps-puppetserver its own facts realm [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) [17:28:30] (03CR) 10AKhatun: [C:03+1] Prefetch the LAC revision_feature_counts artifact on group1 wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359282 (https://phabricator.wikimedia.org/T432604) (owner: 10Ottomata) [17:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12480158 (10phaultfinder) [17:30:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by akhatun@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359282 (https://phabricator.wikimedia.org/T432604) (owner: 10Ottomata) [17:31:14] (03Merged) 10jenkins-bot: Prefetch the LAC revision_feature_counts artifact on group1 wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359282 (https://phabricator.wikimedia.org/T432604) (owner: 10Ottomata) [17:31:25] !log codesearch: fixed the zombie process leak in hound - containers now run with --init (1358873) which reaps orphaned git processes; codesearch1001/2001 went from 160k+ zombies to 0 Now applying to codesearch10 (Cloud VPS, serves the current public codesearch), search incomplete for ~1h while it reindexes T268199 [17:31:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:31:32] T268199: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199 [17:31:42] !log akhatun@deploy1003 Started scap sync-world: Backport for [[gerrit:1359282|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695)]] [17:31:51] T432604: IR2.1.2d: Feature count data product [Attribution API] - https://phabricator.wikimedia.org/T432604 [17:31:53] T436695: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695 [17:36:14] !log akhatun@deploy1003 akhatun, otto: Backport for [[gerrit:1359282|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:37:22] !log akhatun@deploy1003 akhatun, otto: Rolling back deployment [17:38:33] (03PS1) 10Aqu: airflow: fold raw PVC releases into chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359340 (https://phabricator.wikimedia.org/T433696) [17:39:39] 10ops-eqiad, 06SRE, 06DC-Ops, 06Wikidata Platform Team, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8]) - https://phabricator.wikimedia.org/T423314#12480242 (10bking) a:05Jclark-ctr→03bking [17:39:40] !log akhatun@deploy1003 Failed scap sync-world: Backport for [[gerrit:1359282|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias (T432604 T436695)]] (duration: 07m 58s) [17:39:49] T432604: IR2.1.2d: Feature count data product [Attribution API] - https://phabricator.wikimedia.org/T432604 [17:39:50] T436695: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695 [17:40:00] 10ops-eqiad, 06SRE, 06DC-Ops, 06Wikidata Platform Team, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8]) - https://phabricator.wikimedia.org/T423314#12480265 (10bking) [17:41:25] (03PS2) 10Kamila Součková: dsh+common: remove references to deploy2002 [puppet] - 10https://gerrit.wikimedia.org/r/1314073 (https://phabricator.wikimedia.org/T426222) [17:43:17] (03CR) 10Kamila Součková: "Going to decom it this week, so should be fine now. (Thanks ghost Scott :D)" [puppet] - 10https://gerrit.wikimedia.org/r/1314073 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [17:44:16] !log rzl@apt1002:~$ sudo -i reprepro -C component/envoy-future include bookworm-wikimedia /home/rzl/envoyproxy_1.40.1-1_amd64.changes [17:44:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:45:12] (03CR) 10RLazarus: [C:03+1] dsh+common: remove references to deploy2002 [puppet] - 10https://gerrit.wikimedia.org/r/1314073 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [17:45:52] (03CR) 10Eric Gardner: [C:03+1] "LGTM for backport today" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357787 (owner: 10DErenrich) [17:46:03] (03PS1) 10AKhatun: Revert "Prefetch the LAC revision_feature_counts artifact on group1 wikipedias" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359344 [17:47:30] Hi, just FYI, akhatun and I were deploying a mw config change: gerrit:1359282, but then realized that (due to train pause last week), the code we needed wasn't yet on group1 wikis. So, we hit roll back after the mwdebug server step in spider pig. [17:47:31] It looks like the rollback was successful, but the spiderpig UI is stuck waiting for confirmation about creating a revertpatch. There is no way to confirm? [17:47:31] akhatun is going to submit a manual revert and we will merge that and leave it be. I'm going to hit STOP in spiderpig UI to see if we can unblock spiderpig deployments. [17:47:51] (03CR) 10AKhatun: [C:03+2] Revert "Prefetch the LAC revision_feature_counts artifact on group1 wikipedias" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359344 (owner: 10AKhatun) [17:48:46] (03Merged) 10jenkins-bot: Revert "Prefetch the LAC revision_feature_counts artifact on group1 wikipedias" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359344 (owner: 10AKhatun) [17:49:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by otto@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356976 (https://phabricator.wikimedia.org/T441426) (owner: 10Snwachukwu) [17:51:36] !log otto@deploy1003 Started scap sync-world: Backport for [[gerrit:1356976|EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426)]] [17:51:44] T441426: DE1.3.2: Implement "Thanks" - https://phabricator.wikimedia.org/T441426 [17:53:34] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_eqiad and A:cp - 9.2.16 upgrade (T441945) [17:53:36] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_eqiad and A:cp - 9.2.16 upgrade (T441945) [17:56:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:56:05] !log otto@deploy1003 otto, ebysans: Backport for [[gerrit:1356976|EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:57:35] !log otto@deploy1003 otto, ebysans: Continuing with deployment [18:00:05] dduvall and dancy: That opportune time for a MediaWiki train - Utc-7 Version deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T1800). [18:00:09] !log brett@puppetserver1001 conftool action : set/pooled=no; selector: name=dns4003.* [18:01:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [18:01:42] Hi window! FYI, sandraebele and I are almost done syncing a mw config change. its 51% k8s progress now [18:02:37] (03CR) 10Kamila Součková: [C:03+2] dsh+common: remove references to deploy2002 [puppet] - 10https://gerrit.wikimedia.org/r/1314073 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [18:03:18] !log otto@deploy1003 Finished scap sync-world: Backport for [[gerrit:1356976|EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (T441426)]] (duration: 11m 41s) [18:03:25] T441426: DE1.3.2: Implement "Thanks" - https://phabricator.wikimedia.org/T441426 [18:03:42] done ty! [18:03:48] ottomata: ty! [18:04:39] !log otto@deploy1003 helmfile [staging] START helmfile.d/services/eventgate-main: sync [18:04:43] dancy: o/ (can't help but hear "welcome to your doom" in 16bit audio this week) [18:04:50] !log roll restarting eventgate-main to pick up stream config change [18:04:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:05:01] !log otto@deploy1003 helmfile [staging] DONE helmfile.d/services/eventgate-main: sync [18:05:30] (03PS1) 10RLazarus: envoy-future: Update to 1.40.1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359349 [18:06:13] (03CR) 10Dzahn: "done! ok now?" [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) (owner: 10Dzahn) [18:06:55] (03CR) 10RLazarus: [V:03+2] "`" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359349 (owner: 10RLazarus) [18:07:22] hmm, looks like the spiderpig train interface is indeed confused by the partial wmf.1 train and wmf.2 rollout this week [18:07:37] (03CR) 10Kamila Součková: [C:03+1] envoy-future: Update to 1.40.1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359349 (owner: 10RLazarus) [18:07:45] !log otto@deploy1003 helmfile [eqiad] START helmfile.d/services/eventgate-main: sync [18:08:06] !log otto@deploy1003 helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync [18:08:21] (03CR) 10RLazarus: [V:03+2 C:03+2] envoy-future: Update to 1.40.1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359349 (owner: 10RLazarus) [18:08:21] !log otto@deploy1003 helmfile [codfw] START helmfile.d/services/eventgate-main: sync [18:08:25] PROBLEM - Ensure traffic_server is running for instance backend on cp1114 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/traffic_server -M --httpport 3128 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [18:08:34] oh right, we're finishing wmf.1 this week (double checking with thcipriani) [18:08:45] !log otto@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync [18:08:47] * thcipriani summoned [18:09:25] RECOVERY - Ensure traffic_server is running for instance backend on cp1114 is OK: PROCS OK: 1 process with args /usr/bin/traffic_server -M --httpport 3128 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [18:09:32] dduvall: yeah, finishing out wmf.1 (should be on group0 already), so rolling that out to group1 tomorrow, everywhere next day [18:09:44] thcipriani: spiderpig is a little sad that the task is declined but otherwise it should function to finish up wmf.1 this week [18:09:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12480516 (10phaultfinder) [18:09:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12480517 (10phaultfinder) [18:10:21] hrm, I wonder if changing the dates on the train task would help it [18:11:14] (I've been wary to touch it since train tasks are technically a phabricator plugin) [18:11:27] i think it's ok. the warning banners are doing there job by having me double check the special plan :D [18:12:13] heh, "See secret plans, if applicable" [18:12:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/PageImages] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358456 (https://phabricator.wikimedia.org/T441253) (owner: 10Krinkle) [18:13:23] (03CR) 10Kamila Součková: [C:03+1] "looks good to my untrained eyes" [puppet] - 10https://gerrit.wikimedia.org/r/1358726 (https://phabricator.wikimedia.org/T428909) (owner: 10Clément Goubert) [18:13:51] !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns4003.* [18:16:18] dduvall: Nod. We'll see how things go tomorrow. [18:17:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [18:22:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [18:22:17] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_eqiad and A:cp - 9.2.16 upgrade (T441945) [18:29:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12480589 (10phaultfinder) [18:29:44] FIRING: [4x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [18:32:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://wikifeeds.svc.eqiad.wmnet:4101 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [18:36:04] (03CR) 10Ladsgroup: [C:03+2] Make exiv calls thread safe [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359335 (https://phabricator.wikimedia.org/T442547) (owner: 10Jgiannelos) [18:37:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://wikifeeds.svc.eqiad.wmnet:4101 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [18:39:26] (03Merged) 10jenkins-bot: Make exiv calls thread safe [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359335 (https://phabricator.wikimedia.org/T442547) (owner: 10Jgiannelos) [18:46:43] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_eqiad and A:cp - 9.2.16 upgrade (T441945) [18:49:50] !log Deploying Refinery at 4a515e4 for change 1355145/T434136 Add per wiki filtering to edits/v3/per_editor [18:49:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:56:14] !log ebysans@deploy1003 Started deploy [analytics/refinery@4a515e4] (hadoop-test): Deploying Refinery Patch 1355145 and Ticket T434136 [18:56:22] T434136: AQS - Add per wiki filtering to edits/v3/per_editor - https://phabricator.wikimedia.org/T434136 [18:56:58] !log ebysans@deploy1003 Finished deploy [analytics/refinery@4a515e4] (hadoop-test): Deploying Refinery Patch 1355145 and Ticket T434136 (duration: 00m 43s) [18:57:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [18:59:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12480715 (10phaultfinder) [18:59:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:00:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:00:59] (03CR) 10Majavah: [C:03+1] puppet-diffs: give cloudinfra-cloudvps-puppetserver its own facts realm [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) (owner: 10Dzahn) [19:01:23] (03CR) 10RLazarus: [V:03+2 C:03+2] "Tested locally:" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1358779 (owner: 10Jforrester) [19:03:01] !log ebysans@deploy1003 Started deploy [analytics/refinery@4a515e4]: Deploying Refinery Patch 1355145 [analytics/refinery@4a515e4f] [19:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12480737 (10phaultfinder) [19:04:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:06:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:08:05] !log ebysans@deploy1003 Finished deploy [analytics/refinery@4a515e4]: Deploying Refinery Patch 1355145 [analytics/refinery@4a515e4f] (duration: 05m 03s) [19:11:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:15:52] (03CR) 10RLazarus: [V:03+2 C:03+2] "And built and published in production (ahead of the weekly run):" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1358779 (owner: 10Jforrester) [19:17:16] !log Deployed Refinery using Scap, and then deployed onto hdfs [19:17:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:21:05] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12480808 (10phaultfinder) [19:31:38] (03CR) 10Jforrester: "Thanks!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1358779 (owner: 10Jforrester) [19:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:35:12] !log sukhe@cumin1004 START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough [19:41:09] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12480937 (10phaultfinder) [19:46:39] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_codfw and A:cp - 9.2.16 upgrade (T441945) [19:46:44] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_codfw and A:cp - 9.2.16 upgrade (T441945) [19:48:16] !log sukhe@cumin1004 END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough [19:53:17] (03PS1) 10Ottomata: [eventgate] Bump to v1.34.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359381 (https://phabricator.wikimedia.org/T442177) [19:54:51] (03PS4) 10Jdlrobson: Merge master into wmf.1 [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358888 (https://phabricator.wikimedia.org/T438390) [20:00:04] RoanKattouw, urbanecm, TheresNoTime, and cjming: #bothumor I � Unicode. All rise for UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T2000). [20:00:04] _Gerges: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:03:43] <_Gerges> hre [20:03:55] <_Gerges> here [20:06:16] 10ops-codfw, 06DC-Ops, 10decommission-hardware: decommission/return to spares sretest2005.codfw.wmnet - https://phabricator.wikimedia.org/T442638 (10RobH) 03NEW [20:07:05] !log robh@cumin1004 START - Cookbook sre.hosts.decommission for hosts sretest2005.codfw.wmnet [20:08:00] o/ [20:08:02] _Gerges: hello [20:08:10] RhinosF1 told me you got a patch :) [20:09:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12481083 (10phaultfinder) [20:09:59] (03CR) 10TrainBranchBot: [C:03+2] "Approved by hashar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359334 (https://phabricator.wikimedia.org/T442578) (owner: 10GergesShamon) [20:10:33] fun, the Wikimedia Debug extension does not work anymore for me under Firefox [20:10:50] (03Merged) 10jenkins-bot: [arwiki] Remove uploader user group and merge it with autoreview [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359334 (https://phabricator.wikimedia.org/T442578) (owner: 10GergesShamon) [20:11:01] !log robh@cumin1004 START - Cookbook sre.dns.netbox [20:11:13] !log hashar@deploy1003 Started scap sync-world: Backport for [[gerrit:1359334|[arwiki] Remove uploader user group and merge it with autoreview (T442578)]] [20:11:21] T442578: Merge "uploader" group into "autoreview" group in arwiki - https://phabricator.wikimedia.org/T442578 [20:12:24] <_Gerges> Is there a need to run a script that removes all users in the 'uploader' group? [20:13:13] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_codfw and A:cp - 9.2.16 upgrade (T441945) [20:13:48] _Gerges: do you need to copy members or just empty the old group [20:14:45] !log rzl@apt1002:~$ sudo -i reprepro copy bookworm-wikimedia bullseye-wikimedia kubeconform [20:14:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:15:15] <_Gerges> just empty the old group [20:15:36] !log hashar@deploy1003 gergesshamon, hashar: Backport for [[gerrit:1359334|[arwiki] Remove uploader user group and merge it with autoreview (T442578)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:15:55] _Gerges: https://www.mediawiki.org/wiki/Manual:EmptyUserGroup.php then [20:16:03] hashar: should be able to run it [20:16:03] today I learned about that script :b [20:16:17] the change is on the test servers, anyone can validate it? [20:16:26] robh@cumin1004 decommission (PID 3448160) is awaiting input [20:17:07] I guess via https://ar.wikipedia.org/wiki/Special:ListGroupRights [20:17:14] but the WikimediaDebug extension is broken for me for some reason [20:17:49] _Gerges: are you able to test ^ [20:17:59] !log robh@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - robh@cumin1004" [20:17:59] <_Gerges> WikimediaDebug working on chrome [20:18:34] <_Gerges> I tested patch [20:19:18] and it is good? :) [20:19:36] !log hashar@deploy1003 gergesshamon, hashar: Continuing with deployment [20:19:59] !log robh@cumin1004 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - robh@cumin1004" [20:19:59] !log robh@cumin1004 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [20:20:00] !log robh@cumin1004 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts sretest2005.codfw.wmnet [20:20:07] 10ops-codfw, 06DC-Ops, 10decommission-hardware: decommission/return to spares sretest2005.codfw.wmnet - https://phabricator.wikimedia.org/T442638#12481150 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by robh@cumin1004 for hosts: `sretest2005.codfw.wmnet` - sretest2005.codfw.wmnet (**FAIL*... [20:20:31] <_Gerges> @hasher yes [20:20:48] <_Gerges> @hashar [20:20:50] nice. Your change is being deployed to the servers [20:21:24] <_Gerges> you can script https://www.mediawiki.org/wiki/Manual:EmptyUserGroup.php [20:21:35] possibly [20:21:47] well I do have access to run the script [20:21:54] but I don't remember how to run the scripts [20:22:49] !log ayounsi@cumin1004 START - Cookbook sre.dns.netbox [20:23:19] hashar: https://wikitech.wikimedia.org/wiki/Mwscript [20:24:00] and the group to remove user from is "uploader" ? [20:24:21] <_Gerges> yes [20:25:20] !log hashar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1359334|[arwiki] Remove uploader user group and merge it with autoreview (T442578)]] (duration: 14m 07s) [20:25:23] <_Gerges> Wouldn't it have been better to run it before deploy? [20:25:29] T442578: Merge "uploader" group into "autoreview" group in arwiki - https://phabricator.wikimedia.org/T442578 [20:25:33] oh it even support giving a reason [20:25:36] mwscript emptyUserGroup.php --wiki=arwiki uploader --create-log --log-reason 'uploader user group merged into autoreview - T442578' [20:26:12] which I think will add entries to Special:Log [20:26:17] not sure whether it is a good idea [20:27:38] !log ayounsi@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: normalize - ayounsi@cumin1004" [20:27:41] Removing users from uploader... [20:27:41] ...done! Removed 7 users in total. [20:28:02] hashar: should probably !log too [20:28:40] yeah good point [20:28:48] !log arwiki: mwscript emptyUserGroup.php --wiki=arwiki uploader --create-log --log-reason 'uploader user group merged into autoreview - T442578' [20:28:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:29:01] internally we have an audit log of those commands [20:29:08] but it never hurts to log publicly as well [20:29:27] I posted a comment on the Phab task at https://phabricator.wikimedia.org/T442578#12481170 [20:29:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12481173 (10phaultfinder) [20:30:00] Lgtm [20:30:17] let me know if anything looks broken. I am still around for the next 30 minutes or so [20:30:27] thank you RhinosF1 and _Gerges! [20:30:42] ayounsi@cumin1004 netbox (PID 3451336) is awaiting input [20:31:08] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: normalize - ayounsi@cumin1004" [20:31:08] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [20:31:15] Your welcome, thank you to you too! [20:32:06] !log bking@cumin2003 START - Cookbook sre.dns.netbox [20:37:32] bking@cumin2003 netbox (PID 1278432) is awaiting input [20:39:22] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix IPv6 PTR - bking@cumin2003" [20:39:26] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix IPv6 PTR - bking@cumin2003" [20:39:26] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [20:41:23] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_codfw and A:cp - 9.2.16 upgrade (T441945) [20:52:22] 10ops-codfw, 06SRE, 06DC-Ops, 10decommission-hardware: decommission/return to spares sretest2005.codfw.wmnet - https://phabricator.wikimedia.org/T442638#12481237 (10RobH) 05Open→03Resolved a:03RobH manual dns run to fix decom and set host to inventory as it has no use at this time in prod but may... [20:58:26] I'm going to make use of the deploy window in 2 mins. Looks like nothing is in flight? [20:58:57] (03PS3) 10Jdlrobson: Enable Vector 2022 CTA on test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358273 (https://phabricator.wikimedia.org/T441705) [20:59:00] (03CR) 10Jdlrobson: Enable Vector 2022 CTA on test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358273 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [20:59:28] (03PS4) 10Jdlrobson: Enable Vector 2022 CTA on test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358273 (https://phabricator.wikimedia.org/T441705) [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261027T2100) [21:01:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358888 (https://phabricator.wikimedia.org/T438390) (owner: 10Jdlrobson) [21:02:47] (03Merged) 10jenkins-bot: Merge master into wmf.1 [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358888 (https://phabricator.wikimedia.org/T438390) (owner: 10Jdlrobson) [21:03:15] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1358888|Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585)]] (1.48.0-wmf.1) [21:03:28] That's a lot of tickets! [21:03:47] T438390: Support unsave flow via configpopover - https://phabricator.wikimedia.org/T438390 [21:03:48] T438405: Update mobile collections view items to match spec - https://phabricator.wikimedia.org/T438405 [21:03:48] T438411: Support renaming collections - https://phabricator.wikimedia.org/T438411 [21:03:49] T438413: Support deleting collections - https://phabricator.wikimedia.org/T438413 [21:03:49] T440609: Visual refinements to NavigationBar component - https://phabricator.wikimedia.org/T440609 [21:03:50] T440858: ReadingLists tooltip appears too far from anchor on narrower screens - https://phabricator.wikimedia.org/T440858 [21:03:51] T440995: Bookmark CTA: Blue dot gets cut off by hatnote - https://phabricator.wikimedia.org/T440995 [21:03:51] T441249: Visual refinements for create/rename collection (mobile and desktop) - https://phabricator.wikimedia.org/T441249 [21:03:52] T441468: Change bookmark label and tooltip when custom lists is enabled - https://phabricator.wikimedia.org/T441468 [21:03:53] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [21:03:53] T441696: Desktop CTA: Show bookmark for logged-out users on desktop - https://phabricator.wikimedia.org/T441696 [21:03:54] T441698: Desktop CTA: Update CtaDialog UI - https://phabricator.wikimedia.org/T441698 [21:03:54] T441700: Desktop CTA: Style pulsating blue dot for desktop - https://phabricator.wikimedia.org/T441700 [21:03:55] T441702: Desktop CTA: Hide homepage tour popover - https://phabricator.wikimedia.org/T441702 [21:03:56] T441797: tests: Follow-ups on Vitest test runner transformation - https://phabricator.wikimedia.org/T441797 [21:03:56] T441918: Add page to collection via bookmark button popover should copy page to the collection - https://phabricator.wikimedia.org/T441918 [21:03:57] T442075: URLs are broken on non-English sites - https://phabricator.wikimedia.org/T442075 [21:03:57] T442419: [Accessibility] ReadingList anonymous CTA Popovers need to be reachable via the keyboard - https://phabricator.wikimedia.org/T442419 [21:03:58] T442585: [Regression] Creating a new collection from a collection page leads to broken experience - https://phabricator.wikimedia.org/T442585 [21:04:02] Crikey [21:04:39] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1358888|Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:06:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:06:52] !log kamila@cumin1004 START - Cookbook sre.hosts.decommission for hosts deploy2002.codfw.wmnet [21:07:04] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [21:07:58] !log kamila@cumin1004 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts deploy2002.codfw.wmnet [21:08:52] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1358888|Merge master into wmf.1 (T438390 T438405 T438411 T438413 T440609 T440858 T440995 T441249 T441468 T441477 T441696 T441698 T441700 T441702 T441797 T441918 T442075 T442419 T442585)]] (1.48.0-wmf.1) (duration: 05m 37s) [21:09:36] T438390: Support unsave flow via configpopover - https://phabricator.wikimedia.org/T438390 [21:09:37] T438405: Update mobile collections view items to match spec - https://phabricator.wikimedia.org/T438405 [21:09:37] T438411: Support renaming collections - https://phabricator.wikimedia.org/T438411 [21:09:38] T438413: Support deleting collections - https://phabricator.wikimedia.org/T438413 [21:09:38] T440609: Visual refinements to NavigationBar component - https://phabricator.wikimedia.org/T440609 [21:09:39] T440858: ReadingLists tooltip appears too far from anchor on narrower screens - https://phabricator.wikimedia.org/T440858 [21:09:40] T440995: Bookmark CTA: Blue dot gets cut off by hatnote - https://phabricator.wikimedia.org/T440995 [21:09:40] T441249: Visual refinements for create/rename collection (mobile and desktop) - https://phabricator.wikimedia.org/T441249 [21:09:41] T441468: Change bookmark label and tooltip when custom lists is enabled - https://phabricator.wikimedia.org/T441468 [21:09:42] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [21:09:42] T441696: Desktop CTA: Show bookmark for logged-out users on desktop - https://phabricator.wikimedia.org/T441696 [21:09:43] T441698: Desktop CTA: Update CtaDialog UI - https://phabricator.wikimedia.org/T441698 [21:09:44] T441700: Desktop CTA: Style pulsating blue dot for desktop - https://phabricator.wikimedia.org/T441700 [21:09:44] T441702: Desktop CTA: Hide homepage tour popover - https://phabricator.wikimedia.org/T441702 [21:09:45] T441797: tests: Follow-ups on Vitest test runner transformation - https://phabricator.wikimedia.org/T441797 [21:09:45] T441918: Add page to collection via bookmark button popover should copy page to the collection - https://phabricator.wikimedia.org/T441918 [21:09:46] T442075: URLs are broken on non-English sites - https://phabricator.wikimedia.org/T442075 [21:09:47] T442419: [Accessibility] ReadingList anonymous CTA Popovers need to be reachable via the keyboard - https://phabricator.wikimedia.org/T442419 [21:09:47] T442585: [Regression] Creating a new collection from a collection page leads to broken experience - https://phabricator.wikimedia.org/T442585 [21:11:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:11:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358273 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [21:12:12] (03PS1) 10Kamila Součková: alias deployment.codfw.wmnet to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1359395 (https://phabricator.wikimedia.org/T426222) [21:12:45] (03Merged) 10jenkins-bot: Enable Vector 2022 CTA on test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358273 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [21:14:19] !log sudo cumin "gitlab-runner*" "systemctl restart ferm" on Tyler's request [21:14:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:16:41] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1358273|Enable Vector 2022 CTA on test wikis (T441705)]] [21:16:48] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [21:17:41] (03PS1) 10DLynch: editcheck-headless: upgrade image to 2026-10-27-200626-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359396 (https://phabricator.wikimedia.org/T432286) [21:19:29] (03CR) 10CI reject: [V:04-1] editcheck-headless: upgrade image to 2026-10-27-200626-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359396 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [21:22:42] (03PS1) 10Jforrester: mathoid: Bump from 2026-05-12-175031-production to 2026-10-27-201041-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359397 [21:23:52] (03CR) 10DLynch: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359396 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [21:25:06] (03CR) 10CI reject: [V:04-1] mathoid: Bump from 2026-05-12-175031-production to 2026-10-27-201041-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359397 (owner: 10Jforrester) [21:27:24] hmm I was expecting this config deploy to be quick but I guess it has run scap - is that because of the Depends-On in the commit message? [21:30:20] brennen: any ideas ? ^ I'm also not sure if spiderpig is stuck.. it hasn't updated me in a while now (13mins). [21:31:40] ok it's doing stuff again.. just the first time I've experienced this much time for a config-only change. [21:32:01] Jdlrobson: yeah, not sure about duration, but was just about to say it seems to be rolling along now [21:32:32] Since the config patch depends on an already backported change, I suspect it's just re-deploying that change redundantly? [21:36:43] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1358273|Enable Vector 2022 CTA on test wikis (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:36:50] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [21:37:49] (03PS1) 10Jforrester: [abstractwiki-rust] Pin wasi-libc to backports too [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359399 [21:40:00] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [21:40:33] (03PS1) 10Krinkle: [WIP] varnish: Write tests for `?download` [puppet] - 10https://gerrit.wikimedia.org/r/1359400 [21:42:28] (03PS1) 10Jdlrobson: Fix typo in config name [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359401 (https://phabricator.wikimedia.org/T441705) [21:45:25] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 654.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:45:44] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359402 [21:46:40] (03CR) 10CI reject: [V:04-1] mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359402 (owner: 10PipelineBot) [21:47:13] (03CR) 10VolkerE: [C:03+1] Fix typo in config name [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359401 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [21:50:23] (03CR) 10Jasmine: [C:03+1] alias deployment.codfw.wmnet to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1359395 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [21:51:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:51:41] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1358273|Enable Vector 2022 CTA on test wikis (T441705)]] (duration: 35m 00s) [21:51:45] 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648 (10cfoster) 03NEW [21:51:49] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [21:51:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359401 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [21:52:56] (03Merged) 10jenkins-bot: Fix typo in config name [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359401 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [21:53:02] 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12481876 (10cfoster) [21:53:24] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1359401|Fix typo in config name (T441705)]] [21:56:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:57:59] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1359401|Fix typo in config name (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:58:06] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [21:59:57] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [22:00:12] (03CR) 10Jasmine: alias deployment.codfw.wmnet to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1359395 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [22:02:55] (03CR) 10Jasmine: alias deployment.codfw.wmnet to deploy2003 (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1359395 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [22:05:22] (03CR) 10Jasmine: alias deployment.codfw.wmnet to deploy2003 (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1359395 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [22:07:11] (03CR) 10Jdlrobson: Enable Vector 2022 CTA on test wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358273 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [22:07:29] (03PS1) 10Majavah: gitlab_runner: Use ferm::join_hosts to resolve host names [puppet] - 10https://gerrit.wikimedia.org/r/1359404 [22:07:52] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1359401|Fix typo in config name (T441705)]] (duration: 14m 28s) [22:08:00] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [22:08:20] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9561/co" [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (owner: 10Majavah) [22:09:18] (03PS2) 10Majavah: gitlab_runner: Use ferm::join_hosts to resolve host names [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (https://phabricator.wikimedia.org/T439046) [22:09:34] (03PS5) 10Jdlrobson: Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech Bug: T441705 Change-Id: Ie045af6a21b06540762c303919521f8978661211 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359380 (https://phabricator.wikimedia.org/T441705) [22:09:38] (03PS6) 10Jdlrobson: Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359380 (https://phabricator.wikimedia.org/T441705) [22:24:15] 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12481962 (10kzimmerman) Approved as Corbin's manager [22:25:34] (03CR) 10Dzahn: [C:03+2] puppet-diffs: give cloudinfra-cloudvps-puppetserver its own facts realm [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) (owner: 10Dzahn) [22:31:07] (03CR) 10Dzahn: [C:03+2] "Thanks for the review and for setting the realm override in Horizon. I will now wait until tomorrow when the first uploaded with the new r" [puppet] - 10https://gerrit.wikimedia.org/r/1355152 (https://phabricator.wikimedia.org/T441067) (owner: 10Dzahn) [22:43:01] (03PS1) 10Dzahn: planet: set an identifiable User-Agent [puppet] - 10https://gerrit.wikimedia.org/r/1359406 (https://phabricator.wikimedia.org/T442071) [22:43:25] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.17 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:44:16] (03CR) 10Dzahn: [C:03+2] planet: set an identifiable User-Agent [puppet] - 10https://gerrit.wikimedia.org/r/1359406 (https://phabricator.wikimedia.org/T442071) (owner: 10Dzahn) [22:49:07] !log planet2003,planet1003 - re-enabled puppet after gerrit:1359406 T442071 and being disabled for somet ime [22:49:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:57:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [23:00:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:05:56] (03CR) 10Dzahn: [C:03+1] cdn: exclude miscweb from ATSBackendErrorsHigh [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) (owner: 10Jelto) [23:08:14] (03PS3) 10Dzahn: rsyslog: send php8.5-fpm logs to Logstash [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [23:09:20] (03PS4) 10Dzahn: rsyslog: remove lookup entries for php7.0-fpm [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [23:10:08] (03CR) 10Dzahn: "rebased! then adjusted commit message to what is left that it does after another change partially did this already." [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [23:10:28] (03CR) 10Ahmon Dancy: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359402 (owner: 10PipelineBot) [23:10:38] (03CR) 10RLazarus: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359397 (owner: 10Jforrester) [23:12:45] (03CR) 10Dzahn: "Acknowledged" [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [23:17:20] (03PS1) 10Santiago Faci: Deploy Test Kitchen UI v2.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359409 (https://phabricator.wikimedia.org/T442521) [23:17:31] (03CR) 10Dzahn: [C:03+1] "php7.0-fpm last shipped with Debian stretch and Moritz removed stretch support from puppet in August 2023 (54c22f0a33), after the last str" [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [23:18:25] (03CR) 10Clare Ming: [C:03+2] Deploy Test Kitchen UI v2.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359409 (https://phabricator.wikimedia.org/T442521) (owner: 10Santiago Faci) [23:18:34] (03CR) 10Santiago Faci: [C:03+2] test-kitchen: drop the phabricator_api_token secret [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359276 (https://phabricator.wikimedia.org/T442521) (owner: 10Brouberol) [23:20:20] (03Merged) 10jenkins-bot: Deploy Test Kitchen UI v2.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359409 (https://phabricator.wikimedia.org/T442521) (owner: 10Santiago Faci) [23:20:24] (03Merged) 10jenkins-bot: test-kitchen: drop the phabricator_api_token secret [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359276 (https://phabricator.wikimedia.org/T442521) (owner: 10Brouberol) [23:21:49] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Tab completion for cookbook names - https://phabricator.wikimedia.org/T367230#12482226 (10Lizimedia) Just as an experiment, I Rewrote It In Rust™ using [[https://docs.rs/ruff_python_parser|ruff_python_parser]] and the difference is... [23:23:12] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [23:23:25] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 656.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:23:31] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [23:25:52] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12482235 (10phaultfinder) [23:26:29] (03CR) 10RLazarus: [V:03+2 C:03+2] [abstractwiki-rust] Pin wasi-libc to backports too [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359399 (owner: 10Jforrester) [23:30:41] jouncebot: nowandnext [23:30:41] No deployments scheduled for the next 2 hour(s) and 29 minute(s) [23:30:41] In 2 hour(s) and 29 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T0200) [23:31:01] (03PS1) 10Ladsgroup: ffmpeg: Actually apply the -speed argument [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359411 [23:31:14] (03PS1) 10Ladsgroup: ffmpeg: Actually apply the -speed argument [extensions/TimedMediaHandler] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359412 [23:31:24] (03CR) 10Ladsgroup: [C:03+2] ffmpeg: Actually apply the -speed argument [extensions/TimedMediaHandler] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359412 (owner: 10Ladsgroup) [23:31:29] (03CR) 10Ladsgroup: [C:03+2] ffmpeg: Actually apply the -speed argument [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359411 (owner: 10Ladsgroup) [23:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:37:12] (03Merged) 10jenkins-bot: ffmpeg: Actually apply the -speed argument [extensions/TimedMediaHandler] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359412 (owner: 10Ladsgroup) [23:37:47] (03Merged) 10jenkins-bot: ffmpeg: Actually apply the -speed argument [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359411 (owner: 10Ladsgroup) [23:38:23] !log ladsgroup@deploy1003 Started scap sync-world: Backport for ffmpeg: Actually apply the -speed argument ([[gerrit:1359411|1.48.0-wmf.1]], [[gerrit:1359412|1.47.0-wmf.24]]) [23:40:20] (03CR) 10RLazarus: [V:03+2 C:03+2] "Done:" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359399 (owner: 10Jforrester) [23:40:57] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply [23:41:21] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply [23:42:57] !log ladsgroup@deploy1003 ladsgroup: Backport for ffmpeg: Actually apply the -speed argument ([[gerrit:1359411|1.48.0-wmf.1]], [[gerrit:1359412|1.47.0-wmf.24]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:43:52] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [23:49:32] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for ffmpeg: Actually apply the -speed argument ([[gerrit:1359411|1.48.0-wmf.1]], [[gerrit:1359412|1.47.0-wmf.24]]) (duration: 11m 08s)