[00:00:05] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:28] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [00:00:32] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:43] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [00:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [00:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:19:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.75% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:24:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.61% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:24:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.87% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:28:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-work [00:28:13] odfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2108.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2345.codfw.wmnet, w [00:28:13] worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2125.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [00:29:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.85% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:30:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2055.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-work [00:30:03] odfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2370.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2088.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2109.codfw.wmnet, wikikube-worker2199.codfw.wmnet, wikikube-worker2194.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2046.codfw.wmnet, wikikube-worker2154.codfw.wmnet, w [00:30:03] worker2142.codfw.wmnet, wikikube-worker2347.codfw.wmnet, wikikube-worker2203.codfw.wmnet, wikikube-worker2166.codfw.wmnet, wikikube-worker2072.codfw.wmnet, wikikube-worker2362.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [00:30:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:31:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:39:02] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1372995 [00:39:02] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1372995 (owner: 10TrainBranchBot) [00:39:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2148.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-work [00:39:13] odfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2340.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2039.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2273.codfw.wmnet, w [00:39:13] worker2213.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2065.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [00:39:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.41% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:40:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-work [00:40:03] odfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2313.codfw.wmnet, wikikube-worker2055.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2089.codfw.wmnet, wikikube-worker2302.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2127.codfw.wmnet, wikikube-worker2106.codfw.wmnet, wikikube-worker2104.codfw.wmnet, wikikube-worker2146.codfw.wmnet, w [00:40:03] worker2374.codfw.wmnet, wikikube-worker2335.codfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2317.codfw.wmnet, wikikube-worker2207.codfw.wmnet, wikikube-worker2260.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [00:41:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:41:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:42:25] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:43:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:45:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [00:45:39] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1372995 (owner: 10TrainBranchBot) [00:48:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:50:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [00:55:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-work [00:55:03] odfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2133.codfw.wmnet, wikikube-worker2075.codfw.wmnet, wikikube-worker2308.codfw.wmnet, w [00:55:03] worker2348.codfw.wmnet, wikikube-worker2115.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2365.codfw.wmnet, wikikube-worker2058.codfw.wmnet, wikikube-worker2330.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [00:56:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:04:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.95% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:05:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:09:10] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1372996 [01:09:10] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1372996 (owner: 10TrainBranchBot) [01:15:41] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1372996 (owner: 10TrainBranchBot) [01:15:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.85% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:21:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2353.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-work [01:21:03] odfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2035.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2367.codfw.wmnet, wikikube-worker2199.codfw.wmnet, wikikube-worker2115.codfw.wmnet, wikikube-worker2342.codfw.wmnet, wikikube-worker2152.codfw.wmnet, wikikube-worker2366.codfw.wmnet, wikikube-worker2068.codfw.wmnet, w [01:21:03] worker2188.codfw.wmnet, wikikube-worker2057.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2111.codfw.wmnet, wikikube-worker2330.codfw.wmnet, wikikube-worker2153.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [01:22:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:25:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-work [01:25:13] odfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2089.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2268.codfw.wmnet, w [01:25:13] worker2252.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2332.codfw.wmnet, wikikube-worker2208.codfw.wmnet, wikikube-worker2035.codfw.wmnet, wikikube-worker2328.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [01:26:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:34:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [01:44:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590586 (10phaultfinder) [01:51:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2155.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-work [01:51:03] odfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2370.codfw.wmnet, wikikube-worker2034.codfw.wmnet, wikikube-worker2354.codfw.wmnet, wikikube-worker2129.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2210.codfw.wmnet, wikikube-worker2265.codfw.wmnet, w [01:51:03] worker2290.codfw.wmnet, wikikube-worker2303.codfw.wmnet, wikikube-worker2349.codfw.wmnet, wikikube-worker2153.codfw.wmnet, wikikube-worker2288.codfw.wmnet, wikikube-worker2338.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [01:52:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:56:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-work [01:56:03] odfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2160.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2179.codfw.wmnet, w [01:56:03] worker2296.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2268.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [01:57:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [01:59:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590602 (10phaultfinder) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590644 (10phaultfinder) [02:39:38] FIRING: GnmiInterfaceCountersDrop: ... [02:39:38] cr3-ulsfo is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr3-ulsfo:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [02:50:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590660 (10phaultfinder) [02:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [02:56:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2191.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2075.codfw.wmnet, wikikube-work [02:56:03] odfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2374.codfw.wmnet, wikikube-worker2138.codfw.wmnet, wikikube-worker2298.codfw.wmnet, wikikube-worker2367.codfw.wmnet, wikikube-worker2354.codfw.wmnet, wikikube-worker2115.codfw.wmnet, wikikube-worker2042.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2145.codfw.wmnet, wikikube-worker2331.codfw.wmnet, wikikube-worker2051.codfw.wmnet, wikikube-worker2103.codfw.wmnet, w [02:56:03] worker2338.codfw.wmnet, wikikube-worker2250.codfw.wmnet, wikikube-worker2299.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2302.codfw.wmnet, wikikube-worker2050.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [02:57:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:09:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-work [03:09:03] odfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2249.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2275.codfw.wmnet, wikikube-worker2284.codfw.wmnet, wikikube-worker2209.codfw.wmne [03:09:03] ube-worker2129.codfw.wmnet, wikikube-worker2167.codfw.wmnet, wikikube-worker2115.codfw.wmnet, wikikube-worker2074.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2186.codfw https://wikitech.wikimedia.org/wiki/PyBal [03:10:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 15h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [03:15:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [03:18:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [03:20:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [03:21:15] FIRING: PingLossPercent: Blackbox probe packet loss 13.54% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:33:28] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [03:34:03] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [03:42:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-work [03:42:03] odfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2062.codfw.wmnet, w [03:42:03] worker2282.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2346.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [03:43:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [04:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [04:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590720 (10phaultfinder) [04:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [04:06:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-work [04:06:03] odfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2039.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2164.codfw.wmnet, w [04:06:03] worker2179.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2268.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [04:07:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [04:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:24:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590733 (10phaultfinder) [04:41:10] FIRING: BFDdown: BFD session down between cr1-codfw and fe80::669:8f07:ece:f4e7 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:44:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590735 (10phaultfinder) [04:46:10] RESOLVED: BFDdown: BFD session down between cr1-codfw and fe80::669:8f07:ece:f4e7 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [05:23:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [05:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [05:44:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590752 (10phaultfinder) [06:23:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [06:39:38] FIRING: GnmiInterfaceCountersDrop: ... [06:39:38] cr3-ulsfo is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr3-ulsfo:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [06:50:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [06:58:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [06:59:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590773 (10phaultfinder) [07:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 11h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:20:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [07:20:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [07:20:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [07:21:15] FIRING: PingLossPercent: Blackbox probe packet loss 9.375% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [07:22:50] (03CR) 10Ayounsi: [C:03+1] "Not tested but overall lgtm! Thanks. Before it's live please also add it to the doc on https://wikitech.wikimedia.org/wiki/Netbox" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [07:25:35] PROBLEM - Ensure traffic_manager is running for instance backend on cp5022 is CRITICAL: PROCS CRITICAL: 3 processes with args /usr/bin/traffic_manager --nosyslog https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [07:26:35] RECOVERY - Ensure traffic_manager is running for instance backend on cp5022 is OK: PROCS OK: 1 process with args /usr/bin/traffic_manager --nosyslog https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [07:34:38] 10ops-codfw, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445741 (10phaultfinder) 03NEW [07:39:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590806 (10phaultfinder) [07:54:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590809 (10phaultfinder) [08:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [08:05:56] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [08:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:56:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-work [08:56:03] odfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2308.codfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2275.codfw.wmnet, wikikube-worker2271.codfw.wmnet, wikikube-worker2358.codfw.wmnet, wikikube-worker2175.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2078.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2312.codfw.wmnet, w [08:56:03] worker2363.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2106.codfw.wmnet, wikikube-worker2104.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [08:56:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-work [08:56:13] odfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2345.codfw.wmnet, w [08:56:13] worker2198.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2065.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [08:58:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [08:58:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [09:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:49:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590826 (10phaultfinder) [09:51:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-work [09:51:03] odfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2258.codfw.wmnet, wikikube-worker2199.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2358.codfw.wmnet, wikikube-worker2331.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2051.codfw.wmnet, wikikube-worker2001.codfw.wmnet, wikikube-worker2078.codfw.wmnet, wikikube-worker2205.codfw.wmnet, w [09:51:03] worker2061.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2334.codfw.wmnet, wikikube-worker2127.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [09:52:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [10:19:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590830 (10phaultfinder) [10:21:15] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [10:25:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [10:39:38] FIRING: GnmiInterfaceCountersDrop: ... [10:39:38] cr3-ulsfo is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr3-ulsfo:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:50:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:54:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590854 (10phaultfinder) [10:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [11:00:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [11:05:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [11:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 7h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [11:21:15] FIRING: PingLossPercent: Blackbox probe packet loss 9.583% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [11:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590864 (10phaultfinder) [11:26:04] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:39:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590865 (10phaultfinder) [11:59:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590874 (10phaultfinder) [12:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:13] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:18] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:27] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:30] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:38] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [12:05:56] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [12:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:13:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.4% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:20:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-work [12:20:13] odfw.wmnet, wikikube-worker2312.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2157.codfw.wmnet, w [12:20:13] worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [12:21:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [12:38:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.63% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:39:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590911 (10phaultfinder) [12:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:45:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [13:04:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590939 (10phaultfinder) [13:12:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.61% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:15:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [13:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:49:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12590988 (10phaultfinder) [13:51:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.84% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:56:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.84% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:14:23] RESOLVED: GnmiInterfaceCountersDrop: ... [14:14:23] cr3-ulsfo is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr3-ulsfo:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:18:37] !log cmooney@cumin1004 START - Cookbook sre.network.peering with action 'configure' for AS: 11686 [14:18:52] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 11686 [14:19:20] !log cmooney@cumin1004 START - Cookbook sre.network.peering with action 'configure' for AS: 24530 [14:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591004 (10phaultfinder) [14:21:31] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 24530 [14:29:39] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [14:34:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591013 (10phaultfinder) [14:36:01] FIRING: [2x] PingLossPercent: Blackbox probe packet loss 12.08% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [14:43:13] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [14:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.71% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:43:16] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.71% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:50:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.57% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:53:10] FIRING: BFDdown: BFD session down between cr2-eqsin and 103.102.166.148 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591020 (10phaultfinder) [14:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.75% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:58:10] RESOLVED: BFDdown: BFD session down between cr2-eqsin and 103.102.166.148 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:05:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [15:05:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [15:05:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:09:20] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [15:09:24] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [15:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 3h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [15:16:07] (03PS1) 10Southparkfan: haproxy: Allow MathoidToolforgeBot to send requests to Mathoid [puppet] - 10https://gerrit.wikimedia.org/r/1373031 (https://phabricator.wikimedia.org/T271001) [15:16:41] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [15:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591031 (10phaultfinder) [15:29:39] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [15:30:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:30:39] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [15:30:42] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [15:33:43] (03CR) 10Bugreporter: "A more long-term solution is to rewrite krinklebot to use Mathoid CLI. So it can work after Mathoid is undeployed." [puppet] - 10https://gerrit.wikimedia.org/r/1373031 (https://phabricator.wikimedia.org/T271001) (owner: 10Southparkfan) [15:39:47] (03CR) 10Krinkle: [C:04-1] "This LGTM, but setting CR-1 because this is specifically for Beta Cluster only. The production one has no exemptions and will be shutdown " [puppet] - 10https://gerrit.wikimedia.org/r/1373031 (https://phabricator.wikimedia.org/T271001) (owner: 10Southparkfan) [15:44:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:49:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:49:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591037 (10phaultfinder) [16:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [16:06:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2187.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-work [16:06:13] odfw.wmnet, wikikube-worker2134.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2151.codfw.wmnet, wikikube-worker2105.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2214.codfw.wmnet, wikikube-worker2370.codfw.wmnet, wikikube-worker2308.codfw.wmnet, wikikube-worker2335.codfw.wmnet, wikikube-worker2138.codfw.wmnet, wikikube-worker2173.codfw.wmnet, wikikube-worker2194.codfw.wmnet, w [16:06:13] worker2163.codfw.wmnet, wikikube-worker2095.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2108.codfw.wmnet, wikikube-worker2193.codfw.wmnet, wikikube-worker2147.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:07:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:32:28] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:32:32] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:37:26] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:37:29] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:49:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591085 (10phaultfinder) [16:56:10] FIRING: BFDdown: BFD session down between cr2-eqsin and 103.102.166.148 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:01:10] RESOLVED: BFDdown: BFD session down between cr2-eqsin and 103.102.166.148 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:04:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591092 (10phaultfinder) [17:12:56] (03PS1) 10Southparkfan: changeprop: Remove unused RESTBase URI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1373037 (https://phabricator.wikimedia.org/T436477) [17:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591107 (10phaultfinder) [17:33:08] (03PS1) 10Southparkfan: changeprop Beta Cluster: Use correct URIs and zones, remove purge_host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1373039 (https://phabricator.wikimedia.org/T429497) [17:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:37:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:41:01] FIRING: [2x] PingLossPercent: Blackbox probe packet loss 10.21% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [17:42:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:59:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591128 (10phaultfinder) [18:14:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591141 (10phaultfinder) [18:19:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-work [18:19:03] odfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2151.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2133.codfw.wmnet, w [18:19:03] worker2324.codfw.wmnet, wikikube-worker2018.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2013.codfw https://wikitech.wikimedia.org/wiki/PyBal [18:19:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-work [18:19:13] odfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, w [18:19:13] worker2215.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2297.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [18:20:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:22:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:31:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:47:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:51:01] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:52:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:59:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591155 (10phaultfinder) [19:01:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:01:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [19:05:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.167% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [19:05:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [19:05:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [19:05:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [19:06:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:10:41] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12591163 (10phaultfinder) [19:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 23h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:19:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591199 (10phaultfinder) [19:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [19:30:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:40:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [19:44:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [19:54:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591254 (10phaultfinder) [20:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [20:09:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [20:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:16:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:20:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2102.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2058.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube- [20:20:03] 70.codfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2371.codfw.wmnet, wikikube-worker2167.codfw.wmnet, wikikube-worker2358.codfw.wmnet, wikikube-worker2175.codfw.wmnet, wikikube-worker2192.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2309.codfw.wmnet, wikikube-worker2006.codfw.wmnet, wikikube-worker2068.codfw.wmnet, wikikube-worker2168.codfw.wmnet, wikikube-worker2331.codfw.wmne [20:20:03] ube-worker2067.codfw.wmnet, wikikube-worker2306.codfw.wmnet, wikikube-worker2349.codfw.wmnet, wikikube-worker2111.codfw.wmnet, wikikube-worker2368.codfw.wmnet, wikikube-worker2250.codfw https://wikitech.wikimedia.org/wiki/PyBal [20:21:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:21:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:34:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591313 (10phaultfinder) [20:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:19:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591345 (10phaultfinder) [21:34:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:39:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:45:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2155.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2049.codfw.wmnet, wikikube-worker2035.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-work [21:45:03] odfw.wmnet, wikikube-worker2209.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2129.codfw.wmnet, wikikube-worker2360.codfw.wmnet, wikikube-worker2192.codfw.wmnet, wikikube-worker2168.codfw.wmnet, wikikube-worker2195.codfw.wmnet, wikikube-worker2145.codfw.wmnet, wikikube-worker2153.codfw.wmnet, wikikube-worker2001.codfw.wmnet, wikikube-worker2288.codfw.wmnet, wikikube-worker2299.codfw.wmnet, wikikube-worker2185.codfw.wmnet, w [21:45:03] worker2171.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker2355.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2055.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [21:46:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:59:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591366 (10phaultfinder) [22:00:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube- [22:00:03] 13.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2037.codfw.wmnet, wikikube-worker2194.codfw.wmnet, wikikube-worker2129.codfw.wmnet, wikikube-worker2167.codfw.wmnet, wikikube-worker2342.codfw.wmnet, wikikube-worker2130.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2366.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2178.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2067.codfw.wmne [22:00:03] ube-worker2153.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2251.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2344.codfw.wmnet, wikikube-worker2292.codfw https://wikitech.wikimedia.org/wiki/PyBal [22:01:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:04:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2136.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-work [22:04:03] odfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2037.codfw.wmnet, wikikube-worker2286.codfw.wmnet, wikikube-worker2275.codfw.wmnet, wikikube-worker2167.codfw.wmne [22:04:03] ube-worker2341.codfw.wmnet, wikikube-worker2360.codfw.wmnet, wikikube-worker2358.codfw.wmnet, wikikube-worker2186.codfw.wmnet, wikikube-worker2210.codfw.wmnet, wikikube-worker2366.codfw https://wikitech.wikimedia.org/wiki/PyBal [22:05:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:32:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2191.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2133.codfw.wmnet, wikikube-work [22:32:03] odfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2308.codfw.wmnet, wikikube-worker2199.codfw.wmnet, wikikube-worker2152.codfw.wmnet, wikikube-worker2042.codfw.wmnet, wikikube-worker2358.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2068.codfw.wmnet, wikikube-worker2347.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2153.codfw.wmnet, wikikube-worker2288.codfw.wmnet, wikikube-worker2274.codfw.wmnet, w [22:32:03] worker2289.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2313.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2105.codfw.wmnet, wikikube-worker2144.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [22:33:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:52:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-work [22:52:13] odfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, w [22:52:13] worker2177.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2198.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [22:53:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2136.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2088.codfw.wmnet, wikikube-worker2037.codfw.wmnet, wikikube- [22:53:03] 67.codfw.wmnet, wikikube-worker2034.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2046.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2186.codfw.wmnet, wikikube-worker2210.codfw.wmnet, wikikube-worker2309.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2051.codfw.wmnet, wikikube-worker2064.codfw.wmnet, wikikube-worker2131.codfw.wmnet, wikikube-worker2111.codfw.wmne [22:53:03] ube-worker2243.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2055.codfw https://wikitech.wikimedia.org/wiki/PyBal [22:53:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:54:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:54:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591451 (10phaultfinder) [22:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [23:05:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [23:05:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [23:05:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [23:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 19h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [23:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [23:30:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [23:30:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [23:31:31] PROBLEM - Ensure trafficserver_exporter is running for instance backend on cp5022 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [23:32:31] RECOVERY - Ensure trafficserver_exporter is running for instance backend on cp5022 is OK: PROCS OK: 1 process with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [23:50:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [23:55:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas