[00:00:08] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:17] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:21] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:28] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [00:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:33:57] (03PS1) 10Krinkle: Rdbms: Fix inverted APCU freshness condition in LoadMonitor [core] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1373153 (https://phabricator.wikimedia.org/T440885) [00:38:49] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1373157 [00:38:49] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1373157 (owner: 10TrainBranchBot) [00:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:44:21] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1373157 (owner: 10TrainBranchBot) [01:08:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1373168 [01:08:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1373168 (owner: 10TrainBranchBot) [01:09:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591687 (10phaultfinder) [01:16:10] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1373168 (owner: 10TrainBranchBot) [01:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [01:39:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591721 (10phaultfinder) [02:03:08] !log setting compaction throughput to 40MiB (from 20MiB) & concurrent compactors to 15 (from 10) — T445661 [02:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:03:13] T445661: sessionstore: rapid increase in utilization - https://phabricator.wikimedia.org/T445661 [02:03:22] !log setting compaction throughput to 40MiB (from 20MiB) & concurrent compactors to 15 (from 10); sessionstore1004-a — T445661 [02:03:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:04:31] !log start major compaction; sessionstore1004-a — T445661 [02:04:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:04:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591751 (10phaultfinder) [02:08:59] !log set compaction throughput to 40MiB & concurrent compactors to 15; sessionstore[2004-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet — T445661 [02:09:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:09:03] T445661: sessionstore: rapid increase in utilization - https://phabricator.wikimedia.org/T445661 [02:09:16] !log starting split compaction; sessionstore[2004-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet — T445661 [02:09:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2136.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-work [02:13:03] odfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2370.codfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2298.codfw.wmnet, wikikube-worker2088.codfw.wmnet, wikikube-worker2286.codfw.wmnet, wikikube-worker2034.codfw.wmnet, wikikube-worker2129.codfw.wmnet, wikikube-worker2167.codfw.wmnet, wikikube-worker2316.codfw.wmnet, wikikube-worker2152.codfw.wmnet, w [02:13:03] worker2360.codfw.wmnet, wikikube-worker2210.codfw.wmnet, wikikube-worker2265.codfw.wmnet, wikikube-worker2310.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2306.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [02:13:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-work [02:13:13] odfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2340.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2372.codfw.wmnet, w [02:13:13] worker2289.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2313.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [02:14:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:14:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:24:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591776 (10phaultfinder) [02:34:38] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445684#12591790 (10phaultfinder) [02:40:53] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:44:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591796 (10phaultfinder) [02:45:53] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [03:09:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591839 (10phaultfinder) [03:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 15h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [03:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [03:30:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [03:50:15] FIRING: PingLossPercent: Blackbox probe packet loss 9.583% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:51:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2091.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2308.codfw.wmnet, wikikube-work [03:51:03] odfw.wmnet, wikikube-worker2317.codfw.wmnet, wikikube-worker2286.codfw.wmnet, wikikube-worker2367.codfw.wmnet, wikikube-worker2275.codfw.wmnet, wikikube-worker2034.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2051.codfw.wmnet, wikikube-worker2111.codfw.wmnet, wikikube-worker2362.codfw.wmnet, wikikube-worker2107.codfw.wmnet, wikikube-worker2205.codfw.wmnet, wikikube-worker2243.codfw.wmnet, w [03:51:03] worker2170.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2312.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2092.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [03:52:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:54:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591873 (10phaultfinder) [03:56:15] FIRING: KubernetesDeploymentUnavailableReplicas: ... [03:56:15] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [03:56:15] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [04:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [04:05:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-work [04:05:13] odfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2289.codfw.wmnet, w [04:05:13] worker2065.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2089.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [04:06:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [04:09:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591886 (10phaultfinder) [04:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:29:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2353.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2089.codfw.wmnet, wikikube-worker2302.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2075.codfw.wmnet, wikikube-work [04:29:13] odfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2056.codfw.wmnet, wikikube-worker2106.codfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2208.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2275.codfw.wmnet, wikikube-worker2271.codfw.wmnet, wikikube-worker2108.codfw.wmnet, wikikube-worker2154.codfw.wmnet, wikikube-worker2193.codfw.wmnet, wikikube-worker2366.codfw.wmnet, wikikube-worker2006.codfw.wmnet, w [04:29:13] worker2188.codfw.wmnet, wikikube-worker2331.codfw.wmnet, wikikube-worker2038.codfw.wmnet, wikikube-worker2067.codfw.wmnet, wikikube-worker2077.codfw.wmnet, wikikube-worker2011.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [04:30:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [04:35:03] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1017 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [04:36:03] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1017 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [04:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:04:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12591943 (10phaultfinder) [05:20:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2340.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-work [05:20:13] odfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2130.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2160.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2339.codfw.wmnet, w [05:20:13] worker2110.codfw.wmnet, wikikube-worker2156.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2127.codfw.wmnet, wikikube-worker2049.codfw.wmnet, wikikube-worker2266.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [05:21:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-work [05:21:03] odfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2333.codfw.wmnet, wikikube-worker2049.codfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2298.codfw.wmnet, wikikube-worker2037.codfw.wmnet, wikikube-worker2199.codfw.wmnet, wikikube-worker2194.codfw.wmnet, wikikube-worker2046.codfw.wmnet, wikikube-worker2360.codfw.wmnet, wikikube-worker2042.codfw.wmnet, w [05:21:03] worker2192.codfw.wmnet, wikikube-worker2366.codfw.wmnet, wikikube-worker2068.codfw.wmnet, wikikube-worker2188.codfw.wmnet, wikikube-worker2303.codfw.wmnet, wikikube-worker2349.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [05:21:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:22:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:24:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-work [05:24:13] odfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2151.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2105.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2214.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2056.codfw.wmnet, wikikube-worker2208.codfw.wmnet, wikikube-worker2335.codfw.wmnet, wikikube-worker2090.codfw.wmnet, w [05:24:13] worker2173.codfw.wmnet, wikikube-worker2194.codfw.wmnet, wikikube-worker2284.codfw.wmnet, wikikube-worker2163.codfw.wmnet, wikikube-worker2294.codfw.wmnet, wikikube-worker2115.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [05:25:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:33:05] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2329.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-work [05:33:05] odfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2249.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2075.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2056.codfw.wmnet, wikikube-worker2049.codfw.wmnet, wikikube-worker2267.codfw.wmne [05:33:05] ube-worker2138.codfw.wmnet, wikikube-worker2351.codfw.wmnet, wikikube-worker2088.codfw.wmnet, wikikube-worker2317.codfw.wmnet, wikikube-worker2037.codfw.wmnet, wikikube-worker2258.codfw https://wikitech.wikimedia.org/wiki/PyBal [05:34:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [05:48:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-work [05:48:13] odfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2340.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2325.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, w [05:48:13] worker2248.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [05:49:05] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-work [05:49:05] odfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2276.codfw.wmnet, w [05:49:05] worker2296.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2133.codfw.wmnet, wikikube-worker2324.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [05:50:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:50:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:56:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-work [05:56:13] odfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2108.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2287.codfw.wmnet, w [05:56:13] worker2130.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2314.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [05:57:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2179.codfw.wmnet, wikikube-worker2018.codfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-work [05:57:03] odfw.wmnet, wikikube-worker2367.codfw.wmnet, wikikube-worker2129.codfw.wmnet, wikikube-worker2342.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2210.codfw.wmnet, wikikube-worker2309.codfw.wmnet, wikikube-worker2057.codfw.wmnet, wikikube-worker2145.codfw.wmnet, wikikube-worker2277.codfw.wmnet, wikikube-worker2349.codfw.wmnet, wikikube-worker2058.codfw.wmnet, wikikube-worker2153.codfw.wmnet, wikikube-worker2078.codfw.wmnet, w [05:57:03] worker2243.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2263.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [05:57:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:58:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [05:59:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445741#12592003 (10phaultfinder) [06:05:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2356.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-work [06:05:03] odfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2088.codfw.wmnet, wikikube-worker2034.codfw.wmnet, wikikube-worker2186.codfw.wmnet, wikikube-worker2265.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2365.codfw.wmnet, wikikube-worker2349.codfw.wmnet, wikikube-worker2330.codfw.wmnet, wikikube-worker2288.codfw.wmnet, w [06:05:03] worker2170.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2355.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [06:06:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:13:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-work [06:13:13] odfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2251.codfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2140.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2094.codfw.wmnet, w [06:13:13] worker2258.codfw.wmnet, wikikube-worker2286.codfw.wmnet, wikikube-worker2163.codfw.wmnet, wikikube-worker2294.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2288.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [06:16:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2191.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-work [06:16:03] odfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2075.codfw.wmnet, wikikube-worker2018.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2298.codfw.wmnet, wikikube-worker2212.codfw.wmnet, wikikube-worker2194.codfw.wmnet, wikikube-worker2209.codfw.wmnet, wikikube-worker2074.codfw.wmnet, wikikube-worker2152.codfw.wmnet, w [06:16:03] worker2142.codfw.wmnet, wikikube-worker2064.codfw.wmnet, wikikube-worker2365.codfw.wmnet, wikikube-worker2349.codfw.wmnet, wikikube-worker2338.codfw.wmnet, wikikube-worker2299.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [06:16:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:17:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:21:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2172.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-work [06:21:13] odfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2143.codfw.wmnet, wikikube-worker2050.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2300.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2109.codfw.wmnet, wikikube-worker2326.codfw.wmne [06:21:13] ube-worker2013.codfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2208.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-worker2138.codfw.wmnet, wikikube-worker2351.codfw https://wikitech.wikimedia.org/wiki/PyBal [06:22:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:24:03] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-work [06:24:03] odfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2164.codfw.wmnet, w [06:24:03] worker2272.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2114.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [06:24:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12592008 (10phaultfinder) [06:26:03] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:35:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 6.458% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [06:41:00] FIRING: PingLossPercent: Blackbox probe packet loss 10.42% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [06:45:13] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2276.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2187.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-work [06:45:13] odfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2340.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, w [06:45:13] worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2298.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [06:46:13] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [06:49:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12592015 (10phaultfinder) [06:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [07:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 11h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [07:30:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:36:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [08:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261122T0800) [08:00:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [08:00:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [08:00:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [08:01:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [08:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [08:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:32:18] 06SRE: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768 (10Novem_Linguae) 03NEW [08:35:21] 06SRE: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592084 (10Kudpung) I had also mentioned this to @Novem_Linguae . I am in Thailand. [08:41:12] 06SRE: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592087 (10ClaudineChionh) I've been experiencing this, and I'm in Australia, I think I see at least two other ESEAP-based editors in the [[ https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)... [08:41:14] 06SRE: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592088 (10Chipmunkdavis) Adding to Thailand, I have also specifically heard this is a problem from users in Australia, Indonesia, and Singapore. It has been very consistent for a couple of days now. Repor... [08:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:52:44] 06SRE: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592092 (10Dragoniez) Multiple users on jawiki have reported this issue as well ([[ https://ja.wikipedia.org/wiki/Wikipedia:%E3%83%90%E3%82%B0%E3%81%AE%E5%A0%B1%E5%91%8A#%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B... [09:09:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12592097 (10phaultfinder) [09:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:45:51] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592107 (10phaultfinder) [10:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12592110 (10phaultfinder) [10:07:14] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - ml-staging-ctrl_6443: Servers ml-staging-ctrl2001.codfw.wmnet are marked down but pooled: linkrecommendation-external_4006: Servers wikikube-worker2102.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2152.codfw.wmnet, wikikube-worker2107.co [10:07:14] t are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [10:08:14] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [10:18:46] 06SRE: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592117 (10tstarling) Seems quite consistent to me, e.g. https://en.wikipedia.org/wiki/Barack_Obama loading in 77 seconds for 425 KB. And the page text appears progressively over that time, not all at once... [10:22:01] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592118 (10Peachey88) [10:25:25] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592121 (10tstarling) I paged SRE. [10:27:26] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592123 (10tstarling) {F116770103,width=800} [10:29:52] !ack [10:29:52] 8760 (ACKED) Manual (paged) by Tim Starling (tstarling@wikimedia.org): codfw -> eqsin packet loss, eqsin users complaining T445768 [10:29:53] T445768: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768 [10:38:23] !log bump ospf metric to 5000 on eqsin-codfw HE transport - T445768 [10:38:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:38:27] T445768: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768 [10:39:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12592141 (10phaultfinder) [10:41:15] FIRING: PingLossPercent: Blackbox probe packet loss 8.542% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [10:41:46] (03PS1) 10Blake: common: update link weight [homer/public] - 10https://gerrit.wikimedia.org/r/1373262 [10:43:56] (03CR) 10Ayounsi: [C:03+1] common: update link weight [homer/public] - 10https://gerrit.wikimedia.org/r/1373262 (owner: 10Blake) [10:43:59] (03CR) 10Ayounsi: [C:03+2] common: update link weight [homer/public] - 10https://gerrit.wikimedia.org/r/1373262 (owner: 10Blake) [10:44:43] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445684#12592154 (10phaultfinder) [10:45:55] (03Merged) 10jenkins-bot: common: update link weight [homer/public] - 10https://gerrit.wikimedia.org/r/1373262 (owner: 10Blake) [10:46:20] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592157 (10Hason-LEK-SIN) If it's limited to ESEAP, is it because servers in Singapore are down? Are there servers in Japan? (Which Metawiki page de... [10:49:01] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592166 (10Blake) Hi folks, we now expect this to be mitigated, please let us know if that isn't the case. [10:49:51] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592168 (10Novem_Linguae) >>! In T445768#12592157, @Hason-LEK-SIN wrote: > If it's limited to ESEAP, is it because servers in Singapore are down? Ar... [10:50:54] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592169 (10phaultfinder) [10:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [11:01:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.542% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [11:05:24] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592186 (10tstarling) 05Open→03Resolved a:03ayounsi >>! In T445768#12592168, @Novem_Linguae wrote: > I doubt an entire server or data cent... [11:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 7h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [11:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [11:30:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:00:05] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:15] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:26] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:00:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [12:00:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [12:00:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [12:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [12:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:42:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [12:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:50:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.06% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.06% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:00:38] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592253 (10phaultfinder) [13:06:57] (03PS3) 10Tryvix1509: itwiki: Add 'autopatrol', 'editautopatrolprotected' and 'patrol' to 'mover' user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1372605 (https://phabricator.wikimedia.org/T445622) [13:09:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [13:14:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [13:30:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [13:34:44] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592299 (10phaultfinder) [13:35:05] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:36:04] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-work [13:36:04] odfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2164.codfw.wmnet, w [13:36:04] worker2272.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2151.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [13:37:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [13:37:04] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [13:49:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.375% from eqiad to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [13:59:42] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592310 (10phaultfinder) [14:00:30] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [14:10:39] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592326 (10phaultfinder) [14:43:06] (03PS1) 10Andrew Bogott: Update codfw1dev horizon release [puppet] - 10https://gerrit.wikimedia.org/r/1373301 [14:44:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [14:45:29] (03CR) 10Andrew Bogott: [C:03+2] Update codfw1dev horizon release [puppet] - 10https://gerrit.wikimedia.org/r/1373301 (owner: 10Andrew Bogott) [14:50:51] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592350 (10phaultfinder) [14:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.49% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:59:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 3h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [15:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:20:52] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445619#12592386 (10phaultfinder) [15:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [15:28:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.47% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:30:16] (03PS1) 10Ameisenigel: eowikibooks: install quiz extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1373311 (https://phabricator.wikimedia.org/T444080) [15:30:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:34:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12592402 (10phaultfinder) [15:52:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [15:54:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:57:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:02:16] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [16:03:33] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:04:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:08:33] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:11:11] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:28] (03PS1) 10Cathal Mooney: eqsin: de-pref eqsin transport to eqiad over HE [homer/public] - 10https://gerrit.wikimedia.org/r/1373324 (https://phabricator.wikimedia.org/T445769) [16:15:24] (03CR) 10Cathal Mooney: [C:03+2] eqsin: de-pref eqsin transport to eqiad over HE [homer/public] - 10https://gerrit.wikimedia.org/r/1373324 (https://phabricator.wikimedia.org/T445769) (owner: 10Cathal Mooney) [16:16:29] (03Merged) 10jenkins-bot: eqsin: de-pref eqsin transport to eqiad over HE [homer/public] - 10https://gerrit.wikimedia.org/r/1373324 (https://phabricator.wikimedia.org/T445769) (owner: 10Cathal Mooney) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:57] (03PS1) 10Krinkle: FlaggableWikiPage: Fix completely broken cache in stableVersionIsSynced [extensions/FlaggedRevs] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1373325 (https://phabricator.wikimedia.org/T347123) [16:18:14] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-work [16:18:14] odfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2358.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2340.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, w [16:18:14] worker2314.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2355.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2206.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:20:14] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:22:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from eqiad to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [16:27:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [16:37:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins