[18:22:51] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2105-2115,2124-2127].codfw.wmnet [18:25:22] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:26:30] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for brookecamarda - https://phabricator.wikimedia.org/T441893#12474136 (10Brookecamarda) Thank you Daniel! I am hoping to gain access to both LDAP and “analytics-privatedata-users” level. Let me know if you have any further questions, thank you! [18:26:43] (03PS1) 10Arlolra: Turn off "report a visual" bug on wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358832 (https://phabricator.wikimedia.org/T430970) [18:27:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [18:30:25] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [18:30:52] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [18:31:01] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2105-2115,2124-2127].codfw.wmnet [18:31:06] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2105-2115,2124-2127].codfw.wmnet [18:31:10] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for brookecamarda - https://phabricator.wikimedia.org/T441893#12474152 (10Dzahn) Hi @Brookecamarda, so.. for the LDAP part please go via the link above to request it within the Identity Management System. You will need that regardless of the steps on... [18:31:38] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2128-2142].codfw.wmnet [18:31:59] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [18:32:23] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [18:34:25] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_eqsin and not P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) [18:36:30] (03CR) 10AKhatun: [C:03+1] Prefetch the LAC revision_feature_counts artifact on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357360 (https://phabricator.wikimedia.org/T436695) (owner: 10Ottomata) [18:39:50] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2128-2142].codfw.wmnet [18:40:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by akhatun@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357360 (https://phabricator.wikimedia.org/T436695) (owner: 10Ottomata) [18:41:20] (03Merged) 10jenkins-bot: Prefetch the LAC revision_feature_counts artifact on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357360 (https://phabricator.wikimedia.org/T436695) (owner: 10Ottomata) [18:41:43] !log akhatun@deploy1003 Started scap sync-world: Backport for [[gerrit:1357360|Prefetch the LAC revision_feature_counts artifact on testwiki (T436695)]] [18:41:48] T436695: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695 [18:46:09] !log akhatun@deploy1003 akhatun, otto: Backport for [[gerrit:1357360|Prefetch the LAC revision_feature_counts artifact on testwiki (T436695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:48:17] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2128-2142].codfw.wmnet [18:48:22] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2128-2142].codfw.wmnet [18:48:56] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2143-2157].codfw.wmnet [18:57:15] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2143-2157].codfw.wmnet [19:00:55] (03CR) 10Jgreen: [C:03+2] Add _dnsauth.donate cname to wikipedia.org for Digicert DNS validation [dns] - 10https://gerrit.wikimedia.org/r/1358240 (https://phabricator.wikimedia.org/T440185) (owner: 10Jgreen) [19:01:32] !log akhatun@deploy1003 akhatun, otto: Continuing with deployment [19:03:05] !log jgreen@dns2006 START - running authdns-update [19:05:34] !log jgreen@dns2006 END - running authdns-update [19:05:42] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_eqsin and not P{cp5001.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) [19:05:58] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2143-2157].codfw.wmnet [19:06:03] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2143-2157].codfw.wmnet [19:06:35] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2158-2172].codfw.wmnet [19:07:23] !log akhatun@deploy1003 Finished scap sync-world: Backport for [[gerrit:1357360|Prefetch the LAC revision_feature_counts artifact on testwiki (T436695)]] (duration: 25m 40s) [19:07:27] T436695: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695 [19:09:39] (03PS1) 10RLazarus: Bump v1.40.1 to bookworm-wikimedia (was bullseye-wikimedia) [debs/envoyproxy] (v1.40) - 10https://gerrit.wikimedia.org/r/1358838 (https://phabricator.wikimedia.org/T436338) [19:11:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:12:20] (03PS1) 10Bking: WIP: matomo: Use an initContainer to copy static assets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) [19:12:33] (03PS2) 10Bking: WIP: matomo: Use an initContainer to copy static assets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) [19:13:10] (03CR) 10RLazarus: [C:03+2] Bump v1.40.1 to bookworm-wikimedia (was bullseye-wikimedia) [debs/envoyproxy] (v1.40) - 10https://gerrit.wikimedia.org/r/1358838 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [19:14:10] (03CR) 10CI reject: [V:04-1] WIP: matomo: Use an initContainer to copy static assets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [19:14:53] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2158-2172].codfw.wmnet [19:17:46] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:17:56] (03CR) 10Bking: [C:03+2] Matomo: Support `trusted_hosts` array variable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356987 (https://phabricator.wikimedia.org/T436186) (owner: 10Bking) [19:18:45] (03PS3) 10Bking: WIP: matomo: Use an initContainer to copy static assets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) [19:20:21] (03CR) 10CI reject: [V:04-1] WIP: matomo: Use an initContainer to copy static assets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [19:23:06] (03PS4) 10Bking: WIP: matomo: Use an initContainer to copy static assets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) [19:23:34] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2158-2172].codfw.wmnet [19:23:39] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2158-2172].codfw.wmnet [19:24:10] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet [19:24:39] (03CR) 10CI reject: [V:04-1] WIP: matomo: Use an initContainer to copy static assets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [19:24:58] (03PS2) 10Krinkle: labs: Remove redundant wgSkipSkins override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1254463 [19:25:17] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [19:25:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346336 (https://phabricator.wikimedia.org/T228380) (owner: 10Awight) [19:25:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1254463 (owner: 10Krinkle) [19:25:29] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [19:26:07] (03Merged) 10jenkins-bot: Remove unused StatsD configuration and instruments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346336 (https://phabricator.wikimedia.org/T228380) (owner: 10Awight) [19:26:10] (03Merged) 10jenkins-bot: labs: Remove redundant wgSkipSkins override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1254463 (owner: 10Krinkle) [19:26:33] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1346336|Remove unused StatsD configuration and instruments (T228380)]], [[gerrit:1254463|labs: Remove redundant wgSkipSkins override]] [19:26:38] T228380: Tech debt: sunsetting of Graphite - https://phabricator.wikimedia.org/T228380 [19:30:51] !log krinkle@deploy1003 krinkle, awight: Backport for [[gerrit:1346336|Remove unused StatsD configuration and instruments (T228380)]], [[gerrit:1254463|labs: Remove redundant wgSkipSkins override]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:32:04] (03PS1) 10Krinkle: tests: Restore test coverage for `ServiceConfig->getLocalService` [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358842 [19:32:22] (03CR) 10Krinkle: Remove unused StatsD configuration and instruments (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346336 (https://phabricator.wikimedia.org/T228380) (owner: 10Awight) [19:32:30] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet [19:35:05] !log jhathaway@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mx-out2001.wikimedia.org with reason: new-kernel [19:37:39] !log krinkle@deploy1003 krinkle, awight: Continuing with deployment [19:37:54] (03PS1) 10RLazarus: aptrepo: Add component/envoy-future to bookworm-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/1358844 (https://phabricator.wikimedia.org/T436338) [19:37:57] (03PS1) 10Ladsgroup: thumbor: Make timeout more strict [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358845 (https://phabricator.wikimedia.org/T431768) [19:38:11] (03CR) 10Krinkle: [C:03+2] tests: Restore test coverage for `ServiceConfig->getLocalService` [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358842 (owner: 10Krinkle) [19:39:12] (03Merged) 10jenkins-bot: tests: Restore test coverage for `ServiceConfig->getLocalService` [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358842 (owner: 10Krinkle) [19:39:28] (03PS1) 10Jdlrobson: Fix recentlyDonated captures non-donors [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358846 (https://phabricator.wikimedia.org/T441244) [19:39:55] (03PS1) 10Jdlrobson: Fix recentlyDonated captures non-donors [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358847 (https://phabricator.wikimedia.org/T441244) [19:40:37] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet [19:40:42] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2173-2179,2184-2190].codfw.wmnet [19:41:19] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2191-2204].codfw.wmnet [19:41:30] !log jhathaway@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out1001.wikimedia.org with reason: new-kernel [19:43:21] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346336|Remove unused StatsD configuration and instruments (T228380)]], [[gerrit:1254463|labs: Remove redundant wgSkipSkins override]] (duration: 16m 48s) [19:43:27] T228380: Tech debt: sunsetting of Graphite - https://phabricator.wikimedia.org/T228380 [19:44:16] !log jhathaway@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in1001.wikimedia.org with reason: new-kernel [19:44:54] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [19:45:12] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [19:46:05] !log jhathaway@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in2001.wikimedia.org with reason: new-kernel [19:49:15] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [19:49:39] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [19:56:00] (03PS1) 10JHathaway: mx: use production, rather than aggregate network list [puppet] - 10https://gerrit.wikimedia.org/r/1358850 (https://phabricator.wikimedia.org/T442063) [19:56:10] (03CR) 10JHathaway: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1358850 (https://phabricator.wikimedia.org/T442063) (owner: 10JHathaway) [19:58:35] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2191-2204].codfw.wmnet [20:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for UTC late backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261026T2000). [20:00:05] SomeRandomDev and Krinkle: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:14] Hi [20:01:43] (03PS1) 10Eric Gardner: [WIP] Add Creative Commons license icons for the share card [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358851 (https://phabricator.wikimedia.org/T439162) [20:01:45] (03PS1) 10Eric Gardner: [WIP] testwiki: Enable the share card [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358852 (https://phabricator.wikimedia.org/T439162) [20:01:47] (03PS1) 10Eric Gardner: [WIP] Enable the share card on group1 Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358853 (https://phabricator.wikimedia.org/T439162) [20:01:49] (03PS1) 10Eric Gardner: [WIP] Enable the share card on all Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358854 (https://phabricator.wikimedia.org/T439162) [20:01:56] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_ulsfo and not P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [20:02:07] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [20:03:25] (03CR) 10JHathaway: "Arzhel, can you take a look to see if the change looks reasonable?" [puppet] - 10https://gerrit.wikimedia.org/r/1358850 (https://phabricator.wikimedia.org/T442063) (owner: 10JHathaway) [20:06:06] 10ops-eqiad, 06DC-Ops: Data entry for cables on new switch - https://phabricator.wikimedia.org/T442447 (10VRiley-WMF) 03NEW [20:06:23] 10ops-eqiad, 06DC-Ops: Data entry for cables on new switch - https://phabricator.wikimedia.org/T442447#12474656 (10VRiley-WMF) 05Open→03Resolved This has been completed [20:06:42] RoanKattouw, urbanecm, TheresNoTime, cjming: Hi, anybody around for the deployment window? [20:06:54] SomeRandomDev: i can deploy today [20:07:00] what do we have here [20:07:03] Cool [20:07:07] I have a revert in LiquidThreads [20:07:08] no types improvement? [20:07:13] Nope :( [20:07:16] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2191-2204].codfw.wmnet [20:07:21] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2191-2204].codfw.wmnet [20:07:24] (03CR) 10Urbanecm: [C:03+2] Revert "Improve types" [extensions/LiquidThreads] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358445 (https://phabricator.wikimedia.org/T441875) (owner: 10SomeRandomDeveloper) [20:07:25] the error is reproducible at https://strategy.wikimedia.org/wiki/Thread:Category_talk:Proposals_related_to_citations/Possible_goals_of_a_new_citation_proposal I think [20:07:27] okay then [20:07:52] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet [20:07:53] o/ [20:08:01] Krinkle: i'll ping you when done? [20:08:03] I've done mine earlier already :) [20:08:08] ah [20:08:08] yeah done [20:08:13] then i won't :D [20:08:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/LiquidThreads] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358445 (https://phabricator.wikimedia.org/T441875) (owner: 10SomeRandomDeveloper) [20:09:14] (03CR) 10Dzahn: [V:03+1 C:03+2] "all boxes checked and not related to hardware keys." [puppet] - 10https://gerrit.wikimedia.org/r/1358309 (https://phabricator.wikimedia.org/T440031) (owner: 10Dzahn) [20:09:28] (03Merged) 10jenkins-bot: Revert "Improve types" [extensions/LiquidThreads] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358445 (https://phabricator.wikimedia.org/T441875) (owner: 10SomeRandomDeveloper) [20:10:12] Krinkle: unexpected code pulled from you :( [20:10:24] https://www.irccloud.com/pastebin/tJMTKTec/ [20:10:43] (03CR) 10Lizimedia: [C:03+2] instructions to run debmonitor locally [software/debmonitor] - 10https://gerrit.wikimedia.org/r/1358307 (owner: 10Lizimedia) [20:11:06] seems test only [20:11:12] proceeding [20:11:27] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1358445|Revert "Improve types" (T441875)]] (1.48.0-wmf.1) [20:11:30] (03Merged) 10jenkins-bot: instructions to run debmonitor locally [software/debmonitor] - 10https://gerrit.wikimedia.org/r/1358307 (owner: 10Lizimedia) [20:11:33] T441875: Error: Call to a member function isTopmostThread() on null - https://phabricator.wikimedia.org/T441875 [20:11:47] urbanecm: ack, I was going to do another deploy to clear that but forgot. [20:11:49] sorry about that [20:12:03] fortunately easy to judge from the diff :) [20:14:07] (03CR) 10Ladsgroup: [C:03+2] thumbor: Make timeout more strict [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358845 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup) [20:15:46] !log urbanecm@deploy1003 somerandomdeveloper, urbanecm: Backport for [[gerrit:1358445|Revert "Improve types" (T441875)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:15:53] (03Merged) 10jenkins-bot: thumbor: Make timeout more strict [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358845 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup) [20:15:53] SomeRandomDev: can you test, please [20:15:55] ? [20:16:10] https://strategy.wikimedia.org/wiki/Thread:Category_talk:Proposals_related_to_citations/Possible_goals_of_a_new_citation_proposal loads fine with mwdebug, so looks like it works [20:16:13] it errors without mwdebug [20:16:16] ageeed [20:16:17] !log urbanecm@deploy1003 somerandomdeveloper, urbanecm: Continuing with deployment [20:16:18] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet [20:16:19] proceeding [20:16:58] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to analytics-privatedata-users, level 3, for eniroomand - https://phabricator.wikimedia.org/T440031#12474717 (10Dzahn) @eniroomand Your Kerberos principal has been created. Please look for a... [20:17:46] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:19:10] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to analytics-privatedata-users, level 3, for eniroomand - https://phabricator.wikimedia.org/T440031#12474718 (10Dzahn) 05In progress→03Resolved a:03Dzahn I also merged and deployed... [20:20:59] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [20:21:08] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [20:21:57] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1358445|Revert "Improve types" (T441875)]] (1.48.0-wmf.1) (duration: 10m 30s) [20:22:02] T441875: Error: Call to a member function isTopmostThread() on null - https://phabricator.wikimedia.org/T441875 [20:22:52] urbanecm: thanks for deploying! any chance you could check on logstash whether the errors have fully stopped ocurring? if so I could resolve the task [20:22:59] sure! [20:23:12] SomeRandomDev: you don't have logstash acc? [20:23:17] nope [20:23:56] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [20:24:09] someone should fix that [20:24:16] SomeRandomDev: do you want one? since your security acc went through, it should be a no brainer [20:24:27] ^ [20:24:49] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet [20:24:54] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2205-2215,2242-2243,2248].codfw.wmnet [20:24:58] i wouldn't mind having access, but I don't really have a lot of use cases for checking logstash [20:25:12] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [20:25:22] FIRING: [3x] SwitchCoreInterfaceDown: Switch core interface down - ssw1-a1-eqiad:lag1 (Core: asw2-a-eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [20:25:24] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2249-2262].codfw.wmnet [20:26:24] charts for this error look good :) https://usercontent.irccloud-cdn.com/file/XoH2QvL7/image.png [20:26:43] last at Oct 26, 2026 @ 20:19:36.809 [20:26:46] nice, thx [20:27:00] !log urbanecm@deploy1003 mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97002) [20:27:05] T438366: Run GrowthExperiments:revalidateLinkRecommendations.php on all Wikimedia wikis - https://phabricator.wikimedia.org/T438366 [20:27:42] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [20:28:30] SomeRandomDev: tbh, there's no reason for someone trusted with acl*security to not have logstash. even if it is to check error fixes here and there [20:28:57] feel free to fill it in. i'd happily support. [20:29:25] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose # T438366 [20:30:12] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [20:30:57] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [20:32:28] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:32:42] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:32:50] 10SRE-tools, 06Infrastructure-Foundations: Debmonitor: backend-changeable settings are stored in the browser's session storage - https://phabricator.wikimedia.org/T240457#12474813 (10Lizimedia) here's the fields, and my assessment of them: | field |... [20:32:58] urbanecm: okay, is there a phabricator form for requesting access or what's the process? wikitech only mentions requesting it through IDM but I'm not in any LDAP groups [20:33:56] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2249-2262].codfw.wmnet [20:35:03] SomeRandomDev: you'd need the `logstash-access` group, so filling it within https://idm.wikimedia.org should work. [20:36:04] i'd link the sec ticket request in the comment field (it will make confirming NDA status easier) [20:36:30] okay, I can try that, though I'm not sure if my NDA would cover logstash as well since I think it's specifically for security tasks [20:36:52] mutante: might know if `logstash-access` also needs `nda` in Logstash (and the fact one is a ticket other is IDM is not helping :-/) [20:40:08] (03PS1) 10AKhatun: revision-feature-counts lac-lambda: Add test wikis [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358864 (https://phabricator.wikimedia.org/T442448) [20:41:49] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2249-2262].codfw.wmnet [20:41:53] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2249-2262].codfw.wmnet [20:42:22] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2263-2276].codfw.wmnet [20:43:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:43:35] (03PS2) 10AKhatun: revision-feature-counts lac-lambda: Add test wikis [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358864 (https://phabricator.wikimedia.org/T442448) [20:44:03] urbanecm: might be worth a ping to Eric Mill as he approved and sponsored SRD's access [20:44:39] RhinosF1: if further approval is needed, then possibly :) [20:48:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:50:27] 10SRE-tools, 06Infrastructure-Foundations: Debmonitor: backend-changeable settings are stored in the browser's session storage - https://phabricator.wikimedia.org/T240457#12474889 (10Lizimedia) further testing hasn't actually produced any errors caused by column configuration mismatches. Deleting one of the co... [20:50:51] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2263-2276].codfw.wmnet [20:51:50] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_ulsfo and not P{cp4037.ulsfo.wmnet} and A:cp - 9.2.16 upgrade (T441945) [20:52:33] (03PS1) 10Herron: admin: herron add hw-backed laptop key and remove legacy key [puppet] - 10https://gerrit.wikimedia.org/r/1358865 [20:55:41] (03PS7) 10Bking: matomo: Use an initContainer to copy static assets, customize httpd-fcgi config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) [20:56:46] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_drmrs and A:cp - 9.2.16 upgrade (T441945) [20:56:51] (03PS8) 10Bking: matomo: Use an initContainer to copy static assets, customize httpd-fcgi config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) [20:56:56] (03CR) 10CI reject: [V:04-1] matomo: Use an initContainer to copy static assets, customize httpd-fcgi config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [20:56:57] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_drmrs and A:cp - 9.2.16 upgrade (T441945) [20:58:08] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2263-2276].codfw.wmnet [20:58:13] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2263-2276].codfw.wmnet [20:58:29] (03PS9) 10Bking: matomo: Use an initContainer to copy static assets, customize httpd-fcgi config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358839 (https://phabricator.wikimedia.org/T436003) [20:58:41] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2277-2290].codfw.wmnet [21:00:05] alexsanford, Reedy, sbassett, Maryum, and manfredi: May I have your attention please! Weekly Security deployment window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261026T2100) [21:01:40] Have several sec patches that need to be deployed. Let me know if there’s anything pending, otherwise I’m going to start. [21:04:01] sbassett: nothing from me [21:06:27] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2277-2290].codfw.wmnet [21:07:30] (03PS1) 10Lizimedia: don't restore column settings [software/debmonitor] - 10https://gerrit.wikimedia.org/r/1358866 (https://phabricator.wikimedia.org/T240457) [21:14:52] ACKNOWLEDGEMENT - MD RAID on wikikube-worker2285 is CRITICAL: CRITICAL: State: degraded, Active: 1, Working: 1, Failed: 0, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T442456 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [21:14:59] 10ops-codfw, 06SRE, 06DC-Ops: Degraded RAID on wikikube-worker2285 - https://phabricator.wikimedia.org/T442456 (10ops-monitoring-bot) 03NEW [21:15:48] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2277-2290].codfw.wmnet [21:15:53] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2277-2290].codfw.wmnet [21:16:26] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2291-2304].codfw.wmnet [21:21:06] (03PS1) 10Dzahn: codesearch: skip the hound index dirs in delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1358869 (https://phabricator.wikimedia.org/T441991) [21:22:48] (03CR) 10BCornwall: cache::haproxy: set low prio QoS to all files from upload (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1358150 (https://phabricator.wikimedia.org/T440990) (owner: 10Fabfur) [21:23:01] !log Deployed security fix for T442373 and T441029 to wmf.24 and wmf.1 [21:23:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:23:08] (03PS2) 10Dzahn: codesearch: skip the hound index dirs in delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1358869 (https://phabricator.wikimedia.org/T441991) [21:28:04] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2291-2304].codfw.wmnet [21:28:49] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358870 [21:28:51] (03CR) 10Dzahn: [C:03+2] codesearch: skip the hound index dirs in delete-zombie-locks [puppet] - 10https://gerrit.wikimedia.org/r/1358869 (https://phabricator.wikimedia.org/T441991) (owner: 10Dzahn) [21:31:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=codfw&var-cluster=text&var-origin=eventstreams.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [21:32:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [21:33:10] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_drmrs and A:cp - 9.2.16 upgrade (T441945) [21:35:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:35:11] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [21:35:58] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2291-2304].codfw.wmnet [21:36:03] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2291-2304].codfw.wmnet [21:36:18] !incidents [21:36:19] 8653 (ACKED, 69h 45m old) Manual (paged) by SBassett (sbassett@wikimedia.org): Need to confirm envoy and nginx config are protective in Wikimedia production [21:36:19] 8658 (UNACKED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet codfw) [21:36:33] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2305-2318].codfw.wmnet [21:36:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=codfw&var-cluster=text&var-origin=eventstreams.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [21:37:47] getting 5xx errors from stream.wikimedia.org. started 21:32 UTC [21:37:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [21:38:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wm [21:38:14] ikube-worker1278.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1 [21:38:14] d.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [21:38:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wm [21:38:24] ikube-worker1345.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1007.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1 [21:38:24] d.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [21:40:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:41:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [21:44:49] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2305-2318].codfw.wmnet [21:46:26] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:48:25] (03PS1) 10Dzahn: codesearch: run the hound containers with --init [puppet] - 10https://gerrit.wikimedia.org/r/1358873 (https://phabricator.wikimedia.org/T268199) [21:49:06] PROBLEM - Check if active EventStreams endpoint is delivering messages. on alert1002 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [21:49:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1345.eqiad.wm [21:49:24] ikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1 [21:49:24] d.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [21:49:40] !log Deployed security fix for T442018 to wmf.24 and wmf.1 [21:49:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:51:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [21:52:06] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2305-2318].codfw.wmnet [21:52:10] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2305-2318].codfw.wmnet [21:52:42] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2319-2332].codfw.wmnet [21:53:21] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventstreams.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqiad&var-cluster=text&var-origin=eventstreams.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [21:57:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:57:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:57:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:00:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1380.eqiad.wm [22:00:14] ikube-worker1320.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1136.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1 [22:00:14] d.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:00:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wm [22:00:24] ikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1076.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1296.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1 [22:00:24] d.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1353.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:00:37] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2319-2332].codfw.wmnet [22:02:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:02:23] (03CR) 10Dzahn: "I have temp disabled puppet on codesearch10 to verify this." [puppet] - 10https://gerrit.wikimedia.org/r/1358873 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [22:02:34] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_drmrs and A:cp - 9.2.16 upgrade (T441945) [22:03:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:03:16] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-text_esams and A:cp - 9.2.16 upgrade (T441945) [22:03:22] !log brett@cumin2003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-upload_esams and A:cp - 9.2.16 upgrade (T441945) [22:03:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:06:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1116.eqiad.wm [22:06:14] ikube-worker1007.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1 [22:06:14] d.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:06:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1380.eqiad.wm [22:06:24] ikube-worker1315.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1 [22:06:24] d.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1135.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:07:04] what's with the eqiad wikikube workers alerts? is this related to all the codfw depooling? [22:07:18] just trying to sort out the background noise [22:07:40] jasmine_: ^ ? [22:07:57] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2319-2332].codfw.wmnet [22:08:02] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2319-2332].codfw.wmnet [22:08:31] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2333-2346].codfw.wmnet [22:08:51] !log Deployed security fix for T440638 to wmf.24 and wmf.1 [22:08:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:09:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:11:54] (03PS1) 10Neriah: Allow those with CheckUser right to view private filters on WMF projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) [22:12:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1280.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1049.eqiad.wm [22:12:24] ikube-worker1079.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1135.eqiad.wmnet, wikikube-worker1 [22:12:24] d.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1336.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-worker1142.eqiad.wmnet, wikikube-worker1286.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:13:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:13:25] (03CR) 10Dreamy Jazz: [C:03+1] "We may want to remove the wiki-specific overrides that grant `abusefilter-view-private` to the `checkuser` group (such as English Wikipedi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) (owner: 10Neriah) [22:13:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 27 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) (owner: 10Neriah) [22:15:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:16:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:16:32] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2333-2346].codfw.wmnet [22:16:49] (03CR) 10Neriah: "Done" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) (owner: 10Neriah) [22:17:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [22:17:44] Deployment eventstreams-production in eventstreams at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=eventstreams&var-deployment=eventstreams-production - ... [22:17:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [22:18:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1340.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:18:46] (03PS2) 10Neriah: Allow those with CheckUser right to view private filters on WMF projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) [22:19:08] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [22:19:28] (03CR) 10Dreamy Jazz: [C:03+1] Allow those with CheckUser right to view private filters on WMF projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358880 (https://phabricator.wikimedia.org/T442464) (owner: 10Neriah) [22:21:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:21:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wm [22:21:14] ikube-worker1050.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1 [22:21:14] d.wmnet, wikikube-worker1261.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1147.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:21:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [22:24:07] (03PS1) 10Jdlrobson: ReadingList: Fix instrument for clicking on overflow icon [extensions/WikimediaEvents] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358885 (https://phabricator.wikimedia.org/T441477) [22:24:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:24:22] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2333-2346].codfw.wmnet [22:24:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:24:26] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2333-2346].codfw.wmnet [22:24:57] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2347-2360].codfw.wmnet [22:25:10] (03PS1) 10Jdlrobson: ReadingList: Fix instrument for clicking on overflow icon [extensions/WikimediaEvents] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358886 (https://phabricator.wikimedia.org/T441477) [22:25:22] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [22:26:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [22:29:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1066.eqiad.wmnet, wikikube-worker1286.eqiad.wmnet, wikikube-worker1299.eqiad.wmnet, wikikube-worker1123.eqiad.wmnet, wikikube-worker1361.eqiad.wmnet, wikikube-worker1114.eqiad.wmnet are marked down but pooled ht [22:29:14] kitech.wikimedia.org/wiki/PyBal [22:29:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1291.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1015.eqiad.wm [22:29:24] ikube-worker1309.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-worker1125.eqiad.wmnet, wikikube-worker1048.eqiad.wmnet, wikikube-worker1138.eqiad.wmnet, wikikube-worker1158.eqiad.wmnet, wikikube-worker1381.eqiad.wmnet, wikikube-worker1065.eqiad.wmnet, wikikube-worker1120.eqiad.wmnet, wikikube-worker1321.eqiad.wmnet, wikikube-worker1007.eqiad.wmnet, wikikube-worker1257.eqiad.wmnet, wikikube-worker1122.eqiad.wmnet, wikikube-worker1 [22:29:24] d.wmnet, wikikube-worker1276.eqiad.wmnet, wikikube-worker1113.eqiad.wmnet, wikikube-worker1073.eqiad.wmnet, wikikube-worker1249.eqiad.wmnet, wikikube-worker1123.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:30:06] !incidents [22:30:06] 8653 (ACKED, 70h 39m old) Manual (paged) by SBassett (sbassett@wikimedia.org): Need to confirm envoy and nginx config are protective in Wikimedia production [22:30:07] 8659 (ACKED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet eqiad) [22:30:07] 8658 (RESOLVED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet codfw) [22:31:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:34:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:36:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [22:37:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1315.eqiad.wm [22:37:14] ikube-worker1132.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1070.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-worker1272.eqiad.wmnet, wikikube-worker1 [22:37:14] d.wmnet, wikikube-worker1369.eqiad.wmnet, wikikube-worker1164.eqiad.wmnet, wikikube-worker1154.eqiad.wmnet, wikikube-worker1354.eqiad.wmnet, wikikube-worker1381.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:37:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1121.eqiad.wm [22:37:24] ikube-worker1116.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1274.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1 [22:37:24] d.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:38:03] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-upload_esams and A:cp - 9.2.16 upgrade (T441945) [22:38:27] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2347-2360].codfw.wmnet [22:39:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:40:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:41:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [22:42:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:43:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wm [22:43:14] ikube-worker1384.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1330.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1 [22:43:14] d.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:43:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1051.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1155.eqiad.wm [22:43:24] ikube-worker1278.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1007.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1 [22:43:24] d.wmnet, wikikube-worker1282.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:45:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:45:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:45:40] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2347-2360].codfw.wmnet [22:45:45] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2347-2360].codfw.wmnet [22:46:16] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2361-2374].codfw.wmnet [22:47:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:48:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1347.eqiad.wm [22:48:14] ikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1 [22:48:14] d.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1288.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:48:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1121.eqiad.wm [22:48:24] ikube-worker1116.eqiad.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1053.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1 [22:48:24] d.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:50:06] PROBLEM - Check if active EventStreams endpoint is delivering messages. on alert1002 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [22:50:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:50:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:51:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [22:54:17] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2361-2374].codfw.wmnet [22:54:36] I'm going to use the readers window in 6mins if no objections [22:56:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [22:59:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wm [22:59:14] ikube-worker1281.eqiad.wmnet, wikikube-worker1274.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1263.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1 [22:59:14] d.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [22:59:26] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1291.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1320.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1016.eqiad.wm [22:59:26] ikube-worker1260.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1135.eqiad.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1336.eqiad.wmnet, wikikube-worker1305.eqiad.wmnet, wikikube-worker1076.eqiad.wmnet, wikikube-worker1253.eqiad.wmnet, wikikube-worker1342.eqiad.wmnet, wikikube-worker1040.eqiad.wmnet, wikikube-worker1154.eqiad.wmnet, wikikube-worker1 [22:59:26] d.wmnet, wikikube-worker1381.eqiad.wmnet, wikikube-worker1240.eqiad.wmnet, wikikube-worker1153.eqiad.wmnet, wikikube-worker1346.eqiad.wmnet, wikikube-worker1321.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261026T2300) [23:00:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358885 (https://phabricator.wikimedia.org/T441477) (owner: 10Jdlrobson) [23:00:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358847 (https://phabricator.wikimedia.org/T441244) (owner: 10Jdlrobson) [23:00:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358846 (https://phabricator.wikimedia.org/T441244) (owner: 10Jdlrobson) [23:00:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358787 (https://phabricator.wikimedia.org/T429288) (owner: 10Jdlrobson) [23:00:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358786 (https://phabricator.wikimedia.org/T429288) (owner: 10Jdlrobson) [23:00:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358886 (https://phabricator.wikimedia.org/T441477) (owner: 10Jdlrobson) [23:01:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:01:22] !log Deployed security fixes for T440493 and T441721 to wmf.24 and wmf.1 [23:01:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:01:26] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:01:31] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2361-2374].codfw.wmnet [23:01:36] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2361-2374].codfw.wmnet [23:01:41] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp2001.codfw.wmnet [23:01:55] (03Merged) 10jenkins-bot: ReadingList: Fix instrument for clicking on overflow icon [extensions/WikimediaEvents] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358885 (https://phabricator.wikimedia.org/T441477) (owner: 10Jdlrobson) [23:02:13] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp2001.codfw.wmnet [23:03:37] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-text_esams and A:cp - 9.2.16 upgrade (T441945) [23:03:41] (03Merged) 10jenkins-bot: Fix recentlyDonated captures non-donors [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358847 (https://phabricator.wikimedia.org/T441244) (owner: 10Jdlrobson) [23:03:43] (03Merged) 10jenkins-bot: Fix recentlyDonated captures non-donors [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358846 (https://phabricator.wikimedia.org/T441244) (owner: 10Jdlrobson) [23:03:46] (03Merged) 10jenkins-bot: Update account creation instrumentation mechanism [extensions/WikimediaEvents] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358787 (https://phabricator.wikimedia.org/T429288) (owner: 10Jdlrobson) [23:03:48] (03Merged) 10jenkins-bot: Update account creation instrumentation mechanism [extensions/WikimediaEvents] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1358786 (https://phabricator.wikimedia.org/T429288) (owner: 10Jdlrobson) [23:03:49] (03Merged) 10jenkins-bot: ReadingList: Fix instrument for clicking on overflow icon [extensions/WikimediaEvents] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358886 (https://phabricator.wikimedia.org/T441477) (owner: 10Jdlrobson) [23:04:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1279.eqiad.wm [23:04:14] ikube-worker1338.eqiad.wmnet, wikikube-worker1056.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1289.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1147.eqiad.wmnet, wikikube-worker1272.eqiad.wmnet, wikikube-worker1142.eqiad.wmnet, wikikube-worker1 [23:04:14] d.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1051.eqiad.wmnet, wikikube-worker1055.eqiad.wmnet, wikikube-worker1154.eqiad.wmnet, wikikube-worker1066.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:04:22] !log jdlrobson@deploy1003 Started scap sync-world: Backport for ReadingList: Fix instrument for clicking on overflow icon (T441477) ([[gerrit:1358885|1.47.0-wmf.24]], [[gerrit:1358886|1.48.0-wmf.1]]), Fix recentlyDonated captures non-donors (T441244) ([[gerrit:1358847|1.48.0-wmf.1]], [[gerrit:1358846|1.47.0-wmf.24]]), Update account creation instrumentation mechanism (T429288) ([[gerrit:1358787|1.48.0-wmf.1]], [[gerrit:13 [23:04:23] 58786|1.47.0-wmf.24]]) [23:04:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1322.eqiad.wmnet, wikikube-worker1079.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1244.eqiad.wm [23:04:24] ikube-worker1352.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1336.eqiad.wmnet, wikikube-worker1143.eqiad.wmnet, wikikube-worker1305.eqiad.wmnet, wikikube-worker1057.eqiad.wmnet, wikikube-worker1081.eqiad.wmnet, wikikube-worker1164.eqiad.wmnet, wikikube-worker1140.eqiad.wmnet, wikikube-worker1087.eqiad.wmnet, wikikube-worker1154.eqiad.wmnet, wikikube-worker1066.eqiad.wmnet, wikikube-worker1 [23:04:24] d.wmnet, wikikube-worker1312.eqiad.wmnet, wikikube-worker1240.eqiad.wmnet, wikikube-worker1117.eqiad.wmnet, wikikube-worker1271.eqiad.wmnet, wikikube-worker1359.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:04:31] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [23:04:31] T441244: recentlyDonated captures non-donors - https://phabricator.wikimedia.org/T441244 [23:04:32] T429288: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288 [23:05:30] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Gracefully handle Phabricator being inaccessible during Spicerack calls - https://phabricator.wikimedia.org/T442374#12475337 (10Volans) Phabricator support in Spicerack is a mere proxy to what's supported in wmflib. In v2.0.0 (2025-06-17) of wmflib it w... [23:05:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [23:06:06] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp2001.codfw.wmnet [23:06:08] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp2001.codfw.wmnet [23:06:09] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-codfw [23:06:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:07:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:07:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:09:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1132.eqiad.wm [23:09:24] ikube-worker1072.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1044.eqiad.wmnet, wikikube-worker1286.eqiad.wmnet, wikikube-worker1267.eqiad.wmnet, wikikube-worker1241.eqiad.wmnet, wikikube-worker1299.eqiad.wmnet, wikikube-worker1150.eqiad.wmnet, wikikube-worker1 [23:09:24] d.wmnet, wikikube-worker1147.eqiad.wmnet, wikikube-worker1292.eqiad.wmnet, wikikube-worker1156.eqiad.wmnet, wikikube-worker1321.eqiad.wmnet, wikikube-worker1067.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:09:32] !log jdlrobson@deploy1003 jdlrobson: Backport for ReadingList: Fix instrument for clicking on overflow icon (T441477) ([[gerrit:1358885|1.47.0-wmf.24]], [[gerrit:1358886|1.48.0-wmf.1]]), Fix recentlyDonated captures non-donors (T441244) ([[gerrit:1358847|1.48.0-wmf.1]], [[gerrit:1358846|1.47.0-wmf.24]]), Update account creation instrumentation mechanism (T429288) ([[gerrit:1358787|1.48.0-wmf.1]], [[gerrit:1358786|1.47.0-w [23:09:32] mf.24]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:09:42] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [23:09:42] T441244: recentlyDonated captures non-donors - https://phabricator.wikimedia.org/T441244 [23:09:43] T429288: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288 [23:10:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1280.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1291.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1259.eqiad.wm [23:10:14] ikube-worker1155.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1334.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1148.eqiad.wmnet, wikikube-worker1364.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1016.eqiad.wmnet, wikikube-worker1273.eqiad.wmnet, wikikube-worker1 [23:10:14] d.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1157.eqiad.wmnet, wikikube-worker1358.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:10:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [23:11:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:11:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:11:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:11:29] (03CR) 10Dzahn: [C:03+2] codesearch: run the hound containers with --init [puppet] - 10https://gerrit.wikimedia.org/r/1358873 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:12:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:13:36] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [23:14:41] (03PS1) 10Urbanecm: services_proxy:linkrecommendation: Increase timeout to 31s [puppet] - 10https://gerrit.wikimedia.org/r/1358890 (https://phabricator.wikimedia.org/T442468) [23:15:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1144.eqiad.wmnet, wikikube-worker1042.eqiad.wmnet, wikikube-worker1383.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1260.eqiad.wmnet, wikikube-worker1263.eqiad.wm [23:15:14] ikube-worker1307.eqiad.wmnet, wikikube-worker1313.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1037.eqiad.wmnet, wikikube-worker1160.eqiad.wmnet, wikikube-worker1135.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1272.eqiad.wmnet, wikikube-worker1305.eqiad.wmnet, wikikube-worker1081.eqiad.wmnet, wikikube-worker1 [23:15:14] d.wmnet, wikikube-worker1267.eqiad.wmnet, wikikube-worker1035.eqiad.wmnet, wikikube-worker1241.eqiad.wmnet, wikikube-worker1342.eqiad.wmnet, wikikube-worker1329.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:16:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [23:18:32] Jdlrobson: just fyi, if you’re doing another sync-world after this, two security patches will be going out with it :) [23:18:52] (two new security patches) [23:19:25] sbassett: hmm? I'm almost done [23:19:52] you want to use the rest of the window? [23:20:06] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [23:21:17] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for ReadingList: Fix instrument for clicking on overflow icon (T441477) ([[gerrit:1358885|1.47.0-wmf.24]], [[gerrit:1358886|1.48.0-wmf.1]]), Fix recentlyDonated captures non-donors (T441244) ([[gerrit:1358847|1.48.0-wmf.1]], [[gerrit:1358846|1.47.0-wmf.24]]), Update account creation instrumentation mechanism (T429288) ([[gerrit:1358787|1.48.0-wmf.1]], [[gerrit:1 [23:21:17] 358786|1.47.0-wmf.24]]) (duration: 16m 54s) [23:21:24] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [23:21:24] T441244: recentlyDonated captures non-donors - https://phabricator.wikimedia.org/T441244 [23:21:25] T429288: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288 [23:21:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [23:22:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:22:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [23:24:06] sbassett: im all done. [23:25:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:25:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1079.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1007.eqiad.wmnet, wikikube-worker1094.eqiad.wmnet, wikikube-worker1071.eqiad.wmnet, wikikube-worker1161.eqiad.wmnet, wikikube-worker1149.eqiad.wmnet, wikikube-worker1365.eqiad.wm [23:25:24] ikube-worker1135.eqiad.wmnet, wikikube-worker1309.eqiad.wmnet, wikikube-worker1044.eqiad.wmnet, wikikube-worker1305.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1253.eqiad.wmnet, wikikube-worker1035.eqiad.wmnet, wikikube-worker1312.eqiad.wmnet, wikikube-worker1040.eqiad.wmnet, wikikube-worker1154.eqiad.wmnet, wikikube-worker1354.eqiad.wmnet, wikikube-worker1158.eqiad.wmnet, wikikube-worker1381.eqiad.wmnet, wikikube-worker1 [23:25:24] d.wmnet, wikikube-worker1120.eqiad.wmnet, wikikube-worker1346.eqiad.wmnet, wikikube-worker1316.eqiad.wmnet, wikikube-worker1303.eqiad.wmnet, wikikube-worker1274.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:26:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1051.eqiad.wmnet, wikikube-worker1280.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1384.eqiad.wmnet, wikikube-worker1121.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1049.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1016.eqiad.wm [23:26:14] ikube-worker1071.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1270.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1365.eqiad.wmnet, wikikube-worker1278.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1344.eqiad.wmnet, wikikube-worker1272.eqiad.wmnet, wikikube-worker1044.eqiad.wmnet, wikikube-worker1057.eqiad.wmnet, wikikube-worker1 [23:26:14] d.wmnet, wikikube-worker1050.eqiad.wmnet, wikikube-worker1035.eqiad.wmnet, wikikube-worker1241.eqiad.wmnet, wikikube-worker1158.eqiad.wmnet, wikikube-worker1087.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:26:28] (03CR) 10Volans: "Post-merge comments" [cookbooks] - 10https://gerrit.wikimedia.org/r/1358763 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [23:27:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [23:30:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:30:33] (03PS1) 10Urbanecm: GrowthExperiments: Set link recommendation service timeout to 32s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358891 (https://phabricator.wikimedia.org/T442468) [23:31:03] (03PS2) 10Urbanecm: GrowthExperiments: Set link recommendation service timeout to 32s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358891 (https://phabricator.wikimedia.org/T442468) [23:31:24] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:38:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:42:39] (03PS1) 10Dreamy Jazz: WikimediaAntiAbuse: Promote vandalism queue out of alpha in enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358893 (https://phabricator.wikimedia.org/T442379) [23:51:06] PROBLEM - Check if active EventStreams endpoint is delivering messages. on alert1002 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [23:52:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1144.eqiad.wmnet, wikikube-worker1281.eqiad.wmnet, wikikube-worker1007.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1247.eqiad.wmnet, wikikube-worker1279.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1015.eqiad.wmnet, wikikube-worker1340.eqiad.wm [23:52:14] ikube-worker1119.eqiad.wmnet, wikikube-worker1081.eqiad.wmnet, wikikube-worker1337.eqiad.wmnet, wikikube-worker1267.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1329.eqiad.wmnet, wikikube-worker1087.eqiad.wmnet, wikikube-worker1354.eqiad.wmnet, wikikube-worker1293.eqiad.wmnet, wikikube-worker1074.eqiad.wmnet, wikikube-worker1065.eqiad.wmnet, wikikube-worker1370.eqiad.wmnet, wikikube-worker1078.eqiad.wmnet, wikikube-worker1 [23:52:14] d.wmnet, wikikube-worker1319.eqiad.wmnet, wikikube-worker1124.eqiad.wmnet, wikikube-worker1156.eqiad.wmnet, wikikube-worker1359.eqiad.wmnet, wikikube-worker1321.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:55:14] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:58:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1042.eqiad.wmnet, wikikube-worker1268.eqiad.wmnet, wikikube-worker1118.eqiad.wmnet, wikikube-worker1259.eqiad.wmnet, wikikube-worker1347.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1380.eqiad.wmnet, wikikube-worker1371.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1334.eqiad.wm [23:58:14] ikube-worker1260.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1244.eqiad.wmnet, wikikube-worker1340.eqiad.wmnet, wikikube-worker1377.eqiad.wmnet, wikikube-worker1119.eqiad.wmnet, wikikube-worker1135.eqiad.wmnet, wikikube-worker1333.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1130.eqiad.wmnet, wikikube-worker1267.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1253.eqiad.wmnet, wikikube-worker1 [23:58:14] d.wmnet, wikikube-worker1087.eqiad.wmnet, wikikube-worker1357.eqiad.wmnet, wikikube-worker1308.eqiad.wmnet, wikikube-worker1271.eqiad.wmnet, wikikube-worker1165.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:58:24] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - eventstreams_4892: Servers wikikube-worker1268.eqiad.wmnet, wikikube-worker1304.eqiad.wmnet, wikikube-worker1306.eqiad.wmnet, wikikube-worker1155.eqiad.wmnet, wikikube-worker1345.eqiad.wmnet, wikikube-worker1116.eqiad.wmnet, wikikube-worker1036.eqiad.wmnet, wikikube-worker1315.eqiad.wmnet, wikikube-worker1132.eqiad.wmnet, wikikube-worker1136.eqiad.wm [23:58:24] ikube-worker1358.eqiad.wmnet, wikikube-worker1072.eqiad.wmnet, wikikube-worker1338.eqiad.wmnet, wikikube-worker1307.eqiad.wmnet, wikikube-worker1159.eqiad.wmnet, wikikube-worker1287.eqiad.wmnet, wikikube-worker1162.eqiad.wmnet, wikikube-worker1360.eqiad.wmnet, wikikube-worker1272.eqiad.wmnet, wikikube-worker1044.eqiad.wmnet, wikikube-worker1142.eqiad.wmnet, wikikube-worker1127.eqiad.wmnet, wikikube-worker1373.eqiad.wmnet, wikikube-worker1 [23:58:24] d.wmnet, wikikube-worker1312.eqiad.wmnet, wikikube-worker1292.eqiad.wmnet, wikikube-worker1124.eqiad.wmnet, wikikube-worker1156.eqiad.wmnet, wikikube-worker1275.eqiad.wmnet, wikikube-wo https://wikitech.wikimedia.org/wiki/PyBal [23:58:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://eventstreams.svc.eqiad.wmnet:4892 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures