[18:36:06] test wm-bot logging [18:42:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:44:07] (03CR) 10Krinkle: Redesign 503 pages (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356470 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup) [18:47:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:49:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:50:05] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 66%, RTA = 5377.08 ms [18:50:31] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - thanos-web_443: Servers titan1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:51:53] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [18:52:11] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - thanos-web_443: Servers titan1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:52:31] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:53:09] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:54:03] RESOLVED: [3x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:54:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:59:30] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [18:59:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:05:46] (03PS1) 10Snwachukwu: EventStreamConfig: Declare the mediawiki.user_thank.v1 stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356976 (https://phabricator.wikimedia.org/T441426) [19:06:33] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:08:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:13:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:18:24] 10SRE-Access-Requests: Prod Access Key - https://phabricator.wikimedia.org/T441724 (10Mstyles) 03NEW [19:18:47] FIRING: [3x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 7d 17h 47m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:18:59] (03CR) 10Hashar: [C:03+2] Point Zuul plugin to stable-3.10 and update it [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1356923 (https://phabricator.wikimedia.org/T441684) (owner: 10Hashar) [19:19:02] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [19:19:39] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [19:19:40] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [19:20:23] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [19:20:54] 10SRE-Access-Requests: Add FIDO-backed SSH keys for mstyles - https://phabricator.wikimedia.org/T441724#12451046 (10Mstyles) [19:28:03] (03Merged) 10jenkins-bot: Point Zuul plugin to stable-3.10 and update it [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1356923 (https://phabricator.wikimedia.org/T441684) (owner: 10Hashar) [19:30:04] (03PS2) 10Dzahn: admin: add brennen to codesearch-roots [puppet] - 10https://gerrit.wikimedia.org/r/1355898 (https://phabricator.wikimedia.org/T441371) [19:32:03] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for brennen - https://phabricator.wikimedia.org/T441371#12451142 (10Dzahn) @LSobanski I see your approval on Gerrit, thanks! Can you leave it here too? @thcipriani please approve as Brennen's manager :) [19:32:14] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for brennen - https://phabricator.wikimedia.org/T441371#12451144 (10Dzahn) a:03thcipriani [19:32:38] (03CR) 10Dzahn: "sorry, wrong ticket was linked. fixed now. gtg" [puppet] - 10https://gerrit.wikimedia.org/r/1355898 (https://phabricator.wikimedia.org/T441371) (owner: 10Dzahn) [19:33:12] (03CR) 10Dzahn: "just needs approvals on ticket and approval from Tyler" [puppet] - 10https://gerrit.wikimedia.org/r/1355898 (https://phabricator.wikimedia.org/T441371) (owner: 10Dzahn) [19:33:57] (03CR) 10Dzahn: "@mmuhlenhoff@wikimedia.org this one has all approvals, ok to merge?" [puppet] - 10https://gerrit.wikimedia.org/r/1355898 (https://phabricator.wikimedia.org/T441371) (owner: 10Dzahn) [19:35:44] (03CR) 10Dzahn: [C:03+1] "@mmuhlenhoff@wikimedia.org @slyngshede this has all approvals, good to merge?" [puppet] - 10https://gerrit.wikimedia.org/r/1355775 (https://phabricator.wikimedia.org/T441068) (owner: 10Dzahn) [19:36:38] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for krinkle - https://phabricator.wikimedia.org/T441068#12451186 (10Dzahn) 05Open→03In progress [19:36:48] (03PS1) 10Mstyles: admin: Add FIDO SSH keys for mstyles [puppet] - 10https://gerrit.wikimedia.org/r/1356982 (https://phabricator.wikimedia.org/T441724) [19:37:50] (03CR) 10CI reject: [V:04-1] admin: Add FIDO SSH keys for mstyles [puppet] - 10https://gerrit.wikimedia.org/r/1356982 (https://phabricator.wikimedia.org/T441724) (owner: 10Mstyles) [19:38:23] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Remaining access pieces for Eva to use Jupyter - https://phabricator.wikimedia.org/T440031#12451200 (10Dzahn) [19:38:47] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:39:08] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Remaining access pieces for Eva to use Jupyter - https://phabricator.wikimedia.org/T440031#12451206 (10Dzahn) Tagging Data-Engineering to increase visibility per https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#analytics-privatedata-users... [19:39:42] (03PS1) 10Dwisehaupt: fr-tech: shift fundraising read database to writer [dns] - 10https://gerrit.wikimedia.org/r/1356984 (https://phabricator.wikimedia.org/T419556) [19:42:51] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Remaining access pieces for Eva to use Jupyter - https://phabricator.wikimedia.org/T440031#12451235 (10Dzahn) a:05eniroomand→03None [19:42:55] (03CR) 10Jgreen: [C:03+1] fr-tech: shift fundraising read database to writer [dns] - 10https://gerrit.wikimedia.org/r/1356984 (https://phabricator.wikimedia.org/T419556) (owner: 10Dwisehaupt) [19:45:07] (03CR) 10Dzahn: [V:04-1] "minor issue here with the CI downvote. it's a trailing whitespace. 12:37:33 FAIL:2841 trailing spaces" [puppet] - 10https://gerrit.wikimedia.org/r/1356982 (https://phabricator.wikimedia.org/T441724) (owner: 10Mstyles) [19:47:44] (03PS2) 10Dzahn: admin: Add FIDO SSH keys for mstyles [puppet] - 10https://gerrit.wikimedia.org/r/1356982 (https://phabricator.wikimedia.org/T441724) (owner: 10Mstyles) [19:48:45] (03CR) 10Dzahn: "just fixed white space to make CI pass. unrelated to needed approvals." [puppet] - 10https://gerrit.wikimedia.org/r/1356982 (https://phabricator.wikimedia.org/T441724) (owner: 10Mstyles) [19:50:02] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mstyles - https://phabricator.wikimedia.org/T441724#12451284 (10Dzahn) 05Open→03In progress [19:52:16] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mszwarc - https://phabricator.wikimedia.org/T441400#12451288 (10Dzahn) 05Open→03In progress @mszwarc looks like it's back to you to verify the new keys work and check off that box ;) [19:57:58] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users, lvl 1 for OBenhmida - https://phabricator.wikimedia.org/T440464#12451311 (10Dzahn) The user has already been created back in June in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1303499 All there is left to do now is to ad... [19:58:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:00:04] RoanKattouw, urbanecm, TheresNoTime, and cjming: OwO what's this, a deployment window?? UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261019T2000). nyaa~ [20:00:04] Krinkle and arlolra: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:01:57] o/ [20:02:02] o/ [20:02:11] I did mine already, go ahead arlolra [20:02:20] ok, thanks [20:03:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [extensions/Linter] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356949 (https://phabricator.wikimedia.org/T441307) (owner: 10Arlolra) [20:03:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:05:26] (03Merged) 10jenkins-bot: Ignore lints when parsing non-lintable content models [extensions/Linter] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356949 (https://phabricator.wikimedia.org/T441307) (owner: 10Arlolra) [20:05:40] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1356949|Ignore lints when parsing non-lintable content models (T441307)]] (1.47.0-wmf.24) [20:05:44] T441307: div tag in HTML comment on CSS page causes false positive Linter error - https://phabricator.wikimedia.org/T441307 [20:06:46] (03PS1) 10Dzahn: admin: add obenhmida to analytics-privatedata-users, lvl1 [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) [20:07:49] (03CR) 10CI reject: [V:04-1] admin: add obenhmida to analytics-privatedata-users, lvl1 [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) (owner: 10Dzahn) [20:08:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:09:11] (03PS1) 10Bking: WIP: Matomo: Support `trusted_hosts` array variable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356987 (https://phabricator.wikimedia.org/T436186) [20:09:42] !log arlolra@deploy1003 arlolra: Backport for [[gerrit:1356949|Ignore lints when parsing non-lintable content models (T441307)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:10:19] !log arlolra@deploy1003 arlolra: Continuing with deployment [20:12:14] (03PS2) 10Bking: WIP: Matomo: Support `trusted_hosts` array variable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356987 (https://phabricator.wikimedia.org/T436186) [20:13:35] (03PS1) 10Krinkle: OutputPageHooks: Avoid DB query for talk page exists() on every pageview [extensions/PageAssessments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356988 (https://phabricator.wikimedia.org/T347123) [20:13:47] FIRING: [7x] GnmiInterfaceCountersDrop: asw1-b3-magru is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [20:13:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:13:56] (03CR) 10CI reject: [V:04-1] WIP: Matomo: Support `trusted_hosts` array variable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356987 (https://phabricator.wikimedia.org/T436186) (owner: 10Bking) [20:14:31] (03PS2) 10Dzahn: admin: add obenhmida to analytics-privatedata-users, lvl1 [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) [20:15:34] (03CR) 10CI reject: [V:04-1] admin: add obenhmida to analytics-privatedata-users, lvl1 [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) (owner: 10Dzahn) [20:15:39] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1356949|Ignore lints when parsing non-lintable content models (T441307)]] (1.47.0-wmf.24) (duration: 09m 59s) [20:15:44] T441307: div tag in HTML comment on CSS page causes false positive Linter error - https://phabricator.wikimedia.org/T441307 [20:16:21] (03PS3) 10Dzahn: admin: add obenhmida to analytics-privatedata-users, lvl1 [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) [20:16:35] Done with my deploy, if anyone else needs to go. Otherwise, I think we're done with the window [20:17:15] arlolra_: ok, I've got another one now :) [20:17:25] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:17:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/PageAssessments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356988 (https://phabricator.wikimedia.org/T347123) (owner: 10Krinkle) [20:17:44] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:18:40] (03CR) 10Dzahn: [C:03+1] "ticket has approval. can have a review / sanity check on this?" [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) (owner: 10Dzahn) [20:18:47] (03Merged) 10jenkins-bot: OutputPageHooks: Avoid DB query for talk page exists() on every pageview [extensions/PageAssessments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356988 (https://phabricator.wikimedia.org/T347123) (owner: 10Krinkle) [20:19:33] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users, lvl 1 for OBenhmida - https://phabricator.wikimedia.org/T440464#12451394 (10Dzahn) 05Open→03In progress [20:20:03] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users, lvl 1 for OBenhmida - https://phabricator.wikimedia.org/T440464#12451398 (10Dzahn) User needs to be converted from "ldap_only admin" to "shell user without shell" :) patch in review! [20:22:02] > fatal: unable to access 'https://gitlab.wikimedia.org/repos/releng/release.git/': The requested URL returned error: 500 [20:22:03] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1356988|OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123)]] (1.47.0-wmf.24) [20:22:07] aborting and re-trying [20:22:07] T347123: Reduce database queries on parsercached logged-out page views (Sep 2023) - https://phabricator.wikimedia.org/T347123 [20:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:24:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:26:06] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1356988|OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:27:46] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:28:07] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:28:50] !log krinkle@deploy1003 krinkle: Continuing with deployment [20:29:06] krinkle: T441247. :-( Maybe it will be better tomorrow [20:34:10] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1356988|OutputPageHooks: Avoid DB query for talk page exists() on every pageview (T347123)]] (1.47.0-wmf.24) (duration: 12m 07s) [20:34:15] T347123: Reduce database queries on parsercached logged-out page views (Sep 2023) - https://phabricator.wikimedia.org/T347123 [20:35:32] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1001.eqiad.wmnet [20:37:23] (03PS1) 10Dillon: PersonalDashboard: Clean up config and set QA values for test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) [20:37:43] !log btullis@cumin1004 START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema [20:39:29] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1001.eqiad.wmnet [20:40:00] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1002.eqiad.wmnet [20:43:08] dancy: ack [20:43:14] (03PS1) 10Dzahn: gerrit: ensure future gerrit is NOT generating host keys [puppet] - 10https://gerrit.wikimedia.org/r/1356992 (https://phabricator.wikimedia.org/T441113) [20:43:50] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:43:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1002.eqiad.wmnet [20:44:11] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:45:08] (03CR) 10Dwisehaupt: [C:03+2] fr-tech: shift fundraising read database to writer [dns] - 10https://gerrit.wikimedia.org/r/1356984 (https://phabricator.wikimedia.org/T419556) (owner: 10Dwisehaupt) [20:45:19] !log dwisehaupt@dns1005 START - running authdns-update [20:47:40] !log dwisehaupt@dns1005 END - running authdns-update [20:47:44] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd1003.eqiad.wmnet [20:48:54] (03CR) 10Lerickson: "Ooh.... is this backwards-compatible with an old index, or do we need to update the index and the qlever server version simultaneously?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1355527 (owner: 10Trueg) [20:49:21] !log dancy@deploy1003 Installing scap version "4.301.0" for 3 host(s) [20:51:00] (03CR) 10Lerickson: Point both deployments to `wdqs-update-pipeline`. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356516 (https://phabricator.wikimedia.org/T441491) (owner: 10Lerickson) [20:51:16] !log dancy@deploy1003 Installation of scap version "4.301.0" completed for 3 hosts [20:51:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd1003.eqiad.wmnet [20:52:53] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2001.codfw.wmnet [20:54:23] !log btullis@cumin1004 END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema [20:54:39] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards [20:54:43] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:55:05] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards [20:55:41] (killed first run due to not being in tmux, but accidentally hit ctrl+c twice so it didnt output the fail message; there's only one run ongoing) [20:56:42] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2001.codfw.wmnet [20:56:51] RECOVERY - Host an-worker1154 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [20:56:56] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2002.codfw.wmnet [20:58:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [20:58:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:58:55] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:58:58] FIRING: NELHigh: Elevated Network Error Logging events (tcp.timed_out) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELHigh [20:59:13] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet [20:59:13] sirenbot: !ack [21:00:05] alexsanford, Reedy, sbassett, Maryum, and manfredi: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Weekly Security deployment window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261019T2100). [21:00:22] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards [21:00:25] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [21:00:29] !log bking@cumin2003 END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards [21:00:57] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2002.codfw.wmnet [21:01:00] (03PS1) 10Jdlrobson: Add Storage::clearCache() to allow clearing the cache per user [extensions/GlobalPreferences] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356993 (https://phabricator.wikimedia.org/T427942) [21:01:29] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 6.315 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [21:02:21] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-etcd2003.codfw.wmnet [21:02:50] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, restore commons categories lost due to the empty 20261017 weekly dump) xfer categories from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling both afterwards [21:03:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:03:58] RESOLVED: NELHigh: Elevated Network Error Logging events (tcp.timed_out) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELHigh [21:04:01] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet [21:04:31] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl2001.codfw.wmnet [21:06:22] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-etcd2003.codfw.wmnet [21:08:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [21:08:55] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet [21:09:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:09:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl2001.codfw.wmnet [21:09:42] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet [21:11:06] preparing for security deploy [21:11:31] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 7.729 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [21:12:42] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet [21:12:45] (03PS1) 10Dwisehaupt: fr-tech: shift fundraising read database back to frdb1008 [dns] - 10https://gerrit.wikimedia.org/r/1356995 (https://phabricator.wikimedia.org/T419556) [21:13:12] (03CR) 10Dzahn: [C:03+2] codesearch: move envoy from port 443 to 7443 [puppet] - 10https://gerrit.wikimedia.org/r/1356533 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [21:13:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:13:55] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:14:08] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:14:18] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet [21:14:25] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:14:26] (03CR) 10Jgreen: [C:03+1] fr-tech: shift fundraising read database back to frdb1008 [dns] - 10https://gerrit.wikimedia.org/r/1356995 (https://phabricator.wikimedia.org/T419556) (owner: 10Dwisehaupt) [21:14:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet [21:18:05] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet [21:18:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:19:11] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:20:44] FIRING: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnsta [21:23:41] (03CR) 10Dwisehaupt: [C:03+2] fr-tech: shift fundraising read database back to frdb1008 [dns] - 10https://gerrit.wikimedia.org/r/1356995 (https://phabricator.wikimedia.org/T419556) (owner: 10Dwisehaupt) [21:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:23:52] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl2002.codfw.wmnet [21:23:59] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet [21:24:01] !log dwisehaupt@dns1005 START - running authdns-update [21:25:44] RESOLVED: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUns [21:26:22] !log dwisehaupt@dns1005 END - running authdns-update [21:27:42] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet [21:28:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:30:33] (03CR) 10Ryan Kemper: "catching up on these" [puppet] - 10https://gerrit.wikimedia.org/r/1355380 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [21:30:45] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl2002.codfw.wmnet [21:31:27] !log Deployed security fix for T441248 [21:31:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:32:25] (03PS1) 10Btullis: deployment_server: Restrict airflow kubeconfigs to team groups [puppet] - 10https://gerrit.wikimedia.org/r/1356996 (https://phabricator.wikimedia.org/T429558) [21:32:26] (03PS1) 10Btullis: admin: Make airflow-deployers a managed list [puppet] - 10https://gerrit.wikimedia.org/r/1356997 (https://phabricator.wikimedia.org/T429558) [21:32:43] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet [21:33:44] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet [21:34:11] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:35:28] (03PS1) 10Jasmine: NEL alerts: exclude HeadlessChrome [puppet] - 10https://gerrit.wikimedia.org/r/1357001 (https://phabricator.wikimedia.org/T374563) [21:36:27] (03CR) 10CI reject: [V:04-1] NEL alerts: exclude HeadlessChrome [puppet] - 10https://gerrit.wikimedia.org/r/1357001 (https://phabricator.wikimedia.org/T374563) (owner: 10Jasmine) [21:36:33] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet [21:36:46] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet [21:38:55] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:39:39] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet [21:40:04] (03PS2) 10Btullis: deployment_server: Restrict airflow kubeconfigs to team groups [puppet] - 10https://gerrit.wikimedia.org/r/1356996 (https://phabricator.wikimedia.org/T429558) [21:40:04] (03PS2) 10Btullis: admin: Make airflow-deployers a managed list [puppet] - 10https://gerrit.wikimedia.org/r/1356997 (https://phabricator.wikimedia.org/T429558) [21:40:15] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1356997 (https://phabricator.wikimedia.org/T429558) (owner: 10Btullis) [21:40:18] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1356996 (https://phabricator.wikimedia.org/T429558) (owner: 10Btullis) [21:40:36] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet [21:43:55] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:43:58] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet [21:45:55] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:47:47] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet [21:50:29] !log Deployed security fix for T440498 [21:50:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:50:55] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:59:03] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Data Engineering topics [puppet] - 10https://gerrit.wikimedia.org/r/1355378 (https://phabricator.wikimedia.org/T435284) [21:59:03] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Search topics [puppet] - 10https://gerrit.wikimedia.org/r/1355379 (https://phabricator.wikimedia.org/T435284) [21:59:03] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Wikidata Platform topics [puppet] - 10https://gerrit.wikimedia.org/r/1355380 (https://phabricator.wikimedia.org/T435284) [21:59:04] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Experiment Platform topics [puppet] - 10https://gerrit.wikimedia.org/r/1355381 (https://phabricator.wikimedia.org/T435284) [21:59:05] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Reader Experience topics [puppet] - 10https://gerrit.wikimedia.org/r/1355382 (https://phabricator.wikimedia.org/T435284) [21:59:06] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Editing topics [puppet] - 10https://gerrit.wikimedia.org/r/1355383 (https://phabricator.wikimedia.org/T435284) [21:59:10] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Infra Foundations topics [puppet] - 10https://gerrit.wikimedia.org/r/1355384 (https://phabricator.wikimedia.org/T435284) [21:59:14] (03PS3) 10Ryan Kemper: kafka-jumbo: declare Observability topics [puppet] - 10https://gerrit.wikimedia.org/r/1355385 (https://phabricator.wikimedia.org/T435284) [22:00:56] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:03:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:05:55] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:08:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:08:55] !log Deployed security fix for T440946 [22:08:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:13:47] FIRING: [14x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [22:13:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:15:56] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:18:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:24:31] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:27:37] !log Deployed security fix for T441632 [22:27:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:28:47] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:28:52] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:30:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:34:25] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 2.227 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:39:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:40:08] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for brennen - https://phabricator.wikimedia.org/T441371#12451982 (10brennen) [22:40:27] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 2.862 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:40:56] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:41:25] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:42:50] !log Deployed security fix for T441639 [22:42:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:43:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:46:11] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:49:31] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 7.704 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:51:11] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:53:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:54:27] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 4.087 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [22:56:11] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:56:28] (03PS3) 10Dzahn: trafficserver: add maps for codesearch [puppet] - 10https://gerrit.wikimedia.org/r/1342804 (https://phabricator.wikimedia.org/T268199) [22:58:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261019T2300) [23:00:10] (03CR) 10Dzahn: "envoy 7443 is live and search through envoy was tested on codesearch1001" [puppet] - 10https://gerrit.wikimedia.org/r/1342804 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:00:29] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 5.309 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:01:16] (deploying a few changes in reader deploy window) [23:03:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356918 (https://phabricator.wikimedia.org/T427942) (owner: 10Jdlrobson) [23:03:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/GlobalPreferences] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356993 (https://phabricator.wikimedia.org/T427942) (owner: 10Jdlrobson) [23:03:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1355866 (https://phabricator.wikimedia.org/T435564) (owner: 10Jdlrobson) [23:03:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:03:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:04:28] (03Merged) 10jenkins-bot: Add Storage::clearCache() to allow clearing the cache per user [extensions/GlobalPreferences] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356993 (https://phabricator.wikimedia.org/T427942) (owner: 10Jdlrobson) [23:05:29] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 5.194 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:06:43] (03CR) 10CI reject: [V:04-1] Clear cache in syncDonorStatus.php after updating global preferences [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356918 (https://phabricator.wikimedia.org/T427942) (owner: 10Jdlrobson) [23:07:21] (03Merged) 10jenkins-bot: DonorIdentification: Preserve donor preferences in sync [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1355866 (https://phabricator.wikimedia.org/T435564) (owner: 10Jdlrobson) [23:08:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:08:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:09:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356918 (https://phabricator.wikimedia.org/T427942) (owner: 10Jdlrobson) [23:12:24] (03Merged) 10jenkins-bot: Clear cache in syncDonorStatus.php after updating global preferences [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1356918 (https://phabricator.wikimedia.org/T427942) (owner: 10Jdlrobson) [23:16:28] sbassett: are you around? [23:16:50] concurrent prep is locked by sbassett [23:17:28] jdlrobson: my scap sync-world is almost done [23:17:29] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 4.981 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:18:47] FIRING: [3x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 7d 13h 47m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [23:18:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:20:29] (03PS1) 10Dwisehaupt: fr-tech: shift frdata services to codfw [dns] - 10https://gerrit.wikimedia.org/r/1357009 (https://phabricator.wikimedia.org/T419556) [23:20:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:26:56] (03CR) 10Dwisehaupt: [C:03+2] fr-tech: shift frdata services to codfw [dns] - 10https://gerrit.wikimedia.org/r/1357009 (https://phabricator.wikimedia.org/T419556) (owner: 10Dwisehaupt) [23:27:29] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 5.363 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:29:14] !log Deployed security fix for T441038 [23:29:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:30:07] !log dwisehaupt@dns1004 START - running authdns-update [23:30:38] Jdlrobson: all yours [23:32:31] !log dwisehaupt@dns1004 END - running authdns-update [23:32:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:35:23] hmmm sbassett any idea how to debug this? [23:35:25] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 0.280 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:35:34] 23:33:50 prep failed: Patch a != b: wikimedia/parsoid/src/Wt2Html/TT/AttributeExpander.php != src/Wt2Html/TT/AttributeExpander.php (scap version: 4.301.0) (duration: 01m 23s) 23:33:50 backport failed: Command '['/usr/bin/scap', 'prep', 'auto']' returned non-zero exit status 1. (scap version: 4.301.0) (duration: 01m 33s) [23:36:25] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:37:06] (03CR) 10Jdlrobson: Drop MediaViewerMobileBeta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1348770 (https://phabricator.wikimedia.org/T436799) (owner: 10Jdlrobson) [23:37:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1348770 (https://phabricator.wikimedia.org/T436799) (owner: 10Jdlrobson) [23:37:25] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:38:10] (03Merged) 10jenkins-bot: Drop MediaViewerMobileBeta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1348770 (https://phabricator.wikimedia.org/T436799) (owner: 10Jdlrobson) [23:38:47] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:39:11] hmm sbassett it seems like scap is in a broken state. Is anyone around that can help me leave this in a good state for the next person? [23:39:28] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1357013 [23:39:28] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1357013 (owner: 10TrainBranchBot) [23:41:11] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:46:34] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1357013 (owner: 10TrainBranchBot) [23:47:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:48:29] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 6.445 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:49:17] @dancy I will need to step away in 13 minutes but tagging you as train conductor (cc hashar ). [23:53:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [23:55:30] jdlrobson: Taking a look [23:56:58] thanks dancy