[00:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0000) [00:04:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506577 (10phaultfinder) [00:05:49] scap is almost done [00:07:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:09:17] PROBLEM - Check unit status of kube-reload-certs on deploy2003 is CRITICAL: CRITICAL: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [00:10:48] !log Deployed security fix for T440469, T441384, T440735, T440501, T439716 [00:10:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:11:09] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 653.34 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:12:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:12:33] scap finished [00:15:14] jouncebot: nowandnext [00:15:15] For the next 0 hour(s) and 44 minute(s): Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0000) [00:15:15] In 1 hour(s) and 44 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0200) [00:15:24] if the readers window isn't in use, I'll run another set of envoy deploys [00:17:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:20:36] maryum: ^ if you click the grafana link, does the time of that increase line up with your deployment? [00:20:59] rzl: looking into it now [00:21:12] might be nothing, it's been noisy for the past week or so [00:21:32] hmm will investigate [00:22:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:22:15] upgrading that to, it's almost certainly nothing, unless your deploy was touching something cache-related :) except that it might be triggered by a scap deploy [00:22:32] please don't spend too much time chasing it down, but if you *were* doing something memcache-related I'd love to konw [00:22:33] *know [00:23:47] but whatever this was, it started at ~16:02, so I'm a little curious how that compares to your scap timestamps [00:24:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12506706 (10phaultfinder) [00:25:13] So if anything that's the second scap, not the first one, since the first one had finished by 15:24 PST [00:25:35] I'm looking at those patches now to see if there's anything that would spike memcached traffic [00:25:46] second one, yeah [00:25:50] thanks Roan [00:27:20] again, just to say, if anything this is related to an off-and-on ghost we've been chasing for a while, we're looking for a possible connection but not a smoking gun :) not worth a ton of effort, just a glance [00:27:38] (unless this is a totally unrelated memcache breakage, but it doesn't smell that way) [00:28:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:31:03] rzl: Yeah I don't see anything related in the deployed patches. None of them involve caching directly, and none of them touch hot code paths, they're all in code that is run infrequently (e.g. actions only privileged users can take, stuff like that) [00:31:36] okay, cool. maybe just related to the churn of the deployment (in which case, interesting that nothing happened for scap #1) or maybe just a coincidence [00:31:42] appreciate you taking a look! [00:33:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:38:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:41:07] going ahead with those envoy deploys [00:41:22] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/apertium: apply [00:41:56] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/apertium: apply [00:42:14] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1361600 [00:42:14] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1361600 (owner: 10TrainBranchBot) [00:42:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:44:09] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.37 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:46:02] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/chart-renderer: apply [00:46:27] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply [00:46:37] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [00:46:43] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [00:46:52] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/citoid: apply [00:47:12] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/citoid: apply [00:47:38] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply [00:47:54] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1361600 (owner: 10TrainBranchBot) [00:47:56] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply [00:48:38] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/cxserver: apply [00:49:00] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [00:49:06] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/data-gateway: apply [00:49:25] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply [00:50:08] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/developer-portal: apply [00:50:25] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [00:51:11] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/device-analytics: apply [00:51:27] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply [00:52:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:53:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:53:33] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/echostore: apply [00:53:49] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/echostore: apply [00:53:56] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/edit-analytics: apply [00:54:13] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply [00:54:18] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply [00:55:26] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply [00:55:32] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/editor-analytics: apply [00:55:46] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply [00:56:31] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply [00:57:12] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply [00:57:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [00:58:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:00:23] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/eventstreams: apply [01:00:58] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply [01:01:05] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/geo-analytics: apply [01:01:24] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply [01:01:58] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/kartotherian: apply [01:02:15] RESOLVED: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [01:03:01] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply [01:03:07] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply [01:03:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:03:27] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply [01:03:31] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply [01:03:56] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply [01:04:05] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply [01:04:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:05:37] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply [01:05:44] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [01:09:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:10:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:12:16] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1361602 [01:12:16] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1361602 (owner: 10TrainBranchBot) [01:13:03] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [01:13:14] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/media-analytics: apply [01:13:33] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply [01:13:55] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [01:14:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506825 (10phaultfinder) [01:16:05] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [01:16:19] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [01:17:00] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [01:18:22] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/page-analytics: apply [01:18:41] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply [01:18:50] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/proton: apply [01:19:16] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1361602 (owner: 10TrainBranchBot) [01:20:02] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [01:20:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:20:15] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/push-notifications: apply [01:20:45] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply [01:21:29] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/recommendation-api: apply [01:21:55] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply [01:22:02] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/sessionstore: apply [01:22:18] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply [01:22:48] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/shellbox: apply [01:23:17] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [01:23:24] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply [01:23:59] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply [01:24:10] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/shellbox-media: apply [01:24:26] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply [01:24:43] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [01:25:04] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [01:25:12] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [01:25:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:25:42] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [01:25:48] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply [01:26:24] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply [01:26:31] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/termbox: apply [01:27:10] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/termbox: apply [01:27:25] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/toolhub: apply [01:27:59] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [01:28:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:28:23] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [01:28:47] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [01:28:52] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [01:29:11] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [01:29:33] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [01:30:12] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [01:30:20] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/zotero: apply [01:30:43] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/zotero: apply [01:33:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:35:15] FIRING: [3x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [01:35:16] done charlieing for today [01:40:15] RESOLVED: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [01:44:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506890 (10phaultfinder) [01:48:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:50:57] FIRING: ProbeDown: Service mw-api-ext-next:4455 has failed probes (http_mw-api-ext-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext-next:4455 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:53:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:55:57] RESOLVED: ProbeDown: Service mw-api-ext-next:4455 has failed probes (http_mw-api-ext-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext-next:4455 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:58:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:59:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0200) [02:04:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:07:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:10:02] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12506927 (10phaultfinder) [02:10:28] (03CR) 10Lerickson: "Thanks! One question." [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) (owner: 10Gmodena) [02:11:34] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.48.0-wmf.3 [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361607 (https://phabricator.wikimedia.org/T442221) [02:11:36] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.48.0-wmf.3 [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361607 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:12:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:13:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:18:29] (03Merged) 10jenkins-bot: Branch commit for wmf/1.48.0-wmf.3 [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361607 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [02:21:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:23:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:26:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:28:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:33:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:38:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:40:12] FIRING: [2x] ProbeDown: Service people1005:30443 has failed probes (http_security_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:43:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:47:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:49:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506963 (10phaultfinder) [02:52:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0300) [03:09:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506971 (10phaultfinder) [03:26:31] PROBLEM - statsv Varnishkafka log producer on cp6012 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [03:27:31] RECOVERY - statsv Varnishkafka log producer on cp6012 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [03:32:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:37:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0400) [04:14:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [04:15:26] (03Merged) 10jenkins-bot: MathML+MathJax rollout to phase 6 (remaining large Wikipedias) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [04:16:14] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1311061|MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] [04:16:20] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [04:24:21] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1311061|MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [04:24:27] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [04:24:57] FIRING: ProbeDown: Service mw-api-ext-next:4455 has failed probes (http_mw-api-ext-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext-next:4455 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:29:57] RESOLVED: ProbeDown: Service mw-api-ext-next:4455 has failed probes (http_mw-api-ext-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext-next:4455 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:33:03] !log krinkle@deploy2003 krinkle: Continuing with deployment [04:40:48] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1311061|MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s) [04:40:55] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [05:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0500) [05:00:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [05:02:28] !log mwpresync@deploy2003 Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s) [05:05:03] (03PS1) 10Krinkle: Purge mathoid embeds via RejectParserCacheValue hook [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361722 (https://phabricator.wikimedia.org/T271001) [05:08:59] (03PS1) 10Jdlrobson: Revert "Remove mobile watchlist styles" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361723 [05:09:45] (03PS2) 10Jdlrobson: Revert "Remove mobile watchlist styles" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361723 [05:10:24] (03PS1) 10Jdlrobson: Revert "footer: Hide separator before a hidden last footer place" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361724 (https://phabricator.wikimedia.org/T387797) [05:14:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12507068 (10phaultfinder) [05:20:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361187 (https://phabricator.wikimedia.org/T443033) (owner: 10Tryvix1509) [05:20:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [05:47:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [05:52:15] FIRING: [7x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [05:57:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [06:08:53] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12507090 (10Marostegui) >>! In T443208#12503051, @Papaul wrote: > @CWilliams-WMF We will have to reboot the node after the change Yeah, this is the main problem. @Papaul can this process be changed and then th... [06:10:06] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12507091 (10Marostegui) @Papaul maybe we should also check this upon first installation of DB hosts? [06:11:46] 10ops-codfw, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12507095 (10Marostegui) @Papaul is this host under warranty? It may still be, as it was purchased in 2024. Not sure if Dell is going to be helpful but maybe worth checking with them. [06:14:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12507100 (10phaultfinder) [06:17:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [06:23:17] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12507118 (10Marostegui) p:05Medium→03High This host has crashed again. Can we get Dell involved here? This host is under warrany ` ----------------------------------------------------------------------------... [06:24:41] (03PS1) 10Ayounsi: Enable multipath for PyBal BGP group [homer/public] - 10https://gerrit.wikimedia.org/r/1361729 (https://phabricator.wikimedia.org/T443003) [06:27:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [06:36:15] FIRING: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [06:40:12] FIRING: [2x] ProbeDown: Service people1005:30443 has failed probes (http_security_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:41:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [06:46:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [06:49:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12507160 (10phaultfinder) [06:54:29] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 3/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:55:29] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:58:10] FIRING: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0700) [07:00:05] marostegui, cezmunsta, and federico3: It is that lovely time of the day again! You are hereby commanded to deploy Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0700). [07:00:15] FIRING: [3x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [07:03:10] RESOLVED: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:05:15] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [07:10:15] RESOLVED: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [07:10:31] (03CR) 10JMeybohm: [C:03+2] kubernetes-generic: Alert on coredns failing uptream health checks [alerts] - 10https://gerrit.wikimedia.org/r/1355559 (https://phabricator.wikimedia.org/T439575) (owner: 10JMeybohm) [07:13:23] (03Merged) 10jenkins-bot: kubernetes-generic: Alert on coredns failing uptream health checks [alerts] - 10https://gerrit.wikimedia.org/r/1355559 (https://phabricator.wikimedia.org/T439575) (owner: 10JMeybohm) [07:14:25] 10ops-codfw, 06DC-Ops, 06ServiceOps: Degraded RAID on wikikube-worker2285 - https://phabricator.wikimedia.org/T441252#12507177 (10JMeybohm) [07:14:26] 10ops-codfw, 06SRE, 06DC-Ops: Degraded RAID on wikikube-worker2285 - https://phabricator.wikimedia.org/T442456#12507175 (10JMeybohm) →14Duplicate dup:03T441252 [07:23:59] RECOVERY - MD RAID on wikikube-worker2285 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [07:33:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [07:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:38:15] RESOLVED: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [07:54:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12507217 (10phaultfinder) [07:55:23] (03CR) 10Arthur taylor: Stop exempting item/property creation from hCaptcha (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [08:00:05] Amir1, urbanecm, and awight: OwO what's this, a deployment window?? UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0800). nyaa~ [08:00:05] vs1: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:01:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:06:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:08:25] (03CR) 10Muehlenhoff: [C:03+2] Remove non-hardware ssh key for wmde-leszek [puppet] - 10https://gerrit.wikimedia.org/r/1361542 (https://phabricator.wikimedia.org/T442763) (owner: 10WMDE-leszek) [08:11:18] (03CR) 10Muehlenhoff: [C:03+2] Add site.pp/preseed entries for new puppet servers and Ganeti nodes in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1361411 (https://phabricator.wikimedia.org/T434699) (owner: 10Muehlenhoff) [08:12:35] (03Abandoned) 10Slyngshede: mw-web: upsize for single-DC serving [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344016 (https://phabricator.wikimedia.org/T435462) (owner: 10Slyngshede) [08:13:31] !log incresing disk in arclamp2001 [08:13:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:14:04] (03PS1) 10Muehlenhoff: Failover irc.w.o for reboot [dns] - 10https://gerrit.wikimedia.org/r/1361799 [08:14:27] !log deleting stale .tmp logs from 2025 in arclamp2001 [08:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:17:01] (03CR) 10Slyngshede: [C:03+2] P:tofurkey enable Tofurkey for DRMRS [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [08:24:28] !log increasing disk size in arclamp1001 [08:24:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:29:20] o/ if no one is deploying atm, I'd like to do a private deploy [08:31:25] (03PS1) 10Rafka Aditia: Set the project namespace (via sitename) for Sundanese Wikipedia (suwiki) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361803 (https://phabricator.wikimedia.org/T443174) [08:32:57] o/ [08:33:23] oh jk, I was about to start a private deploy but haven't done anything yet. vs1 when you're done w/your deploy, could you ping me? [08:33:38] 06SRE, 06Infrastructure-Foundations, 10netops: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12507287 (10ayounsi) Good idea, we can also log them for some time before dropping them to be sure we're not about to block something we... [08:33:41] Okay, but I need a deployer. [08:34:09] (03CR) 10Ayounsi: [C:03+1] Remove repo sync definition for Routinator on Bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1361408 (https://phabricator.wikimedia.org/T438122) (owner: 10Muehlenhoff) [08:39:19] Amir1: Are you available for backport [08:41:31] if not, I think Tran: should go ahead. N [08:42:03] (sorry, while I can technically backport, I'm not comfortable approving the patches for backport) [08:42:24] but I can get my private deploy out of the way in that case [08:42:26] I mean, your private deploy [08:42:34] ack, going to start [08:44:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:44:30] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for jnuche [puppet] - 10https://gerrit.wikimedia.org/r/1361465 (https://phabricator.wikimedia.org/T442761) (owner: 10Muehlenhoff) [08:46:38] (03CR) 10Muehlenhoff: [C:03+2] Remove repo sync definition for Routinator on Bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1361408 (https://phabricator.wikimedia.org/T438122) (owner: 10Muehlenhoff) [08:48:12] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12507295 (10MoritzMuehlenhoff) 05Open→03Resolved All done! [08:48:26] (03CR) 10Dpogorzelski: [C:03+2] kserve-llmisvc-resources: add a Gateway for LLMInferenceService routes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361478 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [08:49:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:50:38] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [08:50:44] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [08:53:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:54:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12507308 (10phaultfinder) [08:57:30] (03PS1) 10Effie Mouzeli: site.pp: add mc20[56-73] [puppet] - 10https://gerrit.wikimedia.org/r/1361808 (https://phabricator.wikimedia.org/T436272) [08:58:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:58:30] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps, and 2 others: Q1:rack/setup/install mc20[56-73] - https://phabricator.wikimedia.org/T436272#12507326 (10jijiki) Thank you @Jhancock.wm ! Here is my suggestion: | Row | Racks | Notes | --- | --- | --- | A | A3, A6 | 2 servers in A6 (`A4 not usable:` mc-gp2004) |... [08:58:58] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org [08:59:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:59:18] (03PS1) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1361809 (https://phabricator.wikimedia.org/T439509) [09:00:26] (03CR) 10CI reject: [V:04-1] P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1361809 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [09:00:44] morning, the train is currently blocked on two security patches failing to apply, don't know enough about them to attempt a rebase myself so we need to wait for a bit for someone more knowledgeable to come online [09:01:11] also, the train events are gone from the calendar so the bot is not pinging about it, funny [09:01:28] (03CR) 10Ladsgroup: [C:03+2] thumbor-plugins: Rebuild against latest security updates in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1361328 (owner: 10Muehlenhoff) [09:01:39] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9588/console" [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [09:01:51] (03CR) 10Trueg: [C:03+1] "The alerts look reasonable to me." [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) (owner: 10Gmodena) [09:02:09] (03CR) 10Ladsgroup: [C:03+1] "I've said it many times, I say it again: I will kill ircstreams with my bare hands." [dns] - 10https://gerrit.wikimedia.org/r/1361799 (owner: 10Muehlenhoff) [09:02:21] (03CR) 10Klausman: [C:03+1] raid: Treat hot spare drives as healthy in the broadcom check [puppet] - 10https://gerrit.wikimedia.org/r/1361526 (https://phabricator.wikimedia.org/T441831) (owner: 10Btullis) [09:02:33] !log Deployed patch for T441675 [09:02:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:02:38] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9589/console" [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [09:03:28] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org [09:03:37] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9590/co" [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [09:03:37] (03CR) 10Jelto: [C:03+1] "looks good, two nits in the comments" [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [09:04:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:04:16] done with my private deploy [09:04:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12507352 (10phaultfinder) [09:05:13] jouncebot: now [09:05:14] For the next 1 hour(s) and 54 minute(s): MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T0900) [09:05:22] calendar fixed [09:08:02] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org [09:09:13] (03Merged) 10jenkins-bot: thumbor-plugins: Rebuild against latest security updates in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1361328 (owner: 10Muehlenhoff) [09:09:15] RESOLVED: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:10:17] (03PS1) 10Dpogorzelski: experimental: route llm-qwen3-14b-fp8 through the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361810 (https://phabricator.wikimedia.org/T436650) [09:12:01] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance [09:12:04] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json [09:12:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:12:31] (03PS1) 10Muehlenhoff: Remove legacy key for mgerlach [puppet] - 10https://gerrit.wikimedia.org/r/1361811 (https://phabricator.wikimedia.org/T443038) [09:12:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org [09:17:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:17:40] (03CR) 10Bartosz Wójtowicz: [C:03+1] experimental: route llm-qwen3-14b-fp8 through the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361810 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:18:17] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json [09:19:22] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12507394 (10CWilliams-WMF) @Marostegui I would guess that before that we should change: ` racadm>>get BIOS.SysProfileSettings.ProcCStates [Key=BIOS.Setup.1-1#SysProfileSettings] ProcCStates=Enabled racadm>>get... [09:20:20] (03CR) 10Dpogorzelski: [C:03+2] experimental: route llm-qwen3-14b-fp8 through the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361810 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:20:37] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] experimental: route llm-qwen3-14b-fp8 through the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361810 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:21:17] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [09:22:16] (03CR) 10AikoChou: ml-services: place jina-embeddings on 24G GPU partitions (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361475 (https://phabricator.wikimedia.org/T443256) (owner: 10Gkyziridis) [09:22:45] FIRING: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:23:00] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:24:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12507419 (10phaultfinder) [09:26:13] (03PS1) 10Btullis: ceph: Remove dummy keys for the unused osd.$HOSTNAME entities [labs/private] - 10https://gerrit.wikimedia.org/r/1361813 (https://phabricator.wikimedia.org/T437717) [09:26:51] (03PS1) 10Dpogorzelski: kserve-llm-inference: keep the endpoint picker out of the mesh [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361814 (https://phabricator.wikimedia.org/T436650) [09:27:03] (03PS1) 10Btullis: ceph: Stop ordering ceph::osds after the osd.$HOSTNAME keyring [puppet] - 10https://gerrit.wikimedia.org/r/1361815 (https://phabricator.wikimedia.org/T437717) [09:27:06] (03PS1) 10Btullis: ceph: Remove the unused osd.$HOSTNAME and client.rgw.$HOSTNAME entities [puppet] - 10https://gerrit.wikimedia.org/r/1361816 (https://phabricator.wikimedia.org/T437717) [09:27:14] (03CR) 10Btullis: [C:03+2] raid: Treat hot spare drives as healthy in the broadcom check [puppet] - 10https://gerrit.wikimedia.org/r/1361526 (https://phabricator.wikimedia.org/T441831) (owner: 10Btullis) [09:27:45] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:28:19] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json [09:29:21] (03PS10) 10Jelto: sre.k8s: add upgrade-kubelets-inplace cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) [09:32:35] (03CR) 10Jelto: sre.k8s: add upgrade-kubelets-inplace cookbook (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [09:32:45] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:35:12] (03PS1) 10Dpogorzelski: kserve-llm-inference: fix the endpoint picker NetworkPolicy selector [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361817 (https://phabricator.wikimedia.org/T436650) [09:36:22] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for mgerlach [puppet] - 10https://gerrit.wikimedia.org/r/1361811 (https://phabricator.wikimedia.org/T443038) (owner: 10Muehlenhoff) [09:37:13] (03CR) 10Bartosz Wójtowicz: [C:03+1] kserve-llm-inference: keep the endpoint picker out of the mesh [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361814 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:37:45] RESOLVED: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:37:46] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12507459 (10MoritzMuehlenhoff) >>! In T443038#12499213, @andrea.denisse wrote: > Hi @MGerlach , once you confirm that SSH access works with the Yubikey we can remove the old key... [09:38:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json [09:43:11] (03CR) 10Dpogorzelski: [C:03+2] kserve-llm-inference: keep the endpoint picker out of the mesh [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361814 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:43:43] (03CR) 10Bartosz Wójtowicz: [C:03+1] kserve-llm-inference: fix the endpoint picker NetworkPolicy selector [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361817 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:44:15] (03CR) 10Dpogorzelski: [C:03+2] kserve-llm-inference: fix the endpoint picker NetworkPolicy selector [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361817 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:44:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12507472 (10phaultfinder) [09:48:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json [09:48:42] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance [09:48:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json [09:52:13] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-presto1006 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [09:54:00] (03PS1) 10Dpogorzelski: kserve-llm-inference: render router route rules per service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361819 (https://phabricator.wikimedia.org/T436650) [09:57:16] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json [09:59:10] (03CR) 10Bartosz Wójtowicz: [C:03+1] kserve-llm-inference: render router route rules per service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361819 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:59:58] (03CR) 10Dpogorzelski: [C:03+2] kserve-llm-inference: render router route rules per service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361819 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [10:00:09] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-presto1015 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [10:00:23] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:01:01] (03CR) 10Effie Mouzeli: [C:03+1] changeprop: Wire in mesh.networkpolicy.egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352101 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [10:02:37] (03PS2) 10Jelto: spiderpig: add mediawiki-event-enrichment deployments [puppet] - 10https://gerrit.wikimedia.org/r/1360938 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [10:03:48] (03PS1) 10Muehlenhoff: Remove legacy keys for arthurtaylor and tarrow [puppet] - 10https://gerrit.wikimedia.org/r/1361820 (https://phabricator.wikimedia.org/T442763) [10:04:19] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12507546 (10BTullis) 05Open→03Resolved This is now working as expected. The Icinga check is green when drives are configured as either global hot s... [10:04:33] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:07:12] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy keys for arthurtaylor and tarrow [puppet] - 10https://gerrit.wikimedia.org/r/1361820 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [10:07:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json [10:08:23] (03PS1) 10Awight: Switch awight to a fido2 key [puppet] - 10https://gerrit.wikimedia.org/r/1361821 (https://phabricator.wikimedia.org/T442763) [10:09:36] (03PS1) 10Jelto: microsites: fix security blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1361823 (https://phabricator.wikimedia.org/T443305) [10:11:52] (03PS1) 10Muehlenhoff: Record LDAP access for lucian [puppet] - 10https://gerrit.wikimedia.org/r/1361824 [10:13:38] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for lucian [puppet] - 10https://gerrit.wikimedia.org/r/1361824 (owner: 10Muehlenhoff) [10:13:39] (03PS4) 10JMeybohm: sre.k8s: Add cookbook for apiserver inplace upgrades [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) [10:14:26] (03CR) 10Blake: [C:03+2] changeprop: Wire in mesh.networkpolicy.egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352101 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [10:14:54] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12507641 (10phaultfinder) [10:15:50] !log installing imagemagick security updates [10:15:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:15:59] (03Merged) 10jenkins-bot: changeprop: Wire in mesh.networkpolicy.egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352101 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [10:16:46] (03CR) 10JMeybohm: [C:03+1] sre.k8s: add upgrade-kubelets-inplace cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [10:17:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json [10:19:31] (03PS1) 10Dpogorzelski: kserve-llm-inference: serve the endpoint picker's ext_proc in plaintext [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361825 (https://phabricator.wikimedia.org/T436650) [10:19:52] (03PS1) 10Ladsgroup: thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361826 [10:19:57] (03CR) 10Jelto: [C:03+2] spiderpig: add mediawiki-event-enrichment deployments [puppet] - 10https://gerrit.wikimedia.org/r/1360938 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [10:21:36] (03CR) 10JMeybohm: sre.k8s: Add cookbook for apiserver inplace upgrades (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [10:22:11] !log blake@deploy2003 helmfile [staging] START helmfile.d/services/changeprop: apply [10:22:36] !log blake@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop: apply [10:22:44] !log blake@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop: apply [10:22:45] !log urbanecm@deploy2003 mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97103) [10:22:48] !log installing git-lfs security updates [10:22:54] T438366: Run GrowthExperiments:revalidateLinkRecommendations.php on all Wikimedia wikis - https://phabricator.wikimedia.org/T438366 [10:22:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:23:03] !log urbanecm@deploy2003 mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose # T438366 [10:23:19] !log blake@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [10:23:19] (03CR) 10Jelto: [C:03+1] "lgtm" [cookbooks] - 10https://gerrit.wikimedia.org/r/1358223 (https://phabricator.wikimedia.org/T437480) (owner: 10JMeybohm) [10:23:22] !log urbanecm@deploy2003 mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose # T438366 [10:23:27] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop: apply [10:23:50] !log urbanecm@deploy2003 mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose # T438366 [10:23:58] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop: apply [10:25:32] (03CR) 10Bartosz Wójtowicz: [C:03+1] kserve-llm-inference: serve the endpoint picker's ext_proc in plaintext [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361825 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [10:27:15] (03CR) 10Dpogorzelski: [C:03+2] kserve-llm-inference: serve the endpoint picker's ext_proc in plaintext [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361825 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [10:27:17] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361826 (owner: 10Ladsgroup) [10:27:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json [10:27:40] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [10:27:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json [10:27:49] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:28:49] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for aikochou - https://phabricator.wikimedia.org/T442367#12507724 (10isarantopoulos) Aiko has updated her patch with a key from Yubikey instead of the mac enclave so we're ready to go! [10:29:22] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:29:49] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:30:23] (03PS1) 10JMeybohm: k8s::package: Don't force install new package versions [puppet] - 10https://gerrit.wikimedia.org/r/1361829 (https://phabricator.wikimedia.org/T427069) [10:30:30] !log blake@deploy2003 helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [10:30:46] !log blake@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [10:30:56] !log urbanecm@deploy2003 mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97106) [10:31:03] T438366: Run GrowthExperiments:revalidateLinkRecommendations.php on all Wikimedia wikis - https://phabricator.wikimedia.org/T438366 [10:31:08] !log blake@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [10:31:17] !log urbanecm@deploy2003 mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose # T438366 [10:31:47] !log blake@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [10:32:27] (03CR) 10JMeybohm: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361829 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [10:32:32] (03CR) 10Ladsgroup: [C:03+2] thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361826 (owner: 10Ladsgroup) [10:33:02] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [10:33:48] (03CR) 10Btullis: [V:03+2 C:03+2] ceph: Remove dummy keys for the unused osd.$HOSTNAME entities [labs/private] - 10https://gerrit.wikimedia.org/r/1361813 (https://phabricator.wikimedia.org/T437717) (owner: 10Btullis) [10:33:49] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [10:34:05] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json [10:34:07] (03Merged) 10jenkins-bot: thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361826 (owner: 10Ladsgroup) [10:34:15] FIRING: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:34:26] (03PS1) 10Muehlenhoff: Remove legacy keys for mlitn and ssastry [puppet] - 10https://gerrit.wikimedia.org/r/1361830 (https://phabricator.wikimedia.org/T442758) [10:36:39] !log cwilliams@cumin1004 START - Cookbook sre.mysql.pool pool db1228: Repooling [10:36:50] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12507767 (10ops-monitoring-bot) Starting pool of db1228 by cwilliams@cumin1004: Repooling [10:39:15] RESOLVED: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:39:34] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for aikochou - https://phabricator.wikimedia.org/T442367#12507791 (10achou) [10:40:12] FIRING: [2x] ProbeDown: Service people1005:30443 has failed probes (http_security_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:42:29] !log ladsgroup@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [10:42:42] !log ladsgroup@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [10:42:50] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy keys for mlitn and ssastry [puppet] - 10https://gerrit.wikimedia.org/r/1361830 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [10:42:52] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361187 (https://phabricator.wikimedia.org/T443033) (owner: 10Tryvix1509) [10:43:12] !log ladsgroup@deploy2003 helmfile [codfw] START helmfile.d/services/thumbor: apply [10:43:16] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355072 (https://phabricator.wikimedia.org/T440179) (owner: 10Tryvix1509) [10:44:07] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json [10:46:14] (03PS2) 10Mahmoud-abdelsattar: Stop exempting item/property creation from hCaptcha [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) [10:46:53] !log ladsgroup@deploy2003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [10:47:04] (03CR) 10Mahmoud-abdelsattar: Stop exempting item/property creation from hCaptcha (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [10:47:37] (03CR) 10Hnowlan: [C:03+2] "One extra qualifier: we'll need to run `systemctl reenable arclamp-log.service` after this change is merged to create the `wants` relation" [puppet] - 10https://gerrit.wikimedia.org/r/1360876 (https://phabricator.wikimedia.org/T441510) (owner: 10Hnowlan) [10:47:58] (03PS1) 10Gehel: admin: new SSH key for passimacopoulos backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1361832 (https://phabricator.wikimedia.org/T442756) [10:48:44] (03PS2) 10Gehel: admin: new SSH key for passimacopoulos backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1361832 (https://phabricator.wikimedia.org/T442756) [10:49:00] (03CR) 10Arthur taylor: [C:03+1] "Looks good - let's do it!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [10:50:38] (03CR) 10Arnaudb: [C:03+1] "looks good to me!" [puppet] - 10https://gerrit.wikimedia.org/r/1361823 (https://phabricator.wikimedia.org/T443305) (owner: 10Jelto) [10:51:07] (03CR) 10Muehlenhoff: [C:03+2] Failover irc.w.o for reboot [dns] - 10https://gerrit.wikimedia.org/r/1361799 (owner: 10Muehlenhoff) [10:51:28] !log jmm@dns1004 START - running authdns-update [10:53:18] (03PS1) 10Kamila Součková: shellbox: bump image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361833 [10:53:31] !log ladsgroup@deploy2003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [10:53:43] !log jmm@dns1004 END - running authdns-update [10:54:09] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json [10:54:15] FIRING: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:54:50] !log ladsgroup@deploy2003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [10:56:06] (03PS1) 10Tiziano Fogli: metamonitoring: make public endpoint script aware of silenced DMS alerts [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) [10:57:45] (03CR) 10Kamila Součková: [C:03+2] Revert "shellbox: revert all back to 8.3 flavors" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361446 (owner: 10Kamila Součková) [10:58:01] (03CR) 10Hnowlan: [C:03+2] "Correction: `sudo systemctl reenable excimer-log.service excimer-wall-log.service`. Done on both hosts, `want` links look correct." [puppet] - 10https://gerrit.wikimedia.org/r/1360876 (https://phabricator.wikimedia.org/T441510) (owner: 10Hnowlan) [10:58:25] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1361832 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [10:59:13] (03Merged) 10jenkins-bot: Revert "shellbox: revert all back to 8.3 flavors" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361446 (owner: 10Kamila Součková) [10:59:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:59:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12507876 (10phaultfinder) [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1100) [11:00:55] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361833 (owner: 10Kamila Součková) [11:00:56] (03PS1) 10Arnaudb: gerrit: raise the GerritHigh4xxRatio threshold to 25% [alerts] - 10https://gerrit.wikimedia.org/r/1361836 [11:02:06] (03CR) 10Kamila Součková: [C:03+2] shellbox: bump image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361833 (owner: 10Kamila Součková) [11:03:03] (03PS1) 10Ameisenigel: T443269 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361837 [11:03:43] 06SRE, 06Infrastructure-Foundations, 10netops: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12507890 (10cmooney) >>! In T443214#12507287, @ayounsi wrote: > Good idea, we can also log them for some time before dropping them to be... [11:03:58] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [11:04:11] (03Merged) 10jenkins-bot: shellbox: bump image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361833 (owner: 10Kamila Součková) [11:04:11] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json [11:04:30] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance [11:04:46] (03PS2) 10Ameisenigel: liwiktionary: Create Appendix namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361837 (https://phabricator.wikimedia.org/T443269) [11:05:04] (03CR) 10Brouberol: [C:03+1] "Validated out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1361832 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [11:06:10] (03CR) 10Effie Mouzeli: [C:03+1] deployment_server/mw-*: switch next and migration to PHP 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1361437 (https://phabricator.wikimedia.org/T439955) (owner: 10Kamila Součková) [11:09:43] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance [11:09:45] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json [11:12:38] !log installing chromium security updates [11:12:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:15:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json [11:19:36] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1358720 (https://phabricator.wikimedia.org/T442367) (owner: 10AikoChou) [11:19:45] (03PS4) 10Hnowlan: miscweb: add apps.wikipedia.org configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356269 (https://phabricator.wikimedia.org/T440988) [11:19:50] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for aikochou [puppet] - 10https://gerrit.wikimedia.org/r/1358720 (https://phabricator.wikimedia.org/T442367) (owner: 10AikoChou) [11:21:50] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling [11:21:57] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12507983 (10ops-monitoring-bot) Completed pooling of db1228 by cwilliams@cumin1004: Repooling [11:22:47] (03PS1) 10Ameisenigel: eswiki: add editprotected to botadmin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361843 (https://phabricator.wikimedia.org/T443272) [11:24:17] jouncebot: nowandnext [11:24:17] For the next 0 hour(s) and 35 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1100) [11:24:17] In 1 hour(s) and 35 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1300) [11:25:42] (03PS1) 10Muehlenhoff: Add FIDO keys for jdlrobson and rsilvola [puppet] - 10https://gerrit.wikimedia.org/r/1361844 (https://phabricator.wikimedia.org/T442758) [11:25:45] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json [11:26:20] (03PS1) 10Samtar: security.txt: Fix link to Acknowledgments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361845 (https://phabricator.wikimedia.org/T443423) [11:26:36] (03CR) 10Btullis: [C:03+2] ceph: Stop ordering ceph::osds after the osd.$HOSTNAME keyring [puppet] - 10https://gerrit.wikimedia.org/r/1361815 (https://phabricator.wikimedia.org/T437717) (owner: 10Btullis) [11:26:51] (03CR) 10CI reject: [V:04-1] Add FIDO keys for jdlrobson and rsilvola [puppet] - 10https://gerrit.wikimedia.org/r/1361844 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [11:27:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361843 (https://phabricator.wikimedia.org/T443272) (owner: 10Ameisenigel) [11:27:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361837 (https://phabricator.wikimedia.org/T443269) (owner: 10Ameisenigel) [11:28:20] (03CR) 10Hnowlan: [C:03+1] "😎" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361572 (https://phabricator.wikimedia.org/T442195) (owner: 10ArielGlenn) [11:28:45] (03PS1) 10Ladsgroup: Language: Avoid creating all converters just to find the parent [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361846 (https://phabricator.wikimedia.org/T443175) [11:29:03] (03PS1) 10Ladsgroup: Language: Avoid creating all converters just to find the parent [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361847 (https://phabricator.wikimedia.org/T443175) [11:29:37] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-d6-eqiad [11:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12508034 (10phaultfinder) [11:29:57] (03CR) 10Klausman: [C:03+1] ceph: Stop ordering ceph::osds after the osd.$HOSTNAME keyring [puppet] - 10https://gerrit.wikimedia.org/r/1361815 (https://phabricator.wikimedia.org/T437717) (owner: 10Btullis) [11:30:33] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad [11:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:35:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 21.57% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:35:47] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json [11:36:41] (03PS2) 10Muehlenhoff: Add FIDO keys for jdlrobson and rsilvola [puppet] - 10https://gerrit.wikimedia.org/r/1361844 (https://phabricator.wikimedia.org/T442758) [11:37:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:38:38] (03PS1) 10Ladsgroup: Add --listfile to transcode maint scripts [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361850 (https://phabricator.wikimedia.org/T435744) [11:38:38] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12508080 (10cmooney) [11:39:12] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361850 (https://phabricator.wikimedia.org/T435744) (owner: 10Ladsgroup) [11:39:42] (03CR) 10Effie Mouzeli: [C:03+1] changeprop: node 20 to node 26 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361572 (https://phabricator.wikimedia.org/T442195) (owner: 10ArielGlenn) [11:39:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12508087 (10phaultfinder) [11:39:46] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for jdlrobson and rsilvola [puppet] - 10https://gerrit.wikimedia.org/r/1361844 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [11:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 21.57% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:42:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:43:10] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aikochou - https://phabricator.wikimedia.org/T442367#12508101 (10MoritzMuehlenhoff) 05In progress→03Resolved a:03achou The new key has been deployed, I'm resolving the task [11:45:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json [11:45:58] (03Abandoned) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1361809 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [11:46:01] (03Merged) 10jenkins-bot: Add --listfile to transcode maint scripts [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361850 (https://phabricator.wikimedia.org/T435744) (owner: 10Ladsgroup) [11:46:08] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance [11:46:10] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json [11:46:27] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1361850|Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) [11:46:34] T435744: Bitrate for transcoded videos has gone through the roof (40mbps 480p?) - https://phabricator.wikimedia.org/T435744 [11:46:35] T373546: Migrate off HLS mov/mp4 experiment to chunked WebM MPEG-DASH - https://phabricator.wikimedia.org/T373546 [11:47:50] (03CR) 10Slyngshede: [V:03+1 C:03+2] P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [11:48:35] (03CR) 10Btullis: [C:03+2] ceph: Remove the unused osd.$HOSTNAME and client.rgw.$HOSTNAME entities [puppet] - 10https://gerrit.wikimedia.org/r/1361816 (https://phabricator.wikimedia.org/T437717) (owner: 10Btullis) [11:49:18] (03CR) 10Muehlenhoff: [C:03+2] admin: Add second new HW backed SSH key for addshore & rm old [puppet] - 10https://gerrit.wikimedia.org/r/1361545 (https://phabricator.wikimedia.org/T442758) (owner: 10Addshore) [11:49:29] !log cmooney@cumin1004 START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie [11:49:41] btullis: ok to merge your patch along? [11:50:05] (03CR) 10Blake: [C:03+1] "\o/" [puppet] - 10https://gerrit.wikimedia.org/r/1361808 (https://phabricator.wikimedia.org/T436272) (owner: 10Effie Mouzeli) [11:50:24] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only) [11:50:28] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only) [11:50:49] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1361850|Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:51:21] RECOVERY - Host an-worker1206 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [11:51:29] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [11:51:37] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [11:52:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json [11:56:16] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004" [11:56:19] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004" [11:56:19] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:56:30] !log cmooney@cumin1004 START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors [11:56:33] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors [11:57:25] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361850|Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s) [11:57:31] T435744: Bitrate for transcoded videos has gone through the roof (40mbps 480p?) - https://phabricator.wikimedia.org/T435744 [11:57:32] T373546: Migrate off HLS mov/mp4 experiment to chunked WebM MPEG-DASH - https://phabricator.wikimedia.org/T373546 [11:58:22] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1206 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [11:59:00] (03PS1) 10Vgutierrez: liberica: Remove include_services [puppet] - 10https://gerrit.wikimedia.org/r/1361853 [11:59:47] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12508160 (10phaultfinder) [12:01:47] (03CR) 10CI reject: [V:04-1] liberica: Remove include_services [puppet] - 10https://gerrit.wikimedia.org/r/1361853 (owner: 10Vgutierrez) [12:02:14] (03PS1) 10Filippo Giunchedi: team-wmcs: silence promql/series for CloudVirtDisabled [alerts] - 10https://gerrit.wikimedia.org/r/1361854 [12:02:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json [12:05:42] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only) [12:05:46] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only) [12:07:56] !log cmooney@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage [12:10:26] (03PS2) 10Vgutierrez: liberica: Remove include_services [puppet] - 10https://gerrit.wikimedia.org/r/1361853 [12:12:23] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json [12:12:50] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage [12:13:06] (03PS2) 10Blake: debug.json: Add k8s-mw-pretrain debug backends [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361855 (https://phabricator.wikimedia.org/T427666) [12:13:21] (03PS1) 10Bartosz Wójtowicz: ml-services: add qwen38-27b to llm to test vLLM 0.32 on MI300X [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361856 (https://phabricator.wikimedia.org/T442138) [12:13:32] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361853 (owner: 10Vgutierrez) [12:22:04] (03PS1) 10Jelto: Update staging-codfw to kubernetes 1.34 [puppet] - 10https://gerrit.wikimedia.org/r/1361859 (https://phabricator.wikimedia.org/T427069) [12:22:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json [12:22:43] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance [12:23:43] (03CR) 10Jelto: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361859 (https://phabricator.wikimedia.org/T427069) (owner: 10Jelto) [12:25:46] (03PS1) 10Muehlenhoff: Remove legacy key of jnuche [puppet] - 10https://gerrit.wikimedia.org/r/1361860 (https://phabricator.wikimedia.org/T442761) [12:28:01] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance [12:28:04] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json [12:28:51] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org [12:29:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12508314 (10phaultfinder) [12:31:00] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie [12:31:34] (03CR) 10Giuseppe Lavagetto: [C:03+1] P:wmcs::waf: Add known fingerprints file [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) (owner: 10Majavah) [12:32:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org [12:34:04] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json [12:35:23] (03PS2) 10Michael Große: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) [12:35:24] (03CR) 10Michael Große: [C:04-1] "Not ready yet. This requires Ib6a2b1fac9ed2b09031079efc81d3caf1616a28d to be in production and coordination with Product and Community Com" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [12:36:39] (03PS1) 10Ozge: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361862 (https://phabricator.wikimedia.org/T442141) [12:37:26] (03CR) 10Jelto: [C:03+1] "lgtm thank you. However we had some internal discussion of moving all of miscweb to the aux cluster (T442414). So it could make sense to d" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356269 (https://phabricator.wikimedia.org/T440988) (owner: 10Hnowlan) [12:38:56] (03PS2) 10Jelto: Update staging-codfw to kubernetes 1.34 [puppet] - 10https://gerrit.wikimedia.org/r/1361859 (https://phabricator.wikimedia.org/T427069) [12:40:15] (03CR) 10Majavah: [C:03+2] P:wmcs::waf: Add known fingerprints file [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) (owner: 10Majavah) [12:40:52] hi, blockers should have been resolved, I'm moving ahead with the train [12:41:37] (03PS1) 10TrainBranchBot: group0 to 1.48.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361863 (https://phabricator.wikimedia.org/T442221) [12:41:40] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by jnuche@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361863 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [12:41:43] (03CR) 10Jelto: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361859 (https://phabricator.wikimedia.org/T427069) (owner: 10Jelto) [12:42:31] (03Merged) 10jenkins-bot: group0 to 1.48.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361863 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [12:43:39] scap errored out, train is stopped again [12:44:06] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json [12:44:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12508382 (10phaultfinder) [12:45:43] (03CR) 10Clément Goubert: changeprop-jobqueue: Enable service mesh and mw-jobrunner listener (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [12:45:48] (03PS5) 10Scott French: changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) [12:46:32] (03PS1) 10Slyngshede: P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) [12:46:52] (03CR) 10Clément Goubert: [C:03+1] debug.json: Add k8s-mw-pretrain debug backends [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361855 (https://phabricator.wikimedia.org/T427666) (owner: 10Blake) [12:47:26] jouncebot: nowandnext [12:47:26] No deployments scheduled for the next 0 hour(s) and 12 minute(s) [12:47:26] In 0 hour(s) and 12 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1300) [12:47:38] (03CR) 10CI reject: [V:04-1] P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [12:47:48] (03CR) 10Ladsgroup: [C:03+2] Language: Avoid creating all converters just to find the parent [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361847 (https://phabricator.wikimedia.org/T443175) (owner: 10Ladsgroup) [12:47:53] (03CR) 10Ladsgroup: [C:03+2] Language: Avoid creating all converters just to find the parent [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361846 (https://phabricator.wikimedia.org/T443175) (owner: 10Ladsgroup) [12:48:05] (03CR) 10Effie Mouzeli: "tabs vs spaces I think :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361855 (https://phabricator.wikimedia.org/T427666) (owner: 10Blake) [12:48:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361847 (https://phabricator.wikimedia.org/T443175) (owner: 10Ladsgroup) [12:48:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361846 (https://phabricator.wikimedia.org/T443175) (owner: 10Ladsgroup) [12:49:02] (03PS2) 10Slyngshede: P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) [12:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12508398 (10phaultfinder) [12:49:56] (03PS3) 10Blake: debug.json: Add k8s-mw-pretrain debug backends [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361855 (https://phabricator.wikimedia.org/T427666) [12:50:10] (03CR) 10CI reject: [V:04-1] P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [12:51:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [12:51:47] (03PS3) 10Slyngshede: P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) [12:52:01] (03Merged) 10jenkins-bot: Language: Avoid creating all converters just to find the parent [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361847 (https://phabricator.wikimedia.org/T443175) (owner: 10Ladsgroup) [12:52:12] (03CR) 10Blake: "Thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361855 (https://phabricator.wikimedia.org/T427666) (owner: 10Blake) [12:52:54] (03CR) 10CI reject: [V:04-1] P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [12:53:15] (03PS1) 10Cathal Mooney: lvs1020: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361867 (https://phabricator.wikimedia.org/T440859) [12:54:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json [12:54:47] (03PS4) 10Slyngshede: P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) [12:54:51] (03Merged) 10jenkins-bot: Language: Avoid creating all converters just to find the parent [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361846 (https://phabricator.wikimedia.org/T443175) (owner: 10Ladsgroup) [12:56:07] (03CR) 10Clément Goubert: [C:03+1] changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [12:56:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [12:56:20] (03CR) 10Slyngshede: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [12:56:24] (03CR) 10Ozge: ml-services: place jina-embeddings on 24G GPU partitions (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361475 (https://phabricator.wikimedia.org/T443256) (owner: 10Gkyziridis) [12:58:11] (03CR) 10Ozge: [C:03+1] ml-services: place jina-embeddings on 24G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361475 (https://phabricator.wikimedia.org/T443256) (owner: 10Gkyziridis) [12:58:48] (03PS1) 10Muehlenhoff: Remove legacy key of kemayo [puppet] - 10https://gerrit.wikimedia.org/r/1361869 (https://phabricator.wikimedia.org/T442902) [12:58:54] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1361829 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [13:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1300) [13:01:21] (03PS3) 10JMeybohm: Update staging-codfw to kubernetes 1.34 [puppet] - 10https://gerrit.wikimedia.org/r/1361859 (https://phabricator.wikimedia.org/T427069) (owner: 10Jelto) [13:01:24] (03CR) 10JMeybohm: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361859 (https://phabricator.wikimedia.org/T427069) (owner: 10Jelto) [13:01:49] (03CR) 10Arnaudb: [C:03+2] "gitlab is now behind CDN" [cookbooks] - 10https://gerrit.wikimedia.org/r/1320977 (https://phabricator.wikimedia.org/T430655) (owner: 10Arnaudb) [13:01:53] (03CR) 10Arnaudb: [C:03+2] sre.gitlab: fix rollback crash and minor pre-existing issues [cookbooks] - 10https://gerrit.wikimedia.org/r/1320978 (https://phabricator.wikimedia.org/T430655) (owner: 10Arnaudb) [13:02:49] (03PS2) 10Cathal Mooney: lvs1020: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361867 (https://phabricator.wikimedia.org/T440859) [13:03:05] ok, I think the problem could be related to the presync not getting far enough last night [13:03:52] Amir1: I see you were trying a backport, is it ok if I take over for the train? [13:04:11] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json [13:04:15] hi, sure thing [13:04:18] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance [13:04:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json [13:04:20] thx [13:04:37] sorry, it failed and I didn't notice [13:04:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12508458 (10phaultfinder) [13:04:53] (03Merged) 10jenkins-bot: sre.gitlab.failover: check discovery CNAMEs and gitlab-ssh records [cookbooks] - 10https://gerrit.wikimedia.org/r/1320977 (https://phabricator.wikimedia.org/T430655) (owner: 10Arnaudb) [13:04:56] (03Merged) 10jenkins-bot: sre.gitlab: fix rollback crash and minor pre-existing issues [cookbooks] - 10https://gerrit.wikimedia.org/r/1320978 (https://phabricator.wikimedia.org/T430655) (owner: 10Arnaudb) [13:05:38] (03CR) 10Daniel Kertesz: [C:03+1] liberica: Remove include_services [puppet] - 10https://gerrit.wikimedia.org/r/1361853 (owner: 10Vgutierrez) [13:05:46] * jnuche sighs [13:05:56] one of the sec patches is still failing to apply... [13:06:03] (03CR) 10JMeybohm: [C:03+2] k8s::package: Don't force install new package versions [puppet] - 10https://gerrit.wikimedia.org/r/1361829 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [13:07:51] (03PS5) 10Slyngshede: P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) [13:08:04] (03CR) 10Jelto: [C:03+2] microsites: fix security blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1361823 (https://phabricator.wikimedia.org/T443305) (owner: 10Jelto) [13:08:18] (03PS1) 10Cathal Mooney: lvs1019: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361872 (https://phabricator.wikimedia.org/T440859) [13:08:39] (03PS6) 10Slyngshede: P:idp config chances to support CAS 8 [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) [13:08:50] :/ [13:09:14] (03PS1) 10Joal: Revert webrequest retention to 90 days [puppet] - 10https://gerrit.wikimedia.org/r/1361873 (https://phabricator.wikimedia.org/T437597) [13:09:16] jnuche: then is it okay if I push the wmf.1 patch? in return I try to look at the rebase of the sec patch afterwards :D [13:09:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12508507 (10phaultfinder) [13:10:12] (03CR) 10Slyngshede: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361864 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [13:10:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json [13:11:08] Amir1: deal! [13:11:16] 🤝 [13:11:22] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12508510 (10cmooney) Guys many thanks for opening the sub-tasks. Right now this is a blocker for the eqiad row a/b network migration... [13:11:51] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12508514 (10cmooney) [13:13:03] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key of kemayo [puppet] - 10https://gerrit.wikimedia.org/r/1361869 (https://phabricator.wikimedia.org/T442902) (owner: 10Muehlenhoff) [13:13:27] (03CR) 10Btullis: [C:03+2] Revert webrequest retention to 90 days [puppet] - 10https://gerrit.wikimedia.org/r/1361873 (https://phabricator.wikimedia.org/T437597) (owner: 10Joal) [13:14:04] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902#12508547 (10MoritzMuehlenhoff) 05In progress→03Resolved a:03Dzahn >>! In T442902#12503358, @DLynch wrote: > @Dzahn All seems to work, you can remove the old key when you... [13:16:13] okay, even wmf.1 fails. I have no choice but to look at the sec patch [13:17:49] 06SRE, 10LDAP-Access-Requests: Grant Access to ldap/nda for Aroraakhil - https://phabricator.wikimedia.org/T443439 (10MGerlach) 03NEW [13:18:05] 06SRE, 10LDAP-Access-Requests: Grant Access to ldap/nda for Aroraakhil - https://phabricator.wikimedia.org/T443439#12508583 (10MGerlach) [13:18:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:19:38] (03PS1) 10Muehlenhoff: Add FIDO keys for wmde-fisch and tchin [puppet] - 10https://gerrit.wikimedia.org/r/1361877 (https://phabricator.wikimedia.org/T442758) [13:19:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12508590 (10phaultfinder) [13:20:12] FIRING: [2x] ProbeDown: Service people1005:30443 has failed probes (http_security_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:20:32] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json [13:20:41] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12 [13:20:44] (03CR) 10CI reject: [V:04-1] Add FIDO keys for wmde-fisch and tchin [puppet] - 10https://gerrit.wikimedia.org/r/1361877 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [13:20:44] (03PS1) 10Klausman: profile/manifest: Bump pinned kernel for ML GPU machines [puppet] - 10https://gerrit.wikimedia.org/r/1361878 [13:20:45] 06SRE, 10LDAP-Access-Requests: Grant Access to ldap/nda for Aroraakhil - https://phabricator.wikimedia.org/T443439#12508595 (10MGerlach) [13:20:51] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet [13:21:32] (03CR) 10Ozge: [C:03+1] ml-services: add qwen38-27b to llm to test vLLM 0.32 on MI300X (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361856 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [13:22:24] (03CR) 10Klausman: [V:03+1] "PCC SUCCESS (NOOP 1 CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1361878 (owner: 10Klausman) [13:22:33] !log kamila@deploy2003 Started scap deploy-service shellbox: update to 8.5 + new image [13:22:33] !log kamila@deploy2003 scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [13:23:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:23:41] !log kamila@deploy2003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [13:23:47] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: add qwen38-27b to llm to test vLLM 0.32 on MI300X (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361856 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [13:24:17] (03PS1) 10Atsuko: airlfow-dumps: live-tracing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361881 (https://phabricator.wikimedia.org/T443252) [13:25:12] RESOLVED: [2x] ProbeDown: Service people1005:30443 has failed probes (http_security_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:25:19] (03Merged) 10jenkins-bot: ml-services: add qwen38-27b to llm to test vLLM 0.32 on MI300X [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361856 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [13:25:20] !log kamila@deploy2003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [13:25:44] (03CR) 10Brouberol: [C:03+1] airlfow-dumps: live-tracing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361881 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [13:26:28] (03CR) 10Effie Mouzeli: [C:03+1] debug.json: Add k8s-mw-pretrain debug backends [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361855 (https://phabricator.wikimedia.org/T427666) (owner: 10Blake) [13:26:28] (03CR) 10Urbanecm: [C:03+1] "LGTM" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361862 (https://phabricator.wikimedia.org/T442141) (owner: 10Ozge) [13:27:17] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [13:27:23] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [13:27:25] (03CR) 10Effie Mouzeli: [C:03+2] site.pp: add mc20[56-73] [puppet] - 10https://gerrit.wikimedia.org/r/1361808 (https://phabricator.wikimedia.org/T436272) (owner: 10Effie Mouzeli) [13:27:33] !log kamila@deploy2003 Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s) [13:27:51] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [13:28:02] (03CR) 10Atsuko: [C:03+2] airlfow-dumps: live-tracing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361881 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [13:28:06] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361885 [13:28:15] FIRING: [7x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:28:39] (03PS1) 10Dpogorzelski: kserve-llm-inference: send router services' public traffic to the Gateway [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361886 (https://phabricator.wikimedia.org/T436650) [13:28:56] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [13:29:01] (03PS1) 10Tryvix1509: Add missing groupOverrides for sysop and bot on ltwiki. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361887 (https://phabricator.wikimedia.org/T440216) [13:29:03] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361888 [13:29:32] (03Merged) 10jenkins-bot: airlfow-dumps: live-tracing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361881 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [13:29:46] issue with the patches is solved [13:29:46] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361885 (owner: 10Matthias Mullie) [13:29:49] !log jnuche@deploy2003 Started scap sync-world: testwikis to 1.48.0-wmf.3 refs T442221 [13:29:52] re-running pre-sync [13:29:55] T442221: 1.48.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T442221 [13:30:04] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361888 (owner: 10Matthias Mullie) [13:30:10] (03PS2) 10Klausman: profile/manifest: Bump pinned kernel for ML GPU machines [puppet] - 10https://gerrit.wikimedia.org/r/1361878 [13:30:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json [13:31:16] !log jayme@cumin1004 END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet [13:31:25] !log jayme@cumin1004 END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12 [13:31:38] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1361878 (owner: 10Klausman) [13:31:39] (03CR) 10Rsilvola: [V:03+2 C:03+2] security.txt: Fix link to Acknowledgments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361845 (https://phabricator.wikimedia.org/T443423) (owner: 10Samtar) [13:31:40] (03CR) 10Klausman: [V:03+1] "PCC SUCCESS (NOOP 1 CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1361878 (owner: 10Klausman) [13:32:02] (03CR) 10Klausman: [V:03+2 C:03+2] profile/manifest: Bump pinned kernel for ML GPU machines [puppet] - 10https://gerrit.wikimedia.org/r/1361878 (owner: 10Klausman) [13:32:44] (03Merged) 10jenkins-bot: security.txt: Fix link to Acknowledgments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361845 (https://phabricator.wikimedia.org/T443423) (owner: 10Samtar) [13:33:00] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12508661 (10Marostegui) Yeah, my question was more in the sense of, can we apply whatever needs to be applied (which I guess it is what you posted) and then reboot whenever would be convenient for us. Asking bec... [13:33:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:33:15] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:33:16] (03CR) 10Jaime Nuche: [C:03+1] Remove legacy key of jnuche [puppet] - 10https://gerrit.wikimedia.org/r/1361860 (https://phabricator.wikimedia.org/T442761) (owner: 10Muehlenhoff) [13:33:34] (03CR) 10Bartosz Wójtowicz: [C:03+1] kserve-llm-inference: send router services' public traffic to the Gateway [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361886 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [13:33:44] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet [13:33:46] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet [13:34:06] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet [13:34:08] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet [13:34:30] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12 [13:34:32] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361887 (https://phabricator.wikimedia.org/T440216) (owner: 10Tryvix1509) [13:34:39] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet [13:34:41] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet [13:35:00] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key of jnuche [puppet] - 10https://gerrit.wikimedia.org/r/1361860 (https://phabricator.wikimedia.org/T442761) (owner: 10Muehlenhoff) [13:35:09] (03CR) 10JMeybohm: [C:03+2] Update staging-codfw to kubernetes 1.34 [puppet] - 10https://gerrit.wikimedia.org/r/1361859 (https://phabricator.wikimedia.org/T427069) (owner: 10Jelto) [13:35:50] moritzm: ok to merge :0 [13:36:14] moritzm: ah, wasn't in your chain aparently [13:36:25] (03CR) 10Dpogorzelski: [C:03+2] kserve-llm-inference: send router services' public traffic to the Gateway [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361886 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [13:36:44] (03CR) 10Arnaudb: "I think that can be abandoned as well" [puppet] - 10https://gerrit.wikimedia.org/r/1352297 (https://phabricator.wikimedia.org/T440177) (owner: 10Dzahn) [13:36:47] I'm the Lucky Luke of Puppet deploys, merging quicker than my shadow [13:38:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:38:56] lol [13:40:37] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json [13:40:55] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance [13:41:17] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance [13:41:19] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1271 (T441633)', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json [13:42:51] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet [13:42:53] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet [13:42:56] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12 [13:43:00] (03PS2) 10Muehlenhoff: Add FIDO keys for wmde-fisch and tchin [puppet] - 10https://gerrit.wikimedia.org/r/1361877 (https://phabricator.wikimedia.org/T442758) [13:43:15] FIRING: [7x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:43:31] (03CR) 10Vgutierrez: [C:03+2] liberica: Remove include_services [puppet] - 10https://gerrit.wikimedia.org/r/1361853 (owner: 10Vgutierrez) [13:43:57] (03PS1) 10Klausman: profile/manifest: Fix brainfart in in previous commit [puppet] - 10https://gerrit.wikimedia.org/r/1361929 [13:44:10] (03CR) 10Klausman: [V:03+2 C:03+2] profile/manifest: Fix brainfart in in previous commit [puppet] - 10https://gerrit.wikimedia.org/r/1361929 (owner: 10Klausman) [13:44:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12508729 (10phaultfinder) [13:45:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1271 (T441633)', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json [13:47:29] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [13:48:15] RESOLVED: [7x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:53:22] (03PS1) 10Klausman: profile/manifest: Fix version of linux-base from previous commits [puppet] - 10https://gerrit.wikimedia.org/r/1361934 [13:53:30] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for wmde-fisch and tchin [puppet] - 10https://gerrit.wikimedia.org/r/1361877 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [13:53:34] (03CR) 10Klausman: [V:03+2 C:03+2] profile/manifest: Fix version of linux-base from previous commits [puppet] - 10https://gerrit.wikimedia.org/r/1361934 (owner: 10Klausman) [13:53:45] FIRING: [7x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:55:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json [13:56:37] jnuche: can I move forward with my patches? [13:57:18] Amir1: please stand-by, presync is still running and then I have to move the train to group0 [13:57:34] are they urgent? we could slip you in between those two steps [13:58:01] no, they are not. Just don't want to leave them around since they are merged [13:58:14] I think they'll automatically get deployed with your deploys anyway [13:58:39] (03CR) 10Tiziano Fogli: "LGTM overall, just a couple of inline comments. Thx" [alerts] - 10https://gerrit.wikimedia.org/r/1361473 (https://phabricator.wikimedia.org/T442710) (owner: 10Hnowlan) [13:58:45] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [13:58:54] yep, if you merged into them into wmf-3 [13:59:50] !log jelto@cumin1004 START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12 [14:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: May I have your attention please! UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1400) [14:00:05] Nvdtn19, vs1, Mahmoud-WMDE, and matthiasmullie: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:06] !log jelto@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet [14:00:13] o/ [14:00:19] please hang on, train is running [14:00:38] !log jelto@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet [14:02:56] Is there anyone here? Sorry but I forgot my backport yesterday, I've rescheduled and are here today. [14:03:57] jelto@cumin1004 upgrade-kubelets-inplace (PID 449130) is awaiting input [14:04:27] Nvdtn19: Let the train go first. [14:05:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97138 and previous config saved to /var/cache/conftool/dbconfig/20261103-140523-ladsgroup.json [14:06:49] !log jnuche@deploy2003 Finished scap sync-world: testwikis to 1.48.0-wmf.3 refs T442221 (duration: 37m 00s) [14:06:55] o/ [14:06:56] please wait [14:07:06] still need to deploy the train to group0 [14:12:15] * TheresNoTime is around to deploy for those that can't self-deploy (but noting the train ^) [14:13:25] ha, scratch my last message, my first attempt to go to group0 already updated the versions file and the pre-sync has moved the train all the way to group0 [14:13:38] logs look good, please go ahead with backports, and thanks for your patience [14:13:56] ack :) [14:14:04] who needs a deployer? Nvdtn19 ? [14:14:25] TheresNoTime: Yes [14:14:34] (poking Amir1 bc he was also in line) [14:15:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1271 (T441633)', diff saved to https://phabricator.wikimedia.org/P97139 and previous config saved to /var/cache/conftool/dbconfig/20261103-141525-ladsgroup.json [14:15:44] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1276.eqiad.wmnet with reason: Maintenance [14:15:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1276 (T441633)', diff saved to https://phabricator.wikimedia.org/P97140 and previous config saved to /var/cache/conftool/dbconfig/20261103-141545-ladsgroup.json [14:16:33] Amir1: can go after you if you'd like? [14:16:47] jnuche: my patches will go with your deploy so nothing on my side [14:17:41] okay I will start with Nvdtn19 and one of vs1's, and it should pick them up too? [14:18:17] my Internet connection was lost. Did I miss something? [14:19:03] vs1: I am deploying your IP cap patch along with Nvdtn19's patch [14:19:29] That's okay. wmdebug installed. [14:19:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1276 (T441633)', diff saved to https://phabricator.wikimedia.org/P97141 and previous config saved to /var/cache/conftool/dbconfig/20261103-141953-ladsgroup.json [14:20:54] !log samtar@deploy2003 Started scap sync-world: Backport for [[gerrit:1359140|Enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1361187|Lift IP cap for edit-a-thon (T443033)]] [14:25:20] !log samtar@deploy2003 nvdtn19, samtar, tryvix1509: Backport for [[gerrit:1359140|Enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1361187|Lift IP cap for edit-a-thon (T443033)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:25:28] vs1: Nvdtn19: will you need to test either of these changes? I don't believe the IP cap patch will be testable at least [14:25:46] IP cap isn't testable. [14:25:50] (ack) [14:26:09] But I'm wondering how NewUserMessage can be tested. [14:27:16] I'm not sure it can, easily - will proceed will sync (fyi Nvdtn19) [14:27:22] !log samtar@deploy2003 nvdtn19, samtar, tryvix1509: Continuing with deployment [14:28:24] I will create a new account to test whether it will send the messages. Is it fine? [14:29:17] mwdebug is a somewhat client side iirc [14:29:34] Nvdtn19: I'll let it sync and then you can do that to test [14:29:37] while NewUserMessage is a server-side function (queue job) [14:29:56] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1276', diff saved to https://phabricator.wikimedia.org/P97142 and previous config saved to /var/cache/conftool/dbconfig/20261103-142955-ladsgroup.json [14:33:10] !log samtar@deploy2003 Finished scap sync-world: Backport for [[gerrit:1359140|Enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1361187|Lift IP cap for edit-a-thon (T443033)]] (duration: 12m 16s) [14:33:22] Nvdtn19: Give it a test now :) [14:34:28] vs1: doing your other 2 patches now [14:34:34] Okay. [14:34:37] Ok, I'm now testing [14:35:10] fyi vivoy' NewUserMessage works as excepted: https://vi.wikivoyage.org/wiki/Th%E1%BA%A3o_lu%E1%BA%ADn_Th%C3%A0nh_vi%C3%AAn:5XaHyuz2uH [14:36:10] where's wikibugs [14:37:00] vs1 Wait. I configured $wgNewUserMessageOnFirstEdit to true [14:37:08] taavi: ... good question [14:37:48] !log samtar@deploy2003 Started scap sync-world: Backport for [[gerrit:1355072|Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887|Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] [14:38:44] !log klausman@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve101[2-5]*} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [14:38:47] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1012.eqiad.wmnet [14:39:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1276', diff saved to https://phabricator.wikimedia.org/P97143 and previous config saved to /var/cache/conftool/dbconfig/20261103-143957-ladsgroup.json [14:40:37] Please be aware that we are about to repool eqiad [14:40:55] slyngs: I have a deploy in progress, assuming thats fine? [14:41:29] That should not matter, he say optimistically :-) [14:41:44] :D [14:42:13] !log samtar@deploy2003 samtar, tryvix1509: Backport for [[gerrit:1355072|Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887|Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:42:27] vs1: your 2 other patches can be tested now ^ [14:42:38] Testing... [14:43:52] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1012.eqiad.wmnet [14:43:56] My patches include i18n changes, so they'll take awhile. They're low-risk, though (nothing in active use) - any change they could ride along with another to-be-deployed change to cut down on overall time? [14:44:22] (03PS2) 10Tiziano Fogli: metamonitoring: make public endpoint script aware of silenced DMS alerts [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) [14:44:30] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12509011 (10Marostegui) Sounds good Papaul. Let's apply that change and we can take care of the reboot and leave it running for some weeks to see how it behaves. [14:44:34] (03CR) 10Hnowlan: [C:03+2] "I think the rebased version of this change is safe to merge - apologies for any disruption." [puppet] - 10https://gerrit.wikimedia.org/r/1356954 (owner: 10Hnowlan) [14:44:39] probably with https://gerrit.wikimedia.org/r/c/1361447/, but Mahmoud-WMDE doesn't seem to be here for that deployment [14:45:04] taavi: oof, K-Lined? [14:45:10] uh oh [14:46:22] matthiasmullie: I'll set them merging now [14:46:30] ty! [14:48:20] !log slyngshede@cumin1004 START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] [14:48:23] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] [14:48:30] My patch has some issues. Although $wgNewUserMessageOnFirstEdit is true, it still sends welcome to the newly created account without any edits first. Idk what is happening but TheresNoTime: you can suspend my patch for review later. [14:49:15] jnuche: FYI with the patches backported, we no longer load Crimean Tatar language converter mapping on every page view \o/ https://performance.wikimedia.org/arclamp/svgs/daily/2026-11-02.excimer-wall.index.reversed.svgz?s=CrhExceptions&x=269.1&y=1106 Thanks! [14:49:35] Nvdtn19: hm, your patch is deployed now - does your patch need to be reverted or will you follow-up with a fix? [14:49:58] vs1: how is your patch testing? [14:50:00] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1276 (T441633)', diff saved to https://phabricator.wikimedia.org/P97144 and previous config saved to /var/cache/conftool/dbconfig/20261103-145000-ladsgroup.json [14:50:02] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1012.eqiad.wmnet [14:50:03] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1012.eqiad.wmnet [14:50:07] !log slyngshede@cumin1004 START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: Datacenter services switchover - T439676 [14:50:08] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [14:50:08] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1013.eqiad.wmnet [14:50:33] still testing ltwiki change [14:50:36] TheresNoTime: It needs to be reverted [14:50:43] vs1: ack [14:50:50] Nvdtn19: ack, will revert after this deploy [14:53:35] go ahead, test ok [14:53:40] !log samtar@deploy2003 samtar, tryvix1509: Continuing with deployment [14:54:51] after this deploy is done, I will do matthiasmullie's 2 patches along with Nvdtn19's revert [14:55:06] grand, thanks [14:56:38] Amir1: "we no longer load Crimean Tatar language converter mapping on every page view" One of those sentences you will only ever hear around these parts, love it :) [14:57:03] (03Merged) 10jenkins-bot: eventgate: bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362065 (https://phabricator.wikimedia.org/T440968) (owner: 10A-pizzata) [14:57:33] xD [14:57:38] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12509132 (10Papaul) @Marostegui i already did the changes on db1217 yesterday [14:58:10] (03PS1) 10JMeybohm: kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) [14:59:01] (03Merged) 10jenkins-bot: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361862 (https://phabricator.wikimedia.org/T442141) (owner: 10Ozge) [14:59:05] (03CR) 10CI reject: [V:04-1] kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [14:59:19] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12509143 (10Marostegui) Good thanks! [14:59:31] !log samtar@deploy2003 Finished scap sync-world: Backport for [[gerrit:1355072|Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887|Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s) [14:59:38] T440179: Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace - https://phabricator.wikimedia.org/T440179 [14:59:39] T440216: Enable extended confirmed user group on Lithuanian Wikipedia - https://phabricator.wikimedia.org/T440216 [14:59:50] (03PS2) 10JMeybohm: kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) [15:00:03] vs1: deployed [15:00:05] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1500) [15:00:12] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet [15:00:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362064 (owner: 10Samtar) [15:00:27] thanks! [15:01:09] (03CR) 10Jelto: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:01:30] (03Merged) 10jenkins-bot: Revert "Enable NewUserMessage for viwikivoyage" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362064 (owner: 10Samtar) [15:02:03] !log samtar@deploy2003 Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885|1.48.0-wmf.1]], [[gerrit:1361888|1.48.0-wmf.3]]), [[gerrit:1362064|Revert "Enable NewUserMessage for viwikivoyage"]] [15:02:10] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951#12509166 (10Dzahn) Key has been verified-out-of-band [15:03:59] (03CR) 10Arnaudb: [C:03+2] gerrit: raise the GerritHigh4xxRatio threshold to 25% [alerts] - 10https://gerrit.wikimedia.org/r/1361836 (owner: 10Arnaudb) [15:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12509183 (10phaultfinder) [15:04:50] 10ops-eqiad, 06SRE, 06DC-Ops, 06Machine-Learning-Team (Q2 FY2026-27): hw troubleshooting: complete power removal for ml-serve1015 - https://phabricator.wikimedia.org/T441146#12509184 (10isarantopoulos) [15:06:08] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet [15:06:09] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet [15:06:15] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet [15:06:53] (03Merged) 10jenkins-bot: gerrit: raise the GerritHigh4xxRatio threshold to 25% [alerts] - 10https://gerrit.wikimedia.org/r/1361836 (owner: 10Arnaudb) [15:07:28] (03PS2) 10Majavah: confd: Add support for authenticating to backend [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) [15:07:28] (03PS3) 10Majavah: haproxy: Move check-haproxy-map to a reusable class [puppet] - 10https://gerrit.wikimedia.org/r/1362053 [15:07:28] (03PS3) 10Majavah: P:wmcs::novaproxy: Test fetching data from requestctl [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) [15:07:29] !log cwilliams@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance [15:07:37] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12509216 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=28965ed7-dcee-4e88-aebc-b38925ce884c) set by cwilliams@cumin1004 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Maint... [15:09:50] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - T439676 [15:09:55] T439676: Day 8 - Source datacenter pooling (Thu. Nov. 5th - 2026) - https://phabricator.wikimedia.org/T439676 [15:10:06] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12509222 (10CWilliams-WMF) a:05CWilliams-WMF→03None ` Firmware Version = 7.00.60.00 Firmware Build = 27 Last Firmware Update = 02/09/2024 13:13:29 Hardware Version = 0.01 MAC Addres... [15:10:40] !log cwilliams@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance [15:10:47] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12509241 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=178fff39-7eb2-4311-ad3e-24df851b1939) set by cwilliams@cumin1004 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Maint... [15:11:11] (03PS4) 10Majavah: P:wmcs::novaproxy: Test fetching data from requestctl [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) [15:11:44] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:12:20] !log cwilliams@cumin1004 START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet [15:12:20] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet [15:15:45] (03PS4) 10Sbisson: ArticleGuidance: Enable the reference risk model in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357358 (https://phabricator.wikimedia.org/T442164) [15:16:19] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet [15:16:42] (03PS1) 10Tchanders: Enable link spam queue on Special:AbuseReview for alpha testing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362074 (https://phabricator.wikimedia.org/T443432) [15:17:58] matthiasmullie: will you want to test these changes? [15:18:15] nah [15:18:20] ack :) [15:18:32] not a big deal if there's an issue :) [15:19:34] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [15:20:06] 10SRE-Access-Requests: Add FIDO-backed SSH keys for ttaylor - https://phabricator.wikimedia.org/T443457 (10ttaylor) 03NEW [15:20:14] (03CR) 10Arnaudb: [C:03+1] remove load balancer IPs for Gitlab [dns] - 10https://gerrit.wikimedia.org/r/1361944 (https://phabricator.wikimedia.org/T430113) (owner: 10AOkoth) [15:20:44] (03CR) 10Arnaudb: [C:03+1] dns.admin: remove gitlab-addrs resource [cookbooks] - 10https://gerrit.wikimedia.org/r/1361941 (https://phabricator.wikimedia.org/T430113) (owner: 10AOkoth) [15:20:47] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [15:23:18] !log samtar@deploy2003 samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885|1.48.0-wmf.1]], [[gerrit:1361888|1.48.0-wmf.3]]), [[gerrit:1362064|Revert "Enable NewUserMessage for viwikivoyage"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:23:58] !log samtar@deploy2003 samtar, mlitn: Continuing with deployment [15:26:47] (03PS1) 10Nvdtn19: Re-enable NewUserMessage for viwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362080 (https://phabricator.wikimedia.org/T442485) [15:27:38] (03PS1) 10JMeybohm: Revert "Update staging-codfw to kubernetes 1.34" [puppet] - 10https://gerrit.wikimedia.org/r/1362081 (https://phabricator.wikimedia.org/T427069) [15:28:10] 10ops-codfw, 06SRE, 06DC-Ops: SM rail swap - sretest2010 - https://phabricator.wikimedia.org/T443460 (10RobH) 03NEW p:05Triage→03High [15:28:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362080 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [15:28:43] (03PS1) 10David Caro: node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T442893) [15:29:14] 10ops-codfw, 06SRE, 06DC-Ops: SM rail swap - sretest2010 - https://phabricator.wikimedia.org/T443460#12509373 (10RobH) a:03Papaul @papaul, this should have landed at codfw shipping earlier this week for swap! Please swap and let us know if the new rails work, as we'll change all our SM 24LFF bay hosts to... [15:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1530) [15:30:29] (03CR) 10Jelto: [C:03+1] "lgtm 😭" [puppet] - 10https://gerrit.wikimedia.org/r/1362081 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:30:39] (03CR) 10JMeybohm: [C:03+2] Revert "Update staging-codfw to kubernetes 1.34" [puppet] - 10https://gerrit.wikimedia.org/r/1362081 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [15:30:51] (03CR) 10David Caro: "Built it locally, will build it also in docker-regsitry and test in lima-kilo, but ready to review." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T442893) (owner: 10David Caro) [15:31:09] (03PS1) 10Gehel: admin: new SSH key for htriedman and matmarex, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362083 (https://phabricator.wikimedia.org/T442756) [15:32:30] !log jelto@cumin1004 END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12 [15:33:28] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1361503 (https://phabricator.wikimedia.org/T442758) (owner: 10Ottomata) [15:33:29] (03CR) 10Tryvix1509: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362080 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [15:33:30] (03CR) 10Muehlenhoff: [C:03+2] otto - add yubikey ssh public key; also remove unused key [puppet] - 10https://gerrit.wikimedia.org/r/1361503 (https://phabricator.wikimedia.org/T442758) (owner: 10Ottomata) [15:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:35:29] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 60%, RTA = 2130.15 ms [15:35:43] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.15 ms [15:35:48] (03CR) 10Muehlenhoff: [C:03+1] "Keys have been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1361550 (https://phabricator.wikimedia.org/T443359) (owner: 10Bearloga) [15:35:49] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH keys for bearloga [puppet] - 10https://gerrit.wikimedia.org/r/1361550 (https://phabricator.wikimedia.org/T443359) (owner: 10Bearloga) [15:35:51] !log samtar@deploy2003 Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885|1.48.0-wmf.1]], [[gerrit:1361888|1.48.0-wmf.3]]), [[gerrit:1362064|Revert "Enable NewUserMessage for viwikivoyage"]] (duration: 33m 48s) [15:35:56] !log jayme@cumin1004 START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14 [15:36:01] (03CR) 10Filippo Giunchedi: confd: Add support for authenticating to backend (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) (owner: 10Majavah) [15:36:11] matthiasmullie: done :) [15:36:18] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet [15:36:18] Thanks Sammy! [15:36:20] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet [15:36:31] (03CR) 10Filippo Giunchedi: [C:03+1] haproxy: Move check-haproxy-map to a reusable class [puppet] - 10https://gerrit.wikimedia.org/r/1362053 (owner: 10Majavah) [15:36:46] !log dancy@deploy2003 Installing scap version "4.311.0" for 155 host(s) [15:37:03] I kicked off a visual diff run to compare php 8.3 vs php 8.5 .. I don't expect it to cause any load issues, but just in case, FYI. [15:37:58] (03PS2) 10Ahmon Dancy: profile::puppet::agent: ignore system CAs in SSL re-anchor check [puppet] - 10https://gerrit.wikimedia.org/r/1361577 (https://phabricator.wikimedia.org/T443373) [15:39:05] (03CR) 10Ahmon Dancy: "Hi Andrew. Can you +2 this for me? It's a following to a prior change that you also approved." [puppet] - 10https://gerrit.wikimedia.org/r/1361577 (https://phabricator.wikimedia.org/T443373) (owner: 10Ahmon Dancy) [15:39:26] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:39:47] !log jayme@cumin1004 END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14 [15:40:30] (03PS1) 10Gehel: admin: new SSH key for samwalton and kcvelaga, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362084 (https://phabricator.wikimedia.org/T442756) [15:40:45] !log dancy@deploy2003 Installation of scap version "4.311.0" completed for 155 hosts [15:41:47] (03PS3) 10Cathal Mooney: lvs1020: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361867 (https://phabricator.wikimedia.org/T440859) [15:42:26] !log otto@deploy2003 Started scap deploy-service eventgate-analytics: v1.38.0 for T440968 [15:42:26] !log otto@deploy2003 scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics [15:42:31] T440968: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968 [15:43:17] (03PS2) 10Hnowlan: swagger: Make alerting message more useful [alerts] - 10https://gerrit.wikimedia.org/r/1361473 (https://phabricator.wikimedia.org/T442710) [15:43:17] (03PS1) 10Clément Goubert: basic-app: Introduce basic chart for basic apps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) [15:43:28] (03CR) 10Hnowlan: "Thanks for the suggestions!" [alerts] - 10https://gerrit.wikimedia.org/r/1361473 (https://phabricator.wikimedia.org/T442710) (owner: 10Hnowlan) [15:44:04] !log otto@deploy2003 scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics [15:44:31] !log otto@deploy2003 scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics [15:44:55] !log otto@deploy2003 scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics [15:45:41] (03PS4) 10Majavah: haproxy: Move check-haproxy-map to a reusable class [puppet] - 10https://gerrit.wikimedia.org/r/1362053 [15:46:13] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet [15:46:16] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet [15:47:04] (03PS2) 10Muehlenhoff: admin: Add FIDO SSH key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [15:47:12] (03PS2) 10Clément Goubert: basic-app: Introduce basic chart for basic apps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) [15:47:53] 10ops-codfw, 06SRE, 06DC-Ops: SM rail swap - sretest2010 - https://phabricator.wikimedia.org/T443460#12509588 (10RobH) No shipping tracking data in the ticket, so I've asked the support case for the info directly. CC'd Papaul and Jenn on the thread. [15:48:08] 10ops-codfw, 06SRE, 06DC-Ops: SM rail swap - sretest2010 - https://phabricator.wikimedia.org/T443460#12509592 (10RobH) a:05Papaul→03Jhancock.wm [15:48:42] (03CR) 10CI reject: [V:04-1] admin: Add FIDO SSH key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [15:48:52] (03PS1) 10Kamila Součková: shellbox: upgrade to PHP 8.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362086 (https://phabricator.wikimedia.org/T440468) [15:49:17] (03CR) 10Majavah: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1362053 (owner: 10Majavah) [15:51:00] (03PS3) 10Muehlenhoff: admin: Add FIDO SSH key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [15:51:03] (03CR) 10Majavah: [C:03+2] haproxy: Move check-haproxy-map to a reusable class [puppet] - 10https://gerrit.wikimedia.org/r/1362053 (owner: 10Majavah) [15:51:23] (03PS1) 10Ayounsi: BGP: implement 'dont_export' on circuits [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1362088 (https://phabricator.wikimedia.org/T439408) [15:51:42] (03PS1) 10Dpogorzelski: amd-gpu: set the compute partition mode through sysfs [puppet] - 10https://gerrit.wikimedia.org/r/1362090 (https://phabricator.wikimedia.org/T436928) [15:52:07] (03PS3) 10Majavah: confd: Add support for authenticating to backend [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) [15:52:08] (03PS5) 10Majavah: P:wmcs::novaproxy: Test fetching data from requestctl [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) [15:52:52] (03CR) 10CI reject: [V:04-1] amd-gpu: set the compute partition mode through sysfs [puppet] - 10https://gerrit.wikimedia.org/r/1362090 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [15:53:04] !log otto@deploy2003 scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics [15:53:06] (03PS4) 10Majavah: confd: Add support for authenticating to backend [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) [15:53:06] (03PS6) 10Majavah: P:wmcs::novaproxy: Test fetching data from requestctl [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) [15:53:51] !log otto@deploy2003 Finished scap deploy-service eventgate-analytics: v1.38.0 for T440968 (duration: 11m 24s) [15:53:55] T440968: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968 [15:54:04] (03PS2) 10Dpogorzelski: amd-gpu: set the compute partition mode through sysfs [puppet] - 10https://gerrit.wikimedia.org/r/1362090 (https://phabricator.wikimedia.org/T436928) [15:54:12] (03CR) 10CI reject: [V:04-1] confd: Add support for authenticating to backend [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) (owner: 10Majavah) [15:54:39] (03CR) 10Blake: basic-app: Introduce basic chart for basic apps (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) (owner: 10Clément Goubert) [15:54:55] (03PS5) 10Majavah: confd: Add support for authenticating to backend [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) [15:54:56] (03PS7) 10Majavah: P:wmcs::novaproxy: Test fetching data from requestctl [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) [15:56:00] (03CR) 10Majavah: confd: Add support for authenticating to backend (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) (owner: 10Majavah) [15:56:55] (03CR) 10Klausman: [C:03+1] amd-gpu: set the compute partition mode through sysfs [puppet] - 10https://gerrit.wikimedia.org/r/1362090 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [15:57:06] (03CR) 10Dpogorzelski: [C:03+2] amd-gpu: set the compute partition mode through sysfs [puppet] - 10https://gerrit.wikimedia.org/r/1362090 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [15:58:24] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Q1:rack/setup/install mc20[56-73] - https://phabricator.wikimedia.org/T436272#12509661 (10Jhancock.wm) those look good to me, ty! [15:59:03] jouncebot: nowandnext [15:59:03] For the next 0 hour(s) and 0 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1530) [15:59:03] In 0 hour(s) and 0 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1600) [15:59:10] FIRING: [2x] BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:00:05] jelto, arnoldokoth, mutante, and arnaudb: May I have your attention please! SRE Collaboration Services office hours. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1600) [16:01:52] (03CR) 10Clément Goubert: basic-app: Introduce basic chart for basic apps (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) (owner: 10Clément Goubert) [16:02:17] (03PS2) 10Kamila Součková: shellbox: upgrade to PHP 8.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362086 (https://phabricator.wikimedia.org/T440468) [16:04:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-magru and fe80::b6f9:5d02:e530:c943 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:04:26] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:05:00] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet [16:05:01] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet [16:05:06] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet [16:05:27] !log ozge@deploy2003 helmfile [staging] START helmfile.d/services/linkrecommendation: apply [16:05:46] (03PS1) 10Ottomata: changeprop-jobqueue: Add rule for edit_suggestions_counts LAC prefetch [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362092 (https://phabricator.wikimedia.org/T443442) [16:06:12] (03CR) 10Btullis: [C:03+2] admin: new SSH key for htriedman and matmarex, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362083 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:06:13] (03PS1) 10Ottomata: Prefetch edit_suggestions_counts with EventBus LinkedArtifacts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362093 (https://phabricator.wikimedia.org/T443442) [16:06:37] !log ozge@deploy2003 helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [16:06:54] (03PS2) 10Ottomata: Prefetch edit_suggestions_counts with EventBus LinkedArtifacts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362093 (https://phabricator.wikimedia.org/T443442) [16:07:18] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work, 06Machine-Learning-Team (Q2 FY2026-27): QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12509695 (10Papaul) We are still working with SM, they requested to provide them with some log files and screenshots, which we did y... [16:08:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:08:56] !log ozge@deploy2003 helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply [16:10:13] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet [16:10:36] !log ozge@deploy2003 helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:58] !log ozge@deploy2003 helmfile [codfw] START helmfile.d/services/linkrecommendation: apply [16:13:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:13:09] (03PS1) 10Ottomata: changeprop-jobqueue: Remove editSuggestionCountsPrecompute rule [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362096 (https://phabricator.wikimedia.org/T443442) [16:13:30] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1359873 (https://phabricator.wikimedia.org/T442725) (owner: 10Alex.sanford) [16:13:32] (03CR) 10Muehlenhoff: [C:03+2] Update SSH keys for alexsanford [puppet] - 10https://gerrit.wikimedia.org/r/1359873 (https://phabricator.wikimedia.org/T442725) (owner: 10Alex.sanford) [16:13:40] !log ozge@deploy2003 helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply [16:15:41] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for alexsanford - https://phabricator.wikimedia.org/T442725#12510704 (10MoritzMuehlenhoff) 05Open→03Resolved a:03ASanford-WMF The keys have been merged into Puppet [16:15:47] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) (owner: 10Majavah) [16:16:12] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet [16:16:13] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet [16:16:13] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve101[2-5]*} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [16:16:17] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q3:rack/setup/install ms-fe202[5-8] - https://phabricator.wikimedia.org/T439292#12510848 (10MatthewVernon) a:05MatthewVernon→03None I've checked `modules/profile/data/profile/installserver/preseed.yaml` and ms-fe202[1-9] is alread... [16:17:31] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q3:rack/setup/install ms-fe102[5-8] - https://phabricator.wikimedia.org/T439293#12510976 (10MatthewVernon) a:05MatthewVernon→03None I've checked `modules/profile/data/profile/installserver/preseed.yaml` and ms-fe102[1-9] is alread... [16:17:50] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9593/co" [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) (owner: 10Majavah) [16:19:05] (03CR) 10Majavah: [V:03+1 C:03+2] confd: Add support for authenticating to backend [puppet] - 10https://gerrit.wikimedia.org/r/1356966 (https://phabricator.wikimedia.org/T438971) (owner: 10Majavah) [16:19:35] (03CR) 10Btullis: [C:03+2] admin: new SSH key for samwalton and kcvelaga, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362084 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [16:20:09] (03PS1) 10Muehlenhoff: Add FIDO keys for aranyap and astein [puppet] - 10https://gerrit.wikimedia.org/r/1362099 (https://phabricator.wikimedia.org/T443041) [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:22:44] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for aranyap and astein [puppet] - 10https://gerrit.wikimedia.org/r/1362099 (https://phabricator.wikimedia.org/T443041) (owner: 10Muehlenhoff) [16:23:05] btullis: I'll merge your patch along [16:24:24] Many thanks. [16:24:37] !log otto@deploy2003 Started scap deploy-service eventgate-all: v1.38.0 for T440968 [16:24:37] !log otto@deploy2003 scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [16:24:41] T440968: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968 [16:25:04] 10SRE-Access-Requests: Add FIDO-backed SSH keys for dang - https://phabricator.wikimedia.org/T443471 (10dang) 03NEW [16:26:38] !log otto@deploy2003 scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [16:26:54] !log dancy@deploy2003 Installing scap version "4.312.0" for 2 host(s) [16:27:08] !log otto@deploy2003 scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [16:27:35] !log otto@deploy2003 scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [16:28:12] (03PS1) 10Jdlrobson: styles: Restore style to Special:MobileOptions [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362101 (https://phabricator.wikimedia.org/T363032) [16:28:58] !log dancy@deploy2003 Installation of scap version "4.312.0" completed for 2 hosts [16:29:05] 10SRE-Access-Requests: Add FIDO-backed SSH keys for dang - https://phabricator.wikimedia.org/T443471#12511106 (10dang) 05Open→03Declined I'm in the oct 2026 migration effort; I shouldn't have created this ticket [16:29:09] (03PS2) 10Ayounsi: BGP: implement 'dont_export' on circuits [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1362088 (https://phabricator.wikimedia.org/T439408) [16:29:31] (03Abandoned) 10Jdlrobson: Revert "Remove mobile watchlist styles" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361723 (owner: 10Jdlrobson) [16:30:02] ottomata: I accidentally installed a new release of scap while your deployment was active. If it goes crazy, just restart it. [16:36:25] !log otto@deploy2003 scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [16:36:57] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet [16:39:20] !log otto@deploy2003 Finished scap deploy-service eventgate-all: v1.38.0 for T440968 (duration: 14m 43s) [16:39:24] T440968: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968 [16:40:57] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet [16:43:41] (03PS1) 10Muehlenhoff: Add FIDO keys for dbrant [puppet] - 10https://gerrit.wikimedia.org/r/1362107 (https://phabricator.wikimedia.org/T442758) [16:43:45] (03PS1) 10Kosta Harlan: hCaptcha: Allow hCaptcha for Translate editor API edits [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362108 (https://phabricator.wikimedia.org/T442735) [16:44:12] (03CR) 10Lerickson: wdqs: add WDQS v2 (qlever) alerts (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) (owner: 10Gmodena) [16:44:31] (03CR) 10BCornwall: "I would love to get this merged in eventually but as it currently stands we sadly cannot without providing similar Prometheus checks with " [puppet] - 10https://gerrit.wikimedia.org/r/1344231 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [16:44:46] (03CR) 10BCornwall: [C:04-2] haproxy: remove check_procs check in favour of native systemd alerting [puppet] - 10https://gerrit.wikimedia.org/r/1344231 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [16:45:27] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1362107 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:45:29] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for dbrant [puppet] - 10https://gerrit.wikimedia.org/r/1362107 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:46:44] (03PS1) 10Kevin Bazira: services_proxy: raise tts-section-generator timeouts to 30 minutes [puppet] - 10https://gerrit.wikimedia.org/r/1362111 (https://phabricator.wikimedia.org/T441983) [16:50:40] (03CR) 10Klausman: [C:03+1] services_proxy: raise tts-section-generator timeouts to 30 minutes [puppet] - 10https://gerrit.wikimedia.org/r/1362111 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [16:51:31] (03CR) 10Kamila Součková: [C:03+2] shellbox: upgrade to PHP 8.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362086 (https://phabricator.wikimedia.org/T440468) (owner: 10Kamila Součková) [16:52:53] (03PS1) 10Muehlenhoff: Record LDAP access for gaiarajan [puppet] - 10https://gerrit.wikimedia.org/r/1362112 [16:53:03] (03Merged) 10jenkins-bot: shellbox: upgrade to PHP 8.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362086 (https://phabricator.wikimedia.org/T440468) (owner: 10Kamila Součková) [16:54:46] (03Abandoned) 10Kamila Součková: shellbox: pilot 1 replica on 8.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1355124 (https://phabricator.wikimedia.org/T440468) (owner: 10Kamila Součková) [16:56:51] 10ops-codfw, 06SRE, 10SRE-swift-storage, 10Ceph, and 2 others: Q1:rack/setup/install apus-be200[7-9] - https://phabricator.wikimedia.org/T436180#12511361 (10MatthewVernon) a:05MatthewVernon→03None OK, I've gone through the setup for this task and T436181, and I think we're good to go as-is: `manifests/... [16:58:05] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for gaiarajan [puppet] - 10https://gerrit.wikimedia.org/r/1362112 (owner: 10Muehlenhoff) [16:58:25] !log kamila@deploy2003 Started scap deploy-service shellbox: bump also shellbox score to php 8.5 [16:58:25] !log kamila@deploy2003 scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [16:58:55] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 10Ceph, and 2 others: Q1:rack/setup/install apus-be100[7-9] - https://phabricator.wikimedia.org/T436181#12511382 (10MatthewVernon) a:05MatthewVernon→03None I've checked our current preseed setup (more detailed notes in T436180), and I think we're good-to-go from... [16:59:14] !log kamila@deploy2003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [17:00:05] jhathaway and rzl: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Puppet request window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1700). [17:00:05] No Gerrit patches in the queue for this window AFAICS. [17:00:23] (03PS1) 10Gehel: admin: new SSH key for jiawang, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362114 (https://phabricator.wikimedia.org/T442756) [17:00:35] !log kamila@deploy2003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [17:01:25] (03CR) 10Btullis: [C:03+2] admin: new SSH key for jiawang, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362114 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [17:01:27] (03CR) 10Btullis: [V:03+2 C:03+2] admin: new SSH key for jiawang, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362114 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [17:01:29] (03PS3) 10JMeybohm: kubelet: --register-schedulable flag has been removed in 1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1362069 (https://phabricator.wikimedia.org/T427069) [17:01:37] (03CR) 10Btullis: [C:03+2] admin: new SSH key for jiawang, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362114 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [17:03:17] !log kamila@deploy2003 Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s) [17:04:14] (03PS2) 10Tchanders: Enable link spam queue on Special:AbuseReview for alpha testing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362074 (https://phabricator.wikimedia.org/T443432) [17:04:18] (03PS1) 10Sbisson: Load ArticleGuidance on all Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362118 (https://phabricator.wikimedia.org/T442035) [17:05:02] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362118 (https://phabricator.wikimedia.org/T442035) (owner: 10Sbisson) [17:05:09] (03PS1) 10Kamila Součková: Revert "scap: use php8.3 multiversion for scap php operations" [puppet] - 10https://gerrit.wikimedia.org/r/1362119 [17:05:35] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357358 (https://phabricator.wikimedia.org/T442164) (owner: 10Sbisson) [17:06:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [17:06:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355485 (https://phabricator.wikimedia.org/T437054) (owner: 10Thiemo Kreuz (WMDE)) [17:09:13] (03CR) 10Hnowlan: metamonitoring: make public endpoint script aware of silenced DMS alerts (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) (owner: 10Tiziano Fogli) [17:12:35] (03PS3) 10Clément Goubert: basic-app: Introduce basic chart for basic apps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) [17:12:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:13:01] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps: Rerack mc1057, mc1061, mc1069 - https://phabricator.wikimedia.org/T443209#12511527 (10VRiley-WMF) Hey @jijiki We would love to do this. However, as it turns out, we are still working on the A/B switch upgrade and A5 doesn't have that capability yet. [17:13:03] (03PS1) 10Jforrester: tables-catalogue: List abstractwiki_integrations [puppet] - 10https://gerrit.wikimedia.org/r/1362124 (https://phabricator.wikimedia.org/T443295) [17:14:06] (03CR) 10Cathal Mooney: [C:03+1] network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [17:15:47] (03CR) 10Btullis: [C:03+2] airflow-main: add phabricator token secret [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360500 (https://phabricator.wikimedia.org/T434047) (owner: 10Aqu) [17:17:37] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:17:39] (03PS1) 10AKhatun: revision feature counts: increase timeout 20s to 30s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362126 (https://phabricator.wikimedia.org/T443482) [17:17:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:17:42] (03Merged) 10jenkins-bot: airflow-main: add phabricator token secret [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360500 (https://phabricator.wikimedia.org/T434047) (owner: 10Aqu) [17:17:46] jouncebot: nowandnext [17:17:46] For the next 0 hour(s) and 42 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1700) [17:17:46] In 0 hour(s) and 42 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1800) [17:17:50] Going to use scap shortly [17:18:06] nothing in the puppet window, although I might use the infra window [17:18:20] (depends on whether I can get my stuff together beforehand) [17:18:34] (03CR) 10AKhatun: "Do we need any other chnages? In metrics, etc? I don't think so?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362126 (https://phabricator.wikimedia.org/T443482) (owner: 10AKhatun) [17:20:10] (03CR) 10Clément Goubert: basic-app: Introduce basic chart for basic apps (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) (owner: 10Clément Goubert) [17:20:34] (03PS4) 10Clément Goubert: basic-app: Introduce basic chart for basic apps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) [17:21:53] (03CR) 10Ssingh: "Removing my -1." [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [17:22:29] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:23:54] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1187.mgmt:22 - https://phabricator.wikimedia.org/T443184#12511629 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Reseated cable [17:24:20] (03PS6) 10Blake: changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [17:24:33] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:24:40] (03CR) 10Blake: changeprop-jobqueue: Enable service mesh and mw-jobrunner listener (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [17:25:16] (03CR) 10Ssingh: [C:03+1] "Looks good. Note that these IPs have to be removed from Netbox as well and the Netbox DNS cookbook has to be run. Let us know if you need " [dns] - 10https://gerrit.wikimedia.org/r/1361944 (https://phabricator.wikimedia.org/T430113) (owner: 10AOkoth) [17:26:58] (03PS3) 10Blake: changeprop: Enable service mesh and mw-api-int-async listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352075 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [17:27:10] (03PS4) 10Blake: changeprop: Enable service mesh and mw-api-int-async listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352075 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [17:29:44] (03PS1) 10ArielGlenn: Add my public key to mediawiki.org/keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362130 (https://phabricator.wikimedia.org/T441081) [17:30:15] 10ops-codfw, 06SRE, 06DC-Ops: SM rail swap - sretest2010 - https://phabricator.wikimedia.org/T443460#12511681 (10Jhancock.wm) re offline convo: checked at the dock. they haven't gotten it yet but will email when they do. [17:30:44] (03CR) 10Ssingh: [C:03+1] "Given that we have three services pending for move to IPIP and that we don't have a clear timeline on when they will be moved, and as long" [puppet] - 10https://gerrit.wikimedia.org/r/1361867 (https://phabricator.wikimedia.org/T440859) (owner: 10Cathal Mooney) [17:32:24] (03CR) 10ArielGlenn: "Someone should verify my key out of band. I am available for google meet, poke on slack as needed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362130 (https://phabricator.wikimedia.org/T441081) (owner: 10ArielGlenn) [17:32:27] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:33:19] (03PS1) 10Santiago Faci: Deploy Test Kitchen UI v2.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362133 (https://phabricator.wikimedia.org/T405467) [17:35:21] (03PS1) 10Jdlrobson: downloadPageAction: Add the download link before the bookmark [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362134 (https://phabricator.wikimedia.org/T443387) [17:38:16] (03CR) 10Ssingh: [C:03+1] lvs1019: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361872 (https://phabricator.wikimedia.org/T440859) (owner: 10Cathal Mooney) [17:39:12] (03CR) 10Ottomata: [C:03+1] revision feature counts: increase timeout 20s to 30s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362126 (https://phabricator.wikimedia.org/T443482) (owner: 10AKhatun) [17:40:28] (03CR) 10Ottomata: [C:03+1] "no should be fine!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362126 (https://phabricator.wikimedia.org/T443482) (owner: 10AKhatun) [17:42:45] (03PS3) 10Tchanders: Enable link spam queue on Special:AbuseReview for alpha testing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362074 (https://phabricator.wikimedia.org/T443432) [17:44:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tchanders@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362074 (https://phabricator.wikimedia.org/T443432) (owner: 10Tchanders) [17:45:47] (03Merged) 10jenkins-bot: Enable link spam queue on Special:AbuseReview for alpha testing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362074 (https://phabricator.wikimedia.org/T443432) (owner: 10Tchanders) [17:46:21] !log tchanders@deploy2003 Started scap sync-world: Backport for [[gerrit:1362074|Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] [17:46:26] T443432: AbuseReview: Rollout for link spam classifier - https://phabricator.wikimedia.org/T443432 [17:47:37] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:49:54] (03CR) 10Cathal Mooney: [C:03+1] "LGTM! Very nice work. Sorry it took some time to parse it all and work out what was going on fully. Can't see any flaws in the logic re" [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [17:50:37] !log tchanders@deploy2003 tchanders: Backport for [[gerrit:1362074|Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:56:28] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance [17:56:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json [17:57:16] (03PS1) 10Daimona Eaytoy: CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362142 (https://phabricator.wikimedia.org/T440359) [17:57:57] Krinkle: https://www.mediawiki.org/w/index.php?title=Template:Main_page&diff=next&oldid=8691791 this edit broke the test servers checks for scap [17:58:07] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12511969 (10Jhancock.wm) confirmed it's safe to update firmware. then sending it off to dell. cause that will be the first thing they ask us to do. [17:58:35] It's looking for the "MediaWiki is a collaboration and documentation platform" title and for some reason it no longer finds it [17:59:25] !log tchanders@deploy2003 tchanders: Continuing with deployment [17:59:56] Dreamy_Jazz: https://gerrit.wikimedia.org/g/operations/puppet/+/bf2928c0348179df9d94be5da3b8005e739824cc/modules/profile/files/httpbb/appserver/test_main.yaml#68 [17:59:59] Dreamy_Jazz: not exactly [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1800) [18:00:12] https://www.mediawiki.org/w/index.php?title=MediaWiki%3AMainpage-title&diff=8691994&oldid=5003672 [18:00:14] this edit did [18:00:25] Ah :D [18:00:51] reverted [18:04:33] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json [18:05:02] Thanks! [18:05:21] !log tchanders@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362074|Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s) [18:05:25] T443432: AbuseReview: Rollout for link spam classifier - https://phabricator.wikimedia.org/T443432 [18:06:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.792% from codfw to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:06:10] (03CR) 10Clare Ming: [C:03+2] Deploy Test Kitchen UI v2.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362133 (https://phabricator.wikimedia.org/T405467) (owner: 10Santiago Faci) [18:07:33] (03Merged) 10jenkins-bot: Deploy Test Kitchen UI v2.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362133 (https://phabricator.wikimedia.org/T405467) (owner: 10Santiago Faci) [18:07:58] Any concerns with deploying mw-config in infra window? cc: @ottomata [18:08:09] deploy hosts need a reboot, anyone using them or can I use the infra window? [18:08:20] akhatun: you can go first [18:08:26] Thanks! [18:10:48] (03CR) 10TrainBranchBot: [C:03+2] "Approved by akhatun@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362126 (https://phabricator.wikimedia.org/T443482) (owner: 10AKhatun) [18:11:34] (03Merged) 10jenkins-bot: revision feature counts: increase timeout 20s to 30s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362126 (https://phabricator.wikimedia.org/T443482) (owner: 10AKhatun) [18:11:56] !log akhatun@deploy2003 Started scap sync-world: Backport for [[gerrit:1362126|revision feature counts: increase timeout 20s to 30s (T443482)]] [18:12:01] T443482: Increase revision feature counts timeout - https://phabricator.wikimedia.org/T443482 [18:12:49] (03PS1) 10Kamila Součková: Configure cookie-based enrollment in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362144 (https://phabricator.wikimedia.org/T441689) [18:14:33] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:14:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json [18:15:08] jouncebot: nowandnext [18:15:08] For the next 0 hour(s) and 44 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1800) [18:15:08] In 0 hour(s) and 44 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1900) [18:15:30] Amir1: theres a backport ongoing then deploy hosts are going to reboot [18:15:43] noted, mine is not urgent [18:16:01] !log akhatun@deploy2003 akhatun: Backport for [[gerrit:1362126|revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:16:10] FIRING: [2x] BFDdown: BFD session down between cr1-magru and 195.200.68.161 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:16:47] !log akhatun@deploy2003 akhatun: Continuing with deployment [18:17:37] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:17:55] !log cjming@deploy2003 Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging [18:17:55] !log cjming@deploy2003 scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next [18:18:42] (I'll save my infra-window thing for later, belatedly) [18:18:58] !log cjming@deploy2003 Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s) [18:20:39] (03PS1) 10Eevans: linked-artifacts: raise revision_feature_counts timeout to 20s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362147 (https://phabricator.wikimedia.org/T443482) [18:21:10] RESOLVED: [2x] BFDdown: BFD session down between cr1-magru and 195.200.68.161 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:21:14] (03CR) 10Ahmon Dancy: [C:04-1] Revert "scap: use php8.3 multiversion for scap php operations" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1362119 (owner: 10Kamila Součková) [18:22:29] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:22:37] !log akhatun@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362126|revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s) [18:22:41] T443482: Increase revision feature counts timeout - https://phabricator.wikimedia.org/T443482 [18:22:45] @Raine: I'm done with deployment. Thanks all! [18:22:46] off I go then [18:22:48] thanks! [18:22:57] (03CR) 10AKhatun: [C:03+1] "TY! MW is deployed. You can deploy this." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362147 (https://phabricator.wikimedia.org/T443482) (owner: 10Eevans) [18:23:12] !log kamila@cumin1004 START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet [18:23:43] (03CR) 10Eevans: [C:03+2] linked-artifacts: raise revision_feature_counts timeout to 20s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362147 (https://phabricator.wikimedia.org/T443482) (owner: 10Eevans) [18:24:37] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json [18:25:02] (03Merged) 10jenkins-bot: linked-artifacts: raise revision_feature_counts timeout to 20s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362147 (https://phabricator.wikimedia.org/T443482) (owner: 10Eevans) [18:25:12] (03CR) 10Cathal Mooney: "Overall LGTM very nice! A few more high-level comments below but I'm not as such opposed to the "toggle" option." [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1360446 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [18:26:20] (03CR) 10David Caro: "Tested it in lima-kilo:" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T442893) (owner: 10David Caro) [18:26:37] (03CR) 10Cathal Mooney: "Nice work! LGTM overall. High-level question about how we approach setting the export policies, direct from Netbox versus a default ther" [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [18:27:25] !log eevans@deploy2003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [18:27:42] !log eevans@deploy2003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [18:28:08] !log eevans@deploy2003 helmfile [codfw] START helmfile.d/services/linked-artifacts: apply [18:28:26] !log eevans@deploy2003 helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply [18:28:34] !log eevans@deploy2003 helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply [18:28:50] !log eevans@deploy2003 helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply [18:29:14] (03PS2) 10David Caro: node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) [18:29:17] RECOVERY - Check unit status of kube-reload-certs on deploy2003 is OK: OK: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:29:20] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362144 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [18:29:43] !log cjming@deploy2003 Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production [18:29:43] !log cjming@deploy2003 scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen [18:31:55] !log cjming@deploy2003 Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s) [18:32:27] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:33:29] !log kamila@cumin1004 START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet [18:34:21] RECOVERY - Check unit status of fundraising-data-import on deploy2003 is OK: OK: Status of the systemd unit fundraising-data-import https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:34:31] !log kamila@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet [18:34:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json [18:34:57] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance [18:34:59] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json [18:36:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from codfw to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:37:37] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:38:35] Krinkle: we can always update that httpbb test, it's only there to make sure we're serving the right website :P if you want to keep the language in https://www.mediawiki.org/w/index.php?title=MediaWiki:Mainpage-title&oldid=8691994 then I'm happy to fix the test [18:39:33] Hm. I'd rather remove it or have it test something different on the page [18:39:43] thre's already htree other pageviews there. I'm not sure why it's testing that sentence [18:40:19] if we want to test MainPage logic I suppose we could test for "wgIsMainPage" [18:41:01] IIRC when we wrote httpbb, we started out by just reusing the same list of URLs we had in the old apache tests -- but those just had URLs, nothing about the responses, so I grabbed some text out of the body that looked relatively unlikely to change :) got away with it until now [18:41:20] e.g. for the enwiki main page I think we look for "wikipedia, the free encyclopedia" [18:41:43] wgIsMainPage seems reasonable [18:43:02] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json [18:43:14] WE do have
From mediawiki.org
[18:43:24] which is on all pageviews, not just the main page [18:44:53] nod [18:46:01] I think part of the thought process was that we might be more likely to make code changes that affected the HTML, so I stuck to English phrases that would probably stick around longer? (and guessed wrong in this case, obviously) [18:46:16] but we're reaching the limit of where the historical context makes any difference. I think either approach is probably fine [18:46:39] !log kamila@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet [18:49:21] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9] [18:49:25] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [18:49:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12512379 (10phaultfinder) [18:50:13] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [18:51:05] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9] [18:52:00] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12512398 (10Jhancock.wm) SR: 233767967 [18:53:04] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json [18:53:43] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [18:58:43] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:00:05] thcipriani and thcipriani: gettimeofday() says it's time for MediaWiki train - Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T1900) [19:03:06] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json [19:05:55] uhhh... /me updates calendar [19:06:06] (03CR) 10Volans: "Approach looks good, couple of questions/doubts inline." [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) (owner: 10Majavah) [19:09:05] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet [19:09:52] (03CR) 10Kamila Součková: [C:03+2] deployment_server/mw-*: switch next and migration to PHP 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1361437 (https://phabricator.wikimedia.org/T439955) (owner: 10Kamila Součková) [19:11:02] !log dancy@deploy2003 Installing scap version "4.313.0" for 2 host(s) [19:13:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json [19:13:26] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance [19:13:28] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json [19:14:28] !log dancy@deploy2003 Installing scap version "4.313.0" for 2 host(s) [19:16:25] !log dancy@deploy2003 Installing scap version "4.313.0" for 1 host(s) [19:17:56] !log dancy@deploy2003 Installing scap version "4.313.0" for 1 host(s) [19:18:42] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [19:19:34] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [19:20:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json [19:24:35] Raine: Is there any maintenance happening on deploy1003 ? [19:24:54] oh ffs I forgot to arm keyholder again didn't I [19:25:03] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [19:25:05] Seems keyholderish [19:25:17] (T443505) [19:25:18] T443505: scap install-world rsync failing on deploy1003.eqiad.wmnet - https://phabricator.wikimedia.org/T443505 [19:25:20] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [19:26:57] dancy: sorry, fixed :-/ [19:27:16] !log dancy@deploy2003 Installing scap version "4.313.0" for 2 host(s) [19:29:00] !log dancy@deploy2003 Installation of scap version "4.313.0" completed for 2 hosts [19:29:06] Raine: Looks good. Thanks! [19:29:33] (03PS10) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (https://phabricator.wikimedia.org/T440168) [19:30:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json [19:30:59] (03CR) 10Andriy.v: [C:03+1] ukwiki: limit "user" namespace to only registered users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [19:32:01] (03PS1) 10Kamila Součková: mw-web: Scale next release to 10% of main [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362162 (https://phabricator.wikimedia.org/T441689) [19:32:46] (03Abandoned) 10Ssingh: dns1005|dns[25]004: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1326341 (https://phabricator.wikimedia.org/T401832) (owner: 10Ssingh) [19:33:46] (03CR) 10Ssingh: [C:03+1] dns.admin: remove gitlab-addrs resource [cookbooks] - 10https://gerrit.wikimedia.org/r/1361941 (https://phabricator.wikimedia.org/T430113) (owner: 10AOkoth) [19:37:21] (03CR) 10Ssingh: "Thanks for checking; the order does not matter much as long as they are both merged together." [puppet] - 10https://gerrit.wikimedia.org/r/1361938 (https://phabricator.wikimedia.org/T430113) (owner: 10AOkoth) [19:37:57] (03CR) 10Ssingh: "I mean, as long as they merged one after the other, in any order and are not pending." [puppet] - 10https://gerrit.wikimedia.org/r/1361938 (https://phabricator.wikimedia.org/T430113) (owner: 10AOkoth) [19:38:52] (03PS3) 10Muehlenhoff: Remove the url-downloader legacy CNAMEs [dns] - 10https://gerrit.wikimedia.org/r/1351840 (https://phabricator.wikimedia.org/T439713) [19:40:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json [19:42:02] (03CR) 10Ssingh: [C:03+1] "Blocker has been resolved and codesearch confirms no other uses of the CNAMEs so I think this can move ahead." [dns] - 10https://gerrit.wikimedia.org/r/1351840 (https://phabricator.wikimedia.org/T439713) (owner: 10Muehlenhoff) [19:42:11] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12512624 (10ssingh) >>! In T439713#12420972, @MoritzMuehlenhoff wrote: >>>! In T439713#12416592, @fnegri wrote: >> I thin... [19:45:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:48:05] 10SRE-Access-Requests: Add FIDO-backed SSH key for sbassett - https://phabricator.wikimedia.org/T443508 (10sbassett) 03NEW [19:49:30] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH key for sbassett - https://phabricator.wikimedia.org/T443508#12512656 (10sbassett) [19:50:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:50:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json [19:50:28] (03PS1) 10Eevans: linked-artifacts: deploy Hoarde v1.5.4 (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362167 (https://phabricator.wikimedia.org/T441744) [19:50:42] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [19:50:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json [19:52:56] (03PS1) 10SBassett: admin: Add FIDO SSH key for sbassett [puppet] - 10https://gerrit.wikimedia.org/r/1362168 (https://phabricator.wikimedia.org/T443508) [19:53:05] (03PS1) 10Ottomata: changeprop-jobqueue: Give LAC revision_feature_counts prefetch its own rule [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357365 (https://phabricator.wikimedia.org/T436695) [19:55:48] (03PS1) 10Gehel: admin: new SSH key for gweld and snowick, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362170 (https://phabricator.wikimedia.org/T442756) [19:57:13] (03CR) 10Eevans: [C:03+2] linked-artifacts: deploy Hoarde v1.5.4 (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362167 (https://phabricator.wikimedia.org/T441744) (owner: 10Eevans) [19:57:34] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json [19:57:43] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH key for sbassett - https://phabricator.wikimedia.org/T443508#12512701 (10sbassett) [19:58:52] (03Merged) 10jenkins-bot: linked-artifacts: deploy Hoarde v1.5.4 (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362167 (https://phabricator.wikimedia.org/T441744) (owner: 10Eevans) [19:59:41] !log eevans@deploy2003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [19:59:59] !log eevans@deploy2003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [20:04:20] (03Abandoned) 10Ssingh: bird: support configuration of IPv6-only addresses [puppet] - 10https://gerrit.wikimedia.org/r/1238073 (owner: 10Ssingh) [20:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:07:05] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951#12512740 (10Dzahn) This task is now also part of T442756. When resolved, please also check the box in the other place. [20:07:36] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json [20:08:03] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12512743 (10Dzahn) 05Open→03In progress [20:11:16] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12512753 (10Dzahn) @aranyap I see your key has been added with the merge above. Let us know if it works for you and then we can remove the old keys and close this out. Cheers! [20:13:30] (03CR) 10Btullis: [C:03+2] admin: new SSH key for gweld and snowick, backed by FIDO2 [puppet] - 10https://gerrit.wikimedia.org/r/1362170 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [20:13:56] (03PS1) 10Codename Noreste: Round 2 on disabling createpage for anonymous users on English Wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362175 (https://phabricator.wikimedia.org/T443449) [20:17:14] I am going to check a patch which was previously tried before but abandoned due to a merge conflict, 1362175 [20:17:22] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1362175 [20:17:38] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json [20:27:40] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json [20:27:48] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance [20:27:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json [20:34:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json [20:36:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:39:55] (03CR) 10Michael Große: [C:04-1] "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [20:41:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:44:32] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json [20:44:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12512976 (10phaultfinder) [20:49:41] (03CR) 10Southparkfan: [C:03+1] "Not tested, but: well done!" [puppet] - 10https://gerrit.wikimedia.org/r/1361577 (https://phabricator.wikimedia.org/T443373) (owner: 10Ahmon Dancy) [20:53:33] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:54:34] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json [20:55:56] (03PS1) 10Ebernhardson: Revert "Reapply "cirrus: Send more_like traffic to eqiad"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362185 [20:56:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362185 (owner: 10Ebernhardson) [20:58:33] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:59:58] (03PS3) 10Michael Große: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T2100). Please do the needful. [21:00:05] sbassett, Ameisenigel, Thiemo_WMDE, and ebernhardson: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:19] o/ [21:00:20] I'm here. [21:00:53] I am here as well [21:02:04] \o [21:02:05] I just have a low-risk config patch [21:02:17] (03CR) 10SBassett: [C:03+1] Enable DBSC report-only mode on group0 and test2wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360542 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [21:02:19] mine is low risk as well [21:02:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360542 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [21:04:06] Two config patches here as well, low risk, only one is important for us to have tomorrow. [21:04:27] (03Merged) 10jenkins-bot: Enable DBSC report-only mode on group0 and test2wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360542 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [21:04:36] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json [21:04:57] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance [21:04:57] !log sbassett@deploy2003 Started scap sync-world: Backport for [[gerrit:1360542|Enable DBSC report-only mode on group0 and test2wiki (T393212)]] [21:04:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json [21:05:04] T393212: Implement Device Bound Session Credentials (DBSC) in MediaWiki - https://phabricator.wikimedia.org/T393212 [21:05:25] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12513053 (10thcipriani) +1: I asked @Southparkfan to make this request, happy to sponser. [21:09:14] !log sbassett@deploy2003 sbassett: Backport for [[gerrit:1360542|Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:09:52] !log sbassett@deploy2003 sbassett: Continuing with deployment [21:11:41] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json [21:14:31] (03CR) 10Bking: [C:03+1] Revert "Reapply "cirrus: Send more_like traffic to eqiad"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362185 (owner: 10Ebernhardson) [21:15:05] almost done [21:16:39] !log sbassett@deploy2003 Finished scap sync-world: Backport for [[gerrit:1360542|Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s) [21:16:43] T393212: Implement Device Bound Session Credentials (DBSC) in MediaWiki - https://phabricator.wikimedia.org/T393212 [21:17:05] and done [21:17:11] looks good for me [21:17:52] can everyone else self-deploy or do i need to help out with the rest of the patches? [21:18:12] I need a deployer. [21:18:39] i have deploy rights, i suspect the rest of these can be shipped together/ [21:20:26] (03PS1) 10WMDE-leszek: Restore running WMDE Analytics scripts [puppet] - 10https://gerrit.wikimedia.org/r/1362194 (https://phabricator.wikimedia.org/T443525) [21:21:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ebernhardson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361843 (https://phabricator.wikimedia.org/T443272) (owner: 10Ameisenigel) [21:21:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ebernhardson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361837 (https://phabricator.wikimedia.org/T443269) (owner: 10Ameisenigel) [21:21:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ebernhardson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [21:21:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ebernhardson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355485 (https://phabricator.wikimedia.org/T437054) (owner: 10Thiemo Kreuz (WMDE)) [21:21:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ebernhardson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362185 (owner: 10Ebernhardson) [21:21:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json [21:22:42] (03Merged) 10jenkins-bot: eswiki: add editprotected to botadmin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361843 (https://phabricator.wikimedia.org/T443272) (owner: 10Ameisenigel) [21:22:45] (03Merged) 10jenkins-bot: liwiktionary: Create Appendix namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361837 (https://phabricator.wikimedia.org/T443269) (owner: 10Ameisenigel) [21:22:51] (03Merged) 10jenkins-bot: Stop exempting item/property creation from hCaptcha [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [21:23:36] (03Merged) 10jenkins-bot: Remove WikibaseTmpMobileEditingUIBetaFeature config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355485 (https://phabricator.wikimedia.org/T437054) (owner: 10Thiemo Kreuz (WMDE)) [21:23:39] (03Merged) 10jenkins-bot: Revert "Reapply "cirrus: Send more_like traffic to eqiad"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362185 (owner: 10Ebernhardson) [21:24:04] !log ebernhardson@deploy2003 Started scap sync-world: Backport for [[gerrit:1361843|eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837|liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447|Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485|Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185|Revert "Reapply "cirrus: Send more_like [21:24:04] traffic to eqiad""]] [21:24:15] T443272: eswiki: Add editprotected right to the botadmin group for approved page protection tasks - https://phabricator.wikimedia.org/T443272 [21:24:15] T443269: Create namespace for "Appendix" on li.wiktionary - https://phabricator.wikimedia.org/T443269 [21:24:16] T440832: [hCaptcha] Enable hCaptcha on the Beta only - https://phabricator.wikimedia.org/T440832 [21:24:16] T437054: [MEX] M5: release mobile editing to all editors - https://phabricator.wikimedia.org/T437054 [21:28:34] !log ebernhardson@deploy2003 ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843|eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837|liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447|Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485|Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185|Reve [21:28:34] rt "Reapply "cirrus: Send more_like traffic to eqiad""]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:29:07] Thiemo_WMDE, Ameisenigel: look as-expected? [21:31:05] One second. For one of the patches everything looks fine. Checking the second. [21:31:45] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json [21:32:09] Loogs good to me [21:32:16] thanks! [21:35:28] My 2nd patch is for the beta cluster. So I guess I can see it only after the next step? [21:35:49] Thiemo_WMDE: hmm, sadly i don't know how beta works anymore. Seems plausible. Sounds safe to move forward? [21:36:06] it used to be a git pull on cron updating beta, but i wouldn't be surprised if that's changed [21:36:09] I'm pretty sure it's safe, yes. [21:36:13] !log ebernhardson@deploy2003 ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment [21:37:01] True, I'm told just merging the config change and waiting a few minutes should be all that's needed for the beta cluster. [21:37:21] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 635.45 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:38:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:41:47] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json [21:42:05] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance [21:42:07] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1274 (T441633)', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json [21:42:46] !log ebernhardson@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361843|eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837|liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447|Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485|Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185|Revert "Reapply "cirrus: Send more_lik [21:42:46] e traffic to eqiad""]] (duration: 18m 42s) [21:42:54] T443272: eswiki: Add editprotected right to the botadmin group for approved page protection tasks - https://phabricator.wikimedia.org/T443272 [21:42:55] T443269: Create namespace for "Appendix" on li.wiktionary - https://phabricator.wikimedia.org/T443269 [21:42:55] T440832: [hCaptcha] Enable hCaptcha on the Beta only - https://phabricator.wikimedia.org/T440832 [21:42:55] T437054: [MEX] M5: release mobile editing to all editors - https://phabricator.wikimedia.org/T437054 [21:43:03] that completes the deploy window [21:43:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:43:33] Just FYI, the Beta config change works as expected. Thanks a lot! [21:46:51] Thanks, ebernhardson ! [21:48:34] np! [21:52:34] Ameisenigel: Thanks for the es.wiki task [21:52:44] My bot is already running and working :D [21:54:21] (03CR) 10AKhatun: changeprop-jobqueue: Give LAC revision_feature_counts prefetch its own rule (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357365 (https://phabricator.wikimedia.org/T436695) (owner: 10Ottomata) [21:59:54] (03PS3) 10Krinkle: Remove Mathoid SVG option from wgMathValidModes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311062 (https://phabricator.wikimedia.org/T271001) [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261103T2200) [22:01:06] 06SRE, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review, 07Puppet (Puppet 7.0): Next steps for Puppet 7 - https://phabricator.wikimedia.org/T330490#12513281 (10Krinkle) [22:02:00] 06SRE, 06Infrastructure-Foundations, 10Puppet-Infrastructure: Puppet 8 readiness - https://phabricator.wikimedia.org/T366900#12513286 (10Krinkle) [22:03:06] o// taking over readers deployment window [22:06:43] (03PS1) 10Herron: logstash: add TitleBlacklist INFO to spam filter [puppet] - 10https://gerrit.wikimedia.org/r/1362207 [22:06:53] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362101 (https://phabricator.wikimedia.org/T363032) (owner: 10Jdlrobson) [22:06:54] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361724 (https://phabricator.wikimedia.org/T387797) (owner: 10Jdlrobson) [22:06:54] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362134 (https://phabricator.wikimedia.org/T443387) (owner: 10Jdlrobson) [22:09:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [22:12:44] (03Merged) 10jenkins-bot: styles: Restore style to Special:MobileOptions [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362101 (https://phabricator.wikimedia.org/T363032) (owner: 10Jdlrobson) [22:12:57] (03CR) 10Herron: [C:03+1] alerting: continue when alerting_hosts hits an error [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355757 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [22:13:33] (03Merged) 10jenkins-bot: Revert "footer: Hide separator before a hidden last footer place" [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1361724 (https://phabricator.wikimedia.org/T387797) (owner: 10Jdlrobson) [22:13:33] (03CR) 10Herron: [C:03+1] alerting: improve removing alertmanager silences [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355701 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [22:13:35] (03Merged) 10jenkins-bot: downloadPageAction: Add the download link before the bookmark [skins/MinervaNeue] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362134 (https://phabricator.wikimedia.org/T443387) (owner: 10Jdlrobson) [22:13:55] (03CR) 10Herron: [C:03+2] Remove Luca from Icinga contact group [puppet] - 10https://gerrit.wikimedia.org/r/1361495 (https://phabricator.wikimedia.org/T443187) (owner: 10Muehlenhoff) [22:14:03] !log jdlrobson@deploy2003 Started scap sync-world: Backport for [[gerrit:1362101|styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724|Revert "footer: Hide separator before a hidden last footer place" (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134|downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) [22:14:11] T363032: Mobile should separate out Saved reading list and editor Watchlist experiences. - https://phabricator.wikimedia.org/T363032 [22:14:11] T387797: Stray dot in footer when page previews or parsoid enabled - https://phabricator.wikimedia.org/T387797 [22:14:12] T443387: Regression: Download icon causes reflow for anonymous users - https://phabricator.wikimedia.org/T443387 [22:14:13] (03CR) 10Herron: [C:03+2] rsyslog: remove lookup entries for php7.0-fpm [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [22:15:21] (03PS1) 10Ladsgroup: Avoid unnecessary/aborted requests in DASH mode [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362212 (https://phabricator.wikimedia.org/T442988) [22:18:10] !log jdlrobson@deploy2003 jdlrobson: Backport for [[gerrit:1362101|styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724|Revert "footer: Hide separator before a hidden last footer place" (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134|downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki [22:18:10] /Mwdebug). Changes can now be verified there. [22:21:05] !log jdlrobson@deploy2003 jdlrobson: Continuing with deployment [22:24:04] (03PS1) 10DLynch: VisualEditor: Precompute edit suggestion counts on 5% of jawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362215 (https://phabricator.wikimedia.org/T442767) [22:27:28] !log jdlrobson@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362101|styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724|Revert "footer: Hide separator before a hidden last footer place" (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134|downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s) [22:27:36] T363032: Mobile should separate out Saved reading list and editor Watchlist experiences. - https://phabricator.wikimedia.org/T363032 [22:27:37] T387797: Stray dot in footer when page previews or parsoid enabled - https://phabricator.wikimedia.org/T387797 [22:27:37] T443387: Regression: Download icon causes reflow for anonymous users - https://phabricator.wikimedia.org/T443387 [22:30:12] done [22:30:28] I've got a config patch to throw in, if nobody else wants to use more of this window. [22:30:50] (03PS1) 10Southparkfan: cloud: Switch default cfssl intermediate to discovery2026 [puppet] - 10https://gerrit.wikimedia.org/r/1362216 [22:31:09] Kemayo: all yours [22:32:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362215 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [22:32:52] (03PS2) 10Southparkfan: cloud: Switch default cfssl intermediate to discovery2026 [puppet] - 10https://gerrit.wikimedia.org/r/1362216 [22:33:10] (03Merged) 10jenkins-bot: VisualEditor: Precompute edit suggestion counts on 5% of jawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362215 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [22:33:33] !log kemayo@deploy2003 Started scap sync-world: Backport for [[gerrit:1362215|VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] [22:33:38] T442767: Deploy Headless VE to a second Wikipedia - https://phabricator.wikimedia.org/T442767 [22:34:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [22:35:07] (03PS1) 10Ladsgroup: Enable chunked videos on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362217 (https://phabricator.wikimedia.org/T373546) [22:35:21] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.45 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:38:02] !log kemayo@deploy2003 kemayo: Backport for [[gerrit:1362215|VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:38:36] !log kemayo@deploy2003 kemayo: Continuing with deployment [22:44:23] !log kemayo@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362215|VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s) [22:44:28] T442767: Deploy Headless VE to a second Wikipedia - https://phabricator.wikimedia.org/T442767 [22:48:15] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [22:48:17] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json [22:53:37] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet [22:56:32] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json [22:57:09] (03PS1) 10JHathaway: Remove legacy ssh keys for jforrester [puppet] - 10https://gerrit.wikimedia.org/r/1362222 (https://phabricator.wikimedia.org/T442758) [22:59:04] (03CR) 10JHathaway: [C:03+2] Remove legacy ssh keys for jforrester [puppet] - 10https://gerrit.wikimedia.org/r/1362222 (https://phabricator.wikimedia.org/T442758) (owner: 10JHathaway) [23:01:32] (03PS1) 10JHathaway: Remove legacy ssh keys for otto [puppet] - 10https://gerrit.wikimedia.org/r/1362227 (https://phabricator.wikimedia.org/T442758) [23:02:37] (03PS1) 10JHathaway: Remove legacy ssh keys for dbrant [puppet] - 10https://gerrit.wikimedia.org/r/1362228 (https://phabricator.wikimedia.org/T442758) [23:03:20] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: hw troubleshooting: power-on for wikikube-worker2337.codfw.wmnet - https://phabricator.wikimedia.org/T442714#12513543 (10Jhancock.wm) updated the firmware but still not powering up. gonna push the button in the morning [23:03:22] (03CR) 10JHathaway: [C:03+2] Remove legacy ssh keys for otto [puppet] - 10https://gerrit.wikimedia.org/r/1362227 (https://phabricator.wikimedia.org/T442758) (owner: 10JHathaway) [23:05:22] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a7-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T443335#12513549 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm stopped alerting ~12 hours ago [23:05:31] (03CR) 10JHathaway: [C:03+2] Remove legacy ssh keys for dbrant [puppet] - 10https://gerrit.wikimedia.org/r/1362228 (https://phabricator.wikimedia.org/T442758) (owner: 10JHathaway) [23:06:00] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12513553 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm stopped alerting ~8 hours ago [23:06:34] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json [23:06:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12513559 (10Jhancock.wm) 05Open→03Resolved stopped reporting ~8 hours ago [23:07:36] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c4-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442765#12513570 (10Jhancock.wm) 05Open→03Resolved stopped alerting 2 days ago [23:08:10] (03PS1) 10JHathaway: Remove legacy ssh keys for jdlrobson [puppet] - 10https://gerrit.wikimedia.org/r/1362231 (https://phabricator.wikimedia.org/T442758) [23:08:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12513576 (10Jhancock.wm) alerting, but more level than before. might just need a little more adjustment. will check in the morning [23:09:22] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12513579 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm stopped alerting ~8 hour ago [23:10:57] (03CR) 10JHathaway: [C:03+2] Remove legacy ssh keys for jdlrobson [puppet] - 10https://gerrit.wikimedia.org/r/1362231 (https://phabricator.wikimedia.org/T442758) (owner: 10JHathaway) [23:12:19] 10ops-codfw, 10ops-eqiad, 06SRE, 06DC-Ops: Netbox: ToR and management switch report - https://phabricator.wikimedia.org/T442160#12513590 (10Jhancock.wm) [23:13:09] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 632.29 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:15:40] jouncebot: nowandnext [23:15:41] No deployments scheduled for the next 2 hour(s) and 44 minute(s) [23:15:41] In 2 hour(s) and 44 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T0200) [23:16:15] okay I'm going to deploy a few config changes [23:16:36] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json [23:17:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by musikanimal@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352303 (https://phabricator.wikimedia.org/T440077) (owner: 10MusikAnimal) [23:17:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by musikanimal@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358261 (https://phabricator.wikimedia.org/T442015) (owner: 10Amdrel) [23:17:16] I'm just gearing up for some envoy updates, but those can happen at the same time [23:17:34] okay, phew :) [23:17:55] (03Merged) 10jenkins-bot: CommonSettings: remove redundant CodeMirror config settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352303 (https://phabricator.wikimedia.org/T440077) (owner: 10MusikAnimal) [23:17:58] (03Merged) 10jenkins-bot: metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358261 (https://phabricator.wikimedia.org/T442015) (owner: 10Amdrel) [23:18:22] !log musikanimal@deploy2003 Started scap sync-world: Backport for [[gerrit:1352303|CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261|metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] [23:18:28] T440077: Enable CodeMirror for wikitext by default on all wikis - https://phabricator.wikimedia.org/T440077 [23:18:29] T442015: Wishlist Extension: Disable wishlist submission on 2026-11-09 at 9:00 PST (17:00 UTC) - https://phabricator.wikimedia.org/T442015 [23:22:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and KPN (139.156.127.121) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [23:22:46] !log musikanimal@deploy2003 musikanimal, amdrel: Backport for [[gerrit:1352303|CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261|metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:24:06] you'll see lots of log lines from me, still fine to keep going [23:24:13] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/apertium: apply [23:24:44] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/apertium: apply [23:24:52] 06SRE: Document current expected SSH fingerprint for releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T443537 (10matmarex) 03NEW [23:26:28] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/chart-renderer: apply [23:26:38] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json [23:26:53] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply [23:26:56] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [23:26:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json [23:27:02] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [23:27:06] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [23:27:07] !log musikanimal@deploy2003 musikanimal, amdrel: Continuing with deployment [23:27:18] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/citoid: apply [23:27:33] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/citoid: apply [23:27:43] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply [23:27:57] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply [23:28:09] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/cxserver: apply [23:28:30] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/cxserver: apply [23:28:40] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/data-gateway: apply [23:28:56] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/data-gateway: apply [23:29:01] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/developer-portal: apply [23:29:28] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [23:29:32] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/device-analytics: apply [23:29:56] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/device-analytics: apply [23:30:01] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/echostore: apply [23:30:22] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/echostore: apply [23:30:34] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/edit-analytics: apply [23:30:54] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply [23:31:01] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/editcheck-headless: apply [23:31:56] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply [23:32:11] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/editor-analytics: apply [23:32:27] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply [23:33:19] !log musikanimal@deploy2003 Finished scap sync-world: Backport for [[gerrit:1352303|CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261|metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s) [23:33:24] T440077: Enable CodeMirror for wikitext by default on all wikis - https://phabricator.wikimedia.org/T440077 [23:33:25] T442015: Wishlist Extension: Disable wishlist submission on 2026-11-09 at 9:00 PST (17:00 UTC) - https://phabricator.wikimedia.org/T442015 [23:33:28] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [23:33:56] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [23:34:00] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/geo-analytics: apply [23:34:12] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply [23:34:36] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/kartotherian: apply [23:35:37] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json [23:37:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and KPN (139.156.127.121) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [23:44:45] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/kartotherian: apply [23:44:55] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/lac-lambdas: apply [23:45:11] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply [23:45:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json [23:46:30] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/machinetranslation: apply [23:52:55] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [23:53:17] musikanimal: when you're done, can you let me know? [23:53:27] Amir1: all done! [23:53:39] awesome! [23:53:55] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/media-analytics: apply [23:54:06] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/media-analytics: apply [23:54:09] Amir1: I'm still charlieing the envoys but you're fine to scap [23:54:18] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/miscweb: apply [23:54:32] Awesome <3 [23:55:41] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json [23:56:29] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [23:56:35] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [23:57:15] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [23:58:28] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/page-analytics: apply [23:58:41] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/page-analytics: apply [23:59:23] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/proton: apply [23:59:24] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362212 (https://phabricator.wikimedia.org/T442988) (owner: 10Ladsgroup) [23:59:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362217 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup)