[00:08:34] (03CR) 10Elukey: "I added some folks form Observability fro the prometheus part, so we are on the same page :)" [puppet] - 10https://gerrit.wikimedia.org/r/1116888 (https://phabricator.wikimedia.org/T385530) (owner: 10BryanDavis) [00:10:41] PROBLEM - MariaDB Replica Lag: s1 on db2141 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [00:38:28] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1118253 [00:38:28] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1118253 (owner: 10TrainBranchBot) [00:49:28] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1118253 (owner: 10TrainBranchBot) [01:08:34] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1118255 [01:08:34] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1118255 (owner: 10TrainBranchBot) [01:11:39] FIRING: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic2085-production-search-psi-codfw is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency [01:30:10] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1118255 (owner: 10TrainBranchBot) [01:42:41] FIRING: [3x] SystemdUnitFailed: etcd-backup.service on aux-k8s-etcd2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:44:33] FIRING: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [01:46:29] PROBLEM - Disk space on releases1003 is CRITICAL: DISK CRITICAL - /srv/docker/overlay2/32222d459bf56569cbecc6f5af4737387e4c73b32fd714ad5946423c1b6ed3cb/merged is not accessible: Permission denied https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=releases1003&var-datasource=eqiad+prometheus/ops [02:04:33] RESOLVED: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [02:06:29] RECOVERY - Disk space on releases1003 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=releases1003&var-datasource=eqiad+prometheus/ops [02:09:41] RECOVERY - MariaDB Replica Lag: s1 on db2141 is OK: OK slave_sql_lag Replication lag: 0.19 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [02:36:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:42:49] FIRING: PuppetFailure: Puppet has failed on build2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:01:42] RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:30:39] (03PS1) 10Tim Starling: Revert "API: Use ExpiryDef for action=block expiry parameter" [core] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118262 (https://phabricator.wikimedia.org/T248196) [04:44:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tstarling@deploy2002 using scap backport" [core] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118262 (https://phabricator.wikimedia.org/T248196) (owner: 10Tim Starling) [04:54:20] (03Merged) 10jenkins-bot: Revert "API: Use ExpiryDef for action=block expiry parameter" [core] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118262 (https://phabricator.wikimedia.org/T248196) (owner: 10Tim Starling) [04:54:47] !log tstarling@deploy2002 Started scap sync-world: Backport for [[gerrit:1118262|Revert "API: Use ExpiryDef for action=block expiry parameter" (T248196)]] [04:54:50] T248196: Consolidate logic for parsing expiries - https://phabricator.wikimedia.org/T248196 [05:07:39] !log tstarling@deploy2002 tstarling: Backport for [[gerrit:1118262|Revert "API: Use ExpiryDef for action=block expiry parameter" (T248196)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [05:07:44] T248196: Consolidate logic for parsing expiries - https://phabricator.wikimedia.org/T248196 [05:09:28] !log tstarling@deploy2002 tstarling: Continuing with sync [05:11:39] FIRING: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic2085-production-search-psi-codfw is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency [05:14:07] (03PS3) 10Anzx: tcywiki: add extendedconfirmed usergroup and restriction level [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118235 (https://phabricator.wikimedia.org/T385828) [05:14:35] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118235 (https://phabricator.wikimedia.org/T385828) (owner: 10Anzx) [05:18:51] !log tstarling@deploy2002 Finished scap sync-world: Backport for [[gerrit:1118262|Revert "API: Use ExpiryDef for action=block expiry parameter" (T248196)]] (duration: 24m 04s) [05:18:55] T248196: Consolidate logic for parsing expiries - https://phabricator.wikimedia.org/T248196 [05:31:51] (03PS1) 10KartikMistry: Update cxserver to 2025-02-10-050623-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118264 (https://phabricator.wikimedia.org/T377966) [05:42:41] FIRING: [3x] SystemdUnitFailed: etcd-backup.service on aux-k8s-etcd2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:49:33] FIRING: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [06:09:33] RESOLVED: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [06:42:49] FIRING: PuppetFailure: Puppet has failed on build2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:49:59] PROBLEM - BFD status on cr2-eqdfw is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [06:50:59] RECOVERY - BFD status on cr2-eqdfw is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [08:00:05] Amir1, Urbanecm, and awight: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T0800). [08:00:05] anzx: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:00:08] o/ [08:47:50] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [08:48:27] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [08:49:28] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [08:49:50] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [09:01:57] (03CR) 10Brouberol: envoy: add the analytics-web service to the mesh (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1116760 (https://phabricator.wikimedia.org/T384329) (owner: 10Brouberol) [09:03:30] (03PS2) 10Brouberol: envoy: add the analytics-web service to the mesh [puppet] - 10https://gerrit.wikimedia.org/r/1116760 (https://phabricator.wikimedia.org/T384329) [09:05:48] (03CR) 10Brouberol: envoy: add the analytics-web service to the mesh (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1116760 (https://phabricator.wikimedia.org/T384329) (owner: 10Brouberol) [09:10:59] 10ops-eqiad, 06SRE, 06DC-Ops, 13Patch-For-Review: Decommission dbstore1005 - https://phabricator.wikimedia.org/T351925#10534567 (10Urbanecm) [09:11:39] FIRING: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic2085-production-search-psi-codfw is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency [09:11:51] anzx: are you still looking for a deployer? [09:12:15] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118235 (https://phabricator.wikimedia.org/T385828) (owner: 10Anzx) [09:12:44] urbanecm: yes [09:12:59] anzx: happy to do it now if you want [09:13:28] sure [09:13:34] (03CR) 10Urbanecm: [C:03+2] tcywiki: add extendedconfirmed usergroup and restriction level [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118235 (https://phabricator.wikimedia.org/T385828) (owner: 10Anzx) [09:13:54] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118235 (https://phabricator.wikimedia.org/T385828) (owner: 10Anzx) [09:14:18] (03Merged) 10jenkins-bot: tcywiki: add extendedconfirmed usergroup and restriction level [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118235 (https://phabricator.wikimedia.org/T385828) (owner: 10Anzx) [09:14:40] !log urbanecm@deploy2002 Started scap sync-world: Backport for [[gerrit:1118235|tcywiki: add extendedconfirmed usergroup and restriction level (T385828)]] [09:14:43] T385828: Add "extendedconfirmed" local group rights on tcy.wikipedia - https://phabricator.wikimedia.org/T385828 [09:18:27] !log urbanecm@deploy2002 urbanecm, anzx: Backport for [[gerrit:1118235|tcywiki: add extendedconfirmed usergroup and restriction level (T385828)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [09:18:30] urbanecm: checking [09:18:36] you're too quick [09:20:28] urbanecm: look ok [09:20:33] !log urbanecm@deploy2002 urbanecm, anzx: Continuing with sync [09:23:26] urbanecm: is there a maintenance script for populating user group [09:23:35] what do you mean, populating? [09:24:06] adding already eligible users to user group [09:25:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115006 (https://phabricator.wikimedia.org/T330217) (owner: 10Arthur taylor) [09:25:38] anzx: it will happen automatically, whenever they attempt to edit [09:26:04] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115013 (https://phabricator.wikimedia.org/T330217) (owner: 10Arthur taylor) [09:27:30] !log urbanecm@deploy2002 Finished scap sync-world: Backport for [[gerrit:1118235|tcywiki: add extendedconfirmed usergroup and restriction level (T385828)]] (duration: 12m 49s) [09:27:33] T385828: Add "extendedconfirmed" local group rights on tcy.wikipedia - https://phabricator.wikimedia.org/T385828 [09:27:44] anzx: done [09:27:47] sorry to keep you waiting [09:28:21] urbanecm: thanks [09:28:55] np [09:38:16] (03PS1) 10Aklapper: Remove a condition which always returns false [phabricator/antivandalism] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1118472 [09:42:41] FIRING: [3x] SystemdUnitFailed: etcd-backup.service on aux-k8s-etcd2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:54:33] FIRING: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [10:02:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/CentralAuth] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1117927 (https://phabricator.wikimedia.org/T364866) (owner: 10Gergő Tisza) [10:13:19] RECOVERY - Disk space on stat1008 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=stat1008&var-datasource=eqiad+prometheus/ops [10:14:33] RESOLVED: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [10:16:51] (03PS4) 10Jforrester: Provide a base image for Rust, based on Bookworm using 'rustc-web' now at 1.78 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1102983 (https://phabricator.wikimedia.org/T380807) [10:18:48] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Remove a condition which always returns false (032 comments) [phabricator/antivandalism] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1118472 (owner: 10Aklapper) [10:22:44] (03PS1) 10Joal: Update webrequest_sampled_live turnilo config [puppet] - 10https://gerrit.wikimedia.org/r/1118477 [10:39:26] (03PS2) 10KartikMistry: Update MinT to 2025-02-05-115716-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115314 (https://phabricator.wikimedia.org/T383750) [10:42:49] FIRING: PuppetFailure: Puppet has failed on build2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:49:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1116812 (owner: 10Lucas Werkmeister (WMDE)) [10:57:37] (03PS1) 10Lucas Werkmeister (WMDE): Enable fixed Wikibase RDF on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118484 (https://phabricator.wikimedia.org/T384344) [10:57:38] (03PS1) 10Lucas Werkmeister (WMDE): Enable fixed Wikibase RDF on Test Wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118485 (https://phabricator.wikimedia.org/T384344) [10:57:40] (03PS1) 10Lucas Werkmeister (WMDE): Enable fixed Wikibase RDF everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118486 (https://phabricator.wikimedia.org/T384344) [10:57:41] (03PS1) 10Lucas Werkmeister (WMDE): Remove Wikibase fixed RDF feature flag again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118487 (https://phabricator.wikimedia.org/T384344) [10:57:42] (03PS2) 10Aklapper: Remove a condition which always returns false [phabricator/antivandalism] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1118472 [10:57:57] (03CR) 10Lucas Werkmeister (WMDE): [C:04-2] "DNM yet, needs announcement." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118485 (https://phabricator.wikimedia.org/T384344) (owner: 10Lucas Werkmeister (WMDE)) [10:58:01] (03CR) 10Lucas Werkmeister (WMDE): [C:04-2] "DNM yet, needs announcement." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118486 (https://phabricator.wikimedia.org/T384344) (owner: 10Lucas Werkmeister (WMDE)) [10:58:13] (03CR) 10Lucas Werkmeister (WMDE): [C:04-2] "DNM until the Wikibase code is ready (see Depends-On)." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118487 (https://phabricator.wikimedia.org/T384344) (owner: 10Lucas Werkmeister (WMDE)) [10:58:54] (03CR) 10Aklapper: Remove a condition which always returns false (032 comments) [phabricator/antivandalism] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1118472 (owner: 10Aklapper) [10:59:43] (03CR) 10CI reject: [V:04-1] Remove Wikibase fixed RDF feature flag again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118487 (https://phabricator.wikimedia.org/T384344) (owner: 10Lucas Werkmeister (WMDE)) [11:00:01] (03CR) 10Lucas Werkmeister (WMDE): [C:04-2] "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118487 (https://phabricator.wikimedia.org/T384344) (owner: 10Lucas Werkmeister (WMDE)) [11:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1100) [11:00:11] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Remove a condition which always returns false (031 comment) [phabricator/antivandalism] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1118472 (owner: 10Aklapper) [11:13:42] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] "Should be okay to deploy now. (Unfortunately, I’ll be in a meeting during the deployment window.)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115006 (https://phabricator.wikimedia.org/T330217) (owner: 10Arthur taylor) [11:15:37] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10535068 (10phaultfinder) [11:16:11] (03CR) 10Lucas Werkmeister (WMDE): "I’ll try to be there towards the end of the window, if someone else can +1 this in the meantime :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1116812 (owner: 10Lucas Werkmeister (WMDE)) [11:29:16] FIRING: CertManagerCertNotReady: Certificate default/jayme-debug is not in a ready state (k8s-staging@codfw) - https://wikitech.wikimedia.org/wiki/Kubernetes/cert-manager - https://grafana.wikimedia.org/d/vo5tiJTnz?var-site=codfw&var-cluster=k8s-staging&var-namespace=default - https://alerts.wikimedia.org/?q=alertname%3DCertManagerCertNotReady [11:32:32] (03PS1) 10Ladsgroup: Set file to write both in all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118490 (https://phabricator.wikimedia.org/T384481) [11:32:42] jouncebot: nowandnext [11:32:42] For the next 0 hour(s) and 27 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1100) [11:32:42] In 2 hour(s) and 27 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1400) [11:36:39] RESOLVED: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic2085-production-search-psi-codfw is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency [11:40:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118490 (https://phabricator.wikimedia.org/T384481) (owner: 10Ladsgroup) [11:42:11] (03Merged) 10jenkins-bot: Set file to write both in all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118490 (https://phabricator.wikimedia.org/T384481) (owner: 10Ladsgroup) [11:42:28] !log ladsgroup@deploy2002 Started scap sync-world: Backport for [[gerrit:1118490|Set file to write both in all wikis (T384481)]] [11:42:32] T384481: Set new file tables to write both in production - https://phabricator.wikimedia.org/T384481 [11:45:08] !log ladsgroup@deploy2002 ladsgroup: Backport for [[gerrit:1118490|Set file to write both in all wikis (T384481)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [11:45:24] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118492 [11:46:17] !log ladsgroup@deploy2002 ladsgroup: Continuing with sync [11:52:50] !log ladsgroup@deploy2002 Finished scap sync-world: Backport for [[gerrit:1118490|Set file to write both in all wikis (T384481)]] (duration: 10m 21s) [11:52:53] T384481: Set new file tables to write both in production - https://phabricator.wikimedia.org/T384481 [12:05:18] (03CR) 10FNegri: [C:03+2] [toolforge::harbor] use latest thirdparty/docker [puppet] - 10https://gerrit.wikimedia.org/r/1114007 (https://phabricator.wikimedia.org/T384720) (owner: 10Raymond Ndibe) [12:08:47] (03CR) 10CDanis: [C:03+1] "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1118477 (owner: 10Joal) [12:11:28] (03PS1) 10Dragoniez: viwiki: Restrict the "changetags" permission to the sysop and bot groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118497 (https://phabricator.wikimedia.org/T385960) [12:13:10] (03CR) 10FNegri: [C:03+2] [toolforge::harbor] upgrade harbor v2.10.1 ---> v2.12.2 [puppet] - 10https://gerrit.wikimedia.org/r/1113871 (https://phabricator.wikimedia.org/T358225) (owner: 10Raymond Ndibe) [12:15:19] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1118498 (owner: 10L10n-bot) [12:30:27] Deploying cxserver.. [12:33:24] (03CR) 10KartikMistry: [C:03+2] Update cxserver to 2025-02-10-050623-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118264 (https://phabricator.wikimedia.org/T377966) (owner: 10KartikMistry) [12:34:57] (03Merged) 10jenkins-bot: Update cxserver to 2025-02-10-050623-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118264 (https://phabricator.wikimedia.org/T377966) (owner: 10KartikMistry) [12:52:15] !log kartik@deploy2002 helmfile [staging] START helmfile.d/services/cxserver: apply [12:52:49] !log kartik@deploy2002 helmfile [staging] DONE helmfile.d/services/cxserver: apply [12:56:39] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [12:57:01] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [12:57:48] (03PS1) 10Cyndywikime: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [13:00:48] (03PS1) 10Brouberol: airflow: don't define a Service when no ports are defined [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118508 [13:01:50] !log kartik@deploy2002 helmfile [codfw] START helmfile.d/services/cxserver: apply [13:02:25] !log kartik@deploy2002 helmfile [codfw] DONE helmfile.d/services/cxserver: apply [13:03:11] !log kartik@deploy2002 helmfile [eqiad] START helmfile.d/services/cxserver: apply [13:03:45] !log kartik@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [13:05:49] (03CR) 10Urbanecm: [C:04-1] "the corresponding part of CommonSettings.php needs to be cleaned up as well in this change" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [13:06:43] (03PS1) 10Ladsgroup: lists: Allow excempting ip ranges for export [puppet] - 10https://gerrit.wikimedia.org/r/1118511 (https://phabricator.wikimedia.org/T385271) [13:07:55] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1118511 (https://phabricator.wikimedia.org/T385271) (owner: 10Ladsgroup) [13:12:01] (03CR) 10Stevemunene: [C:03+1] "lgtm!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118508 (owner: 10Brouberol) [13:12:23] (03CR) 10Brouberol: [C:03+2] airflow: don't define a Service when no ports are defined [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118508 (owner: 10Brouberol) [13:12:29] (03PS2) 10Ladsgroup: lists: Allow excempting ip ranges for export [puppet] - 10https://gerrit.wikimedia.org/r/1118511 (https://phabricator.wikimedia.org/T385271) [13:12:38] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1118511 (https://phabricator.wikimedia.org/T385271) (owner: 10Ladsgroup) [13:13:10] !log Updated cxserver to 2025-02-10-050623-production (T377966, T383863, T385552, T369815) [13:13:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:13:17] T377966: Make cxserver Logstash logs readable and reliable - https://phabricator.wikimedia.org/T377966 [13:13:17] T383863: Adjust Google Configuration to expose Cantonese MT instead of Chinese - https://phabricator.wikimedia.org/T383863 [13:13:18] T385552: MinT: Add support for Obolo, Central Dusun, Iban and, South Ndebele - https://phabricator.wikimedia.org/T385552 [13:13:18] T369815: Enable in content Translation the new languages Google Translate supports in June 2024 - https://phabricator.wikimedia.org/T369815 [13:13:40] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [13:14:02] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [13:16:03] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [13:16:52] !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [13:17:03] (03PS3) 10Ladsgroup: lists: Allow excempting ip ranges for export [puppet] - 10https://gerrit.wikimedia.org/r/1118511 (https://phabricator.wikimedia.org/T385271) [13:17:07] (03CR) 10Ladsgroup: [V:03+2 C:03+2] "https://puppet-compiler.wmflabs.org/output/1118511/5470/lists1004.wikimedia.org/index.html" [puppet] - 10https://gerrit.wikimedia.org/r/1118511 (https://phabricator.wikimedia.org/T385271) (owner: 10Ladsgroup) [13:19:35] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10535371 (10phaultfinder) [13:34:38] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10535416 (10phaultfinder) [13:39:09] (03PS1) 10Gergő Tisza: Preserve 'campaign' parameter during authentication [extensions/Campaigns] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118515 [13:39:19] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/Campaigns] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118515 (owner: 10Gergő Tisza) [13:39:52] 06SRE, 06collaboration-services, 10Wikimedia-Mailing-lists: Excempt researcher from hyperkitty monthly export - https://phabricator.wikimedia.org/T385271#10535419 (10Ladsgroup) 05Open→03Resolved [13:41:59] (03PS1) 10Gergő Tisza: Call AuthPreserveQueryParams hook when redirecting to SUL3 domain [extensions/CentralAuth] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118517 [13:42:32] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/CentralAuth] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118517 (owner: 10Gergő Tisza) [13:42:41] FIRING: [3x] SystemdUnitFailed: etcd-backup.service on aux-k8s-etcd2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:59:33] FIRING: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [14:00:04] Lucas_WMDE, Urbanecm, and TheresNoTime: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for UTC afternoon backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1400). [14:00:05] codders, tgr, and Lucas_WMDE: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:14] here! [14:00:19] I can’t deploy, sorry [14:04:01] o/ [14:04:16] I can deploy in ~10 min if no one shows up before [14:10:39] (03CR) 10Jforrester: [C:03+1] "Let's do this." [dumps] - 10https://gerrit.wikimedia.org/r/1108844 (https://phabricator.wikimedia.org/T382069) (owner: 10Ladsgroup) [14:16:01] ok, deploying [14:16:42] codders: can the two patches be deployed together? [14:17:10] yeah. should work [14:18:21] (03PS1) 10Btullis: Update the SSH key for btullis with a new public key [puppet] - 10https://gerrit.wikimedia.org/r/1118520 (https://phabricator.wikimedia.org/T385943) [14:19:33] RESOLVED: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [14:20:17] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tgr@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115006 (https://phabricator.wikimedia.org/T330217) (owner: 10Arthur taylor) [14:20:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tgr@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115013 (https://phabricator.wikimedia.org/T330217) (owner: 10Arthur taylor) [14:21:49] (03Merged) 10jenkins-bot: Remove `tmpAlwaysShowMulLanguageCode` temporary setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115006 (https://phabricator.wikimedia.org/T330217) (owner: 10Arthur taylor) [14:22:04] (03Merged) 10jenkins-bot: Add `enableMulLanguageCode` to replace `tmpEnableMulLanguageCode` [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115013 (https://phabricator.wikimedia.org/T330217) (owner: 10Arthur taylor) [14:22:34] > Gerrit could not merge the change '1115013' as is and could require a rebase [14:22:47] seems like the communication between gerrit and scap is not perfect there [14:22:58] okay. I'll take a look [14:23:09] they both got merged, right? [14:23:11] !log tgr@deploy2002 Started scap sync-world: Backport for [[gerrit:1115006|Remove `tmpAlwaysShowMulLanguageCode` temporary setting (T330217)]], [[gerrit:1115013|Add `enableMulLanguageCode` to replace `tmpEnableMulLanguageCode` (T330217)]] [14:23:15] T330217: MUL - Cleanup soft rollout flag - https://phabricator.wikimedia.org/T330217 [14:23:18] it's fine, just the deploy tool being dim [14:23:29] are they live now? [14:23:37] not yet [14:24:39] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10535534 (10phaultfinder) [14:26:46] (03CR) 10Brouberol: [C:03+1] "Confirmed with Ben out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1118520 (https://phabricator.wikimedia.org/T385943) (owner: 10Btullis) [14:27:17] !log tgr@deploy2002 tgr, arthurtaylor: Backport for [[gerrit:1115006|Remove `tmpAlwaysShowMulLanguageCode` temporary setting (T330217)]], [[gerrit:1115013|Add `enableMulLanguageCode` to replace `tmpEnableMulLanguageCode` (T330217)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [14:28:17] looks good on k8s-mwdebug using the debug toolbar [14:28:36] !log tgr@deploy2002 tgr, arthurtaylor: Continuing with sync [14:35:26] !log tgr@deploy2002 Finished scap sync-world: Backport for [[gerrit:1115006|Remove `tmpAlwaysShowMulLanguageCode` temporary setting (T330217)]], [[gerrit:1115013|Add `enableMulLanguageCode` to replace `tmpEnableMulLanguageCode` (T330217)]] (duration: 12m 14s) [14:35:29] T330217: MUL - Cleanup soft rollout flag - https://phabricator.wikimedia.org/T330217 [14:36:09] thanks, should be live [14:36:22] thank you! looks good [14:36:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:39:41] (03CR) 10Gergő Tisza: [C:03+1] "Maybe someone confused it with the Commons:Upload/pt localization mechanism?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1116812 (owner: 10Lucas Werkmeister (WMDE)) [14:39:47] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tgr@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1116812 (owner: 10Lucas Werkmeister (WMDE)) [14:40:29] (03CR) 10Gergő Tisza: [C:03+2] Fire CentralAuthPostLoginRedirect on SUL3 login [extensions/CentralAuth] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1117927 (https://phabricator.wikimedia.org/T364866) (owner: 10Gergő Tisza) [14:40:41] (03CR) 10Gergő Tisza: [C:03+2] Preserve 'campaign' parameter during authentication [extensions/Campaigns] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118515 (owner: 10Gergő Tisza) [14:40:42] (03Merged) 10jenkins-bot: Remove /pt from ptwikibooks $wgUploadMissingFileUrl [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1116812 (owner: 10Lucas Werkmeister (WMDE)) [14:40:44] (03CR) 10Gergő Tisza: [C:03+2] Call AuthPreserveQueryParams hook when redirecting to SUL3 domain [extensions/CentralAuth] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118517 (owner: 10Gergő Tisza) [14:40:58] !log tgr@deploy2002 Started scap sync-world: Backport for [[gerrit:1116812|Remove /pt from ptwikibooks $wgUploadMissingFileUrl]] [14:42:49] FIRING: PuppetFailure: Puppet has failed on build2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:43:35] !log tgr@deploy2002 tgr, lucaswerkmeister-wmde: Backport for [[gerrit:1116812|Remove /pt from ptwikibooks $wgUploadMissingFileUrl]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [14:47:15] (03PS1) 10Andrew Bogott: Add wmcs-bastionless utility script [puppet] - 10https://gerrit.wikimedia.org/r/1118526 (https://phabricator.wikimedia.org/T379550) [14:48:05] (03Merged) 10jenkins-bot: Fire CentralAuthPostLoginRedirect on SUL3 login [extensions/CentralAuth] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1117927 (https://phabricator.wikimedia.org/T364866) (owner: 10Gergő Tisza) [14:48:07] (03Merged) 10jenkins-bot: Preserve 'campaign' parameter during authentication [extensions/Campaigns] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118515 (owner: 10Gergő Tisza) [14:48:47] !log tgr@deploy2002 tgr, lucaswerkmeister-wmde: Continuing with sync [14:48:53] (03Merged) 10jenkins-bot: Call AuthPreserveQueryParams hook when redirecting to SUL3 domain [extensions/CentralAuth] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118517 (owner: 10Gergő Tisza) [14:49:22] Lucas_WMDE: tested, deployed [14:52:16] (03PS2) 10Andrew Bogott: Add wmcs-bastionless utility script [puppet] - 10https://gerrit.wikimedia.org/r/1118526 (https://phabricator.wikimedia.org/T379550) [14:52:56] !log fnegri@cumin1002 START - Cookbook sre.hosts.reboot-single for host cloudnet1006.eqiad.wmnet [14:53:26] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1118526 (https://phabricator.wikimedia.org/T379550) (owner: 10Andrew Bogott) [14:55:06] (03PS3) 10Andrew Bogott: Add wmcs-bastionless utility script [puppet] - 10https://gerrit.wikimedia.org/r/1118526 (https://phabricator.wikimedia.org/T379550) [14:55:20] !log tgr@deploy2002 Finished scap sync-world: Backport for [[gerrit:1116812|Remove /pt from ptwikibooks $wgUploadMissingFileUrl]] (duration: 14m 21s) [14:55:32] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1118526 (https://phabricator.wikimedia.org/T379550) (owner: 10Andrew Bogott) [14:58:30] !log fnegri@cumin1002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1006.eqiad.wmnet [14:59:33] !log tgr@deploy2002 Started scap sync-world: Backport for [[gerrit:1117927|Fire CentralAuthPostLoginRedirect on SUL3 login (T364866)]], [[gerrit:1118515|Preserve 'campaign' parameter during authentication]], [[gerrit:1118517|Call AuthPreserveQueryParams hook when redirecting to SUL3 domain]] [14:59:36] T364866: Adapt to changes in post-login/signup hooks after switching to a central login wiki - https://phabricator.wikimedia.org/T364866 [15:03:11] !log tgr@deploy2002 tgr: Backport for [[gerrit:1117927|Fire CentralAuthPostLoginRedirect on SUL3 login (T364866)]], [[gerrit:1118515|Preserve 'campaign' parameter during authentication]], [[gerrit:1118517|Call AuthPreserveQueryParams hook when redirecting to SUL3 domain]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [15:06:42] RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:07:42] !log tgr@deploy2002 tgr: Continuing with sync [15:14:21] !log tgr@deploy2002 Finished scap sync-world: Backport for [[gerrit:1117927|Fire CentralAuthPostLoginRedirect on SUL3 login (T364866)]], [[gerrit:1118515|Preserve 'campaign' parameter during authentication]], [[gerrit:1118517|Call AuthPreserveQueryParams hook when redirecting to SUL3 domain]] (duration: 14m 47s) [15:14:24] T364866: Adapt to changes in post-login/signup hooks after switching to a central login wiki - https://phabricator.wikimedia.org/T364866 [15:16:39] !log UTC afternoon deploys done [15:16:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:17:16] FIRING: MediaWikiLatencyExceeded: p75 latency high: codfw mw-web/next (k8s) 1.62s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=next - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [15:22:16] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: codfw mw-web/next (k8s) 1.082s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=next - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [15:24:38] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10535746 (10phaultfinder) [15:32:41] FIRING: CertManagerCertNotReady: Certificate default/jayme-debug is not in a ready state (k8s-staging@codfw) - https://wikitech.wikimedia.org/wiki/Kubernetes/cert-manager - https://grafana.wikimedia.org/d/vo5tiJTnz?var-site=codfw&var-cluster=k8s-staging&var-namespace=default - https://alerts.wikimedia.org/?q=alertname%3DCertManagerCertNotReady [15:47:58] 06SRE, 10SRE Observability (FY2024/2025-Q3): etcd: adapt etcd-backup.py for etcd 3.4 - https://phabricator.wikimedia.org/T385727#10535814 (10herron) Thought about this over the weekend a bit. To summarize my current understanding, we have etcd-backup.py on etcd hosts to creates backups via `etcdctl backup` wh... [16:06:23] tgr|away: cool, thanks! [16:18:38] (03PS1) 10Sergio Gimeno: beta: fix typo in GEApiQueryGrowthTasksLookaheadSize variable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118541 [16:30:04] jan_drewniak: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Wikimedia Portals Update . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1630). [16:30:30] (03PS1) 10Arturo Borrero Gonzalez: wmcs: kernel_errors: don't alert on warning messages [alerts] - 10https://gerrit.wikimedia.org/r/1118547 [16:32:05] (03CR) 10CI reject: [V:04-1] wmcs: kernel_errors: don't alert on warning messages [alerts] - 10https://gerrit.wikimedia.org/r/1118547 (owner: 10Arturo Borrero Gonzalez) [16:34:38] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10535967 (10phaultfinder) [16:38:39] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118548 (https://phabricator.wikimedia.org/T128546) [16:39:13] (03CR) 10Jdrewniak: [C:03+2] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118548 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [16:41:19] (03CR) 10FNegri: [C:04-1] wmcs: kernel_errors: don't alert on warning messages (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1118547 (owner: 10Arturo Borrero Gonzalez) [16:41:28] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118548 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [16:47:28] (03PS2) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [16:50:40] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10536027 (10phaultfinder) [16:51:44] (03PS3) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [16:55:53] !log jdrewniak@deploy2002 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:1112805| Bumping portals to master (T128546)]] (duration: 09m 04s) [16:55:56] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [16:57:10] (03PS1) 10DCausse: Update plugins to opensearch 1.3.20 [software/opensearch/plugins] - 10https://gerrit.wikimedia.org/r/1118553 (https://phabricator.wikimedia.org/T385005) [16:58:14] !log jdrewniak@deploy2002 Synchronized portals: Wikimedia Portals Update: [[gerrit:1112805| Bumping portals to master (T128546)]] (duration: 02m 19s) [17:03:18] (03PS4) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [17:03:51] I am going to start running scripts for today's planned work on https://wikitech.wikimedia.org/wiki/News/2024_Migrating_Wikitech_Account_to_SUL very soon. [17:18:19] (03CR) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [17:24:35] (03CR) 10Urbanecm: [C:03+1] "LGTM, but should be marked as depends-on I9aebdb6476113b29716cee780489f30f3795c25a, as it depends on the A/B variants no longer being used" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [17:26:14] (03PS5) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [17:26:26] (03CR) 10DCausse: [C:04-1] wdqs-categories: enable scrapes for jmx exporter (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1118162 (https://phabricator.wikimedia.org/T385236) (owner: 10Bking) [17:26:30] (03PS6) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [17:29:42] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10536180 (10phaultfinder) [17:30:32] Wikitech: SUL attached 593 accounts with matching names claimed via Striker or Bitu (T161859) [17:30:33] T161859: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859 [17:31:12] * bd808 forgot that `dologmsg` in prod doesn't add the user@host preamble [17:31:50] !log bd808@mwmaint2002 Wikitech: SUL attached 593 accounts with matching names claimed via Striker or Bitu (T161859) [17:36:12] (03PS7) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [17:37:43] (03PS1) 10Urbanecm: [Growth] Deploy Community updates to all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118557 (https://phabricator.wikimedia.org/T384406) [17:40:34] !log bd808@mwmaint2002 Wikitech: Renamed and attached 57 of 82 accounts claimed via Striker where the SUL account was renamed after claiming (T161859) [17:40:37] T161859: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859 [17:42:41] FIRING: [3x] SystemdUnitFailed: etcd-backup.service on aux-k8s-etcd2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:46:27] (03CR) 10Michael Große: [C:03+1] "Looks good, could you add a line to the commit message saying that `GECommunityUpdatesEnabled` is set to `true` by default in the extensio" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118557 (https://phabricator.wikimedia.org/T384406) (owner: 10Urbanecm) [17:46:48] (03PS1) 10Jdlrobson: Add search activity id [extensions/WikimediaEvents] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118558 (https://phabricator.wikimedia.org/T383936) [17:47:51] (03PS2) 10Urbanecm: [Growth] Deploy Community updates to all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118557 (https://phabricator.wikimedia.org/T384406) [17:47:59] (03CR) 10Urbanecm: "Sure. Done!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118557 (https://phabricator.wikimedia.org/T384406) (owner: 10Urbanecm) [17:52:46] (03CR) 10Stoyofuku-wmf: [C:03+1] "Looks good! Do we have a deployer for this?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117620 (https://phabricator.wikimedia.org/T384824) (owner: 10LorenMora) [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1800) [18:00:05] ryankemper: I, the Bot under the Fountain, call upon thee, The Deployer, to do Wikidata Query Service weekly deploy deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1800). [18:03:34] !log bd808@mwmaint2002 Wikitech: Renamed and attached 234 of 385 accounts claimed via Bitu (T161859) [18:03:38] T161859: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859 [18:04:33] FIRING: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [18:08:52] (03CR) 10Michael Große: "Thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118557 (https://phabricator.wikimedia.org/T384406) (owner: 10Urbanecm) [18:10:05] bd808: At least for me, it worked perfectly. Thank you! [18:14:08] (03PS1) 10Ladsgroup: Remove special-casing of CentralAuth for labswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) [18:15:30] (03CR) 10Ladsgroup: "Did we miss a step Taavi? You might know if I messed up something" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) (owner: 10Ladsgroup) [18:22:20] (03CR) 10Bugreporter: "IMO We should remove it only after all accounts are migrated to SUL (which will happen several weeks later). Account autocreation is still" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) (owner: 10Ladsgroup) [18:24:33] RESOLVED: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [18:33:33] (03CR) 10Ladsgroup: "wikitech is a internal but public documentation wiki. It's not a top tier content project." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) (owner: 10Ladsgroup) [18:36:51] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 219, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [18:37:01] PROBLEM - Router interfaces on cr1-codfw is CRITICAL: CRITICAL: host 208.80.153.192, interfaces up: 128, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [18:38:14] jouncebot now [18:38:14] For the next 0 hour(s) and 21 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T1800) [18:38:27] !log dancy@deploy2002 Installing scap version "4.135.0" for 204 host(s) [18:39:17] (03PS2) 10Bking: wdqs-categories: enable scrapes for jmx exporter [puppet] - 10https://gerrit.wikimedia.org/r/1118162 (https://phabricator.wikimedia.org/T385236) [18:40:02] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117620 (https://phabricator.wikimedia.org/T384824) (owner: 10LorenMora) [18:40:53] (03CR) 10Bugreporter: "At least, setting $wgCentralAuthStrict to true will make accounts not yet migrated unable to login." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) (owner: 10Ladsgroup) [18:40:59] (03CR) 10Majavah: "I don't think you can remove `$wgCentralAuthStrict = false;` before all of the unattached accounts have been migrated (which is scheduled " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) (owner: 10Ladsgroup) [18:41:37] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1118162 (https://phabricator.wikimedia.org/T385236) (owner: 10Bking) [18:42:15] (03CR) 10Ladsgroup: "noted. Thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) (owner: 10Ladsgroup) [18:42:44] (03CR) 10Bugreporter: "Obsolete patch, please abandon." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/931570 (owner: 10Majavah) [18:42:49] FIRING: PuppetFailure: Puppet has failed on build2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:43:27] (03Abandoned) 10Majavah: labswiki: Use ExternalStore by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/931570 (owner: 10Majavah) [18:43:35] !log dancy@deploy2002 Installation of scap version "4.135.0" completed for 204 hosts [18:43:52] !log dancy@deploy2002 Started scap sync-world: Testing scap 4.135.0 [18:45:30] (03PS2) 10Ladsgroup: Remove special-casing of CentralAuth for labswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118561 (https://phabricator.wikimedia.org/T161859) [18:46:43] !log dancy@deploy2002 Finished scap sync-world: Testing scap 4.135.0 (duration: 02m 50s) [18:55:51] (03PS1) 10Bernard Wang: Turn on sampling rate for web ab test schemas for basque wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 [18:58:59] (03CR) 10Andrew Bogott: [C:03+1] sysctl: Introduce base::sysctl::inotify helper [puppet] - 10https://gerrit.wikimedia.org/r/1116888 (https://phabricator.wikimedia.org/T385530) (owner: 10BryanDavis) [19:09:59] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 2157379304 and 118 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:15:59] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 5152 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:22:57] (03PS2) 10Bernard Wang: Turn on sampling rate for web ab test schemas for basque and ca wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 [19:32:41] FIRING: CertManagerCertNotReady: Certificate default/jayme-debug is not in a ready state (k8s-staging@codfw) - https://wikitech.wikimedia.org/wiki/Kubernetes/cert-manager - https://grafana.wikimedia.org/d/vo5tiJTnz?var-site=codfw&var-cluster=k8s-staging&var-namespace=default - https://alerts.wikimedia.org/?q=alertname%3DCertManagerCertNotReady [19:43:52] (03PS2) 10Bvibber: Pref off use of gjl_namespace_text field until it's deployed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118184 (https://phabricator.wikimedia.org/T385917) [19:52:49] (03PS1) 10Urbanecm: Community Updates: End pilot experiment [extensions/GrowthExperiments] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118574 (https://phabricator.wikimedia.org/T385338) [20:04:36] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10536653 (10phaultfinder) [20:10:31] (03PS1) 10Urbanecm: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118575 [20:11:56] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117960 (https://phabricator.wikimedia.org/T385833) (owner: 10Acamicamacaraca) [20:12:28] (03CR) 10Urbanecm: [C:03+2] linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118575 (owner: 10Urbanecm) [20:12:53] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/GrowthExperiments] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118574 (https://phabricator.wikimedia.org/T385338) (owner: 10Urbanecm) [20:13:56] (03Merged) 10jenkins-bot: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118575 (owner: 10Urbanecm) [20:14:29] !log urbanecm@deploy2002 helmfile [staging] START helmfile.d/services/linkrecommendation: apply [20:14:41] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10536677 (10phaultfinder) [20:15:33] !log urbanecm@deploy2002 helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [20:16:04] !log urbanecm@deploy2002 helmfile [codfw] START helmfile.d/services/linkrecommendation: apply [20:17:16] !log urbanecm@deploy2002 helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply [20:19:11] !log urbanecm@deploy2002 helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply [20:19:42] (03CR) 10Jdlrobson: [C:04-1] "this should also configure RelatedArticlesABTestEnrollment for cawiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 (owner: 10Bernard Wang) [20:21:10] !log urbanecm@deploy2002 helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply [20:21:47] (03PS1) 10Urbanecm: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118576 [20:21:49] ...and once more [20:21:54] (03CR) 10Urbanecm: [C:03+2] linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118576 (owner: 10Urbanecm) [20:22:42] Hello! Any objections to me doing a quick-ish backport ahead of the formal window? [20:23:01] For this patch: https://gerrit.wikimedia.org/r/1118558 [20:23:03] (03Merged) 10jenkins-bot: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1118576 (owner: 10Urbanecm) [20:23:09] And potentially a config deploy [20:23:33] toyofuku: there's only 30 mins, with how slow gate-and-submit is today, i wouldn't be sure that's going to be enough [20:24:06] Makes sense [20:24:10] (03PS3) 10Bernard Wang: Turn on sampling rate for web ab test schemas for basque and ca wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 [20:24:19] no strong objection though, if you're feeling lucky :) [20:24:28] !log urbanecm@deploy2002 helmfile [staging] START helmfile.d/services/linkrecommendation: apply [20:24:52] hahaha I'm looking at the deployment schedule and thinking I'd rather not chance all those people mad at me. Will keep you all updated on the config deploy - might take _that_ risk [20:24:55] (03CR) 10CI reject: [V:04-1] Turn on sampling rate for web ab test schemas for basque and ca wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 (owner: 10Bernard Wang) [20:25:09] that should be fine :) [20:25:17] !log urbanecm@deploy2002 helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [20:26:08] !log urbanecm@deploy2002 helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply [20:26:55] !log urbanecm@deploy2002 helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply [20:26:59] !log urbanecm@deploy2002 helmfile [codfw] START helmfile.d/services/linkrecommendation: apply [20:27:19] (03PS4) 10Bernard Wang: Turn on sampling rate for web ab test schemas for basque and ca wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 [20:27:40] !log urbanecm@deploy2002 helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply [20:28:33] * urbanecm done with k8s deployment [20:28:34] (03CR) 10CI reject: [V:04-1] Turn on sampling rate for web ab test schemas for basque and ca wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 (owner: 10Bernard Wang) [20:42:20] okay yeah we're gonna wait on the deploys, sorry for the back and forth [20:42:30] But see you all in ~20 for the backport window! [20:47:23] (03CR) 10Jdlrobson: [C:04-1] Turn on sampling rate for web ab test schemas for basque and ca wiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 (owner: 10Bernard Wang) [21:00:04] RoanKattouw, Urbanecm, cjming, TheresNoTime, and kindrobot: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T2100). [21:00:05] bvibber, lucaswerkmeister, toyofuku, Aca, and urbanecm: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:12] i can deploy today [21:00:20] o/ [21:00:21] (03CR) 10Urbanecm: [C:03+2] Community Updates: End pilot experiment [extensions/GrowthExperiments] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118574 (https://phabricator.wikimedia.org/T385338) (owner: 10Urbanecm) [21:00:28] thank you!! [21:00:30] hey lucaswerkmeister! special nick :) [21:00:43] confirming my presence as well [21:00:45] toyofuku: you also mentioned some backport, do you want to do that too? [21:00:47] hey Aca [21:00:58] urbanecm: I can’t deploy today, I’m just an uwu smol bean volunteer with no special rights :3 [21:01:05] nah, we can stick with the one that's on the schedule [21:01:11] ack [21:01:19] Team is still working out exactly what else needs to be deployed - thanks for asking though!! [21:01:25] lucaswerkmeister: disadvantage of putting WMDE in the shell name! :)) [21:02:00] my config change isn’t important btw, so feel free to do Commons and other stuff before ^^ [21:03:21] (03PS2) 10LorenMora: Deploy Vector 2022 skin to next set of wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117620 (https://phabricator.wikimedia.org/T384824) [21:03:27] (03CR) 10Urbanecm: [C:03+2] Deploy Vector 2022 skin to next set of wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117620 (https://phabricator.wikimedia.org/T384824) (owner: 10LorenMora) [21:03:59] (03PS5) 10Acamicamacaraca: SITENAME, project namespace, and timezone change of Serbo-Croatian Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117960 (https://phabricator.wikimedia.org/T385833) [21:04:02] (03CR) 10Urbanecm: [C:03+2] SITENAME, project namespace, and timezone change of Serbo-Croatian Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117960 (https://phabricator.wikimedia.org/T385833) (owner: 10Acamicamacaraca) [21:04:18] (03Merged) 10jenkins-bot: Deploy Vector 2022 skin to next set of wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117620 (https://phabricator.wikimedia.org/T384824) (owner: 10LorenMora) [21:05:00] so many parsoid stuff in CI [21:06:48] (03Merged) 10jenkins-bot: SITENAME, project namespace, and timezone change of Serbo-Croatian Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1117960 (https://phabricator.wikimedia.org/T385833) (owner: 10Acamicamacaraca) [21:07:53] lucaswerkmeister: i'm sure this was discussed somewhere, but...i can't find the "go ahead" / +1 on the change. can you give me some pointers, please? [21:08:12] I haven’t received a go ahead on this change in particular [21:08:23] (03PS1) 10Dbrant: Add app_games event stream. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118577 (https://phabricator.wikimedia.org/T385323) [21:08:26] I assumed that it’s been up long enough for someone to object if they think it shouldn’t go out [21:08:33] fair [21:08:34] !log urbanecm@deploy2002 Started scap sync-world: Backport for [[gerrit:1117960|SITENAME, project namespace, and timezone change of Serbo-Croatian Wiktionary (T385833)]], [[gerrit:1117620|Deploy Vector 2022 skin to next set of wikis (T384824)]] [21:08:34] but I’m also okay with waiting further if you prefer [21:08:39] T385833: SITENAME, project namespace and timezone change of Serbo-Croatian Wiktionary - https://phabricator.wikimedia.org/T385833 [21:08:39] T384824: Deploy Vector 2022 skin to next set of wikis - https://phabricator.wikimedia.org/T384824 [21:11:18] !log urbanecm@deploy2002 urbanecm, lmora, aleksandar: Backport for [[gerrit:1117960|SITENAME, project namespace, and timezone change of Serbo-Croatian Wiktionary (T385833)]], [[gerrit:1117620|Deploy Vector 2022 skin to next set of wikis (T384824)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [21:11:27] toyofuku: Aca: can you test, please? [21:13:01] Yes! Sorry [21:13:20] no worries [21:13:23] Aca: ^^ [21:14:00] Looks good to me, everything is updated accordingly. However, since we updated the namespaces, perhaps we should run namespaceDupes.php as well afterwards [21:14:06] Looks good! [21:14:56] Aca: yep, i'll do that once it syncs [21:15:00] toyofuku: thanks, proceeding! [21:15:02] !log urbanecm@deploy2002 urbanecm, lmora, aleksandar: Continuing with sync [21:15:16] bvibber: hey, around for the window? [21:15:46] (03CR) 10Urbanecm: [C:03+2] "Based on the discussion on the ticket and the change, there seems to be consensus on going ahead with this. Let's move this forward!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1116860 (https://phabricator.wikimedia.org/T322944) (owner: 10Lucas Werkmeister) [21:16:41] (03Merged) 10jenkins-bot: Enable $wgAllowAuthenticatedCrossOrigin on most wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1116860 (https://phabricator.wikimedia.org/T322944) (owner: 10Lucas Werkmeister) [21:18:17] (03Merged) 10jenkins-bot: Community Updates: End pilot experiment [extensions/GrowthExperiments] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118574 (https://phabricator.wikimedia.org/T385338) (owner: 10Urbanecm) [21:21:37] !log urbanecm@deploy2002 Finished scap sync-world: Backport for [[gerrit:1117960|SITENAME, project namespace, and timezone change of Serbo-Croatian Wiktionary (T385833)]], [[gerrit:1117620|Deploy Vector 2022 skin to next set of wikis (T384824)]] (duration: 13m 03s) [21:21:42] T385833: SITENAME, project namespace and timezone change of Serbo-Croatian Wiktionary - https://phabricator.wikimedia.org/T385833 [21:21:42] T384824: Deploy Vector 2022 skin to next set of wikis - https://phabricator.wikimedia.org/T384824 [21:22:15] Vector 2022 on Commons :o [21:23:20] needs a bit of cache to clean up, but yeah! [21:23:36] it’s already showing it for me fwiw [21:23:38] !log Start namespaceDupes.php on shwiktionary (T385833) [21:23:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:23:41] (when logged in, which I suppose helps) [21:23:54] lucaswerkmeister: then you bypass the CDN altogether :) [21:23:57] yeah ^^ [21:24:26] https://commons.wikimedia.org/wiki/Main_Page logged out has legacy vector for me, but a random file has V22 already [21:24:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [21:24:44] (03CR) 10Urbanecm: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [21:24:50] (03CR) 10Urbanecm: [C:03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [21:24:54] main page in private window is v22 for me [21:25:00] not what i wanted to start [21:25:01] but yeah it’ll just take a bit [21:25:16] yep [21:25:59] !log urbanecm@deploy2002 Started scap sync-world: Backport for [[gerrit:1118574|Community Updates: End pilot experiment (T385338)]], [[gerrit:1116860|Enable $wgAllowAuthenticatedCrossOrigin on most wikis (T322944)]] [21:26:04] T385338: Community Updates: End pilot experiment - https://phabricator.wikimedia.org/T385338 [21:26:04] T322944: Allow authenticated requests via OAuth to the Action API from any origin - https://phabricator.wikimedia.org/T322944 [21:28:34] i still think it's tricky to get params for jobs started via mwscript-k8s... [21:29:15] urbanecm: thank you!! [21:29:30] !log urbanecm@deploy2002 urbanecm, lucaswerkmeister: Backport for [[gerrit:1118574|Community Updates: End pilot experiment (T385338)]], [[gerrit:1116860|Enable $wgAllowAuthenticatedCrossOrigin on most wikis (T322944)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [21:29:33] any time :) [21:29:40] lucaswerkmeister: can you test? :) [21:29:44] let’s see if I can test this (probably not very much) [21:29:45] urbanecm: sorry got distracted [21:29:52] here now :D [21:30:05] hey bvibber ! [21:30:11] \o/ [21:30:17] my patch works [21:30:21] waiting for Lucas [21:31:15] urbanecm: should be working, as far as I can test it at least [21:31:24] sounds good enough to me! [21:31:26] !log urbanecm@deploy2002 urbanecm, lucaswerkmeister: Continuing with sync [21:31:28] proceeding [21:31:34] (03PS3) 10Bvibber: Pref off use of gjl_namespace_text field until it's deployed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118184 (https://phabricator.wikimedia.org/T385917) [21:31:34] (the preflight request goes to non-wikimediadebug so I can’t integration-test in a browser, but I tried it with curl ^^) [21:31:46] (same as when it rolled out to testwiki ^^) [21:32:01] (03CR) 10Urbanecm: [C:03+2] Pref off use of gjl_namespace_text field until it's deployed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118184 (https://phabricator.wikimedia.org/T385917) (owner: 10Bvibber) [21:32:11] * lucaswerkmeister waves at bvibber [21:32:18] we'll see for real soon enough :)) [21:32:26] * bvibber waves at lucaswerkmeister [21:32:35] * urbanecm waves in general [21:32:38] :D [21:32:42] ~o~ [21:32:44] (03Merged) 10jenkins-bot: Pref off use of gjl_namespace_text field until it's deployed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118184 (https://phabricator.wikimedia.org/T385917) (owner: 10Bvibber) [21:32:45] (03PS8) 10Cyndywikime: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) [21:32:48] (03CR) 10Urbanecm: [C:03+2] GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [21:33:12] [21:33:32] (03Merged) 10jenkins-bot: GrowthExperiments: End A/B test for Community Updates module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118507 (https://phabricator.wikimedia.org/T385338) (owner: 10Cyndywikime) [21:38:06] !log urbanecm@deploy2002 Finished scap sync-world: Backport for [[gerrit:1118574|Community Updates: End pilot experiment (T385338)]], [[gerrit:1116860|Enable $wgAllowAuthenticatedCrossOrigin on most wikis (T322944)]] (duration: 12m 06s) [21:38:10] T385338: Community Updates: End pilot experiment - https://phabricator.wikimedia.org/T385338 [21:38:10] T322944: Allow authenticated requests via OAuth to the Action API from any origin - https://phabricator.wikimedia.org/T322944 [21:38:11] *tries* [21:38:40] !log urbanecm@deploy2002 Started scap sync-world: Backport for [[gerrit:1118184|Pref off use of gjl_namespace_text field until it's deployed (T385917)]], [[gerrit:1118507|GrowthExperiments: End A/B test for Community Updates module (T385338)]] [21:38:44] T385917: Deploy patch-gjl_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917 [21:38:49] fingers crossed! [21:38:57] \o/ [21:39:11] nothing to test, it's just preemptively disabling a feature flag :D [21:39:19] yep, figured so [21:39:33] woot [21:39:43] hm, not working for me but I can’t figure out why yet [21:40:15] wait [21:40:40] 🤦 https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/bd42efa5d63ec102843072f4837d4b51 is limited to testwiki lol [21:41:11] anyway, the preflight request works and it’s the real request that throws the “can’t use this consumer here bud” error, I think that’s enough to call the CORS part working ^^ [21:41:15] !log urbanecm@deploy2002 urbanecm, cyndywikime, bvibber: Backport for [[gerrit:1118184|Pref off use of gjl_namespace_text field until it's deployed (T385917)]], [[gerrit:1118507|GrowthExperiments: End A/B test for Community Updates module (T385338)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [21:41:16] thanks a lot urbanecm! \o/ [21:41:40] woot [21:41:53] woot woot! [21:42:02] lucaswerkmeister: let me know if you want me to approve a less limited version of the consumer [21:42:12] !log urbanecm@deploy2002 urbanecm, cyndywikime, bvibber: Continuing with sync [21:42:41] FIRING: [3x] SystemdUnitFailed: etcd-backup.service on aux-k8s-etcd2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:46:54] when running mwscript-k8s https://www.irccloud.com/pastebin/0bnaLtUU/ [21:46:58] that...doesn't look good [21:48:16] ouch [21:48:39] (also, nice that it doesn’t say *which* host name :| ) [21:48:55] probably an etcd-related one [21:49:06] seems to be transient though, it didn't happen on the first or the third execution [21:49:08] !log urbanecm@deploy2002 Finished scap sync-world: Backport for [[gerrit:1118184|Pref off use of gjl_namespace_text field until it's deployed (T385917)]], [[gerrit:1118507|GrowthExperiments: End A/B test for Community Updates module (T385338)]] (duration: 10m 28s) [21:49:13] T385917: Deploy patch-gjl_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917 [21:49:13] T385338: Community Updates: End pilot experiment - https://phabricator.wikimedia.org/T385338 [21:49:22] Aca: anyway, seems namespaces are cleared up https://www.irccloud.com/pastebin/lOei4G3h/ [21:50:05] two Acas! [21:51:06] posted on task too: https://phabricator.wikimedia.org/T385833#10536931 [21:51:14] bvibber: should be live [21:51:16] Bruh, got disconnected. Anyway, thankiess! Just saw [21:51:17] anything else, anyone? [21:51:24] woohoo thx urbanecm [21:51:25] Aca51: no problem! [21:54:42] 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10536938 (10phaultfinder) [22:00:05] Reedy, sbassett, Maryum, and manfredi: That opportune time for a Weekly Security deployment window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250210T2200). [22:09:33] FIRING: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [22:29:33] RESOLVED: Wikidata Reliability Metrics - Median loading time alert: - https://alerts.wikimedia.org/?q=alertname%3DWikidata+Reliability+Metrics+-+Median+loading+time+alert [22:42:49] FIRING: PuppetFailure: Puppet has failed on build2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:01:03] (03CR) 10Jdlrobson: [C:03+1] Add search activity id [extensions/WikimediaEvents] (wmf/1.44.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1118558 (https://phabricator.wikimedia.org/T383936) (owner: 10Jdlrobson) [23:10:01] (03PS5) 10Bernard Wang: Turn on sampling rate for web ab test schemas for basque and ca wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 [23:10:45] (03CR) 10CI reject: [V:04-1] Turn on sampling rate for web ab test schemas for basque and ca wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 (owner: 10Bernard Wang) [23:32:41] FIRING: CertManagerCertNotReady: Certificate default/jayme-debug is not in a ready state (k8s-staging@codfw) - https://wikitech.wikimedia.org/wiki/Kubernetes/cert-manager - https://grafana.wikimedia.org/d/vo5tiJTnz?var-site=codfw&var-cluster=k8s-staging&var-namespace=default - https://alerts.wikimedia.org/?q=alertname%3DCertManagerCertNotReady [23:47:04] (03CR) 10Jdlrobson: [C:04-1] Turn on sampling rate for web ab test schemas for basque and ca wiki (034 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1118566 (owner: 10Bernard Wang)