[00:25:29] 06serviceops, 07Datacenter-Switchover: Verify our current wikikube capacity (in both DCs) can handle all our traffic - https://phabricator.wikimedia.org/T371273#10101517 (10Scott_French) Spent a bit of time thinking about this today. If we think in terms of a desired target php-fpm worker utilization and look... [06:19:57] I send a series of welcome messages to new members on their user talk pages. Would this be considered vandalism? (Is it prohibited?) [08:24:05] I don't think anyone in this channel can authoritatively answer that question. Ask in the equivalent of the village pump of your wiki. [08:25:24] Oh, they left. After having asked in other channels too [08:28:11] 06serviceops, 06Release-Engineering-Team: Investigate scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10101970 (10hashar) Branch cut fails ==== The automatic branch cut failed: [[ https://releases-jenkins.wikimedia.org/job/Automatic%20branc... [08:28:13] 06serviceops, 06Release-Engineering-Team: Investigate scap fails to create /srv/mediawiki-staging/php-1.43.0-wmf.20/cache/l10n - https://phabricator.wikimedia.org/T373425#10101973 (10hashar) a:05hashar→03None [08:31:55] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10LPL Essential (LPL Essential 2024 Jul-Sep): Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10101982 (10PWaigi-WMF) p:05Triage→03Medium a:03santhosh [09:31:30] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102107 (10Clement_Goubert) [09:46:54] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102173 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by cgoubert@cumin1002 Renumb... [09:48:18] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102176 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [10:34:17] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102294 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [10:37:56] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102310 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin1002 Renumberin... [10:39:29] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10LPL Essential (LPL Essential 2024 Jul-Sep): Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10102336 (10santhosh) > > 1. **Machine-Translate Content Endpoint** > - **HTTP Verb:** `POST` > - *... [10:47:28] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102361 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from mw2380 to w... [10:48:26] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102363 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wi... [10:48:32] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102364 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [10:49:18] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102365 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wi... [10:49:26] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102366 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [10:55:49] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373591 (10akosiaris) 03NEW [11:02:37] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102411 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by cgoubert@cumin1002 Renumb... [11:06:24] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102413 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [11:13:54] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102420 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wi... [11:52:09] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102445 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikiku... [11:55:16] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102464 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [12:00:52] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10102470 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin1002 Renumberin... [12:52:27] 06serviceops, 06Data-Platform-SRE, 06Infrastructure-Foundations, 06Machine-Learning-Team, 13Patch-For-Review: Migrate the ownership of Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T373526#10102566 (10elukey) >>! In T373526#10100630, @akosiaris wrote: > I se... [13:08:43] 06serviceops, 10Deployments, 10Shellbox, 10Wikibase-Quality-Constraints, and 4 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10102614 (10ItamarWMDE) Is this somehow related to {T362084}? [14:03:41] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373591#10102839 (10hnowlan) [14:13:21] hnowlan: I am deploying the patch for the user-agent on restbase and i will try to redeploy mobileapps with cassandra enabled [14:16:48] nemo-yiannis: o/ did the new password work yesterday? [14:16:55] yes [14:17:24] nemo-yiannis: ack - I am about to go into a meeting but I will keep an eye [14:54:00] i saw the user agents on the tls proxy logs [14:54:09] i am gonna try redeploying mobileapps [15:02:29] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Memcached: Migrate wikitech to main memcached - https://phabricator.wikimedia.org/T371608#10103142 (10jijiki) [15:09:17] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10103156 (10jijiki) [15:11:30] hnowlan: done [15:20:40] nemo-yiannis: all good so far? [15:22:01] i think so [15:22:22] i mean, i checked already yesterday and things worked OK manually other than the fact that RESTBase didn't work as expected [15:22:36] i don't expect something to go horribly wrong with this deployment [15:23:23] i will keep an eye and if its stable we can try sending some traffic from changeprop next week [15:25:11] ack, sgtm. I should be a bit more available next week [15:32:06] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10103278 (10jijiki) [15:32:24] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Memcached: Migrate wikitech to main memcached - https://phabricator.wikimedia.org/T371608#10103286 (10jijiki) [15:32:28] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10103287 (10jijiki) [15:32:38] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10103288 (10jijiki) [15:32:55] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org, 13Patch-For-Review: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10103309 (10jijiki) [15:33:01] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10103310 (10jijiki) [15:33:43] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: mediawiki-config: consolidate labswiki - https://phabricator.wikimedia.org/T371374#10103316 (10jijiki) [15:33:50] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10103273 (10jijiki) [15:38:12] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10103341 (10jijiki) [15:46:34] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Add a banner on wikitech regarding upcoming authentication changes - https://phabricator.wikimedia.org/T373615 (10jijiki) 03NEW [16:08:25] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Add a banner on wikitech regarding upcoming authentication changes - https://phabricator.wikimedia.org/T373615#10103490 (10bd808) Happy to help with this as needed. https://wikitech.wikimedia.org/wiki/MediaWiki:Sitenotice is the common method of running th... [17:27:11] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10103822 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw2401 to wi... [17:28:05] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10103824 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wik... [18:12:23] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10103958 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikub... [18:55:08] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T373591#10104094 (10kamila) [21:11:10] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Add a banner on wikitech regarding upcoming authentication changes - https://phabricator.wikimedia.org/T373615#10104462 (10Bugreporter) Most users do not active edit Wikitech wiki. Only those who want to edit Wikitech in this short transitional period need... [22:58:19] 06serviceops: Regenerate UcfirstOverrides.php for PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T372603#10104750 (10Scott_French) Although a bit of a process, the following will definitely work: 1. Use `mwscript-k8s` to launch `shell.php` in `--attach` mode - when doing this "for real" we'll sel... [23:52:19] 06serviceops, 10MW-on-K8s, 10Shellbox, 10SRE-swift-storage, 13Patch-For-Review: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322#10104888 (10tstarling) For configuration propagation, it turns out to be better to have the Command act as a factory, so we will have: `php $command...