[00:33:56] <wikibugs>	 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10527388 (10Scott_French)
[10:47:17] <wikibugs>	 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782 (10Clement_Goubert) 03NEW
[11:04:00] <wikibugs>	 06serviceops, 10MW-on-K8s: Document the new periodic jobs infrastructure on Wikitech - https://phabricator.wikimedia.org/T385783 (10Clement_Goubert) 03NEW
[11:04:12] <wikibugs>	 06serviceops, 10MW-on-K8s: Document the new periodic jobs infrastructure on Wikitech - https://phabricator.wikimedia.org/T385783#10528212 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium
[12:14:09] <wikibugs>	 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10528477 (10Clement_Goubert)
[12:25:43] <wikibugs>	 06serviceops, 10RESTBase, 10RESTBase Sunsetting, 06Traffic: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528510 (10MSantos) @akosiaris this is ready for work and communication has been sent Today.
[12:38:47] <wikibugs>	 06serviceops, 10RESTBase, 10RESTBase Sunsetting, 06Traffic: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528536 (10Seddon)
[12:56:23] <wikibugs>	 06serviceops, 10Thumbor: Reconsider Thumbor SSIM tests - https://phabricator.wikimedia.org/T385798#10528608 (10hnowlan)
[12:58:18] <wikibugs>	 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10528623 (10Clement_Goubert)
[13:03:00] <wikibugs>	 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1): Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799 (10Clement_Goubert) 03NEW
[13:03:48] <wikibugs>	 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1): Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10528649 (10Clement_Goubert) p:05Triage→03Medium
[13:05:37] <wikibugs>	 06serviceops, 06Data-Persistence, 10MW-on-K8s: Migrate ParserCachePurging jobs to mw-cron - https://phabricator.wikimedia.org/T385800 (10Clement_Goubert) 03NEW
[13:06:39] <wikibugs>	 06serviceops, 10RESTBase, 10RESTBase Sunsetting, 06Traffic: Block traffic to RESTBase /page/related endpoint before it's deprecated - https://phabricator.wikimedia.org/T376297#10528672 (10akosiaris) @msantos. And it's done. Thanks to [requestctl](https://wikitech.wikimedia.org/wiki/Requestctl) web interfac...
[13:55:48] <wikibugs>	 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service: testwiki: Add changeprop rule to send requests to PCS on resource change - https://phabricator.wikimedia.org/T385719#10528725 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos
[14:01:54] <wikibugs>	 06serviceops, 06Content-Transform-Team-WIP, 06Data-Persistence, 10iOS-app-feature-Performance, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#10528739 (10Jgiannelos) Currently we have testwiki and hewiki to send changeprop events to P...
[14:08:53] <wikibugs>	 06serviceops, 06Content-Transform-Team-WIP, 06Data-Persistence, 10iOS-app-feature-Performance, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#10528765 (10Jgiannelos) More specifically here is the events produced from my test:  resourc...
[14:21:20] <wikibugs>	 06serviceops, 10Deployments, 10MW-on-K8s, 06SRE, 13Patch-For-Review: mwscript-k8s gives Invalid value: "<script>": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric c... - https://phabricator.wikimedia.org/T380533#10528799
[14:29:53] <moritzm>	 FYI, I'm temporarily switching kubestagemaster1003 to DRBD
[15:29:29] <wikibugs>	 06serviceops, 10MediaWiki-Platform-Team (Radar): Please review MediaWiki Apache config changes adding new docroot for the auth domain - https://phabricator.wikimedia.org/T385228#10529105 (10matmarex) I would like to help resolve this faster and unblock my team's work. I'll submit that patch, if it's such an ea...
[15:44:18] <wikibugs>	 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1): Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10529183 (10Dreamy_Jazz) Thanks for checking for our input!  > * jobs that should be watched more The `mediamoderation-con...
[16:10:32] <wikibugs>	 06serviceops, 10MW-on-K8s: mwscript-k8s fails when there are parenthesis used in arguments - https://phabricator.wikimedia.org/T385818 (10pmiazga) 03NEW
[16:15:50] <wikibugs>	 06serviceops, 10MW-on-K8s: mwscript-k8s fails when there are parenthesis used in arguments - https://phabricator.wikimedia.org/T385818#10529385 (10Volans) @pmiazga you have weird quotes in your command line: ` $ echo “PMiazga_(WMF)” -bash: syntax error near unexpected token `(' $ echo "PMiazga_(WMF)" PMiazga_(...
[16:16:32] <wikibugs>	 06serviceops, 10MW-on-K8s: mwscript-k8s fails when there are parenthesis used in arguments - https://phabricator.wikimedia.org/T385818#10529387 (10akosiaris) I don't know how you ended up with a `“` character in your shell prompt but this works  `lang=bash deploy2002:~$ mwscript-k8s --comment='T378402' -- Cent...
[16:38:36] <wikibugs>	 06serviceops, 10Page Content Service: Dont propagate server error details to end users - https://phabricator.wikimedia.org/T385821 (10Jgiannelos) 03NEW
[16:39:32] <wikibugs>	 06serviceops, 10Page Content Service: Dont propagate server error details to end users - https://phabricator.wikimedia.org/T385821#10529509 (10Jgiannelos)
[16:42:26] <wikibugs>	 06serviceops, 10MW-on-K8s: mwscript-k8s fails when there are parenthesis used in arguments - https://phabricator.wikimedia.org/T385818#10529518 (10Lucas_Werkmeister_WMDE) (Also `--user` somehow became `—user` with an em dash, probably at the same time.)
[17:07:12] <wikibugs>	 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10529623 (10Scott_French) We've reverted the migration to 0% while T385395 is being investigated.  Depending on how quickly cookie enrolled traffic decays, we may use a patch like https...
[17:15:58] <wikibugs>	 06serviceops, 10MW-on-K8s: mwscript-k8s fails when there are parenthesis used in arguments - https://phabricator.wikimedia.org/T385818#10529691 (10pmiazga) 05Open→03Invalid Sorry, those have to probably copy and paste quirks on macos :/. Quite often I use an editor to prepare the entire command and the...
[17:59:44] <wikibugs>	 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 3 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10529813 (10EAkinloose)
[21:46:16] <wikibugs>	 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10530340 (10Scott_French) Since we're still seeing a long tail of traffic on the 8.1-based next releases, and we do not believe we need PHP_ENGINE-based routing to reproduce the issue i...
[22:24:21] <wikibugs>	 06serviceops, 10Deployments, 10MW-on-K8s, 06SRE: mwscript-k8s gives Invalid value: "<script>": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character - https://phabricator.wikimedia.org/T380533#10530441 (10RLaza...
[22:27:36] <wikibugs>	 06serviceops, 06MediaWiki-Engineering, 06Traffic, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530496 (10Scott_French) p:05Triage→03High Many thanks again, all, for the investigation today.  So, we're a...
[22:29:29] <wikibugs>	 06serviceops, 06MediaWiki-Engineering, 06Traffic, 13Patch-For-Review, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530540 (10Scott_French) @BCornwall - Likely not related to the 8.1 migration, since we didn't start ramping tra...
[23:10:42] <wikibugs>	 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10530756 (10Scott_French) Alright, all production MediaWiki traffic is back on 7.4 as of ~ 22:00 UTC. We don't intend to roll forward again until T385395 is understood and fixed.
[23:17:32] <wikibugs>	 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10530762 (10Scott_French)