[00:13:04] 10Beta-Cluster-Infrastructure: Logins seem not to work after swtich to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349 (10bd808) 03NEW [00:13:20] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10997004 (10bd808) [00:20:29] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10997018 (10bd808) p:05Triage→03High There are //a lot// of `Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER ERROR` errors i... [00:43:33] 10Beta-Cluster-Infrastructure, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08), 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10997047 (10bd808) [00:43:34] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10997048 (10bd808) [00:44:45] 10Beta-Cluster-Infrastructure, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08), 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10997050 (10bd808) {T399349} feels related, but maybe login was already broken and I just didn't notice because I had a work... [12:02:51] 06Release-Engineering-Team, 06collaboration-services, 10mwcli, 06serviceops-radar: Create /nonexistent directory for nobody user in golang images - https://phabricator.wikimedia.org/T331209#10997260 (10Addshore) @Dzahn so I don't actually use these images any more, as the golang ones from docker hub are ac... [14:29:43] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10997334 (10BPirkle) I'm seeing the errors mentioned in the task description when trying to log in to https://en.wikipedia.beta.wmcloud.org/wiki/Main_Page Perso... [16:04:53] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10997373 (10Tgr) Yeah these are both memcached errors. Tokens are stored in the micro stash, which indeed seems broken. ` mwsctgr@deployment-mwmaint03:~$ mwscrip... [16:19:50] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10997386 (10Tgr) Microstash uses the `mcrouter-primary-dc` object cache config, which is ` [ "class" => "MemcachedPeclBagOStuff", "serializer" => "php",... [16:54:52] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10997413 (10Tgr) A manual test provides marginally more information (seems like the error logging of MemcachedPeclBagOStuff is buggy): ` tgr@deployment-mediawiki... [17:32:56] 10Release-Engineering-Team (Priority Backlog 📥), 10Code-Stewardship-Reviews, 07Documentation: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329#10997437 (10A_smart_kitten) >>! In T294329#8085948, @thcipriani wrote: > […] > 2.... [17:35:41] 10Release-Engineering-Team (Priority Backlog 📥), 10Code-Stewardship-Reviews, 07Documentation: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329#10997439 (10Novem_Linguae) Combining those 3 items in one patch is probably fine s... [18:37:08] 10Continuous-Integration-Config: Create a non-voting CI job to flag deprecated code - https://phabricator.wikimedia.org/T388948#10997446 (10Tacsipacsi) >>! In T388948#10967892, @Daimona wrote: >>>! In T388948#10769160, @Michael wrote: >>>>! In T388948#10638982, @Jdforrester-WMF wrote: >>> Loathe as I am to add Y... [18:54:56] 10Phabricator (Upstream), 10Release-Engineering-Team (Radar), 07Developer Productivity, 07Upstream: Add Open Graph support to Phabricator Maniphest Tasks to have link preview on Telegram, Slack, and other messaging apps - https://phabricator.wikimedia.org/T288117#10997460 (10Novem_Linguae) Sounds like an u... [20:56:56] I'm looking at why it is that beta pageviews often take 10-20 seconds to load, when e.g. hitting load.php is fast even for an uncached cache-custed URL. [20:56:58] https://performance.wikimedia.beta.wmcloud.org/excimer/profile/62ff425d0c51178c [20:57:35] It seems a lot of time is spent figuring out thumbnail information. I guess that would be consistent with memcached being broken, i.e. each one a serial API call to Commons. [20:59:32] the repeat calls are still weird, given there is only one image on the Main_Page and the call has a pcTTL (in-process cache) as well.. [21:26:55] 10Continuous-Integration-Config: Create a non-voting CI job to flag deprecated code - https://phabricator.wikimedia.org/T388948#10997516 (10Daimona) >>! In T388948#10997446, @Tacsipacsi wrote: > But then we’re back at adding yet another CI job. So yes, the ideal solution would be a job having three possible resu... [22:24:00] !log Add port 11212 to 'default' security group in deployment-prep, similar to Redis, ref T399349 [22:24:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:24:03] T399349: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349 [22:24:19] !log Add port 11212 for Memcached to 'default' security group in deployment-prep (TCP, IPv4+IPV6), similar to Redis, ref T399349 [22:24:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL