[00:21:25] 06serviceops, 06SRE, 10Wikifeeds: 504 responses (gateway timeout) for /api/rest_v1/feed/featured - https://phabricator.wikimedia.org/T400425#11033423 (10Scott_French) One additional data point: To find out exactly how slow these requests are (if indeed they succeed) I `nsenter`'d the netns of a wikifeeds po... [00:39:18] 06serviceops, 06SRE, 10Wikifeeds: 504 responses (gateway timeout) for /api/rest_v1/feed/featured - https://phabricator.wikimedia.org/T400425#11033433 (10Scott_French) [00:58:33] 06serviceops, 06SRE, 10Wikifeeds: 504 responses (gateway timeout) for /api/rest_v1/feed/featured - https://phabricator.wikimedia.org/T400425#11033437 (10Scott_French) [03:18:51] 06serviceops, 06SRE, 10Wikifeeds, 07Chinese-Sites: 504 responses (gateway timeout) for /api/rest_v1/feed/featured - https://phabricator.wikimedia.org/T400425#11033508 (10Shizhao) [08:07:48] 06serviceops, 10decommission-hardware: decommission mwmaint1002.eqiad.wmnet and mwmaint2002.codfw.wmnet - https://phabricator.wikimedia.org/T400442#11033826 (10Clement_Goubert) a:03jasmine_ [08:07:54] 06serviceops, 10decommission-hardware: decommission mwmaint1002.eqiad.wmnet and mwmaint2002.codfw.wmnet - https://phabricator.wikimedia.org/T400442#11033829 (10Clement_Goubert) [08:07:59] 06serviceops, 10MW-on-K8s: Turn down mwmaint production servers - https://phabricator.wikimedia.org/T397017#11033830 (10Clement_Goubert) [08:09:00] 06serviceops, 10decommission-hardware: decommission mwmaint1002.eqiad.wmnet and mwmaint2002.codfw.wmnet - https://phabricator.wikimedia.org/T400442#11033843 (10Clement_Goubert) @jasmine_ Could you take care of this next week please? Thanks a bunch :) [08:38:06] 06serviceops, 06Structured-Data-Backlog, 10Thumbor: Thumbor workers hang indefinitely when conducting some tiff operations, leading to user-facing error - https://phabricator.wikimedia.org/T374350#11033892 (10Clement_Goubert) Happened again, 2 pods wedged on all tiff conversions on all 4 workers. ` for pod i... [08:43:50] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work: Ensure REST Gateway headers don't conflict with Mediawiki headers - https://phabricator.wikimedia.org/T400346#11033895 (10Clement_Goubert) a:03Clement_Goubert [10:10:37] 06serviceops, 10MediaWiki-extensions-OAuth, 10MediaWiki-Platform-Team (Roadmap), 07Technical-Debt, 07Upstream: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream - https://phabricator.wikimedia.org/T261462#11034161 (10Tgr) I'm trying to make sense of the exact fork situation, with... [11:08:28] 06serviceops, 06Structured-Data-Backlog, 10Thumbor: Thumbor workers hang indefinitely when conducting some tiff operations, leading to user-facing error - https://phabricator.wikimedia.org/T374350#11034298 (10jijiki) Sorry for the driveby comment and it may be a longshot, we keep seeing a lot of `:-` in the... [11:18:44] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11034336 (10Fabfur) The image used for debci building (bullseye) is still affected by the bullseye-backports issue: ` ~ > podman run -it --rm docker-... [11:44:24] 06serviceops, 10MediaWiki-extensions-OAuth, 10MediaWiki-Platform-Team (Roadmap), 07Technical-Debt, 07Upstream: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream - https://phabricator.wikimedia.org/T261462#11034368 (10Tgr) IMO the next step would be to make it clear how the fork d... [13:43:53] 06serviceops: Identify areas covered by the Production Readiness checkilist - https://phabricator.wikimedia.org/T400476 (10jijiki) 03NEW [14:02:30] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Ensure REST Gateway headers don't conflict with Mediawiki headers - https://phabricator.wikimedia.org/T400346#11034736 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium [14:03:08] 06serviceops, 06MW-Interfaces-Team, 07Epic, 07OKR-Work: Central REST gateway for APIs - https://phabricator.wikimedia.org/T400130#11034740 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [14:13:28] 06serviceops: Create a template on wikitech - https://phabricator.wikimedia.org/T400481 (10jijiki) 03NEW [14:22:28] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11034824 (10Scott_French) Thanks, @Fabfur - If this is urgent, a manual rebuild of the [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dock... [14:31:44] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557#11034843 (10Fabfur) Thanks a lot @Scott_French ! This weekend is fine, I'll retry on Monday! [14:34:29] 06serviceops: Create a template on wikitech - https://phabricator.wikimedia.org/T400481#11034859 (10jijiki) [15:15:42] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485 (10RobH) 03NEW [15:16:03] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11035008 (10RobH) [15:16:51] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11035012 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) a... [15:54:59] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work: Capacity plan for REST gateway - https://phabricator.wikimedia.org/T400490 (10hnowlan) 03NEW [20:44:39] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11036005 (10Eevans) [20:48:15] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11036009 (10Eevans)