[09:05:33] 06serviceops, 06MediaWiki-Platform-Team: Set up a rest-gateway deployment for rate limiting testing - https://phabricator.wikimedia.org/T402914#11142783 (10daniel) There's different setups I would want to test here, but considering that we will be using an Evnoy version less than 1.33 for a while, I propose we... [09:17:46] 06serviceops, 13Patch-For-Review: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11142859 (10Ladsgroup) That sounds good to me. Thank you Scott for the investigation! [09:40:54] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Reimage sretest2009 as a wikikube worker and assess performance - https://phabricator.wikimedia.org/T400871#11142997 (10Clement_Goubert) We're not using that tag, go ahead and remove it. [10:32:53] 06serviceops: Increased latency on mw-web, mw-api-int, mw-ext-api - https://phabricator.wikimedia.org/T403606 (10jijiki) 03NEW [10:34:36] 06serviceops: Increased response times linked to Module:Citation/CS1 changes - https://phabricator.wikimedia.org/T403606#11143272 (10jijiki) 05Open→03Resolved p:05Triage→03High [10:48:49] 06serviceops: Monitor Template and Module changes on large wikis and ship events to Logstash - https://phabricator.wikimedia.org/T403608 (10jijiki) 03NEW [12:59:00] 06serviceops, 06cloud-services-team, 06SRE: hosts failing puppet compile due to missing secrets - https://phabricator.wikimedia.org/T274392#11143753 (10fgiunchedi) 05Open→03Invalid The drift between public and private private.git will continue to be an issue until we get serious about secrets managem... [13:07:51] 06serviceops, 06DC-Ops, 06SRE: Reimage sretest2009 as a wikikube worker and assess performance - https://phabricator.wikimedia.org/T400871#11143828 (10Jhancock.wm) [13:12:30] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install wikikube-ctrl2006 - https://phabricator.wikimedia.org/T400661#11143867 (10Jhancock.wm) [13:15:02] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install wikikube-ctrl2006 - https://phabricator.wikimedia.org/T400661#11143881 (10Jhancock.wm) @jasmine_ how do you feel about the server going in row D? doesn't look like we have one in that row. [13:19:05] 06serviceops: Increased response times linked to Module:Citation/CS1 changes - https://phabricator.wikimedia.org/T403606#11143898 (10matmarex) [14:25:13] 06serviceops, 06cloud-services-team, 06SRE: hosts failing puppet compile due to missing secrets - https://phabricator.wikimedia.org/T274392#11144118 (10Dzahn) It was just about a social contract to not forget adding fake secrets when adding real secrets. [16:21:05] 06serviceops, 13Patch-For-Review: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11144609 (10Scott_French) As of ~ 15:58 UTC, `CategoryCountUpdateJob` is now on a dedicated rule with concurrency 50 ([[ https://grafana.wikimedia.org/goto/h_o_Cd9N... [17:11:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11144882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host maps1011.eqiad.wmnet with OS bookworm [17:45:14] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145046 (10VRiley-WMF) [17:56:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145097 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host maps1012.eqiad.wmnet with OS bookworm [18:04:14] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145152 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host maps1011.eqiad.wmnet with OS bookworm completed: - maps1011 (**PASS**) - Rem... [18:36:25] 06serviceops: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421#11145297 (10Scott_French) [18:37:11] 06serviceops: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421#11145298 (10Scott_French) 05In progress→03Resolved [18:47:52] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145333 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host maps1012.eqiad.wmnet with OS bookworm completed: - maps1012 (**PASS**) - Rem... [18:50:07] 06serviceops: Configure mw-next-routing for PHP 8.3 - https://phabricator.wikimedia.org/T403655 (10Scott_French) 03NEW [18:51:11] 06serviceops: Configure mw-next-routing for PHP 8.3 - https://phabricator.wikimedia.org/T403655#11145349 (10Scott_French) [18:51:53] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145354 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host maps1013.eqiad.wmnet with OS bookworm [18:53:25] 06serviceops: Configure mw-next-routing for the PHP 8.3 migration - https://phabricator.wikimedia.org/T403655#11145356 (10Scott_French) [18:54:24] 06serviceops: Configure mw-next-routing for the PHP 8.3 migration - https://phabricator.wikimedia.org/T403655#11145360 (10Scott_French) [19:01:40] 06serviceops: Configure the WikimediaEvents extension for the PHP 8.3 migration - https://phabricator.wikimedia.org/T403657 (10Scott_French) 03NEW [19:35:04] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145493 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host maps1014.eqiad.wmnet with OS bookworm [19:47:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host maps1013.eqiad.wmnet with OS bookworm completed: - maps1013 (**PASS**) - Rem... [19:47:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145535 (10VRiley-WMF) [20:04:30] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663 (10RLazarus) 03NEW [20:30:35] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11145687 (10RLazarus) [20:31:59] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11145695 (10RLazarus) Removed the tracing item > (1.29.10) tracing: Fixed a bug where the OpenTelemetry tracer exports the OTLP request even when no spans are present. after consulting with our tracing exper... [20:32:22] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145696 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host maps1014.eqiad.wmnet with OS bookworm completed: - maps1014 (**PASS**) - Rem... [20:32:44] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145697 (10VRiley-WMF) [20:33:00] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11145701 (10VRiley-WMF) 05Open→03Resolved this is resolved [23:42:51] 06serviceops: Create dedicated changeprop-jobqueue rule for CategoryCountUpdateJob - https://phabricator.wikimedia.org/T402873#11146176 (10Scott_French) 05In progress→03Resolved Nearly 8 hours on from with switch, everything still seems to be working as expected. As promised, resolving this. Thanks again...