[01:33:40] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10079939 (10Krinkle) [02:19:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10079967 (10CDanis) p:05Low→03Triage [02:20:02] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10079969 (10CDanis) [02:20:04] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10079968 (10CDanis) [02:20:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10079965 (10CDanis) I think in light of T370304 we need to prioritize this -- and a proper version, not just creating static placeholder records. The good news is, we already have the data... [07:22:30] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar), 10Sustainability (Incident Followup): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10080170 (10Peachey88) [08:56:08] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10080323 (10JMeybohm) For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug Pod to resolve the PTRs is not really neces... [09:20:59] 06serviceops, 06Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 2 others: Allow connections from PCS to eventgate - https://phabricator.wikimedia.org/T368052#10080374 (10Jgiannelos) 05Open→03Resolved [09:26:03] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q1): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265#10080397 (10fgiunchedi) p:05Medium→03Low Back to low as it turns out I went down t... [09:35:03] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#10080434 (10Mvolz) [09:39:59] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#10080424 (10Mvolz) FYI Citoid upgrade is blocked by dependency on service-runner: https://github.com/wikimedia/service-ru... [09:40:02] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#10080435 (10Mvolz) [09:41:23] 06serviceops, 10observability, 06SRE: aggregate mismatched wikiversions alert - https://phabricator.wikimedia.org/T302832#10080436 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is actually done as the warnings don't spam irc [09:56:51] 06serviceops, 10Deployments, 10Shellbox, 10Wikibase-Quality-Constraints, and 2 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10080652 (10Clement_Goubert) This is most likely caused by envoy terminating before mediaw... [10:12:18] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T372916#10080728 (10Clement_Goubert) Apparently the removal from the puppetserver wasn't properly done by the cookbook, I've done it manually and it should resolve. Sor... [12:20:23] 06serviceops, 06Infrastructure-Foundations, 10netops, 06Traffic: weighted maglev viability for low-traffic services - https://phabricator.wikimedia.org/T368545#10081016 (10Vgutierrez) 05Open→03Resolved [13:25:56] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#10081253 (10Ottomata) FYI, [[ https://gitlab.wikimedia.org/tchin/service-utils/-/tree/main?ref_type=heads | service-utils... [13:27:06] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10081276 (10CDanis) >>! In T344171#10080323, @JMeybohm wrote: > For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug P... [14:11:14] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10081520 (10Andrew) Will this be moot after the move to k8s, or is this blocking the move to k8s? [14:53:24] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T372916#10081763 (10Jhancock.wm) [14:54:25] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T372916#10081757 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm np! things happen. looking out for each other. label has been changed! [14:55:28] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10081765 (10Clement_Goubert) [14:55:57] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T372916#10081766 (10Clement_Goubert) Thank you! [15:00:59] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar), 10Sustainability (Incident Followup): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10081785 (10CDanis) @Ladsgroup @Marosteg... [15:17:10] 06serviceops: kafka-main replacement nodes don't fit kafka-main (storage wise) - https://phabricator.wikimedia.org/T368714#10081850 (10JMeybohm) 05Open→03Resolved [15:29:58] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10081906 (10bd808) >>! In T371592#10081520, @Andrew wrote: > Will this be moot after the move to k8s, or is this block... [15:35:34] 06serviceops, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10081944 (10Clement_Goubert) [16:16:46] 06serviceops, 10Deployments, 10Shellbox, 10Wikibase-Quality-Constraints, and 3 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10082100 (10Lydia_Pintscher) [16:16:58] 06serviceops, 10Deployments, 10Shellbox, 10Wikibase-Quality-Constraints, and 4 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10082102 (10Lydia_Pintscher) [17:00:03] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10082373 (10Andrew) Right, but after the sul migration the ldap extension will be dead code, right? Or does it need to... [18:17:11] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar), 10Sustainability (Incident Followup): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10082614 (10CDanis) p:05Triage→03High [18:17:57] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10082616 (10bd808) >>! In T371592#10082373, @Andrew wrote: > Right, but after the sul migration the ldap extension wil... [18:35:00] 06serviceops, 13Patch-For-Review: Prepare WMF PHP 8.1 packages for Bullseye - https://phabricator.wikimedia.org/T372507#10082651 (10Scott_French) Verified that in a fresh docker-registry.wikimedia.org/bullseye:latest image, I can successfully: ` echo 'deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia... [18:53:31] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10082693 (10Andrew) Yep, that makes sense. I was confused because it popped up in our inbox as though it was an isolat... [21:59:16] 06serviceops, 13Patch-For-Review: Prepare WMF PHP 8.1 packages for Bullseye - https://phabricator.wikimedia.org/T372507#10083213 (10Scott_French) While working through the production image definitions for T372602, I discovered that the three extension packages maintained by WMF (php-luasandbox, php-wmerrors, w... [23:56:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10083332 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1298.eqiad.wmnet with OS bull...