[08:56:09] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10080323 (10JMeybohm) For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug Pod to resolve the PTRs is not really neces... [09:58:08] 06Traffic, 06Data-Engineering, 13Patch-For-Review: Install benthos on single esams host to check performances under higher load - https://phabricator.wikimedia.org/T365968#10080661 (10Fabfur) 05Open→03Declined Superseded by T370668 [09:59:06] 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Install new Benthos instance on cp hosts - https://phabricator.wikimedia.org/T358109#10080656 (10Fabfur) 05In progress→03Declined Superseded by T370668 [10:00:09] 06Traffic, 10Observability-Logging: Add metrics to Benthos - https://phabricator.wikimedia.org/T361845#10080677 (10Fabfur) 05Open→03Declined Superseded by T370668 [10:00:57] 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Benthos loses messages when under high load - https://phabricator.wikimedia.org/T364379#10080666 (10Fabfur) 05Open→03Declined Superseded by T370668 [10:00:59] 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Better Benthos performances - https://phabricator.wikimedia.org/T360454#10080671 (10Fabfur) 05Open→03Declined Superseded by T370668 [10:01:13] 06Traffic, 06Data-Engineering: Upgrade Benthos package on cp hosts - https://phabricator.wikimedia.org/T366031#10080698 (10Fabfur) 05Open→03Declined Superseded by T370668 [10:01:17] 06Traffic, 06Data-Engineering, 10Observability-Logging: Check statsv and eventlogging VarnishKafka instances - https://phabricator.wikimedia.org/T359178#10080682 (10Fabfur) 05Open→03Declined Superseded by T370668 [10:02:07] 06Traffic, 06Data-Engineering, 10Observability-Logging: Umbrella task for Benthos parsing error - https://phabricator.wikimedia.org/T365441#10080693 (10Fabfur) 05Open→03Declined Superseded by T370668 [10:02:27] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: HAProxy log format doesn't support "invalid" request path - https://phabricator.wikimedia.org/T365117#10080687 (10Fabfur) 05Open→03Declined Superseded by T370668 [10:55:38] 10netops, 06Infrastructure-Foundations, 06SRE: Packet loss reflected in NELs for traffic to Reliance Jio Infocomm Ltd over BBIX Singapore - https://phabricator.wikimedia.org/T373015 (10cmooney) 03NEW p:05Triage→03Low [11:23:35] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Packet loss reflected in NELs for traffic to Reliance Jio Infocomm Ltd over BBIX Singapore - https://phabricator.wikimedia.org/T373015#10080912 (10cmooney) Path is now avoided again and packet loss is no longer observable: ` cmooney@bast500... [12:20:24] 10netops, 06Traffic, 06Infrastructure-Foundations, 06serviceops: weighted maglev viability for low-traffic services - https://phabricator.wikimedia.org/T368545#10081016 (10Vgutierrez) 05Open→03Resolved [12:43:44] 06Traffic: Remove katran blockers for low-traffic non-k8s based services - https://phabricator.wikimedia.org/T373020 (10Vgutierrez) 03NEW [12:45:06] 06Traffic: Remove katran blockers for low-traffic non-k8s based services - https://phabricator.wikimedia.org/T373020#10081137 (10Vgutierrez) p:05Triage→03Medium [12:49:08] 06Traffic: Remove katran blockers for low-traffic non-k8s based services - https://phabricator.wikimedia.org/T373020#10081149 (10Vgutierrez) [13:27:06] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10081276 (10CDanis) >>! In T344171#10080323, @JMeybohm wrote: > For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug P... [13:31:34] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10081293 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f69725b2-8f49-49fe-8766-ce7bb9ffa253) set by... [13:33:55] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10081302 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b60d0e72-74ce-4dee-9bed-2acca82f8655) set by... [13:55:33] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Core router error logs: "sshd: Did not receive identification string" from prometheus hosts - https://phabricator.wikimedia.org/T368513#10081413 (10ayounsi) Confirmed that cr1-eqiad stopped generating those logs for 10.64.0.82 (prometheus10... [14:03:18] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Core router error logs: "sshd: Did not receive identification string" from prometheus hosts - https://phabricator.wikimedia.org/T368513#10081484 (10ayounsi) 05Open→03Resolved a:03ayounsi All done ! [14:08:01] 06Traffic, 06Security-Team, 10WMF-General-or-Unknown, 07ContentSecurityPolicy, 13Patch-For-Review: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618#10081492 (10ssingh) Following up on the patch by @TheDJ (thanks!) and per https://gerrit.wikimedia.org/r/c/operations/... [14:17:21] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 10Release-Engineering-Team (Radar): implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10081536 (10Jelto) After reviewing the `DENYLIST` and the nftables logs, we noticed that so... [14:18:49] 06Traffic: Add support for mh-port scheduler flag on pybal - https://phabricator.wikimedia.org/T373027 (10Vgutierrez) 03NEW [14:23:19] 06Traffic, 13Patch-For-Review: Add support for mh-port scheduler flag on pybal - https://phabricator.wikimedia.org/T373027#10081573 (10Vgutierrez) p:05Triage→03Medium [14:32:50] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: PuppetDB import failing for lvs2014 - https://phabricator.wikimedia.org/T372931#10081634 (10cmooney) 05Open→03Resolved a:03cmooney Working ok following update to the puppetdb import script. The issue was not actually caused by Ne... [14:47:27] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10081718 (10cmooney) a:05cmooney→03None Work completed, no issues to report (although I had to downgrade the NIC firm... [14:55:28] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10081765 (10Clement_Goubert) [15:35:34] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10081944 (10Clement_Goubert) [18:09:50] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048 (10Krinkle) 03NEW [18:10:21] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10082590 (10Krinkle) \cc @ladsgroup FYI ^ [18:14:03] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10082609 (10Krinkle) When using [WikimediaDebug](https://wikitech.wikimedia.org/wiki/WikimediaDebug) and selecting a server outside Kube... [18:54:31] 10Wikimedia-Apache-configuration, 06MW-Interfaces-Team, 10MW-on-K8s, 07Regression: https://en.wikipedia.org/api/ 404 Not Found - https://phabricator.wikimedia.org/T373048#10082695 (10Jdforrester-WMF) Also https://www.mediawiki.org/api/ etc. [20:58:11] 06Traffic: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10083046 (10BCornwall) [20:58:41] 06Traffic: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10083048 (10BCornwall)