[01:15:43] <jinxer-wm>	 FIRING: [4x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[01:20:43] <jinxer-wm>	 FIRING: [9x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[01:30:43] <jinxer-wm>	 RESOLVED: [9x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[03:50:43] <jinxer-wm>	 FIRING: [8x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[03:55:43] <jinxer-wm>	 FIRING: [9x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[04:00:43] <jinxer-wm>	 FIRING: [9x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[04:05:43] <jinxer-wm>	 RESOLVED: [9x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[13:30:00] <gehel>	 I'm having another look at T406222. I'll start following https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service in a few minutes unless someone tells me to stop.
[13:30:01] <stashbot>	 T406222: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222
[13:33:17] <gehel>	 if someone could review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1216793
[13:47:51] <wikibugs>	 06Traffic, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work, 13Patch-For-Review: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222#11443942 (10Gehel) deploying following instructions in https://wikitech.wikimedia.org/wiki/LVS#Confi...
[13:48:02] <fabfur>	 lgfm too!
[13:58:35] <gehel>	 fabfur: thx
[13:58:48] <wikibugs>	 06Traffic, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work, 13Patch-For-Review: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222#11443987 (10Gehel) This seems to be working, sending an HTTP 307 redirect to one of the druid node:...
[14:09:19] <gehel>	 fabfur: if you have a minute to check on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1216797 and the prerequisite for merging it are fullfilled.
[14:10:52] <fabfur>	 have all previous steps been successful ?
[14:11:05] <gehel>	 as far as I know, yes.
[14:11:18] <gehel>	 successful test on https://phabricator.wikimedia.org/T406222#11443987
[14:12:09] <fabfur>	 ok so puppet ran on all lvs hosts and pybal has been restarted too I suppose
[14:13:45] <gehel>	 I completed steps 4 and 7 with no error!
[14:14:40] <fabfur>	 lgtm!
[14:14:45] <gehel>	 thx!
[14:17:11] <wikibugs>	 06Traffic, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work, 13Patch-For-Review: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222#11444052 (10Gehel) 05Open→03Resolved
[14:22:04] <wikibugs>	 06Traffic, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work, 13Patch-For-Review: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222#11444059 (10Gehel) HTTP calls to `druid-public-coordinator.svc.eqiad.wmnet:8081` result in an HT...
[15:53:10] <wikibugs>	 06Traffic, 06Security-Team, 10WMF-General-or-Unknown, 07ContentSecurityPolicy, 13Patch-For-Review: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618#11444306 (10ssingh) Thanks for the update @sbassett. Noting that Traffic requested the move to January.
[17:35:46] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143 (10colewhite) 03NEW
[17:36:17] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11444783 (10colewhite)
[17:41:26] <wikibugs>	 06Traffic, 06SRE: Getting forbidden from public CI runners on forgejo with opendatasync - https://phabricator.wikimedia.org/T412142#11444796 (10Reedy)
[18:02:32] <wikibugs>	 06Traffic, 06SRE: Getting forbidden from public CI runners on forgejo with opendatasync - https://phabricator.wikimedia.org/T412142#11444919 (10taavi) Please provide the full HTTP response body, which either details why the request is blocked or contains an internal identifier for us to locate the relevant WAF...
[18:21:37] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11444984 (10ayounsi) Looks like a repeat of {T398433} but for a different switch. As it's not in a VXLAN fabric, we should look at upgrading that one switch.
[18:32:44] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178#11445009 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=2a98251c-6798-469c-a3de-57fcfb13969f) set by cmooney@cumin1003 for 2:00:00 on 17 host(s) and t...
[18:36:43] <wikibugs>	 06Traffic, 06SRE: Getting forbidden from public CI runners on forgejo with opendatasync - https://phabricator.wikimedia.org/T412142#11445020 (10EvanCarroll) ` <p>Our servers are currently under maintenance or experiencing a technical issue</p> </div> </div> <div class="footer"><p>If you report this error to th...
[18:39:23] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11445028 (10colewhite) List of affected hosts: ` asw1-b3-magru asw1-b4-magru asw1-bw27-esams asw1-by27-esams cloudsw1-b1-codfw lsw1-a2-codfw lsw1-a3-codfw lsw1-a4-codfw lsw1-a...
[18:41:45] <tchin>	 Hello! Where do rate limits for k8 services live? I'm trying to figure out eventgate-analytics-external rate limits and there was a patch in varnish for it that was abandoned because it would be 'better served by confd rules' but I don't know where that is
[18:42:16] <sukhe>	 tchin: what kind of rate limits are we talking about, sorry?
[18:48:12] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178#11445044 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1aee9e7e-d36b-4c56-8cac-746f48098c6f) set by cmooney@cumin1003 for 2:00:00 on 2 host(s) and th...
[18:48:28] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11445046 (10ayounsi) Thanks, that's a lot !! I guess we will have to start 2026 with a switch upgrade campaign...
[18:48:42] <tchin>	 @sukhe I didn't even know there were different kinds but I'm basically trying to find what's the maximum amount of requests per second intake-analytics.wikimedia.org is limited to. Right now according to grafana the service gets 4.3k requests per second but that might get doubled or tripled
[18:49:14] * tchin Somewhat related is this task https://phabricator.wikimedia.org/T407570
[18:50:09] <sukhe>	 tchin: ok thanks. can you share the "better served by confd rules"?
[18:50:31] * tchin It was a comment left on this abandoned patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/768028
[18:50:36] <sukhe>	 thanks, reading
[18:51:01] <sukhe>	 ah well that was from some time ago and things have changed
[18:51:11] <sukhe>	 let me check the current VCL and other rate-limits
[18:51:27] <bblack>	 is the context actually  https://phabricator.wikimedia.org/T407570 or something else?
[18:52:00] <sukhe>	 tchin: nothing on our end. but intake-analytics needs to be checked with #wikimedia-analytics, so please consider that
[18:53:26] * tchin @bblack it's related to that, but the tickets are restricted since it's about bot detection, but I can add you to it if you want
[18:54:29] * tchin Whoops wrong reply, so there's no rate limits on that endpoint at all? It would just be a k8 scaling problem
[18:55:21] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11445078 (10ayounsi) a:03ayounsi
[18:56:02] <sukhe>	 tchin: yes, nothing in the VCL or the other place where we put ratelimits. but analytics/k8s may have their own set of ratelimits and we (I?) am not sure about those
[19:02:49] <tchin>	 Cool thanks! eventgate is on wikikube not the analytics cluster so I guess I'll have to ask whoever manages that
[19:03:42] <mutante>	 there are ratelimits in the kubernetes service mesh configuration
[19:03:49] <mutante>	 tchin: ask over in #wikimedia-serviceops
[19:05:00] <mutante>	 this repo could potentially include them: https://gerrit.wikimedia.org/r/q/project:operations/deployment-charts
[19:05:19] <mutante>	 maybe modules/mesh/configuration_1.14.1.tpl
[19:10:05] <wikibugs>	 06Traffic, 10DNS, 10wikimediafoundation.org, 07IPv6: wikimediafoundation.org does not support IPv6 - https://phabricator.wikimedia.org/T403269#11445107 (10SCampos-WMF) Hi @BCornwall, The IPv6 address is `2a04:fa87:fffd::c000:4280`   `host wikimediafoundation-org.go-vip.net` `wikimediafoundation-org.go-vip....
[19:13:24] <wikibugs>	 06Traffic, 06SRE: Getting forbidden from public CI runners on forgejo with opendatasync - https://phabricator.wikimedia.org/T412142#11445149 (10EvanCarroll) Full response body from the runner,  ` <html lang="en"> <meta charset="utf-8"> <title>Wikimedia Error</title> <style> * { margin: 0; padding: 0; } body {...
[19:27:16] <wikibugs>	 06Traffic, 06SRE: Getting forbidden from public CI runners on forgejo with opendatasync - https://phabricator.wikimedia.org/T412142#11445164 (10Dzahn) @EvanCarroll  thanks! we are able to detect the rule that was causing this and made an edit to exempt you. Please try again now.
[19:31:44] <wikibugs>	 06Traffic, 06SRE: Getting forbidden from public CI runners on forgejo with opendatasync - https://phabricator.wikimedia.org/T412142#11445177 (10EvanCarroll) Joy! Thanks a ton! Sent over a LinkedIn req, I'll give you a shout out and write a recommendation for the prompt assistance.
[19:37:34] <wikibugs>	 06Traffic, 06SRE: Getting forbidden from public CI runners on forgejo with opendatasync - https://phabricator.wikimedia.org/T412142#11445188 (10taavi) 05Open→03Resolved a:03Dzahn
[20:56:26] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Nokia: how to approach schema differences in SR-Linux versions - https://phabricator.wikimedia.org/T412157 (10cmooney) 03NEW p:05Triage→03High
[23:03:15] <wikibugs>	 06Traffic, 07Essential-Work, 05MW-1.46-notes (1.46.0-wmf.5; 2025-12-02), 06Test Kitchen (Experiment Platform Sprint 16): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11445845 (10Sfaci) The ticket is `BLOCKED` because there wa...