[09:21:43] <jinxer-wm>	 FIRING: [5x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[09:26:43] <jinxer-wm>	 FIRING: [18x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[09:36:43] <jinxer-wm>	 RESOLVED: [18x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[14:24:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[14:27:33] <wikibugs>	 06Traffic, 10Hiddenparma, 13Patch-For-Review: Add ipblock-source objects and logic - https://phabricator.wikimedia.org/T402014#11353838 (10JMeybohm) For the migration of existing ipblock objects that are currently maintained by `fetch_external_clouds_vendors_nets.py` I suggest to: - Add calling `requestctl f...
[14:29:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[16:33:30] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11354358 (10Sfaci) I had a meeting with @BBlack from Traffic team and we talked about the question...
[16:46:50] <wikibugs>	 06Traffic, 06SRE, 05FY2025-26 WE3.3 Engaging core audiences, 06Reader Experience Team (REx Sprint 8 [Q2 Oct 21-Nov 3]): [Reading Lists] Monitor potential performance impact of Reading Lists for Web - https://phabricator.wikimedia.org/T397526#11354405 (10Jdrewniak) 05Open→03Resolved
[17:19:35] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11354523 (10mpopov) Thank you for summarizing and sharing your conversations! This is great.  I th...
[17:26:10] <wikibugs>	 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11354548 (10JVanderhoop-WMF) This is excellent. Thank you @Sfaci and @BBlack   > Register an exper...
[17:32:32] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar): Telegram previews broken since unified mobile routing - https://phabricator.wikimedia.org/T409575 (10BCornwall) 03NEW
[18:07:54] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11354607 (10RobH) I accidentally pasted the Day 1 update on a subtask:  >>! In T405945#11351182, @RobH wrote: > Day 1 of migrations update: >  > * 58 hosts mo...
[18:33:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[18:38:03] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11354665 (10RobH) Day 2 update:  * 73 servers moved today, 169 servers remain. * We (again) focused on moving hosts that did not require any specific scheduli...
[18:38:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[18:46:08] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11354723 (10Jclark-ctr) @cmooney   Few things we ran into   an-worker1136 Failed to ping after migration.  changed cable port old and new showed link moved ba...
[21:38:11] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11355132 (10RobH)
[22:02:53] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar): Telegram previews broken since unified mobile routing - https://phabricator.wikimedia.org/T409575#11355182 (10Krinkle)
[22:17:11] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar): Telegram previews broken since unified mobile routing - https://phabricator.wikimedia.org/T409575#11355215 (10Krinkle) Looking at the hostname breakdown:  {F70007795 height=300}  I notice Telegram exclusive makes connections to our m-dot hostnames. This is true his...
[23:25:14] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team, 07Upstream: Telegram previews broken since unified mobile routing - https://phabricator.wikimedia.org/T409575#11355374 (10Krinkle) p:05Triage→03Low >>! In T409575#11355215, @Krinkle wrote: > […] their crawler both forcibly rewrites your link //and// neglects to follo...