[01:44:15] 06Traffic, 06SRE: cp_upload @ eqsin cascading failures, February 2021 - https://phabricator.wikimedia.org/T274888#11373140 (10Pppery) Anything left to do here? [01:44:20] 06Traffic, 06SRE: cache_upload cache policy + large_objects_cutoff concerns - https://phabricator.wikimedia.org/T275809#11373142 (10Pppery) [01:59:25] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-Needs-Improvement: test_matching_vlan() function crashing in Netbox network report - https://phabricator.wikimedia.org/T339133#11373180 (10Pppery) [07:43:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3070&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [07:48:43] FIRING: [16x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [07:58:43] RESOLVED: [16x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [11:27:58] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Map video and other large files to 'low-priority' network Qos queue - https://phabricator.wikimedia.org/T410133 (10cmooney) 03NEW p:05Triage→03Low [11:44:49] 06Traffic, 06Data-Engineering, 07Essential-Work, 10Experimentation Lab Radar: Cookie % has been rejected because it is foreign and does not have the "Partitioned" attribute - https://phabricator.wikimedia.org/T375256#11374053 (10Back_ache) {F70214945} added screenshot for clarity and to show this is still... [12:14:29] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: No free IPs on public1-ulsfo vlan (Nov 2025) - https://phabricator.wikimedia.org/T410047#11374122 (10cmooney) @ssingh I made a patch and can kick off the changes in Netbox and on the routers next week for this. However I wonde... [12:52:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [12:57:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [17:05:28] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 06Release-Engineering-Team (Radar): Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11374995 (10Dzahn) A proxy is running on 14 VMs, 2 in each of the 7 POPs. What is missing is the load-balancing part. [18:52:44] 10netops, 06Infrastructure-Foundations, 06SRE: Audit and verify all cloudcephosd have their primary interface tagged and access to cloud-storage vlan - https://phabricator.wikimedia.org/T409690#11375257 (10cmooney) btw I haven't forgotten about this I'll get to it next week [19:56:52] what's up with cp2043? it's been emailing root@ about an expired debmonitor certificate for a while [19:58:50] has been reimaged Oct 16 - is that when the emails started? https://phabricator.wikimedia.org/T392851#11281808 [20:01:13] in other tickets like https://phabricator.wikimedia.org/T345057 you fixed that by enabling puppet ? [20:02:04] maybe lets leave a comment on that ticket above [20:56:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [21:01:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS