[05:33:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: BAD PEM3 on cr2-codfw - https://phabricator.wikimedia.org/T394868#10855645 (10Papaul) ` UPDATE HAS BEEN ADDED: Dear Juniper Networks Customer, Your replacement part associated with RMA R200568010 Item # 100 has been successfu... [08:31:10] 06Traffic, 10Liberica: Test katran forwarding plane on lvs1013 - https://phabricator.wikimedia.org/T395228 (10Vgutierrez) 03NEW [08:31:15] 06Traffic, 10Liberica: Test katran forwarding plane on lvs1013 - https://phabricator.wikimedia.org/T395228#10855814 (10Vgutierrez) p:05Triage→03Medium [08:36:41] XioNoX, topranks do we currently perform ICMP offloading in eqiad? more specifically for the ncredir-lb VIPs [08:37:03] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#10855831 (10ayounsi) mr update : > Thanks for your patience. I’ve observed similar errors in the lab setup, even though the latest documentation confirms that the featu... [08:37:23] vgutierrez: where we do icmp offloading, it's only on text-lb [08:37:30] XioNoX: awesome, thanks [08:37:46] I'm working on testing katran on lvs1013 that's currently handling ncredir@eqiad [08:37:53] and I need ICMP hitting it [08:39:04] vgutierrez: looking forward to decom the ping offload hack we have on the routers and the dedicated VMs :) [08:39:43] oh yeah that's planned with the move to Liberica isn't it? [08:39:49] nice [08:40:00] yep [08:40:11] https://phabricator.wikimedia.org/T367973 [08:40:14] XDP will take care of ICMP [08:40:58] 06Traffic: Replace ping offload servers with eBPF - https://phabricator.wikimedia.org/T367973#10855858 (10Vgutierrez) [08:40:59] 06Traffic, 10Liberica, 13Patch-For-Review: Replace current L4LB with with Katran-based alternative - https://phabricator.wikimedia.org/T332027#10855859 (10Vgutierrez) [08:41:32] 06Traffic, 10Liberica: Replace ping offload servers with eBPF - https://phabricator.wikimedia.org/T367973#10855865 (10Vgutierrez) p:05Triage→03Low [09:23:24] FIRING: [3x] SystemdUnitFailed: wmfuniq-experiment-fetcher.service on cp7006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:28:25] FIRING: [26x] SystemdUnitFailed: wmfuniq-experiment-fetcher.service on cp1103:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:31:51] 06Traffic: Review fetch_external_clouds_vendors_nets script to improve Azure fetcher - https://phabricator.wikimedia.org/T395236 (10Fabfur) 03NEW [09:33:24] FIRING: [44x] SystemdUnitFailed: wmfuniq-experiment-fetcher.service on cp1103:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:38:24] FIRING: [57x] SystemdUnitFailed: wmfuniq-experiment-fetcher.service on cp1102:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:41:14] ^^ this should be fixed as soon as puppet finishes running on cp servers [09:43:25] FIRING: [2x] SystemdUnitFailed: wmfuniq-experiment-fetcher.service on cp2041:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:48:25] RESOLVED: [2x] SystemdUnitFailed: wmfuniq-experiment-fetcher.service on cp2041:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:52:48] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: Fetch experiments configuration programmatically from CDN servers - https://phabricator.wikimedia.org/T395001#10856155 (10Vgutierrez) 05Open→03Resolved Fetcher is working as expected, currently dropping the experiments configuration on /tmp: ` (1... [10:30:38] 06Traffic: Can't download Azure cloud prefixes anymore - https://phabricator.wikimedia.org/T395127#10856198 (10Fabfur) 05In progress→03Resolved Patch deployed and service manually run on puppetmaster1001, all fine with Azure prefixes download [12:22:24] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Stage and configure new Juniper switches in codfw rows E/F - https://phabricator.wikimedia.org/T394021#10856524 (10cmooney) @Jhancock.wm hey I'm having some problems reaching cagefive2001 over management. The IP it is assigned is not respo... [13:35:56] 06Traffic, 06SRE, 13Patch-For-Review: Lower geodns TTLs for dyna.wm.org and upload.wm.org from 300s (5 min) to 180s (3 min) - https://phabricator.wikimedia.org/T394312#10856773 (10ssingh) [15:49:37] 07HTTPS, 06SRE, 06Traffic-Icebox, 07Wikimedia-Performance-recommendation: Enable HTTP/3 (QUIC) support on Wikimedia servers - https://phabricator.wikimedia.org/T238034#10857202 (10ssingh)