[07:59:13] 10netops, 06Infrastructure-Foundations, 07Documentation: https://wikitech.wikimedia.org/wiki/Out-of-band_network out of date - https://phabricator.wikimedia.org/T379465#10311213 (10ayounsi) 05Open→03Resolved a:03ayounsi Updated :) [08:05:25] FIRING: SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:09:04] ^^ me [08:10:25] FIRING: [2x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:15:25] FIRING: [7x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:20:25] FIRING: [13x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:59:22] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikipedia, zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10311503 (10Bawolff) > I'm pretty sure you're not allowed to link to o... [10:12:00] 10netops, 06Infrastructure-Foundations, 06SRE: Extend sre.network.configure-switch-interfaces cookbook to add sflow and qos config - https://phabricator.wikimedia.org/T379549#10311548 (10cmooney) I discussed this briefly with @ayounsi on irc and while this is probably a good idea it won't, as things stand, p... [10:15:25] FIRING: [13x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:20:25] FIRING: [13x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:09:41] 06Traffic: Investigate transport errors in eqsin - https://phabricator.wikimedia.org/T379611 (10Fabfur) 03NEW [11:15:25] FIRING: [13x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:20:25] FIRING: [13x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:37:54] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10311981 (10cmooney) @Jgreen @Dwisehaupt I think we have broadly two options for how to proceed today: **O... [14:10:17] 06Traffic, 10ops-magru, 06SRE: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged) - https://phabricator.wikimedia.org/T376737#10312400 (10RobH) **Vital Date Update** I failed to get this filed before I went away for a week, and now its too short notice to get it filed today. I'... [14:31:56] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10312547 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=fd1b13c3-25ae-42de-a138-bb1a39... [14:45:25] FIRING: [13x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:50:25] FIRING: [13x] SystemdUnitFailed: haproxykafka.service on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:04:33] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10312686 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6d3e8237-b81b-47ec-a63c-afd9f7... [16:23:58] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10313129 (10cmooney) Migration work is now complete, bastion and all hosts are reachable again following th... [23:18:19] 06Traffic: Upgrade Varnish from 6.0.11 to 6.0.13 - https://phabricator.wikimedia.org/T379699 (10BCornwall) 03NEW [23:19:13] 06Traffic: Upgrade Varnish from 6.0.11 to 6.0.13 - https://phabricator.wikimedia.org/T379699#10314994 (10BCornwall) 05Open→03In progress p:05Triage→03High [23:23:33] 06Traffic: Upgrade Varnish from 6.0 to 7.1 - https://phabricator.wikimedia.org/T378737#10315000 (10BCornwall) [23:23:34] 06Traffic: Upgrade Varnish from 6.0.11 to 6.0.13 - https://phabricator.wikimedia.org/T379699#10314999 (10BCornwall)