[02:02:25] FIRING: SystemdUnitCrashLoop: varnish-frontend-slowlog.service crashloop on cp4044:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [02:07:25] RESOLVED: SystemdUnitCrashLoop: varnish-frontend-slowlog.service crashloop on cp4044:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [02:11:07] :( [03:44:25] FIRING: SystemdUnitCrashLoop: varnish-frontend-slowlog.service crashloop on cp4044:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [03:49:25] RESOLVED: SystemdUnitCrashLoop: varnish-frontend-slowlog.service crashloop on cp4044:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [05:41:35] 06Traffic: upgrade to trafficserver 9.2.9 - https://phabricator.wikimedia.org/T388035#10622095 (10Vgutierrez) [07:05:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [07:10:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [10:28:00] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#10622824 (10Ladsgroup) Progress update: Now 1% of production thumbnails (and 100% of test wikis) are using thumbnail steps which means they are going to b... [10:28:19] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#10622826 (10Ladsgroup) a:03Ladsgroup I guess I'm doing this. [11:09:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10622967 (10cmooney) @VRiley-WMF I've fixed up the scs links in netbox now, moving the cables to the new devices. To set the boxes to 'active' (needed b... [11:21:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10623041 (10cmooney) [11:30:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10623075 (10cmooney) @robh did Myriad send us the license files for these two new QFX5120-48Y switches? [13:21:00] 06Traffic, 06Infrastructure-Foundations: requestctl bandwidth limit has incorrect syntax - https://phabricator.wikimedia.org/T388529 (10Joe) 03NEW [16:05:31] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on  - https://phabricator.wikimedia.org/T388561 (10TRIBU_sig) 03NEW Closing this task as invalid due to missing information. [16:08:14] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on - https://phabricator.wikimedia.org/T388562 (10TRIBU_sig) 03NEW [16:17:37] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on  - https://phabricator.wikimedia.org/T388562#10624721 (10Aklapper) 05Open→03Declined Declining as there is no Wikimedia Affiliate supporting. Please see https://wikitech.wikimedia.org/wiki/Maps/External_usage : //maps.wikimedia.org tile... [16:24:42] 06Traffic, 06Infrastructure-Foundations: requestctl bandwidth limit has incorrect syntax - https://phabricator.wikimedia.org/T388529#10624763 (10Vgutierrez) I don't think applying limits per stream is enough as a single HTTP/2 connection can contain multiple streams so probably we need to group at the very lea... [16:53:32] 06Traffic, 10Citoid, 06Editing QA, 06Editing-team, and 3 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10624921 (10EAkinloose) [17:23:08] 06Traffic: upgrade to trafficserver 9.2.9 - https://phabricator.wikimedia.org/T388035#10625125 (10BCornwall) [18:59:07] 06Traffic: varnish-frontend-slowlog.service fails with Varnish 7.1 - https://phabricator.wikimedia.org/T388597 (10BCornwall) 03NEW [18:59:37] 06Traffic: varnish-frontend-slowlog.service fails with Varnish 7.1 - https://phabricator.wikimedia.org/T388597#10625883 (10BCornwall) 05Open→03In progress p:05Triage→03High [20:01:25] FIRING: SystemdUnitCrashLoop: varnish-frontend-slowlog.service crashloop on cp4044:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [20:06:25] RESOLVED: SystemdUnitCrashLoop: varnish-frontend-slowlog.service crashloop on cp4044:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [21:40:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10626536 (10Papaul) lsw2-e8-codfw transceiver and cable type ` --{ + state }--[ interface ethernet-1/10 ]-- transceiver {... [22:39:38] 06Traffic, 06MW-Interfaces-Team: Log Api-User-Agent header in Turnilo - https://phabricator.wikimedia.org/T373871#10626734 (10nshahquinn-wmf) (This is a request to add new fields to a dataset rather than a report of a data bug.)