[06:12:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [06:47:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [10:27:10] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad: row C/D switch refresh configuration task - https://phabricator.wikimedia.org/T402588#11240440 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=da4ec6fc-e51f-4967-b0f1-8ef51813239b) set by cmooney@cumin1003 for 0:10:00 on 5 host(s) and... [12:19:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [12:31:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11240827 (10Jclark-ctr) I have pulled 2 Servers from decom i have wiped all configs from them and racked and cabled sretest1005 Rack... [12:49:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [13:30:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11241087 (10cmooney) >>! In T405560#11240827, @Jclark-ctr wrote: > I have pulled 2 Servers from decom i have wiped all configs fro... [15:13:19] btw topranks -- this is coming in OpenSSH 10.x https://cvsweb.openbsd.org/src/usr.bin/ssh/readconf.c#rev1.284 [15:13:46] and in general more extensive support for setting ipqos stuff on both client and server side [15:17:20] hmm ok [15:17:41] kind of flipping the default by making interactive as “ef” [15:17:53] and resetting non-interactive to default [15:18:22] surprised they’d be bothered, most anyone who respects the markings will probably have a custom policy or remark [15:19:14] “EF” is actually a terrible idea as on many an enterprise network that will go into a priority queue with zero buffer and very small bw allocation - used for VoIP only [15:19:31] but most places aren’t gonna let a server set that anyway [15:19:38] thanks for the link!! [15:21:05] topranks: there is a long thread about it on the mailing list :D [15:21:23] that exact point was raised and might also have made it to the release notes [16:48:55] FIRING: MaxConntrack: Max conntrack at 81.85% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [16:53:55] RESOLVED: MaxConntrack: Max conntrack at 84.33% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [18:55:55] FIRING: MaxConntrack: Max conntrack at 83.41% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:00:55] RESOLVED: MaxConntrack: Max conntrack at 83.03% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:50:55] FIRING: MaxConntrack: Max conntrack at 80.23% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:55:55] RESOLVED: MaxConntrack: Max conntrack at 80.05% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack