[02:28:40] FIRING: [6x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:29:09] FIRING: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [02:33:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:34:09] RESOLVED: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [02:38:40] FIRING: [11x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:43:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:48:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:28:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:33:40] FIRING: [14x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:48:40] FIRING: [7x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:53:40] RESOLVED: [7x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:32:23] 10netops, 06Infrastructure-Foundations: Set MTU on mr1 interfaces - https://phabricator.wikimedia.org/T359320#10224664 (10ayounsi) a:05ayounsi→03Papaul [12:38:19] 06Traffic: liberica puppetization for existing lvs instances - https://phabricator.wikimedia.org/T377127 (10Vgutierrez) 03NEW [12:38:40] 06Traffic: liberica puppetization for existing lvs instances - https://phabricator.wikimedia.org/T377127#10225641 (10Vgutierrez) p:05Triage→03Medium [12:41:25] 06Traffic: liberica puppetization - https://phabricator.wikimedia.org/T377127#10225657 (10Vgutierrez) [12:41:36] forget the existing lvs instances part..... our current lvs runs on bullseye [12:53:09] 10netops, 10Ceph, 06Infrastructure-Foundations: cephosd advertised v6 prefix flapping - https://phabricator.wikimedia.org/T376697#10225721 (10cmooney) I see the healthcheck fails quite regularly `lines=10 cmooney@cephosd1002:~$ grep DOWN /var/log/anycast-healthchecker/anycast-healthchecker.log 2024-10-13 20:... [12:56:28] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10225742 (10aborrero) >>! In T375847#10217807, @cmooney wrote: >>>! In T375847#10195673, @aborrero wrote: >> `lang=shell-session >> roo... [13:47:20] 06Traffic, 10conftool, 13Patch-For-Review: Integrate requestctl haproxy rules into our TLS terminator - https://phabricator.wikimedia.org/T370745#10225982 (10Joe) 05Open→03Resolved [13:49:12] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10225985 (10cmooney) We definitely want to use DHCPv6 (stateful) for address assignment. So OpenStack is in control of what IPs are us... [14:01:38] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10226051 (10cmooney) So.... maybe this is normal for DHCPv6? Re-reading the reddit post and looking at the setup on the VM it seems li... [14:44:45] 10netops, 06Infrastructure-Foundations: Transient DOWN alert on cr2-magru - https://phabricator.wikimedia.org/T374401#10226253 (10ayounsi) 05Open→03Resolved a:03ayounsi Closing this. Please re-open if it happens again. [14:51:59] 10netops, 06Infrastructure-Foundations, 06SRE-OnFire, 10Sustainability (Incident Followup): Juniper: regularly run `request system configuration rescue save` - https://phabricator.wikimedia.org/T376005#10226302 (10joanna_borun) p:05Triage→03Low a:03ayounsi [14:57:00] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06serviceops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10226327 (10cmooney) >>! In T375845#10182322, @JMeybohm wrote: >> It might be possible though to migrate to a new IPPool that... [15:01:11] 10netops, 10Ceph, 06Infrastructure-Foundations: cephosd advertised v6 prefix flapping - https://phabricator.wikimedia.org/T376697#10226356 (10joanna_borun) p:05Triage→03Medium [15:04:02] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06serviceops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10226362 (10JMeybohm) >>! In T375845#10226327, @cmooney wrote: > If we do this we probably need to allocate a new single pool,... [15:52:36] elukey: happy to see that you didn't need to increase docker-registrry timeouts in ATS :) [20:21:32] 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 13Patch-For-Review, 10Wiki-Setup (Delete / Redirect): redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#10227207 (10Pppery) 05Open→03Resolved a:03Pppery