[02:17:48] FIRING: [4x] PuppetFailure: Puppet has failed on lvs1019:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:02:57] 10netops, 06Infrastructure-Foundations: magru HE down (oct 2026) - https://phabricator.wikimedia.org/T441591 (10ayounsi) 03NEW p:05Triage→03High [06:03:27] 10netops, 06Infrastructure-Foundations: magru HE down (oct 2026) - https://phabricator.wikimedia.org/T441591#12446549 (10ayounsi) [06:17:48] FIRING: [4x] PuppetFailure: Puppet has failed on lvs1019:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:46:58] elukey: we got some issues with lvs hosts failing to reach pki.svc.eqiad.wmnet [07:48:19] vgutierrez: good morning to you :) pki1002 is depooled for mainteance (that afaics didn't happen) but pki1003 should be up taking traffic. What are you seeing? [07:48:35] puppet failing on lvs hosts due to being unable to reach pki infra [07:49:08] lvs1019/lvs1020, lvs2013 and lvs2014 [07:50:34] I wonder if it's just a matter of lvs hosts failing to use a VIP [07:51:10] bast1004 is able to reach pki.discovery.wmnet:443 as expected [07:53:07] ah they are failing to get their cert checked/renewed via puppet [07:53:43] do we know when did it start? [07:53:58] on this puppet run for lvs1019: https://puppetboard.wikimedia.org/report/lvs1019.eqiad.wmnet/bb10f9b30d9553e881605614bece3159ce7e1083 [07:54:13] Oct 18 2026 - 06:47:42 [07:55:16] ok so definitely not related to anything that we've done manually [07:55:47] I hope not :D [07:56:11] hosts reach pki.discovery.wmnet on every puppet run? [07:56:17] or just when they need to renew the cert? [07:57:46] it looks like the latter... we can't use the discovery record for lvs hosts [07:58:47] that wouldn't be an issue for liberica/katran [07:59:00] but its' for pybal/ipvs [08:00:09] ok so we have seen it only yesterday because the certs needed a renew, makes sense [08:00:31] do you know why the discovery record is not reachable from lvs? [08:00:54] sort-of self reference [08:01:00] ipvs failing to handle `lo` traffic [08:01:33] and it was working before since pki.discovery was basically a cname to pki1002/2002 [08:01:38] yep [08:01:48] what a lovely morning [08:02:01] can we override the pki host for those 4 hosts via hiera? [08:03:16] yeah definitely, but we'd need to always keep in mind that depooling a node means side effects for lvs [08:03:24] that is not the end of the world [08:03:56] the key is profile::pki::client::signer_host: pki.discovery.wmnet [08:04:52] since pki1002 is having some weird sporadic freeze issue, we could go with pki1003 and pki2003 (those are VMs on ganeti) [08:05:02] sounds good [08:05:08] let me get the CR ready for you [08:05:13] do you want me to do it or you'll do it? [08:05:15] ah ok :) [08:05:29] do you have the phab task of the migration to lvs handy? [08:05:53] https://phabricator.wikimedia.org/T436809 [08:10:19] PCC running at the moment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1356779 [08:12:48] elukey: ready for you :D [08:14:22] you are good to go [08:14:44] 06Traffic: Upgrade to HAProxy 3.2.26 - https://phabricator.wikimedia.org/T441435#12446872 (10dkertesz) 05Open→03In progress a:03dkertesz [08:15:47] elukey: merging, thx <3 [08:20:25] triggering a puppet run on the impacted hosts.. recoveries should follow soonTM [08:22:48] FIRING: [4x] PuppetFailure: Puppet has failed on lvs1019:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:23:05] 06Traffic: Upgrade to HAProxy 3.2.26 - https://phabricator.wikimedia.org/T441435#12446894 (10dkertesz) [08:25:26] lol.. alerts are lagging behind.. puppet is now happy on the 4 hosts [08:27:48] FIRING: [4x] PuppetFailure: Puppet has failed on lvs1019:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:28:12] super [08:42:48] FIRING: [3x] PuppetFailure: Puppet has failed on lvs1020:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:47:48] RESOLVED: [2x] PuppetFailure: Puppet has failed on lvs2013:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:35:04] 06Traffic, 06ServiceOps: Trial of weighted dnsdisc for mw-web-ro switchover (gradual ro traffic shift) - https://phabricator.wikimedia.org/T441436#12448123 (10Vgutierrez) Looks good, please consider that if you wanna go over 16 slices you need to adjust [[ https://docs.trafficserver.apache.org/en/9.2.x/admin-g... [12:17:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12448324 (10Jclark-ctr) Cabled the 4x breakout cables to patch panels [13:00:28] 06Traffic: revisit disabling virtualization on Dell hosts - https://phabricator.wikimedia.org/T441020#12448569 (10Papaul) @CDobbins cp2047 doesn't like version 1.30.60 for the IDRAC, it is breaking the IDRAC. I will look into it more this week. Thank you [13:15:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12448657 (10ayounsi) Opened {T441671} to audit them. The GS748Tv5 is end of life ( https://www.netgear.com/support/product/gs748tv5... [14:07:35] 06Traffic, 10Lift-Wing: Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12449143 (10ssingh) Hi @klausman / @DPogorzelski-WMF: We are planning to work on this for the reasons mentioned above and also to prevent the extra work in T440859. Are there any concerns from yo... [14:10:10] 06Traffic, 06Data-Platform-SRE, 10Lift-Wing, 06Machine-Learning-Team: Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12449172 (10klausman) [14:15:29] 06Traffic, 06Data-Platform-SRE, 10Lift-Wing, 06Machine-Learning-Team: Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12449204 (10klausman) Since I was confused for a moment, for clarification, this is in addition to/independent of the control plane side of thing... [14:40:12] 06Traffic, 06ServiceOps: Trial of weighted dnsdisc for mw-web-ro switchover (gradual ro traffic shift) - https://phabricator.wikimedia.org/T441436#12449348 (10CDanis) [17:33:43] FIRING: [2x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [17:35:40] FIRING: VarnishChildRestarted: varnish-text restarted on cp3071 - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000330/varnish-machine-stats?orgId=1&viewPanel=66&var-server=cp3071&datasource=esams%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DVarnishChildRestarted [17:36:09] depooling, cp3071 doesn't seem healthy [17:36:40] ok it's fine [17:38:47] FIRING: [11x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [17:42:40] 06Traffic, 06ServiceOps: Trial of weighted dnsdisc for mw-web-ro switchover (gradual ro traffic shift) - https://phabricator.wikimedia.org/T441436#12450732 (10MLechvien-WMF) p:05Triage→03High a:03RLazarus @RLazarus could you please work on this task with @CDanis once you're back? Depending on the progr... [17:48:43] FIRING: [11x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [17:53:43] RESOLVED: [11x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [19:08:49] 06Traffic, 06ServiceOps: Trial of weighted dnsdisc for mw-web-ro switchover (gradual ro traffic shift) - https://phabricator.wikimedia.org/T441436#12451001 (10CDanis) [20:59:03] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from CA) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [21:03:58] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from CA) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [21:35:40] FIRING: VarnishChildRestarted: varnish-text restarted on cp3071 - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000330/varnish-machine-stats?orgId=1&viewPanel=66&var-server=cp3071&datasource=esams%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DVarnishChildRestarted [23:07:01] 06Traffic, 07Voice & Tone: Status code 429 response does not print contact information wrapped in angle brackets - https://phabricator.wikimedia.org/T440648#12452086 (10KormiSK) I've been encountering this too before I figured out how to do it correctly. Interesting that it's only been opened recently. I also...