[00:01:55] RESOLVED: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:10:53] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Packet loss between magru and codfw on the EdgeUno link - https://phabricator.wikimedia.org/T445280#12574067 (10RLazarus) [08:55:31] 06Traffic, 13Patch-For-Review: Synchronize and rotate TCP Fastopen keys for various use-cases - https://phabricator.wikimedia.org/T355446#12574602 (10SLyngshede-WMF) 05In progress→03Resolved [10:48:16] 06Traffic, 10HaproxyKafka: haproxykafka fails to start after a system reboot - https://phabricator.wikimedia.org/T444888#12574999 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez haproxykafka now starts smoothly on system reboot: ` -- Boot 22b87d20b43d4e8bb7977c99ddc4a73b -- Nov 18 10:45:49 cp7016 sys... [12:39:54] 06Traffic, 06Data-Engineering: Let kapow_estimate be available in Turnilo and datalake - https://phabricator.wikimedia.org/T445104#12575524 (10JAllemandou) [12:40:43] 06Traffic, 06Data-Engineering: Let kapow_estimate be available in Turnilo and datalake - https://phabricator.wikimedia.org/T445104#12575544 (10JAllemandou) [12:40:45] 06Traffic, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: Add `x_kapow_estimate` to `webrequest` - https://phabricator.wikimedia.org/T445221#12575542 (10JAllemandou) →14Duplicate dup:03T445104 [13:27:48] 06Traffic, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Let kapow_estimate be available in Turnilo and datalake - https://phabricator.wikimedia.org/T445104#12575766 (10JAllemandou) [14:15:10] o/ I'd like to enable a miscweb service on ATS/varnish please, no unusual config https://gerrit.wikimedia.org/r/c/operations/puppet/+/1356300 [14:26:45] ty <3 [14:26:49] hnowlan: looks good... no cache-control header on / though [14:27:01] oh, will bring that up with the team [14:48:51] 06Traffic, 06Data-Platform-SRE, 06Wikidata Platform Team (Sprint 11 (2026/11/04)): ATS: route WDQSv2 public /sparql endpoints through REST Gateway - https://phabricator.wikimedia.org/T443756#12576385 (10BTracy-WMF) 05Open→03Resolved [14:55:04] hashar: it looks like aadc0d40f2 broke check_pybal_ipvs_diff [14:55:49] sukhe: ^^ [14:56:55] UNKNOWN: Status code 404 returned while getting http://localhost:9090/pools/b'mw-api-int_4446' for example [14:58:52] 06Traffic, 13Patch-For-Review: the configuration file for check_rp_filter_disabled can be too large for icinga - https://phabricator.wikimedia.org/T443771#12576496 (10dkertesz) 05In progress→03Resolved the check has been fully migrated to Prometheus [15:01:07] let me send a CR... [15:18:18] sorry, weird [15:18:24] I can fix it soon too [15:21:07] sukhe: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1371029 [15:25:15] * sukhe checks what I missed [15:25:42] hmm [15:26:10] the import thingie wasn't exploding in our face by pure luck.. probably cause prometheus also loads urllib under the hood [15:26:40] vgutierrez: I guess you have tested this unlike me? :P [15:30:18] vgutierrez@lvs1017:~$ ./check_pybal_ipvs_diff.py --pybal-url http://10.64.131.17:9090/pools --prometheus-url http://10.64.131.17:9100/metrics [15:30:18] OK: no difference between hosts in IPVS/PyBal [15:30:57] thanks for fixing my breakage [15:31:19] always happy to help a manager in distress [15:31:57] lol [16:09:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12576952 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1cabde64-fe83-4580-872f-9377642e823b) set by robh@cumin1004 for 0:15:00 on 1 hos... [16:24:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12577046 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a4baed93-0245-4e40-a8bb-91f54a1a00ed) set by robh@cumin1004 for 0:15:00 on 1 hos... [16:51:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12577187 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=12440d2d-85cd-43bf-9492-d1a5a19a0ece) set by robh@cumin1004 for 0:15:00 on 1 hos... [18:39:36] 06Traffic, 06DC-Ops, 10ops-esams: cp3074 hardware failure (possibly CMOS battery) - https://phabricator.wikimedia.org/T445380 (10ssingh) 03NEW [18:58:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12577843 (10RobH) [19:00:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12577855 (10RobH) Actual migrations of production hosts from the old to the new switch stacks began today with a dbtest host (in a1) and all the cirrussearch... [19:00:30] 06Traffic, 13Patch-For-Review: Migrate Sophroid to IPIP - https://phabricator.wikimedia.org/T441429#12577857 (10ssingh) 05Open→03Resolved This has been done in the CR above and the cookbook has been run. PyBal pools for sophroid_4252 seem happy so boldly marking this as resolved. [19:09:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12577911 (10RobH) 05Open→03Resolved all of this info is on the google sheet, so resolving this task [21:36:41] 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12578691 (10cmooney) [21:36:43] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12578692 (10cmooney) [21:39:00] 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12578713 (10cmooney) Juniper have after much slow back-and-forth reproduced and acknowledged this bug. They are tracking it as PR1959424 however the...