[02:18:41] 06Traffic, 10Page Content Service, 06Wikipedia-Android-App-Backlog: [[2025 Coeur d'Alene shooting]] showing old version in Android app - https://phabricator.wikimedia.org/T398243#10961735 (10jeremyb-phone) this particular article is no longer so outdated. but it was like this for hours. I'll keep an eye out... [06:18:20] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10961884 (10ayounsi) @Jgreen please sync up with us on IRC for the Netops part. [09:49:50] 06Traffic: Consider using a dedicated TLS certificate for upload.w.o - https://phabricator.wikimedia.org/T394484#10962324 (10Vgutierrez) After the switch on ulsfo we are currently saving one RTT per handshake with and without SNI present on the ClientHello, client side used to generate the requests is openssl s_... [10:20:33] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10962442 (10Vgutierrez) chassis power status seems to be deprecated, `get System.ServerPwr` doesn't provide specific information about the status but `serveraction power... [10:48:18] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10962540 (10Vgutierrez) currently spicerack IPMI module uses `chassis power status` as part of its [[ https://doc.wikimedia.org/spicerack/master/_modules/spicerack/ipmi.... [11:03:44] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10962585 (10Vgutierrez) @Jhancock.wm could you try re-running the cookbook again and see if we get more progress now? thanks [13:08:38] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963400 (10ayounsi) [13:13:33] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963414 (10Jgreen) [13:26:19] 06Traffic, 06Movement-Insights, 10Data-Engineering (Q4 2025 April 1st - June 30th): NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#10963470 (10mforns) [13:52:25] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963580 (10Jgreen) 05Open→03Resolved a:03Jgreen [13:53:03] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963585 (10Jgreen) p:05Triage→03Medium [13:54:51] Is there a way to go from a varnish xid to the actual request? I assume we don't log everything? [13:56:31] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10963607 (10Jgreen) [13:56:54] claime: we don't AFAIK [13:57:01] claime: it's a 5xx response? [13:57:06] 429 [13:57:34] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Test prototype fundraising pybal replacement based on haproxy + anycast-healthchecker. - https://phabricator.wikimedia.org/T373942#10963611 (10Jgreen) [15:24:19] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#10964266 (10BCornwall) [15:35:26] 06Traffic, 10Liberica, 13Patch-For-Review: Switch to katran as forwarding plane on non-core DCs - https://phabricator.wikimedia.org/T396561#10964327 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=177a0107-d947-4693-94c9-ef613b43531f) set by vgutierrez@cumin1002 for 1 day, 0:00:00 on 1 ho... [15:44:53] 06Traffic, 10Liberica, 13Patch-For-Review: Switch to katran as forwarding plane on non-core DCs - https://phabricator.wikimedia.org/T396561#10964378 (10Vgutierrez) [16:05:32] 06Traffic, 06Content-Transform-Team, 10Page Content Service, 06Wikipedia-Android-App-Backlog: [[2025 Coeur d'Alene shooting]] showing old version in Android app - https://phabricator.wikimedia.org/T398243#10964438 (10Seddon) [16:32:22] 06Traffic, 06Content-Transform-Team, 10Page Content Service, 06Wikipedia-Android-App-Backlog: [[2025 Coeur d'Alene shooting]] showing old version in Android app - https://phabricator.wikimedia.org/T398243#10964578 (10Vgutierrez) purge infrastructure on the CDN has been working as expected, no hiccups there... [16:36:32] 06Traffic, 06Content-Transform-Team, 10Page Content Service, 06Wikipedia-Android-App-Backlog: [[2025 Coeur d'Alene shooting]] showing old version in Android app - https://phabricator.wikimedia.org/T398243#10964593 (10A_smart_kitten) There are a few bug reports (of varying age/characteristics) for the gener... [22:56:34] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10965689 (10Jhancock.wm) got a pass. but... ` Testing Redfish API connection to cp2043 (10.193.1.85) [IDRAC.2.11.SYS057] Exporting Server Configuration Profile. [1/30,... [22:58:08] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10965692 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host cp2043.codfw.wmnet with OS bullseye [22:58:21] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10965693 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host cp2043.codfw.wmnet with OS bullseye executed with errors: - c... [23:04:33] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10965714 (10Jhancock.wm) that didn't take long. same issue i saw on sretest2006 ` Exception raised while executing cookbook sre.hosts.reimage: Traceback (most recent c...