[01:14:17] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10Ottomata) IIRC, the decision was to wait until the new year, so as not to risk a mistake while people were out on holidays. I'm about... [07:19:03] 10HTTPS, 10SRE, 10Traffic-Icebox, 10Upstream: Support ECH on Wikimedia servers - https://phabricator.wikimedia.org/T205378 (10Diskdance) [07:44:08] 10HTTPS, 10SRE, 10Traffic-Icebox, 10Wikimedia-Performance-recommendation: Enable HTTP/3 (QUIC) support on Wikimedia servers - https://phabricator.wikimedia.org/T238034 (10Diskdance) [08:39:12] 10netops, 10Infrastructure-Foundations, 10SRE: cr1-esams:fpc0 errors - https://phabricator.wikimedia.org/T346779 (10ayounsi) 05Open→03Resolved All good. [08:41:03] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade asw1-eqsin - https://phabricator.wikimedia.org/T332395 (10ayounsi) a:03ayounsi [10:02:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade asw1-eqsin - https://phabricator.wikimedia.org/T332395 (10ayounsi) Latest Junos recommended has been copied to /var/tmp/ Next steps: downtime the site and proceed with the upgrade : https://wikitech.wikimedia.org/wiki/Juniper_switch... [11:29:37] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) Next steps to create a production grade routed cluster: # {T353935} # Assign a private and optionally public IPv4 and v6 range for codfw # Add a Hiera key `pro... [11:34:17] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10MoritzMuehlenhoff) Let's start the routed ganeti setup directly on Bookworm (IOW reimage ganeti2033/2024 after the move); the regular Ganeti clusters are still on Bullse... [12:47:01] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) Will do, thanks. For prefix allocation I'm suggesting the following, let me know what you think (especially @cmooney ! ) * eqiad * 10.64.24.0/23 - private1-v... [14:49:18] 10Traffic, 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10CodeReviewBot) vgutierrez merged https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/11 clamper: Provide... [15:28:30] 10Traffic, 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10Vgutierrez) https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/11 performs integration tests using qemu... [15:46:03] 10Traffic: HAProxy 2.6.16 CPU spikes on cp3066 - https://phabricator.wikimedia.org/T354424 (10Vgutierrez) [15:46:30] 10Traffic: HAProxy 2.6.16 CPU spikes on cp3066 - https://phabricator.wikimedia.org/T354424 (10Vgutierrez) p:05Triage→03Medium [15:46:41] 10Traffic, 10Upstream: HAProxy 2.6.16 CPU spikes on cp3066 - https://phabricator.wikimedia.org/T354424 (10Vgutierrez) [16:54:04] 10Traffic, 10Release-Engineering-Team, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) [17:19:03] 10Traffic, 10Release-Engineering-Team, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10dancy) Try adding ` tags: [memory-optimized] ` to the failing job. [22:05:24] 10Traffic, 10Release-Engineering-Team, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) @dancy You're a wizard. Thanks so much for this, it's now happily building. :) I added some docs to https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Ru... [22:05:35] 10Traffic, 10Release-Engineering-Team, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) 05Open→03Resolved a:03BCornwall [22:19:24] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10CodeReviewBot) brett closed https://gitlab.wikimedia.org/repos/sre/pybal/-/merge_requests/3 Release 1.15.14 [22:20:17] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10CodeReviewBot) brett reopened https://gitlab.wikimedia.org/repos/sre/pybal/-/merge_requests/3 Release 1.15.14 [22:20:45] 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10CodeReviewBot) brett closed https://gitlab.wikimedia.org/repos/sre/pybal/-/merge_requests/3 Release 1.15.14 [23:15:56] 10Traffic, 10Release-Engineering-Team, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) Hm, I'm still running into this every now and then. @dancy, Is there a possibility of raising the amount of memory even more?