[01:21:05] 10Traffic, 10Operations, 10Performance-Team (Radar): Enable mwdebug routes for noc.wikimedia.org - https://phabricator.wikimedia.org/T233768 (10Krinkle) [01:22:14] 10Traffic, 10Operations, 10Performance-Team (Radar): Enable mwdebug routes for noc.wikimedia.org - https://phabricator.wikimedia.org/T233768 (10Krinkle) [01:52:10] 10Traffic, 10Operations: rack/setup/install ganeti400[123] - https://phabricator.wikimedia.org/T226444 (10Dzahn) ^ Fixed by @papaul. Confirmed it works now to change password via IPMI from remote. [02:16:55] 10Traffic, 10Operations, 10ops-eqiad: cp1085 - IPMI not working - https://phabricator.wikimedia.org/T231525 (10Dzahn) [02:28:52] 10Traffic, 10Operations, 10ops-eqiad: cp1085 - IPMI not working - https://phabricator.wikimedia.org/T231525 (10Dzahn) @Papaul confirmed this looks like it needs onsite to drain the power. I asked @Vgutierrez about depooling this. Could i up the priority a bit due to the relation to T147074? [02:29:31] 10Traffic, 10Operations, 10ops-eqiad: cp1085 - IPMI not working - https://phabricator.wikimedia.org/T231525 (10Dzahn) p:05Normal→03High [04:02:36] 10HTTPS, 10Traffic, 10Operations, 10Upstream: Enable ESNI support on Wikimedia servers - https://phabricator.wikimedia.org/T205378 (10Shizhao) [06:25:49] <_joe_> heads up, I'm merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/538864/ [06:26:06] <_joe_> and then restarting the low-traffic LVS servers [08:44:49] 10Traffic, 10Anti-Harassment, 10CheckUser, 10MediaWiki-User-management, 10Operations: Users editing from 127.0.0.1 (due to experimenting with ATS terminating TLS) - https://phabricator.wikimedia.org/T233657 (10Vgutierrez) [08:44:52] 10Traffic, 10Operations, 10Patch-For-Review: varnish-fe is handling X-Forwarded-For differently when ats is in front of it - https://phabricator.wikimedia.org/T233667 (10Vgutierrez) 05Open→03Resolved [08:44:55] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [08:45:03] _joe_: ack :) [09:26:33] 10Traffic, 10Operations: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 (10Vgutierrez) [10:45:16] 10Traffic, 10Operations, 10Patch-For-Review: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 (10Vgutierrez) [10:48:19] !log Switch from nginx to ats-tls on cp4022 - T231433 [10:48:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:48:24] T231433: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 [10:48:34] wrong window... but same result O:) [11:09:27] 10Traffic, 10Operations, 10Patch-For-Review: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 (10Vgutierrez) [11:34:32] 10Traffic, 10Operations, 10Patch-For-Review: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 (10Vgutierrez) [11:54:17] 10Traffic, 10Operations, 10Patch-For-Review: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 (10Vgutierrez) [12:25:13] 10Traffic, 10Operations, 10Patch-For-Review: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 (10Vgutierrez) [15:40:44] 10Traffic, 10Operations, 10Performance-Team (Radar): Enable mwdebug routes for noc.wikimedia.org - https://phabricator.wikimedia.org/T233768 (10herron) p:05Triage→03Normal [15:47:02] 10Traffic, 10Analytics, 10Operations, 10observability: Publish tls related info to webrequest via varnish - https://phabricator.wikimedia.org/T233661 (10herron) p:05Triage→03Normal [15:57:00] XioNoX: are there any special steps necessary for rebooting the new netflow* instances? Could you add a note to https://wikitech.wikimedia.org/wiki/Service_restarts please? [15:58:22] moritzm: nothing special to do [15:59:48] ack, thx [16:32:41] 10Traffic, 10Operations, 10Wikidata, 10serviceops, and 3 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10Dzahn) 05Stalled→03Declined @JanZerebecki as the original reporter. Please see T99531#5406014 and all the other comments above. This has b... [16:38:36] i closed the wikiba.se to wmf ticket as declined. i didn't see any more cleanups left to be done in puppet or dns and wmde guys deleted the cloud vps part [16:38:59] so i just took the liberty to tell the original author it's declined after discussion between management , as it is [16:39:18] (and he knows anyways) [16:39:42] if there is anything else i dont see please reopen [16:40:52] sad how we had to revert stuff but it is what it is and at least be done with it instead of stalled forever [18:47:16] 10Traffic, 10Operations: GRE MTU mitigations - Tracking - https://phabricator.wikimedia.org/T232602 (10ayounsi) @BBlack @faidon let me know when is a good time to remove that MSS hack on the routers. To be done one router at a time with time in between for the sessions to re-establish. Will also drain NTT/Teli... [19:02:41] 10Traffic, 10Operations, 10Puppet: Puppet systemd::mask is an anti pattern that has unwanted side effect - https://phabricator.wikimedia.org/T233839 (10hashar) Adding #traffic team since `systemd::mask` has been introduced for trafficserver / tlsproxy.