[09:54:26] today's fun discovery: T442818 [09:54:27] T442818: Reclaim floating IPs assigned to deleted projects - https://phabricator.wikimedia.org/T442818 [09:57:39] doh [09:57:47] nice catch! [13:29:11] I imagine that after opening egress some projects in the unsafe list are now safe thanks to unattended upgrades. Has someone already re-audited all of those to see? [13:32:52] most likely not many because for the easy ones we did upgrade the packages ourselves, but we can surely check [13:33:24] makes sense, ok. [13:33:32] it's a short list, I'll re-run now since it's easy [13:35:17] if you run it with -q instead of -s [13:35:22] it's best on larger set of hosts [13:35:29] just prints the php=$value bit [13:35:38] andrewbogott: i have [13:36:03] ok :) [13:36:19] everything still locked is in the spreadsheet, due to having containers etc that unattended-upgrades can't reach [13:36:33] (or has a magnum cluster we can't see well into.) [13:36:47] I'm impatient to restore the sg APIs but I fear there's going to be a long, long tail for this last handful... [13:38:30] which is why i added that bullet point to the meeting in a few moments :P [13:44:36] any objections to me deploying T439554? It was scheduled to be deployed on the very day of the great poweroff :) [13:44:36] T439554: Read the opted-in tools from components-api - https://phabricator.wikimedia.org/T439554 [13:50:03] dhinus: i'd maybe wait for aputhin's outage comms to go out first before announcing new features, but otherwise no issues from my side [13:50:17] We won't announce it yet, just turn it on [13:50:26] It's opt-in anyhow [13:50:38] announcement is tentatively planned for next week [13:51:00] why the delay? [13:51:04] I thought was for today [13:51:18] ? you mean announcing the new features? [13:51:54] well https://phabricator.wikimedia.org/T439554#12407760 says we will announce it the moment it goes live, is that not true? [13:52:13] ahh sorry mixing announcements up [13:52:30] I thought you said the upgrade/security announcment was delayed, my bad :D [13:53:04] not anymore, we can update that [13:53:10] that was the plan before the world caught fire :-P [13:53:13] fair :P [13:53:17] dhinus: then go ahead ihmo [13:53:31] ok thanks, I'll post a comment in phab as well [13:56:37] volans: that one just got sent, short delay due to interruptions hehe [13:57:43] I was gonna say something about it being likely our last update on the PHP incident thread but I won't because jinxing is a real threat these days [14:43:48] task for the proxy-level blocking thing is T442870 [14:43:50] T442870: Add admin-only disable function to individual Cloud VPS web proxies - https://phabricator.wikimedia.org/T442870 [15:00:37] updated: https://wikitech.wikimedia.org/wiki/Incidents/2026-10-21_PHP_security_update/Community#FAQs [15:36:07] andrewbogott: do you happen to have a non-global-admin codfw1dev test account handy? (and have a moment to test something for me) [15:39:57] yes, what's the test? [15:40:23] in the 'testlabs' project, can you do anything about the 'wmcs-proxy-test.taavivaananen.fi' web proxy? [15:40:57] this is how I picture andrew's collection of openstack credentials: https://www.gettyimages.in/detail/photo/man-with-bunch-of-keys-on-keychain-mid-section-royalty-free-image/200500649-001 [15:41:13] taavi: the UI thinks I can. Let me make sure I'm not accidentally admin there... [15:41:22] dhinus: it's really only like 4 or 5 I think [15:41:26] :D [15:41:49] andrewbogott: yeah, the UI is going to think that, I mostly care if the backend will actually let you do it [15:42:04] ah, I see, then I'll try to edit it [15:43:04] taavi: 403 [15:43:13] cool [15:43:18] and same thing with deletion i assume? [15:43:42] Error: Unable to delete proxy: wmcs-proxy-test.taavivaananen.fi [15:43:49] We need to block creation of new proxies too don't we? [15:44:09] you should not be able to create a proxy with the same name? [15:45:32] But I can create a new proxy pointing to the same cursed backend and bypass the block right? [15:45:38] I mean, not that any users are likely to know about the new one [15:48:22] yeah, although that's ihmo less critical [15:49:48] If we assume that the thread is mostly from robots, a robot could find the new proxy pretty quickly. But your change is definitely the main thing! [15:50:21] *threat [15:56:23] ok, patches in https://phabricator.wikimedia.org/T442870 are up for review [16:17:02] taavi: I've forgotten how all this works apparently... where do the map files in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1360462/2/modules/profile/files/wmcs/novaproxy/novaproxy-update-map.sh come from? Are they dumped out of the db periodically, or is that just a once-in-a-blue-moon bulk import system? [16:18:34] andrewbogott: the script you are looking at creates those files, it runs on a systemd timer once a minute or so. and puppet creates those files as empty if they don't exist to make sure that haproxy doesn't crash. [16:19:07] oh oh this is the part that feeds into haproxy [16:19:31] I'm caught up now, got lost in the change from redis [16:19:48] yeah no redis involved anymore [16:21:06] in modules/openstack/files/util/wmcs-webproxy.py -- if there was no update_proxy before did the UI 'edit' them by just deleting and making a new one? [16:21:07] if you want to test the CLI, there's a copy in my home directory in cloudcontrol2010-dev.codfw.wmnet [16:21:27] wmcs-webproxy.py is the admin script that exists on cloudcontrols that uses the API, it has no relation to the horizon panel [16:22:05] So there just wasn't a way to edit from the cli before [16:23:01] yeah [16:23:07] I think this has a lot more pieces since I last worked on it. But clearly better. [16:23:34] tbh i think the main change was the move from redis+nginx to haproxy with those static map files [16:24:34] so it's (cli or horizon) <-> api -> map files -> haproxy [16:24:38] yes [16:24:53] well api <-> database <-> map files [16:25:11] that database moved from a local sqlite file to being in the cloudinfra mariadb cluster some tiem ago [16:26:25] Because the map files are dumped out of the db on a cron rather than written synchronously by the api... [16:27:19] which I guess is to support haproxy running on a separate system from the API, or having multiple haproxy nodes? [16:27:31] yes [16:27:53] anyway the changes look fine :) I'm just trying to update my brain [16:28:24] yeah I'm just waiting for CI to pass after fixing that typo, then will merge and deploy [16:45:16] all deployed, and updated the docs even [17:19:33] andrewbogott about sending a follow up about node upgrades, please feel free to. I have to logoff now but I think it doesn't hurt. [17:20:56] ok! [17:36:55] the "maintainer alerts" feature (T432860) is now available for all users. the email to cloud-announce will go out next week, but you can start enabling it in your tools :) docs are at https://wikitech.wikimedia.org/wiki/Help:Toolforge [17:36:56] T432860: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860 [17:41:24] sorry wrong link, docs are at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Alerts