[09:23:15] maintain-dbusers seems unhappy because the paws nfs server proxy entry is broken (due to missing security group rules?) [09:45:00] are you looking at it? I was going to have a look (as people complained), I think toolforge might be having issues there too [09:54:58] looking, let me know if I'm stepping on your toes [09:59:55] dcaro: please do [10:02:25] this should fix the tools side while I fix the paws things https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359773 [10:02:50] damn auto-formatter :/ [10:10:12] doesn't that completely hide the failures if a backend fails? right now we get a prometheus alert for the systemd service failing which is useful [10:11:03] yep [10:11:08] to some extent [10:11:14] (still in the logs) [10:12:32] no-one's going to go looking at the logs without a reason :P [10:12:47] agree, we should move the alerts to maintaindbusers prometheus stats instead [10:12:52] or split it into 3 [10:14:08] we could even split the 'collect new accounts to the database' and 'provision accounts from the database' parts to separate processes/timers [10:14:26] anyway, did you check if the paws failure is just a missing security group rule or if it is something else? [10:14:31] I think I'll add a prometheus stat like '_run_success' [10:14:43] looking at that now [10:14:58] (will leave the "how to fail and notice" for after) [10:15:32] if it's just that we can add the security group rule, otherwise i'm fine with merging your patch for now [10:15:47] a middle ground: continue populating and set exit code non-zero [10:16:07] (i don't want to merge that and then forget doing anything else, since ihmo having it fail loudly is better in the general place) [10:16:16] godog: or that [10:17:10] yeah the alert can/will flap either way [10:17:21] the nfs does not include the port 443, looking [10:17:37] taavi: I'm not forgetting, I'm just fixing the account creation first [10:18:45] yeah i think https://horizon.wikimedia.org/project/security_groups/ce70e040-182c-49c8-a14a-f27b27843f84/ just needs rules for port 80 from cloud vps to make the proxy work again [10:22:44] dcaro: any reason not to run the restore security groups script for paws? I think it's still mostly in the egress-only lockdown state [10:22:50] I'll look at the backup failures on cloudbackup2003 [10:23:18] taavi: I did run it before, did not want to start debugging it, we can run it again sure [10:23:24] manually changed the rules now though [10:23:39] it started creating accounts [10:47:54] this adds the fix and the prometheus stat on top https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359775 this adds the new alert https://gerrit.wikimedia.org/r/c/operations/alerts/+/1359779 [10:48:57] looking now at the restore script [10:57:43] https://usercontent.irccloud-cdn.com/file/jNFJeZfL/image.png [11:00:35] * dcaro trying to setup the local env to run the alert tests [11:07:28] all tests cleared, now just waiting for reviews [11:14:27] I suspect that the rule restoration in paws might mess up with the magnum rules [11:16:00] it wants to delete the ones that were created yesterday ype [11:16:16] yes that's possible, the restore script assumes we want to turn back to the state we had before the incident [11:16:35] and that rules created after the incident were "temporary" rules that we created while debugging, so they can be removed [11:17:10] so it should recreate magnum rules from before the incident, but if new and different ones were created after, it will not work [11:17:38] magnum does create and delete a bunch yep [11:17:42] if the new ones match the old ones, it should not delete them though [11:17:59] it creates new security groups completely [11:18:04] ah :/ [11:19:07] hmm, it might be helpful even if we can just specify which secgroup to restore [11:19:18] that should be easy right? [11:19:57] yes I think so, let me have a look [11:20:14] though I think that we don't need any more for paws [11:21:30] if it's just paws, it might be easier to fix it manually yeah [11:21:43] yep, just paws [11:31:17] dhinus: you are working on the paws markdown thing right? [11:31:53] yes, still fighting with helm in the local dev env :/ [11:32:06] can we use codfw to test instead? [11:32:12] is paws up in codfw? [11:32:41] it's not working no :/, there was some dns issue in one of the workers, and I was unable to ssh to it [11:32:47] (the ssh key did not work) [11:33:11] we can try again if you want, maybe 4 eyes are able to fix it [11:33:16] (though I'm about to go for lunch) [11:33:25] you can try deploying the latest code (there were a couple changes) [11:33:42] maybe it just starts working xd [11:34:42] otherwise I'll start working on getting toolsbeta up, so we can do the regular deploy flow to keep deploying stuff to toolforge [11:35:42] no if codfw is not working I think it's easier if I continue on the local env, I would like to get it working anyway [11:35:51] if you can bring up toolsbeta I think that will be useful! [11:40:31] I was not able to make it work locally last time I tried :/ [11:40:48] but we should fix it yep [11:41:53] aputhin: where was the task "template" we wanted to use? I just created a few and forgot to use it :/ [12:44:23] Just add "Impact"and "Definition of Done" to it, that's all [13:33:34] we could move those paws security group rules to be managed by tofu in the paws repo, even [13:39:26] +1 for that yep, well, not the magnum, the nfs/bastion [13:39:45] aputhin: ack [13:40:13] also paws just showed me a popup that there is a new JupyterLab version available, we should probably disable those as individual users can't do anything to upgrade (and we don't want software phoning home ihmo) [13:41:55] I saw! but I thought I had the latest :/ [13:42:27] * taavi files a ticket [13:42:28] https://hub.jupyter.org/helm-chart/ is 4.4.4 [13:42:56] anyhow, thanks yep [13:44:25] T442717 [13:44:25] T442717: PAWS should not show update notifications for users - https://phabricator.wikimedia.org/T442717 [13:59:08] I'm going to start brining toolsbeta up [14:03:51] dcaro: ah I was just about to do that myself :D so thank you [14:10:21] bah, https://phabricator.wikimedia.org/P97014. any opinions on whether we want to adjust the tests or add back the ability to run 'webservice status' without a type somehow? [14:11:13] hmm, the webservice satus seems like a very convenient shortcut [14:11:24] can it read the service.manifest thingie? [14:11:51] (and just complain if it's not there) [14:13:53] https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/129 i think is the simplest solution [14:17:06] I wonder if that would mislead people thinking that they have no webservice but then when trying to start it will fail as it already exists or something [14:17:39] oh of course, it already reads the manifest, so if it gets there means there's no manifest right? [14:18:04] the only case in which the latter `if args.wstype is None:` block is hit (which the patch is modifying) is when there is no manifest, and the only case when there is no manifest is when there is no web service running [14:18:21] ack, LGTM [14:25:41] dcaro: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1480 [14:26:07] taavi: LGTM [14:26:42] do I just manually update the checkout in /data/project/test/toolforge-deploy/ if I want to use those when deploying an updated .deb? [14:27:41] ah nvm, the cookbook updates that clone [14:27:55] yep, there's also a couple options to change branches and such [14:28:17] webservice tests passed on toolsbeta, moving on to tools [14:29:36] hmm https://phabricator.wikimedia.org/P97015 [14:29:39] \o/ [14:30:13] xd too early [14:30:41] let me know if you want any help [14:32:33] worked on a retry [14:34:57] :/ ok [15:03:57] FYI swithover in progress in prod-land [15:12:08] switchover successful afaict [15:12:30] don't jinx it [15:13:40] taavi: the paws NFS security group was extremely permissive (allowing entry to all ports from all hosts) so I made a new project-local one. Didn't realize that other things read that filesystem. Sorry it messed with you [15:52:31] welp, I found why the harbor alerts have been firing [15:52:38] target=https://tools-harbor.wmcloud.org/api/v2.0/ping err="Get \"https://[2a02:ec80:a000:1::1d]/api/v2.0/ping\": dial tcp [2a02:ec80:a000:1::1d]:443: connect: no route to host" [15:52:43] Oct 22 09:07:07 proxy-5 Keepalived_vrrp[521]: (vrrp_ipv6): entering FAULT state (src address not configured) [15:53:04] tl;dr the cloud vps web proxies have had broken ipv6 since the downtime [15:54:16] :facepalm: thanks! [15:55:29] was there an alert for it too? [15:55:51] not that i can see [15:56:19] sigh [15:56:21] should I create a task to add it? [15:56:41] more we should fix keepalived to not try to start too early, i think [15:57:28] sure, I'll let you decide [16:14:28] paws seems to have lost its security group rules permitting metricsinfra scraping [16:22:09] that's quite possible, as I did not restore the gorups because it wanted to delete the magnum things that were created [16:22:33] let's just restore the defaults i'd say https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:23:41] this one does not show the diff? [16:23:48] (the tofu apply diff thingie) [16:24:04] yeah i'm waiting for a.ndrewbogott to finish deploying his tofu patch before running the cookbook [16:58:28] what is 10.244.0.0/16 and why does paws have a ssh rule from it? [17:08:53] taavi: tofu is clean now [17:09:41] and... I'm out again. Ping if needed! [17:17:26] taavi: paws bastion? [17:17:34] or all vms? [17:17:44] (is it a magnum thing?) [17:17:45] the default security group [17:17:50] might be [17:18:10] hmm... weird that it would be in the default though, magnum creates it's own sec groups [17:20:50] dcaro: bit of phabricator digging finds https://phabricator.wikimedia.org/T182722 so it's probably a leftover from an anchient setup? i'll just remove it [17:21:26] +1 [17:25:34] checking in, paws status? [17:30:00] bliviero: paws is available for most use cases, except that paws-public (access to other users public notebooks) remains unavailable [17:31:26] thank you taavi [17:43:06] * dcaro off [17:43:08] cya on tuesday! [20:04:05] re: patchdemo (T442664), we think we're good to open things back up now. [20:04:05] T442664: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664