[01:09:55] !log bsadowski1@tools-bastion-15 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [01:09:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [01:10:53] derenrich: I caught one of the failures in the front haproxy log and it seems to confirm that a 500 response came out of the Kubernetes ingress. That haproxy front proxy is configured to always return the same error page, so that doesn't help debug things a lot. [01:11:29] If you have not yet started a Phab task to track things you probably should so we have somewhere to keep notes. [05:13:01] bd808: https://phabricator.wikimedia.org/T410352 [10:00:18] !log admin switch cloudcephosd1049 to single nic - T399180 [10:00:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:00:27] T399180: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180 [14:59:24] !log tools.wikibugs $ toolforge jobs restart irc # to get it to join channels after the bouncer pod was moved to a different node [14:59:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [14:59:49] ^ bd808: seems like wikibugs's auto channel joining logic gets broken after a znc restart [15:35:13] Toolforge timeout? [15:38:39] !log copypatrol copypatrol-backend-prod-02 sudo systemctl restart copypatrol-backend-check-changes [15:38:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Copypatrol/SAL [15:52:09] !status Investigating Toolforge web issues [16:00:18] The problem looks to be at least partly a traffic flood towards *.toolforge.org that is filling up the active session quota in the HAProxy edge layer. [16:07:48] !status Ok [16:20:13] thanks all for fixing my toolforge issue so quickly. things look good now [16:21:54] derenrich: thanks for helping us find growing pains in the new haproxy edge config [16:37:05] +1 to that :) [16:58:11] !log bd808@tools-bastion-14 tools.keystone-browser Replaced ingress-only redirect with tool-containers/redirect:latest webservice [16:58:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.keystone-browser/SAL [17:17:12] jelto: is it Ok if I reboot tcp-proxy-test.devtools? [17:18:03] andrewbogott: he is not currently "in office" but I will say ..just do it [17:18:16] ok! [17:18:17] thx [17:18:58] !log devtools rebooting tcp-proxy-test.devtools to reset the MTU (currently live-migration is broken for this and a few other VMs) [17:19:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:04:34] !log lucaswerkmeister@tools-bastion-15 tools.lexeme-forms deployed c74254856c (fix skiplink visibility) [20:04:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [20:05:25] !log lucaswerkmeister@tools-bastion-15 tools.wd-image-positions deployed 2c255742e4 (fix skiplink visibility) [20:05:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [20:05:33] !log lucaswerkmeister@tools-bastion-15 tools.ranker deployed f234dfee7a (fix skiplink visibility) [20:05:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [20:09:05] if you used cookiecutter-toolforge to create a Toolforge tool in the last year, please apply the following accessibility fix that I accidentally introduced with the Bootstrap 5 migration: https://github.com/lucaswerkmeister/cookiecutter-toolforge/commit/e6f4bfe13e [20:09:23] (hence the above three deployments. apparently I didn’t migrate the rest of my Python/Bootstrap tools to Bootstrap 5 yet, I should do that at some point)