[05:35:43] !log tools.dannys712-bot `toolforge jobs restart uncategorize-afc` after pulling 3d4e469a3e94aa70805e8a5a5f9d1519cadc8c5b [05:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dannys712-bot/SAL [05:48:19] !log tools.dannys712-bot ` toolforge jobs restart stub-name` after pulling b96b77533ec27e11c98e8bcf8f503584a0fc71aa [05:48:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dannys712-bot/SAL [13:45:39] !log tools hard reboot tools-k8s-control-7 [13:45:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:36:35] !log tools.dannys712-bot `toolforge jobs restart stub-name` after pulling ed29eb994ca2dfe14d3e597e05c21e85840fa2cc [14:36:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dannys712-bot/SAL [14:39:39] !log tools.dannys712-bot `toolforge jobs restart stub-name` after pulling 1500718f9dbd5588e6e41047b4004857ca3fa647 [14:39:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dannys712-bot/SAL [16:57:45] !status Toolforge Kubernetes backplane issues [16:58:06] oh, andrew did it already. I'm slow [17:02:08] if an admin is around could you restart https://quickcategories.toolforge.org/ ? [17:02:36] there's a k8s outage [17:03:09] better yet, a quickcategories maintainer is around (me) [17:03:29] ah, okay [17:03:40] but it sounds like there’s no point in me doing much at the moment [17:04:29] …yikes, since when have all those /healthz been in my uwsgi.log [17:04:41] did I deploy the health check thingy but not the patch that excludes them from the log 🤦 [17:04:42] The Toolforge Kubernetes backplane is basically down at the moment [17:06:05] ouch, the uwsgi.ini has to be in www/python/? not www/python/src/? [17:06:11] that means it’s outside of the git repo :( [17:06:21] (totally unrelated, sorry, I just noticed it now) [17:06:38] yeah, I have to symlink it seperately [17:06:58] if that’s true I can basically scrap https://github.com/lucaswerkmeister/cookiecutter-toolforge/commit/621944e6a7aa7944cafa42efef03b9b2fccbbf01 [17:07:07] or file a phab task to also look in www/python/src/, I guess [17:07:23] Or move to buildpacks ;) [17:08:23] not sure I’m ready to force that on all cookiecutter-toolforge users [17:09:05] feels like it should be easy enough to add another check to https://gerrit.wikimedia.org/g/operations/docker-images/toollabs-images/+/11ab9d8cb33ce14c1b21fd94fa9d72a1b8729457/shared/python/webservice-runner#14 [17:09:07] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/toollabs-images/+/refs/heads/master/shared/python/webservice-runner#14 is the place to extend where we look for uwsgi.ini [17:09:12] jinx :D [17:09:49] alright, I’ll file / upload things in that direction later [17:09:52] /me afk [17:09:56] * bd808 owes Lucas a Coke® [17:11:29] Toolforge Kubernetes backplane seems happier [17:20:52] quickcategories loads for me now [17:42:04] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [17:42:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [18:14:56] filed T367345 (not sure I distinguished between Use case(s) and Benefits correctly but hopefully it makes more or less sense ^^) [18:20:57] for one of my tools, I have individual symlinks for $git/src, $git/uwsgi.ini, and $git/venv [18:21:25] for the other webservice I just symlinked the whole git root directory to ~/www/python [19:41:28] !log tools Rebuilding all shared Docker containers. This will among other things apply the fix for T367345. [19:41:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:41:31] T367345: webservice-runner: Also read uwsgi.ini from ~/www/python/src/ - https://phabricator.wikimedia.org/T367345 [19:41:49] \o/ [19:42:13] and then I’ll also have to restart my webservices, right? [19:43:15] @lucaswerkmeister: yeah, once the new containers are in the registry you will need to restart your pods to pick them up. [20:27:59] @lucaswerkmeister: the new images should be in the registry now. [20:28:07] alright, let’s see… [20:28:41] !log lucaswerkmeister@tools-bastion-13 tools.quickcategories restarted webservice to pull in new image for T367345 [20:28:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [20:28:59] hm, uwsgi.log is empty… I guess that means the log rotation setting got picked up ^^ [20:29:23] yeah there’s also a uwsgi.log.1718224123 now [20:30:19] and now it’s getting populated again and I don’t see any /healthz requests, so I think it’s working \o/ [20:30:23] thanks a lot bd808! [20:30:56] nice. Happy to be the +2 and deployment monkey for that fix :) [22:57:59] !log bd808@tools-bastion-12 tools.schedule-deployment Built new image from 52a9d4a4 (T366763) [22:58:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schedule-deployment/SAL [22:59:12] !log bd808@tools-bastion-12 tools.schedule-deployment Restart to pick up new container image (T366763) [22:59:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schedule-deployment/SAL [23:22:06] anyone know what the preferred way to tell Magnus that petscan is down is these days? [23:43:39] no idea tbh [23:43:54] I know he has the Buggregator now (http://magnusmanske.de/wordpress/?p=658), but that doesn’t answer the question of what the *preferred* way to report bugs is