[08:52:07] <dcaro>	 !log tools added 'disable-ssl' to tools replica.my.cnf
[08:52:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:04:09] <dcaro>	 !log toolsbeta bumped the object quota to 100G, like tools
[12:04:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:17:32] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.ranker deployed 3226a38be4 (l10n updates: ps)
[15:17:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL
[15:22:47] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed c35d575859 (l10n updates: nb)
[15:22:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[18:17:28] <hauskater>	 !help I've followed all steps in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_Pywikibot_scripts_(advanced) but I keep getting "No module named 'packaging'" in the job logs. Rebuilt venv for python3.13, no .bash_profile file, and checked that packaging is indeed installed in the venv. Any help would be appreciated, thanks.
[18:17:28] <wm-bot>	 If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[18:18:30] <wurgl>	 curl "https://services.dnb.de/sru/authorities" <-- fails within webservice php8.2 shell and works on tools-bastion-13 (I mentiones this problem here on Aug 30th, 9;46 UTC, it fixed itself magically)
[18:19:04] <wurgl>	 Error message: curl: (7) Failed to connect to services.dnb.de port 443 after 207 ms: Couldn't connect to server
[18:21:13] <dcaro>	 hauskater: can you share the tool you are running it in, and the exact commands you are running?
[18:21:28] <hauskater>	 Hi dcaro sure
[18:21:51] <hauskater>	 I'm using toolforge, and scheduling jobs using toolforge-jobs
[18:22:22] <dcaro>	 hauskater: Sure, what is the tool you are logged in as? (you had to run `become ...`, what is that `...`?)
[18:22:33] <hauskater>	 oh sorry, it's called 'mabot'
[18:23:06] <hauskater>	 I started a `webservice python3.13 shell` to check pwb version, python and pip
[18:23:41] <hauskater>	 all looks okay, yet everything keeps failing; no idea what may be happening
[18:24:32] <hauskater>	 I assume the python3.13 toolforge image has the pip `packaging` installed? Or just needs it in my tool's venv?
[18:25:15] <dcaro>	 wurgl: I can reproduce from one of the workers, do you mind creating a task to follow up? I'll test the rest in the meantime
[18:25:33] <dcaro>	 hauskater: it has pip yep
[18:25:57] <dcaro>	 hauskater: but you still have to `source /path/to/venv/bin/activate`
[18:26:20] <hauskater>	 dcaro: yup, that should be happening
[18:28:36] <dcaro>	 wurgl: we get connection refused `* connect to 193.175.100.222 port 443 failed: Connection refused`, is it possible that they are blocking traffic from the toolforge workers?
[18:29:54] <dcaro>	 hauskater: I don't see a `toolforge webservice` command in `mabot` history, but several `pwb`, note that you will not be able to run `pwb` directly from the bastion
[18:30:49] <hauskater>	 dcaro: via `webservice python3.13 shell` plus source pwbvenv/bin/activate
[18:31:02] <hauskater>	 for testing only
[18:31:12] <wurgl>	 Deutsche Nationalbibliothek? I don't think so. I had this error at 7:45 (UTC) in the morning, I had no errors at 17:46 (UTC) but I have them now
[18:31:59] <dcaro>	 hauskater: this is working for me
[18:32:14] <dcaro>	 https://www.irccloud.com/pastebin/UNdUWxk9/
[18:36:10] <dcaro>	 wurgl: traffic seems to be getting there, traceroute shows the traffig going out of our network, and we get a 'connection refused', that is not a timeout, that means it got somewhere that refused the connection :/
[18:36:57] <wurgl>	 I will contact them
[18:38:01] <dcaro>	 it works currently from the bastion, and it's taking also the same route (traceroute)
[18:38:20] <dcaro>	 wait no, it's a bit different
[18:39:36] <wurgl>	 Aug 30 11:45:36 <lucaswerkmeister>	mtr suggests connections from the bastion are going out through vlan-legacy.cloudinstances2b-gw.svc.eqiad1.wikimedia.cloud while connections from a random worker (tools-k8s-worker-112) go through vxlan-dualstack.cloudinstances2b-gw.svc.eqiad1.wikimedia.cloud, maybe something is broken in the new dualstack vxlan…
[18:39:50] <wurgl>	 Jepp, there is a difference
[18:39:56] <dcaro>	 both get to germany though `14  kr-ddbffm8.x-win.dfn.de (188.1.242.210)  33.648 ms  33.904 ms  32.641 ms`
[18:40:03] <dcaro>	 through that same node
[18:40:43] <dcaro>	 it might be possible though that something is broken
[18:41:20] <dcaro>	 it has to be subtle though, or other things would break too, and the fact that you say it worked before, makes it weirder, let me try all the workers
[18:41:21] <hauskater>	 dcaro: Hmm, I did a toolforge jobs reload and apparently this has fixed the issue
[18:41:38] <hauskater>	 since I updated these to 3.13 from 3.11
[18:41:48] <hauskater>	 checking logs right now to confirm, etc
[18:45:46] <dcaro>	 hauskater: nice :)
[18:46:35] <wurgl>	 dcaro: Maybe they are blocking me temporary because of to many requests within some time?
[18:46:50] <dcaro>	 might be
[18:49:53] <dcaro>	 wurgl: all the workers failed to connect yep
[19:22:54] <dcaro>	 wurgl:fyi. curl is working again in the workers
[19:23:53] <wurgl>	 So, maybe I need some delay between each access