[03:37:05] <wm-bb>	 <ashotjanibekyan> Hi everyone. Did something change with Toolforge recently? All my jobs are failing but when I try to run them manually, they work.
[03:38:48] <gry>	 what error message do you get
[03:40:33] <wm-bb>	 <ashotjanibekyan> I'm getting the following email
[03:40:33] <wm-bb>	 <ashotjanibekyan> We wanted to notify you about the activity of some jobs in the 'ashbot' Toolforge tool.
[03:40:34] <wm-bb>	 <ashotjanibekyan> * Job 'old-unsource' (cronjob) (emails: onfailure) had 2 events:
[03:40:36] <wm-bb>	 <ashotjanibekyan>   -- Pod 'old-unsource-28778588-qkhmc'. Phase: 'pending'. Container state: 'terminated'. Start timestamp 2024-09-19T03:08:04Z. Finish timestamp 2024-09-19T03:08:04Z. Exit code was '2'. With reason 'Error'.
[03:40:37] <wm-bb>	 <ashotjanibekyan>   -- Pod 'old-unsource-28778588-qkhmc'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-09-19T03:08:04Z. Finish timestamp 2024-09-19T03:08:04Z. Exit code was '2'. With reason 'Error'.
[03:40:39] <wm-bb>	 <ashotjanibekyan> but there is nothing in old-unsource.err file
[08:03:03] <dcaro>	 ashotjanibekyan can you open a task for it? we did an bumpy k8s upgrade lately and had a couple outages that might have left some pods in a weird state
[09:51:05] <arturo>	 !log admin [codfw1dev] play with neutron default security group rules (delete, create them, etc) T375111
[09:51:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[09:51:10] <stashbot>	 T375111: openstack: clarify default security group semantics - https://phabricator.wikimedia.org/T375111
[12:40:58] <wm-bb>	 <ashotjanibekyan> how do I do it? You mean at phabricator? (re @wmtelegram_bot: <dcaro> ashotjanibekyan can you open a task for it? we did an bumpy k8s upgrade lately and had a couple outages that might have ...)
[13:43:50] <dcaro>	 ashotjanibekyan: yep, you can create it under https://phabricator.wikimedia.org/project/view/539/ project (toolforge)
[13:44:31] <dcaro>	 guidelines here https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_report_a_bug
[13:45:11] <dcaro>	 (make sure to add the tool name, the output you pasted before and the commands you used to trigger it correctly, the more info the easier it will be to debug xd)
[16:10:08] <dcaro>	 !log rebooting tools-k8s-worker-nfs-24 it's stuck without network
[16:10:08] <stashbot>	 dcaro: Unknown project "rebooting"
[16:10:12] <dcaro>	 oops
[16:10:18] <dcaro>	 !log tools rebooting tools-k8s-worker-nfs-24 it's stuck without network
[16:10:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:31:32] <DreamRimmer>	 Hello,  I created a new tool named dreamtools but the replica.my.cnf file is not being created automatically. How to fix it?
[17:36:14] <dcaro>	 DreamRimmer: It might take a minute to create, how long has it been since you created the tool?
[17:40:08] <dcaro>	 oh, it just got created
[17:40:17] <dcaro>	 (the replica.my.cnf file)
[17:41:01] <DreamRimmer>	 yesterday
[17:41:10] <dcaro>	 hmm... I think that the process creating them got stuck, just restarted it and it created the file and such :/
[17:41:14] <dcaro>	 we should have caught that
[17:41:34] <dcaro>	 (unfortunately it was actually replying to the monitoring system saying it was alive...)
[17:41:51] <dcaro>	 DreamRimmer: can you try now? it should be there
[17:44:24] <DreamRimmer>	 yes, thanks :)