[07:58:13] <kbch>	 I'm having trouble starting the techdoc-dashboard tool after using --use-latest-versions. Seems like the location of the Java binary in the Clojure buildpack changed again. I used to be able to just call "java" in the Procfile, then had to change to "/app/.jdk/bin/java", and with the latest version neither works. Any suggestion on how to find the right binary?
[08:35:21] <wm-bb>	 <lucaswerkmeister> there’s `/workspace/.jdk/bin/java` (and also a few others; you can run the docker image manually and then check `find / -name java` inside)
[08:35:38] <wm-bb>	 <lucaswerkmeister> but there should probably be a phabricator task about it not being in the `$PATH`, that sounds like a bug to me
[08:49:04] <kbch>	 Thanks, I'll have a look and see if this fixes it. There's already a task: https://phabricator.wikimedia.org/T411486 - I'll add a comment that it happened again.
[08:51:49] <wm-bb>	 <lucaswerkmeister> alright, thanks!
[09:05:01] <kbch>	 Using /workspace/.jdk/bin/java worked, thank you!
[09:25:07] <wm-bb>	 <lucaswerkmeister> \o/
[13:32:16] <wm-bb>	 <Roy> Yesteday, I started up a toolforge cron job.  It has been running fine for the most part, but I see in my error log one failure:
[13:32:28] <wm-bb>	 <Roy> WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 114593102230754635671774431008
[13:32:29] <wm-bb>	 <Roy> ERROR: Retrying failed OAuth authentication for wikipedia:en: The authorization headers in your request are not valid: Nonce already used: 114593102230754635671774431008
[13:32:42] <wm-bb>	 <Roy> What's going on there?
[13:33:14] <wm-bb>	 <Roy> I'm using pywikibot with owner-only OAuth credentials.
[15:06:10] <bd808>	 @Roy: that error means that pywikibot glitched the OAuth handshake somehow and resent a one time pad value (nonce) with a second request. Probably some transient environmental cause unless you are seeing it happen over and over.
[16:26:38] <wm-bb>	 <Cyberpower678> My VMs seems to have become unreachable.  I hit bastion just fine, but the proxying over to my VM seems to just hang
[16:27:43] <wm-bb>	 <Cyberpower678> Nvm, they're starting to load agin
[16:27:44] <dcaro>	 Cyberpower678: can you share which vm is having the issue? Do you get any logs from horizon?
[16:28:18] <dcaro>	 :)
[16:33:09] <wm-bb>	 <Cyberpower678> cyberbot-exec-iabot-01
[16:33:39] <wm-bb>	 <Cyberpower678> A notoriously broken VM that's due to be scrapped in the foreseeable future but needs to exist for the time being.
[16:39:28] <mutante>	 Cyberpower678: imho, just click reboot (soft or hard) in Horizon and it's most likely back
[16:39:51] <wm-bb>	 <Cyberpower678> I do that.  But I have to do that as many s 4 times per day.
[16:40:06] <wm-bb>	 <Cyberpower678> Other times, I may only need to do it once a week.
[16:40:06] <mutante>	 oh, i thought it's once every few weeks
[16:40:28] <mutante>	 maybe it's running out of RAM
[16:41:31] <wm-bb>	 <Cyberpower678> Very possible, but there's other oddities happening on it that the other VMs, even if low on RAM, don't exhibit.  I get kernal log explosions that drive the disk to 100% and such.
[16:41:45] <mutante>	 you can reboot it one more time and then look at /var/log/syslog   like  "grep killer" /var/log/syslog
[16:41:57] <mutante>	 to see if you got any " invoked oom-killer" in there
[16:42:40] <mutante>	 I see, yea, in that case, try to put some logs that look relevant on a paste bin
[16:45:10] <wm-bb>	 <Cyberpower678> Oh wow oom-killer is hard at work.  Admittedly, IABot is red-lined and pushing limits.  That's being worked on though.  I am planning to scrap the VM in the foreseeable future.  It is deprecated anyway.
[16:45:57] <mutante>	 ok, yea, in that case.. you can only get a VM with more RAM or fix the software or split things up between multiple machines. I guess.
[16:46:24] <wm-bb>	 <Cyberpower678> WIP. 😊
[16:46:43] <mutante>	 we don't know if it just legit needs that much RAM or if there is a memory leak due to some software bug that could be fixed
[16:46:44] <perryprog>	 Someone should make a version of downloadmoreram that's FLOSS, so we can host it on toolforge...
[16:47:46] <mutante>	 :) sounds like a plan
[18:03:13] <dcaro>	 the hackathon is not far ;)
[18:08:37] <mutante>	 Cyberpower678: I did not expect this to work.. but looks like I got a JetBrains license for you :p
[18:27:44] <_Gerges>	 Ping
[19:48:03] <wm-bb>	 <GergesShamon> What is the maximum count of cores that can be run in a container in Toolforge?
[19:56:47] <wm-bb>	 <Yetkin> In site-wide CSS pages, we are using " var(--background-color-target-text)" to define the background color of a container in dark mode. How to pass a default value for bright mode?
[20:28:41] <wm-bot>	 !log difronzo@tools-bastion-15 tools.wikivisage deployed v0.7.1 (by DiFronzo on GitHub)
[20:28:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikivisage/SAL
[20:33:11] <addshore>	 not exactly "urget" but the sooner the better :D https://phabricator.wikimedia.org/T421292 (As it says I might want to poke people toward it)