[00:03:46] There is not a single line of python code in my codebase (so, this must be from toolforge). [00:27:38] Another question, is it possible to get all of my access logs of the tool? `toolforge webservice logs` does not show all of them [00:31:48] which do you think are missing? (re @nokibsarkar: Another question, is it possible to get all of my access logs of the tool? toolforge webservice logs does not show all of them) [00:33:54] like, my tool started toolforge journey at least from 27th may. But upon `head access.log` I see it starts from 3rd June [00:34:27] maybe that just means that older log was pruned somehow. [00:34:32] or rotated [00:34:36] btw, access.log was dumped from that command [00:35:16] if rotated, where can i find them? [00:37:07] nginx: [emerg] mkdir() "/app/.heroku/php/var/run/nginx/fastcgi_temp" failed (2: No such file or directory) [00:37:18] is also seen on my logs [07:46:09] !log quarry delete redis pod stuck in Completed with no futher explanation why [07:46:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [13:13:35] !log admin add AAAA record to openstack.codfw1dev.wikimediacloud.org T347148 [13:13:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:13:41] T347148: Determine how to monitor services in cloud-private / cloudlb - https://phabricator.wikimedia.org/T347148 [13:13:55] oops wrong task :/ [13:14:14] !log admin add AAAA record to openstack.codfw1dev.wikimediacloud.org T379282 [13:14:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:14:20] T379282: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282 [14:05:12] Taavi: Have you guys considered using ipv6 unique local addresses for the part that is supposed to not be internet facing? [14:15:26] @MaartenDammers: the endpoint referenced there is in fact an internet-facing one :-) but no, I don't think that was ever talked about [15:21:42] !log bd808@tools-bastion-12 tools.wikibugs Built new image from git hash 64a8e04d [15:21:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [15:27:41] I am facing issue in gitlab CI/CD >> https://gitlab.wikimedia.org/nokibsarkar/campwiz-frontend/-/jobs/531647 [15:30:04] Retried and the same OOM error. Seems like gitlab is refusing to build nextjs [15:30:28] "FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory" -- You might try adding `tags: memory-optimized` to get a bit more ram. https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner#GitLab_Runner_types [15:32:44] where should i add this tag? [15:39:35] Got it [15:59:55] (new Toolforge user question): I am confused about why my tool (webservice) doesn't seem to get updated when I push new code to GitLab, then login to Toolforge, become my tool, trigger a new build and restart the webservice. Am I missing a step? Docs consulted: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images/My_first_Buildpack_Python_tool and [15:59:55] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images [16:00:32] which tool? [16:00:51] https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics [16:00:59] https://tech-doc-metrics.toolforge.org/ [16:01:11] that is the update workflow for a buildservice tool, so something "interesting" may be happening [16:01:49] i have gotten to the point where i know just enough to make "interesting" things happen but not enough to know they are happening ;-) [16:02:24] something interesting's happening, with that namespace trying to run both a buildservice and non-buildservice webservice at the same time [16:03:05] hahah yep that is definitely my fault. i originally was following the old docs and then when i couldn't get the tool to run i switched tactics so i probably left a trail of errors in my wake...SORRY [16:03:12] that project has a `www/python/src/service.template` with `type: python3.11` which might be confusing things [16:03:31] i would stop the webservice properly, move that file to somewhere different, then try starting it again [16:03:44] where should I move that file? [16:04:20] `mv service.template service.template.py311` or something [16:04:34] okay I'll do that [16:04:42] or just delete it, or update that backend to buildservice [16:05:08] you might want to just get rid of the `www/python` directory if you're not using that method of deploying it [16:12:11] That worked, thanks! [16:23:14] i am still getting oomkilled (but not always). sometimes it builds [16:30:34] Because a lot of the "firewalling" in ipv4 is actually based on the fact that NAT is mostly unidirectional and for v6 you have to explicit firewalling. (re @wmtelegram_bot: @MaartenDammers: the endpoint referenced there is in fact an internet-facing one :-) but no, I don't think that was ever...) [16:30:55] People forget or mess up and you end up with security holes. [16:39:08] nokibsarkar: Do you have a link to a failing job that ran on the memory optimized runners? [17:01:50] https://gitlab.wikimedia.org/nokibsarkar/campwiz-frontend/-/jobs/531649 (re @wmtelegram_bot: nokibsarkar: Do you have a link to a failing job that ran on the memory optimized runners?) [17:02:59] Thanks. That job says that it ran on a non-memory-optimized runner, based on this part of the page: `Runner: #1480 (-WhWGubP) Cloud Runners, running in Digital Ocean K8s` [17:03:36] (It would have said "memory optimized" in there somewhere if it had run in the right place) [17:04:27] I'm looking at https://gitlab.wikimedia.org/nokibsarkar/campwiz-frontend/-/commit/dd302c78f123cdd6e964c3628143e2e597849367 ( the commit associated with that job) and I don't see any memory-optimized tags. [17:05:01] nokibsarkar: can you point me to the commit where you added the memory-optimized tag? [17:13:03] I see you added it in https://gitlab.wikimedia.org/nokibsarkar/campwiz-frontend/-/commit/469e8d12251cb015d7a10e231c1f3dfca58d71e9 and the corresponding `build` job did run successfully: https://gitlab.wikimedia.org/nokibsarkar/campwiz-frontend/-/jobs/531727 [17:13:34] The link you originally provided was for an old job before you added the tags. I'm interesting in jobs that failed _after_ you added the tags. [17:43:55] Sorry, did not notice. Here is the build >> https://gitlab.wikimedia.org/nokibsarkar/campwiz-frontend/-/jobs/531759 [18:37:26] thx [20:18:43] nokibsarkar: As a fallback you can try using the `wmcs` tag instead.