[07:49:52] akosiaris: m1 switchover happening in 10 minutes! [07:51:03] * akosiaris hides [07:52:31] I've disabled puppet on backup1001 [07:52:43] so bacula dir doesn't auto-restart [07:52:48] now that not job was running [07:55:07] tbh, for etherpad I am gonna be a bit more risky today. I 'll wait at least 30s to see if it will pick up the change on its own, or at least commit harakiri so that systemd may restart it [07:55:30] it's pretty early in the EU morning so it should impact very few users. Plus.. it commits harakiri anyway every 3-5 days [07:55:37] akosiaris: cool [07:55:54] active (running) since Sun 2020-06-21 00:00:04 UTC; 4 days ago [07:56:22] 4secs after midnight... coincidence? [08:39:40] volans: oh, hah. side-effect of removing py36 tox support is that i can't run tests on my dev machine (which is ubuntu 18.04, with python 3.6.9) [08:40:11] End of Life :D [08:40:26] what I do is to have all python versions so I run tox with all of them all the time [08:40:30] that's a bit overkill I know [08:40:38] looks like there's a ppa that will allow me to do this [08:40:46] but I feel I have to, YMMV ofc [08:40:58] and we can re-add 3.6 if we feel it's needed [08:41:29] (https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa) [08:42:00] kormat: we also have some stuff in house [08:42:03] let me show it to you [08:42:37] https://people.debian.org/~paravoid/python-all/ [08:42:42] please note the username ;) [08:42:57] ah hah :) [08:43:05] that's what we use in CI [08:43:10] to make CI run on all python versions [08:44:05] not sure if it will work on ubuntu [08:48:19] volans: oh also i found a wonderful thing while cleaning this up: https://phabricator.wikimedia.org/P11660 [08:48:53] wut?!?! [08:49:08] yeaah [08:49:08] that should not happen... [08:49:14] yeah [08:50:20] oh wow. it actually happened again on a fresh setup [08:51:28] volans: how does tox decide what binary to use to create a given version'd venv? [08:51:47] .tox/py37-tests/bin/python3 worked correctly [08:52:02] from the py38 name [08:52:13] which python3.8 ? [08:52:20] gives you anything? [08:52:25] /usr/bin/python3.8 [08:52:42] and if you run that? [08:52:46] you get 3.8 or 3.6? [08:52:51] 3.8.3 [08:53:15] virtualenv --python=python3.8 foo [08:53:18] does the right thing? [08:53:29] like source the activate and check the python inside [08:54:00] yep, that works correctly too [08:54:12] when you say "fresh setup" you mean after a rm -rf .tox? [08:54:15] yep [08:54:19] damn [08:54:43] by default tox would create a venv per environment in envlist [08:54:53] but we force it to re-use the same to be quicker [08:54:57] in the envdir directive [08:54:58] py38: {toxworkdir}/py38-tests [08:55:21] which tox version do you have? [08:55:34] I don't see how that could fail / get you a 3.6 [08:55:36] let's take this to PM to avoid tox-ifying this channel [08:56:19] lol [09:12:40] jayme: thanks! it's great :) [09:15:29] volans: kormat: there is some black magic in how tox pick the python version to use :/ [09:16:05] hashar: we figured it out in the end; tox 2.5.0 was broken, tox 3.14.x was broken, tox 3.15.x works [09:16:11] nice [09:16:21] though CI still has 3.10.0 [09:16:28] and CI lacks python3.8 :\ [09:16:48] we have the backport for buster... [09:17:02] for local run, I went some hack to use whatever "python3" is available on the machine but on CI use various python version [09:17:14] I use pyenv [09:17:28] I added you as a reviewer on some black change https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/607512/1/tox.ini [09:17:46] yeah pyenv. Maybe we should use that instead of the python debian package for CI [09:23:39] hashar: I'm not sure what you want to achieve with venv-py3 [09:24:30] the idea is that locally one probably just has one python installed and I am trying to avoid running commands ofr the other commands [09:25:02] maybe I should just use skip-missing-interpreters instead [09:25:11] yeah that's what we usually do [09:25:58] then locally one would have to invoke the proper pyenv eg py37-unit [09:26:05] instead of the straightforward "unit" one ;] [09:26:42] to run lint and unit you just run 'tox' plain [09:26:47] to run the optional ones yes [09:27:28] also envdir is defined both in the envdir top level directive and the sub-section ones [09:27:32] so not sure what happens there [09:28:15] ah yeah that is the case for lint, the sub section one wins [11:58:37] Is it reasonable for mediawiki instances that need to access things like APIs (the Microsoft PhotoDNA API in this case) to use the webproxys in their respective DCs or is there another pattern they should follow? [12:03:31] yeah, this is handled by the urldownloader* instances, there are CNAMEs for the active host, you can use url-downloader.eqiad and url-downloader.codfw (or url-downloader (which points to .eqiad ATM)) [12:03:49] oh, cool [12:33:49] volans: i swear i didn't break it: https://integration.wikimedia.org/ci/job/spicerack-tox-publish/187/console [12:35:30] yeah, I did [12:35:36] patch incoming [12:37:43] kormat: https://gerrit.wikimedia.org/r/#/c/integration/config/+/607778 [12:38:16] hashar: if by any chance you have a sec for it ^^^ :) [12:38:27] sure [12:42:04] volans: done! [12:42:55] thanks a lot! [12:48:10] hashar: is there a way to re-trigger the post-merge build via a comment on gerrit? [12:50:16] volans: nop [12:50:29] volans: but I can retrigger it usin gthe zuul command line [12:50:48] if not too much trouble thanks, otherwise not critical or anything [12:50:51] just need the number of the last changes merged :) [12:51:02] also would be nice to use a generic environment name ;] [12:51:27] software/spicerack/+/603434 [12:51:31] yeah saw your comment [12:53:20] volans: https://integration.wikimedia.org/ci/job/spicerack-tox-publish/188/ [12:54:37] great! thanks a lot hashar! [17:20:46] question about puppet running automatically: if it fails, does it try repeatedly and fail or does it do that and alert as well? because I had a failed puppet run since probably yesterday and I never got an alert for it [17:21:09] puppet agent [17:21:48] it will try every half hour [17:22:24] it will eventually alert; puppet failures used to be very spammy on IRC and now there's only an IRC alert right away for large numbers of hosts failing; there's eventually (days?) an alert for hosts persistently failing [17:22:32] ah that explains it [17:22:50] (see -operations for more context sorry :) [17:22:54] when I'm making puppet changes I often just ssh into the host and run-puppet-agent immediately after puppet-merge'ing, since it's all too common to have made some mistake that pcc didn't catch [17:23:04] and that shortens the debug/fix cycle [17:23:23] yeah I do that as well. in this case, I made a change to private *after* doing that and then it failed and I never looked because there was nothing to merge :) [17:23:45] and I guess the only reason we got the alert now was because that it was compounded by the other hosts failing [17:23:52] ahhhh [17:24:06] private repo changes are... tricky. [17:25:56] ha yeah. but anyway, thanks for clearing up this doubt. I couldn't figure out what I did wrong or what went wrong [17:54:05] we used to have individual puppet alerts/notifications in the past, yes they were considered spammy but also caught more stuff earlier, ack [17:54:32] that's why i was talking about "getting under the threshold" again before we see a recovery of the global check [17:56:06] +1 to always running puppet agent manually right after puppet-merge but a change in the private repo like that is a special case of course.. but it's rare [17:59:00] yeah.