[04:48:31] In 10 minutes we'll failover phabricator master, so we'll set phabricator in read only for a few minutes [06:44:06] in 5 minutes we will do maintenance on phabricator, some minutes of unavailability could happen [06:57:18] if you still see some phabricator errors, please report [10:15:27] https://phabricator.wikimedia.org/T247966#6392501 [10:15:42] * godog cries in dh_params [10:22:03] godog: just pad it with 0's! [10:23:31] I like your optimism kormat! hold my params while I recompile nrpe [10:24:20] yes in jessie's version of nrped you can't specify an external file :( [10:25:02] urk [12:05:20] who was telling me about the 'wonders' of debdeploy? methinks it was _joe_ [12:05:37] <_joe_> I said nothing about its wonders [12:06:08] _joe_: close enough [12:06:16] can you tell me why it's being like it is: https://phabricator.wikimedia.org/P12289 ? [12:07:20] kormat: you need to either pass a cumin alias using `-s` or a cumin query with -Q [12:07:47] try ` sudo debdeploy deploy -u 2020-08-18-wmfmariadbpy.yaml -Q 'cumin2001.codfw.wmnet'` [12:08:03] <_joe_> jbond42: that's not the problem [12:08:17] (i've updated the paste to remove the distraction :) [12:08:42] oh yes wasn't looking properly :) [12:10:31] doesn't look like wmfmariadbpy 0.4 is available kormat ? [12:11:03] https://phabricator.wikimedia.org/P12289#69035 [12:11:52] i wonder if i've hit a corner-case. in 0.3 there was a metapackage called `wmfmariadbpy` (same name as the source package). in 0.4 i've removed that. [12:12:05] https://phabricator.wikimedia.org/P12289#69037 [12:12:33] jbond42: yeah that's the deprecated metapackage [12:14:21] possibly a corner case then yeah [12:14:22] kormat: have you run apt-get update? [12:14:27] volans: yes [12:14:29] sorry for the trivial question :) [12:14:30] kormat: i dont know enough about the innards of depdeploy but that would be my first guess [12:14:33] just to exclude it [12:16:03] i'll have a look at the debdeploy source to see if there's something obviuos [12:16:06] *obvious [12:16:46] kormat: shout if you need a hand, I'm familiar with that code, not enough to tell you what the problem could be though [12:19:22] i wonder if i should be using the 'transition' feature [12:21:15] ah haha: https://phabricator.wikimedia.org/P12289#69038 [12:21:45] https://i.kym-cdn.com/entries/icons/original/000/007/508/neildegrasse.jpg [12:21:47] alrighty, cumin it is! [12:21:55] godog: :D [13:16:45] can i have a volunteer to review this 1-line CR? https://gerrit.wikimedia.org/r/c/operations/puppet/+/620931 [13:16:51] (needed to unbreak puppet on cumin hosts) [13:18:35] cdanis: cheers. that was surprisingly helpful [13:19:11] 💜 [13:19:38] btw that patch is well within the set of patches it seems acceptable to self-+2 around here [13:20:40] oh, yeah. fair point. [15:51:52] <_joe_> definitely. [15:54:14] <_joe_> btw, if you deel we're too lax with the self-merges, we can discuss the matter [16:17:08] kormat: thanks for the quick response on the mysql grant ticket! [16:52:19] hii [16:53:53] I have an issue running docker-pkg on contint2001 [16:54:02] cause it is configured with http_proxy: http://webproxy.codfw.wmnet.wmnet:8080 [16:54:20] there is an extra .wmnet in the name. File last changed Aug 18 13:58 [16:56:27] hashar: looks like from this https://gerrit.wikimedia.org/r/c/operations/puppet/+/620784 [16:56:44] http_proxy_host: webproxy.%{::site}.wmnet in common.yaml [16:56:48] nice [16:56:55] that probably broke a bunch of things then [16:57:08] ottomata: ^^ ;] [16:59:43] I am inclined to get it reverted [17:00:54] oh [17:00:56] oh man [17:00:57] sorry [17:01:07] turns out we can just remove the extra .wmnet :] [17:01:09] i think i had forgotten that i had that change in that patch even [17:01:13] i was working on it so long [17:01:18] hehe [17:01:40] i looked at PCC for so many things but not that [17:01:41] yikes [17:01:41] sorry [17:01:43] um [17:01:54] wait %{::site} is e.g. codfw.wmnet [17:01:55] ? [17:01:57] not codfw? [17:02:00] anyway yes let's fix it [17:02:10] making patch [17:02:14] ottomata: I did ) [17:02:17] oh' [17:02:18] https://gerrit.wikimedia.org/r/c/operations/puppet/+/621029 Restore http_proxy broken by be4a47b [17:02:46] I am not sure what is the impact though [17:02:47] OHHHHH i had an extra in there [17:02:49] sorry [17:03:12] as to what have broken since the change got deployed. But probably not much or we would have a bunch of alarms here and there [17:03:30] yeah [17:03:36] yikes [17:03:37] ok [17:09:29] ottomata: can you review and apply that patch please? ;) [17:15:31] mutante: you're welcome :) [17:16:09] have to head out for dinner, but in short would be nice to have http_proxy fixed please https://gerrit.wikimedia.org/r/c/operations/puppet/+/621029 [17:16:13] * hashar escapes to eat [17:40:05] <_joe_> oh god how is it things still work? [17:40:09] <_joe_> (that proxy change) [17:40:41] <_joe_> ottomata: I think it's better if for such changes (involving common.yaml) you ask/wait for a review [17:41:35] <_joe_> having said that, it's very late for me [17:41:46] <_joe_> can anyone in a correct TZ merge that change ASAP? [17:42:28] <_joe_> cdanis / rzl ^^ [17:49:42] merged [17:50:35] ran puppet on contint2001 as well [18:05:51] just back from lunch, thanks cdanis [18:10:00] <_joe_> thanks cdanis :) [18:12:09] <_joe_> and ofc thanks hasharDinner [18:15:12] cdanis: rzl: _joe_: thanks ;] [18:16:40] it looks like it did break our rpki scrape monitoring? [18:17:41] at least, the breakage of such correlates [18:18:51] they do use the proxies to scrape the RIRs so seems unsurprising [18:27:00] Anybody here have a working setup for cookbook dev locally? No matter what I do, which venv or version of python (3.7 or 3.8), I cannot make the `cookbook -l` command do anything but fail to import the cookbooks module. [18:28:25] bstorm: I haven't tried to do it myself, but just to be sure, have you seen https://wikitech.wikimedia.org/wiki/Spicerack/Cookbooks#Creating_your_local_environment already? [18:28:56] last I tried that procedure it worked [18:29:02] but it's been a while [18:29:25] Yes I did all that [18:29:39] But, running from the tox env does the same thing [18:29:51] I've used various versions of the config.yaml [18:30:15] I've also tried running from a pipenv that uses a pyenv py3.7 😆 [18:30:33] https://www.irccloud.com/pastebin/NuFMGGF2/ [18:30:38] Always get that [18:31:11] That's from my virtualenv from pipenv, but you get the idea [18:31:27] All the venvs I've used produce the same result once I get the versions right of the various libraries [18:32:02] cookbooks is a plain checkout of the gerrit repo right now [18:33:16] I can just accept that it will not run locally, but that feels dreadful when it comes to trying things and debugging [18:34:43] I briefly got a venv that worked by producing no output instead of an error...which is still wrong, but it seemed like progress. That was from a particular version of pyparser I think [18:38:19] bstorm: can you also paste your config.yaml? [18:39:52] https://www.irccloud.com/pastebin/h56CHNSp/ [18:40:21] lol, looks very reasonable [18:40:39] Yup. That's my checkout. I've tried other locations for base_dir and it gives the same error [18:40:56] I'm sure that this is the case, but, /home/bstorm/src/wmf/cookbooks/cookbooks/__init__.py exists right? [18:41:09] yes [18:41:18] It's a largely unmodified checkout [18:41:43] https://www.irccloud.com/pastebin/64m9qVcd/ [18:41:53] ayup [18:44:32] I am still scratching my head [18:45:24] Does it work for you? [18:45:40] I've noticed this is quite sensitive to some python library versions [18:45:59] If it works for someone here, I may be able to simply compare a pip freeze :) [18:46:59] This is my current virtualenv's pip freeze: [18:47:18] I'm re-creating an environment now [18:47:35] https://www.irccloud.com/pastebin/Od1hqnAF/ [18:50:15] mine is https://phabricator.wikimedia.org/P12297 [18:50:17] and it works [18:50:24] but I'm on Linux, not MacOS... [18:52:31] diff at https://phabricator.wikimedia.org/P12298 [18:52:36] yours == lhs, mine == rhs [18:53:51] Ok, I'll play with that :) [18:53:55] Thanks! [18:54:49] how did `-e git+ssh://b=wikimedia_spicerack` wind up in yours? [18:56:33] Me trimming the line [18:56:33] bstorm: not on my laptop now, do you need any help? [18:56:43] wikitech describe the setup [18:56:51] Yeah, I did it, and it won't work basically [18:57:05] Let me give you my tox venv's pip freeze [18:57:19] most of the diff is test packages [18:57:48] https://www.irccloud.com/pastebin/2OnCkVeS/ [18:58:03] cdanis: that's more similar...but still doesn't work :) [18:58:39] did the compiler stop taking class names instead of hostnames? it used to be "C:" you could put in the list of nodes, right? it seems to compile but match no hosts now [18:59:37] mutante: is that in the compiler itself, or just in the `Hosts: ` line in the patch description for use with `check experimental`? [19:00:17] cdanis: in the compiler UI at https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build?delay=0sec [19:01:08] sorry really can't help right now, I'll ping you later if I can bstorm [19:01:39] the experimental check sometimes said success to me when it actually failed, so that made me go back to usually not putting hosts in the commit message [19:10:54] actually the diffs match perfectly now [19:10:54] volans: basically, I follow the instructions on wikitech on my mac and the `cookbook` command always produces the output above [19:11:26] it cannot find the cookbooks module despite the correct config 🤷🏻 [19:13:57] <_joe_> bstorm: my dev env for cookbooks is amazing. It's called cumin2001 [19:14:06] LOL [19:14:11] <_joe_> it looks quite literally like production [19:14:48] cumin2001, or as we call it, "staging" [19:16:41] <_joe_> not to be confused with 'beta' [19:18:03] * bstorm goes to install this on linux to see if it works and then will just give up if so [19:18:26] I know volans uses a Mac too so that can't be the only issue [19:20:44] <_joe_> cdanis: I'm sure he has a very peculiar setup [19:26:18] Ugh. My virtual machine needs to finish unattended upgrades and reboots and nonsense. I'll deal with that later :-/ [19:48:28] FYI, I did the steps on a linux VM, and it works [19:48:34] :-/ [19:48:52] So there's something in python on mac that makes it not work...somewhere [19:56:39] bstorm: I guess there is something slightly broken due to the virtualenvs ;] [19:57:07] cause the bin/cookbook is in a virtualenv but spicerack is outside of it ? [19:57:12] Doesn't seem like it. I created them exactly the same in both environments (and in several other ways). [19:57:37] anyway you might use PYTHONVERBOSE=x to print the import statements. That is sometime helpful [19:57:44] On linux, it's fine. I suspect there's some library or something in a linux install that is not there or differently versioned than on mac [19:57:59] I have several pythons on mac too...no version has worked [19:58:15] Makes me want to try my windows python just for the masochism [20:02:21] It ultimately dies here: `import 'spicerack.cookbook' # <_frozen_importlib_external.SourceFileLoader object at 0x100ad3250>` [20:02:38] https://www.irccloud.com/pastebin/2ku0xd4l/ [20:03:52] And that should be controlled by the config, theoretically [20:25:27] I think I found the problem. I just made a stupid mistake in my config [20:25:33] *sigh* [20:26:42] Yup it works. I didn't even see it above when I pasted it. [20:26:51] s/home/Users/ [20:26:54] Damn you macos! [20:27:01] Thanks all! [21:45:13] ahahah sigh [21:45:18] I missed it too [21:45:49] we should probably just add a call to os.path.isdir() :)