[07:13:32] <_joe_> "a derivative over 10 minutes" is an interesting concept :P [07:17:28] good morning, today is hadoop upgrade day! [07:17:45] we are prepping the work (draining the cluster, take the last backup, etc..) [07:17:57] we should be able to start around 10:30/11 CET more or less [07:35:42] exciting, hopefully in a good sort of way! [07:35:53] what improvements does the new version bring? [07:40:16] <_joe_> tangentially, I'm going to be out for part of the morning [07:40:27] <_joe_> gotta thanks my bank for saving me from the fun [07:48:02] someone knows how I can downtime the whole of esams? [07:48:35] godog maybe? ^ [07:57:36] volans: or is it possible with a cookbook? [07:59:48] XioNoX: yes use the downtime cookbook... but... [08:00:15] trying to figure out what's the cumin syntax for all of esams [08:00:20] it will not downtime network devices IIRC and other alerts unbound to devices [08:00:28] like a URL chexk [08:00:36] A:esams [08:00:51] volans: VIPs neither? [08:01:21] If their parwnt is not a devixe no in it's cirrent form [08:01:33] maybe from the UI is easier if we have a common parent [08:01:49] I can do the VIPs via the UI [08:01:50] on mobile right now so can't check [08:01:55] ok, thx [08:09:43] alright, brace yourself, esams switch stack is going to go down, hopefully for no more than 20min [08:12:00] Uptime: 475d1h53m8s [08:12:09] XD [08:13:19] * volans bracing [08:20:48] upgrade done, now it would be nice if the interfaces were showing up [08:21:05] here they are [08:21:10] and bast3005 replies to ping [08:21:22] things should start recovering [08:22:23] so about 11min downtime, not bad [08:30:27] XioNoX: we are having issues with phab [08:30:30] hashar: ^ [08:30:52] marostegui: ? [08:31:40] hashar: from the proxy side everything looks up [08:32:46] "Woe! This request had its journey cut short by unexpected circumstances (Can Not Connect to MySQL)." [09:36:51] volans: hurrah re: black & spicerack [09:39:30] volans: ohh. i didn't know that `py3` was accepted by tox. that's neat. [09:41:03] should use the default py3 you have not requiring to force everyone to have the same python version [09:41:56] 👍 [09:47:29] volans: weirdly, `tox` doesn't run py3-style for me [09:48:07] kormat: on purpose [09:48:11] it runs the check [09:48:14] not the formatter [09:48:24] volans: i'm saying it doesn't run the check [09:48:53] kormat: is it part of tox -lv ? [09:48:56] py3-style -> Style consistency checker (Python 3) [09:49:09] ohh, nevermind. i have a `.envrc` that sets `TOXENV` [09:49:17] lol [09:49:56] (direnv ftw) [09:50:31] and yes py3-style should run with tox, py3-format not, sorry for the confusion [09:51:05] np :) [09:51:35] i think i'll steal the `py3-` approach for wmfmariadb [09:51:50] it'll fail if anyone's default python3 version is 3.5, [09:51:55] but in that case they're asking for pain anyway [09:54:25] uff. tox is happily running the py39- stuff. but i do not have python 3.9 [09:54:42] what do you mean? [09:55:01] ``` [09:55:04] $ ./.tox/py39-tests/bin/python --version [09:55:04] Python 3.8.5 [09:55:04] ``` [09:55:20] the version of tox is too old to know about python 3.9, so it just makes shit up [09:55:33] you can force a minversion [09:56:29] volans: of? tox? i'm unclear how that would help [09:57:25] you said tox is too old [09:57:48] volans: i'm saying the version of tox that ships with ubuntu 20.04 is older than python 3.9. [09:58:09] so having py39 in tox.ini is never going to work right [09:58:29] same goes for debian stable [09:58:39] the implication being: we should remove py39 from tox.ini [09:58:54] because it won't work for anyone who isn't running debian testing/unstable [09:59:31] I can't test it right now but I doubt that's the case [09:59:42] tox version doesn't have to "know" about python 3.9 [10:00:14] https://tox.readthedocs.io/en/latest/config.html#tox-environments [10:04:11] ahh. my memory is out of date. until v3.13 (in 2018) it had a hard-coded list of supported versions. https://tox.readthedocs.io/en/latest/changelog.html#v3-1-3-2018-08-03 [10:04:44] ok, then it's just broken. it's supposed to use python3.9, or skip it if the interpreter is missing. it does neither [10:05:08] weird, try to nuke the .tox directory [10:05:13] i did [10:05:15] is this in spicerack or wmfmariadb ? [10:05:20] spicerack [10:05:43] weird, care to open a task? I'll have a look [10:06:26] sure [10:08:48] <_joe_> kormat: sorry, did we ever give you the impression we care about anyone but people running debian sid? [10:08:59] <_joe_> if that's the case, I apologize. We really don't. [10:09:08] _joe_: i know _you_ don't. i had some hope for the less damaged :) [10:09:20] <_joe_> :* [10:09:25] :D [10:09:32] <_joe_> I'm now running ubuntu stable on my laptop btw [10:09:46] volans: https://phabricator.wikimedia.org/T274232 [10:10:10] _joe_: 20.04? or 20.10? [10:10:26] <_joe_> I meant LTS, yes [10:10:33] <_joe_> although I'm tempted to jump ship [10:10:33] ok :) [10:34:39] headsup: I'll reboot cumin2001 in ~ 5 minutes, please use cumin1001 for cookbooks/cumin/debdeploy for a bit [10:36:13] ack [10:36:14] thx [11:06:19] kormat: thanks the for finding! this is happening in CI too [11:06:24] volans: ah hah! :) [11:07:19] I thought they'd added py39 to the images [11:09:31] volans: it gets worse! even with py3.9 available, it _still_ uses py3.8 here [11:10:14] that is weirder, uses the right oen for me [11:10:23] $ .tox/py39-tests/bin/python --version [11:10:23] Python 3.9.1 [11:10:42] and 3.8.2 and 3.7.7 respectively for the others [11:10:52] volans: what version of tox do you have? [11:11:04] I upgraded the other day, 3.21.3 [11:11:23] and my default python3 is currently 3.9.1 [11:11:28] so might alter things [11:13:00] let me try changing that [11:15:44] looking through the tox changelog, i don't see anything relevant [11:17:00] all works here with default py3 to 3.7, let me downgrade tox and do some binary search of when it breaks [11:18:26] I am doing some bacula management/cleanup, ignore any backup errors that could appear any time soon [11:40:02] volans: kormat: for python3.9 I guess wee need to upgrade tox https://phabricator.wikimedia.org/T274232 [11:40:10] oh and read the backlog now bah [11:40:23] and I don't even know whether we have python3.9 available [11:41:15] hashar: ahh. i missed that entry in the changelog for tox. [11:41:29] well at some point they made it to support any future python versio [11:41:34] but that seems to have regressed somehow [11:44:06] and apparently we have a mix of jobs using tox on Stretch and others on Buster bah :\ [11:44:17] anyway lunch time [11:44:28] will follow the task but Iam otherwise busy this afternoon [13:42:38] volans: huh. does py3- work for you in CI? it breaks for me because it's using python 3.5 🥀 [13:44:00] kormat: CI works, I think it's running on buster that one, let me check [13:45:14] yep 3.7.3 AFAICT [13:45:18] how can you tell/change that? [13:45:31] for CI? [13:45:34] yes [13:46:07] I'm using the tox-docker template [13:46:39] looking at the console in CI it's running [13:46:39] docker-registry.wikimedia.org/releng/tox-buster:0.1.0 [13:47:28] ahh! ok. mine says `tox-mysqld:0.5.0-s2` [13:47:47] which seems to be based on the base (squeeze) tox-docker image [13:48:02] the definitons are in integration/config [13:48:05] yes [15:06:47] <_joe_> kormat: switch the image you depend on to tox-buster [15:07:35] _joe_: i have a CR ready to do just that, now that i finally figured out how [15:07:53] <_joe_> lmk if you need a review [15:10:11] _joe_: cheers! added you to https://gerrit.wikimedia.org/r/c/integration/config/+/662992 [18:32:06] James_F: ping, just updated the patch with the changes you proposed, I'm trying to run the docker-pkg (first time), but let me know if I should change/adapt anything, thanks! [18:43:32] Is there a way to remove entries from the SAL? In this case I made a small typo in a log line which I corrected in the subsequent message, but ideally I'd be able to nuke the wrong message to restore balance to the universe [18:43:46] It looks like https://wikitech.wikimedia.org/wiki/Server_Admin_Log can be edited directly but wasn't sure if the SAL propagates to other places as well [18:45:46] ryankemper: yea, you can edit the wiki but it already posted to Twitter [18:45:55] i would still edit the wiki thogh [18:46:18] mutante: ack! and then does the searchable SAL website just pull from wiki as its source of truth? [18:46:20] did that for small typos before [18:47:22] ryankemper: I would think so.. at least [18:47:29] it is https://twitter.com/wikimedia_sal btw [18:49:12] mutante: Interestingly it's still showing up in the toolforge SAL, although it's possible there's some caching happening or something [18:49:42] (https://sal.toolforge.org/production?p=0&q=ryankemper&d= still has for example the line saying `Small typo in previous SAL log message` which I removed along with the previous message) [18:50:53] ryankemper: I don't know much about toolforge SAL, but i think that's completely separate from the prod SAL [18:50:56] "It reads data from an Elasticsearch cluster hosted in Toolforge that is populated by Stashbot from IRC !log messages sent in various freenode channels used by the Wikimedia projects. [18:51:19] i think -cloud knows more [18:52:01] ack, makes sense if it's just aggregating the stashbot log messages (wouldn't reflect any wiki changes) [18:55:34] I did not even realize it takes the input from the "prod SAL" and merges them [18:55:51] assumed it was like the separate SAL for each cloud VPS project [18:55:55] just for the tool project [18:56:12] where toolforge is just one project among others [19:01:48] The sal.toolforge.org tool is part of Stashbot and reads from an elasticsearch index in the Toolforge es cluster. The only connection between the wikitech SAL pages and the sal tool is that Stashbot puts data into both of them based on the !log irc messages. [19:02:20] sort of kind of explained at https://wikitech.wikimedia.org/wiki/Tool:Stashbot [19:38:45] simple (I think) puppet question, what does a leading `::` mean in puppet syntax? I imagine something to do with a relative "path" so-to-speak [19:39:25] For example `class role::elasticsearch::relforge` has the line `include ::profile::standard`. I see a `./modules/profile/manifests/standard.pp` file, so does `::` actually just mean "start at the base"? [19:39:49] ryankemper: yes [19:39:59] ack, thanks [19:40:49] ryankemper: the leading :: makes explict that you don't want a locally scoped thing to override. The lexical scope rules in Puppet (or at lest some version of Puppet) are pretty loose [19:48:59] ryankemper: similarly with variables it means "top-scope" as well. Like a global variable $::fqdn or $::site a opposed to local variables just for inside this class. sometimes jenkins/the style check will give you the "WARNING: top-scope variable being used without an explicit namespace" if it's confused by something like $fqdn or $site or you use a variable that just doesn't exist due to a typo.. [19:49:05] due to that [19:50:26] mutante: got it, so in the example you gave, the top-scope variable is the one actually being used but jenkins/linter is pointing out that you might be under the impression you're using a local variable? [19:52:27] let's say I am trying to use a normal local variable but make a typo.. so I refer to a variable that has never been set. then puppet doesn't find it locally and assumes it must be from another scope, resulting in that lint warning that a top-scope variable is being used but you did not tell it a namespace, like "::" [19:53:44] or it would say it if you use the global variable $fqdn but don't write it as $::fqdn which makes it explicit you are talking about the global one [21:26:24] FYI pattern matching (a.k.a. switch case) is coming to the language: https://mail.python.org/archives/list/python-dev@python.org/message/SQC2FTLFV5A7DV7RCEAR2I2IKJKGK7W3/ [21:26:44] for a practical example of how it will look like once implemented see https://www.python.org/dev/peps/pep-0636/ [21:48:18] looks pretty powerful [21:49:05] indeed! [21:49:46] one of the things I like about python is this whole border world it inhabits between OO and functional, where both idioms are possible [21:49:53] how do I get PCC to work on a patch that relies on hiera being set in the private repo? https://gerrit.wikimedia.org/r/c/operations/puppet/+/662807/ [21:50:22] "Error: Function lookup() did not find a value for the name 'docker_registry_ha::restricted_push_password'" https://puppet-compiler.wmflabs.org/compiler1001/27948/registry2001.codfw.wmnet/change.registry2001.codfw.wmnet.err [21:50:55] legoktm: there's a fake private repo that's actually public, that you have to merge a patch to with fake secrets [21:50:58] legoktm: labs-private [21:51:02] yeah that :) [21:51:20] such a horribly named repo these days :/ [21:51:34] thanks [21:51:55] cloud-public-but-dont-let-the-seeming-passwords-scare-you? [21:52:05] bd808: you didn't manage to rename this one! :-P [21:52:09] lol maybe better yes [21:52:35] nope, that is a "labs" that I never got too excited about breaking [21:52:42] cloud-private-that-is-actually-public-and-mocks-other-private-parts [21:53:00] mocking private parts is not a nice thing to do! [21:53:02] -and-some-of-the-mocks-are-not-mocks [21:53:26] language issue... *mocks-other-private-bits [21:54:25] cloud-puppet-faux-private-hack? [21:54:41] but seriously, does our gerrit or future gerrit replacement let us rename repos and alias the old URI without breaking things for legacy uses? [21:55:21] gerrit no, but I would actually be surprised if gitlab did not support transparent renaming [21:55:43] gitlab has proper renaming support [21:56:02] https://docs.gitlab.com/ee/user/project/repository/index.html#redirects-when-changing-repository-paths [21:56:24] bd808: the best named repo is OnionsPorFavor.git [21:56:49] oops, I never finished that project [21:56:54] heh [21:57:26] I'm going to add labs/private to https://codesearch.wmcloud.org/puppet/ [21:57:34] +1 legoktm [21:58:08] legoktm: where's the list of included repos? [21:58:38] click "advanced" and it shows you the list of repos per search profile [21:58:54] and https://gerrit.wikimedia.org/r/plugins/gitiles/labs/codesearch/+/refs/heads/master/write_config.py is what generates it [21:59:57] ack, and what's the rationale of adding or not a repo? [22:00:38] if someone finds it useful :) [22:00:39] spite :) [22:02:16] I could see the operations/cookbooks potentially useful where sometimes a host might be hardcoded (although discouraged) for some reason [22:11:12] ok, we can add it [22:11:27] also, not sure if someone linked https://www.youtube.com/watch?v=hgI0p1zf31k yet [22:11:58] <_joe_> legoktm: we have a strict policy against linking nerd songs or taylor swift in this team [22:12:03] <_joe_> :P [22:12:16] <_joe_> joking, of course Taylor Swift is ok [22:12:59] haha [22:37:04] we prefer if you link this though https://www.youtube.com/watch?v=dQw4w9WgXcQ [22:37:11] (good night :-P) [22:54:35] <_joe_> I know that url by memory apergos, sorry [22:55:03] <_joe_> it's both great and sad. [22:55:19] What about Rick Astley? [22:56:58] <_joe_> tabbycat: just that I can recognize the rickroll youtube url, it tells you how many times I've seen it :P [22:57:23] <_joe_> (also rick astley was *really* bad) [22:58:14] _joe_: tbh, I prefer Julio Iglesias :)