[00:30:00] Anyone ran into issues with `enable-puppet-agent -e` refusing to re-enable puppet? It seems that the reason given isn't the same, but I can't spot any differences...not sure if there's some weird ascii stuff going on [00:30:17] https://www.irccloud.com/pastebin/0DVuS0ok/ [00:30:32] https://www.irccloud.com/pastebin/JyQEwGIm/ [00:30:52] ryankemper: https://phabricator.wikimedia.org/T272539 [00:31:27] cdanis: that explains it! thanks, will use the force flag [00:32:45] yeah, we should get it fixed this week :) [04:59:08] step 1: draw a circle, step 2: draw the rest of the owl. :D [04:59:09] https://phabricator.wikimedia.org/T273023#6779134 [07:21:09] <_joe_> Krinkle: SREs and CSS don't go well together [07:21:28] <_joe_> actually I think what pushed me father towards the backend was my deep hatred of css and the box model [07:23:37] hehe, it's a funny little language for sure. [07:25:40] _joe_: I'm mainly just happy this proposal didn't "win". https://en.wikipedia.org/wiki/JavaScript_Style_Sheets [07:25:58] It would have combined your two favourite frontend technologies [07:26:11] <_joe_> ahah indeed [07:26:46] <_joe_> I mean, I can stand javascript per-se, it's using it in browsers that's the real pain, unless you don't care about older devices [07:27:38] I don't think coding for new browsers is notably easier, I'd say that about CSS though, where things have more signigicantly changed with new primitives. [07:28:09] <_joe_> Krinkle: they keep introducing more incompatibilities/incongruence between js interpreters? [07:28:17] JS is hard unless you don't care about laggy intermittent partially-failing and slow unencouraging experiences :P [07:28:33] <_joe_> yeah that too [07:28:43] e.g. latest JS in a modern browser, has all of the above if not done well. [07:29:10] <_joe_> and btw, I completely stopped keeping myself up to date with modern js frameworks [07:29:33] <_joe_> given there is a new one every two years everyone swears is the definitive one, I just lost hope to be able to follow [07:29:50] there are new APIs introduced in the HTML and DOM indeed, such as Service Workers, Nav Timing, CORS, COOP, COEP, CSP etc, but that's more HTML/frontend in general than JS is a language. [07:30:07] <_joe_> yeah js as a language is very simple [07:30:24] as* [07:30:39] <_joe_> it's the whole "js to drive an HTML application" part that is extremely complex [07:30:41] the newer syntax additions are pretty ignorable imho. [07:31:54] right, the problem is that the browser allows one to mix logic and UI in the same thread. it makes it appealing to a lesser experienced or diciplined developer to manage their state in the DOM, or to read/write intemittently instead separating the two. [07:32:27] When you separate the two, it becomes similar to a native app with render updates sent to the DOM and being painted the next frame. [08:59:20] kormat: yes CI can do timed / nightly builds but we just dont have any ;D [08:59:40] kormat: I guess or most high traffic repositories it is not that helpful (puppet.git , mediawiki/core etc) [09:00:10] for mediawiki extensions/skins we have some bots updating translations or dependencies and catch up with build errors this way [09:00:49] for the rest well, when a new change is send after a while, it might randomly breaks for reasons unrelated to the change [09:00:54] and folks fix it up [09:02:27] and for wmfmariadbpy, the dependencies in test-requirements.txt do not have any version limiations and the build is thus subject to a breakage whenever one of the requirement get updated by upstream [09:03:14] guess you can add some upper bound limit or even freeze them [09:51:42] Krinkle: thanks for the patches as jo.e said "SREs and CSS don't go well together" so any help with those templates gets much <3 from me <3 <3 [09:55:33] <_joe_> jbond42: as proved by the puppet compiler output :D [09:55:55] :D [09:56:48] <_joe_> tbf klaxon has a nice interface, but chris spent a lot of effort (and used a pretty nice css framework too) [09:59:13] _joe_: you raise a good point chris (in doing so) has now promoted himeself to the SRE CSS SME ;) [10:06:25] * volans is super happy to surrender that title from the time of debmonitor [10:07:22] <_joe_> jbond42: +1 [10:10:14] hashar: i could lock down the versions, but it doesn't feel like the right approach. there's 62 dependencies, all of which would need to be specified in order to prevent this kind of thing from happening. that many version specs is unmaintainable [10:12:11] <_joe_> kormat: I'm not sure I follow [10:14:19] which bit? :) [10:21:07] <_joe_> you want nightly builds, but you can't use frozen requirements? [10:21:45] <_joe_> my approach usually is that I create frozen requirements, check that using those all tests pass, and update those from time to time [10:23:14] <_joe_> vgutierrez: I am about to create a phab component to group all envoy activity under. If you have related tasks, please add the #envoy tag to them [10:23:25] ack [10:24:29] _joe_: can you show me an example project? [10:25:18] <_joe_> kormat: IIRC docker-pkg does, or maybe that's just the deployment repo, I don't recall [10:25:30] <_joe_> but that's why the deployment repo is what we actually build [10:26:59] <_joe_> https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/docker-pkg/deploy/+/refs/heads/master [10:27:16] kormat: the thing is that if you depends on "flake8" your test suite is subject to arbitrary failures whenever flake8 release a new version and some new lint nitpicks [10:27:54] _joe_: i'm not familiar with 'deploy' repos. is there a doc somewhere? [10:28:03] kormat: nothing you can do beside fixing the flake8 version. Typically to a minor version ( flake8==3.7.* ) or an explicit one [10:28:32] <_joe_> kormat: disregard it, it's tangential for you, those are used for scap3 deployments [10:28:38] ah, i see [10:28:51] <_joe_> my point is, it's possible to declare your requirements this way [10:29:05] <_joe_> but also, I think cumin takes the approach for tests that hashar was talking about [10:29:08] _joe_: docker-pkg/deploy has frozen-requirements.txt, docker-pkg itself does not [10:29:17] kormat: for composer / npm repositories most come from a package lockfile that pins all dependencies. For other repositories, we accept that the test build can magically fail due to a version change and folks then send a differnet change that address the issue [10:29:26] <_joe_> kormat: yeah because that never gets deployed to production [10:30:19] <_joe_> kormat: the alternative is to just pin the test dependencies, as hashar was mentioning, see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/cumin/+/refs/heads/master/setup.py [10:30:54] <_joe_> if you don't want to thaw all of your deps using frozen requirements, but just avoid new linting behaviour to make ci fail [10:31:05] _joe_: that looks like it pins direct test dependencies, but not indirect ones. which means it can still fail [10:31:16] but yeah, it presumably helps [10:31:38] <_joe_> kormat: that's the testing library author's failure, and yes, it can happen ofc [10:32:17] hmm. this can get complicated when you're supporting multiple python versions [10:32:33] i guess maybe you pin to the versions supported by the oldest python version, and hope that's ok [10:33:52] hashar: i'm well aware that unpinned packages can get updates which cause failures. that's exactly why i was asking for a way to detect this has happened :) [10:33:53] or use different set of requirements for each python versions :D [10:34:00] * kormat winces [10:34:05] for the dependencies of dependencies, I guess that can be frozen via a pip freeze maybe [10:34:10] which should capture the whole set of deps [10:35:17] so yeah essentially, pin the dependencies :] [11:08:33] akosiaris, I will be doing the heze/helium cleanup this week- I will be adding you to reviewers, but let me know if to tune spam level up or down [11:08:44] (I know you are bussy with other stuff) [11:08:52] *busy [12:37:30] jynus: Sure, feel free to add me. Thanks for the heads up [12:38:26] it is not like it is super-important, but we had delayed it for long AND I want to consider your input, as will be very valuable [13:32:57] re: the pip dependency versioning discussion earlier, i've come to the conclusion that all of the cures are worse than the disease, so i'm just going to leave it as-is. [14:39:58] FYI, I just updated https://wikitech.wikimedia.org/wiki/Analytics/Data_access, with some clarifying docs, hopefully that will help when users make SRE access request tickets, and also for SRE to fulfil them. [14:51:34] _joe_: yes they do :P [14:55:35] <_joe_> cdanis: we know you do [14:56:02] I mean, I still cheated and used bulma.io (which is quite nice) and wrote perhaps 6 lines of CSS myself [14:56:14] but such frameworks exist, and they're good and helpful :) [14:56:46] <_joe_> oh sure :) [16:01:43] ryankemper: holy crap someone reads my templates??!@!? ref T273009 ;D [16:01:44] T273009: Update decommission phab template to remove unneeded steps - https://phabricator.wikimedia.org/T273009 [16:01:57] fixing now, thx for catching! [16:08:23] :P [16:08:24] thanks! [21:31:29] mutante: on VMs we're getting... [21:31:31] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Base::Debdeploy]: parameter 'filter_services' entry 'default_value' expects an Array value, got Hash (file: /etc/puppet/modules/profile/manifests/base.pp, line: 130, column: 5) on node abogott-cinderdev.testlabs.eqiad1.wikimedia.cloud [21:31:38] I wonder if that's a puppet version thing? [21:32:13] it seems to work on a metal host [21:37:15] andrewbogott: sigh.. compiled on * but still does not cover this [21:37:33] looking [21:38:00] It's not clear to me what's wrong, I see {default_value => {}} working elsewhere [21:49:24] weird, I changed it from {} to [] and got this diff: [21:49:28] https://www.irccloud.com/pastebin/uV2ZS9vV/ [21:49:40] so some how default_value is getting /inside/ the hash [21:50:37] if I remove the two middle args "Hash, 'hash'," then it works as I would expect [21:50:41] not that they seem wrong to me... [21:50:49] mutante, some possible light ^^ [21:54:15] ok, the issue is that we're mixing two idioms, the new args-in-hash method and the old positional-args method [21:55:58] I'll make a patch [21:58:31] andrewbogott: please wait, I am already uploading one, let's just revert the one line [21:58:50] I had another idea but I don't know how to compile on VPS [21:59:13] it would have made sense if this was a hiera() line before, but it wasn't [21:59:17] I believe this is the correct solution [21:59:17] https://gerrit.wikimedia.org/r/c/operations/puppet/+/659087 [21:59:34] but reverting is also ok with me :) [22:00:13] note how it was already lookup() before my change though [22:00:28] yes. Docs here: https://puppet.com/docs/puppet/7.3/hiera_automatic.html [22:00:52] "Note: Giving a hash of options containing default_value at the same time as giving a lambda means that the lambda wins." [22:01:30] It's pretty weird that lookup() takes totally different arg sets and tries to guess what kind of usage based on the first few [22:04:51] andrewbogott: alright, how could we test it? I am compiling your change on a prod host right now [22:05:12] I think my patch is missing a ) [22:05:52] I'm testing on abogott-puppetclient.testlabs/abogott-puppetmaster.testlabs [22:05:56] it does not break on mwdebug1003 [22:06:00] but I'm less convinced about my fix now... [22:07:10] ok, it works for my test, I just made a c/p error [22:07:45] cool. how did you do that? using your own puppetmaster and locally editing files? [22:07:49] yeah [22:07:52] i see [22:09:27] andrewbogott: well, it was noop on a random prod host and the issue somehow only affects VMs and if it looks good to you [22:09:35] https://puppet-compiler.wmflabs.org/compiler1002/27709/ [22:09:43] there's pcc on a VM w/stretch [22:09:59] so yeah, I think we should merge it [22:10:01] want me to? [22:10:22] yes please [22:10:37] interesting how you can use compiler :o [22:10:42] but cool [22:11:11] if there is an issue we'll just revert that one line to how it was before [22:11:17] that's a very old VM that's still registered under .wmflabs [22:11:23] and then i'll get john to review that problem [22:13:10] andrewbogott: works for me on a cloud VPS :) [22:13:18] cool [22:13:19] thanks for the fix [22:13:23] sorry about interrupting your dinner [22:14:23] no no.. sorry for breaking it. I tried to compile it on everything but this wasn't expected of course [22:14:34] touching "base" is just .. well... touchy [22:14:43] soon we are done with this :) [22:14:53] it's the last few cases now