[06:17:45] Maybe it's just my inability to cope with phab but I got sick of always having to scroll over all those gerritbot comments. So utilizing my superior javascript skills I came up with a userscript to make them "inline" without loosing information https://phabricator.wikimedia.org/P11638 [06:20:43] jayme: do you have a screenshot of what it looks like? [06:21:35] Now that phab shows all the related CR at the top of the task, I'm not even sure we should keep those gerrit comments [06:23:44] XioNoX: added one to the "phist" ...lol [06:24:28] that looks nice! [06:25:52] jayme: didn't even know I needed this, thanks! :D [06:26:58] :-) yw [06:31:19] jayme: that looks great [06:33:53] I can't make it work :( [06:34:21] jayme: It'll be great for my pc but how could it be installed on a mobile? [06:34:49] Or should we just have something similar added as a phab plugin behind a feature flag? [06:35:50] XioNoX: I'm using it with greasemonkey in Firefox. Did not do anything special...I guess [06:36:05] that's what I have too [06:36:30] RhinosF1: No idea about mobile. I don't think that mobile browsers have addon support like that [06:36:49] I don't remember ever finding one [06:37:58] XioNoX: hm...strange. I'm on FF 77, Greasemonkey 4.9 fwiw ...maybe throw a "console.log()" in there to see if it even gets executed [06:39:12] nevermind, it works :) [06:41:19] great [06:41:25] jayme: might be useful to do the same for the !log comments, there is a lot of redundant information like "Mentioned in SAL (#wikimedia-operations) [2020-02-11T20:08:08Z]" etc.. [06:42:01] Sure. You have a task at hand that has SAL comments? [06:43:46] jayme: https://phabricator.wikimedia.org/T255098 has one [06:44:04] https://phabricator.wikimedia.org/p/Stashbot/ will find you more [06:45:13] thanks. Will take a look when I have a spare minute [06:54:38] thx! [07:20:13] * volans jayme [07:20:33] nice, but you know that now you're cursed... you showed JS skills ;) [07:21:50] yeah..."skills" :D ... I can do text replacement and basic flow control. Bet you do as well ;-) [07:21:55] I'm wondering if there is a better way to achieve the same result within phab itself [07:22:06] jayme: no in JS :-P [07:22:09] *not in [07:22:15] haha [07:28:56] volans: mutante might know, we have a few phab addons but it might be best behind a feature flag [07:32:00] I think it would require significant tinkering with Phab and with the Gerrit integration [07:36:40] Majavah: it's just js though converting a comment. Why not just create a feature flag that does the exact same thing that jayme's script does? [07:55:04] FYI, I'll disable puppet in eqiad for ~ 5 mins in a bit to unblock the migration of the VM to a new node [08:36:59] hurm. i submitted a CR to reenable notifications for a host (db1088), ran puppet on icinga1001, but icinga still shows notifications are disabled for that host [08:37:39] kormat: have you run it multiple times on icinga and the host itself? [08:38:03] oh, crud. i keep forgetting about that indirect path. [08:41:15] tsh tsh...you already forgot our monday chat? :-P [08:41:26] willfully [08:42:36] marostegui: that fixed it, :sigh:. thanks [08:43:03] kormat: good! [09:30:57] volans: what's wrong with this query? `sudo cumin "P:mariadb::mysql_role%role = master and R:profile::mariadb::section = s1"` [09:31:23] each individual clause works, and they both contain db1083, but when combined i get the empty set [09:31:39] two resources same puppetdb query, you need to use the global grammar P{} and P{} [09:32:03] ahh. thanks! [09:32:19] puppetdb allows to query a single resource per query, I should add some check to cumin to fail witha good message if that's detectable [09:32:28] that'd be nice :) [09:37:19] I didn't know screen -x will match any screen name even if you don't specify the whole name of the screen [09:37:21] that's useful [09:37:52] * kormat waves byobu at marostegui, without much hope [09:38:15] * marostegui wonders who kormat is [10:48:13] I was checking an unrelated commons issue and just saw this: https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=22&fullscreen&orgId=1&from=now-30d&to=now-1m&var-DC=eqiad&var-prometheus=eqiad%20prometheus%2Fops [10:48:19] is this something I should worry about? [10:48:43] (swift server errors) [11:58:37] I'll take a look shortly [12:22:23] jynus: seems benign, those can be incremented by bad client payload too [12:22:33] ok [12:22:41] so one thing that led me to that [12:22:59] is that we saw a large increase on the image table (file metadata) [12:23:02] in the last week [12:23:11] and the upload of originals seem to match that [12:23:31] there could be some kind of intense batch upload in the last week [12:24:42] https://grafana.wikimedia.org/d/000000034/media?panelId=24&fullscreen&orgId=1&refresh=5m [12:25:11] nothing worrying but it was noticed by monitoring of db growth [12:25:39] *nod* yeah I wouldn't be surprised if it was e.g. GLAM activity [12:41:46] XioNoX: here you go https://phabricator.wikimedia.org/P11638 [12:52:17] jayme: looks great [12:52:22] * RhinosF1 just installed ot [12:52:26] it* [13:33:26] I am running a big and complex schema change on s6 master, if you notice something strange on frwiki, jawiki or ruwiki, please ping me [14:03:41] sukhe, jayme: https://wikitech.wikimedia.org/wiki/User:Filippo_Giunchedi/Pontoon#Master_setup is now updated (and godog's pontoon role is merged into production) to make it more... possible to test puppet changes [14:03:55] let me know if the docs suck, let godog know if he sucks [14:04:54] kormat: who do we complain to about you, though? [14:05:01] marostegui: do we expect impact on the replicas? just wondering if I need to keep an eye out [14:05:04] cdanis: marostegui [14:05:10] 🤔 [14:05:13] apergos: nope, we shouldn't [14:05:17] cool [14:05:27] haha kormat [14:05:54] kormat: nice [14:06:13] cdanis: I already have that email from you pending to read, what was the subject: I am tired of kormat or something similar? I can take a look later if you like [14:07:06] marostegui: it's okay, there's not much to read, the content of the email is just https://www.youtube.com/watch?v=GwQW3KW3DCc [14:07:15] kormat: thank you! I will have a look at it [14:07:19] nice work [14:07:53] is it me or is it getting hot in here 🔥🔥🔥 [14:08:45] cdanis: hahahahahah [19:40:42] mutante: I'm guessing the exec block here https://gerrit.wikimedia.org/g/operations/puppet/+/production/modules/systemd/manifests/sysuser.pp#35 needs to be moved out into another file that is just `include`d when a systemd::sysuser is created? [19:43:50] cdanis: hmm, i don't know. usually a duplicate declaration error gives 2 different places but in this case it's both the same line in the same file [19:45:03] mutante: right, because the Exec is part of the expansion of the class declaration [20:28:37] mutante: https://gerrit.wikimedia.org/r/c/operations/puppet/+/607609 should fix it [20:35:53] been staring at an issue for a while and cant work out the problem, its not urgent but second set of eyes appreciated. [20:35:56] https://gerrit.wikimedia.org/r/c/operations/puppet/+/607543/16..17/modules/profile/manifests/wikidough.pp [20:36:08] PS 16 works https://puppet-compiler.wmflabs.org/compiler1001/23459/ [20:36:26] PS17 dosn't https://puppet-compiler.wmflabs.org/compiler1002/23460/ [20:36:53] to me the change should make no difference but gussing i must be missing something [20:39:07] jbond42: sorry that this is driving you crazy as well :( [20:39:41] no worries :) [20:41:09] * jbond42 PS17 also fails on compiler1001 [20:43:17] jbond42: ... *what* [20:43:40] jbond42: namespace collision [20:43:49] between the private (labs/private) repo stuff [20:43:55] included in include passwords::wikidough::dnsdist [20:44:05] volans: the hiera lookup is not the thing that changes [20:44:13] just the local paramter variable name [20:44:29] yes, but that paramter is the same name of profile::wikidough::dnsdist_webserver [20:44:32] in the labs/private [20:44:46] look at https://puppet-compiler.wmflabs.org/compiler1002/23460/malmok.wikimedia.org/change.malmok.wikimedia.org.err [20:44:49] points to line 9 [20:45:08] oh so its doing automatic paramter lookup but without the merge stratagy [20:45:21] which is overriding the actual lookup call? [20:46:19] don't ask me to explain why puppet does what it does :D [20:47:15] its intresting though it means that in a lot of places we are allready doing automatic paramter lookups and the lookup call is being ignored. [20:47:35] I think is more subtle here [20:47:41] which i think makes sense as hiera is prefered to the in module default [20:47:44] you include passwords::wikidough::dnsdist [20:47:57] on line 9 [20:48:10] that is a class, has nothing to do with the dnsdist_webserver name [20:48:14] and has no collisions with that [20:48:58] ok, PEBCAK on my side for line 9 [20:49:08] it's line 9 of the role, not the profile :/ [20:50:32] so it's complaining that whatever resolves for "dnsdist_webserver", has not the expected struct with password and api_key [20:51:02] yes the reason its complaining is because its not useing th 'merge' =>hash stratagy [20:51:25] but the value in hiera, assuming is getting that one from labs/private, has password and api_key [20:51:29] it only occurs when the variable is named dnsdist_webserver which is the only time automatic paramter lookups is used [20:51:38] (for some more context, this was working fine earlier, so it certainly has something to do with the names unless there was other puppet stuff that changed in the meantime) [20:51:47] and by default that uses a merge stratagy of unique [20:53:16] jbond42: but howcome the 2 keys disappear? [20:53:28] it should merge the values in https://gerrit.wikimedia.org/r/c/operations/puppet/+/607543/17/hieradata/role/common/wikidough.yaml with those in the private [20:53:36] the full hash has 4 values [20:53:46] https://puppet-compiler.wmflabs.org/compiler1001/23459/malmok.wikimedia.org/index.html [20:54:07] ahhh got it, I didn't saw the change to that name too [20:54:28] yes sorry i noticed it was missing from the 16..17 diff [20:54:49] no prob [20:55:02] anyway i think this has helped me see what the issue is [20:55:09] o thanks :) [20:55:13] *so [20:55:22] glad to have unintentionally helped :D [20:56:10] :) sukhe im not sure of the best fix yet i dont think you can specify a merge stratagy in side hieraa with our version but will check tomorrow [20:56:37] jbond42: that's fine, thanks for looking. it's a cosmetic fix so nothing critical but I appreciate the time you have put in this! [20:57:05] np [20:57:05] thanks to volans for clearing the doubt :) [20:57:12] ahahah [20:57:43] when I made the first commit and pcc failed, I for one was convinced there was a typo somewhere [21:06:49] * sukhe out [21:17:29] cdanis: thanks! i will take a look. actually had a power outage here [21:18:45] uff