[10:11:09] <_joe_> FWIW, I don't always agree with shellcheck [10:11:21] <_joe_> sometimes its reccommendations are pretty useless [10:21:48] _joe_: in the ps i have added only the error states cause a -1. not and warnign are still displayed but also happy to suppress completly. [10:22:22] s/not/note/ [10:23:54] <_joe_> error states are fine [10:24:00] <_joe_> that was what I wanted to suggest :) [10:24:16] <_joe_> the warnings are sometimes questionable [10:25:01] great and yes i agree re warnings [10:31:34] <_joe_> jbond42: I forgot volans didn't implement it, sorry [10:31:41] <_joe_> he would've made notices vote -2 [10:32:19] warnings -4 and errors -8 ofc [10:32:20] :D lol [10:33:16] <_joe_> jbond42: he never disappoints [10:35:37] :) [10:37:22] <_joe_> volans: but what about the lenght of the script? I think we need an algorhythm [10:40:11] are we using black yet? [10:40:44] mark: for bash? :D [10:40:54] why are we using bash :P [10:41:09] yeah, what's wrong with plain /bin/sh. ;) [10:41:24] there you go kormat: now you have to convert everything! [10:41:27] :-P [10:42:04] careful what you wish for [10:44:29] volans: speaking of, i'll be sending you a CR later today ;) [10:45:12] kormat: are you reserving a slot in the CR queue in advance? :-P [10:47:07] 80% of a slot [10:47:29] lol, slot are atomic :D [10:47:41] see how much of a pain it is [10:48:36] <_joe_> ahah [13:39:10] for a second I've read "sloth are atomic", which seemed like a fairly unusual statement [14:00:24] 🦥 ⚛ [14:01:23] 👌 [14:07:35] i had to put our current team staffing at 30.8 [14:07:41] for annual planning [14:09:47] mark: put 154/5 :-P [14:09:48] * volans hides [14:10:21] thankfully he runs at only 80% speed [14:10:40] or, maybe, I can just catch up with him on friday [14:24:37] vgutierrez@puppetmaster1001:~$ sudo -i puppet-merge [14:24:37] /usr/local/bin/puppet-merge: line 17: [: too many arguments [14:24:37] /usr/local/bin/puppet-merge: line 22: [: puppetmaster1001.eqiad.wmnet: integer expression expected [14:24:43] jbond42: is that expected? [14:25:03] vgutierrez: no did you get that just now? [14:25:06] yes [14:25:11] I got the expected diff output [14:25:18] but I'm not sure if I should proceed merging :) [14:25:39] no let me take a look at it [14:25:45] ack, releasing the lock.. [14:25:48] jbond42: need doublequotes on those vars on line 17 [14:25:48] please cancle your merge [14:25:54] now that the arrays are joined [14:26:19] and on line 22 you need != instead of -ne [14:26:19] cdanis thanks [14:27:42] jbond42: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/603487 [14:28:12] thanks yuo happy for me to merge that [14:28:31] never mind i see yuo are [14:29:29] vgutierrez: merging yours along with cdanis fix [14:29:36] thx [14:33:48] I am about to self-merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/601460 as without it you can specify Systemd::Servicenames that exist in production; if people want to talk about the policy question implicit in it let's do that as a followup [14:35:43] cdanis: *"you can't specify" ? [14:35:56] yes [14:36:06] look at the patch, example one that doesn't work in the patch description ;) [14:36:09] php7.2-fpm.service [14:36:51] cdanis: you wrote " you can specify ... names that exist in production" above [14:37:00] oh, sorry, yes [14:37:02] cannot [14:37:03] i'm asking did you mean to write that you cannot currently specify those.. ok :) [15:05:02] godog: VMs in the 'swift' project seem pretty broken… is that project still a going concern? [15:07:22] (pinging you because the only VMs there were made by you) [15:21:07] chasemp: Did I break orch-buster.orch.eqiad.wmflabs or was it broken all along? (I'm reviewing things I can't reach with cumin and that's one of them) [15:21:52] andrewbogott: I just spun that up I think last week? it shouldn't be broken but also I hadn't gotten a chance to do much with it if it is then it can be killed no worries [15:22:11] thanks, I'll delete and you can recreate at your convenience [15:22:11] thanks [15:27:05] andrewbogott: what do you mean with broken in this context ? [15:27:25] godog: a great many puppet failures, also not responding to cumin [15:27:42] I'm working on the cumin thing now but wondering if it's worth it [15:28:15] andrewbogott: re: cumin I think that'd be https://phabricator.wikimedia.org/T254041 (?) [15:28:25] I'll check the list of VMs now [15:29:38] andrewbogott: yeah I'll nuke those two VMs in the swift project [15:29:46] that's easy! Thank you :) [15:30:28] andrewbogott: np, though for cumin also instances in the monitoring project will have the same problem [15:30:51] godog: right now I can access them, so the keys must be there [15:31:31] oh ok, sounds good [15:32:10] * andrewbogott is down to only one VM that I can't manage with cumin [16:52:53] https://github.com/RadeonOpenCompute/ROCm#heterogeneous-compute-interface-for-portability [16:52:59] really nice --^ [17:10:30] godog: speaking of 'monitoring,' any chance you could do a rebase and resolve diffs on your puppetmaster? Everything using pontoon-puppet-01 is drifting pretty far afield [17:10:39] (to the extent that the puppetmaster there is mostly nonfunctional now) [17:10:57] alternatively I can just move you over to a working branch now and let you rebase later when you need your changes again [23:09:56] So, I'm performing an elasticsearch rolling upgrade/restart, and I've identified the code block that's blowing up [23:10:08] In order to debug I'd like to go with the classic approach of inserting some print statements [23:10:27] Is there an easy way to run an amended version of a given cookbook? Is there precedent here? (I imagine there must be) [23:11:21] ryankemper: do you really need to run an amended version of the cookbook? It seems to me that you just need to check if the prometheus query wors [23:11:24] *works [23:11:41] Ah, that does seem much simpler :) [23:11:44] I agree, that is what I need to check [23:12:02] in order to do that you can get spicerack loaded into a python repl [23:12:04] and play with it [23:12:08] follow https://etherpad.wikimedia.org/p/volans-tmp [23:12:34] s is a spicerack instance with dry_run=True, sreal is one with dry_run=False [23:13:15] Thanks! I'll start with trying out https://wikitech.wikimedia.org/wiki/Prometheus#Access_Prometheus_web_interface and seeing if I can use the web interface to run a test query [23:13:47] so s.prometheus() is what prometheus is in that piece of code you pasted [23:14:06] and you can test s.prometheus().query(...) [23:15:04] I'm about to go to bed ryankemper, I hope this allow you to find out the issue [23:15:49] volans: Thanks, that's really helpful. Instead of manually running the prom query I'll open up the repl following your steps [23:17:20] yw :) [23:19:11] Looks like the prometheus query is returning just `[]` [23:23:39] Changing the query to use a dc of `codfw` instead of `eqiad` makes the query work. That's interesting, because the cookbook is hardcoded to use `eqiad` which implies it should absolutely be working [23:25:53] So to recap, a query of `kafka_burrow_partition_lag{ exported_cluster="main-codfw", group="change-prop-cirrusSearchElasticaWrite", topic="codfw.mediawiki.job.cirrusSearchElasticaWrite"} for active_dc codfw` executed on `prometheus.svc.codfw.wmnet` returns a proper response [23:26:27] A query of `query is kafka_burrow_partition_lag{ exported_cluster="main-eqiad", group="change-prop-cirrusSearchElasticaWrite", topic="eqiad.mediawiki.job.cirrusSearchElasticaWrite"} for active_dc eqiad` on `prometheus.svc.eqiad.wmnet` does return a 200 response but the result is the empty list `[]`