[08:56:01] hello hello [08:56:24] * kormat retreats [08:56:37] ahahaha [08:56:49] I am going to open a task like https://phabricator.wikimedia.org/T248093 and ack all the warnigns for mcrouter [08:56:52] didn't find another one [09:04:57] done [09:36:18] <_joe_> elukey: thanks [14:49:47] puppeteers: i have a list of items. i want to set a variable to true if f(item) is true for any item in the list. is there a good way of doing this? [14:51:03] kormat: map() and any() exist :) [14:51:34] also reduce [14:51:36] https://puppet.com/docs/puppet/5.5/function.html#any [14:51:42] I think any() is exactly what you want? [14:52:35] cdanis: seems like it. now i'm just trying to understand how to lambda [14:54:28] maybe this will work: https://phabricator.wikimedia.org/P14549 [14:55:14] did you want .each or .any ? [14:55:25] oops, right, any. [14:55:45] fixed [14:56:00] I'm also not sure if any is a method on the array itself (it might be) [14:56:11] or if it is just a global function that takes an iterable as an argument [14:56:18] the docs show both [14:56:37] the examples for any() are unfortunate in that they use some syntax i'm unfamiliar with [14:56:49] `(lookup('somekey', Hash)` [14:56:59] s/^\(// [16:50:16] FYI I will be taking over sretest1002 and test its reimage [16:50:38] let me know if you're using it and I should hold off (both last and w show no activity) [17:32:24] elukey, deployment-eventlog05.deployment-prep.eqiad.wmflabs has been failing puppet runs for 5 months or so; it depends on some classes and templates that don't exist anymore. [17:32:35] Shall I delete the VM or would you like a ticket to work on fixing things? [17:32:48] (pinging you because you created the VM, sorry if it's someone else's baby now) [17:36:52] andrewbogott: hi! if you don't mind opening a task I'd be grateful! [17:37:26] sure thing [17:39:31] T276122 [17:39:31] T276122: broken puppet on deployment-eventlog05.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T276122 [19:42:20] o/ anyone with conftool knowledge about? [19:42:50] bits and pieces, what's up? [19:43:04] as a user, sure [19:43:22] we're looking at re-pointing the phatality deploy at the new cluster [19:44:13] currently, it's looking at the old hosts `confctl select 'cluster=logstash,service=kibana' get` [19:45:12] would the right way to go be to update dsh.yaml to set `cluster: kibana7 and service: kibana7`? [19:46:41] are you adding new hosts to an existing cluster [19:46:48] or creating an entirely new cluster [19:47:34] re-pointing the phatality deploy to the correct set of hosts [19:47:37] based on "the old hosts" and the select command [19:47:58] shdubsh: I believe that's correct for repointing scap [19:48:04] it looks like `confctl select 'cluster=kibana7,service=kibana7' get` gives us the right set of hosts [19:48:08] (this is more of a scap question than a conftool question) [19:49:38] cdanis: thanks, sounds right to me as well [19:50:43] ah, so it's already a separate cluster and also server and new service and cluster have the same name. [19:50:55] yes, it seems right to edit dsh.yaml like that [19:52:04] puppet should edit the scap config on deployment server, indeed scap config [19:52:22] we are also about to put the deployment server into maintenance though [19:55:22] right on, thanks all :) [19:55:29] shdubsh: I am talking to Mukunda anyways in a few minutes, can merge it, but it would not be a good time to deploy [19:55:44] that alright? are we just unblocking him? [19:56:15] mutante: yes, this is for him [19:58:00] shdubsh: we can do it right after the switch or so [19:58:15] :) [19:58:23] great! :) [19:58:45] mutante: that patch just changes teh dsh_targets so that phatality can deploy to the right hosts [19:59:07] I'll take care of deploying it but I'll need to have an sre around in case it goes horribly wrong [19:59:40] (previous deployments crashed logstash's elasticsearch instance and required a manual restart, thanks to OOM killer) [20:00:14] given that right this moment we have our scheduled window to switch deployment servers.. [20:00:36] yeah phatality deployment can wait ;) [20:00:40] could we do that first and avoid adding additional things [20:00:52] alright [20:01:01] yeah of course I wasn't in a rush to deploy phatality, especially given that it's somewhat iffy [21:33:32] shdubsh: btw, do you know if/when Kibana 5>7 will be applied to beta cluster? [21:33:53] unsure whether and for how long to on-board and support tooling on both [21:35:21] Krinkle: good question. There has been some discussion about it, but no plans made as of yet. Might be worth pinging lmata for prioritization. [21:35:30] given the long list of UX landmines and traps including mutually exclusive workflows between the versions, it adds significant confusion/complexity when onboarding staff into logstash practices I found on the two recnet onboarding sessions I had (more later this/next quarter) [21:36:53] urgh, yeah that could get confusing for sure :( [21:38:59] Krinkle: any of that tooling should we be looking at as well? [21:40:41] tooling involves browser extensions for assisting in task creation, and video tutorials and general docs and best practices for how to create dashboard, filter bubbles, and which buttons not to press for risk of crashing the browser and/or producing queries that don't work. [21:41:11] also the whole "thing.keyword" that's new is still something I haven't quite figured out what to make of [21:42:28] previously "thing" did I think a word-boundary term search so it effectively supported a prefix search if using quotes and making sure the prefix is a whole word (e.g. thing:"Error from /foo/bar" found "Error from /foo/bar/baz.php line 123 for user 456") [21:42:37] and "thing.raw" was previously an all-or-nothing index [21:42:44] I don't know where the keyword fits in yet [21:43:40] I can explain the .keyword suffix. It is what .raw used to be. It's the field captured in the index as keyword type. In the legacy cluster, .raw was the suffix we gave to the same data type. [21:46:15] I'm not aware of any browser extensions. If that's something we supported at one time, it long predates me. [21:57:25] shdubsh: if 'we' means SRE then no, you (plural) did not developer or maintain browser extensions interacting with Kibana or Phabricator. [21:58:07] Phatality was the Logstash plugin developed by RelEng, but due to inability to deploy updates to a 20 line JS files without OOMing logstash updates were effectively forbidden and the API changed in Kibana 7 so we'll need to develop it afresh [21:58:14] probably as a browser extension client-side. [21:59:06] until then we're back to a few dozen clicks and copying-pasting manouvres for each prod error task [22:00:03] Krinkle: ah, we are aware of Phatality at least. Hopefully an updated plugin will be ready to deploy soon. [22:00:35] WRT the OOMs, we hope those are a remnant of the past. [22:00:55] ok, do releng know that? [22:01:45] yes, working with t.wentyafterfour [22:01:57] awesome [22:13:55] help, is someone around to review/deploy a core router change? [22:14:10] found out we need this: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/667718/1/templates/cr/firewall.conf [22:14:43] to allow new deployment servers in scap ACLs [22:15:34] right now appservers can scap pull from the new deploy server but f.e. from stat1007 you can't curl deploy1002 but you can curl deploy1001 [22:16:04] mutante: it's missing it's v6 counterpart [22:16:09] grep deploy1001 on that file [22:16:46] thanks volans, yes, amending [22:20:38] mutante: +1ed, do you know how to deploy it? [22:22:29] volans: reading the docs, first: homer "cr*" diff ? [22:23:29] first merge and run puppet on the cumin host where you would run the homer command [22:23:45] ack, merging.thx [22:24:53] puppet pulled from the homer repo. done [22:26:12] running the "diff" [22:26:17] ack [22:28:39] it's going through 13 devices for cr*. most are empty diff but not all. got the diff at the end, looks good to me [22:29:03] if you know already to which CR device will be applied run the commit just for those [22:29:43] i didnt know before but the diff told me it's just cr1-eqiad and c2-eqiad [22:30:25] [cumin1001:~] $ homer 'cr2-eqiad.wikimedia.org' commit [22:30:47] you can do 'cr*-eqiad*' [22:30:51] ah, I am supposed to add a message as well [22:30:52] also you need a commit message [22:31:06] I'm wondering why just eqiad and not codfw [22:31:50] [cumin1001:~] $ homer 'cr*eqiad.wikimedia.org' commit "adding new deployment servers to scap firewall T265963" [22:31:52] T265963: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 [22:31:54] deploying [22:32:01] k [22:32:43] volans: maybe because it's for analytics vlan and that is not a thing in codfw [22:32:55] ahh right if it's only for analytics [22:32:55] issue happened on stat1007 but not on mw* [22:33:02] normal mw deploy was fine [22:33:12] but not when others tried to deploy stat1007 [22:33:38] I had to type yes for each device. done now [22:33:54] yep, the diff might be different for each device [22:34:01] i'll ask them to try again, sec [22:35:20] volans: actually, already pretty confidemt it worked because now i can: [stat1007:~] $ curl deploy1002.eqiad.wmnet [22:35:31] and before I couldn't [22:35:32] thank you [22:48:57] no prob, anytime :) [22:50:38] oh man. I am glad this is done. switching deployment server has only happened like twice, and tin was a couple years ago. a bunch of moving parts there [22:50:45] all seems good though now