[00:00:25] it is possible to amend the mapping template to explicitly handle this field special though. [00:00:28] see properties: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/logstash/templates/logstash_7.0-1.json [00:01:36] note that amending this template will only affect indexes created after the amended template is deployed [00:02:35] shdubsh: so the default is `type": "text", "norms": "false"` for all properties of a json document? [00:02:54] yes [00:03:10] and this means creating foobar.keyword with terms/word boundaries, and foobar is not indexed by itself [00:03:27] I see host and level use `{ "type" : "keyword","index" : true, "doc_values" : true, "copy_to": "all" },` [00:03:34] but I'm not sure what the difference is with the default [00:04:25] foobar in that case is indexed as natural language, which means it breaks up the string on break characters and indexes each item broken out [00:06:08] host and level do not have keyword equivalents because they are treated as keyword type natively [00:08:53] hm.. I see [09:20:16] moritzm: thank you for the udplog update, but I had just tricked Majavah into learning how to build Debian packages! ;-) [09:21:48] I wasn't aware of that secret plan, sorry :-) mostly wanted to unblock the mwlog migration [09:22:30] but there's still a lot of potential to teach, my upload is just a rebuild with the boost build fix, he can still add a systemd unit, switch to dh, debsrc 3.0 etc :-) [09:23:33] indeed [09:37:29] how long does it take for removed icinga checks to disappear from the web u/i? [09:37:56] i ran puppet twice on the monitored node, and then twice on icinga [09:38:15] the puppet diff on icinga correctly showed the checks being removed [09:38:20] but the web u/i still shows them [09:38:34] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=db1138 - the "MariaDB Replica" 'services' [09:40:42] kormat: on which host did you run puppet? alert1001? [09:40:50] volans: icinga1001 [09:40:57] is.. that the wrong host now? [09:41:02] yes, since quite a while [09:41:10] 🤦‍♀️ [09:41:15] the HUGE motd didn't spoil it? [09:41:28] volans: well, see, i used cumin to run it, so no motd visible :P [09:41:37] icinga.wikimedia.org. 253 IN CNAME alert1001.wikimedia.org. [09:41:50] volans: dcaro: hi, since you are my python gurus we found a good one on CI yesterday. tox is installed with python2 which ends up being the default python used in tox venvs . Unless one goes with dirty tricks You might want to read my report at https://phabricator.wikimedia.org/T276384#6880938 :] [09:44:25] volans: amazing - running puppet on the right node actually changed things! :) [09:44:26] thanks <3 [09:44:34] it's magic! [09:52:48] just a heads-up folks: i'm currently rolling out monitoring changes for db hosts. it Shouldn't(*) cause any issues [09:52:58] (* This is the elukey definition of Shouldn't. YMMV) [09:58:34] ahahhaah [10:02:43] hashar: interesting! [10:08:34] volans: dcaro: I guess we will get tox in ci to be installed with pip3 / python3 to at least have a sane default basepython ;] [10:10:55] probably simpler than the other way around yep, though we should move out of python2 everywhere imo, it's officially unmaintained [10:19:27] how would CI for py2 projects work though? [10:36:22] please remind me which is the recommended way to install packages with puppet? [10:36:42] https://wikitech.wikimedia.org/wiki/Puppet_coding#Different_ways_to_install_apt_packages_with_puppet says package, but I think there is a few abstractions on top of that we use [10:41:56] jynus: ensure_packages T266479 [10:41:56] T266479: Puppet Proposal to remove require_package - https://phabricator.wikimedia.org/T266479 [10:42:39] thanks, volans, I will add it to the wiki [10:43:19] jbond42 is authoritative on that, so check with him :) [10:45:33] yes i would use ensure_packages and ideally put it at the top of the manifest unless there is a $reason not to [10:46:23] yeah, no problem, I am trying to express that on top of the package parameters discussion [10:46:45] jbond42: the question is if we should use always that or package is still a valid option in some cases [10:47:48] i would replace method 1 in the link above with `ensure_packages(['htop'])` [10:48:08] so if you just want to ensure =>present|installed then use ensure_packages [10:48:16] I am going to say to use ensure packages, and then leave the rest unchanged, noting those are low level options [10:48:51] ack that sounds good to me [10:54:46] this is enough for what I needed: https://wikitech.wikimedia.org/wiki/Puppet_coding#Different_ways_to_install_apt_packages_with_puppet [10:55:00] people can improve on later edits :-) [10:56:15] tlgtm thanks jynus (i also updated method 7 with the updated resource) [11:10:27] volans: I would like to do a cumin + spicerack release, now that a bunch of the fixes and new functionality for wmcs toolforge/vps cookbooks is merged, What's the process? [11:11:08] dcaro: I can take care of the spicerack one, for the cumin one I need to get something else finished before releasing it [11:11:40] volans: can I help anyhow? would be nice to learn how to do it [11:13:30] I run a script, it's a bit tailored to my setup, it requires an account on pypi and also I'm signing all the releases with my gpg [11:14:06] my pypi account user is dcaro [11:14:58] my gpg key fingerprint is '718083A2AC8B314FB4CE11714071C7E1D26269C3' [11:15:03] ^ is that enough? [11:20:57] for what? the release script is totally experimental and tailored to my local env [11:21:41] volans has made his own ball, and it only works at his home [11:22:21] it's more that I didn't had time to make it generic enough and uses a config file per repo that are not committd [11:22:27] because experimental [11:23:03] it also requires a script in the build host and one in the apt host to build the deb package and then reprepro it [11:23:31] but at least I use the same script for all the python packages I manage, because they share the same setup [11:27:05] volans: WTB cumin feature: if i start a cumin run, but forget to press 'y' for more than 5 minutes, page me [11:28:03] kormat: where should it get your number? [11:28:19] volans: that's easy. i'm number 1. you can hard-code that as a constant. [11:28:37] volans: I'll add a note in the spicerack roadmap to add/generalize that script, ack? [11:29:38] kormat: ack [11:30:21] I think it would be good if more than one person can release ;) [11:33:44] I agree in general, but is not just fixing the script. It's also a matter of who's responsabile for any follout of its deploy to production. What/how to test it when releasing it, coordinate with long-running cookbooks, etc. [11:35:03] that's ok, let's write that down [11:37:32] the only way of having more than one person being able to release, is well, sharing that responsibility and knowledge [11:39:41] yes and AFAIK SRE I/F is responsible for that area, but I'm not a manager [11:41:33] let's get one then :), I'll open a phab task to start a chat about it [11:51:29] volans: T276443 [11:51:29] T276443: Formalize and share the spicerack/cumin realease process - https://phabricator.wikimedia.org/T276443 [11:51:57] ack [12:26:20] hi all, operations/puppet CRs for https://gerrit.wikimedia.org/r/668386 and https://gerrit.wikimedia.org/r/668384 would be appreciated, thanks! [12:30:03] Majavah: just checking pcc [12:35:01] thanks jbond42! [12:35:34] Majavah: merged [12:55:38] https://gerrit.wikimedia.org/r/668392 is in the same category, would appreciate it getting merged too, thanks [12:56:53] Majavah: looking [12:59:34] Majavah: merged! [12:59:54] thanks, I think that's a Jessie VM ready to be removed [14:25:23] jbond42: is there a way to convert a boolean to an integer in puppet? [14:25:50] mm, maybe stdlib::bool2num [14:27:17] looks like we don't use this in operations/puppet at all, which isn't very reassuring :) [14:27:17] kormat: yes there is that (although it dosn;t have the stdlib:: prefix) you can also cast using Integer(Boolean) [14:27:57] ok cool, that seems simplest. thanks [14:28:25] i.e. it would be just bool2num($bool) but yes using Integer is not dependent on stdlib so i would go that way [14:28:43] not you can also cast to a Float(bool) [14:28:52] *note [14:30:06] 👍 [14:51:31] <_joe_> kormat: what are you trying to do? [14:51:51] _joe_: innocent ol' me? nothing! [14:52:08] <_joe_> no I mean why would you cast a boolean to integer in puppet [14:52:54] to replace doing $is_master? { true => 0, false => 1 } [14:52:59] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668444/ [14:56:22] <_joe_> kormat: ok, a case of adapting the type in puppet to what will go in the file [14:56:58] <_joe_> not your fault, but my personal preference would've been to keep read_only true/false there, and translate it in the template for the configuration [14:57:17] <_joe_> I was perplexed for this reason :) [14:57:20] _joe_: yeah. the literal bool values would also work, but we don't seem to do that in our mysql configs in general [14:57:26] ah i see [14:57:37] how _would_ you translate it in the template? [14:57:50] <_joe_> the templates are ruby [14:57:57] that's what i was afraid of [14:57:59] no ty [14:58:23] <_joe_> ahahah [14:59:19] <_joe_> read_only = <%= @is_master && 1 || 0 %>; [14:59:25] <_joe_> so elegant, so clear [14:59:36] <_joe_> combining the best of perl and malbolge [14:59:42] also wrong, best of all worlds! [15:00:03] (mariadb::misc is the only profile that uses the logic of is_master == !read_only) [15:00:22] because misc sections are even more special than i had realised before today [15:00:43] <_joe_> ahaha [15:50:59] which etcd version is the production cluster using? [15:54:17] <_joe_> Majavah: interesting question! [15:54:28] <_joe_> most clusters are on etcd v3 [15:54:35] <_joe_> but one is still on v2 [15:54:55] <_joe_> also for mediawiki we still use the v2 datastore even on version 3 [15:55:49] _joe_: Majavah volunteered to upgrade a bunch of Jessie things in deployment-prep! :) [15:56:10] <_joe_> <3 [15:56:38] <_joe_> Majavah: so I would suggest you speak with me and jayme, maybe tomorrow? regarding etcd [15:56:43] _joe_: I have deployment-etcd-01 on beta cluster, which is a Jessie VM using etcd v2.2, and I'm trying to figure out what exactly uses it and is it safe to upgrade to latest buster 3.2, which still requires at least two hops [15:56:48] <_joe_> as I would like to try and move to buster [15:56:55] <_joe_> yes, it should be safe [15:57:02] <_joe_> also we are about to do so in production too [15:57:18] <_joe_> so we are eager to assist with it [15:57:22] sure, why not [15:57:28] <_joe_> but I have a meeting in minutes, sorry [16:12:32] jbond42, dcaro: I've released spicerack to APT and PyPi but as there are long-running cookbooks running on cumin2001 and lot of people using cumin1001 right now I'd ping you back when will be a good time to test the new release on cumin2001. Most likely will be Monday at this point [16:12:52] \o/ thanks! [16:13:05] thanks :) [16:14:02] fwiw this happened couple of hours ago but then got meeting and was hoping to deploy it now, but it' snot a good time [17:27:03] Majavah: _joe_: did we "agree" on talking tomorrow? :-) [17:30:07] jayme: I'm happy to talk to you if it's helpful, but we didn't agree on any specifics [17:31:25] ah, okay. I just wanted to catch up as I was in a meeting as well. Would like to leave now, so if you're fine with that we can talk tomorrow [17:31:40] what TZ are you in? [17:34:48] something pretty compatible to j.oe and me it seems, I'll try to ping you tomorrow o/ [17:36:51] UTC+2 [17:37:31] but yes, feel free to ping tomorrow [17:38:31] sweet, will do! [19:09:07] mutante: could you restart apache/php on doc1001 one more time? A few minor syncs have happened and it's still stale unfortunately. still not sure why this keeps hapening :) [19:28:06] Krinkle: m.utante just posted that he's out today but I can restart em -- is it both apache and php? any preferred order? I don't have context on this yet [19:28:29] rzl: either way, yep! [19:32:43] Krinkle: {{done}} [19:33:58] thx, that worked [19:34:32] rad [20:00:50] https://lwn.net/Articles/844224/ is a really great primer on low-level threading/smp/memory-model/locks/lockless things, the whole series will probably be awesome, but it's not all written yet [22:50:06] mutante: o/ just made this change to the http proxy docs. let me know if that makes sense to you: https://wikitech.wikimedia.org/w/index.php?title=HTTP_proxy&type=revision&diff=1901919&oldid=1885661