[00:12:09] the mind boggles at the logic at work there. [00:14:12] #httpd says "can't expect much from Debian maintainers" and that they should have RTFMed and there are historic reasons for _having an option called IncludeOptional that is identical to Include but also not always optional_.. :p [00:14:36] i just said if i wanted it to be non-optional i'd use Include .. shrug [00:14:37] huzzah [00:15:05] seems like a slightly bizarre response [00:15:15] but reifying historical accidents isn't uncomming I guess. [00:16:49] right. well, yea. the Debian bashing is also common in #httpd heh [00:18:08] debian has a weirdly negative reputation in some circles. [00:18:37] the fix for this is to always install modsecurity-crs in addition to libapache2-mod-security2 and never the second one by itself [00:19:20] then the mystery left for me will just be how the appservers got to where they are [00:19:43] * mutante touches mwmaint but nothing mw* [07:51:35] akosiaris: ready for the m2 failover in 10 minutes? [07:51:49] marostegui: yes [07:51:57] \o/ [08:31:06] druid upgrade completed, indexations are recovering now [08:32:03] netflow realtime ingestion is working, but it will take one/two hours to not have any hole in data visibile in turnilo (batch jobs need to fill the realtime data lost) [08:32:08] XioNoX: --^ [08:32:28] elukey: great! [08:32:40] elukey: any exciting new feature I can understand? [08:34:08] XioNoX: the update list is gigantic (we moved from 0.12.3 to 0.19), I mostly focused on the security part etc.. They simplified a lot of things, and added more features like new SQL operators etc.. [08:34:24] ok :) [08:34:56] also improved caching, UIs based on druid should be faster now [08:35:33] but new features might be useful for https://phabricator.wikimedia.org/T254332 [08:35:42] we'll see (Nuria scheduled it for this quarter) [08:36:00] great :) [12:35:25] how do nrpe scripts get run? i see a workaround in one of our scripts (https://github.com/wikimedia/puppet/commit/1da9fed5276413043a6c44c2888a56c143d7f4e0), and i'm wondering if the simple fix i have is sufficient [12:59:24] <_joe_> kormat: IIRC with a clean environment [12:59:44] <_joe_> that's why os.getlogin() doesn't work when run via nrpe [13:03:51] hmm. os.getlogin says "Return the name of the user logged in on the controlling terminal of the process.". there presumably isn't a controlling terminal in this case [13:05:39] `pwd.getpwuid(os.getuid())[0]` would probably work better [13:07:24] cdanis: yeah. [13:07:49] os.getlogin seems dumb, and worse, the docs say maybe use getpass.getuser() instead, which just TRUSTS THE LOGNAME ENV VAR. like wtf. [13:08:01] anyway kormat they're invoked by nrped [13:08:11] if you're feeling especially foolhardy you can go look at the source [13:08:33] i looked at the config; it runs as nagios/nagios, and it doesn't have a sudo config, [13:08:46] so the scripts themselves must also run as nagios/nagios [13:08:54] which makes things sufficiently simple. [13:08:57] I believe so, yes [13:10:21] also kormat you've reminded me of this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/499028 which I'll link just for the horror [13:12:02] cdanis: .... nooooo .... [13:12:36] and akosiaris signed off on it [13:14:56] and volans! [13:15:43] i can't tell which is worse. volans giving a +1 with no comment, or akosiaris giving a +1 with "on, nice find!". [13:17:03] do you want to know how I diagnosed this kormat ? [13:17:10] i'm torn [13:17:13] this is really the best part [13:17:19] but i assume you're going to share anyway :P [13:17:22] I was trying to figure out *why* nscad was getting stuck [13:17:38] so, a few times when it did get stuck, I attached strace to the running process [13:17:45] which would always un-stick it! [13:18:01] oh god [13:18:26] which, well, the semantics of PTRACE_ATTACH: [13:18:28] Since attaching sends SIGSTOP and the tracer usually suppresses it, [13:18:30] this may cause a stray EINTR return from the currently executing sys‐ [13:18:32] tem call in the tracee, as described in the "Signal injection and [13:18:34] suppression" section. [13:20:07] cdanis: i'm impressed. and appalled. so the usual, i guess [13:21:52] kormat: I think that a-ha moment is actually thanks to elly; I was ranting about this to her and, well, she's implemented her own debugger because of course she has [13:22:05] but of course [13:42:26] what did I do? [13:42:56] * volans catching on various backlogs at once [13:43:48] volans: You Know What You Did [13:43:58] no I don't :D [13:44:42] implausible deniability [15:41:07] herron: hey, for the new host of fermium, do you think it would be able to have multiple disks so drop them once we migrate to mailman3 and use a proper database instead? idea by Reedy [16:06:15] Amir1: it's a VM, so we're pretty flexible in adding/ditching diskspace [16:07:21] hey Amir1, also I think we'd migrate over to the new version using a new host, and that being the case wouldn't need to worry about mailman3 provisions on the mailman2 system [16:08:01] moritzm: oh I don't know much about ganeti but at least in open stack I'm not able to add/remove disk space, I need to throw it away and make a new one [16:08:25] herron: That sounds good to me! [16:08:54] I'm close to the get exim4 working on the cloud setup (haven't puppetized much yet) [16:09:07] nice! [16:11:14] once it's done, it'd be great to migrate a couple of fully public lists (if possible) to see how the size on these VMs change [16:11:53] so we get a ballpark idea how big the DB on mailman3 in production will be when we migrate everything [16:12:36] makes sense, fwiw we're at ~200G currently on the filesystem [16:15:27] also, it's not too hard to puppetize, lots of packages and couple of files in /etc/mailman3/ but that's all. The exim4 config will be quite a pain, I basically gave up and kept only ACL of the old mailman (which is the most important part) [16:15:56] will update you once I got it fully puppetized [16:33:31] exim config and pain in the same sentence? sounds about right haha :) [16:33:32] sounds good! [16:38:11] I literally had nightmares about exim4 configs