[00:25:28] <mutante>	 MediaWiki's CommonSettings.php is almost like another bash.org/quip collection
[00:25:33] <mutante>	  656         # Because I hate having to find print drivers -- tomasz
[00:25:33] <mutante>	  657         $wgFileExtensions[] = 'ppd';
[00:26:04] <Reedy>	 heh
[00:26:06] <mutante>	  636         # delphine made me do it!!!!! --brion
[00:26:06] <mutante>	  637         $wgFileExtensions[] = 'eps';
[00:28:11] <mutante>	 "wmv" - "Temporary for office work :P"
[03:03:23] <vgutierrez>	 https://www.irccloud.com/pastebin/C4gZeLM2/
[03:03:38] <vgutierrez>	 jbond42: ^^compiler1003 is having issues fetching changes
[08:36:37] <vgutierrez>	 godog, ema: we already got the ats-tls status panel on the frontend traffic grafana dashboard... to provide the availability one we need https://gerrit.wikimedia.org/r/c/operations/puppet/+/548954 merged, let me know if it's looking good
[09:16:10] <godog>	 vgutierrez: for sure! will do
[09:26:29] <jbond42>	 vgutierrez: sorry thats seems to be some testing i neglected to clean up.  should be fixed now
[09:30:46] <vgutierrez>	 np, thx
[11:13:57] <arturo>	 vgutierrez bblack I just added you as reviewer in this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/549058
[11:14:18] <arturo>	 we got generated a new SSL cert by globalsign which apparently uses a new CA we didn't have in puppet
[11:14:29] <arturo>	 see https://phabricator.wikimedia.org/T237066#5639813
[15:43:06] <shdubsh>	 jbond42: I think there's a problem with this CS: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/549094/
[15:43:27] <shdubsh>	 2019-11-06 15:42:05.985343 ERROR puppetlabs.facter - error while resolving custom fact "puppet_config": undefined local variable or method `sections' for #<Facter::Util::Resolution:0x00558dac6f1460>
[15:43:29] <shdubsh>	 Did you mean?  section
[15:44:30] <jbond42>	 shdubsh: thanks fixing now
[15:44:39] <cdanis>	 rlazarus: ^
[15:44:57] <rlazarus>	 aha thanks
[15:52:49] <shdubsh>	 jbond42: that did the trick, thanks!
[15:54:41] <jbond42>	 np thanks, just need to wait for everything to pick up the change.  looking at the cron spam it seems everything was effected so better to wait it out then try and force it and risk overloading puppet
[19:59:18] <mutante>	 a change in the rsync module broke puppet on the gerrit replica
[20:00:07] <mutante>	 it's about the allowed_hosts. what used to be a hostname now "cannot be converted to Numeric"
[20:07:05] <cdanis>	 yes mutante
[20:07:07] <cdanis>	 I'm working on it
[20:07:19] <cdanis>	 I made the assumption that $hosts_allow was used everywhere as an array
[20:07:19] <mutante>	 i just saw your change. ack!
[20:07:21] <cdanis>	 this is not the case
[20:08:45] <mutante>	 yea, i remember looking up hosts_allow in rsyncd and it can be IPv4 or IPv6 or hostname or hostname pattern
[20:08:59] <mutante>	 re: the type
[20:20:22] <cdanis>	 mutante: okay, think I have the change working right now for both cases as they exist in production, and for the spec tests
[20:20:34] <cdanis>	 a large PCC run looks good so far
[20:21:10] <mutante>	 cdanis: it's just 2 hosts and i can run them to confirm after merge
[20:21:22] <mutante>	 sounds good!
[20:21:31] <cdanis>	 yeah, it certainly fixes those two hosts
[20:21:34] <mutante>	 thanks for adding that
[20:25:38] <XioNoX>	 "I made the assumption that [insert logical thing there]"     "this is not the case"
[20:25:38] <mutante>	 +1 (we could use Variant[Ip_address, Fqdn,.. but we would still need to allow String for the patterns and didn't want to nitpick more :)
[20:26:15] <mutante>	 makes sense that you always have to allow localhost now for stunnel... *nod*
[20:30:03] <cdanis>	 yeah...
[20:30:10] <cdanis>	 it's even more complicated since we allow so much
[20:30:32] <cdanis>	 because if we allowed just FQDNs, it would be very simple to translate those into stunnel doing certificate subject name verification
[20:30:53] <cdanis>	 so now stunnel-wrapping is kind of a 'downgrade' for those
[20:31:04] <mutante>	 we dont have to support everything that is allowed in hosts_allow of rsyncd
[20:31:09] <cdanis>	 (all it checks is that the cert presented by the other side is signed by the CA)
[20:34:30] <mutante>	 i am only using fqdn i think
[20:34:39] <cdanis>	 good to know :)
[20:35:37] <cdanis>	 gerrit2001 fixed
[20:36:32] <mutante>	 changing a few from IP to host names should be ok, but i dont know how many. looks like profile::mediawiki::scap_proxy  is one
[20:36:39] <mutante>	 cool, thanks!
[20:37:20] <cdanis>	 yeah I'm trying to extract that from puppetdb now
[20:38:22] <mutante>	 moscovium also fixed. puppetboard:  0 nodes failed
[20:41:05] <cdanis>	 hm I guess we aren't really 'losing' anything because we have also ferm rules
[20:42:18] <cdanis>	 ❌cdanis@puppetdb1001.eqiad.wmnet ~/resources 🕞🍵 jq '.[] | select(.type == "Rsync::Server::Module") | {"node": .certname, "title": .title, "hosts_allow":.parameters.hosts_allow}' *
[20:42:37] <cdanis>	 jq is good stuff (once you half-wrap your head around the syntax)
[20:43:24] <bblack>	 I keep trying to love jq, but I do get confuzzled by the syntax often
[20:43:48] <cdanis>	 it is always a lot of trial and error for me
[20:56:01] <cdanis>	 looking at https://phabricator.wikimedia.org/P9544 most of them are either unspecified or are FQDNs, so that's nice
[20:57:19] <mutante>	 yep, looks like it's mostly scap/deploy related where it isn't
[23:03:34] <mutante>	 so weird how a single appserver keeps sending cronspam every day at the same time. exactly once per 24 hours but nothing  in /var/spool/cron/crontabs matches that. it's removed like from all the other servers it looks. yet mw2225 is the special case
[23:03:41] <mutante>	 ghost cron
[23:06:04] <mutante>	 ! systemd timer .. guess this one was used to test switching from cron
[23:08:58] <mutante>	 except it also doesn't match any of the timers..