[07:22:56] 10Traffic, 06Labs, 06Operations, 07Puppet, 07Technical-Debt: Convert all of our site.pp/roles to the role/profile paradigm - https://phabricator.wikimedia.org/T159412#3097172 (10Joe) @Ciencia_Al_Poder care to explain why did you remove the "easy" tag? In general, I'd like to see a comment explaining act... [10:43:14] 10Traffic, 06Labs, 06Operations, 07Puppet, 07Technical-Debt: Convert all of our site.pp/roles to the role/profile paradigm - https://phabricator.wikimedia.org/T159412#3066827 (10Ciencia_Al_Poder) @Joe I found the easy tag is not suitable/applicable to this task, that's why I removed it Feel free to poke... [10:47:03] 10Traffic, 06Labs, 06Operations, 07Puppet, 07Technical-Debt: Convert all of our site.pp/roles to the role/profile paradigm - https://phabricator.wikimedia.org/T159412#3066827 (10MoritzMuehlenhoff) I'd say let's add a few gerrit links of style conversions which have already landed to the task description,... [11:47:11] 10Traffic, 06Operations, 10Pybal: pybal stops logging - https://phabricator.wikimedia.org/T160405#3097560 (10ema) [11:47:24] 10Traffic, 06Operations, 10Pybal: pybal stops logging - https://phabricator.wikimedia.org/T160405#3097572 (10ema) p:05Triage>03Normal [11:57:11] 07HTTPS, 10Traffic, 06Operations, 10Pywikibot-core, and 2 others: Prepare pywikibot for http -> https switch in entity uri - https://phabricator.wikimedia.org/T159956#3097590 (10ema) [12:54:50] 10netops, 06Operations: Audit and cleanup border-in ACL on core routers - https://phabricator.wikimedia.org/T160055#3087244 (10faidon) The first part is true and I couldn't figure out why -- I know why I removed the IPv6 multicast ranges (basically: NDP), but the question about 224/4 still remains. It may have... [15:40:30] 10Traffic, 06Operations, 10Pybal, 13Patch-For-Review: pybal stops logging - https://phabricator.wikimedia.org/T160405#3098227 (10ema) On machines where pybal doesn't log anything, the pybal process is indeed not doing any meaningful writes to standard out/err: ``` $ sudo strace -e trace=write -e write=1,2... [15:58:25] bblack, ema: Linux 4.9 seems fine on cp1008 so far, how about migrating e.g. one cache/maps host next? [16:03:27] moritzm: yeah should be fine, maybe go ahead and do one of maps + misc and let that sit a few days? [16:03:53] e.g. cp4001 (misc) + cp4011 (maps) [16:04:24] +1 [16:04:37] ok, I'll take care of that tomorrow [16:04:54] thanks! :) [16:05:31] bblack: I'm trying to figure out what's going on with pybal's logging T160405, do you remember seeing this behavior in the past? [16:05:31] T160405: pybal stops logging - https://phabricator.wikimedia.org/T160405 [16:08:44] ema: no idea off the top of my head [16:08:59] ema: is a python thread crashing maybe, or just rsyslog / systemd issues? [16:10:06] bblack: I don't think rsyslog/systemd are involved, the process is not writing to fd 1 or 2 [16:10:20] might be some thread crashing but there are no logs heh [16:41:01] <_joe_> ema: this is the first time i see this [16:41:11] <_joe_> what's the last thing logged, btw? [16:42:20] _joe_: normal stuff, see neodymium.eqiad.wmnet:~ema/last-pybal-logs [16:42:30] <_joe_> ok [16:43:22] _joe_: not that I have reasons to believe that it would solve this problem but should we perhaps upgrade to twisted 16.2.0 (in backports)? [16:44:01] <_joe_> ema: test it thoroughly [16:44:09] of course [16:44:19] <_joe_> we had some (minor) issues when upgrading a major twisted version [16:44:34] <_joe_> I'm pretty sure it's ok now [16:44:43] I've upgraded pybal-test2001 and pybal didn't cry, that seems like a good start :) [16:44:52] <_joe_> oh for sure [16:45:07] <_joe_> be sure that pybal does all sorts of monitoring there [16:45:16] right [16:45:17] <_joe_> we had issues with some of the monitors too [16:46:03] _joe_: later on we could also port the logging code to twisted.logger [16:46:33] <_joe_> it is using twisted's logging under the hood IIRC [16:56:37] 10Traffic, 06Operations, 10Pybal: Upgrade twisted on load balancers to 16.2.0 - https://phabricator.wikimedia.org/T160433#3098548 (10ema) [16:56:56] 10Traffic, 06Operations, 10Pybal: Upgrade twisted on load balancers to 16.2.0 - https://phabricator.wikimedia.org/T160433#3098571 (10ema) p:05Triage>03Normal [17:35:50] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Operations, and 2 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3098754 (10AndyRussG) >>! In T154954#3095397, @ema wrote: > Uhm, 3K purges 50-200 times a day seem too many for banner u... [17:47:25] 10Traffic, 10DNS, 06Operations, 10Parsoid, 13Patch-For-Review: Separate subdomain for parsoid visual diff test service on ruthenium - https://phabricator.wikimedia.org/T159995#3098777 (10Dzahn) @ema does that seem ok to you? review of patches would be great if you have the time