[00:22:09] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: donation page format error - https://phabricator.wikimedia.org/T162300#3170209 (demon) >>! In T162300#3169967, @Ejegg wrote: > Hey @demon, sorry for the delay. Someone on fr-tech sh... [13:18:16] Fundraising-Backlog: tweak process-control email template to be inbox friendlier - https://phabricator.wikimedia.org/T162687#3171261 (Jgreen) [14:58:10] Fundraising-Backlog: Help Ellery understand recurring - https://phabricator.wikimedia.org/T92461#3171603 (DStrine) Open>declined [15:06:04] Fundraising-Backlog: Estimate the proportion of people who send us complaints vs are affected by an issue - https://phabricator.wikimedia.org/T120722#1859982 (DStrine) In my time managing problems like this, Ive never seen a CS/QA/Engineering team actually accomplish this model. You can't really predict an... [16:00:36] Caminando por el campo una vaca me encontré [16:00:53] Como no tenía nombre, Banner Sequence le pondré [16:01:00] Oh Banner Sequence, la vaca eres tú [16:01:10] Si te jalo de la cola, ¿qué dices tú? [16:02:01] AndyRussG: hehe sort of reminds me of Hal's rendition of Daisy Bell [16:02:08] heheh [16:02:22] It goes to the tune of "O Susannah". [16:02:45] After that, Banner Sequence is supposed to sing it, and put someone else's name in place [16:14:56] fundraising-tech-ops, Operations, ops-ulsfo: rack/setup frbackup2001 - https://phabricator.wikimedia.org/T162469#3172075 (Papaul) [16:28:35] Fundraising Sprint Ghostbusting , Fundraising-Backlog, FR-Ingenico, MediaWiki-extensions-DonationInterface, and 2 others: iDEAL cancel returns donor to Thank You page - https://phabricator.wikimedia.org/T131904#3172145 (DStrine) p:High>Normal [16:28:54] Anyone around to babysit donatewiki as we move back to wmf.19 this morning? [16:29:00] cf T162300 [16:29:00] T162300: donation page format error - https://phabricator.wikimedia.org/T162300 [16:29:05] fr-tech: ^ [16:29:09] Fundraising Sprint Ghostbusting , Fundraising-Backlog, FR-Ingenico, MediaWiki-extensions-DonationInterface, and 2 others: iDEAL cancel returns donor to Thank You page - https://phabricator.wikimedia.org/T131904#2182480 (DStrine) We should recheck if this happens after the new integration. [16:29:29] RainbowSprinkles: For sure--what time are you thinking? [16:29:37] Basically now :p [16:30:07] I'm around. I can keep an eye on the logs and such. [16:30:24] I start the train about 1030, so preferably before then :) [16:30:51] Fundraising Sprint Ghostbusting , Fundraising-Backlog, FR-Ingenico, MediaWiki-extensions-DonationInterface, and 2 others: iDEAL cancel returns donor to Thank You page - https://phabricator.wikimedia.org/T131904#3172172 (Sjoerddebruin) We do have a few people saying (on the OTRS helpdesk) that the... [16:31:26] RainbowSprinkles: right on, thanks for the warning. I'll be in -operations and can poke at the wiki as you go. [16:31:36] Ok awesome, let's do it! [16:32:22] pretty sure ejegg fixed everything relevant yesterday [16:33:27] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Recurring-Donations, Spike: Spike: Errors while charging recurring GlobalCollect might be handled wrong - https://phabricator.wikimedia.org/T105857#3172180 (DStrine) p:High>Normal [16:34:26] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Recurring-Donations, Spike: Spike: Errors while charging recurring GlobalCollect might be handled wrong - https://phabricator.wikimedia.org/T105857#1453031 (DStrine) dropping this to normal since we haven't... [16:38:57] awight: Ok, moving back here because noise and we gotta figure this out. [16:39:05] This is clearly doing the same thing on wmf.18 and wmf.19 [16:39:08] So what gives? :) [16:39:17] I'm out of the looop [16:39:35] The bug we saw was that there were and meta tags in the templates and system messages [16:39:48] which were blocked by the security upgrade and caused major rendering fail [16:39:54] Is there something else going on? [16:40:03] What are we fixing? [16:40:19] RainbowSprinkles: was it broken again? with the html tags? [16:40:48] Yes [16:40:52] But then I rolled back to wmf.18 [16:40:54] And I still see tags [16:40:59] link? [16:41:03] i ran some edit.php script yesterday for ejegg that was supposed to regenerate the affected templates [16:41:05] I'm in https://donate.wikimedia.org/wiki/Special:FundraiserLandingPage?uselang=en&country=NL&1 [16:41:05] https://donate.wikimedia.org/wiki/Special:FundraiserLandingPage?uselang=en&country=NL [16:41:11] wat tha [16:41:21] O_o [16:41:21] Ahhhhh, f'ing cache [16:41:28] The &1 makes sense now [16:41:33] We gotta bust the cache here [16:41:33] Wait what is the difference between [16:41:34] uh [16:41:35] argh [16:41:46] Ok ok, this makes total sense to me now [16:42:01] (well, sorta. special pages ain't usually cached at the edge) [16:42:04] But clearly that's what's up [16:42:08] where is the broken one coming from? Your experimental .19 deploy? [16:42:18] They're all on wmf.18 again, I rolled back [16:42:25] roger that [16:42:53] So, in theory we can roll forward to wmf.19 again, and try busting the cache? Also: curious why a special page is getting cached... [16:43:01] so was .19 causing brokenness, or is that cached from yesterday [16:43:09] +1 on trying rolling forward again [16:43:22] and I donno, let me search the source for cache headers [16:44:17] $out->setSquidMaxage( $wgFundraiserLandingPageMaxAge ); [16:44:30] Yeahhhhh, ok. So it is cached in varnish [16:44:45] that is a hard cache to bust right? [16:44:51] So what we can do here is roll forward, then use the purge script thingie to kill it? [16:44:59] let's do it [16:45:00] purgeList or w/e it's called [16:45:16] i donno where that needs to be run from, but willing to learn [16:45:26] ejegg might have a better answer about whether this is for sure fixed or not... [16:45:40] as in the templates, and whatever it was that had to happen to them :) [16:45:57] He seemed happy with the fixin', but +1 it would be nice to ask [16:46:25] awight: It's just a general maintenance script, so tin/terbium is fine :) [16:46:30] XenoRyet: I'm doing some organization in phab and I see this task https://phabricator.wikimedia.org/T89016 I'm wondering if you are fixing that now? [16:46:47] looking [16:46:53] RainbowSprinkles: oh dang--apparently there's a ruthless backdoor to allow banning: https://wikitech.wikimedia.org/wiki/Varnish#How_to_execute_a_ban_across_a_cluster [16:47:03] but I think purge is good enough for us [16:47:32] A ban isn't what we want--that'd be "don't cache this crap ever" iirc [16:47:43] If you're setting a maxage, I'm assuming you do want it cached ;-) [16:48:00] I think it's only previous to the ban. But agreed that we should use the more mundane tool. [16:48:17] dstrine: That might have some overlap with the work I'm doing now, but it doesn't sound like exactly the same problem. [16:48:31] ok I'll leave it open [16:49:30] dstrine: Yea, looks like what I'm doing will get at the point ejegg was making, but there's a lot of other stuff going on in that ticket. [16:50:01] that's cool, just thought I'd try to tidy up if I could [16:50:26] RainbowSprinkles: for your review: mwscript purgeList.php donatewiki ; then I specify the URL on stdin [16:50:31] Yea, good times. I'll leave a comment on that ticket when I get this fix working, since it is partially relevant. [16:51:40] MBeat: ccogdill: fyi some donatewiki landing pages are temporarily hosed [16:51:54] RainbowSprinkles: I'm going for it. [16:51:56] ty awight [16:52:09] RainbowSprinkles: the purge command works. [16:52:22] MBeat: ccogdill: k and just like that, the problem might be fixed. [16:52:30] cool [16:52:52] RainbowSprinkles: we're purged. re-roll .19 at your leisure [16:53:39] Ok, one minute [16:53:51] cwd|brb: hey, you want to get us to python 3 so we have something exciting to test in the next .deb? https://gerrit.wikimedia.org/r/#/c/346708/ [16:55:20] Ok, back to wmf.19, and looking good! [16:55:58] https://donate.wikimedia.org/wiki/Special:FundraiserLandingPage?uselang=en&country=NL&100 [16:56:01] Not so sure [16:56:14] I'll purge the main page again to say I did... [16:56:27] RainbowSprinkles: borken. [16:56:49] Gah, what? [16:56:52] It looked fine to me [16:56:56] it was cached. [16:56:57] try again [16:57:05] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 8 others: [Spike] investigate contribution tracking data (was Engage import fail... - https://phabricator.wikimedia.org/T146295#2656564 [16:57:12] Son of a bitch [16:58:29] RainbowSprinkles: It's possible that we fixed the occurrences on paymentswiki but not donatewiki. [16:58:40] Are you able to run the train without us? [16:58:44] I *can* [16:58:52] We'll just be running 3 branches instead of 2 [16:58:59] I'll try to get in there and fix the templates [16:59:05] just... in case I don't make it in time. [16:59:08] that *socks* [16:59:12] Ok, reverting back to wmf.18 again [17:00:19] fr-tech: Peace, n.: [17:00:19] In international affairs, a period of cheating between two [17:00:19] periods of fighting. [17:00:19] -- Ambrose Bierce, "The Devil's Dictionary" [17:00:19] -- discuss. [17:00:21] 245 pages with "html" [17:00:23] oh boy [17:00:35] cwd|brb: You said there was a shortcut using edit.php ? [17:01:01] argh no the-wub [17:02:08] awight: yes i will merge and build python3 today [17:02:39] fundraising-tech-ops, Operations, ops-ulsfo, Patch-For-Review: rack/setup frbackup2001 - https://phabricator.wikimedia.org/T162469#3172311 (Papaul) [17:03:00] ok only 19 pages might actually have html tags. [17:03:08] in the MediaWiki namespace. [17:03:27] awight: RainbowSprinkles - there is a wikitext file i can provide [17:03:36] i ran it on one of the payments servers [17:03:49] oh good [17:03:54] Do you know how we were patching these pages? [17:04:01] https://donate.wikimedia.org/w/index.php?search=%3Chtml%3E&title=Special%3ASearch&profile=advanced&fulltext=1&ns8=1 [17:04:18] the command was `php maintenance/edit.php -m Template:quote < /tmp/quote.wikitext` [17:04:24] legoktm: <3 cool, that's the list I was working through [17:04:57] Not even sure how I would fix something like https://donate.wikimedia.org/wiki/MediaWiki:Resources/validation.js [17:05:09] fundraising-tech-ops, Operations, ops-ulsfo, Patch-For-Review: rack/setup frbackup2001 - https://phabricator.wikimedia.org/T162469#3172320 (Papaul) a:Papaul>Jgreen @Jgreen Complete let me know if you have any questions. [17:05:37] I think the key part is that it's being included via a system message [17:05:55] so {{MediaWiki:Resources/validation.js}} by itself is OK [17:06:15] but if the thing htat it's in is being outputted by wfMessage(...)->parse() then it's not OK [17:06:20] harrr [17:06:42] k reading FundraiserLandingPage "code" [17:08:35] cwd: Can I see quote.wikitext? [17:09:37] legoktm: I don't see any usage like that: https://github.com/wikimedia/mediawiki-extensions-FundraiserLandingPage/blob/master/FundraiserLandingPage.body.php [17:09:43] legoktm: Sorry to drag you into it! [17:10:13] The only wfMessage call is for 'donate_interface-make-your-donation' which contains plain text [17:10:46] Funky template magic is output using addWikiText, would that have the same new anti-html safety mechanism? [17:12:59] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog: Fix raw HTML tags breaking 1.29.0-wmf19 security release - https://phabricator.wikimedia.org/T162716#3172368 (awight) [17:13:18] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Release-Engineering-Team: Fix raw HTML tags breaking 1.29.0-wmf19 security release - https://phabricator.wikimedia.org/T162716#3172381 (awight) p:Triage>Unbreak! [17:13:22] awight: https://etherpad.wikimedia.org/p/quote.wikitext [17:13:30] ty [17:13:35] oh dear [17:13:35] dint see anything sensitive... [17:13:44] iono wtf that's about. [17:13:48] it ipsum? [17:13:55] s/it/the/ [17:14:17] I don't understand what happened yesterday when this bug came up--do you remember working on donatewiki, or just payments? [17:16:10] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Release-Engineering-Team: Fix raw HTML tags breaking 1.29.0-wmf19+ - https://phabricator.wikimedia.org/T162716#3172390 (demon) [17:16:15] no big deal. I'm trying to reproduce it locally [17:16:22] hmm [17:16:30] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Release-Engineering-Team: Fix raw HTML tags breaking 1.29.0-wmf19+ - https://phabricator.wikimedia.org/T162716#3172368 (demon) (minor tweak to title, it's not a security release, and donatewiki would be affected on all release... [17:17:41] awight: i have never touched donate [17:17:56] i think ejegg just needed to run that on one of the live boxes [17:18:07] to get the templates to render, to import them to donate? something like that... [17:18:13] huh [17:18:17] k [17:18:38] fundraising-tech-ops, Operations, ops-ulsfo, Patch-For-Review: rack/setup frbackup2001 - https://phabricator.wikimedia.org/T162469#3172430 (Papaul) The server is plugged in port 8 on pfw1-codfw [17:21:00] RainbowSprinkles: can you move donatewiki to wmf.19 on one of the mwdebug hosts? [17:21:12] Hmm, can I? [17:21:50] prep the change on tin and then scap pull on that host I think? [17:26:54] Live on mwdebug1002, I think [17:27:34] wgHostname":"mwdebug1002" ok it doesn't look broken to me [17:27:42] [17:27:50] still on wmf.18 :/ [17:28:27] +1 [17:28:32] Bleh, it pulled from deployment.eqiad.wmnet, which means it doesn't work unless I scap pull on tin first too [17:28:43] (which potentially lets it roll out everywhere) [17:28:55] Anyone happen to know if this html stuff comes from an extension? I'm having a hard time reproducing locally. [17:28:57] Er, maybe not? I dunno [17:29:01] Wikiversions stuff is weird [17:29:47] manually edit wikiversions.php on mwdebug1002? [17:31:10] Ugh, ok [17:31:59] done [17:32:28] broken now, thanks :) [17:33:05] Heh, that's a first [17:33:07] "Thanks for breaking" [17:33:08] lol [17:33:42] :) I see it too [17:34:37] No chance of leaving $wgRawHtml on for donatewiki? [17:36:45] Not my call [17:38:31] https://donate.wikimedia.org/wiki/Template:Lp-layout-default?action=edit [17:38:37] the top call is what's triggering it [17:39:10] I was wrong earlier, OutputPage::addWikiText() will trigger it [17:39:17] because of public function addWikiText( $text, $linestart = true, $interface = true ) { [17:41:01] oh, hold up [17:41:04] ojala I'm finally able to repro locally [17:41:13] $anonPO->setAllowUnsafeRawHtml( false ); [17:41:32] from OutputPage::parserOptions() [17:41:41] so tl;dr: OutputPage::addWikiText() can't use tags [17:41:53] [17:41:57] I don't understand the comment [17:42:06] of course it can be in the just do it from the special page code? [17:42:11] This whole donatewiki template house of cards relies on rawhtml [17:42:28] Yeah I'm happy to shuffle stuff into code to emit meta tags [17:42:47] there's at least 6 more instances of it [17:43:07] and it's not just for meta tags AFAICT [17:43:13] it's to include .js and .css wholesale [17:43:41] all of this code should really be...code. in gerrit. [17:43:47] ino [17:43:58] This is 100% debt [17:44:24] legoktm: I noticed that donatewiki has $wgRawHtml = true [17:44:43] but that allowunsafe(false) thing you found is overriding the setting, I guess? [17:44:46] yes [17:44:49] k [17:45:02] because there were places where parameters would get interpreted as wikitext [17:45:11] so if you did ?foobar=... you now had an XSS [17:45:36] We've locked down edit access to donatewiki, fwiw [17:46:49] ah. Reading bawolff's explanation now. [17:47:09] https://gerrit.wikimedia.org/r/#/c/336389/ [17:51:00] I tweaked my local code to emit the content of the illegal raw html. there are 34 occurrences on this one page [17:54:46] :( [17:54:54] I don't have more time to help sorry, I have to get ready for school [17:55:09] legoktm: This is a horrific hack, but what if we add a config flag to prevent system messages from using URL params [17:55:19] how would that work? [17:55:20] legoktm: no worries, thanks for all the help so far! [17:55:34] Not sure cos I don't understand how that mechanism works yet [18:03:33] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Release-Engineering-Team: Fix raw HTML tags breaking 1.29.0-wmf19+ - https://phabricator.wikimedia.org/T162716#3172647 (awight) See https://phabricator.wikimedia.org/T156184 [18:17:07] (PS1) Awight: Fix sanitization regex [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/347656 (https://phabricator.wikimedia.org/T162716) [18:17:15] fr-tech: ^ [18:20:23] I'm doing a full scap, so donatewiki will briefly go back to wmf.18 on mwdebug1002. I'll revert afterwords. [18:20:42] (PS1) Awight: Kludge an exception to allow raw HTML in system messages [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/347660 (https://phabricator.wikimedia.org/T156184) [18:21:05] RainbowSprinkles: I have a fix ready for CR so we shouldn't need mwdebug1002 any more. Thanks! [18:21:38] awight: so this makes sure there is nothing to the left and right of the target? [18:22:20] cwd: the first fix, yeah. The second one is when things get fun. [18:22:45] * cwd winces [18:24:34] awight: why is the - moved to the front of the group? [18:29:20] hyphen is special in regex, I donno wtf the author intended. [18:29:42] I actually can't untangle what that garbage will actually do [18:29:48] actual for real. [18:30:05] cos the first parenthesis was being interpreted as a delimiter, not a capture expression. surprise! [18:30:21] And the hyphen is supposed to create a character range, but after a backslash all bets are off. [18:31:53] i think you just summed up the problem with regex irl [18:32:07] hehe [18:32:12] I enjoy them [18:32:15] don't know why. [18:32:23] oh they are the most fun [18:32:30] the world runs on regex [18:32:35] i'm not denying it [18:32:43] but every one that gets added kills a kitten [18:33:11] I strictly write "/x" now [18:33:24] a newline and a paragraph of comments for each damn clause [18:34:39] SO HUNGREE [18:40:57] (PS2) Awight: Fix sanitization regex [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/347656 (https://phabricator.wikimedia.org/T162716) [18:40:59] (PS2) Awight: Kludge an exception to allow raw HTML in system messages [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/347660 (https://phabricator.wikimedia.org/T156184) [18:52:25] (CR) Awight: "@bawolff @legoktm" [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/347660 (https://phabricator.wikimedia.org/T156184) (owner: Awight) [19:11:52] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#3172888 (faidon) [19:11:54] fundraising-tech-ops, Patch-For-Review: rename backup4001 to frbackup4001 and jessie reimage - https://phabricator.wikimedia.org/T158220#3172887 (faidon) [19:28:04] fr-tech: Feel free to look askance at https://gerrit.wikimedia.org/r/#/c/347656/ [19:37:40] Fundraising-Backlog, FR-Ingenico: Fix alphabetization of list of iDEAL banks on NL form - https://phabricator.wikimedia.org/T162613#3173078 (DStrine) [19:38:03] (PS5) Awight: Update to python3 [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346708 [19:38:05] (PS4) Awight: Job spec can include tags [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346806 [19:38:07] (PS5) Awight: README fixups [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346810 [19:38:09] (PS5) Awight: Support job as either a positional or named argument [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346811 [19:38:11] (PS9) Awight: Record job state and history in a file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346833 [19:38:13] (PS4) Awight: Rename confusing use of "job tag" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347525 [19:38:15] (PS3) Awight: Ignore a debian packaging byproduct [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347530 [19:38:29] Fundraising-Backlog: process-control lock contention cronspam - https://phabricator.wikimedia.org/T162738#3173084 (cwdent) [19:38:45] awight: ^ [19:38:49] cwd: thanks! [19:38:55] :) [19:38:57] Now I'm allowed to work on it ;-) [19:44:32] (PS1) Awight: Fix cronspam: Lock errors should be reported the same way as other exceptions [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347673 [19:45:04] Sigh. It's way too easy to work with this Python app, there's just no job security AFAICT. [19:51:47] (PS2) Awight: Fix cronspam: Lock errors should be reported the same way as other exceptions [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347673 [19:52:03] aand I just jinxed it [19:52:30] (PS3) Awight: Fix cronspam: Lock errors should be reported the same way as other exceptions [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347673 [19:52:44] why phabricator link not found... [19:52:53] (PS4) Awight: Fix cronspam: Lock errors should be reported the same way as other exceptions [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347673 (https://phabricator.wikimedia.org/T162738) [19:52:53] "T" [19:53:14] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Patch-For-Review: process-control lock contention cronspam - https://phabricator.wikimedia.org/T162738#3173182 (awight) [19:53:56] (CR) jerkins-bot: [V: -1] Fix cronspam: Lock errors should be reported the same way as other exceptions [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347673 (https://phabricator.wikimedia.org/T162738) (owner: Awight) [19:54:23] awight: heh recently that still linked to bugzilla [19:55:13] (PS5) Awight: Fix cronspam: Lock errors should be reported the same way as other exceptions [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/347673 (https://phabricator.wikimedia.org/T162738) [19:55:22] oh neat! [19:58:08] (CR) Ejegg: [C: 2] add license file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346641 (owner: Awight) [19:58:37] (Merged) jenkins-bot: add license file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346641 (owner: Awight) [20:10:02] huh, that paypal audit parser still seems to be doing lookups. or something slow, anyway [20:10:44] haaar [20:10:53] Could be firewalled? [20:11:04] it's not dying [20:11:37] but I thought I fixed the fact that it tried to look up names at all, when we already have them from the file [20:12:08] The logs might say somet things [20:12:36] oh hey, that fix isn't deployed [20:13:11] haha nice debugging! [20:15:29] XenoRyet: feel like testing the CRM's ability to handle normalized recurring paypal messages? [20:15:50] got stuff to deploy in tools and smashpig [20:16:56] Sure, what do we have to do? [20:18:16] deploy the things and slow-start the recurring consumer [20:19:04] Alright, let's do it. [20:20:01] oh hey, i should kill that paypal audit job so we have some audit txns left to test with [20:20:10] Good call [20:20:15] cwd can you kill 107634 on the new civi box? [20:20:22] or Jeff_Green ^ ? [20:20:29] looking [20:20:37] me2 [20:20:41] thanks! [20:20:44] you want us to strace it? [20:20:58] strace? [20:21:11] i.e. see what it's up to using strace [20:21:20] ah, no need [20:21:26] k [20:21:48] It's running fine, I just realized I wanted to stop before it parses the whole file [20:22:12] so we can deploy the new code and have some stuff left to parse [20:22:13] no process-control slay feature yet? [20:22:19] it's stopped [20:22:23] nope, that was deemed not-mvp [20:22:29] oic [20:22:30] thanks Jeff_Green ! [20:22:33] np [20:22:37] ejegg: Jeff_Green - i want to say awight was able to kill something the other day? [20:22:39] XenoRyet: want to prep a tools deploy? [20:22:42] something i didn't think he'd be able to? [20:22:50] oh, freal? [20:22:56] huh [20:23:02] Yeah I still don't know why it worked [20:23:07] I tried and couldn't with a simple 'kill XXX' [20:23:12] It could have been amazingly coincidental timing. [20:23:22] yeah perhaps [20:23:26] did you just kill the pid as your own user? [20:23:31] AFAIR yeah [20:23:38] afaik processes do not have very granular permissions [20:23:48] I may have killed the parent process fwiw [20:23:59] but still shouldn't have been able to send it any signals AIUI [20:24:03] same owner [20:24:16] hrm, what if we whitelisted sudo -u jenkins kill ? [20:24:48] I like the thought of having a consistent kill interface, kill-job paypal_audit_parser [20:25:09] we'll eventually want higher-level tools like kill-and-disable --group queue [20:25:12] yeah it would be better to get the feature into process-control, for that reason ^^^ and also for logging and cleanup [20:25:25] * awight snuck an extra reason in. [20:27:20] yeah...reasonable :( [20:27:23] :) [20:27:29] :S [20:28:19] i wonder if there could be some sudo madness that keeps it attached to your own user since you started the process in sudo [20:28:25] XenoRyet: let me know when you've got a deploy ready and want to try out the new scheduler stuff. We'll need to un-schedule the recurring queue consumer and set it to manual with a low threshold [20:28:54] Yea, prepping the deploy now, I'll let you know when it's ready. [20:28:57] Jeff_Green: i bet there is...probably gets ugly though [20:29:07] log perms and everything [20:29:11] yeah [20:29:21] sudo is a tool that i'm not sure we are all better for having [20:29:24] just thinking of rational explanations for awight's situation [20:29:39] oh, yeah, i have no idea what went on there [20:30:25] Looking at my bash history, it was a straight-up "kill" [20:30:44] I think I killed at the python parent [20:31:04] started with process-control? [20:31:09] yeah [20:32:10] sure looks like the sudo wrapper was already in place--April 4 [20:32:15] (PS1) XenoRyet: Merge branch 'master' into HEAD [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/347687 [20:32:20] yeah as i remember it was [20:32:31] would have certainly been owned by jenkins [20:32:41] ejegg: when you just tried you got, operation not permitted or some such? [20:33:08] Is it possible that I killed the sh wrapper script? [20:33:21] hehe that might work [20:33:23] good question [20:33:25] (CR) XenoRyet: [C: 2] Merge branch 'master' into HEAD [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/347687 (owner: XenoRyet) [20:33:31] (Merged) jenkins-bot: Merge branch 'master' into HEAD [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/347687 (owner: XenoRyet) [20:33:39] if that's the case, I'm not sure it's bad [20:36:22] oh, that's interesting [20:36:34] we should verify that [20:37:03] cwd yeah Operation not permitted [20:37:09] but I tried killing the child process [20:39:12] * cwd ponders the abuelo [20:39:43] u can make a job with command: /bin/sleep 100 to poke at... [21:01:19] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Allow use of Civimails - https://phabricator.wikimedia.org/T162747#3173338 (Eileenmcnaughton) [21:01:48] fr-tech i'll be a couple minutes late to mtg [21:02:57] AndyRussG: meeting? [21:04:34] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Allow use of Civimails - https://phabricator.wikimedia.org/T162747#3173359 (ggellerman) p:Triage>Normal [21:06:39] Fundraising-Backlog: tweak process-control email template to be inbox friendlier - https://phabricator.wikimedia.org/T162687#3173363 (ggellerman) p:Triage>Normal [21:10:00] Fundraising-Backlog: process-control shouldn't cronspam - https://phabricator.wikimedia.org/T162646#3173380 (ggellerman) [21:10:02] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Patch-For-Review: process-control lock contention cronspam - https://phabricator.wikimedia.org/T162738#3173378 (ggellerman) [21:12:20] Fundraising Sprint Asbestos Removal 2016, Fundraising Sprint Bloodletting 2016, Fundraising Sprint Testing on Production, Fundraising Sprint Unbreaking Now, and 10 others: [Epic] Do not show donation form error message: "No processors available". Fi... - https://phabricator.wikimedia.org/T117872#3173385 [21:12:40] Fundraising Sprint Asbestos Removal 2016, Fundraising Sprint Bloodletting 2016, Fundraising Sprint Testing on Production, Fundraising Sprint Unbreaking Now, and 9 others: [Epic] Do not show donation form error message: "No processors available". Fix... - https://phabricator.wikimedia.org/T117872#3173386 [21:13:41] Fundraising-Backlog, FR-Ingenico: Fix alphabetization of list of iDEAL banks on NL form - https://phabricator.wikimedia.org/T162613#3173390 (ggellerman) p:Triage>Normal [21:15:14] Fundraising-Backlog: Replace deprecated transaction.commit_manually in DjangoBannerLoader - https://phabricator.wikimedia.org/T162504#3173395 (ggellerman) p:Triage>Normal [21:18:02] Fundraising-Backlog: Banner impressions loader exception - https://phabricator.wikimedia.org/T162466#3173399 (ggellerman) p:Triage>Low [21:20:33] Fundraising Sprint Asbestos Removal 2016, Fundraising Sprint Bloodletting 2016, Fundraising Sprint Testing on Production, Fundraising Sprint Unbreaking Now, and 9 others: [Epic] Do not show donation form error message: "No processors available". Fix... - https://phabricator.wikimedia.org/T117872#3173402 [21:21:22] Fundraising-Backlog: run-job --list should indicate which are currently scheduled - https://phabricator.wikimedia.org/T162413#3173405 (ggellerman) p:Triage>Low [21:22:19] Fundraising-Backlog, FR-Smashpig: Are we spamming the pending db with iDEAL donations? - https://phabricator.wikimedia.org/T162412#3173406 (ggellerman) p:Triage>Low [21:22:50] Fundraising-Backlog, FR-Ingenico: Fix alphabetization of list of iDEAL banks on NL form - https://phabricator.wikimedia.org/T162613#3168643 (Ejegg) Good old ASCIIbetical order, A-Za-z. Should be an easy fix, though. [21:24:52] (PS21) AndyRussG: [WIP] Custom campaign mixin param handlers [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/343953 (https://phabricator.wikimedia.org/T144453) [21:26:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Clean up Civi bulk import temporary files - https://phabricator.wikimedia.org/T162392#3173417 (ggellerman) p:Triage>Normal [21:27:00] (PS9) AndyRussG: [WIP] Banner sequence campaign mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/344988 (https://phabricator.wikimedia.org/T144453) [21:28:57] (CR) jerkins-bot: [V: -1] [WIP] Banner sequence campaign mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/344988 (https://phabricator.wikimedia.org/T144453) (owner: AndyRussG) [21:29:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Clean up Civi bulk import temporary files - https://phabricator.wikimedia.org/T162392#3173418 (ggellerman) Open>Invalid [21:29:58] Fundraising-Backlog: run-job --list should indicate which are currently scheduled - https://phabricator.wikimedia.org/T162413#3173419 (Ejegg) Open>declined Eh, easy enough to peek at the crontab.d file [21:30:03] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Clean up Civi bulk import temporary files - https://phabricator.wikimedia.org/T162392#3161282 (Eileenmcnaughton) There is a layer of security here in that the user id is in the file name & only the user in the file name can ac... [21:51:54] (PS1) Ejegg: Get rid of subscription paths [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/347772 [21:59:44] So, the sprint name for "T" will be Tinder Returning Random Animal Paddock Escapee Buddy Finder"? [22:00:56] O_O [23:05:54] Fundraising-Backlog, UI-Standardization: CentralNotice Banner Sequence Feature: we need a drag/drop icon - https://phabricator.wikimedia.org/T162756#3173718 (DStrine) [23:11:44] awight: fr-tech: got any suggestions for better class names, rather than MixinCtlGrpController and MixinCtlGrp? In the following patch we have, also, as subclasses BannerSequenceCtlGrpController and BannerSequenceControlGroup... As well as BannerSequenceCtlGrpModel, and these widgets: BktSequenceContainer, BktSequenceWidget, StepWidget [23:12:59] AndyRussG: All I was hoping for were the vowels bck [23:19:01] (PS1) Ejegg: Don't crash on missing ct_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/347787 [23:19:11] XenoRyet: that should fix the KeyError ^^ [23:20:11]