[00:34:16] awight if you are not over Benevity review I pushed up 2 more gerrits - although it is end of day for you now isn't it? [00:35:15] eileen1: oh, sure! nah, I'm working for another hour, thanks for the concern though [00:37:12] :-0 [00:43:05] (CR) Awight: [C: 2] Broaden the definition of 'employed by' an org to include soft credits. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340408 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [00:47:03] (Merged) jenkins-bot: Broaden the definition of 'employed by' an org to include soft credits. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340408 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [00:50:06] (CR) Awight: [C: 2] "Quite the edge case!" (4 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340429 (owner: Eileen) [00:51:57] thanks awight! [00:53:06] awight: You broke beta ;P [00:53:23] awight: DI change seems to work, still pinching myself [00:53:37] Reedy: Ayiii! [00:53:44] * Reedy makes a mw-config commit [00:53:48] (Merged) jenkins-bot: Benevity update, handle situation where the Benevity email might be work but wmf email is home. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340429 (owner: Eileen) [00:53:53] Reedy: oh man [00:54:00] right, I had that mediawiki-config patch queued [00:54:15] 340421 master Deprecate DonationInterface i18n messages [00:54:30] fwiw. I'm sure you've already done the same patch tho [00:54:53] I was about to make one [00:55:08] I don't think the variable needs to exist in InitialiseSettings.php anymore either [00:55:11] Reedy: Are you tooled up to deploy, or shall I? [00:55:14] perfect. [00:55:53] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340448 [00:56:20] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340448 (owner: Eileen) [00:56:58] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340448 (owner: Eileen) [00:59:44] !log Update CiviCRM from 04b49b0393295610c530dc29cb0218e6755ef24f to b3f6eef513cf2ea2c48379f9e0bcd4b02b18e379 [00:59:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:17:46] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3063312 (awight) [01:18:57] eileen1: High five on the code to prevent double-billing recurring donors when our date math fails! [01:19:12] gotta run :D [02:06:02] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3063457 (Eileenmcnaughton) These look like poor calculations on our code's part, but... [02:06:49] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3063458 (Eileenmcnaughton) Query to find them: ``` SELECT * FROM ( SELECT cr.con... [02:10:35] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3063462 (Eileenmcnaughton) Here is a query just to get past the blockage without reso... [02:12:39] (PS1) Eileen: Adjust contribution date on 4 recurring contributions to get past blockage. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340457 (https://phabricator.wikimedia.org/T159298) [02:13:06] I've put up a fix that will adjust (not 100% correctly but acceptably) the 4 affected contributions, should hush the fail mail [03:45:42] (PS1) Eileen: Rerun the geocoding. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340459 (https://phabricator.wikimedia.org/T158271) [03:47:03] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we make the nickname field searchable? - https://phabricator.wikimedia.org/T158784#3063548 (RLewis) Thanks @Eileenmcnaughton this is great! [04:06:05] (PS2) Eileen: Rerun the geocoding. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340459 (https://phabricator.wikimedia.org/T158271) [05:03:45] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: We still have contacts whose merge is blocked on mis-rounded geocodes - https://phabricator.wikimedia.org/T158271#3031937 (Eileenmcnaughton) I found over 1 million on both live and stag... [05:07:22] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Wikimedia-CentralNotice-Administration, User-JSeddon: CentralNotice Product Roadmap - https://phabricator.wikimedia.org/T159305#3063639 (AndyRussG) [05:07:25] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: What should we do to fix data where blank addresses have overwritten valid addresses - https://phabricator.wikimedia.org/T153917#3063640 (Eileenmcnaughton) [05:07:53] * AndyRussG stops stalking Seddon in Phab [05:07:57] cya! [05:53:39] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Waiting for Godot, and 2 others: Blank address data being collected & stored as an address - https://phabricator.wikimedia.org/T153804#3063661 (Eileenmcnaug... [06:00:55] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Wikimedia-CentralNotice-Administration, Patch-For-Review: CentralNotice: banners not showing on mobile versions of mediawiki.org, Wikidata and Wikisource - https://phabricator.wikimedia.org/T158030#3063664 (Jseddon) [06:17:22] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Waiting for Godot, and 2 others: Blank address data being collected & stored as an address - https://phabricator.wikimedia.org/T153804#3063666 (Eileenmcnaug... [12:01:00] Wikimedia-Fundraising-Campaigns: Fundraise in India - https://phabricator.wikimedia.org/T131118#3064192 (Pcoombe) [14:44:51] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#3064525 (Jgreen) [15:16:42] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#3064576 (Jgreen) [15:23:42] fundraising-tech-ops: set up firewall policies for barium replacement civi1001 - https://phabricator.wikimedia.org/T159336#3064590 (Jgreen) [15:54:10] fundraising-tech-ops: set up firewall policies for barium replacement civi1001 - https://phabricator.wikimedia.org/T159336#3064670 (Jgreen) commit aeb68e44ab27a17c62a8079740402b3088fd573a Author: Jeff Green Date: Wed Mar 1 10:25:53 2017 -0500 assign IP for civi1001.frack.eqiad.... [16:01:20] fundraising-tech-ops: set up firewall policies for barium replacement civi1001 - https://phabricator.wikimedia.org/T159336#3064686 (Jgreen) commit f5b42748c4393b7b9736638fafdbc17b67273b9f Author: Jeff Green Date: Wed Mar 1 15:58:47 2017 +0000 pfw and iptables policies for civi1... [16:02:39] fundraising-tech-ops: set up firewall policies for barium replacement civi1001 - https://phabricator.wikimedia.org/T159336#3064687 (Jgreen) Netops could you please deploy the updated pfw policy when you have a chance? Boron's webdrop updated too. [16:03:15] fundraising-tech-ops, Operations, netops: set up firewall policies for barium replacement civi1001 - https://phabricator.wikimedia.org/T159336#3064690 (Jgreen) a:Jgreen>None [16:31:07] fundraising-tech-ops, Operations, netops: reassign wmf7010/frpm1001 to host "civi1001.frack.eqiad.wmnet" - https://phabricator.wikimedia.org/T159342#3064786 (Jgreen) [16:52:54] hi fr-tech! [16:53:45] wb ejegg, how was the party? [16:54:57] Fundraising-Backlog, fundraising-tech-ops: database access to additional CPS analyst - https://phabricator.wikimedia.org/T155746#3064896 (Jgreen) Open>Resolved afaik this is done. please reopen if there are any issues. [16:57:47] cwd massive and extended [16:58:00] and with weird racial dimensions [16:58:47] two of the symbols of the carnaval are young men in blackface with mouths stained red [16:58:59] wow [16:59:00] yikes [16:59:02] and 'la negrita', a stereotyped black woman [16:59:25] i had no idea [16:59:34] about colombia in that respect [16:59:35] neither did I, till I got here! [16:59:48] it's a super diverse city and celebration [17:01:05] i tried asking a bunch of locals what they thought about the caricatures [17:01:24] and nobody seemed to understand why i thought there was a problem [17:01:46] heh [17:01:53] but... anyway, tons of amazing music [17:01:58] and wild costumes [17:02:41] men and women of all shapes and ages in brilliantly colored skintight costumes - the parades lasted 6 hours each day [17:03:12] got a ton of photos to sift through. I'll try to get 'em online this week [17:04:00] awesome [17:04:05] would love to see those [17:13:16] so... let's see what's up with this recurring gc failmail [17:17:30] (CR) Ejegg: "This doesn't seem terrible, but I'd prefer loosening the restrictions to a 5 day window." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340457 (https://phabricator.wikimedia.org/T159298) (owner: Eileen) [18:00:21] fr-tech: A committee is a life form with six or more legs and no brain. [18:00:21] -- Lazarus Long, "Time Enough For Love" [18:00:21] -- discuss. [18:07:09] ooh, super ancient tube fossils [18:10:36] fr-tech i'll be in the talk hangout in one minute [18:12:14] i'm hacking on a puppet thing but will try to drop in [18:12:34] I'm either getting weird network messages, or the room is empty, or both [18:12:42] "Requesting to join the video call..." [18:12:48] ejegg: btw, the resultswitcher changes went out yesterday and do not appear to have exploded [18:13:48] cwd nice! [18:14:16] cwd still soured on matrix / riot app for audio chat? [18:14:17] Luck of the draw, you say! [18:14:29] i may be imagining this but i put through a couple donations and thought there was noticeably less delay before the thank you page [18:14:42] ejegg: i'd be down to try it again [18:14:52] i can see that self hosted is not a good option yet, unfortunately [18:16:28] trying to start a voice call in this channel [18:16:43] oh right, we have a matrix version of the chan too [18:16:43] I'm in the channel, AFAICT [18:16:48] d'oh. [18:17:01] I haven't gotten the schooling yet--will the Riot app work? [18:17:11] awight trying it now myself! [18:17:15] it usta [18:17:22] moi aussi [18:17:49] there are some upsides to taking calls on phone [18:17:58] since it's inherently a compromised platform [18:18:04] i don't feel as bad loading it up with crap [18:18:45] hmm, not sure call doesn't start, though [18:18:48] I'm "calling" at the minute [18:19:18] yah, mine just hung at that too :/ [18:20:05] drat [18:20:11] imo old fashioned conference calls are probably the best signal to noise ratio still [18:20:18] with what, though? [18:20:31] gotta VOIP to the other hemisphere anyway [18:20:45] android will do it natively [18:21:07] That's just a branded VOIP chat room, innit? [18:21:13] ok, hangouts it iregular old 33-way calling ? [18:21:14] no idea [18:22:05] blerp [18:22:56] oh shoot, scrum of scrums time anyway. [18:23:01] Any news? [18:23:33] fr-tech ^^ [18:24:04] nothing i can think of [18:24:49] ejegg: a mini-hava nagila for Reedy for noticing that we could deprecate donationinterface_langonly.php, perhaps [18:25:09] ejegg: Do you see any of my text in the matrix channel? [18:25:31] that's the one where you lift the celebrant up in the chair, right? :) [18:25:46] awight: yeah, I see a couple messages [18:26:07] ejegg: nothing here, thx! [18:26:25] I tried initiating a video call from desktop and still no dice [18:31:11] Well, we always have etherpad [18:32:02] (CR) Awight: [C: 2] Adjust contribution date on 4 recurring contributions to get past blockage. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340457 (https://phabricator.wikimedia.org/T159298) (owner: Eileen) [18:33:11] AndyRussG: did you say you'd tracked down the slow CN saves to a message cache issue? [18:35:43] ejegg: yes, it is indeed due to the change in i18n messagecache handling. Actually affects all edits of long messages to the Mediawiki: namespace [18:36:14] ejegg: T158084 [18:36:14] T158084: Mediawiki namespace pages, including CentralNotice banners, are slow to save - https://phabricator.wikimedia.org/T158084 [18:37:13] My first inclination would be that the deferred job in core run after the page returns the http response (postsend). it's an option in our PHP deferred job class [18:37:40] I'll make a tentative WIP-tagged core patch, see what people think [18:38:17] thanks! [18:38:38] (Merged) jenkins-bot: Adjust contribution date on 4 recurring contributions to get past blockage. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340457 (https://phabricator.wikimedia.org/T159298) (owner: Eileen) [18:45:16] (PS1) Awight: Merge branch master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340545 [19:03:03] https://joeyh.name/blog/entry/removing_everything_from_github/ [19:05:33] awight: thanks! looks like you're deploying the date-shift patch [19:08:13] cwd|afk: seems like a bad move [19:08:24] ejegg: yep, in a minute! [19:08:39] awight: on github's part? [19:09:15] if it was actually deliberate, they deserve whatever happens [19:09:20] gotta go cut some wood! [19:11:38] cwd|afk: yah. Somehow looks like they're capitalizing on the accident of being a monopoly, though I'm not seeing what the benefit could be. [19:11:45] or just carelessness [19:12:00] These BSD armchair lawyer guys do smell grumpy [19:12:55] sounds bad [19:15:59] (CR) Awight: [C: 2] Merge branch master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340545 (owner: Awight) [19:16:06] (Merged) jenkins-bot: Merge branch master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340545 (owner: Awight) [19:16:26] (CR) Ejegg: [C: 2] Rerun the geocoding. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340459 (https://phabricator.wikimedia.org/T158271) (owner: Eileen) [19:18:29] !log update civicrm from b3f6eef513cf2ea2c48379f9e0bcd4b02b18e379 to 58c8c06eee577edd8caa4a33b2c48fbd7651a005 [19:18:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:19:54] !log applying civicrm db migration wmf_civicrm:7465 [19:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:21:46] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations, Patch-For-Review: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3065396 (awight) Wow, thanks for all the queries! Creating a s... [19:21:56] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising Sprint Unbreaking Now, and 3 others: Longterm fix + regression test for T144489 - https://phabricator.wikimedia.org/T144557#3065398 (awight) [19:21:58] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations, Patch-For-Review: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3065397 (awight) Open>Resolved [19:22:12] (Merged) jenkins-bot: Rerun the geocoding. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340459 (https://phabricator.wikimedia.org/T158271) (owner: Eileen) [19:22:50] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Fix Feb 28 recurring glitch - https://phabricator.wikimedia.org/T159363#3065412 (awight) [19:22:59] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Fix Feb 28 recurring glitch - https://phabricator.wikimedia.org/T159363#3065412 (awight) [19:23:49] Fundraising Sprint Deferential Equations, Fundraising-Backlog: PHP-Queue: remove IndexedFifo and KeyValue interfaces - https://phabricator.wikimedia.org/T159175#3065428 (awight) a:awight [19:24:03] Fundraising Sprint Deferential Equations, Fundraising-Backlog, MediaWiki-Cache, MediaWiki-extensions-CentralNotice, Patch-For-Review: Mediawiki namespace pages, including CentralNotice banners, are slow to save - https://phabricator.wikimedia.org/T158084#3065429 (awight) a:AndyRussG [19:24:29] fr-tech: Anything I can help with? [19:25:02] awight one more patch for iDEAL / caching [19:25:30] https://gerrit.wikimedia.org/r/339330 [19:25:43] want to check that we [19:25:47] re [19:26:06] actually getting a cache object from localClusterInstance [19:26:14] and not just the no-op version [19:26:27] but... I'm not sure where that log line will go [19:27:38] ejegg: great, checking it [19:27:56] thanks! [19:31:39] ah, k, that was the same cache I switched the IP velocity filter to, and it seems to still be working [19:42:49] (CR) Awight: [C: -1] Use HashBag instead of EmptyBag (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/339330 (https://phabricator.wikimedia.org/T128692) (owner: Ejegg) [19:59:12] (PS5) Ejegg: Use HashBag instead of EmptyBag [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/339330 (https://phabricator.wikimedia.org/T128692) [20:00:54] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising-Backlog, FR-Smashpig, Spike: Spike: investigate smashpig next steps - https://phabricator.wikimedia.org/T156863#2988148 (awight) I'm not sure if I should rename this spike title to reflect the much... [20:04:23] awight: this comment is your monologue in the broadway adaptation of your wmf memoirs https://phabricator.wikimedia.org/T156863#3065592 [20:05:37] Fundraising Tech Backlog, Fundraising-Backlog: decide whether banner screenshot is still usable/useful - https://phabricator.wikimedia.org/T156853#2987911 (awight) We've reviewed this a few times, and I believe this is what we know: * Nobody from fr-online uses this service. * They manually take screensh... [20:06:58] cwd: Thanks, man :) [20:08:04] How I feel about industry best practices: http://img.timeoutshanghai.com/201208/20120810033954379_Medium.jpg [20:08:45] haha [20:11:35] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising-Backlog, FR-Smashpig, Spike: Spike: Collaborate on a rough roadmap for SmashPig development - https://phabricator.wikimedia.org/T159375#3065672 (awight) [20:12:18] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising-Backlog, FR-Smashpig, Spike: Spike: Write requirements for SmashPig - https://phabricator.wikimedia.org/T159376#3065685 (awight) [20:12:33] fr-tech: Anyone feel like IRC/etherpadding? https://etherpad.wikimedia.org/p/SmashPig_requirements [20:22:18] interesting article - I read it a year ago but FB just brought it up as a memory https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html?_r=5 [20:26:13] * awight wags fist at paywall [20:29:50] ah right [20:30:37] yeah that sucks. I did subscribe to one of the ones that's been under attack lately as a 'pay for what you care for, journalism matters' purchase [20:31:02] surprised such an old article is behind the paywall [20:34:03] i am 100% on board with paying for content [20:34:20] it's the only part of the internet that will survive when the advertising bubble bursts [20:34:47] that being said paying for content online is still a pretty crappy user experience [20:37:57] Ooh, this is fun: https://www.researchgate.net/publication/272793687_Is_Requirements_Engineering_Inherently_Counterproductive [20:38:45] > An interdisciplinary literature [20:38:46] review revealed substantial evidence that while sensemaking [20:38:46] improves design performance, problem structuring reduces [20:38:46] design performance. [20:39:00] ... not entirely sure what that means yet :) [20:39:31] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: What should we do to fix data where blank addresses have overwritten valid addresses - https://phabricator.wikimedia.org/T153917#3065786 (Eileenmcnaughton) dev db is fine for investigating this on as... [20:40:20] sounds like there's a downside to sensemaking [20:42:31] requirements engineering = waterfall = a very sad dstrine [20:48:57] cwd: I'm not seeing that part [20:48:57] Fundraising Sprint Costlier Alternative, Fundraising-Backlog, FR-Smashpig, Spike: Spike: Collaborate on a rough roadmap for SmashPig development - https://phabricator.wikimedia.org/T159375#3065794 (DStrine) [20:48:57] Fundraising Sprint Costlier Alternative, Fundraising-Backlog, FR-Smashpig, Spike: Spike: Write requirements for SmashPig - https://phabricator.wikimedia.org/T159376#3065795 (DStrine) [20:48:57] awight: just the phrasing [20:48:57] dstrine: I'm happy to explore the alternatives, but check out the discussion here: https://phabricator.wikimedia.org/T156863 [20:48:57] My instinct is that we need a roadmap and better definition of what SmashPig is and does [20:48:57] I have enough information from T156863 to satisfy discussions for at least the March timeframe. [20:48:57] T156863: Spike: investigate smashpig next steps - https://phabricator.wikimedia.org/T156863 [20:48:58] awight: what about releasing it one little spec compliant component at a time? [20:49:11] I think we can close T156863 but I will wait to discuss it at standup. [21:08:37] haha [21:08:46] researchpapering [21:13:39] Fundraising-Backlog, fundraising-tech-ops: upgrade samarium from precise to jessie - https://phabricator.wikimedia.org/T156848#3065897 (awight) For reference, the public website hosted on samarium is: https://frdata.wikimedia.org/ [21:16:41] i remember there was one called freshmaking [21:16:45] i think it was K4 [21:22:05] yep [21:49:37] awight: guessing that thing is trying to pop from an empty queue? [21:49:53] which should be fine but something something [21:50:13] Doesn't look like it. [21:50:19] This isn't the PHP queue code [21:50:23] at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1136) [21:50:30] seems more filey than queuey [21:50:58] Here's the offending Jenkins code, [21:50:59] at hudson.util.ProcessTree$Linux$LinuxProcess.getEnvironmentVariables(ProcessTree.java:592) [21:53:28] Not the exact version of the code apparently, but maybe this is the line: https://github.com/jenkinsci/jenkins/blob/stable-1.424/core/src/main/java/hudson/util/ProcessTree.java#L672 [21:54:52] yeah dang [21:56:04] "environ" file? gah [21:56:22] I guess we just write it up as another example of gremlins? [21:56:52] heh, not my favorite resolution... [21:58:39] Fundraising-Backlog, FR-Paypal: New PayPal audit failure: cannot fetch transaction details - https://phabricator.wikimedia.org/T159385#3066002 (awight) [21:59:19] reason: isJava [21:59:57] static public final Boolean reason = Framework().adapterFactory(fuckingState).isJava() [22:00:20] heh that's more like it [22:04:28] import com.newCompany.logging.factories.adapters.stubs.interfaces.properties.dynamicProperties [22:04:54] * awight scatters chicken bones [22:08:37] awight: it seems weird that it only barfed after it had emptied the queue doesn't it? [22:09:42] There's a logline after the queue is emptied, though [22:09:49] 2017-03-01T20:41:41+00:00 [INFO ] {SmashPig-ConsumePendingQueue::SPCID-0091251356} Processed 6 pending messages in 0 seconds. [22:10:02] and it seems that Jenkins was in the midst of killing the runner: [22:10:09] at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:402) [22:10:12] at hudson.Launcher$LocalLauncher.kill(Launcher.java:721) [22:16:36] freaky [22:16:50] not just the thread? i don't know how to read this stuff [22:19:04] https://github.com/jenkinsci/jenkins/blob/stable-1.424/core/src/main/java/hudson/model/AbstractBuild.java#L481 [22:20:50] Oh hey, more to the point, https://github.com/jenkinsci/jenkins/blob/stable-1.424/core/src/main/java/hudson/util/ProcessTree.java#L529 [22:20:52] these comments are not confidence inspiring [22:20:59] /proc/PID/environ [22:21:22] You still have procfs or whatever on the new OS? [22:21:54] And also, it could have simply been a race condition somewhere in this wonderful Java [22:22:09] like, the task died just before Jenkins tried to kill it [22:22:27] but cleaned up its pid file? [22:24:40] that's done by the system [22:25:28] I'm facepalming all over that "getFile" was ever a good function name for "get something random under the "/proc/PID" directory [22:25:52] * awight begins to suspects the authors of being "paid" volunteers [22:26:05] got to wash up before sprint planning... [22:26:14] haha [22:26:34] light-blue collar work. [22:28:01] * awight stops looking for paid protester memes in disgust [22:31:15] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: What should we do to fix data where blank addresses have overwritten valid addresses - https://phabricator.wikimedia.org/T153917#3066093 (Eileenmcnaughton) We have the following actions on blank addre... [22:44:11] awight: touche [22:50:31] awight: i thought the jvm managed its own "threads" [22:50:58] but i have never had any serious exposure to java [22:55:53] It does, but this is an external process [22:56:37] the php one? [22:56:45] but isn't that not what's dying? [22:57:33] So far, my guess is that it's Java trying to kill off the external PHP process that it launched. [22:58:52] I just confirmed to self that /proc/PID/environ is accessible by the process owner. Happy to not worry until this happens again. [22:59:03] what i'm wondering is, if java sets the pidfile with a hacky function call, would the system know to clean it up? [22:59:19] * ejegg vaguely remembers some problem with how jenkins kills things - related to that shutdown handler in python? [23:00:08] /proc/* is out of Java's paws, luckily. Only the OS writes to those files [23:03:43] oh i get it, that code is just reading the file [23:08:54] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: What should we do to fix data where blank addresses have overwritten valid addresses - https://phabricator.wikimedia.org/T153917#3066143 (Eileenmcnaughton) Time to construct blank addresses table = Qu... [23:11:30] awight: there it is again [23:14:10] Fundraising-Backlog: error opening Annual Report in Safari Version 9.1.1 (10601.6.17) - https://phabricator.wikimedia.org/T159389#3066148 (MBeat33) [23:53:29] Fundraising Sprint Deferential Equations, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: turn on 3DS for Sweden, Norway and Denmark - https://phabricator.wikimedia.org/T158357#3066202 (Ejegg) Open>Resolved p:Triage>Normal [23:53:50] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 5 others: Resultswitchers: process return before busting out of frames - https://phabricator.wikimedia.org/T153972#3066208 (cwden... [23:54:11] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising-Backlog, FR-PayPal-ExpressCheckout, Patch-For-Review: Paypal EC missing payment method - https://phabricator.wikimedia.org/T157618#3066210 (Ejegg) Open>Resolved [23:54:14] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising-Backlog, FR-Smashpig, Spike: Spike: investigate smashpig next steps - https://phabricator.wikimedia.org/T156863#2988148 (DStrine) Open>Resolved [23:54:57] Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising Sprint Licking Cookies, and 6 others: [Epic] Support Express Checkout recurring donations - https://phabricator.wikimedia.org/T134446#3066214 (Ejegg) [23:55:00] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising-Backlog, and 4 others: PayPal EC response processor needs to act on "recurring" flag - https://phabricator.wikimedia.org/T153720#3066212 (Ejegg) Open>Re... [23:55:39] Fundraising Sprint Deferential Equations, Fundraising-Backlog: Teach CPS to save and aggregate banner history data - https://phabricator.wikimedia.org/T159173#3059111 (DStrine) Open>Resolved [23:59:42] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: [EPIC] new round of common dedupe conflicts - https://phabricator.wikimedia.org/T153019#3066218 (DStrine)