[00:36:58] Fundraising-Backlog, FR-Amazon, MediaWiki-extensions-DonationInterface: Amazon gateway - crash on getCaptureDetails failiure? - https://phabricator.wikimedia.org/T182735#3832995 (Ejegg) [00:44:52] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Fatal error in Amount formatting - https://phabricator.wikimedia.org/T182737#3833028 (Ejegg) [01:48:34] Fundraising-Backlog, Support-and-Safety: Revoke centralnotice-admin for AWight (WMF) - https://phabricator.wikimedia.org/T168428#3833115 (revi) I thought WMF advanced permission is #support-and-safety business? (While I can just remove the permission, I'm leaving the permission as is for SuSa to take car... [16:36:57] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Make cURL timeout configurable for Silverpop API stuff - https://phabricator.wikimedia.org/T182795#3834700 (Ejegg) [17:02:05] AndyRussG: ejegg meeting? [17:22:06] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-DonationInterface: Run Phan on CentralNotice and DonationInterface - https://phabricator.wikimedia.org/T182714#3834875 (Ejegg) [17:29:13] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-DonationInterface: Run Phan on CentralNotice and DonationInterface - https://phabricator.wikimedia.org/T182714#3834913 (Ejegg) Our code seems to be overwhelming this when I run it locally on a trunkwiki checkout with all of... [18:02:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835068 (LeanneS) [18:11:52] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3835143 (Mholloway) [18:31:49] ejegg: hiya [18:32:08] hi cwd! [18:32:15] Bout to step into Scrum of scrums [18:32:24] aah np [18:32:35] got any news for the rest of engineering? [18:32:38] just looking at some civi logging, nothing urgent [18:32:45] nothing i can think of, thanks [19:07:16] cwd what's up with the civi logging? [19:07:30] Got some good ideas of places to reduce logspam? [19:10:02] Ejegg: just about to eat lunch but I’ll message you in a bit! [19:10:12] cool cool [19:26:28] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-WMF-Audit, Patch-For-Review: Ingenico WX audit parser taking crazy long, mysql connect dying - https://phabricator.wikimedia.org/T182605#3835471 (Ejegg) Open>Resolved a:Ejegg [19:30:45] (PS1) Ejegg: CRM-21521: fix nested multipart handling [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/398107 [19:31:06] Just updating our local version to sync with what's submitted upstream ^^^ [20:07:21] mepps I think the 'reload to see boards when new user' issue is a race condition that'll likely only happen locally [20:08:21] i.e. when the inserts into the DB take longer than the logged-in page loading and making the next request for board info [20:08:45] we could do something like flag the user as new and add a delay to that first request if we were really concerned [20:09:40] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), Patch-For-Review: Blocked cookie detection not working on payments-wiki - https://phabricator.wikimedia.org/T180975#3835595 (Ejegg)... [20:18:43] ejegg: do you have any wifi problems with your x1? [20:19:17] i'm going to conservatively estimate that i've spent 3 hours trying to connect to wifi since yesterday [20:21:36] oh damn, no, I haven't had any issues like that lately [20:21:46] what firmware version do you have? [20:22:41] firmware-iwlwifi: [20:22:43] Installed: 20170823-1 [20:22:45] Candidate: 20170823-1 [20:23:09] lemme see, I might still be using the one I had to sideload at install time [20:23:33] aah [20:23:42] so i've got 20161130-3 installed [20:23:50] but lemme check what else there is in the dir [20:23:54] i have foudn plenty of complaints of generally unreliable wifi with this chipset [20:24:12] but the behavior is eerily similar to my last laptop [20:24:25] and it would not be the first time a bug has jumped the air gap for me [20:25:09] ok, the thing I had to install was iwlwifi-8265-26.ucode [20:25:12] in /lib/firmware [20:25:27] cwd what 8265 files do you have there? [20:26:32] huh, i have 21 22 27 31 [20:26:35] but not 26 [20:26:40] hi ejegg i'm looking at a different bug [20:26:42] of course i have no idea what that means [20:26:43] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835641 (Eileenmcnaughton) @LeanneS how does the data look when there is a different currency? Is there just an extra 'currency' column. It would be kinda good to get a... [20:26:50] it's with the refresh to board links [20:27:26] mepps oh, what's that? [20:27:35] not loading the default board on login? [20:28:10] no, if you refresh with one of the boards in the hash it doesn't load the board [20:28:29] cwd want to try with 26? [20:29:44] cwd https://ejegg.com/iwlwifi-8265-26.ucode [20:29:52] md5sum: 2b93ebba6db2c6bbb7a8f06b0ccae9f2 [20:29:53] ejegg: thanks! [20:29:57] i'll give it a shot [20:30:43] ejegg: as far as civi logging, what i was looking at is /var/log/debug [20:30:55] during peak times that file was getting massive, ~8G [20:31:05] ah, hmm [20:31:11] it seems to have slowed down with the banners [20:31:15] so it is 'debug' logging [20:31:22] yeah [20:31:31] it looked like per query resultsets [20:31:33] maybe only writes [20:31:39] ahh, right [20:31:44] lemme take a look [20:31:59] groovy [20:32:12] we were also seeing all the same stuff copied to /v/l/messages, jeff killed that [20:32:31] so the disk is not in danger atm but good to get ahead of it anyway [20:32:57] cwd what do the resultset lines look like? [20:33:18] not the 'Contribution result from civicrm_contribution_add():' [20:33:22] ? [20:33:47] checking... [20:37:02] gotta relocate; back in a bit! [20:37:06] ejegg: yeah i think that is wat i [20:37:11] what i am seeing [20:37:38] Contribution result from civicrm_contribution_add [20:37:44] yarp...lots of those [20:43:11] so ejegg what i'm looking at is that on refresh with a hash in the url the board doesn't load but i am finding the behavior to be sporadic [21:11:16] mepps ok, I wonder if that's because the hash change event is firing before the info request comes back [21:11:24] we could do something with promises there [21:12:07] cwd ok, that line is being output to the console and picked up by process-control logging [21:12:40] Let me see if we can keep doing that without also logging it to debug [21:12:42] so what i'm seeing is a pool closed error on the first call (to the default board) and then it doesn't get to the board in the hash--it doesn't use the change event, it goes through the code on initial load [21:12:50] ejegg: gotcha [21:13:05] it's not a huge deal but if it is easy to fix it would be a win [21:13:11] mepps ah, ok [21:13:42] what version of persistence.js are you using mepps? [21:13:51] your patch ejegg [21:14:53] althoguh i can't find your patch now [21:15:19] https://gerrit.wikimedia.org/r/397904 [21:15:35] (Abandoned) Ejegg: Bad submodule SHA commit [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/397886 (owner: Ejegg) [21:17:46] (PS2) Ejegg: Tweak informationsharing message to be more upfront [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/397637 (owner: Pcoombe) [21:17:56] (CR) Ejegg: [C: 2] "Thanks pcoombe!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/397637 (owner: Pcoombe) [21:19:19] (PS2) Ejegg: Redirect Adyen skin [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/393633 (owner: Mepps) [21:19:35] (CR) Ejegg: "PS2: deleted .DS_Store files" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/393633 (owner: Mepps) [21:19:59] (PS3) Ejegg: Redirect Adyen skin [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/393633 (owner: Mepps) [21:20:12] (CR) Ejegg: [C: 2] "Thanks mepps!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/393633 (owner: Mepps) [21:20:56] cwd huh, it's all handled by the same watchdog call [21:21:24] ejegg: it is also possible that if they are tagged right we can filter at syslog [21:23:06] cwd ok, so during spreadsheet imports we hit that function via the web UI rather than process-control [21:23:33] (Merged) jenkins-bot: Tweak informationsharing message to be more upfront [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/397637 (owner: Pcoombe) [21:23:39] so we'd lose it if it's not syslogged [21:24:00] but... I don't think we've ever used that file for debugging [21:24:20] and spreadsheet imports are the kind of thing we can easily retry after turning debug logging back on [21:24:46] ejegg: well, it is probably not too big of a deal [21:24:55] a more material disk space concern there is probably /tmp [21:25:08] (Merged) jenkins-bot: Redirect Adyen skin [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/393633 (owner: Mepps) [21:25:58] we got rid of the duplication to messages which was the egregious part [21:27:45] cwd the only file in /tmp in use is Raw Recipient Data Export Dec 13 2017 05-51-02 AM 1464.zip [21:28:23] cool, we sould make something rotate those out huh? [21:28:54] head home from town, back soon! [21:29:24] cwd yep - there is a ticket - you bumped it but not in sprint yet [21:30:08] (PS4) Ejegg: Debugging for mystery FK errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 [21:30:50] ^^^ Checking for native txn rollbacks at a bunch of places to try to narrow down those FK errors [21:32:18] ejegg: I'm ok with that change - but if I approve now you'll deploy now & we can keep an eye that it's not tooo bouncy? [21:33:35] eileen: sure [21:33:51] (CR) Eileen: [C: 2] "I think this will be ok but it makes me a little nervous so let's watch closely when it goes out" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 (owner: Ejegg) [21:34:22] eileen I realized i forgot to cherry-pick your stripe .install patch, will pull that in too [21:34:45] ah ok - the payment type? [21:34:51] yah [21:34:54] we really want to get back to merging [21:35:04] hopefully tomorrow! [21:35:23] btw my contact type change is looking OK - I need to port some core changes tho & get tests running for it in gerrit [21:36:25] nice! [21:37:41] (Merged) jenkins-bot: Debugging for mystery FK errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 (owner: Ejegg) [21:40:44] (PS1) Ejegg: Apply drupal formatting to wmf_civicrm.install [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398131 [21:40:46] (PS1) Ejegg: Add stripe payment instrument. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398132 (https://phabricator.wikimedia.org/T172423) [21:40:48] (PS1) Ejegg: Add stripe to initial install [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398133 [21:40:50] (PS1) Ejegg: Debugging for mystery FK errors [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398134 [21:41:04] (CR) Ejegg: [C: 2] Add stripe to initial install [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398133 (owner: Ejegg) [21:41:06] (CR) Ejegg: [C: 2] Apply drupal formatting to wmf_civicrm.install [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398131 (owner: Ejegg) [21:41:08] (CR) Ejegg: [C: 2] Debugging for mystery FK errors [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398134 (owner: Ejegg) [21:41:10] (CR) Ejegg: [C: 2] Add stripe payment instrument. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398132 (https://phabricator.wikimedia.org/T172423) (owner: Ejegg) [21:42:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835868 (LeanneS) @Eileenmcnaughton the data looks exactly the same without ind. Unfortunately they don't indicate in the CSVs so I have to try to match them up based o... [21:43:53] (Merged) jenkins-bot: Apply drupal formatting to wmf_civicrm.install [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398131 (owner: Ejegg) [21:43:57] (Merged) jenkins-bot: Add stripe payment instrument. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398132 (https://phabricator.wikimedia.org/T172423) (owner: Ejegg) [21:43:59] (Merged) jenkins-bot: Add stripe to initial install [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398133 (owner: Ejegg) [21:44:01] (Merged) jenkins-bot: Debugging for mystery FK errors [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/398134 (owner: Ejegg) [21:48:19] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835883 (Eileenmcnaughton) OK - so we have the converted amount & the currency & we calculate the original amount? [21:52:30] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835887 (Ejegg) Argh, we don't get the original amount? I guess if we don't have their email it's ok, but @CCogdill_WMF likes to mention previous donations in the origi... [21:52:51] ooh, shiny new f_c_u output! [21:53:08] woot [21:53:12] oh hey, and just one 3-digit code to enter, nice [21:53:33] also it doesn't bother to prompt if there are no changes to review [21:53:42] cool, cool [21:54:27] !log updated civicrm from 85a8526eb8 to 798e24671b [21:54:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:54:43] I think it's fixed in terms of impossible submodule hashes too, those should show up at the review stage and cause a fatal error before anything other than git metadata gets touched [21:54:53] very nice [21:55:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835901 (LeanneS) We have the CSV that contains the original currency amount. We then receive the wire in USD so I figure out the converted amounts for each gift using... [21:56:12] lemme know if you see anything weird or broken [21:56:21] will do [21:58:22] hey ejegg i just downloaded your latest patch with the addition to data.js and that causes the board not to load on initial login or reload [21:58:44] oh shoot [21:58:52] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835903 (Eileenmcnaughton) ie you take the total of the spreadsheet & calculate the exchange rate used? how many currencies might be in a spreadsheet? [21:59:32] mepps so it breaks something with getting the default? [22:00:33] hmm so i fixed it on login but it is now consistently broken on login [22:00:52] (PS3) Mepps: Avoid closing connections prematurely [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/397904 (owner: Ejegg) [22:01:40] i meant broken on reload [22:02:09] ok [22:04:04] checking the updated patch... [22:04:57] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Different currencies for Benevity import - https://phabricator.wikimedia.org/T182808#3835904 (LeanneS) To clarify, they include the amount, but not the actual currency. So it may say 5.00 but not indicate if it's EUR, CAD, etc. I have to piece it togeth... [22:07:28] mepps that version is crashing with TypeError: connection.query(...).then is not a function [22:07:38] oh hey, b/c of the data.js change [22:07:49] right, not using promises there [22:08:27] or... no, it's still async [22:08:51] right, but that's supposed to be a 3rd param, not a .then [22:09:37] hmm weird ejegg, locally that's been working and seemed to fix the error i saw [22:09:50] ??? [22:09:59] i know i just cleared my caches too... [22:10:02] do you have another patch in there to swap data.js over to promises? [22:10:08] nope [22:10:09] though [22:10:20] the code base should be consistent [22:10:47] i wonder whether we should creating a function in persistence.js for data.js to call [22:10:54] ok, i'm seeing 'POOL_CLOSED' in there [22:11:00] so that we limit where the mysql calls are made [22:11:14] and make sure it's consistent [22:13:38] i guess we could... [22:16:23] mepps should we be using a pool and connection.release() instead? [22:17:02] I'm wondering if we have a couple different codepaths sharing connections [22:17:15] the errors i'm seeing would indicate that [22:18:55] i tried to use one earlier actually but was having trouble getting it working, but i can try again [22:19:12] yep, mysql-promise uses pools under the hood [22:20:04] connection.release() doesn't work with the way we've set it up though [22:20:11] oh? [22:20:15] the function doesn't exist [22:21:06] ah dang [22:21:39] i'm signing off in 10 ejegg but i can keep looking at this tomorrow [22:21:42] ohhh, and mysql-promise's 'end' function actually closes down the whole pool! [22:22:27] huh, actually, looks like mysql-promise calls release() on connections after it's done with each call [22:22:38] maybe we just need to limit the pool size? [22:24:06] default connection limit is 10 [22:24:22] cwd do you know if we were seeing more than 10 connections? [22:24:49] ahh, so all the dangling connections are probably from data.js! [22:24:52] ejegg: yeah definitely more than 10 [22:25:19] ejegg ahhh [22:25:39] mepps I bet if we do like you suggest and have data.js call a fn in persistence.js to run the query [22:25:48] we can roll back all the .end() stuff [22:25:53] awesome ejegg [22:25:57] and just rely on that pool [22:26:02] makes sense [22:26:10] i can do that tomorrow [22:26:14] rockin [22:27:13] i disabled 802.11n and wifi already seems more reliable... [22:27:18] (Abandoned) Ejegg: Avoid closing connections prematurely [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/397904 (owner: Ejegg) [22:27:49] cwd on the 26 firmware? or still using 31 ? [22:30:24] wow nevermind it was just an errant 2 minutes of adequate performance [22:30:33] :P [22:30:52] i added the .26 but don't actually know what it's using [22:31:15] cwd oh, right, you'd probably need to move anything later out of the way for it it use that one [22:31:28] this is insane [22:31:39] i thought it couldn't possibely get worse than my xps [22:31:43] sounds frustrating to say the least [22:31:43] this is like 5x worse [22:32:11] restart network manager, reload driver, restart router, restart laptop, ad infinitum [22:32:26] after 10 minutes it will connect for 2 minutes [22:32:44] anyway i'll try rolling back the firmware [22:42:18] yeah, it's been rock solid for me since I got it working after install [22:42:27] (knock wood) [23:06:50] (PS1) Eileen: Add extension to change contact types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/398176 (https://phabricator.wikimedia.org/T156193) [23:08:04] Fundraising Sprint Kickstopper, Wikimedia-Fundraising-CiviCRM, FR-Paypal, FR-WMF-Audit: Follow up with Paypal on audit regeneration, enable parser - https://phabricator.wikimedia.org/T167828#3836095 (Popsicleface048) Open>declined [23:09:38] network-manager pegged cpus and froze computer [23:12:04] (CR) jerkins-bot: [V: -1] Add extension to change contact types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/398176 (https://phabricator.wikimedia.org/T156193) (owner: Eileen) [23:13:29] Fundraising Sprint Vaporwerewolf, Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: New Civi Import - https://phabricator.wikimedia.org/T172423#3836148 (jkim_wikimedia) @Eileenmcnaughton Thanks so much- looks great! Will let you know h... [23:13:41] Fundraising Sprint Kickstopper, Wikimedia-Fundraising-CiviCRM, FR-Paypal, FR-WMF-Audit: Follow up with Paypal on audit regeneration, enable parser - https://phabricator.wikimedia.org/T167828#3836154 (Reedy) declined>Open [23:15:53] gonna wander a bit [23:25:52] that was another fun solid hour of trying to connect to wifi [23:26:02] changed firmware, same behavior [23:26:49] not sure what to do at this point, guess i'll ask OIT [23:31:05] aw man... no chance it's the router? [23:32:06] nah same behavior with every network [23:32:20] even my phone which is sitting next to the computer [23:58:25] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice device selector checkboxes look all weird - https://phabricator.wikimedia.org/T168910#3836251 (Pcoombe) @AndyRussG Can you take a look at this one please? It's annoying since the labels now have full width, and people are accidentally...