[00:00:13] (PS10) Eileen: WIP [alphaish] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) [00:04:29] (CR) jerkins-bot: [V: -1] WIP [alphaish] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:06:29] (PS1) Eileen: Switch to phpleage csv Writer package. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467847 (https://phabricator.wikimedia.org/T204008) [00:13:13] (CR) jerkins-bot: [V: -1] Switch to phpleage csv Writer package. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467847 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:16:51] (CR) Ejegg: [C: 1] "Looks good, but it might change behavior on hitting an empty row. Not sure if we get many of those..." (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:22:34] (CR) Ejegg: "Looks good, just one place to remove the param" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467569 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:23:24] ejegg|afk: I’m looking at the bit where I switch to using the csv writer - https://csv.thephpleague.com/ & it’s crashing on the fclose bit - I can’t see anything in their docs to encourage it & wondering whether it is necessary - it seems to be ‘good practice’ but only really matters if you finish writing the csvs materially before the end of script [00:26:24] (PS7) Eileen: Switch to csv package [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) [00:27:01] (CR) Eileen: "Hmm I made that change because of a test fail but I just undid it & the test passed - have removed & resubmitted to see what zuul thinks" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:30:13] (CR) jerkins-bot: [V: -1] Switch to csv package [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:34:35] (CR) Ejegg: "d'oh! The !== [] check is all that we can use, I guess. It'll never be false. Reader.csv's fetchOne returns" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:36:00] eileen even in php 7.2 var_dump( (array) NULL ) gives you array(0) { } [00:36:04] (CR) Eileen: "TBH the empty array handling isn't necessary when I add the batching later one" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:36:33] ejegg: the empty array handling is actually not required once I get to the full batching [00:36:44] eileen: ah, right on [00:36:55] I’ll put it back for now [00:37:04] for this patch, might as well just replace the !== FALSE with !== [] [00:37:09] but the final version starts by getting a row count [00:37:18] since (array) $iterator->current() will never return FALSE [00:41:27] (PS8) Eileen: Switch to csv package [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) [00:41:37] ejegg: ok switched to that [00:43:02] (PS5) Eileen: Import code tweak - Use headers property rather than passing headers variable [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467569 (https://phabricator.wikimedia.org/T204008) [00:44:28] (CR) Ejegg: [C: 2] Switch to csv package [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463195 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:44:33] (PS2) Eileen: Drupal code formatting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467827 [00:44:58] (CR) Eileen: "ejegg I fixed that param" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467569 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:45:55] nice, fetchOne() survives into Reader v9.0 [00:46:49] (CR) Ejegg: [C: 2] Import code tweak - Use headers property rather than passing headers variable [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467569 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:47:21] (CR) Ejegg: [C: 2] Drupal code formatting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467827 (owner: Eileen) [00:47:40] (PS4) Ejegg: Extract the minor functions that mark the outcomes of the row processing. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467595 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:47:46] (CR) Ejegg: [C: 2] Extract the minor functions that mark the outcomes of the row processing. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467595 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:50:50] ejegg: yeah that’s kind of a hard thing about using more packages - they tend to be a bit more tied to php versions [00:51:24] so CiviCRM, for instance, can’t use this csv package I don’t think as there isn’t one release that supports all the php versions civi supports [00:51:46] & civi is stuck with 5.6 support for quite a while based on stats [00:52:19] (PS5) Ejegg: Handle NULL for url rather than expect a duff value [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467597 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:52:26] (CR) Ejegg: [C: 2] Handle NULL for url rather than expect a duff value (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467597 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [00:53:22] (PS4) Eileen: Extract setMessages function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467596 (https://phabricator.wikimedia.org/T204008) [00:54:42] (PS3) Eileen: Do our validation & header instantiation in construct, as long as we have the field uri [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467833 (https://phabricator.wikimedia.org/T204008) [00:54:56] (PS3) Eileen: Instantiate all output files at the start. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467832 (https://phabricator.wikimedia.org/T204008) [00:55:05] (PS2) Eileen: Use function to write csv rows. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467845 (https://phabricator.wikimedia.org/T204008) [00:58:02] ejegg: what do you think about the importance of the fclose - it’s failing tests here https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/467847/ & it’s not that obvious how to do it in phpleague writer - which makes me think they don’t expect you to [01:00:48] huh, but it's failing the tests? Like they check for files left open? [01:01:50] ejegg: no - it’s failing because it’s trying to close a file but we now have writer objects instead [01:04:13] ah, in closeFilesAndSetMessages? [01:05:36] eileen: looks like the underlying SplFileObject closes itself on destruct [01:05:45] so yeah, all taken care of for us [01:06:01] ok I am pretty sure only tests would be hurt if there was an issue [01:07:54] (PS2) Eileen: Switch to phpleage csv Writer package. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467847 (https://phabricator.wikimedia.org/T204008) [01:08:29] (CR) jerkins-bot: [V: -1] Switch to phpleage csv Writer package. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467847 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [01:09:30] (PS5) Eileen: Extract setMessages function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467596 (https://phabricator.wikimedia.org/T204008) [01:09:32] (PS4) Eileen: Do our validation & header instantiation in construct, as long as we have the field uri [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467833 (https://phabricator.wikimedia.org/T204008) [01:09:34] (PS4) Eileen: Instantiate all output files at the start. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467832 (https://phabricator.wikimedia.org/T204008) [01:09:36] (PS3) Eileen: Use function to write csv rows. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467845 (https://phabricator.wikimedia.org/T204008) [01:09:38] (PS3) Eileen: Switch to phpleage csv Writer package. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467847 (https://phabricator.wikimedia.org/T204008) [01:13:26] (PS11) Eileen: WIP [alphaish] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) [01:14:59] (PS12) Eileen: WIP [alphaish] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) [01:17:37] (PS1) Ejegg: Uncancel more PayPal recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467851 (https://phabricator.wikimedia.org/T206343) [01:22:13] (CR) jerkins-bot: [V: -1] Switch to phpleage csv Writer package. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467847 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [01:22:15] (PS1) Eileen: Switch to permitting rowCount & offset in import function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467852 (https://phabricator.wikimedia.org/T204008) [01:22:20] (CR) jerkins-bot: [V: -1] WIP [alphaish] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [01:28:44] (CR) jerkins-bot: [V: -1] Switch to permitting rowCount & offset in import function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467852 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [01:36:04] (PS2) Eileen: Switch to permitting rowCount & offset in import function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467852 (https://phabricator.wikimedia.org/T204008) [01:37:10] (PS4) Eileen: Switch to phpleage csv Writer package. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467847 (https://phabricator.wikimedia.org/T204008) [01:38:08] (PS3) Eileen: Switch to permitting rowCount & offset in import function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467852 (https://phabricator.wikimedia.org/T204008) [01:38:48] (PS1) Eileen: Apply drupal formatting to RefundFile [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467854 [01:47:29] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/467857 [01:48:39] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/467857 (owner: Ejegg) [01:50:01] (PS13) Eileen: WIP [alphaish] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) [01:54:14] (CR) jerkins-bot: [V: -1] WIP [alphaish] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [01:56:08] (PS1) Ejegg: Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467858 [01:56:11] (CR) Ejegg: [C: 2] Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467858 (owner: Ejegg) [01:59:20] (CR) jerkins-bot: [V: -1] Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467858 (owner: Ejegg) [02:02:00] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/467857 (owner: Ejegg) [02:02:02] (CR) jerkins-bot: [V: -1] Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467858 (owner: Ejegg) [02:03:19] (CR) Ejegg: [C: 2] Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467858 (owner: Ejegg) [02:06:06] (Merged) jenkins-bot: Update DonationInterface [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467858 (owner: Ejegg) [02:10:31] !log updated payments-wiki from 7fb1aae963 to 0385ad02a7 [02:10:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:29:51] Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, and 3 others: Invalid locale sent to new Ingenico API - https://phabricator.wikimedia.org/T206093 (Ejegg) Thanks for pointing that out, @Jgleeson!... [02:30:37] good night eileen. I'll pick up on the CR in the morning [03:10:53] (PS1) Eileen: Make output files batch friendly. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467864 (https://phabricator.wikimedia.org/T204008) [03:29:58] Fundraising Sprint Pluto is still a planet, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Sasquatches can't find us either, and 3 others: Create opt-IN page for fundraisingEmailUnsubscri... - https://phabricator.wikimedia.org/T200576 [04:41:59] (PS14) Eileen: Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) [04:46:09] (CR) jerkins-bot: [V: -1] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [04:50:21] (PS15) Eileen: Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) [04:54:32] (CR) jerkins-bot: [V: -1] Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) (owner: Eileen) [04:59:41] (PS16) Eileen: Functional import using batch api. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463898 (https://phabricator.wikimedia.org/T204008) [07:11:18] (PS3) Hashar: Augment precision of updatelist time [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467685 (https://phabricator.wikimedia.org/T207241) [08:28:01] fundraising-tech-ops, Patch-For-Review: rename saiph.frack.codfw.wmnet to frpig2001.frack.codfw.wmnet and reimage with Debian Stretch - https://phabricator.wikimedia.org/T203521 (ayounsi) [09:46:11] (PS5) Jgleeson: Handle invalid GC SET_PAYMENT response and fix some xml formatting issues in stub responses. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/458208 (https://phabricator.wikimedia.org/T203126) (owner: Ejegg) [09:55:06] (CR) Jgleeson: Handle invalid GC SET_PAYMENT response and fix some xml formatting issues in stub responses. (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/458208 (https://phabricator.wikimedia.org/T203126) (owner: Ejegg) [13:31:26] (PS15) Jgleeson: WIP: Backend Stats Comparison Script [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/456016 (https://phabricator.wikimedia.org/T198752) [13:31:40] changing workspace as battery is running low! [14:41:03] hey AndyRussG :) [14:41:13] jgleeson: morning! [14:41:41] hows it goin? [14:41:59] eh not tooo bad [14:42:18] I'm sad because yesterday evening I cooked a whole batch of rice for two meals with the kids [14:42:23] (I get them today after school) [14:42:27] and I overcooked it [14:42:37] so now I have a lot of mushy rice in the fridge [14:42:56] other than that everything's okok [14:42:59] how are u? [14:44:25] ah nooo [14:44:33] can it be saved somehow? [14:44:50] I guess it hardens when chilled no? [14:45:02] might remove some of the moisture if then heated in a pan [14:45:49] I used to make paella a lot and have a similar problem [14:46:49] I'm good, still getting use to 28c in october and how to ask for things in greek, so every day is a learning experience [14:49:34] jgleeson: sounds like an adventure! [14:49:36] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: can't save any changes to specific records' prospect tabs - https://phabricator.wikimedia.org/T206242 (CaitVirtue) I can't reproduce this consistently anymore either (I could last week, but can't this week). @DStrine in that case, is it best to close i... [15:02:26] fr-tech any news to share for scrum of scrums? [15:02:35] none from me! [15:02:48] nick jgleeson|away [15:04:01] ejegg: nothing her, thx! [15:04:16] k, cool [15:07:49] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal: subscr_modify messages lead to failmail - https://phabricator.wikimedia.org/T206650 (Ejegg) I notice that in the Civi code, the subscr_modify type is commented out, with the note that 'at the moment, we are not accepting modification mess... [15:08:34] Fundraising-Backlog: Make sure audit parsers don't run at the same time as the disk array check - https://phabricator.wikimedia.org/T206571 (Ejegg) p:Triage>High [15:09:18] Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog: Make sure audit parsers don't run at the same time as the disk array check - https://phabricator.wikimedia.org/T206571 (Ejegg) Pulling this into sprint to make sure it gets done before the start of next month [15:14:41] Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Delay in French mobile banners showing up in Banner Allocation - https://phabricator.wikimedia.org/T206562 (AndyRussG) Checked a couple more thi... [15:37:50] Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Delay in French mobile banners showing up in Banner Allocation - https://phabricator.wikimedia.org/T206562 (AndyRussG) **Summary** What this is... [15:44:18] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal: subscr_modify messages lead to failmail - https://phabricator.wikimedia.org/T206650 (Ejegg) hmm, the commented-out code actually looks pretty decent. Might be OK to add a test and un-comment it. That might also solve T146309. [15:55:06] ejegg: are you able to attend the tech priorities meeting today? They are super interested in the es-419 task [15:55:23] dstrine: yep, I'll be there [15:55:29] thanks! [16:17:50] (CR) Hashar: "check experimental" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/459765 (https://phabricator.wikimedia.org/T203084) (owner: Hashar) [16:22:25] (CR) jerkins-bot: [V: -1] Test merge plugin prepend-repositories [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/459765 (https://phabricator.wikimedia.org/T203084) (owner: Hashar) [16:32:01] Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog, MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), Patch-For-Review: Payment form amount field improvements - https://phabricator.wikimedia.org/T206873 (Ejegg) Open>... [16:44:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig: Deal with PayPal messages with old_subscr_id - https://phabricator.wikimedia.org/T207304 (Ejegg) [16:54:14] (PS4) Hashar: Augment precision of updatelist time [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467685 (https://phabricator.wikimedia.org/T207241) [16:57:39] (PS1) C. Scott Ananian: Replace deprecated untidy OutputPage::addWikiText() method [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/468031 (https://phabricator.wikimedia.org/T198214) [17:43:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 10923 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 112 days - memory use is 9.64M (peak 15.51M, 0.18% of max, fragmentation 1.55%), connected_slaves is 2, donations is 238, jobs is 0, jobs-adyen is 6, jobs-paypal is 26, payments-antifraud is 176, payments-init is 191, pending is 90, refund is 0, unsubscribe is 22 [17:43:36] hmmmm [17:43:40] ejegg: more paypal fail? [17:48:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 10927 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 112 days 5 minutes - memory use is 9.49M (peak 15.51M, 0.17% of max, fragmentation 1.45%), connected_slaves is 2, donations is 60, jobs is 0, jobs-adyen is 3, jobs-paypal is 39, payments-antifraud is 130, payments-init is 147, pending is 70, refund is 0, unsubscribe is 23 [17:48:27] fack, 11k ? [17:53:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 10936 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 112 days 10 minutes - memory use is 9.49M (peak 15.51M, 0.17% of max, fragmentation 1.46%), connected_slaves is 2, donations is 262, jobs is 0, jobs-adyen is 12, jobs-paypal is 24, payments-antifraud is 59, payments-init is 72, pending is 74, refund is 0, unsubscribe is 26 [17:54:13] ejegg: aren't there banners up? doesn't donations look a little low for that? [17:54:19] possible they are going to recurring by mistake? [17:57:40] cwd it was just a one hour test [17:58:11] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 10943 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 112 days 15 minutes - memory use is 9.56M (peak 15.51M, 0.16% of max, fragmentation 1.35%), connected_slaves is 2, donations is 32, jobs is 0, jobs-adyen is 16, jobs-paypal is 35, payments-antifraud is 173, payments-init is 201, pending is 82, refund is 0, unsubscribe is 29 [17:58:15] and there's a new paypal audit file with more than 13k bogus cancellations [17:58:36] so... maybe we shut off paypal audit processing till they can fix their file generation? [17:58:45] anyway, let's ack that warning for now [18:00:22] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: recurring is 10943 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 112 days 15 minutes - memory use is 9.56M (peak 15.51M, 0.16% of max, fragmentation 1.35%), connected_slaves is 2, donations is 32, jobs is 0, jobs-adyen is 16, jobs-paypal is 35, payments-antifraud is 173, payments-init is 201, pending is 82, refund is 0, unsubscri [18:00:22] leston Bad audit file from paypal [18:50:22] (CR) Jgleeson: [V: 1 C: 1] "Looks good to me although not sure if I can test it locally without setting up some broken records?" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467851 (https://phabricator.wikimedia.org/T206343) (owner: Ejegg) [18:51:08] ejegg, happy to +2 that once you've confirmed processor_id 1 is paypal [18:57:16] thanks jgleeson, let me just take a second look [18:57:53] ah, I should tack on the records from today's audit file too [18:57:59] (CR) Hashar: "REL1_27 patch got approved ( https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/467895/ ) :]" [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/467685 (https://phabricator.wikimedia.org/T207241) (owner: Hashar) [18:58:49] jgleeson: ooh, actually, I don't think we WANT to set that processor_id [18:58:57] no idea why I had that in there :( [18:59:23] ah ok [18:59:34] (CR) Hashar: [C: 1] "The patch does fix the issue we encountered with amazon/login-and-pay being in a different VCS." [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/459762 (https://phabricator.wikimedia.org/T203084) (owner: Hashar) [18:59:37] guess I copy/pasted it? [19:00:38] cool [19:00:44] for me I have adyen as processor ID 1 [19:00:58] so I just wanted to double check whether or not it was a local/prod env difference [19:01:05] I guess I could of checked on frdev [19:01:19] either way, looks like you don't need it [19:01:20] right, if we did want to set that it would be best to look it up by name, anyway [19:01:29] like we do with the status IDs [19:01:33] oops, meeting time [19:01:45] (PS2) Ejegg: Uncancel more PayPal recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467851 (https://phabricator.wikimedia.org/T206343) [19:01:53] anyway, there it is without the processor_id update [19:02:03] cool, will +2 [19:03:15] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 112 days 1 hours - memory use is 2.46M (peak 15.51M, 0.08% of max, fragmentation 2.70%), connected_slaves is 2, donations is 160, jobs is 0, jobs-adyen is 9, jobs-paypal is 12, payments-antifraud is 134, payments-init is 140, pending is 22, recurring is 245, refund is 0, unsubscribe is 1 [19:06:07] (PS1) Ejegg: Add the es-419 language code to support South American Spanish [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/468067 (https://phabricator.wikimedia.org/T112889) [19:08:01] jgleeson: ah, I could add the records from today too [19:08:06] one sec, I'll do that [19:10:48] (PS3) Ejegg: Uncancel more PayPal recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467851 (https://phabricator.wikimedia.org/T206343) [19:11:34] ok, looks like the bogus cancels are all clustered in time, at least [19:15:57] ok cool [19:37:06] (CR) Jgleeson: [C: 2] Uncancel more PayPal recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467851 (https://phabricator.wikimedia.org/T206343) (owner: Ejegg) [19:43:48] (Merged) jenkins-bot: Uncancel more PayPal recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/467851 (https://phabricator.wikimedia.org/T206343) (owner: Ejegg) [19:45:05] (CR) Jgleeson: [C: 2] "Looks good and nice work on the cli commands to pull in all the locales to map! test also passing for me." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/466930 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [19:45:52] (CR) Hashar: "check experimental" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/459765 (https://phabricator.wikimedia.org/T203084) (owner: Hashar) [19:47:17] (Merged) jenkins-bot: Special fallback logic for IngenicoLocale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/466930 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [19:49:43] (CR) jerkins-bot: [V: -1] Test merge plugin prepend-repositories [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/459765 (https://phabricator.wikimedia.org/T203084) (owner: Hashar) [19:53:01] g'night fr-tech [20:05:21] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468081 [20:05:24] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468081 (owner: Ejegg) [20:06:08] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468081 (owner: Ejegg) [20:10:51] !log updated fundraising CiviCRM from 4cc21d61c5 to 5eac0634e6 [20:10:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:47:41] ejegg: did you get all the review you needed? [20:50:20] eileen_: thanks for asking! [20:50:27] I think there's one patch still needing it [20:50:42] (in crm anyway) [20:50:48] https://gerrit.wikimedia.org/r/465546 [20:51:39] looks like you got the batch API stuff pretty well nailed down [20:51:43] cool I’ll take a look [20:51:54] yeah I’m happy with the batch api stuff now [20:52:24] coming back to it I could understand better how the api worked & why the passing between batches is a bit tricksy [20:52:53] that 'context' has to remember a lot of stuff! [20:53:09] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/468088 [20:53:27] yeah I didn’t initially grok the fact that is has to be stored to a table between batch runs [20:53:47] eileen_: ah right, only the serializable stuff survives [20:53:49] One nice thing I found is that you can actually download the csvs while it’s running without it falling over [20:54:00] oh? [20:54:23] these are the ones you're writing with the phpleague package? [20:54:57] Yeah - but it’s the fact that a) batch api allows you to have progress messages that update & b) opening a csv file from a link doesn’t go to a new page [20:57:30] right on [20:57:52] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/468088 (owner: Ejegg) [20:58:24] drupal security release may be about to drop https://www.drupal.org/psa-2018-10-17 [20:58:38] thanks for the heads up [20:59:05] yeah - someone just posted that in civicrm drupal channel [20:59:19] hmm, we should put fr-tech on a mailing list for announcements like that [21:00:11] so that's 3 hrs from now [21:00:17] I'll be getting out of class then [21:04:40] ejegg: I’ll keen an eye out for it [21:05:27] ejegg: just looking at this smashpig on & wondering if the keys smash pig expects are documented anywhere [21:06:19] eileen_: they should be the same as the format of the queue messages [21:06:28] https://wikitech.wikimedia.org/wiki/Fundraising/Queue_messages [21:06:46] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/468088 (owner: Ejegg) [21:06:47] (whose documentation should at least be moved over, and probably updated) [21:07:35] ip address isn’t actually on that list [21:08:27] It would be good to have those docs in code & perhaps generated to somewhere else [21:09:03] oh, the queue / smashpig format should be user_ip [21:09:23] heh, which is missing [21:09:47] :-) [21:13:42] hah this is funny - out driveway needs more gravel as it’s pretty rutted & the neighbour just popped over & offered us some. His neighbour ordered too much & gave him the extra but it was more than he needed so we are in the hand-me-down chain :-) [21:14:58] sweet, hand-me-down rocks! [21:15:49] lol [21:16:34] I kinda want to get it concreted but it would cost quite a bit so topping up the rocks also makes sense [21:17:10] (it’s longer than a ‘town driveway’) [21:18:23] ejegg: so update https://wikitech.wikimedia.org/wiki/Fundraising/Normalized_donation_messages with user_ip - which section - “Recurring other gateway donations”? [21:19:38] nah, it's just a feature of the donations queue generally [21:19:46] (CR) Eileen: [C: 2] "seems all correct although we should" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/465546 (https://phabricator.wikimedia.org/T202783) (owner: Ejegg) [21:22:19] (CR) Eileen: [C: 2] Round things to the right number of places [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463365 (owner: Ejegg) [21:25:02] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/468097 [21:25:04] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/468097 (owner: Ejegg) [21:26:00] (Merged) jenkins-bot: Send user_ip with recurring payments [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/465546 (https://phabricator.wikimedia.org/T202783) (owner: Ejegg) [21:27:28] (CR) jerkins-bot: [V: -1] Round things to the right number of places [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463365 (owner: Ejegg) [21:27:52] hmm, that's funny [21:34:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig: Deal with PayPal messages with old_subscr_id - https://phabricator.wikimedia.org/T207304 (Ejegg) OK, I just looked up all the messages from this month with had old_subscr_id, and we're not storing a reference to the new subscr_id for an... [21:34:58] ejegg: funny? [21:35:42] ehh, probably more db-related crap [21:35:48] looking on the machine [21:35:55] heck I don’t even think I have an account on wikitech [21:35:59] I brought that test issue up in scrum of scums today [21:36:35] hopefully they'll implement your suggestion about the file per table [21:37:05] well, the sql partition is only at 40% use on that machine, so it's not the issue [21:38:31] ejegg: I found something out with the import tests - see comment at line 239 https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/463898/16/sites/all/modules/offline2civicrm/ChecksFile.php [21:39:03] the time limit? [21:39:06] basically someone added that set_time_limit(0); at some time in the past seemingly to undo the set_time_limit(30); but it makes time out infinite [21:39:06] interesting [21:39:13] which is OK in prod but in our tess [21:39:14] tests [21:39:23] they actually only pass because of it [21:39:46] ie. if you take that line away the tests fail because time out got set to 30 [21:40:11] I’m kinda trying not to think too deeply about it because I don’t want to wind up having to do something… [21:40:12] ooh, so phpunit does its own time limit extension [21:40:27] but then we try to 'extend' it in the importer [21:40:36] it either has one - or we create one because of our attempts to extend it [21:40:44] which make a lot of sense for live.. [21:41:08] ehh, dumb code, but if'n you don't care to touch it now, i'm happy looking the other way [21:41:21] the other way from what :-) [21:41:42] exactly [21:41:58] ok whinge moment [21:42:21] my pet hate of gerrit compared to github is I can’t put screenshots on my prs [21:43:05] heh, you think you need screenshots to impress us? [21:43:20] orrr, i guess they can be illustrative :) [21:43:37] yeah the latter [21:43:50] I just added a bit of text to the form & would add a screenshot [21:45:57] ah crud, looks like that un-cancel script didn't do the trick [21:46:05] :-( [21:47:18] oh silly me, I didn't actually parameterize the dates in the query [21:47:26] so it just re-ran the last ones [21:49:07] (PS1) Ejegg: Fix PayPal un-cancel query [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/468172 (https://phabricator.wikimedia.org/T206343) [21:49:20] eileen_: mind taking a quick peep at that? ^^ [21:49:37] ehh, I suppose I should properly parameterize it [21:50:29] (CR) Eileen: [C: 2] Fix PayPal un-cancel query [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/468172 (https://phabricator.wikimedia.org/T206343) (owner: Ejegg) [21:50:33] nah, it's good as it is [21:50:34] :) [21:50:59] ejegg: I guess params are good code but the inputs are trusted here [21:51:05] yeah [21:51:07] so nice to have [21:51:13] but not required [21:51:34] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/468097 (owner: Ejegg) [21:56:12] (Merged) jenkins-bot: Fix PayPal un-cancel query [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/468172 (https://phabricator.wikimedia.org/T206343) (owner: Ejegg) [21:56:52] fr-tech so, we have a tech-talk on the calender in 5 min. Who's around for it? And what would you want to discuss? [21:57:09] I'm in a noisy cafe right now, but I can move to an outdoor table [21:57:45] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468173 [21:57:55] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468173 (owner: Ejegg) [22:02:21] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/468112 (owner: L10n-bot) [22:02:44] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468173 (owner: Ejegg) [22:02:55] :P [22:03:06] (PS1) Eileen: Add a statement about server load to inform import timing decisions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/468174 (https://phabricator.wikimedia.org/T204008) [22:03:22] oops, needs a vendor update! [22:03:41] ah just checking on tech talk - who else is around? [22:04:47] (PS1) Ejegg: Update SmashPig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/468175 [22:04:52] (CR) Ejegg: [C: 2] Update SmashPig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/468175 (owner: Ejegg) [22:05:11] (PS2) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468173 [22:06:25] eileen_: I think it might just be us two [22:06:35] nobody responded to my IRC ping [22:06:40] is there anything you think we should go over? [22:07:26] Err, want to try turning on redis caching in production? [22:07:54] i am here but gotta jet in 5 [22:08:10] will be on again later tonight [22:08:16] cwd ok, I think we can do it without ops-level access [22:08:24] ejegg: yeah that would be cool - what do we need to do in terms of tracking [22:08:28] as long as the creds are available [22:08:29] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468173 (owner: Ejegg) [22:08:36] I think we need to just make sure the q doesn’t slow down [22:09:05] eileen_: ok, let's turn the qc off for a bit and let some build up [22:09:16] ejegg: will we do that twice? [22:09:18] seem to have had an email send recently [22:09:26] ie. before & after turning it on [22:09:38] actually, it's running pretty flat out right now [22:09:45] also I wonder if we can time a user page load [22:10:06] so we can take the current grafana figures as a baseline pre [22:10:36] I feel like we can time page load via curl but we need to somehow manage the logged inny ness [22:10:52] (which is doable I just don’t know how without googling [22:11:08] eileen_: hmm, and the real benefits will come from the multiple requests that usually make up one page load [22:11:17] yeah [22:14:17] hmm ab is described here https://www.drupal.org/node/282862 but that would be logged out speed [22:16:49] (Merged) jenkins-bot: Update SmashPig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/468175 (owner: Ejegg) [22:18:16] (CR) Ejegg: [C: 2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468173 (owner: Ejegg) [22:19:43] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/468173 (owner: Ejegg) [22:20:38] eileen_: ah, good point [22:21:10] just looking at the ab command - with some tinkering we can maybe get it to act as logged in [22:21:37] eileen_: so, we'd also have to do the requests from civi1001 to get around the browser cert issue [22:22:00] ah derp, of course we would run it from civi1001 [22:22:15] ejegg: I was wondering if it would run ok or not from civi1001 [22:22:24] I guess we can test curl on there to see [22:23:20] does running from civi1001 bypass need for a cert? [22:24:07] hmm trying to ping staging from stagin * Hostname was NOT found in DNS cache [22:24:47] !log updated payments-wiki from 0385ad02a7 to a3892e4ed3 [22:24:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:31:07] !log updated fundraising CiviCRM from 5eac0634e6 to 7b8d33bb4e [22:31:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:54:44] (PS1) Ejegg: Deal with old_subscr_id, fix date for cancel [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/468177 (https://phabricator.wikimedia.org/T207304) [22:55:45] Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Smashpig, Patch-For-Review: Deal with PayPal messages with old_subscr_id - https://phabricator.wikimedia.org/T207304 (Ejegg) p:Triage>Normal a:Ejegg [23:16:10] (PS4) Ejegg: Round things to the right number of places [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463365 [23:16:42] ooh, copypaste errors there! [23:17:57] (PS5) Ejegg: Round things to the right number of places [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463365 [23:19:21] (CR) Ejegg: "eileen oops, looks like the tests were right to fail! I had a dumb copypaste error there." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/463365 (owner: Ejegg)