[01:07:59] PROBLEM - check_gcsip on payments1002 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [01:08:00] PROBLEM - check_gcsip on payments1004 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [01:10:18] RECOVERY - check_gcsip on payments1004 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.439 second response time [01:10:20] RECOVERY - check_gcsip on payments1002 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.418 second response time [11:22:45] hi fr-tech! [11:25:34] (PS25) Mepps: Orphan Slayer Module, getting expected error message [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 [11:43:01] (PS2) Mepps: Clean up duplicate curlWrapper stuff [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/373690 (owner: Ejegg) [11:43:06] (CR) Mepps: [C: 2] Clean up duplicate curlWrapper stuff [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/373690 (owner: Ejegg) [11:44:06] (Merged) jenkins-bot: Clean up duplicate curlWrapper stuff [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/373690 (owner: Ejegg) [11:45:26] (PS3) Mepps: Always clear all database statics in tearDown [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376542 (owner: Ejegg) [11:45:29] (CR) Mepps: [C: 2] Always clear all database statics in tearDown [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376542 (owner: Ejegg) [11:46:20] (Merged) jenkins-bot: Always clear all database statics in tearDown [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376542 (owner: Ejegg) [12:28:58] ejegg/XenoRyet looks like we got a bunch of failmail last night related to the paypal listener [12:37:05] (PS26) Mepps: Orphan Slayer Module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/370225 [12:46:30] (CR) Mepps: Add __toString method for PaymentTransactionResponse (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376585 (owner: Ejegg) [13:00:40] (CR) Mepps: Update SmashPig, don't list php-queue as dep (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376717 (https://phabricator.wikimedia.org/T133556) (owner: Ejegg) [13:02:03] (PS2) Mepps: Finally set a version number - 0.5 [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376775 (https://phabricator.wikimedia.org/T133556) (owner: Ejegg) [13:02:15] (CR) Mepps: [C: 2] "I think we're ready for a version half number!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376775 (https://phabricator.wikimedia.org/T133556) (owner: Ejegg) [13:04:25] (PS3) Mepps: Fix a couple auto-inserted braces, break lines [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376306 (owner: Ejegg) [13:04:29] (CR) Mepps: [C: 2] Fix a couple auto-inserted braces, break lines [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376306 (owner: Ejegg) [13:28:50] (Merged) jenkins-bot: Finally set a version number - 0.5 [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376775 (https://phabricator.wikimedia.org/T133556) (owner: Ejegg) [13:31:07] (Merged) jenkins-bot: Fix a couple auto-inserted braces, break lines [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376306 (owner: Ejegg) [13:39:55] (CR) Mepps: "Ejegg this looks fine but there are some big merge conflicts with master that I wasn't sure how to resolve." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 (owner: Ejegg) [13:43:28] (PS4) Mepps: Short array syntax [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 (owner: Ejegg) [13:44:03] (CR) Mepps: "Tried rebasing, just not sure if I picked the right commit hash." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 (owner: Ejegg) [14:17:49] hi mepps! [14:17:53] Thanks for all the CR [14:18:04] and yeah, that PayPal stuff is annoying [14:18:45] I checked with cwd and Jeff_Green, and our firewall should be allowing access to all the IPs we were trying to contact for the IPN verification [14:19:00] so it looks like the problem is on PayPal's side [14:21:18] Okay cool ejegg [14:21:30] I'll open another ticket with them [14:25:32] (PS5) Ejegg: Short array syntax [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 [14:31:03] (PS11) Ejegg: Move FinalStatus enum to SmashPig, add ValidationAction [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/317653 (https://phabricator.wikimedia.org/T163868) [14:48:51] (PS6) Ejegg: Push PaymentResult down into SmashPig [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/333152 (https://phabricator.wikimedia.org/T163868) [14:48:53] (PS1) Ejegg: Move PaymentError and ValidationError to SmashPig [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/377279 (https://phabricator.wikimedia.org/T163868) [14:52:12] (CR) Ejegg: "Rebase looked good, just needed composer.lock updated for the new content hash. And man, Zuul is crazy backed up today!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/376307 (owner: Ejegg) [14:54:12] (CR) Ejegg: "response inline" (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376717 (https://phabricator.wikimedia.org/T133556) (owner: Ejegg) [14:59:39] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 9 others: [Spike] investigate contribution tracking data (was Engage import fail... - https://phabricator.wikimedia.org/T146295#3597050 [15:14:03] (PS5) Ejegg: Update SmashPig, don't list php-queue as dep [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376717 (https://phabricator.wikimedia.org/T133556) [15:18:32] (PS10) Ejegg: Upgrade to new minfraud Composer package [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/375454 (https://phabricator.wikimedia.org/T128902) [15:19:45] (PS6) Ejegg: Optionally send more Minfraud parameters [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/275043 (https://phabricator.wikimedia.org/T128902) [15:28:15] (PS2) Ejegg: WIP allow alternate configurations with 'variant' option [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) [15:37:38] (PS3) Ejegg: WIP allow alternate configurations with 'variant' option [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) [15:55:18] (CR) Mepps: [C: 2] Update SmashPig, don't list php-queue as dep [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376717 (https://phabricator.wikimedia.org/T133556) (owner: Ejegg) [16:01:09] (Merged) jenkins-bot: Update SmashPig, don't list php-queue as dep [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376717 (https://phabricator.wikimedia.org/T133556) (owner: Ejegg) [16:02:50] woohoo, version numbers! [16:29:36] (CR) Mepps: [C: 2] Move FinalStatus enum to SmashPig, add ValidationAction [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/317653 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [16:37:52] (Merged) jenkins-bot: Move FinalStatus enum to SmashPig, add ValidationAction [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/317653 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [16:54:54] (PS4) Ejegg: WIP allow alternate configurations with 'variant' option [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) [17:25:16] (PS5) Ejegg: Allow alternate configurations with 'variant' option [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) [17:25:51] fr-tech ^^^ seems to let me A/B test forms without city and state [17:26:14] ejegg i'll take a look! [17:26:42] but I'm a little bummed that it moves us away from the grand unified configuration [17:27:08] Anyway, we weren't going to move all that config into SmashPig in the next couple months [17:46:57] (PS6) Ejegg: Allow alternate configurations with 'variant' option [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) [17:46:59] (PS1) Ejegg: UI modules are part of the configuration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/377312 (https://phabricator.wikimedia.org/T173869) [17:47:20] Fundraising Sprint Quill Pencil, Fundraising-Backlog, Patch-For-Review, Spike: Spike: how to get back to A/B testing? - https://phabricator.wikimedia.org/T173869#3597847 (Ejegg) a:Ejegg [17:51:07] phew, that zuul queue is up to almost 2 and a half hours [17:51:22] well, I'mm lunch and see how it is after [18:30:16] (CR) Mepps: "This looks okay, feels a little unclear--maybe it's the word variant? See note about spacing. Still waiting for CI." (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) (owner: Ejegg) [18:33:43] (PS7) Ejegg: Allow alternate configurations with 'variant' option [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) [18:35:08] (CR) Ejegg: "Fixed that whitespace. I'd love to make it clearer - what part seems the muddiest right now?" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) (owner: Ejegg) [18:43:07] (CR) Mepps: "The word 'variant' doesn't feel clear to me--it's where you specify an alternate path to a config file?" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/376850 (https://phabricator.wikimedia.org/T151769) (owner: Ejegg) [19:06:06] mepps want to talk through the terminology / documentation in a chat? [19:06:29] ejegg sure! [19:06:42] cool, I'll hop in the usual hangout in a sec [19:16:57] (CR) XenoRyet: [C: 2] Move PaymentError and ValidationError to SmashPig [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/377279 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [19:18:03] (Merged) jenkins-bot: Move PaymentError and ValidationError to SmashPig [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/377279 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [19:18:24] Thanks XenoRyet [19:18:32] No worries [19:18:37] also, good to see zuul is merging stuff [19:18:46] I guess it's just one class of job that's piling up? [19:19:12] Huh, didn't notice it was having issues this morning. [19:22:04] (CR) XenoRyet: [C: 2] Push PaymentResult down into SmashPig [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/333152 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [19:23:06] (Merged) jenkins-bot: Push PaymentResult down into SmashPig [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/333152 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [19:24:34] I think that ticket is finally ready to go into pending deployment [19:24:53] hehe, awesome [19:28:06] ooh, more deadlocks. let's see what's to blame [19:28:48] ejegg: we are running that mailer import thing atm [19:28:56] could contribute? [19:29:13] cwd oh maybe, it was an email insert failure [19:29:35] you mean the importer was running right at the moment of that latest failmail? [19:29:49] well it's been running for a while [19:30:12] 20 minutes or so [19:30:37] ok, that was probably it then [19:30:49] I didn't see anything else running that would have locked stuff up [19:30:52] deadlocks are no good though [19:30:53] and it was only that one [19:31:03] might have just been a lock wait, lemme see [19:31:37] dang, nope, deadlock [19:33:09] i don't know of any way to prevent those without increasing transaction levels [19:40:00] well, I guess we should make sure we're using transactions wisely, and keeping them small [19:40:37] it's a bummer that two inserts block each other [19:40:49] oh right, autoincrement would be the cause of that [19:43:04] the deadlock? [19:44:20] yeah, i thought maybe two inserts shouldn't lock each other out [19:44:25] (CR) XenoRyet: [C: 2] Retry deadlock / lock wait failures [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376569 (https://phabricator.wikimedia.org/T118487) (owner: Ejegg) [19:44:39] if there wasn't another read on the table [19:50:23] (Merged) jenkins-bot: Retry deadlock / lock wait failures [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/376569 (https://phabricator.wikimedia.org/T118487) (owner: Ejegg) [20:11:41] eileen: o/ [20:11:59] cwd hi [20:12:20] how are things in the southern half? [21:48:57] eileen: i have had the pedal to the metal but i can't make it lag: https://grafana.wikimedia.org/dashboard/db/frack-db?orgId=1&from=now-30m&to=now [21:56:21] 1 second behind, huh? [22:03:26] sigh [22:03:58] the results continue to be frustratingly inconsistent [22:04:03] cwd - I have 1.5 hours o meetings [22:04:14] but was going to try again after that [22:04:21] * cwd sympathizes [22:04:32] :-) [22:04:59] we did enable parallel replication on one host, and there was that surprise reboot of the master on saturday [22:05:22] Well I wonder if restarting it means something has to accumulate before it starts again [22:05:45] the number of variables in this situation is too damn high [22:10:42] https://grafana.wikimedia.org/dashboard/db/frack-db?orgId=1&from=now-30m&to=now [22:11:31] same exact thing happened earlier [22:12:01] perhaps network is the bottleneck for 2001, sometimes [22:18:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: import donor survey responses - https://phabricator.wikimedia.org/T175638#3598735 (DStrine) [22:32:05] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: import donor survey responses - https://phabricator.wikimedia.org/T175638#3598815 (DStrine)