[00:04:29] (PS1) Ejegg: Update libs, add php-queue and predis [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311062 (https://phabricator.wikimedia.org/T145419) [00:05:42] (CR) jenkins-bot: [V: -1] Update libs, add php-queue and predis [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311062 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [00:12:13] Gonna try the new rectifier again. [00:17:54] (PS1) Ejegg: Mirror messages to php-queue backend [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) [00:19:09] (CR) jenkins-bot: [V: -1] Mirror messages to php-queue backend [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [00:20:14] fingers crossed [00:20:53] It runs fine, but we're not deleting "complete" transactions from pending :( [00:21:30] huh. or there's a race condition still [00:22:32] (PS2) Ejegg: Update libs, add php-queue and predis [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311062 (https://phabricator.wikimedia.org/T145419) [00:22:34] (PS2) Ejegg: Mirror messages to php-queue backend [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) [00:23:09] select p.order_id, p.date, i.date from pending p join fredge.payments_initial i on p.order_id=i.order_id where p.gateway='globalcollect'; [00:23:09] hmm, smashpig and crm are all up to date? [00:23:15] mebbe not [00:23:31] let me check on smashpig [00:24:11] so these are all from the payments_init record showing up after pending, AFAICT, however there are lots of loglines from that consumer about deleting pending records. [00:24:37] Also weirdly, these are all 10 minutes old [00:25:23] ah nvm about my timing guesses, the pending db takes date from the message itself [00:25:45] should be 'complete', not 'completed', correct? [00:26:34] yep [00:27:01] argh, the PendingQueueConsumer logging is bogus [00:28:16] yeah, needs ids [00:28:25] +1 [00:28:27] Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, and 3 others: Donation import should merge and delete pending db info - https://phabricator.wikimedia.org/T141477#2642490 (awight) Looks like the P... [00:28:31] This is really close, though! [00:28:38] I think that context stuff shows up in syslog, just not in jenkins [00:28:44] yeah [00:28:55] but since we run that one thru jenkins... [00:28:58] fixing [00:29:04] So... we're culling enough of the pending records that I would feel comfortable leaving the new rectifier up... [00:31:13] hey [00:31:41] (PS1) Ejegg: More details for PendingQueueConsumer log [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311071 [00:31:59] Hi Jeff_Green ! [00:32:01] just back from parents night at school, were you guys able to resolve whatever it was you were pinging about earlier? [00:32:11] (haven't read backscroll) [00:32:12] Never mind about that settings testing [00:32:19] ok [00:32:36] we diagnosed with some extra logging, and the problem is totally inexplicable [00:32:49] but it went away when we stopped using APC [00:32:51] awesome :-$ [00:32:58] yeah... [00:35:35] awight your log details: https://gerrit.wikimedia.org/r/311071 [00:39:23] (CR) Awight: [C: 2] More details for PendingQueueConsumer log [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311071 (owner: Ejegg) [00:44:40] (Merged) jenkins-bot: More details for PendingQueueConsumer log [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311071 (owner: Ejegg) [00:45:57] whoa, Portland has a *monthly* naked bike ride? [00:46:12] that's badass [00:46:26] this city, man... [00:47:12] I always thought SF bay was the west pole of oddity. didn't realize how many other directions it goes [00:47:19] Jeff_Green: We're ready to disable the payments1004 orphan rectifier... you wanna do that with cron, or should I use configuration? [00:47:38] ejegg: I had a pretty weird time on my one visit to the PDX [00:48:11] The Greyhound broke down on the way there and I ended up playing chess with scraps of paper against fellow passenger. for 8 hours [00:48:16] things got crazier from there [00:48:21] awight: for tonight can you config-slay it? tomorrow I'll clean it up [00:48:26] prison style! [00:48:27] Jeff_Green: 10-4, thanks! [00:48:43] thx [00:50:40] !log update paymentswiki config to disable legacy orphan rectifier [00:50:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:51:58] ejegg: Should we disable the audit parser and message archiver... [00:52:24] meh they're daily. [00:53:49] !log remove "pending" from AMQ old message consumer [00:53:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:00:20] awjr: oh yeah, I think the audit parser should be able to mirror stuff as soon as we turn that on [01:00:52] sorry awjr, that was meant for awight [01:01:13] * Jeff_Green heading out. have a good night! [01:01:20] good night Jeff_Green ! [01:01:27] suppose I'll head out too [03:02:16] (CR) Legoktm: [C: 2] Add help link to Special:CentralNotice and Special:CentralNoticeBanners/edit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/234831 (owner: Glaisher) [03:07:32] (CR) jenkins-bot: [V: -1] Add help link to Special:CentralNotice and Special:CentralNoticeBanners/edit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/234831 (owner: Glaisher) [07:14:13] Fundraising Sprint Muggle Baiting, Fundraising Sprint Nitpicking, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 7 others: [Epic] Move orphan rectifier out of payments - https://phabricator.wikimedia.org/T131275#2642868 (awight) [07:14:16] Fundraising Sprint Muggle Baiting, Fundraising Sprint Nitpicking, Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, and 10 others: Run the orphan rectifier job from CRM Jenkins - https://phabricator.wikimedia.org/T141487#2500303 (awight) Open>Resolv... [07:30:49] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, FR-Ingenico, MediaWiki-extensions-DonationInterface: New orphan rectifier balks on CVV="0" - https://phabricator.wikimedia.org/T145846#2642893 (awight) [07:32:09] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, FR-Ingenico, MediaWiki-extensions-DonationInterface: New orphan rectifier balks on CVV="0" - https://phabricator.wikimedia.org/T145846#2642893 (awight) [07:33:24] (PS1) Awight: Don't be fooled by CVV="0" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/311098 (https://phabricator.wikimedia.org/T145846) [07:42:31] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Orphan rectifier stores wrong source_* fields - https://phabricator.wikimedia.org/T145847#2642916 (awight) [07:49:08] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Orphan rectifier not sending to payments-initial nor payments-fraud queues - https://phabricator.wikimedia.org/T145848#2642934 (awight) [08:45:50] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: [Tracking] Stop producing to ActiveMQ - https://phabricator.wikimedia.org/T145660#2642986 (awight) [09:14:24] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Unplanned-Sprint-Work: Ingenico audit is broken by queue trickery - https://phabricator.wikimedia.org/T145852#2643046 (awight) [09:15:05] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Unplanned-Sprint-Work: Ingenico audit is broken by queue trickery - https://phabricator.wikimedia.org/T145852#2643059 (awight) p:Triage>Unbreak! [10:01:14] (CR) Awight: [C: 2] Update libs, add php-queue and predis [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311062 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [10:05:16] (CR) Awight: "Just one thing I don't understand...." (1 comment) [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [10:08:56] (Merged) jenkins-bot: Update libs, add php-queue and predis [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311062 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [10:26:38] 10:23:02 INFO:mwextupdate:running: git submodule update --init --recursive [10:26:39] 10:23:44 No submodule mapping found in .gitmodules for path 'vendor/coderkungfu/php-queue' [10:26:39] 10:24:30 Failed to recurse into submodule path 'FundraisingEmailUnsubscribe' [10:27:40] reedy@ubuntu64-web-esxi:/var/www/wiki/mediawiki/extensions/FundraisingEmailUnsubscribe$ git submodule update --init --recursive [10:27:41] fatal: no submodule mapping found in .gitmodules for path 'vendor/coderkungfu/php-queue' [10:42:50] (PS1) Reedy: Add .gitmodules for vendor/coderkungfu/php-queue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311109 (https://phabricator.wikimedia.org/T145859) [10:46:10] Wikimedia-Fundraising, MediaWiki-extensions-Other, Patch-For-Review: FundraisingEmailUnsubscribe has broken git submodules - https://phabricator.wikimedia.org/T145859#2643267 (Reedy) [11:00:06] (PS1) Reedy: Fix mixed spaces and tabs [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311110 [11:08:10] (PS1) Reedy: Revert "Update libs, add php-queue and predis" [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311111 [11:08:12] (PS1) Reedy: Re-do I7c96ff947b852a819e158faf00b13211b24cf06f [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311112 (https://phabricator.wikimedia.org/T145859) [11:08:18] ^ 2 different ways to fix it [11:09:39] (CR) jenkins-bot: [V: -1] Re-do I7c96ff947b852a819e158faf00b13211b24cf06f [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311112 (https://phabricator.wikimedia.org/T145859) (owner: Reedy) [11:10:56] Bleugh, that's the PHP/composer version issue, isn't it? [11:14:10] (PS2) Reedy: Re-do I7c96ff947b852a819e158faf00b13211b24cf06f [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311112 (https://phabricator.wikimedia.org/T145859) [14:20:28] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi search issues related to dedupe? - https://phabricator.wikimedia.org/T145873#2643683 (MBeat33) [15:25:43] (PS6) Glaisher: Add help link to Special:CentralNotice and Special:CentralNoticeBanners/edit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/234831 [15:27:28] (CR) jenkins-bot: [V: -1] Add help link to Special:CentralNotice and Special:CentralNoticeBanners/edit [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/234831 (owner: Glaisher) [15:30:31] (CR) Glaisher: "Failures doesn't seem to be related to this" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/234831 (owner: Glaisher) [16:08:54] (CR) AndyRussG: "> Failures doesn't seem to be related to this" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/234831 (owner: Glaisher) [17:20:38] AndyRussG: about that CPS conversation, would any of the time ranges on doodle work for you? I was going to at least pull you and Adam into these meetings [17:35:05] dstrine: times are in PDT, right? [17:35:16] I hope my response about not hosting stuff like that was OK :) [17:35:49] Do you have any idea what type of monitoring they're working on? [17:37:03] (PS3) Ejegg: Mirror messages to php-queue backend [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) [17:50:25] (PS1) Ejegg: Only send unsubscribe message via PHP-Queue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311165 (https://phabricator.wikimedia.org/T145419) [17:54:36] (PS2) Ejegg: Only send unsubscribe message via PHP-Queue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311165 (https://phabricator.wikimedia.org/T145419) [18:00:47] AndyRussG: yeah those times are in PDT. I think your response was great. They are not sharing anything yet. I think we'll need to start regular calls with them. [18:24:21] (PS3) Ejegg: Only send unsubscribe message via PHP-Queue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311165 (https://phabricator.wikimedia.org/T145419) [19:10:00] (PS1) Ejegg: Add tests for thank you unsubscribe [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311174 (https://phabricator.wikimedia.org/T145419) [19:11:06] (CR) jenkins-bot: [V: -1] Add tests for thank you unsubscribe [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311174 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [19:11:38] (CR) Ejegg: "Thanks awight! There are tests now, in a descendent patch after removing the Stomp send." (1 comment) [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [19:23:19] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, Patch-For-Review: Migrate FundraisingEmailUnsubscribe to new queue backend - https://phabricator.wikimedia.org/T145419#2644358 (XenoRyet) Confirming I'll work on the consumer side. [19:39:22] (Abandoned) Reedy: Add .gitmodules for vendor/coderkungfu/php-queue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311109 (https://phabricator.wikimedia.org/T145859) (owner: Reedy) [19:41:24] (PS3) Reedy: Fix git submodule issue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311112 (https://phabricator.wikimedia.org/T145859) [19:41:54] (Abandoned) Reedy: Fix git submodule issue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311112 (https://phabricator.wikimedia.org/T145859) (owner: Reedy) [19:42:04] (Abandoned) Reedy: Revert "Update libs, add php-queue and predis" [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311111 (owner: Reedy) [19:43:01] (CR) Ejegg: [C: 2] Fix mixed spaces and tabs [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311110 (owner: Reedy) [19:44:02] (Merged) jenkins-bot: Fix mixed spaces and tabs [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311110 (owner: Reedy) [19:51:19] (PS1) Ejegg: Fix unintended submodule [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311176 (https://phabricator.wikimedia.org/T145419) [19:52:52] (PS4) Ejegg: Mirror messages to php-queue backend [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311070 (https://phabricator.wikimedia.org/T145419) [19:54:52] (PS4) Ejegg: Only send unsubscribe message via PHP-Queue [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311165 (https://phabricator.wikimedia.org/T145419) [19:57:42] (PS2) Ejegg: Add tests for thank you unsubscribe [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311174 (https://phabricator.wikimedia.org/T145419) [20:01:02] (CR) Reedy: [C: 2] Fix unintended submodule [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311176 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [20:04:03] (Merged) jenkins-bot: Fix unintended submodule [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/311176 (https://phabricator.wikimedia.org/T145419) (owner: Ejegg) [20:07:15] (PS14) Ejegg: Quit deleting from pending queue, stop saying limbo [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/301630 (https://phabricator.wikimedia.org/T133433) [20:23:56] (PS7) Ejegg: MariaDB strict mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) [20:24:18] AndyRussG: now with 75% more rationale ^^^ [20:24:52] ejegg: cool beans!! thx! [20:24:59] also fr-tech this cleanup patch is finally ready to merge: https://gerrit.wikimedia.org/r/301630 [20:25:33] k, I should stop camping in this coffeehouse. been an hour since I finished my food & beverage [20:27:58] (Abandoned) Ejegg: Live hack: try restoring inst-args for one thing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311039 (owner: Ejegg) [20:30:14] (PS4) Ejegg: jQuery usage tweaks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 [20:31:42] (CR) jenkins-bot: [V: -1] jQuery usage tweaks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg) [20:48:55] (CR) Ejegg: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg) [20:49:46] (CR) Ejegg: [C: 2] Don't be fooled by CVV="0" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/311098 (https://phabricator.wikimedia.org/T145846) (owner: Awight) [20:50:57] (CR) Ejegg: [C: 2] Revert "Live hack to dump configuration" [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311045 (owner: Awight) [20:51:22] (Merged) jenkins-bot: Don't be fooled by CVV="0" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/311098 (https://phabricator.wikimedia.org/T145846) (owner: Awight) [20:51:24] (Merged) jenkins-bot: Revert "Live hack to dump configuration" [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/311045 (owner: Awight) [20:52:55] (CR) Ejegg: [C: -1] "Let's put the composer.lock change in master" [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/310904 (owner: Awight)