[06:45:09] (CR) jerkins-bot: [V: -1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/598341 (owner: L10n-bot) [06:46:16] (CR) Raimond Spekking: [C: +2] "false positive" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/598341 (owner: L10n-bot) [11:15:05] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: BannerExistenceException due to non-existing CentralNotice banner (after Special:LanguageStats view) - https://phabricator.wikimedia.org/T157997 (Nikerabbit) [11:15:45] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: BannerExistenceException due to non-existing CentralNotice banner (after Special:LanguageStats view) - https://phabricator.wikimedia.org/T157997 (Nikerabbit) [11:16:45] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: BannerExistenceException due to non-existing CentralNotice banner (after Special:LanguageStats view) - https://phabricator.wikimedia.org/T157997 (Nikerabbit) ... [13:21:11] (PS1) Nikerabbit: Speed up BannerMessageGroup's getKeys and getDefinitions. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/598470 (https://phabricator.wikimedia.org/T221119) [14:48:36] (PS19) Ejegg: Use new contribution tracking queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/510757 (https://phabricator.wikimedia.org/T215463) [14:48:44] (PS8) Ejegg: Don't update c_t rows with more than contribution_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/597608 (https://phabricator.wikimedia.org/T253250) [14:56:45] (CR) Jgleeson: [C: +2] Use new contribution tracking queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/510757 (https://phabricator.wikimedia.org/T215463) (owner: Ejegg) [15:02:54] (Merged) jenkins-bot: Use new contribution tracking queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/510757 (https://phabricator.wikimedia.org/T215463) (owner: Ejegg) [15:03:00] (Merged) jenkins-bot: Don't update c_t rows with more than contribution_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/597608 (https://phabricator.wikimedia.org/T253250) (owner: Ejegg) [15:04:08] (PS1) Ejegg: Update ContributionTracking submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/598496 [15:04:45] (CR) Ejegg: [C: +2] Update ContributionTracking submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/598496 (owner: Ejegg) [15:12:12] (CR) jerkins-bot: [V: -1] Update ContributionTracking submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/598496 (owner: Ejegg) [15:12:21] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: DonationInterface should follow PSR-4 - https://phabricator.wikimedia.org/T253569 (Ejegg) [15:12:31] oh man, why did that fail? [15:12:58] dang, phpcs timed out [15:13:12] (CR) Ejegg: [V: +2 C: +2] Update ContributionTracking submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/598496 (owner: Ejegg) [15:13:26] but... all the unit tests passed [15:13:29] so, forcing it [15:15:56] !log updated payments-wiki from 3c465cb11c to d11efeb1cf, put it into maintenance mode [15:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:46:13] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598497 [15:46:18] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598497 (owner: Ejegg) [15:50:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [15:51:11] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:52:09] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:52:17] PROBLEM - check_mysql on frdb1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:52:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [15:52:17] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [15:53:11] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/send_nsca.cfg] [15:55:11] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 239 Threads: 2 Questions: 500 Slow queries: 6 Opens: 454 Flush tables: 1 Open tables: 200 Queries per second avg: 2.092 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [15:56:11] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 1049112 Threads: 1 Questions: 26261694 Slow queries: 679 Opens: 16147 Flush tables: 1 Open tables: 941 Queries per second avg: 25.032 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [15:57:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:57:17] PROBLEM - check_mysql on frdb1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:57:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [15:57:17] RECOVERY - check_puppetrun on frdb1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [15:58:11] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/send_nsca.cfg] [16:00:12] PROBLEM - Host frdb1002 is DOWN: PING CRITICAL - Packet loss = 100% [16:01:21] PROBLEM - Host frdev1001 is DOWN: PING CRITICAL - Packet loss = 100% [16:02:07] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:02:13] RECOVERY - Host frdb1002 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [16:02:15] PROBLEM - check_mysql on frdb1001 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [16:02:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2 No such file or directory) [16:03:11] RECOVERY - check_puppetrun on frqueue2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:04:52] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/598499 [16:05:13] RECOVERY - Host frdev1001 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [16:05:15] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/598499 (owner: Ejegg) [16:05:34] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/598499 (owner: Ejegg) [16:07:11] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1049247 Threads: 1 Questions: 26236128 Slow queries: 630 Opens: 15998 Flush tables: 1 Open tables: 811 Queries per second avg: 25.004 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:07:22] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 314 Threads: 6 Questions: 236 Slow queries: 0 Opens: 4399 Flush tables: 1 Open tables: 200 Queries per second avg: 0.751 [16:08:35] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Donate API should check if maintenance mode is on - https://phabricator.wikimedia.org/T253579 (Ejegg) [16:12:17] RECOVERY - check_mysql on frdb1001 is OK: Uptime: 174 Threads: 1 Questions: 115 Slow queries: 0 Opens: 659 Flush tables: 1 Open tables: 653 Queries per second avg: 0.660 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:24:03] !log updated standalone SmashPig from 2702b04329 to 44690f761c [16:24:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:09:00] !log enabled contribution tracking queue on payments-wiki [17:09:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:17:05] !log updated fundraising CiviCRM from 6b1d5902dd to 737d88a5ee [17:17:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:15:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1368 [18:17:17] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1491 [18:20:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1669 [18:22:17] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1791 [18:25:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1968 [18:27:17] RECOVERY - check_mysql on frdb1001 is OK: Uptime: 8274 Threads: 1 Questions: 5492 Slow queries: 1 Opens: 674 Flush tables: 1 Open tables: 655 Queries per second avg: 0.663 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [18:30:11] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 8858 Threads: 2 Questions: 3671 Slow queries: 295 Opens: 145 Flush tables: 1 Open tables: 127 Queries per second avg: 0.414 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [19:45:34] (PS1) Ejegg: Relax c_t queue update rules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/598530 (https://phabricator.wikimedia.org/T253250) [19:46:51] fr-tech ^^^ might be what we need [19:47:23] would that fix my issues of repeatedily using the same link but changing the amount? :p [19:47:35] cstone probably! [19:48:07] haha [19:48:14] ok, I'm going to eat some lunch [19:48:16] back soon!~ [19:51:18] thanks patch looks good but my test greed is kicking in. I want one that confirms we can overwrite an existing :| [19:51:38] lemme see if I can do that while ejegg|food eats [19:54:24] jgleeson: that sounds like a good instinct yeah [19:56:46] oh hey, looks like we already have one [19:56:52] even better [19:57:37] https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/598530/1/sites/all/modules/queue2civicrm/tests/phpunit/ContributionTrackingQueueTest.php@30 [19:58:18] I'll just tweak that to change the amount maybe [20:00:40] ill brb [20:12:45] hmm looks like the php array union operator (+) behaves in interesting ways [20:13:02] it will add new assoc key items but won't overwrite existing from the array on the right? [20:13:46] really php [20:14:42] The + operator returns the right-hand array appended to the left-hand array; for keys that exist in both arrays, the elements from the left-hand array will be used, and the matching elements from the right-hand array will be ignored. [20:15:08] ok let's swap them around [20:19:52] jgleeson: I think it's ok to leave them as is [20:19:59] the id should be the same in both [20:20:12] and the contribution_id is new in the right-hand array [20:20:39] ohh, sorry, to make a new test [20:20:42] gotit! [20:20:56] just gonna wash dishes and will be back at it [20:21:13] (PS2) Jgleeson: Relax c_t queue update rules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/598530 (https://phabricator.wikimedia.org/T253250) (owner: Ejegg) [20:22:24] ejegg|food: yeah, that test just confirms we can update existing stuff alongside adding new (which is what the contribution_id key was doing) [20:23:21] https://github.com/wikimedia/wikimedia-fundraising-crm/blob/09f7cb230ec86111f6d73b986504fa5fca57bee9/sites/all/modules/queue2civicrm/tests/data/contribution-tracking.json#L2 [20:25:57] hah jgleeson about the + operator the podcast i was listening to at like midnight last night while driving was talking about that too [20:26:17] :) ! [20:29:26] (CR) Jgleeson: [C: +2] "Looks good!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/598530 (https://phabricator.wikimedia.org/T253250) (owner: Ejegg) [20:30:01] I'm gonna break for 15 minutes to check in on the kids. back shortly [20:36:12] (Merged) jenkins-bot: Relax c_t queue update rules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/598530 (https://phabricator.wikimedia.org/T253250) (owner: Ejegg) [20:45:43] thanks jgleeson|baksoon ! [20:46:04] ok, gonna push that out [20:47:52] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598533 [20:48:03] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598533 (owner: Ejegg) [20:49:52] * ejegg gets back into the groove with Stevie Wonder -- Talking Book [20:51:57] (CR) Ejegg: [V: +2 C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598533 (owner: Ejegg) [20:54:25] 'talking book' was the original term for audiobooks [20:56:41] ah lol [20:56:53] before the days of the e- prefix I guess [20:56:59] cassette days [20:57:01] ? [20:57:06] LPs even! [20:57:10] oh yeah! [20:57:36] https://psmag.com/social-justice/a-brief-history-of-talking-books-for-the-blind-work-in-progress-55975 [20:57:55] They invented the LP format for audiobooks [20:58:00] wow [20:58:10] sound quality wasn't as good as 78s but you could record more time [21:01:46] !log updated fundraising CiviCRM from 737d88a5ee to 7380e0e8ce [21:01:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:01:49] hi eileen ! [21:02:07] well that was a great random article ejegg [21:02:08] ok cstone, want to go ahead and run that ct queue consumer? [21:02:11] :) [21:02:11] hey eileen [21:02:14] hey did I misjudge that there would be no meetings this am? [21:02:19] ok! [21:02:30] I figured memorial day but it looks like a normal day for everyone [21:02:34] eileen: oh right, we're just doing the maintenance [21:02:41] but no meetings [21:02:52] cool - looks like I missed a chance to sneak in the db alters [21:03:02] arr, shoot, I forgot [21:03:04] (they aren't urgent though) [21:03:18] it's just dropping a couple indexes? [21:03:24] let's see if there's actually any UI activity [21:03:57] they are quite slow though... [21:04:08] like hours? [21:04:14] 20 mins or so each [21:04:17] running now [21:04:24] cool cstone! [21:04:35] command dispatch complete [21:04:45] they could be consolidated into one I think - I left them as they are as any other way would break dev sites [21:04:56] great cstone ! [21:06:54] eileen: it doesn't look like anyone's actually using the Civi UI right now... [21:07:14] ejegg: should I run it - I don't think it's merged [21:07:44] https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/597663/ [21:08:13] Ahh, got it [21:08:24] eileen: I sent an email out last week about a bug I ran into when updating to the latest version of stock civi. If you get some time today could you check the findings? to add extra some extra confusion to the mix, somehow I managed to get around the database triggers when stepping through the code slowly but I only managed to do this once. However once giving civi trigger permissions it worked [21:08:26] (PS2) Ejegg: Put slow sql statement from civicrm upgrade into wmf_civicrm install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/597663 (owner: Eileen) [21:08:26] every time. [21:08:35] (CR) Ejegg: [C: +2] Put slow sql statement from civicrm upgrade into wmf_civicrm install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/597663 (owner: Eileen) [21:08:57] jgleeson that sounds right [21:09:03] s/some// [21:09:10] jgleeson: yeah - I thought you solved it correctly [21:09:30] locally it IS easier to have that setting so it makes sense in our dev builds [21:09:43] (it would be easier on live too but....) [21:11:04] ok. would it make sense to update our set up locally (for vagrant at least) to enable trigger permissions by default [21:11:51] thanks btw [21:12:54] ejegg if i ran that command right should the new entries be in frdevs table? [21:13:14] cstone yeah, where the heck are they? [21:13:57] I just double checked the numbers from the queue, and they are definitely 80600001-80600010 [21:14:02] jgleeson: yeah I think so [21:14:11] from the civi1001 box right? [21:14:12] didn't skip a zero there [21:14:34] lemme look at the db from the civi box [21:14:51] (Merged) jenkins-bot: Put slow sql statement from civicrm upgrade into wmf_civicrm install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/597663 (owner: Eileen) [21:14:54] I had previously listed the queue messages to screen from redis-cli [21:15:04] and I was just doublechecking with that backscroll [21:15:41] cstone yeah, you ran run-job contribution_tracking_queue_consume, right? [21:15:46] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598535 [21:16:12] dwisehaupt: if I run those ^^ index drop messages per convo above - will it trigger replication alerts [21:16:16] i did not run it that way probably why [21:16:31] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598535 (owner: Eileen) [21:16:41] cstone aha, ok [21:16:42] eileen: it may. but that's ok. we are aware of it. [21:16:49] ok cool [21:17:10] cstone want to run it that way before eileen does the index drops? [21:17:22] dwisehaupt: did you shrink civicrm_contact table already? It might be interesting to see before /after size [21:17:23] yep just ran it [21:17:44] crud, still not seeing the new entries [21:17:53] let's see if the queue still has messages [21:17:56] am i in the wrong folder im in the drupal folder? [21:18:04] drupal folder is right [21:18:16] Successfully completed contribution_tracking_queue_consume. was what i got [21:18:22] well, that run-job command has a -r option to drush in it [21:18:25] hmm hmm [21:18:38] !log civicrm revision is 7380e0e8ce, config revision is 6b05d6bb25 [21:18:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:18:42] so with that -r option it shouldn't matter what folder you run from [21:19:03] let's take a peek at the damaged table [21:19:17] let me know when I can kick off that alter [21:19:21] nope, they're not there [21:19:34] eileen: want to do it now? [21:19:45] ejegg: theyre still on the queue then? [21:20:02] cstone yeah, they're still on the queue [21:20:04] hmmmmm [21:20:22] looking back at the c-t queue consumer code... [21:22:11] all looks fine [21:22:52] (CR) Eileen: [V: +2 C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598535 (owner: Eileen) [21:23:32] do you want to try ejegg maybe im just user erroring it [21:27:37] fr-tech I don't see a batch arg on the ctqc [21:27:54] there isnt jgleeson it needs to be added [21:28:09] (PS1) Eileen: Fix misplaced { [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/598537 [21:28:13] but the slow-start command includes it [21:28:18] I wonder if that is erroring [21:28:27] argh I did the thing where I think it's a class & put it in the last function [21:28:30] i put it there for when its actually there [21:28:52] can someone +2 ^^ [21:29:13] oops [21:29:14] ah ok cstone are you just running the main command? [21:29:14] sure [21:29:19] yeah jgleeson [21:29:25] ahh ok [21:29:37] (CR) Ejegg: [C: +2] Fix misplaced { [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/598537 (owner: Eileen) [21:31:16] ohh, is it because we don't have ct_batch_time set? [21:31:28] it defaults to 0 [21:31:59] hmm, and that means no limit [21:32:08] cstone: I just tried running it again [21:33:18] shall we increase the verbosity of the output? [21:33:31] it is some permission thing? [21:33:41] I don't think so [21:33:51] hmm [21:34:13] let's put a -v or two in the process-control config [21:35:26] and let me double-check that smashpig config [21:36:08] (Merged) jenkins-bot: Fix misplaced { [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/598537 (owner: Eileen) [21:36:34] smashpig config on civi box looks fine [21:36:44] and that same config is working to push to the queue [21:37:05] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598538 [21:37:17] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/598538 (owner: Eileen) [21:38:55] !log civicrm revision changed from 5428c5c449 to d1cd99166f, config revision is 6b05d6bb25 [21:38:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:42:28] isn't there some funky drupal db cache [21:42:40] I wonder if that is stopping the db query [21:42:52] due to the id field change [21:43:37] hmm i did have to just clear mine locally to fix something like that [21:43:56] I have a faint memory of drupal having some big schema cache [21:44:04] and it causing problems before [21:44:27] ejegg: eileen does that ring an bells? [21:44:58] no - my query to alter contact table will be messing with you though [21:45:09] argh [21:46:03] I can kill it if need be - at the moment it's about half way through the first change "Stage: 1 of 2 'copy to tmp table' 31.4% of stage done" [21:46:13] hmmm drush registry cache [21:46:15] what is that [21:46:19] it sounds familiar [21:46:25] no eileen i think we're ok [21:46:31] oh that;'s to do with class caching I think [21:46:33] jgleeson: registry is like autoloader [21:46:44] and the c_t queue code has been deployed for months [21:46:53] ah ok mabye I'm barking up the wrong tree [21:47:04] eileen this queue consumer write to a whole different db [21:47:10] doesn't touch anything in CiviCRM [21:47:11] ejegg: oh cool [21:47:12] ejegg: I was thinking the change to the ct_id table is the issue [21:47:39] jgleeson: that /should/ all be cleared up after drush updb [21:48:00] i'd expect to see some kind of error [21:48:16] even with an extra -v i'm not seeing anything illuminating [21:49:28] hmm, did we skip turning on a module? [21:50:07] yup [21:50:12] doing it via the UI [21:50:32] and undoing the extra -v [21:50:32] ooo [21:51:02] ahhh [21:51:53] * jgleeson is thinking of a tooth paste advert [21:52:02] haha [21:52:19] :) [21:52:34] ok cstone, want to try again? [21:52:41] ok! [21:53:11] OUTPUT! :D [21:53:17] oh man jgleeson it's like 11 PM where you are [21:53:29] i see them in the db! [21:53:34] WOOHOO! [21:53:35] ̿̿ ̿̿ ̿̿ ̿'̿'\̵͇̿̿\з= ( ▀ ͜͞ʖ▀) =ε/̵͇̿̿/’̿’̿ ̿ ̿̿ ̿̿ ̿̿ [21:53:42] jgleeson: always prepared [21:53:47] :) [21:53:49] that's a nice one [21:54:12] ejegg: I've been waiting up to do that [21:54:16] :) :) [21:54:38] and now that's accomplished, you can get some sleep! [21:54:52] srsly, we'll test a bit more and finish up tomorrow [21:54:53] righto. good work folks. catch you all tomorrow! [21:55:01] thanks! [21:55:03] see ya [21:55:07] bye! [21:56:29] looks like there were ~45 of them [21:56:40] cstone ok, I guess we wait for eileen's table alters to finish before we run more consumers [21:56:41] er 40 [21:56:44] ok [21:57:00] exactly 38. haha [21:57:01] well, we could do the payments-init and antifraud I guess [21:57:18] ejegg: I actually triggered the first one separately - so when that is done I'll not run the others & give you a chance [21:57:39] those use ct_id as a pseudo-fk so they could potentially have issues if the front end was wired wrong [21:57:47] eileen: okok, thanks [21:58:11] cstone want to try those two with run-job as well? [21:58:49] paymentsinit_queue_consumer? [22:00:13] ok that ran [22:00:17] Stage: 1 of 2 'copy to tmp table' 60.6% of stage done [22:00:32] and antifraud run [22:00:37] er ran* [22:19:08] bloody hell Stage: 1 of 2 'copy to tmp table' 101% of stage done [22:21:19] it's making sure it's giving 110% to get the job done. :) [22:23:08] dwisehaupt: it's up to 108% now so let's see if it gets there :-) [22:23:19] cool cstone, let's wait to run the rest [22:23:32] welll, I guess the pending qc is also to another db [22:26:51] looks like it just finished. [22:27:31] i'll keep an eye on replication lag and set downtime/ignores appropriately. [22:36:12] yep - there are actually more queries to go but I think ejegg needs to do some stuff now [22:38:32] actually all done now [22:39:54] cool [22:41:58] ok. i need to prep the bbq for dinner. i'll still be around, just afk. if you ping me on slack it should make my phone beep. [22:48:43] cool! ok cstone let's run the rest of the queue consumers [22:49:37] I'm starting with pending [22:49:52] ok [22:50:08] paypal smashpig job runner [22:50:13] (both completed OK) [22:50:51] recurring_queue_consume was ok [22:51:15] refund was OK too, though q was empty [22:51:47] unsubscribe_queue_consume was OK [22:53:55] thank you send running now [22:54:17] also ran optin qc and adyen job runner, but there was nothing to do for either [22:55:57] got my thank you email! [22:56:04] !!! woo! [22:56:12] lessee what other queues have anything [22:56:32] oh yeah, banner history [22:56:55] all looking good [22:57:04] OK, we can try a few more payment methods [22:57:08] I'll try an adyen one-time [22:57:29] looking in the ct table, should yours at least have the contribution_id set? [23:00:01] cstone oho, we need to run the ct_consumer again [23:00:06] oooh yeah [23:00:21] ok, it should now [23:00:31] i should have known that i literally had the same thought process this morning testing hah [23:01:25] hmm still not seeing any 806s with them [23:01:59] fr-tech hey how're we doing? Apologies for being a bit absent this afternoon, I think I didn't mention, my kids are still here because of the COVID case in their mom's apartment building [23:02:25] anyway just finished all the cooking and eating stuff [23:02:53] AndyRussG: we're doing pretty well! [23:03:36] ejegg: yaaaay congrats! [23:03:53] So we're on step 15.2? [23:04:32] yep, and 15.3 concurrently [23:06:39] yaaay [23:06:56] ok, paypal donation worked all the way through [23:08:32] ok just submitted adyen [23:09:39] there is a lottt of tern activity here tonight, fishing must be extra good [23:10:21] holey smokes but there are a lot of redirects in the astropay flow [23:10:55] made it back to the thank you page, though! [23:12:26] lemme put in an amazon donation and then we can turn the queues back on [23:12:59] Is this a correct URL for Amazon? https://donate.wikimedia.org/w/index.php?title=Special:AmazonGateway¤cy=USD&country=US&amount=1&ffname=amazon [23:13:45] AndyRussG: that looks about right, but I've just put one in... [23:14:01] no need to shunt more money through the bezos machine [23:14:42] It gives me a "No such special page" page [23:14:58] oooh, donate.wiki [23:15:03] you need payments.wiki [23:15:15] ah weird I thought I'd put that, ah okok [23:15:53] Ahh I see yeah I'd gone to payments wiki but it redirected me to donate wiki, forgot about that [23:16:25] hmm, i have to help with more cleaning here [23:17:44] ejegg looks like we're more than on time to finishing by Friday! [23:20:13] Hmmm it's still sending me to donate wiki [23:21:40] Ah got it it was the extra /w/ in the URL [23:21:50] ahhh [23:22:17] Anyway, I think we can turn the queue consumers back on [23:22:18] Well it sends me to Amazon [23:22:23] they've been solid so far [23:22:27] should I do something there? [23:22:36] Or you already finished checking that? [23:22:38] AndyRussG: no, I just did and the flow worked for me [23:22:45] yay! [23:23:11] I'm guessing Adyen also worked? [23:23:21] adyen worked for me [23:23:31] cool cstone noting on the Etherpad [23:23:49] ok, so I'mma re-enable the basic queue consumers and job runners [23:24:19] though not the recurring charge jobs nor the audit processors [23:24:29] we can test those tomorrow [23:25:31] ok yay! [23:25:47] ejegg: cstone: what could I do this evening that would be helpful? [23:26:38] ejegg: how were you checking what was in the queues on live? [23:27:35] Isn't there a redis cli thing? [23:27:47] yeah i just dont know where it would be [23:32:45] fr-tech sounds like things are going ok. what's on the list for tomorrow? [23:33:06] cstone yep, I was looking at the redis cli from the queue box itself [23:33:17] dstrine tomorrow we test recurring, imports, and audits [23:33:24] dstrine: step 19 https://etherpad.wikimedia.org/p/ContributionTracking [23:33:27] basic one-time donations are all good [23:33:37] and the basic queue consumers are back up [23:34:28] !log re-enabled fundraising queue consumers and job runners, except audits, dedupe, and recurring [23:34:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:36:29] yay! Congrats everyone! I have a bit of a meeting vortex through most of tomorrow but let me know if I can help with anything. I'm going to feed all the animals. Don't work too late today if you don't have to. [23:37:21] cool cool! [23:37:28] Yeah, I think I'm going to call it a night [23:37:46] ejegg|away: ¡que descanses! [23:39:24] im gona go too, good night everyone [23:39:54] cstone: cya! [23:42:57] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Payments should only send new contribution tracking messages if something changes - https://phabricator.wikimedia.org/T253602 (Ejegg) [23:46:43] good work