[02:56:02] i'm back now. gonna run some of the table compactions. to get them out of the way while we still have traffic off. [03:07:49] No time like the present, eh? [03:27:07] yeah. some are running faster. some slower, but good to get them through. [03:27:52] Have fun! [03:28:19] it's something to run in the background while i catch up on some tv. :) [03:29:58] Catching up on a bit of Westworld myself. I'm liking season 3. [03:30:48] cool. i haven't started season 3 three there yet. currently watching killing eve season 3. [03:32:21] I'll have to put that one on the list. Looks fun. [03:33:06] the first 2 seasons were really good. [04:27:34] If anyone's around, just trying to get ready to help with recurring tests and re-understand all that... For Adyen we do post-payments-form monthly convert, or in-banner monthly convert, or both? [04:49:23] Found a recent banner with cc in-banner convert [04:58:53] ok. that's the last alter i'm running tonight. going to wait for the alter to settle. [05:32:17] PROBLEM - check_mysql on frdb1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2263 [05:35:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2439 [05:37:17] RECOVERY - check_mysql on frdb1001 is OK: Uptime: 48474 Threads: 1 Questions: 382449 Slow queries: 8 Opens: 780 Flush tables: 1 Open tables: 662 Queries per second avg: 7.889 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1 [05:40:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2739 [05:45:11] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 49358 Threads: 2 Questions: 378807 Slow queries: 1684 Opens: 372 Flush tables: 1 Open tables: 199 Queries per second avg: 7.674 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [07:19:39] (CR) Abijeet Patro: [C: +1] "This change makes sense to me. Adding +1 for now, and waiting for any comments from project owners." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/598470 (https://phabricator.wikimedia.org/T221119) (owner: Nikerabbit) [10:33:57] Fundraising-Backlog, Wikimedia-Fundraising, MediaWiki-extensions-FundraiserLandingPage, donate.wikimedia.org, Wikimedia-production-error: "PHP Warning: preg_match() expects parameter 2 to be string" from Special:FundraiserLandingPage - https://phabricator.wikimedia.org/T226456 (Pcoombe) [13:39:22] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising Sprint Ice cream cures everything, and 8 others: CRM contribution tracking writes should use queue and sequence generator - https://phabricator.wikimedia.org/T215463 (Jgr... [13:57:42] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog, iOS-app-feature-Feed: Fundraising banners still showing on iOS app? - https://phabricator.wikimedia.org/T242347 (Pcoombe) [13:59:03] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog, iOS-app-feature-Feed: Fundraising banners still showing on iOS app? - https://phabricator.wikimedia.org/T242347 (Pcoombe) Open→Resolved Thank you. These donations do appear to have dried up now. [14:21:41] (PS1) Pcoombe: Change "Argentinian peso" to "Argentine peso" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/598748 [15:19:27] hi fr-tech! [15:19:42] I'm going to go ahead and run the audit parsers to see how they do [15:21:33] hi ejegg [15:29:06] hi cstone [15:29:14] so the ingenico audit parser is running [15:29:18] and looks like it's going ok [15:30:00] nice [15:36:09] ok, 41822092 is a CID from a donation that just came in via the audit file [15:37:45] trying the paypal audit [15:40:29] mornin [15:43:43] hi dwisehaupt ! [15:43:52] ok, paypal audit parse completed, but no new donations from it [15:44:04] oh. kernel update out. i'm going to restart for that before getting too far into things today. [15:45:38] ejegg: I can make a new recurring ingenico and adyen? [15:46:28] cstone thanks! That would be great [15:47:23] and paypal hah [15:50:40] tzag fr-tech cstone ejegg dwisehaupt [15:50:55] hi AndyRussG ! [15:51:34] fundraising-tech-ops: install renewed frdata.wikimedia.org cert - https://phabricator.wikimedia.org/T253643 (Jgreen) [15:51:52] fundraising-tech-ops: install renewed frdata.wikimedia.org cert - https://phabricator.wikimedia.org/T253643 (Jgreen) p:Triage→Medium [15:53:10] :) [15:53:29] hi AndyRussG [15:53:34] :) [15:54:29] fr-tech for Ingenico and Adyen, there are two different types of recurring flows to test, no? One where you select recurring from the banner, and the other with monthly convert via payments, no? [15:54:47] Or am I understanding something wrong? [15:54:53] AndyRussG: they both end up in the same place [15:55:24] I just made a recurring of all three though [15:56:07] cool cool, shall we run the recurring charge jobs on --slow-start? [15:56:13] they get recurring=1 set in the url AndyRussG [15:57:18] though AndyRussG is right, there is a bit of difference in that the updonate ones end up with a non-recurring first donation that then has to be linked for the first real recurring charge a month later [15:57:49] AndyRussG: the updonate flow only works via ingenico for now [15:58:28] im messing up my ingenico link too its erroring when i try and donate hmm [15:58:38] hmm [15:58:40] spoke too soon on all three, paypal and adyen done [15:58:43] let's see what errors look like [16:00:43] is there an already built ingenico link somewhere [16:03:30] ok ingenico done [16:03:53] thank you emails so speedy! :) [16:06:02] ejegg: [16:06:14] ejegg: cstone okok thx!! [16:18:52] * ejegg is listening to Graham Central Station -- Ain't No 'Bout-A-Doubt It [16:19:17] cstone: have you run any of those charge jobs yet, or should I try them? [16:21:00] * AndyRussG is listening to barely audible kids' youtube video sounds from upstairs as they avoid homework [16:21:30] jejeje [16:22:20] i havent ejegg [16:23:02] im gona get some food ill brb [16:23:35] AndyRussG: ours have taken to using headphones as a means to keep us from knowing when they are skiving off. although we always know since it takes forever for them to do their work. [16:25:55] dwisehaupt heheh hmmm I've been just avoiding nagging and letting them self-organize. At 12 and 14, it helps us all get along much better. Keeping my fingers crossed as regards to actual grades... [16:28:50] yeah. i totally get that. had to focus them a few times to remind them to focus on the big things that count more (like tests and quizzes). [16:29:07] focus is the word of the day today. [16:30:02] definitely trying times for everyone [16:31:14] for sure. especially when it's 25C/77F outside and sunny. :) [16:32:11] gonna run those recurring charge jobs [16:37:31] cstone ejegg is there anything I could do to help out, that wouldn't be like just getting in the way? [16:38:48] ejegg I guess that's etherpad step 21 you're doing? [16:40:00] I feel like the etherpad test steps reads like a list of essential functions that always gotta work. Maybe that could be re-used for doc or a checklist somewhere [16:40:23] ok, legacy ingenico seems fine [16:40:42] AndyRussG: sure, want to make a 'checklist template' someplace on-wiki? [16:42:57] ejegg okok! [16:43:39] fundraising-tech-ops: clean up payments-wiki configuration related to deprecated mysql contribution tracking connection - https://phabricator.wikimedia.org/T253653 (Jgreen) [16:48:44] ok, new recurring thing looks good too [16:49:47] so that's all but the csv imports tested [16:51:06] im back [16:52:52] csv imports are like benevity? [16:58:26] cstone yeah, like benevity [16:59:15] handled via drupal7 code at /admin/import_checks [16:59:41] there's a 'generic individuals' import [16:59:46] fr-tech what about fraud-y-detecting code paths? is there any chance those might get confused in the new system? [16:59:58] not that I can think of AndyRussG [17:00:07] ejegg okok thanks :) [17:00:16] the reports should be checked I guess, and dash [17:00:23] that would be the fredge reports [17:00:41] K where are those again? [17:00:51] MBeat: want to take a look at one of those and see if you can find anything from the last couple days? [17:01:52] ok, found it [17:02:37] MBeat: is it normal to have so many with blank columns in the individual filters? [17:03:45] maybe the recent ones are less normal because there's no campaign [17:04:38] anyway, there are definitely entries [17:05:10] ok, i'm going to fabricate a csv import for a dollar donation from myself [17:09:45] thanks ejegg [17:10:05] tzag fr-tech [17:10:20] hi jgleeson [17:10:31] how's it going? [17:11:03] not too bad! nice weather here so that helps. How about you? [17:11:17] going pretty good [17:11:27] been hot and humid these past couple days [17:11:43] hospital trip was smooth enough and hardly anyone else around in the children facility which made things easier [17:11:49] oh good [17:13:05] it is normal to have mostly zeroes entered for a lot of the filters, ejegg [17:13:26] I scanned one from Sunday for Ingenico and it looks normal, 80500254.1 [17:13:45] MBeat: ok, cool. The ones from post-switchover start with 806 [17:14:05] there aren't a lot yet, though [17:16:20] I do see recent entries in fredge.payments_fraud [17:16:30] I assume from the tests being done [17:29:25] i made it through 6 of the 8 compaction alters last night. the 2 remaining are the long ones on time. at least 84m and 129m runtime each. [17:30:18] unfortunately, my testing on frdb2001/2002 didn't have the alters replicate down, but they definitely do so it will affect replication timing when i run them. [17:32:00] hmm, can't figure out what to put in the 'Payment Instrument' column for my import test to work [17:34:33] aha, got it [17:34:39] Raw Payment Instrumet [17:34:44] *Instrument [17:34:53] Credit Card: Visa [17:35:28] worked, too [17:38:55] contrib id 46561052 [17:39:01] let's see if that got a ct_id [17:39:48] nope, guess not [17:42:10] hmm, is that normal though? [17:44:39] (PS13) Cstone: Drush script to tokenize old recurring GC [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/582883 (https://phabricator.wikimedia.org/T169357) (owner: Ejegg) [17:51:41] ok, looking at other contributions with gateway=generic import [17:51:48] none of them have c_t rows either [17:51:53] so I'll call that step donw [17:51:55] *done [17:55:29] ok, I'm going to re-enable the rest of the jobs [18:00:25] ejegg: cool. i'd like to get these last two alters in before we release the maint window for good. [18:01:04] dwisehaupt: ok, sure [18:01:22] should we stop those jobs again? [18:01:38] oh, I should log that [18:02:01] !log restarted fundraising jobs: recurring charge, audit processing, deduplication [18:02:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:02:21] hmmmm.... i could go either way. [18:02:39] the two tables to go are civicrm_activity and civicrm_mailing_provider_data [18:03:44] well, i'm guessing the jobs won't have much to do in any case [18:04:01] those are replica dbs, so I guess it won't deadlock anywhere [18:04:08] just cause replication alerts? [18:04:35] it will definitely affect replication alerts. [18:04:58] the alter uses the online DDL which should allow reads/writes to the table while the alter goes on. [18:05:12] so the impact should be minimal but just mentioning it. [18:10:07] if everyone is ok, i can go ahead and start the first. [18:11:55] fr-tech I'm gonna put half of today down as a PTO because I've not been around much and I'm now about to go eat. Back shortly. [18:12:23] getting some lunch [18:12:30] sounds good dwisehaupt [18:13:38] jgleeson|dinner: We're still operating under the 50% capacity system, so unless you're dropping a fair bit below that, no need to take PTO for stuff like today. [18:15:17] ok. starting the first of the two. it should finish in 90mins or so. [18:20:55] These alerts mean anything? ** PROBLEM alert - frav1002/check_log_messages is CRITICAL ** [18:21:10] Ohh woops sorry they're old [18:21:45] Yeah this morning I just see our old friend Fail Mail: BAD_AUDIT_LINE (paypal-audit) [18:29:22] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Centralnotices leaves lots of entries in LocalStorage - https://phabricator.wikimedia.org/T197463 (AndyRussG) [18:43:09] thanks for the reminder XenoRyet! [18:43:36] fr-tech hows things been today? Anything I can do to help out on the testing/other front? [19:00:46] fr-tech are meetings cancelled today also? [19:06:31] i dont think so? jgleeson [19:16:28] jgleeson: No, we're having retro and sprint planning at the usual time. [19:16:59] thanks! [19:38:33] fundraising-tech-ops: install renewed frdata.wikimedia.org cert - https://phabricator.wikimedia.org/T253643 (Jgreen) Done, confirmed working. [19:38:38] fundraising-tech-ops: install renewed frdata.wikimedia.org cert - https://phabricator.wikimedia.org/T253643 (Jgreen) Open→Resolved a:Jgreen [19:55:35] just started the last alter. [20:06:09] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1343 [20:28:05] Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, MW-1.35-notes (1.35.0-wmf.34; 2020-05-26): Adyen/iDEAL message getting from DonationInterface to Civi missing ... - https://phabricator.wikimedia.org/T251025 [20:28:09] Fundraising-Backlog, FR-Adyen, Recurring-Donations, Epic: EPIC recurring for Adyen - https://phabricator.wikimedia.org/T238101 (Cstone) [20:28:11] Fundraising Sprint CAPS LOCK CULTS, Fundraising Sprint Dampness, Fundraising Sprint Evil Twins For Everyone, Fundraising Sprint Ice cream cures everything, and 3 others: Disregard Adyen recurring authorization IPNs - https://phabricator.wikimedia.org/T243334 (Cstone) Open→Resolved [20:29:09] Fundraising Sprint Just space out, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Security-Team, and 2 others: SecurityCheck-XSS Suppressed in CentralNotice for Geotargeting - https://phabricator.wikimedia.org/T251269 (DStrine) Open→Resolved [20:29:52] Fundraising Sprint Just space out, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Contribution tracking queue consumer should reject bad updates - https://phabricator.wikimedia.org/T253250 (Ejegg) Open→Resolved [20:30:00] Fundraising Sprint Just space out, Fundraising-Backlog, MW-1.35-notes (1.35.0-wmf.34; 2020-05-26): design edits for the payments maintenance page - https://phabricator.wikimedia.org/T253068 (DStrine) Open→Resolved [20:30:03] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising Sprint Ice cream cures everything, and 8 others: CRM contribution tracking writes should use queue and sequence generator - https://phabricator.wikimedia.org/T215463 (Eje... [20:49:26] Fundraising Sprint K 2020, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Payments should only send new contribution tracking messages if something changes - https://phabricator.wikimedia.org/T253602 (DStrine) [20:49:29] Fundraising Sprint K 2020, Fundraising-Backlog, MediaWiki-extensions-ContributionTracking, MediaWiki-extensions-DonationInterface: Move FundraiserMaintenance to DonationInterface - https://phabricator.wikimedia.org/T253057 (DStrine) [20:49:31] Fundraising Sprint Just space out, Fundraising Sprint K 2020, Fundraising-Backlog, Documentation: Document changes in Contribution Tracking - https://phabricator.wikimedia.org/T252818 (DStrine) [20:49:34] Fundraising Sprint Just space out, Fundraising Sprint K 2020, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Schema-change: CentralNotice: Deploy new regional geotargeting and banner template features - https://phabricator.wikimedia.org/T252596 (DStrine) [20:49:37] Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, Fundraising Sprint K 2020, and 3 others: Frontend - Make employer field autocomplete which integrates with new backend MW emp... - https://phabricator.wikimedia.org/T251200 [20:49:40] Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, Fundraising Sprint K 2020, and 3 others: Backend - Create MW API to expose employer data - https://phabricator.wikimedia.org/T251199 (DStrine) [20:49:42] Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, Fundraising Sprint K 2020, and 3 others: Make Civi matching gifts extension store data in custom fields - https://phabricator.wikimedia.org/T249924 (DStrine) [20:49:44] Fundraising Sprint Fish Fight Frogs for Free French-Fry Frappées, Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, and 4 others: Civi mail To, CC, BCC email additions diffcul... - https://phabricator.wikimedia.org/T248199 [20:49:46] Fundraising Sprint K 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-AutoTY-Email: Civi: annual summary email - add tax deductibility language - https://phabricator.wikimedia.org/T242001 (DStrine) [20:49:48] Fundraising Sprint Fish Fight Frogs for Free French-Fry Frappées, Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, and 6 others: Test migrating old ingencio recurring to new ... - https://phabricator.wikimedia.org/T232017 [20:49:51] Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, Fundraising Sprint K 2020, and 3 others: Investigate how to generate generic email for failed recurring donations - https://phabricator.wikimedia.org/T230064 (DStrine) [20:49:52] Fundraising Sprint Fish Fight Frogs for Free French-Fry Frappées, Fundraising Sprint Heaven is a page on Wikipedia, Fundraising Sprint Ice cream cures everything, Fundraising Sprint Just space out, and 6 others: Create script to migrate existing GlobalColle... - https://phabricator.wikimedia.org/T169357 [20:49:56] Fundraising Sprint Just space out, Fundraising Sprint K 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Create a new CiviCRM extension with an API action to fetch data from SSBInfo - https://phabricator.wikimedia.org/T249922 (DStrine) [21:12:26] should be about half way through the last alter. [21:21:09] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 1155012 Threads: 1 Questions: 27689156 Slow queries: 742 Opens: 16329 Flush tables: 1 Open tables: 931 Queries per second avg: 23.973 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:17:49] (PS14) Cstone: Drush script to tokenize old recurring GC [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/582883 (https://phabricator.wikimedia.org/T169357) (owner: Ejegg) [22:31:21] expecting the query to end at any point now. [22:33:28] as if on queue, it's done :) [22:34:26] fr-tech: that's all the alters. it'll take about 160-180 mins for the replication to clear on the slowest host but no more actions needed on the master db. [22:46:04] Fundraising-Backlog, fundraising-tech-ops: Tables to compact on frdb hosts - https://phabricator.wikimedia.org/T248609 (Dwisehaupt) All complete on frdb1002. We are not going to do the dev dbs on frdev1001 as we rebuild them fairly regularly so they will get compacted at that point. ` # compact_innodb_t... [22:46:27] Fundraising-Backlog, fundraising-tech-ops: Tables to compact on frdb hosts - https://phabricator.wikimedia.org/T248609 (Dwisehaupt) [22:50:33] thanks dwisehaupt ! [22:51:41] Fundraising-Backlog, fundraising-tech-ops: Tables to compact on frdb hosts - https://phabricator.wikimedia.org/T248609 (Dwisehaupt) Minor update to the script that was used with this compacting so that it uses SSL connections by default. ` [frack::puppet] 1fe1e45b Specify default value for ssl option `