[02:30:44] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Stripe import issue - https://phabricator.wikimedia.org/T234556 (LeanneS) @Eileenmcnaughton I was able to get them all imported by dividing into different files. Dividing worked for about half and then I rec... [04:33:05] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Stripe import issue - https://phabricator.wikimedia.org/T234556 (Eileenmcnaughton) @LeanneS it sounds like one or more row might have had an issue but we didn't narrow down which & what :-( [07:05:16] (PS1) AndyRussG: Allow bannerimpressions cells from multiple files [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/541155 (https://phabricator.wikimedia.org/T234737) [07:08:04] Fundraising Sprint Lactose is unusually tolerant, Fundraising Sprint Land before Timeouts, Fundraising Sprint Matt Damon to head up Space Force, Fundraising Sprint Men In Slack, and 16 others: DB schemas (production changes and test DB) and SQL commands to ... - https://phabricator.wikimedia.org/T196564 [13:17:21] PROBLEM - check_procs on frdb1001 is CRITICAL: PROCS CRITICAL: 1391 processes [13:22:21] RECOVERY - check_procs on frdb1001 is OK: PROCS OK: 215 processes [15:14:52] fundraising-tech-ops, Operations, ops-codfw: rack/setup/install frban2001.codfw.wmnet - https://phabricator.wikimedia.org/T234069 (Papaul) [15:55:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [16:00:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [16:05:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [16:10:14] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [16:15:14] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 422130 Threads: 4 Questions: 35177735 Slow queries: 14949 Opens: 2683 Flush tables: 1 Open tables: 200 Queries per second avg: 83.333 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:52:35] Fundraising-Backlog, fundraising-tech-ops: check_log_messages may not be correctly setting or checking it's lock file - https://phabricator.wikimedia.org/T234592 (Jgreen) Open→Resolved Working correctly now. [16:55:32] fundraising-tech-ops: fundraising access request for Nora Nichols - https://phabricator.wikimedia.org/T232623 (Jgreen) @NNichols circling back on this task, did you ever receive your yubikey? [17:14:56] fr-tech I'm gonna release the slow-start command update to the donation queue consumer. there's a few other patches on master not yet deployed. cstone your day_of_month patch and two from ejegg|away which are formatting changes and a single line to allow twig templates access to a new l10 function. Anyone have any objections to me deploying these? [17:15:21] hey jgleeson i was just going to ask you the same thing hah [17:15:26] i dont have any objections [17:16:26] cool! [17:22:18] fundraising-tech-ops, Operations, ops-codfw: rack/setup/install frban2001.codfw.wmnet - https://phabricator.wikimedia.org/T234069 (Papaul) [17:25:12] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install frnetmon1001 - https://phabricator.wikimedia.org/T232137 (Jgreen) [17:25:39] fundraising-tech-ops, Operations, ops-codfw: rack/setup/install frban2001.codfw.wmnet - https://phabricator.wikimedia.org/T234069 (Jgreen) [17:26:02] fundraising-tech-ops, Operations, ops-eqiad: rack/setup/install frban1001.eqiad.wmnet - https://phabricator.wikimedia.org/T234068 (Jgreen) [17:26:22] fundraising-tech-ops, Operations, ops-codfw: rack/setup/install frqueue2001 - https://phabricator.wikimedia.org/T232630 (Jgreen) [17:27:03] That reminds me I should do a DonationInterface deploy today, but I'll wait until jgleeson is done. [17:29:38] (PS1) Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/541316 [17:33:07] fr-tech, once this patch is deployed I'm gonna reduce down the number of messages the current donation queue consumer processes during each cron cycle from 'unlimited' to '1' so I can put a few through myself via the slow-start command [17:33:39] (once I've added the slow-start command to the process-control job also!) [17:33:49] fundraising-tech-ops, Operations, ops-codfw, Patch-For-Review: rack/setup/install frban2001.codfw.wmnet - https://phabricator.wikimedia.org/T234069 (Papaul) [17:34:34] fundraising-tech-ops, Operations, ops-codfw, Patch-For-Review: rack/setup/install frban2001.codfw.wmnet - https://phabricator.wikimedia.org/T234069 (Papaul) @Jgreen you can take this task after you review the mgmt and production DNS [17:36:35] cstone: something just caught my eye. it looks like we were using 'create_date' on the old code but in your update you're using 'start_date'. Is that intentional? https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/541316/-1..1/sites/all/modules/queue2civicrm/recurring/RecurringQueueConsumer.php [17:38:43] jgleeson|brb: yeah that was intentional [17:39:02] the start date would be the actual date the recurring donation was processed [17:48:30] cstone: awesome, thanks for confirming [17:48:53] (CR) Jgleeson: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/541316 (owner: Jgleeson) [17:59:39] (CR) Jgleeson: [C: +2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/541316 (owner: Jgleeson) [18:04:01] ah I forgot that we have to manually merged civicrm patches into deployment due to the CI environment missing required extensions https://integration.wikimedia.org/ci/job/wikimedia-fundraising-crm-composer-php72-docker/837/console [18:05:14] ah yeah [18:07:35] !log Updating civicrm from c12f7bb51f to db7ef10bfa [18:07:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:12:44] fr-tech, is anyone free to hop onto frpm1001 and just review my minor update to the donations_queue_consumer.yaml process-control job? [18:12:56] sure [18:12:59] thanks! [18:17:50] fundraising-tech-ops: fundraising access request for Nora Nichols - https://phabricator.wikimedia.org/T232623 (NNichols) yes thank you! [18:18:27] jgleeson: do you need two -v's [18:20:29] but otherwise looks good [18:25:04] cstone: it looks like we have a variety of verbosity levels across the process-control jobs. on some like pending_queue_consume.yaml we don't use it at all but then on others like ingenico_recurring_charge.yaml we use the single -v which I think was the one I followed. [18:27:24] ah then looks good! [18:27:44] hmm I can't seem to find documentation on how the different levels differ between -v and -vvv [18:28:04] presumably something like error, debug, info [18:28:42] thanks for the review! [18:28:48] right, let's give it a whirl! [18:38:01] looks like it does what it says on the tin [18:39:04] ok I'm gonna up the donation queue consumer job to go back to processing 'unlimited' messages each cron cycle by setting the cron batch size here from '1' back to 'unlimited' [18:39:11] here -> https://civicrm.wikimedia.org/admin/config/queue2civicrm/configure [18:42:01] nice jgleeson ! Thanks for deploying mine too :D [18:42:25] that's done. cstone was anyone waiting to hear when the day_of_month update was deployed? It looks like the phab task link isn't finding the original ticket in the commit message https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/539210/ [18:42:51] thanks I can update that task too, I was just testing it on live [18:43:00] awesome [20:01:20] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: More small fixes needed for Campaign fallback - https://phabricator.wikimedia.org/T234248 (jgleeson) a:jgleeson [20:08:23] cstone: wanna talk through the email funkiness ? [20:08:29] sure [20:08:46] let's rejoin the standup chat [20:09:06] hmm we didn't have a standup chat [20:09:14] backlog refinement even [20:09:20] I have a vauge notion that might be expected behavior for the recur to be different than first date. [20:09:25] in the ops one theres one [20:09:41] yeah i kind of remembered that too and was hunting around [20:09:55] I don't remember where that would be set though. [20:09:57] XenoRyet: do you have a few minutes to pop in? I don't know a great deal about the upsell emails [20:26:57] (PS1) XenoRyet: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/541365 [20:33:56] (CR) XenoRyet: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/541365 (owner: XenoRyet) [20:34:52] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/541365 (owner: XenoRyet) [20:39:55] (PS1) XenoRyet: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/541369 [20:46:08] (CR) XenoRyet: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/541369 (owner: XenoRyet) [20:53:31] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/541369 (owner: XenoRyet) [20:56:01] !log updated payments-wiki from b94da68f7e to d2e2637275 [20:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:49:34] (PS1) Cstone: Format the day_of_month correctly. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/541395 (https://phabricator.wikimedia.org/T228162)