[00:01:09] grafana shows 3 jobs https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?refresh=1m&orgId=1&panelId=10&fullscreen&from=now-5m&to=now but run-job -r shows none [00:02:55] hmm actually 'jobs' is flat 'paypal' has 3 - what is paypal vs jobs [00:03:35] oh, those would be the number of messages in the queues [00:03:51] the 'process-control jobs' is trying to show when each job is running [00:04:00] ejegg: ah ok - I'll turn on Redis now then... [00:04:05] and it looks like all of them are off right now [00:05:14] oh, you can also run 'run-job --list-jobs --only-running on civi1001 [00:05:16] !log process-control config revision is ec8c43ee86 (Redis turned on) [00:05:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:05:22] no wSOD! [00:05:36] :) [00:05:59] wait, i think you just synced process-control again, not civi [00:07:32] you are right [00:08:24] !log civicrm revision is 8a4451f390, config revision is ec8c43ee86 Redis [00:08:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:09:23] ejegg: ok this time it SAYS Redis [00:09:36] :) [00:10:13] ok it looks good to me [00:10:29] you think I need to slow start the queues? I don't think I've done that [00:10:51] sure, let's try one for the heck of it [00:12:28] so that would be run-job --slow-start donations_queue_consume [00:12:41] hmm, shoot, says it has no slow start command [00:12:54] well, pending qc worked, but that doesn't touch civi [00:12:55] well it ran ok :-) [00:13:17] yep, looks fine to me [00:13:26] let's go ahead and turn those jobs back on [00:14:37] !log process-control config revision is 8e215d07f2 (renable jobs) [00:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:16:13] (PS1) Ejegg: Allow 'Z' at end of timestamp [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520344 [00:16:22] AndyRussG: ^^^ should fix it [00:16:31] (CR) jerkins-bot: [V: -1] Allow 'Z' at end of timestamp [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520344 (owner: Ejegg) [00:16:37] oops [00:17:21] oh man, it wants that repo to be up to flake8 standards :P [00:19:19] (PS2) Ejegg: Allow 'Z' at end of timestamp [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520344 [00:19:21] (PS1) Ejegg: flake8 fixes [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520345 [00:34:24] Fundraising Sprint Men In Slack, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Turn on Redis, Functional locks - https://phabricator.wikimedia.org/T222002 (Eileenmcnaughton) a:Eileenmcnaughton [00:41:12] Fundraising Sprint Men In Slack, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Turn on Redis, Functional locks - https://phabricator.wikimedia.org/T222002 (Eileenmcnaughton) Just noting that before redis was enabled the highest peak processing in the last 90 days was 13 donations per second - w... [00:43:28] hmm, i think the ingenico audit parser is missing from the process-control job dashboard [00:44:12] well, that board is a horrible hack. I guess we could make process-control itself output to prometheus format instead of trying to parse the stuff out of logs [00:45:05] eileen: does that omnimail job need a kick? [00:45:13] ejegg: it's curl timeouts [00:45:36] oh right [01:33:47] timeout seems too low tbh [01:38:34] (PS1) Ejegg: WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) [01:39:17] (CR) jerkins-bot: [V: -1] WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) (owner: Ejegg) [01:39:47] (CR) Ejegg: "Hmm, looks like 2to3 added lots of unnecessary parens in print statements" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) (owner: Ejegg) [01:42:08] (CR) Ejegg: "Oops, also needs T213757 done first" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) (owner: Ejegg) [02:02:00] (PS2) Ejegg: WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) [02:02:02] (PS1) Ejegg: Switch to pymysql [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520352 (https://phabricator.wikimedia.org/T213757) [02:02:37] (CR) jerkins-bot: [V: -1] Switch to pymysql [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520352 (https://phabricator.wikimedia.org/T213757) (owner: Ejegg) [02:02:54] (CR) jerkins-bot: [V: -1] WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) (owner: Ejegg) [02:06:09] (PS2) Ejegg: Switch to pymysql [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520352 (https://phabricator.wikimedia.org/T213757) [02:06:42] (CR) jerkins-bot: [V: -1] Switch to pymysql [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520352 (https://phabricator.wikimedia.org/T213757) (owner: Ejegg) [08:54:13] (PS1) Eileen: CiviCRM 5.16 stock [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/520386 [08:59:38] (CR) jerkins-bot: [V: -1] CiviCRM 5.16 stock [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/520386 (owner: Eileen) [13:23:29] hi fr-tech [13:24:02] ejegg: hi! [13:24:26] (CR) Ejegg: "OK, maybe not so drop-in" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520352 (https://phabricator.wikimedia.org/T213757) (owner: Ejegg) [13:27:21] PROBLEM - check_procs on frdb1001 is CRITICAL: PROCS CRITICAL: 1071 processes [13:32:19] RECOVERY - check_procs on frdb1001 is OK: PROCS OK: 213 processes [14:24:25] (PS3) Ejegg: Switch to pymysql [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520352 (https://phabricator.wikimedia.org/T213757) [14:30:44] (PS4) Cstone: Add India specfic fields for dlocal, street_address, city, PAN for fiscal_number, and x_version. Update unit tests to handle street_address and city. Create new donor test data for India. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519510 (https://phabricator.wikimedia.org/T224514) [14:32:35] (CR) jerkins-bot: [V: -1] Add India specfic fields for dlocal, street_address, city, PAN for fiscal_number, and x_version. Update unit tests to handle street_address and city. Create new donor test data for India. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519510 (https://phabricator.wikimedia.org/T224514) (owner: Cstone) [14:34:47] (PS5) Cstone: Add India specfic fields for dlocal, street_address, city, PAN for fiscal_number, and x_version. Update unit tests to handle street_address and city. Create new donor test data for India. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519510 (https://phabricator.wikimedia.org/T224514) [14:41:51] (PS3) Ejegg: WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) [14:41:53] (PS1) Ejegg: Rename queue module to prepare for python 2->3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520457 (https://phabricator.wikimedia.org/T197805) [14:42:46] (CR) jerkins-bot: [V: -1] WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) (owner: Ejegg) [14:45:00] cstone: street_address doesn't need the same isset test? [14:45:11] no it has a default value set [14:45:24] in StreetAddress.php [14:45:50] ah, that's right [14:47:25] ejegg: hey... Just making the ticket for the pgehres thingy... So it was a change in the timestamp format in the log filename, right? [14:48:03] AndyRussG: not in the filename, in the content of the logs [14:49:56] Ah hmmm ok [14:51:07] ejegg: do you have easily accessible where those logs live again? aaargh sorry for being disorganized, looking thru my notes [14:51:42] Oh I think I found it [14:51:49] AndyRussG: oh cool [14:52:07] I always forget - I just looked in the process-control logs [14:52:27] and they have the filenames of the tsvs that are being parsed [14:53:14] Ah heheh ok [14:58:36] ejegg: ok it's that 'Z' at the end that's new [14:59:06] cstone mind just reformatting that git commit message? https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines [14:59:20] sure [14:59:26] thanks! [15:00:17] AndyRussG: yep yep, that's all I could see [15:00:29] but it was enough to make the line not match our regex [15:02:09] okok [15:05:26] (PS6) Cstone: Add India specific fields for dlocal. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519510 (https://phabricator.wikimedia.org/T224514) [15:26:24] so the next step is to figure out when it changed, delete those records from the db, and re-run to backfill [15:26:33] fr-tech: any news for scrum of scrums? [15:32:25] oh shoot, sorry fr-tech, I didn't see the new meeting till just now! [15:32:52] I've got a conflict though - need to hop into the scrum of scrums call [15:35:57] ejegg: thanks! yeah nothing noteworthy here [15:44:03] (PS2) Ejegg: Disable rate limiting in Development Settings [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517393 (https://phabricator.wikimedia.org/T225796) (owner: Hashar) [15:44:12] (CR) Ejegg: [C: +2] Disable rate limiting in Development Settings [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517393 (https://phabricator.wikimedia.org/T225796) (owner: Hashar) [15:44:20] Fundraising-Backlog: pgehres data ingress broken - https://phabricator.wikimedia.org/T227203 (AndyRussG) [15:44:51] (PS4) Ejegg: installer: Detect APC for MainCacheType in CLI installer [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517397 (https://phabricator.wikimedia.org/T225496) (owner: Hashar) [15:45:00] Fundraising-Backlog: banner and landing page data ingress (pgehres) is broken - https://phabricator.wikimedia.org/T227203 (AndyRussG) [15:47:14] (CR) Ejegg: [C: +2] installer: Detect APC for MainCacheType in CLI installer [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517397 (https://phabricator.wikimedia.org/T225496) (owner: Hashar) [15:47:59] (PS3) Ejegg: Allow 'Z' at end of timestamp [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520344 (https://phabricator.wikimedia.org/T227203) [15:50:10] (PS1) AndyRussG: Update regex for new timestamp format in log lines [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520476 (https://phabricator.wikimedia.org/T227203) [15:50:53] (PS2) AndyRussG: [WIP] Update regex for new timestamp format in log lines [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520476 (https://phabricator.wikimedia.org/T227203) [15:51:31] (CR) AndyRussG: "Set as "WIP" because I haven't test it, though it's a very simple change..." [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520476 (https://phabricator.wikimedia.org/T227203) (owner: AndyRussG) [15:51:53] (CR) jerkins-bot: [V: -1] [WIP] Update regex for new timestamp format in log lines [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/520476 (https://phabricator.wikimedia.org/T227203) (owner: AndyRussG) [15:52:15] Fundraising Sprint Men In Slack, Fundraising-Backlog, Patch-For-Review: banner and landing page data ingress (pgehres) is broken - https://phabricator.wikimedia.org/T227203 (AndyRussG) [15:52:28] Fundraising Sprint Men In Slack, Fundraising-Backlog, Patch-For-Review: banner and landing page data ingress (pgehres) is broken - https://phabricator.wikimedia.org/T227203 (AndyRussG) [15:57:42] (Merged) jenkins-bot: Disable rate limiting in Development Settings [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517393 (https://phabricator.wikimedia.org/T225796) (owner: Hashar) [15:57:55] (Merged) jenkins-bot: installer: Detect APC for MainCacheType in CLI installer [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517397 (https://phabricator.wikimedia.org/T225496) (owner: Hashar) [15:59:34] (CR) jenkins-bot: installer: Detect APC for MainCacheType in CLI installer [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517397 (https://phabricator.wikimedia.org/T225496) (owner: Hashar) [15:59:57] (CR) jenkins-bot: Disable rate limiting in Development Settings [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517393 (https://phabricator.wikimedia.org/T225796) (owner: Hashar) [16:02:12] AndyRussG: oops, I put up a patch for that too! https://gerrit.wikimedia.org/r/520344 [16:07:48] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: remove some email domains from export - https://phabricator.wikimedia.org/T227206 (DStrine) [16:19:09] (CR) jenkins-bot: Disable rate limiting in Development Settings [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517393 (https://phabricator.wikimedia.org/T225796) (owner: Hashar) [16:55:29] (CR) jenkins-bot: installer: Detect APC for MainCacheType in CLI installer [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/517397 (https://phabricator.wikimedia.org/T225496) (owner: Hashar) [16:59:46] Fundraising-Backlog: Mysql Pgehres query not rendering Unique Clicks for Japan - https://phabricator.wikimedia.org/T227210 (KHaggard) [17:37:43] sorry to be so late for -talk, fr-tech [17:39:17] ejegg: all good, we agreed to rejoin in 5 'cause stuff also [17:39:26] ah, ok [18:39:16] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 13 others: Reduce recurring TY emails - https://phabricator.wikimedia.org/T213209 (Ejegg) Oh hey... [18:45:23] (PS1) Ejegg: Change name of opt-in field in export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520511 (https://phabricator.wikimedia.org/T225544) [18:51:13] XenoRyet: ^^^ [19:02:40] (PS7) Ejegg: Add India specific fields for dlocal. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519510 (https://phabricator.wikimedia.org/T224514) (owner: Cstone) [19:03:52] (CR) Ejegg: [C: +2] "This works locally, and adding the city and default street address doesn't seem to break the sandbox for other countries. We'll want to ch" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519510 (https://phabricator.wikimedia.org/T224514) (owner: Cstone) [19:04:48] (Merged) jenkins-bot: Add India specific fields for dlocal. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519510 (https://phabricator.wikimedia.org/T224514) (owner: Cstone) [19:22:19] (CR) XenoRyet: [C: +2] "Yep, easy peasy." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520511 (https://phabricator.wikimedia.org/T225544) (owner: Ejegg) [19:22:55] (Merged) jenkins-bot: Change name of opt-in field in export [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520511 (https://phabricator.wikimedia.org/T225544) (owner: Ejegg) [20:01:21] Fundraising-Backlog: Communication: Employer Name field is returning blank values - https://phabricator.wikimedia.org/T227091 (mepps) We talked about this in fortnightly today. @NNichols If you search for Communication:Employer Name Regex [a-zA-Z] that will return only those with values. (Not Empty works too... [20:24:33] Fundraising Sprint Land before Timeouts, Fundraising Sprint Men In Slack, Fundraising-Backlog: Rename Field in IBM export - https://phabricator.wikimedia.org/T225544 (Ejegg) @CCogdill_WMF this is ready to deploy. Can we push it out right now, or should we hold off till you've update things on the IBM... [20:27:46] Fundraising Sprint Land before Timeouts, Fundraising Sprint Men In Slack, Fundraising-Backlog: Rename Field in IBM export - https://phabricator.wikimedia.org/T225544 (CCogdill_WMF) Would it be possible to time this change with T170972 so we only have to remap the fields once? [20:39:36] Fundraising Sprint Land before Timeouts, Fundraising Sprint Men In Slack, Fundraising-Backlog: Rename Field in IBM export - https://phabricator.wikimedia.org/T225544 (Ejegg) Yep, totally. We'll hold off on deploying this one. [21:14:57] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/520549 (owner: L10n-bot) [21:16:12] (PS4) Ejegg: Recurring conversion: wire up close and 'no' buttons [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519544 (https://phabricator.wikimedia.org/T216560) [21:16:14] (PS4) Ejegg: Log line for showing recurring conversion ask [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519545 (https://phabricator.wikimedia.org/T216560) [21:17:38] (PS6) Ejegg: Base class for Donation APIs: step one [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519582 (https://phabricator.wikimedia.org/T216560) [21:17:40] (PS6) Ejegg: Donation API base: make $adapter a class property [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519583 (https://phabricator.wikimedia.org/T216560) [21:17:42] (PS7) Ejegg: Donation API Base: Bring Amazon in [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519584 (https://phabricator.wikimedia.org/T216560) [21:18:24] cstone: it's been a whole month (and more) since we updated the Thank You email text [21:19:02] I think you can finally deploy your code change to omit TY letters for 2nd, 3rd, etc recurring donations [21:20:28] (PS2) Ejegg: Add RecurringConversion interface [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/520149 (https://phabricator.wikimedia.org/T216560) [21:20:30] (PS3) Ejegg: WIP RecurringConversion API [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/519729 (https://phabricator.wikimedia.org/T216560) [21:20:32] (PS2) Ejegg: WIP Ingenico implements RecurringConversion [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/520150 (https://phabricator.wikimedia.org/T216560) [21:28:02] (PS4) Ejegg: Switch to pymysql [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520352 (https://phabricator.wikimedia.org/T213757) [21:28:43] (PS2) Ejegg: Rename queue module to prepare for python 2->3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520457 (https://phabricator.wikimedia.org/T197805) [22:08:20] (PS4) Ejegg: WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) [22:09:03] (CR) jerkins-bot: [V: -1] WIP python3 conversion [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/520350 (https://phabricator.wikimedia.org/T197805) (owner: Ejegg)