[00:00:24] (CR) Eileen: "recheck" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/625727 (https://phabricator.wikimedia.org/T261359) (owner: Eileen) [00:01:37] (Merged) jenkins-bot: Remove recurring_globalcollect from modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/626006 (owner: Eileen) [00:09:19] (CR) Cstone: [C: +2] re-add remaining wmf hacks. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/625728 (https://phabricator.wikimedia.org/T261359) (owner: Eileen) [00:09:59] (CR) Cstone: [C: +2] 5.30-rc upgrade (stock) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/625727 (https://phabricator.wikimedia.org/T261359) (owner: Eileen) [00:10:30] Fundraising-Backlog, Wikimedia-Fundraising-Banners, fr-donorservices: JP form: add translations for One time/Monthly, Other, and Amount - https://phabricator.wikimedia.org/T262361 (MBeat33) [00:12:25] Fundraising-Backlog, Wikimedia-Fundraising-Banners, fr-donorservices: JP form: add translations for One time/Monthly, Other, and Amount - https://phabricator.wikimedia.org/T262361 (MBeat33) p:Triage→Unbreak! [00:16:42] (CR) jerkins-bot: [V: -1] 5.30-rc upgrade (stock) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/625727 (https://phabricator.wikimedia.org/T261359) (owner: Eileen) [00:16:44] (CR) jerkins-bot: [V: -1] re-add remaining wmf hacks. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/625728 (https://phabricator.wikimedia.org/T261359) (owner: Eileen) [00:19:56] (CR) Eileen: [V: +2] "v+2 based on following patch passing & gerrit no longer coping with that" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/625727 (https://phabricator.wikimedia.org/T261359) (owner: Eileen) [00:20:53] (CR) Eileen: "recheck" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/625728 (https://phabricator.wikimedia.org/T261359) (owner: Eileen) [01:10:59] (PS1) Eileen: Submodule update - update to 5.30rc [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/626020 [01:19:31] gerrit not loading for me :-( [01:21:48] (CR) Eileen: [C: +2] Submodule update - update to 5.30rc [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/626020 (owner: Eileen) [01:23:20] (CR) Eileen: [C: +2] "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/626020 (owner: Eileen) [01:29:17] (Merged) jenkins-bot: Submodule update - update to 5.30rc [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/626020 (owner: Eileen) [01:40:43] gerrit is really unresponsive today [01:45:51] :( [01:47:08] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/626025 [01:51:03] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/626025 (owner: Eileen) [01:53:35] (CR) Eileen: [C: +2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/626025 (owner: Eileen) [01:56:41] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog: Rogue contacts in _all_wikimedia database preventing accurate queries - https://phabricator.wikimedia.org/T261705 (Eileenmcnaughton) This commit might be worth reviewing before going too far down this https://gerrit.wikimedia.org/r/c/wik... [01:58:32] !log civicrm revision changed from 4e40a59d42 to cc1f7e6d13, config revision is 4845a229dc [01:58:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:25:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1493 [02:30:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [02:35:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2093 [02:40:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2394 [02:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2693 [02:46:42] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2693 Dwisehaupt Checking on replication lag [02:50:21] eileen: did your update at 0158 UTC include a table alter? looks like what may be causing the lag on frdev1001. [02:50:37] dwisehaupt: I didn't think so [02:50:45] the lag I had wasn't frdev [02:50:52] but I guess ^^ is [02:51:13] yeah, looks like a pretty simple one: /* User : 0 */ALTER TABLE `civicrm`.log_civicrm_activity MODIFY `log_id` bigint(20) unsigned AUTO_INCREMENT [02:52:00] not a big deal, just making sure it's not something worse. and possibly another case where it rusgood on the prod hosts but gets lag on staging. [02:52:37] hmm - odd - I think that's a field civi doesn't know about [02:52:44] which we added for replication [02:54:24] I can't think of any reason why the update I did would have triggered that [02:54:54] ok. i'll look a little more. the timing seems to line up right. [02:56:53] that column was added by Jeff for replication reasons [02:57:10] did the int() size get increased ? [02:57:50] unsure, need to do some checks. [02:59:37] nope. the dev_ column has the same, so looks like it may be a table rebuild. [03:00:20] yeah - but why? [03:01:46] not sure. defintely came from the master. i see the query still on frdb1003 also, but it's stuck in with some know laggy cube rebuilds so it's a bit expected to be slower there. [03:03:32] https://www.irccloud.com/pastebin/ZF4Tir85/ [03:04:13] it's possible it also would modify that table if it's schema was out of sync with the civicrm_activity table - but I see no reason for that to be the case [03:05:21] hmmm... yeah. not sure. [03:08:28] looks like it's just still running on these two. i'm gonna let it roll a little longer and check some things. but also need to get the kids pointed in the direction of bed. otherwise they won't have energy for the ocean tomorrow. [03:11:41] :-) [03:23:50] looks like it's still progressing. at 39% reported instead of 23%. gonna let it roll but should probably dig into what triggered it. [03:23:58] see you all tomorrow. [03:36:29] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Analyse queries running during queue processing - https://phabricator.wikimedia.org/T261688 (Eileenmcnaughton) a:Eileenmcnaughton [03:53:59] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Get drush to accept debug_query context - https://phabricator.wikimedia.org/T262368 (Eileenmcnaughton) [06:22:28] fundraising-tech-ops, Operations, netops, observability: update nagios_nsca configuration in frack for new nsca servers - https://phabricator.wikimedia.org/T262291 (ayounsi) [07:14:35] Fundraising-Backlog, Wikimedia-Fundraising-Banners, fr-donorservices: JP form: add translations for One time/Monthly, Other, and Amount - https://phabricator.wikimedia.org/T262361 (CDenes_WMF) Hmm this is very strange. We proofed this and updated the translations back in July, as there is a comment t... [07:16:59] Fundraising-Backlog, Wikimedia-Fundraising-Banners, fr-donorservices: JP form: add translations for One time/Monthly, Other, and Amount - https://phabricator.wikimedia.org/T262361 (CDenes_WMF) As for the Japanese translations for the strings that are still in English, they reside here (see page 42):... [08:53:05] Fundraising-Backlog, Wikimedia-Fundraising-Banners, fr-donorservices: JP form: add translations for One time/Monthly, Other, and Amount - https://phabricator.wikimedia.org/T262361 (jrobell) When clicking the link Michael added I get the text in English as well, however, I think the language parameter... [12:17:25] Fundraising-Backlog, Wikimedia-Fundraising-Banners, fr-donorservices: JP form: add translations for One time/Monthly, Other, and Amount - https://phabricator.wikimedia.org/T262361 (CDenes_WMF) Thanks Jessica. I also tested out the flow. Starting from the WTG page, I clicked on Japanese (in the langua... [12:24:44] tzag fr-tech! [12:33:56] hmmm looks like we got some queue consumer failmail through the night [12:34:24] I wonder if eileen managed to fix the composer package blocker and pushed out the new code and that's introduced something [12:47:20] hmmm so this isn't new, it's another one of the newish 'Error: UNKNOWN missing essential contribution tracking ID' failures that cstone added a ticket for here https://phabricator.wikimedia.org/T256274 [12:49:32] hmmm so how would a ct message make it to the queue with no ct_id [12:49:34] hmmm :/ [12:53:10] Wikimedia-Fundraising-Banners: Japan banner emojis - https://phabricator.wikimedia.org/T261972 (Pcoombe) Yes @TSkaff I think these are good to go. There's still {T261976} but that's only on a few older systems and I don't think the missing emojis disrupt the message too much [13:18:07] hi jgleeson [13:19:34] hi mepps [13:24:28] how goes it today jgleeson? [13:26:44] not too bad thanks mepps, just trying to get used to standing up a lot :) how about you? [13:27:11] I'm digging into this bug with the missing contribution tracking id that you looked at https://phabricator.wikimedia.org/T256274 [13:27:44] I found one clue, looks like the one I looked at was a large donation. I'm seeing this in the logs 'Notifying of large donation, contribution: 49729289 above threshold 499.99' [13:28:38] which is making me wonder if the reconciliation we do for offline donations is maybe falling over due to larger donations? just a first guess [13:29:50] I found that message in /srv/archive/frlog1001/logs/error-20200909.gz on frlog1001 searching for 49729289 [13:30:39] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog: Error: UNKNOWN missing essential contribution tracking ID - https://phabricator.wikimedia.org/T256274 (jgleeson) a:jgleeson [13:31:56] hmm that's an interesting guess jgleeson.. [13:33:29] hmmm maybe this one wasn't an offline donation [13:33:34] I'm seeing Source [13:33:36] online donation [13:33:38] in civi [13:34:27] oooo [13:34:34] this is a paypal donation [13:34:59] but I didn't see any logs for it on the frontend [13:35:09] interesting jgleeson!! [13:35:22] so I'm now thinking maybe the donor made the donation within paypal [13:35:27] i'm going to go jump rope to work off my over caffeination [13:35:35] nice! [13:35:36] hmm [13:36:36] the donor already has a recurring donation of a smaller amount [13:37:02] I'll log into my own paypal to see if I can make a "one-off" donation through it [13:44:21] hmm I can't see anyway to donate from within paypal [13:44:27] hey fr-tech :) [13:45:00] hey AndyRussG :) [13:45:05] :) [13:58:11] jgleeson if we have an email address attached to our account you could send money to it [14:00:16] mepps: I just checked my recurring payment and I do see donate@wikimedia.org [14:00:27] under the seller info bit [14:00:39] I'll see if I can make a small donation to the email address and watch out for it [14:02:07] looks like we don't allow donations via that route [14:02:12] when I try to send I see: [14:02:14] This recipient accepts PayPal payments only through their website. To complete this payment, please go to their website and follow their instructions, or email the recipient for instructions. [14:02:25] shoot [14:12:58] fundraising-tech-ops, Operations, netops, observability: Add alert[12]001 to network ACLs - https://phabricator.wikimedia.org/T260533 (Jgreen) [14:13:02] fundraising-tech-ops, Operations, netops, observability: update nagios_nsca configuration in frack for new nsca servers - https://phabricator.wikimedia.org/T262291 (Jgreen) Open→Resolved p:Triage→Medium a:Jgreen Config change is deployed to puppet and appears to be working fro... [14:15:15] ahh that's good you ruled that out jgleeson [14:29:57] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog: Error: UNKNOWN missing essential contribution tracking ID - https://phabricator.wikimedia.org/T256274 (jgleeson) I licked this cookie also but couldn't work out what's causing it. I was digging into [[ https://civicrm.wikimedia.org/civ... [14:31:12] ok I'm gonna put that task down now since I'm not progressing much on it [14:31:24] and go back to the adyen iframe variant [14:31:50] strange though how we've got a paypal express donation with no frontend log data, only backend log data (smashpig listener) [14:38:47] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog: Error: UNKNOWN missing essential contribution tracking ID - https://phabricator.wikimedia.org/T256274 (jgleeson) a:jgleeson→None [14:47:16] Wikimedia-Fundraising-Campaigns: Fundraise in India - https://phabricator.wikimedia.org/T131118 (Pcoombe) Open→Resolved a:Pcoombe [15:03:26] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog, fr-donorservices: French TY email sending donors to unsubscribe page in English - https://phabricator.wikimedia.org/T260814 (mepps) @Ejegg my read of the unsubscribe code is that core mediawiki uses fr instead of fr-fr for language c... [15:06:54] hi fr-tech! [15:10:42] hi ejegg [15:10:59] ejegg: :) [15:11:05] Hi ejegg jgleeson AndyRussG [15:11:13] cstone: :) [15:11:26] hi cstone [15:11:38] fr-tech so I'm installing payments inside docker [15:12:00] I checked out the fundraising/REL1_35 branch of core [15:12:13] then did git submodule update --init --recursive [15:12:22] hi AndyRussG [15:12:34] you'll need a couple of as-yet-unmerged changes to DonationInterface [15:12:34] hey :) [15:12:41] ahhh okok [15:12:54] Is that like a lightcandy thing? I'm getting an error for that with composer install [15:12:55] (and FundraiserEmailUnsubscribe if you want tests to pass) [15:13:04] yep, the lightncandy version [15:13:07] The requested package zordius/lightncandy 0.23 exists as zordius/lightncandy[v1.2.5] but these are rejected by your constraint. [15:13:11] and a slight tweak to the invocation [15:13:27] cstone's patch will fix that - one sec for the link [15:13:42] invocation: hocus-poocus instead of hocus-pocus [15:13:44] ? [15:13:47] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/622401 [15:14:18] heh, more like ('hocus', 'pocus') instead of (['hocus', 'pocus') [15:14:42] grr, the old one was supposed to be (['hocus', 'pocus']) [15:17:59] ahhh array or just params, a classic gotcha across languages [15:18:16] ejegg: yay the DI patch got composer install working! thx! [15:24:04] Fun bash prompt inside docker when you run as the default user (uid 1000): I have no name!@e3b783fed804:/var/www/html$ [15:24:26] Fundraising-Backlog, fundraising-tech-ops: SSH and Yubikey setup for HNordeenWMF - https://phabricator.wikimedia.org/T262344 (HNordeenWMF) @Dwisehaupt my Yubikey's Public side: cccccctrnutt Thanks! [15:25:07] fr-tech inside docker, I tried php maintenance/update.php --quick [15:25:09] but it said [15:25:18] your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies [15:25:22] should I do that? [15:28:33] hmm [15:29:24] AndyRussG: you can run `composer update --dry-run` to see what WOULD happen [15:30:57] AndyRussG: oh hmm [15:31:04] I thought we had done that already [15:31:07] the update [15:31:23] and `composer update --lock` should fix the error without you needing update anything [15:31:43] to* [15:31:53] phooey, our crufty HTML verification tests are not working with REL1_#% [15:32:03] err REL1_35 [15:32:09] new keyboard... [15:32:39] I guess DOMDocument can't handle newer HTML elements like