[00:00:54] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Benevity import failed - https://phabricator.wikimedia.org/T181548#3798698 (LeanneS) Interesting! Well it looks like they communicated in the end. I think we may see this again as volume wi... [00:03:24] (PS2) Ejegg: Fix last data point day [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394222 [00:04:58] XenoRyet: ok, that one passes jscs ^^^ [00:07:39] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Figure out a way to increase gateway time outs on import jobs - https://phabricator.wikimedia.org/T181688#3798721 (Eileenmcnaughton) [00:09:00] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Benevity import failed - https://phabricator.wikimedia.org/T181548#3798723 (Eileenmcnaughton) Open>Resolved [00:09:04] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Benevity import failed - https://phabricator.wikimedia.org/T181548#3793810 (Eileenmcnaughton) T181688: Figure out a way to increase gateway time outs on import jobs relates to the time out... [00:18:26] (CR) XenoRyet: [C: 2] Fix last data point day [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394222 (owner: Ejegg) [00:19:36] (Merged) jenkins-bot: Fix last data point day [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394222 (owner: Ejegg) [00:21:54] !log added weekly Ingenico audit processing job in makemissing mode [00:22:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:23:46] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/394230 [00:23:55] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/394230 (owner: Ejegg) [00:29:04] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/394230 (owner: Ejegg) [00:31:00] !log updated fundraising dashboard from 6ee656759561d524c1ed8a15ac4da4d0fce887a7 to 114131713e566069575f481281c74349e6bba336 [00:31:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:34:09] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3798757 (schoenbaechler) @bearND > On second thought. To make this easy, and since we seem to ag... [00:34:44] (PS1) Ejegg: Use api to insert address [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/394231 [00:35:02] (CR) Ejegg: [C: 2] Use api to insert address [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/394231 (owner: Ejegg) [00:36:31] (Merged) jenkins-bot: Use api to insert address [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/394231 (owner: Ejegg) [00:40:39] !log disabled CiviCRM jobs [00:40:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:42:30] !log updated CiviCRM from 0f95f3e37f9651a5fd262bf1f5e39929abee0d88 to e81228f79ee4a70b7a72e8aafbb5ee387da4230d [00:42:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:46:46] it's good to see the all clear for DS on our second day! [00:47:04] I'm going to sign off chat but I'm watching emails [00:48:48] !log re-enabled CiviCRM jobs [00:48:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:00:27] (PS3) Ejegg: Restored previous donation counter code (alongside new) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/393828 (owner: Jgleeson) [01:00:59] (CR) Ejegg: [C: 2] "Thanks, Jack!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/393828 (owner: Jgleeson) [01:04:35] (Merged) jenkins-bot: Restored previous donation counter code (alongside new) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/393828 (owner: Jgleeson) [01:07:03] Fundraising-Backlog, Patch-For-Review: Silverpop to Civi donor country error - https://phabricator.wikimedia.org/T181424#3798779 (Ejegg) @CCogdill_WMF I'm about to deploy this, but I realized it'll also deploy the new contact_hash column (see T177663). Is that OK? [01:08:42] (PS1) Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/394233 [01:10:55] (CR) Ejegg: [C: 2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/394233 (owner: Ejegg) [01:11:04] (Merged) jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/394233 (owner: Ejegg) [01:17:49] !log updated fundraising tools from 6d4b6f388e64ff4dc9a6a501bc8339fc1294fe9f to 6e604fd9f3ca57935e2c35e37b7e85b4ca438d6a [01:17:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:28:58] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3798787 (bearND) It's deployed but it'll take some time for it to be visible at https://en.wikiped... [01:44:39] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-PayPal-ExpressCheckout: Big batch of paypal_ec subscription payments with missing parent subscr_ids - https://phabricator.wikimedia.org/T181692#3798795 (Ejegg) [01:49:46] Fundraising-Backlog, Patch-For-Review: Silverpop to Civi donor country error - https://phabricator.wikimedia.org/T181424#3798807 (CCogdill_WMF) Thanks for the reminder on that. Do it! I'll pause our import for tonight and add the column. [02:18:19] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3798825 (Mholloway) Announcements are live! [02:20:17] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3798828 (Mholloway) Er, the endpoint response is live, that is; the in-app announcements will go l... [03:36:07] (PS1) Eileen: Allow Major gift contacts with conflicting addresses to merge. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394239 (https://phabricator.wikimedia.org/T181088) [03:41:03] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3798863 (bearND) Ok, it's visible now. [04:28:06] Fundraising-Backlog, WMF FR, Patch-For-Review: Silverpop to Civi donor country error - https://phabricator.wikimedia.org/T181424#3798872 (Ejegg) OK, the change is up! There were 225,000 addresses whose country changed with the update. The vast majority had been listed as 'US' due to the geolocated IP... [04:51:25] nick ejegg|away [08:15:41] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-2016-17-Q2-Campaign-Support, Performance, Spike: Spike: Look into transaction isolation level and other tricks for easing db contention - https://phabricator.wikimedia.org/T146821#3799033 (Marostegui) [09:30:28] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3799109 (schoenbaechler) @bearND @Mholloway I'm excited, thanks for the great collaboration. [10:15:22] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3799131 (jrobell) Just jumping in to say Thank you to everyone involved for your help getting thes... [13:09:10] Fundraising Sprint RadioActivewear, Fundraising Sprint Synchronized Screaming, Fundraising Sprint Uptight Piano, Fundraising Sprint Vaporwerewolf, and 5 others: Registering test in impressions - https://phabricator.wikimedia.org/T176334#3799591 (Pcoombe) Thanks @AndyRussG! I added the code above... [13:33:57] Fundraising Sprint RadioActivewear, Fundraising Sprint Synchronized Screaming, Fundraising Sprint Uptight Piano, Fundraising Sprint Vaporwerewolf, and 5 others: Registering test in impressions - https://phabricator.wikimedia.org/T176334#3799678 (Pcoombe) Ah, no wonder we're still getting a few po... [13:48:13] (CR) Mepps: "Hmm, I'm not sure I know enough about what "On hold" means in this context. My assumption would be that we would use the address not on ho" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394239 (https://phabricator.wikimedia.org/T181088) (owner: Eileen) [14:38:42] hi jgleeson! [14:38:55] hey there mepps :) [14:39:23] how are you? feeling better [14:39:37] I've hurt my back again [14:39:40] a little bit, not 100% [14:39:41] oh no! [14:42:27] I'm on painkillers now so it's eased off a bit. I was a little too enthusiastic last night putting together Oscars birthday present. It's a plastic activity house/slide thingy and pushing the sides together required you to use your full body weight so I was basically climbing all over it to click [14:42:39] it together and seems to have triggered my back :\ [14:43:01] hopefully be better for his party tomorrow [14:43:22] good to hear you're better [14:44:11] the fundraising dash cheered me up and took my mind of it for a while :) [15:52:40] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3800134 (bearND) [15:53:46] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3776010 (bearND) [15:55:13] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5000 (threshold c = 5000) [15:55:42] huh [15:55:55] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3800155 (bearND) Updated some of the parameters in the description to reflect the values we ended... [15:57:45] https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&panelId=14&fullscreen&orgId=1&from=now%2Fw&to=now [15:57:49] that does look like a problem [16:00:13] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5038 (threshold c = 5000) [16:02:09] it looks like a ton of bounces [16:02:13] guessing thank yous [16:05:04] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5065 (threshold c = 5000) [16:05:59] i'm going to try flushing the queue [16:06:12] though if this is all undeliverable mail i don't think it will help [16:07:13] Heads up MBeat et al, we just launched mobile and ipad [16:07:21] ty pcoombe [16:09:40] pcoombe: I noticed :P [16:09:44] https://usercontent.irccloud-cdn.com/file/sllms95o/image.png [16:10:13] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5082 (threshold c = 5000) [16:15:04] PROBLEM - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5141 (threshold c = 5000) [16:19:41] ACKNOWLEDGEMENT - check_mailq on civi1001 is CRITICAL: CRITICAL: postfix mailq is 5141 (threshold c = 5000) Casey Dentinger Bounced thank you mail awaiting retries [16:33:20] Seddon: did pcoombe get back to you? [16:35:08] dstrine Seddon : about what? [16:35:54] I see seddonn shared an image [16:36:20] I'm not sure what each of those lines means [16:36:31] there's no legend to the graph [16:36:37] It's just the spike in impressions when we launched mobile [16:37:19] ok I read that out of context/fearing the worst [16:37:48] I thought it could have been error codes [16:41:08] woohoo! [16:44:39] cool, that doubled traffic to paymentswiki [16:46:23] we reached the same level for an hour or so Tuesday, but I think this'll be more sustained [16:49:45] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3800304 (bearND) BTW, Now it's live live. [16:54:42] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3800329 (Mholloway) \o/ {F11091204} [16:58:24] yeah today is gonna be big [17:01:40] ejegg: AndyRussG meeting? [17:12:35] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Recalculate exchange rates for Nov. 25 & 26 - https://phabricator.wikimedia.org/T181735#3800403 (Ejegg) [17:46:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Figure out a way to increase gateway time outs on import jobs - https://phabricator.wikimedia.org/T181688#3800497 (Ejegg) Note: 'Gateway Time-out' means nginx stopped listening, not that apache stopped processing. Though @JGreen says the nginx timeout i... [18:06:36] ejegg: I noticed the donations queue started building once mobile switched on -- [18:06:36] https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&from=now-6h&to=now&panelId=9&fullscreen [18:09:38] Seddon yep, there was also an email that hit around the same time [18:09:53] so we're currently getting donations faster than we can import them into Civi [18:10:04] The import is still running full tilt, though [18:19:29] Fundraising Dash, Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Unplanned-Sprint-Work: dash minification busted under debian stretch - https://phabricator.wikimedia.org/T181467#3800616 (Ejegg) Got a workaround, running requirejs's optimiser directly node [18:29:31] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi access for Fundraising Contractor - https://phabricator.wikimedia.org/T181740#3800623 (LeanneS) [19:07:56] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-iOS-App-Backlog, Reading-Infrastructure-Team-Backlog (Kanban): Some iOS users seeing US and Canada banners - https://phabricator.wikimedia.org/T181743#3800698 (JMinor) [19:11:54] (PS1) Ejegg: Update readme with minify workaround [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394371 (https://phabricator.wikimedia.org/T181467) [19:12:58] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog: Some iOS users seeing US and Canada banners - https://phabricator.wikimedia.org/T181743#3800753 (Mholloway) Must be a client issue. The endpoint response looks correct. [19:13:11] XenoRyet: were you thinking of tackling those extraneous http requests from the dash? [19:14:52] (CR) jerkins-bot: [V: -1] Update readme with minify workaround [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394371 (https://phabricator.wikimedia.org/T181467) (owner: Ejegg) [19:15:17] I hadn't been thinking about anything in particular yet. Actually ended up poking around CentralNotice most of the morning, in case that goes wonky while I'm on-call. [19:17:54] (PS2) Ejegg: Update readme with minify workaround [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394371 (https://phabricator.wikimedia.org/T181467) [19:21:37] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog: Some iOS users seeing US and Canada banners - https://phabricator.wikimedia.org/T181743#3800805 (bearND) a:bearND>None I'm not sure I can help here except make a change to the announcement to turn it off for Canada or something similar. It really... [19:22:15] fr-tech anybody have any insight into those constraint violations? I'm at least as puzzled as eileen [19:25:34] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog: Some iOS users seeing US and Canada banners - https://phabricator.wikimedia.org/T181743#3800811 (JMinor) Thanks guys. Its definitely not everyone on iOS (out of 6 people in the meeting, 2 had this issue, both located in SF). I'll check OTRS this afternoon... [19:27:24] (CR) Eileen: "If we don't unset the conflict in that function it will block the merge. I think it might have be that we are not adequately handling that" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394239 (https://phabricator.wikimedia.org/T181088) (owner: Eileen) [19:37:53] (PS1) Ejegg: Fix logging for duplicate subscr_cancel [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394378 [19:38:17] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog: Some iOS users seeing US and Canada banners - https://phabricator.wikimedia.org/T181743#3800843 (bearND) Ok, I think I figured it out. If my theory is correct then most of California users would be affected. :( The cookie header start with this in Califo... [19:38:33] fr-tech I goofed on the logging when the paypal audit file includes a subscr_cancel for an already cancelled subscription [19:38:56] that was the reason for the rash of bad_audit_line failmails today [19:39:05] not actually paypal's fault this time! [19:39:24] Anyway, I think that patch should fix it: https://gerrit.wikimedia.org/r/394378 [19:39:35] Anyone got a second to take a look? [19:42:27] I've got to run out and grab some food for my wife real fast, but I can take a look when I get back if nobody beats me to it. [19:44:55] looks simple enough [19:44:58] no CI yet tho [19:45:09] is there CI on that repo? [19:45:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10160 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 2 hours - memory use is 11.02M (peak 23.12M, 0.18% of max, fragmentation 1.37%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 395, payments-antifraud is 139, payments-init is 565, pending is 257, recurring is 43, refund is 0, unsubscribe is 97 [19:45:15] err, that line doesn't happen to be covered by tests. [19:45:18] welp there's that [19:45:20] suppose I should add that! [19:45:26] oh hey, 10k in the queue [19:45:32] is that a thing to celebrate? [19:45:44] heh, a mixed blessing [19:45:52] i'll just ack it cause nothing we can do [19:46:14] I could tweak timing a bit [19:46:31] to run faster? [19:47:05] y'know, I DO want to see what happens when we let the ty mailer keep running during the next qc run [19:47:24] now that we have db conflicts not actually dropping messages [19:47:33] probably a lot of that stuff was dialed back during the replag investigation [19:48:40] I notice the address change is now causing the errors to show up the same as the email ones - [19:50:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10080 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 3 hours - memory use is 10.75M (peak 23.12M, 0.19% of max, fragmentation 1.41%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 440, payments-antifraud is 80, payments-init is 292, pending is 129, recurring is 69, refund is 0, unsubscribe is 105 [19:50:20] yep, just saying i feel confident enought to try some more overlappy job runs [19:50:58] think that will reduce the queue lag or should i ack the alert? [19:54:00] cwd that's the hope! [19:54:09] groovy [19:54:11] i'll wait and see [19:55:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10315 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 3 hours - memory use is 11.28M (peak 23.12M, 0.18% of max, fragmentation 1.28%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 381, payments-antifraud is 226, payments-init is 836, pending is 287, recurring is 91, refund is 0, unsubscribe is 110 [19:55:30] !log adjusted timings of Civi jobs to let TY and QC run concurrently [19:55:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:55:51] !log 105 seconds for each job [19:56:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:56:30] * ejegg practices crash position [19:56:47] great idea, or terrible idea? Let's see! [19:57:02] fr-tech heads up, experimenting ^^^ [19:57:31] the TY job does its big nasty select when it starts up [19:57:50] which will still be after the queue consumer stops inserting [19:58:21] At 105 seconds for the queue consumer, that gives the TY job 15 seconds to lock tables before the next QC run starts [19:58:59] the TY job does do selects from tables for each send, but those are all pretty well targeted by ID [19:59:10] so I hope they don't block inserts [20:00:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10461 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 3 hours - memory use is 11.25M (peak 23.12M, 0.18% of max, fragmentation 1.34%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 396, payments-antifraud is 141, payments-init is 546, pending is 198, recurring is 111, refund is 0, unsubscribe is 123 [20:00:37] lessee... looks like maybe an inflection point [20:02:34] ohhh, shoot. that's not actually how allow_overtime works :( [20:03:00] still not allowing concurrency, just not failmailing [20:03:15] well dang [20:04:08] ok, so... [20:04:32] ah rats [20:04:44] got a silly workaround idea [20:05:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10663 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 3 hours - memory use is 11.17M (peak 23.12M, 0.19% of max, fragmentation 1.37%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 373, payments-antifraud is 66, payments-init is 270, pending is 276, recurring is 43, refund is 0, unsubscribe is 21 [20:05:42] have we had a big mailing go out? [20:06:00] db lock contention is the bottleneck for sure? [20:06:00] eileen: mailing + turned on mobile banners [20:06:14] ah [20:06:31] well maybe today will get that nmber 1 slot that we missed yesterday [20:07:12] when mysql hits a lock it does a definite slow down when retrying [20:08:35] but we aren't getting that many fail mails off them [20:08:43] so I guess they are not happening all the time [20:09:31] yeah but in general that's the bottleneck of the queue processing, waiting for locked resources? [20:09:49] nothing has particularly high load [20:10:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10563 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 3 hours - memory use is 11.52M (peak 23.12M, 0.19% of max, fragmentation 1.35%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 389, payments-antifraud is 191, payments-init is 841, pending is 230, recurring is 67, refund is 0, unsubscribe is 35 [20:10:58] cwd so if there is a lock there seems to be a bit of a delay before mysql says 'nup [20:11:31] we have it set to retry 3 times - but what we are seeing is that on the first retry it has rolled something back & it fails & we get a failmail [20:11:43] (although on some jobs not so much) [20:12:14] but, we are not seeing that happening even once an hour so I don't think it is the actual locks that are the issue [20:12:27] try 3 times? doesn't mysql wait for the lock to become available? [20:12:37] or do you mean the job itself retries on a deadlock? [20:12:49] or a lock wait timeout? [20:13:04] on a deadlock - although we are picking up locks too - but there are very few cases so not sure it's an issue [20:13:10] in terms of spee [20:13:11] d [20:13:28] This is more how I remember the queues in previous years [20:13:39] ie. not catching up for a couple of days [20:13:47] heh oh yeah the lag is nothing new [20:13:54] !log re-adjusted Civi job timings. QC every odd min for 105 sec, TY every even min for 70 sec after 45 sec delay [20:13:56] always fun to poke at though [20:13:56] yesterday we were spookily close to realtime [20:14:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:14:33] the master db is new since last year and faster so we should be faster [20:15:10] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 10142 10000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 3 hours - memory use is 11.15M (peak 23.12M, 0.19% of max, fragmentation 1.42%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 415, payments-antifraud is 132, payments-init is 591, pending is 254, recurring is 93, refund is 0, unsubscribe is 40 [20:15:17] inflection! [20:15:47] nice [20:15:55] maybe we should change the alert to 11k :P [20:20:58] (CR) XenoRyet: [C: 2] Fix logging for duplicate subscr_cancel [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394378 (owner: Ejegg) [20:21:27] (Merged) jenkins-bot: Fix logging for duplicate subscr_cancel [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394378 (owner: Ejegg) [20:22:04] ty XenoRyet [20:23:22] ok, looks like we're getting more imported and more thanked every 2 minutes [20:23:39] I think the jobs are slowing each other down a little bit [20:23:53] but it's still more effective overall [20:26:35] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog: Some iOS users seeing US and Canada banners - https://phabricator.wikimedia.org/T181743#3801031 (JMinor) Okay @JoeWalsh has confirmed this is indeed the issue. We actually had a bugfix release planned for the next few days, so we'll patch this in the clie... [20:30:07] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog, iOS-app-v5.7.2: iOS app users in California seeing both US and Canada banners - https://phabricator.wikimedia.org/T181743#3801044 (bearND) > Per this and recent convos with @Fjalapeno I think we'll need to iterate on how we plan, test and target annou... [20:31:31] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog, iOS-app-v5.7.2: iOS app users in California seeing both US and Canada banners - https://phabricator.wikimedia.org/T181743#3801047 (JMinor) > It would be helpful to communicate announcement requests much earlier. Agreed, this is mostly on me. Fundrais... [20:31:35] (PS1) Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/394396 [20:31:45] (CR) Ejegg: [C: 2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/394396 (owner: Ejegg) [20:31:51] (Merged) jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/394396 (owner: Ejegg) [20:34:42] ooh, look at that queue delay drop! [20:35:13] and we're still seeing a good amount of traffic [20:35:25] ejegg: I was just about to ask wth you just did. [20:35:27] :D [20:35:34] K4-713: it's ugly... [20:35:39] Oh good. [20:36:04] overlapping the queue consumer and the TY mailer, except for the first 15 seconds of the TY mailer which does that gnarly select [20:36:14] Aha! [20:36:29] Haa. Ahahaha. [20:36:32] Nice. [20:36:38] they seem to slow each other down a little bit [20:36:58] Yeah, but it's hard to argue with that slope. [20:37:04] i.e., # processed per second of actual run is a bit lower [20:37:19] but yeah, the overall effect seems positive [20:38:07] XenoRyet: mind deploying that audit parser fix? [20:38:20] project is just called 'tools' [20:38:40] Yea, no problem. [20:38:45] thanks! [20:39:44] oh hey, if this works out we can afford to create civimail records for 100% of TY mails [20:42:07] aw drat [20:42:23] there goes a bunch of contention [20:43:06] how the HECK are we going ahead with the email insert when the contact insert is rolled back??? [20:45:44] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3801141 (Pcoombe) Thanks everyone! Can confirm that donations are coming in from this, I'll update... [20:46:01] oh you already merged it to deploy. [20:46:04] right. [20:49:12] !log Updated tools from 6e604fd9f3ca57935e2c35e37b7e85b4ca438d6a to 626fe02a9f5ffcc24c200858644d697607679738 [20:49:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:56:23] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, and 2 others: Run Big English fundraising on apps - https://phabricator.wikimedia.org/T181004#3801151 (JMinor) Thanks for sharing the sheet! Very cool to see the donations coming in... [21:00:10] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 65 days 4 hours - memory use is 7.10M (peak 23.12M, 0.19% of max, fragmentation 2.16%), connected_slaves is 2, donations is 4646, jobs is 0, jobs-adyen is 0, jobs-paypal is 382, payments-antifraud is 151, payments-init is 612, pending is 313, recurring is 119, refund is 1, unsubscribe is 187 [21:15:00] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog, iOS-app-v5.7.2: iOS app users in California seeing both US and Canada banners - https://phabricator.wikimedia.org/T181743#3801205 (Pcoombe) @JMinor The fundraising team just discussed this. Given the volume of app use in California, and the comparativ... [21:17:10] (PS1) Ejegg: Better c_t country backfill [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394406 (https://phabricator.wikimedia.org/T181424) [21:17:35] (CR) jerkins-bot: [V: -1] Better c_t country backfill [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394406 (https://phabricator.wikimedia.org/T181424) (owner: Ejegg) [21:17:52] hmm [21:21:06] ohh we are catching up - did something change or just the natural peak & trough [21:22:17] (PS2) Ejegg: Better c_t country backfill [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394406 (https://phabricator.wikimedia.org/T181424) [21:22:44] XenoRyet: I think that's a little better than yesterday's patch ^^ [21:23:27] looking [21:37:20] (CR) XenoRyet: [C: 2] Better c_t country backfill [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394406 (https://phabricator.wikimedia.org/T181424) (owner: Ejegg) [21:38:09] (Merged) jenkins-bot: Better c_t country backfill [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394406 (https://phabricator.wikimedia.org/T181424) (owner: Ejegg) [21:48:19] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog, iOS-app-v5.7.2: iOS app users in California seeing both US and Canada banners - https://phabricator.wikimedia.org/T181743#3801278 (JMinor) I think we might have missed the deploy window... @bearND is it still possible to change the endpoint? Maybe jus... [21:48:58] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, fundraising-tech-ops: Give Civi Access to Thea - https://phabricator.wikimedia.org/T181527#3801310 (TSkaff) @cwdent Hi Casey, I installed the cert and can login to https://civicrm.wikimedia.org/user but when clicking on Dash or CiviCRM from t... [21:50:17] ejegg, eileen - is that a user permissions thing tskaff is seeing? [21:50:30] i created the account but it might be too basic [21:52:58] cwd not sure - what does she mean by 'dash ' or civi? [21:53:08] tskaff: are you there? [21:53:18] @eileen I am [21:54:00] so you log in & then click on 'dash; on the left hand menu? [21:54:24] Correct [21:54:39] (and/or CiviCRM from left hand menu) [21:54:44] ah I see - you didn't have a role [21:54:45] Same error msg [21:54:48] try now [21:55:01] - are you with donor services? [21:55:11] Ok civiCRM looks good! [21:55:20] No I'm on the online fund team [21:55:29] dash looks better - nice $ amount :-) [21:55:32] But Dash ... nope [21:57:22] hmm [21:57:29] so you click on the dash link and? [21:58:23] I click on the link and 1. Redirected to https://dash.frdev.wikimedia.org/ and 2. Click login and redirected back to https://civicrm.wikimedia.org/oauth/authorize?oauth_token=Zh3LgomSSm5gKMyV8SV7eBgRQ4PFZdGm [21:58:57] & what do you see on that? [21:59:01] (and see "You are not authorized to access this page.") [21:59:17] OK - i'll try a different civi role [21:59:28] Thanks! [21:59:48] done - try again [22:00:57] There it is! Many thanks! [22:01:22] (PS1) Ejegg: WIP refactor dedupe [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394433 [22:01:30] great - cwd ejegg turns out that the CiviCRM Admin role gives the rights to dash [22:01:42] eileen oh? [22:01:42] Donor Services role gives access to civi but not dash [22:01:54] Ideally we would have an inbetween role [22:01:56] ooh, lemme see if I can fix that [22:02:06] shoudl ds have dash access? [22:02:19] sure, that sounds fine [22:02:20] I mean I know some should - but all? [22:02:36] I don't think it would hurt [22:02:48] The ppl that enter checks wouldn't need it [22:02:48] (I don't know how broad the ds net is / whether there should be any nuance … mbeat FYI [22:03:31] So there is an 'Engage' role [22:03:58] I think historically we have given a lot of people civicrm admin because we hadn't quite gotten perms right on the other roles [22:04:35] right, let's see what we can tweak with oauth access [22:04:38] typical DS agent is not on Dash or dev [22:04:54] or in Engage [22:05:14] ejegg: why don't we create a dash role [22:05:25] & allocate that specifically - to make it really clear [22:06:04] that way you can either have CiviCRM admin or any role + dash [22:06:24] (BTW and in Hundertwasser art centre meeting & the bell went off :-) [22:06:27] huh, I just don't see where to tell it what roles can oauth [22:06:33] oh [22:06:35] hmm [22:06:47] well - maybe just log a 'one-day-phab' [22:09:12] fr-tech I'd love to get some feedback on this Silverpop export refactor: https://gerrit.wikimedia.org/r/394433 [22:10:04] cwd / Jeff_Green heads up on db load - I might test that out in my own db on fr-dev later today [22:10:48] :-) [22:11:01] I took a quite look & definitely looks worth trying! [22:14:04] huh interesting, i will check it out [22:18:14] 5th place day now [22:19:22] looks like we'll just edge out yesterday for 2nd [22:19:41] first is just too high :-( [22:32:08] ah drat, just thought of a problem with that refactor [22:32:45] ooh, it's actually a problem with some of the current code [22:34:03] really? [22:34:06] but... it's only a problem when two actually different contacts have the same email address, one primary and one not-primary [22:34:06] yuck [22:34:25] if it still needs review on (my) Monday I can do a review then [22:34:46] ah, this is just a for-fun project, not any kind of rush! [22:34:55] I convinced myself that the MG one wasn't too scary :-) [22:35:08] oh hey, I should look at that! [22:35:21] ah ok - well I'm planning on working my Mon to cover the call [22:36:00] oh right, you're not supposed to be working today, huh? [22:36:15] go plan your art center! [22:39:50] Wikimedia-Fundraising, Wikipedia-iOS-App-Backlog, Patch-For-Review, iOS-app-v5.7.2: iOS app users in California seeing both US and Canada banners - https://phabricator.wikimedia.org/T181743#3801441 (JMinor) Thanks @Mholloway! [22:41:07] ejegg: I'm in the meeting being rude :-) [22:41:27] & working on my laptop - just piping up with off-topic comments every now & then [22:41:47] turns out we need to get 10,000 used bricks [22:41:59] & get them in a state for the building [22:42:21] & every light fitting in the building has to be different [22:45:06] wat [22:45:37] apparently it's part of the requirements to do the Hundertwasser build [22:46:03] rad [22:46:10] home depot end cap clearance items ftw [22:46:15] :-) [22:46:24] nope - litterally they have to be recycled [22:46:46] for the small build we got a tonne out of fireplace that was being dismantled [22:47:00] but need more than that this time [22:50:07] (PS1) Ejegg: WIP more Silverpop export refactor [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394466 [22:59:02] ejegg: did someone turn up the engine for processing donations :P [23:01:03] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-iOS-App-Backlog, Patch-For-Review, and 2 others: iOS app users in California seeing both US and Canada banners - https://phabricator.wikimedia.org/T181743#3801474 (Mholloway) [23:11:16] (PS1) Ejegg: WIP further refactor [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/394494 [23:12:30] Seddon: we put some nitrous in the tank [23:12:56] Getting a few backfires (messages to damaged queue), but it's speedy! [23:13:18] ejegg I hope you are sat on top of the server spitting petrol into it Mad Max style [23:18:05] I live, I die, I live again! [23:18:49] Witness me! [23:24:27] ok, I'm timing that latest change on frdev1001 [23:42:25] well, we passed yesterday in donation count! [23:42:40] total amount is looking less likely [23:48:52] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Ingenico should treat CLP as no-decimal - https://phabricator.wikimedia.org/T181768#3801606 (Ejegg) [23:53:10] oh hey, did i make some replag with that silverpop test? [23:53:24] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-iOS-App-Backlog, Patch-For-Review, and 2 others: iOS app users in California seeing both US and Canada banners - https://phabricator.wikimedia.org/T181743#3801621 (bearND) cache purged as of :31 past the hour. [23:55:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [23:57:01] orly