[03:55:06] Wikimedia-Fundraising-Banners: [Candidate] Foundation logo - https://phabricator.wikimedia.org/T269247 (jbolorinos-ctr) Other than the suggestion above, no blockers to report. This banner is READY TO TEST! [03:56:08] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 3 - https://phabricator.wikimedia.org/T269250 (jbolorinos-ctr) Screenshot Test Results - Desktop: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z97875289f3f7d608910 [04:01:40] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 2 - https://phabricator.wikimedia.org/T269249 (jbolorinos-ctr) Open→Resolved No blockers found. Closing this task as Resolved, this banner variant is READY TO TEST!! @spatton [04:21:15] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 3 - https://phabricator.wikimedia.org/T269250 (jbolorinos-ctr) Open→Resolved No blockers to report. @spatton signing off on QA for this one as well. This banner is READY TO TEST! [08:35:30] Fundraising-Backlog, Wikimedia-Fundraising-Banners, Browser-Support-Apple-Safari: "Donate Now" banner doesn't do anything in Safari - https://phabricator.wikimedia.org/T269529 (Aklapper) Open→Stalled [13:04:55] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Turn off import of RML during peak load - https://phabricator.wikimedia.org/T269185 (krobinson) Hi all, just to chime in here, this is actually causing workflow issues for DS. We are unable to unsubscribe donors... [15:27:18] (CR) Mepps: "This patch was super helpful for me to see! I'm a little curious about naming--do we use numbers in queues elsewhere?" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) (owner: Ejegg) [16:05:10] (CR) Jforrester: [C: +1] build: Updating mediawiki/mediawiki-codesniffer to 34.0.0 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/645853 (owner: Umherirrender) [16:07:39] (CR) Umherirrender: [V: +2] "T224764" [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/645820 (owner: Libraryupgrader) [16:27:40] hi fr-tech! [16:27:49] I'm getting a late start today [16:33:04] hi ejegg! [16:48:11] Fundraising-Backlog, fundraising-tech-ops, FR-Tech-Analytics: Efficiency of querying views - https://phabricator.wikimedia.org/T259731 (Jgreen) This query from today's slow_log takes about 55s on frdev: `SELECT month_received_sorted AS month_received_sorted, gateway_not_null AS gateway_not_nu... [17:53:27] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Turn off import of RML during peak load - https://phabricator.wikimedia.org/T269185 (MBeat33) [17:58:33] Fundraising-Backlog: Outdated Paypal rule contributing to declines - https://phabricator.wikimedia.org/T269611 (EMartin) [18:40:50] fr-tech anyone available to take a quick look at this patch mepps and I made to filter some log spam? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/645177 [18:42:55] i can ejegg [18:43:29] thanks cstone! [18:43:57] looks like that config param was actually consulted in the first draft of the logger [18:44:23] but then in a refactor (in late 2013) we stopped consulting it, and it's been [18:44:26] useless ever since [18:47:33] (PS4) Ejegg: WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [18:47:49] mepps and jgleeson ^^^ is the minimal version of the patch [18:48:00] Just adds a config value for number of queues to use [18:48:05] (CR) jerkins-bot: [V: -1] WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) (owner: Ejegg) [18:48:17] oops, let's see what failed lint there [18:48:42] i was stepping through that a bit late on Friday ejegg to get a better idea of what was going on [18:48:50] nice that there is a more minimal version that still works [18:49:16] looks good ejegg [18:49:26] thanks! [18:51:55] the only minor comment I have is the name fan-out [18:53:02] was thinking of maybe something like "queue-instances" but that might be too general [18:53:34] (PS5) Ejegg: WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [18:54:15] but yeah I guess fan-out is the pattern [18:54:19] so makes sense [18:54:53] https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchanges [18:55:42] actually does fan out mean broadcast to all queues [18:55:45] reading this https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-fanout [18:56:33] (PS6) Ejegg: Adyen capture job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [18:56:45] oh, I'm not sure [18:57:03] maybe simpler to just say capture-job-queue-count ? [18:57:08] and a couple chars shorter [18:57:22] yeah maybe [18:57:33] hmm [18:57:44] ah ok, from that diagram it looks like fanout does send to all [18:57:48] renaming... [18:58:12] (CR) Cstone: [C: +2] Restore log level filtering [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645177 (https://phabricator.wikimedia.org/T269371) (owner: Ejegg) [18:58:50] (Merged) jenkins-bot: Restore log level filtering [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645177 (https://phabricator.wikimedia.org/T269371) (owner: Ejegg) [18:59:22] (PS7) Ejegg: Adyen capture job queue multiplexing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [18:59:28] ok, renamed and rebased [18:59:45] oops, variable still called FanOut [19:00:33] (PS8) Ejegg: Adyen capture job queue multiplexing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [19:04:09] ejegg: looks +2'able but might wanna remove fan-out from cmt msg if we wanna be really picky! :) [19:04:50] (PS9) Ejegg: Adyen capture job queue multiplexing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [19:04:52] edited! [19:05:47] (CR) Jgleeson: [C: +2] "Super straightforward! Nice work" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) (owner: Ejegg) [19:06:19] (Merged) jenkins-bot: Adyen capture job queue multiplexing [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) (owner: Ejegg) [19:07:24] thanks! [19:07:54] OK, I'll deploy that standalone SmashPig and get started on the library update to get the logging fix onto payments [19:13:02] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/646806 [19:13:25] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/646806 (owner: Ejegg) [19:14:07] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/646806 (owner: Ejegg) [19:17:20] Re the silverpop job - if we've started emails it might be making it more of an issue [19:17:30] it might not be 'worse' -just noisier [19:23:12] !log updated standalone SmashPig payments listener from 3029b07004 to e3103b96ca [19:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:42:27] mepps: the mediawiki unit tests session bumps into our meeting :/ [19:42:37] I guess we could watch the first bit at least [19:49:32] I sent an email on this but we just throttled traffic from the main 6C campaign to 60%. [19:50:39] nice tskaff ! [19:50:47] always a good sign!! [19:52:10] tskaff: thanks for the update. that is a very good sign. [19:53:44] wow 2020 is hitting hard on the top scores https://analytics.frdev.wikimedia.org/superset/dashboard/47/ [19:59:27] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog: Turn off failmail for silverpop timeouts. - https://phabricator.wikimedia.org/T226747 (Eileenmcnaughton) a:Eileenmcnaughton I've claimed this - but I'm specifically focussing on catching the sessionId timeouts [19:59:35] jgleeson: ops said they'd be happy postponing it [20:00:10] dstrine: OK to move our ops synch-up a half hour later so that fr-tech can attend a relevant org-wide tech talk? [20:05:21] fr-tech, screen-share Murphy's law [20:05:31] yarp [20:05:40] and also Chrome==new IE [20:05:48] ha [20:09:56] ejegg: oh ok [20:10:11] thank you dstrine ! [20:10:18] fr-tech ^^^ [20:10:49] we're moving the standup half an hour later so we can watch the tech talk on mw unit tests [20:10:57] live stream here: https://www.youtube.com/watch?v=HOWKHUA-wAI [21:03:53] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog: Turn off failmail for silverpop timeouts. - https://phabricator.wikimedia.org/T226747 (Eileenmcnaughton) I think this will work but the challenge is testing it https://github.com/mrmarkfrench/silverpop-php-connector/pull/34 [21:04:04] jgleeson: I think this will work - https://github.com/mrmarkfrench/silverpop-php-connector/pull/34 - I just have to figure out how to test [21:05:33] Fundraising-Backlog: Outdated Paypal rule contributing to declines - https://phabricator.wikimedia.org/T269611 (DStrine) a:DStrine→None [21:19:15] Fundraising-Backlog: Outdated Paypal rule contributing to declines - https://phabricator.wikimedia.org/T269611 (DStrine) @EMartin fr-tech pretty much unanimously voted for #2 "Update the rule logic to decline payments with $0 < amount < $1 to avoid those BA payments." but this is something paypal will have t... [21:23:51] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Adyen: Process Adyen IPN messages in parallel (improve TY delivery times) - https://phabricator.wikimedia.org/T233251 (Ejegg) Code is deployed, but we need to deploy some config changes - Add definitions for jobs-adyen-1, jobs-adyen-2, and j... [21:32:04] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-Banners: Error when trying to complete a recurring paypal transaction - https://phabricator.wikimedia.org/T269303 (DStrine) Open→Resolved [21:33:53] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Code review and release management for CN - https://phabricator.wikimedia.org/T268646 (DStrine) [21:41:49] Fundraising-Backlog, Wikimedia-Fundraising-Banners, Browser-Support-Apple-Safari: "Donate Now" banner doesn't do anything in Safari - https://phabricator.wikimedia.org/T269529 (jbolorinos-ctr) @jpryan00 thanks so much for reporting this! Would you be able to provide the version of Safari you were usi... [21:48:49] Fundraising-Backlog: Outdated Paypal rule contributing to declines - https://phabricator.wikimedia.org/T269611 (EMartin) Thanks David! I'll let them know and ask them to confirm they've done it and update the case. Evelyn [22:01:00] dstrine: not sure what to pickup next, any ideas? [22:02:00] the Swedish form ready for review ejegg ? [22:02:29] jgleeson: sure! It's pretty trivial [22:07:11] ok cool, I'm heading off now but if possible ejegg|brb could you add a note on how to test? and I'll pick it up tomorrow. Thanks in advance! [22:07:43] Wikimedia-Fundraising-Banners: [BETA] M Lg Payments First Trilogy banner - https://phabricator.wikimedia.org/T269631 (Aklapper) [22:10:10] I captured the timeout [22:11:33] and it was different than expected (sigh) [22:14:18] nice eileen ! [22:14:26] how so? [22:14:42] the error code was ... not in the error code element [22:22:03] ah [22:23:17] I just looked at donation q consumer - last run was 346 contacts in 40 secs - so we could definitely take a break on it IMHO to run rml [22:49:27] So how do people feel about this [22:49:27] 1) I disable queue consumers (en masse - using the disable [22:49:27] 2) I run the rml job manually [22:49:27] 3) I revert the disable [22:49:27] 4) we assess how long it took etc [23:04:08] eileen: so i'm clear, we would wait to do 3 until after 2 has completed. is that correct? [23:06:29] yep [23:06:49] cool. yeah. i'm fine with that. i'll be around. [23:06:50] but I'd likely just run once & then revert & assess whether to keep running [23:07:11] ok cstone ejegg|brb [23:07:40] we know that currently we could recover a 30 min backlog at 100% banners in about 30 mins. [23:08:07] so i don't have a lot of fear about this from a technical side. [23:09:25] yep - I just put a message out about it on the ds channel [23:11:54] ejegg: I just pushed a disable-the-queues job for the above - does that need checking by someone else [23:12:03] (I just ran the script) [23:12:17] nope [23:12:37] that should be pretty automatic [23:13:44] !log process-control config revision is f48cdb9184 [23:13:44] temporarily disable queues [23:13:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:17:46] dwisehaupt: I've just added the new adyen queues to my local copy of the puppet repo in my homedir on frpm [23:18:10] latest commit touches modules/fundraising/templates/smashpig/main.yaml.erb [23:18:39] when you get a chance can you copy that to the real repo? [23:18:46] No urgency [23:18:55] oops, gotta run out again... [23:20:30] sure [23:32:20] ejegg|afk: i've pushed out your change. also pushed out the prometheus metric collection for the new queues. no icinga checks yet. [23:32:30] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7479 5000, payments-init is 1902 1500, pending is 2304 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 days 2 hours - memory use is 16.11M (peak 83.87M, 0.28% of max, fragmentation 1.41%), connected_slaves is 2, donations is 2239, jobs is 0, jobs-adyen is 4, jobs-paypal is 1050, recurring is 134, refund is 2, unsubscr [23:34:53] dwisehaupt: that ^^ is redis not mysql? I snuck in a slow-ish sql query [23:35:00] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7479 5000, payments-init is 1902 1500, pending is 2304 2000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 days 2 hours - memory use is 16.11M (peak 83.87M, 0.28% of max, fragmentation 1.41%), connected_slaves is 2, donations is 2239, jobs is 0, jobs-adyen is 4, jobs-paypal is 1050, recurring is 134, refund is 2, [23:35:00] Dwisehaupt known - testing RML so queues will back up. [23:35:47] eileen: yeah. that is redis. expected with the queues off. [23:36:59] ok - it looks like 1 run took around 15 mins - I ran it a second time & when that finishes will turn it back on. My analysis also suggests the run is around 10,000 contacts and the queue is around 200000 [23:37:35] that's not too horrible i think. [23:38:58] yeah - so it would take another 18 runs to catch up I think [23:42:22] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog: Turn off failmail for silverpop timeouts. - https://phabricator.wikimedia.org/T226747 (Eileenmcnaughton) Currently it looks like there are 198516 rml contacts to be imported https://engage4.silverpop.com/lists.do?action=listSummary&listId=18468760... [23:44:11] !log process-control config revision is b3bc1959cd [23:44:11] queues are back on [23:44:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:45:45] queues are back on, queue backed up to around 3000 [23:47:28] the biggest backup was in the antifraud queue. i'll keep an eye on that one but i think it should clear pretty soon. [23:47:34] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 21 days 2 hours - memory use is 7.12M (peak 83.87M, 0.17% of max, fragmentation 1.98%), connected_slaves is 2, donations is 2969, jobs is 0, jobs-adyen is 0, jobs-paypal is 1102, payments-antifraud is 218, payments-init is 159, pending is 7, recurring is 200, refund is 2, unsubscribe is 76 [23:57:34] queues cleared really quick. [23:59:11] I'd be happy to do the whole exercise again to try to get caught up a bit... [23:59:16] ccccccknfuvhjnvgbieilthfriiilrkfirubfccidefb [23:59:59] is there a reason not to test a run with the queues running? it would be nice to see the impact of it under a little load.