[00:29:11] (CR) Eileen: "recheck" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/407196 (owner: Eileen) [00:34:09] (CR) jerkins-bot: [V: -1] Temp hack fix [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/407196 (owner: Eileen) [12:23:59] (Abandoned) MarcoAurelio: Don't allow simple registered users to vote 'verified' [extensions/DonationInterface] (refs/meta/config) - https://gerrit.wikimedia.org/r/407912 (owner: MarcoAurelio) [13:29:37] hey fr-online :) is there a campaign or something going on at the moment that would explain a surge in donations ? [13:31:10] I'm looking at our Prometheus stats and it looks like something is happening on the right-hand side of the chart https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=5s&orgId=1&from=now-2d&to=now&panelId=15&fullscreen [14:10:57] hi jgleeson_ ! [14:11:07] hey ejegg :) [14:12:26] maybe some banner tests? [14:12:45] let's take a look at https://meta.wikimedia.org/wiki/Special:CentralNotice [14:13:16] jgleeson_: yep, looks like there are banners live in Italy and Sweeden [14:13:34] I thnk this is actual fundraising even, not just tests [14:13:47] There's a schedule someplace, I'll make sure you have access to it [14:14:02] ah cool [14:14:09] checkout the very bottom graph https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&from=now%2Fd&to=now%2Fd [14:14:15] that's out processing rates per batch [14:14:48] I'm just trying to remove the rounding [14:15:35] I have a vague memory of precision settings somewhere [14:22:11] ohhh, were you pointing out missing data at the right side of the graph? [14:22:36] ah no, I was trying to understand the spike in donations [14:22:53] as it changed when I was making updates so wanted to make sure I hadn't broken anything :) [14:23:05] it looks ok tho, i double/tripple checked everything [14:23:09] ah, i've just got some visual gitch I guess [14:23:26] can you see the new graph at the bottom https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&from=now%2Fd&to=now%2Fd ? [14:23:43] the time UTC is the same as the last data point, my screen is just showing a bunch of blank future hours to the right [14:23:54] yeah, cool! [14:24:06] ah that's because the filter is today so I think it shows the hours for the remainder of the day [14:24:10] odd I know [14:24:36] those are total donations? [14:25:18] so that processing rate is the measurement of the time it took for the queue consumer process in seconds [14:25:35] ah, ok [14:26:51] I'm still aware that we may need to consider other time factors in this metric [14:27:33] although with the current data you can see the spikes in the time it takes to process the batches, guessing the spikes represent bursts of donations https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&from=now-7d&to=now [14:34:04] jgleeson_: that probably has to do with the duty cycle for the IPN processing job [14:34:21] so, almost all our paypal donations still come in via the legacy integration [14:34:29] which means we don't get a notification from the front end [14:34:32] just from the IPN [14:34:45] then we process a batch of those every 5 minutes [14:35:58] the paypal-specific job just transforms those messages into standard queue messages and passes them along to the appropriate queue [14:36:08] i.e. donations, refunds, recurring [14:36:23] so then our donation queue consumer gets a burst of donations every 5 minutes [14:36:34] ah I see [14:36:38] we could definitely change that though [14:36:52] have the paypal processor run every 2 minutes too [14:37:19] I don't think it's very processor intensive [14:45:30] I've also added in the new label-based donations data to the current graph which at current is invisible, although if you enable them and select the gateway specific filter key and compare it to the "new" equivalent (gateway_new) the only thing that should change is the colour. https://grafana-admin.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1&from=now%2Fd&to=now&panelId=15&fullscreen&edit [15:03:18] morning! [15:04:41] hey AndyRussG :) [15:05:41] hi AndyRussG ! [15:05:58] jgleeson_ ejegg :) [15:14:12] (PS1) Ejegg: Update Amazon SDK fork for TCP proxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408278 (https://phabricator.wikimedia.org/T180168) [15:14:16] fr-tech, when you run unit tests for the payments wiki... do you pass a wiki=paymentswiki flag? [15:14:52] jgleeson_: ooh, good question. I think there's an environment variable that might help there [15:15:59] hmm apt-get dist upgrade this morning, see where that takes me [15:16:06] jgleeson_: try exporting MW_INSTALL_PATH [15:16:08] export MW_INSTALL_PATH=/vagrant/mediawiki-fr [15:16:11] this looks like the one [15:16:13] yep! [15:16:27] is that in our wiki notes yet? [15:16:28] thank you :) [15:16:45] yup located it here https://www.mediawiki.org/wiki/Fundraising_tech#Running_PHPUnit_tests_under_vagrant_(mediawiki-fr) [15:16:50] ah, good [15:16:54] once I located the right wiki :) [15:18:08] AndyRussG: huh, still no package for the 4.15 kernel, just the rcs [15:18:27] ah looks like we've got some php7 warnings tripping up unit tests [15:18:36] will dig into these! [15:18:39] oh huh [15:18:58] Have I been ignoring those locally? [15:19:04] no patching them [15:19:07] where possible [15:20:09] most of the issues I've run into so far have been constant declaration's [15:20:18] so easily resolve with a check [15:20:23] resolved* [15:21:48] hmmm with these being notices [15:22:03] I wonder if it would just be quicker to disbale then as part of the cli command [15:22:15] disable* [15:24:14] pcoombe, this is the video I was talking about at all hands https://www.youtube.com/watch?v=UlNT16gqHyo [15:25:51] fr-tech this seems to work for enabling a TCP proxy in the Amazon SDK: https://github.com/ejegg/login-and-pay-with-amazon-sdk-php/blob/master/PayWithAmazon/HttpCurl.php#L73 [15:26:07] (PS2) Ejegg: Update Amazon SDK fork for TCP proxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408278 (https://phabricator.wikimedia.org/T180168) [15:26:59] Anybody want to check it out before we merge that verison update ^^^ ? [15:27:44] ejegg, all I see there is a composer lock file change [15:28:01] do I need to select something to see other changes? [15:28:15] Yep, the actual code change is in the link i posted just before [15:28:43] got it [15:31:49] looks good to me ejegg [15:31:55] thanks [15:32:19] (CR) Ejegg: [C: 2] Update Amazon SDK fork for TCP proxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408278 (https://phabricator.wikimedia.org/T180168) (owner: Ejegg) [15:41:58] fr-tech, are tests passing for you if you run php phpunit.php --wiki paymentswiki --group DonationInterface on the payments wiki? [15:42:16] firing up vagrant... [15:42:58] ah drat, virtualbox complaining about the kernel module. [15:43:27] ahh, i'm on one of the 4.15 rcs. lemme just reboot real quick [15:43:34] np [15:44:22] (Merged) jenkins-bot: Update Amazon SDK fork for TCP proxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408278 (https://phabricator.wikimedia.org/T180168) (owner: Ejegg) [15:44:55] jgleeson_: Ah cool, yes I'd seen that one before. Wikimedia UK have always been at the forefront of GLAM work [15:54:41] jgleeson_: among other things, looks like the db connection for ContributionTracking is mutated during unit tests [15:55:01] oh wait, is that in my local settings? [16:26:28] ejegg jsut saw the emails about being unable to donate [16:26:35] do we know how they got to that page? [16:26:58] oh i just noticed the attached image [16:27:00] weird [16:27:33] ugh, also we're not catching exceptions correctly there :( [16:27:40] I'll make a couple tickets [16:27:47] so if i just go through wikimediafoundation.org i have no problem [16:28:17] and if i try to go straight to payments.wikimedia.org it redirects me to donatewiki [16:28:19] mepps the issue doesn't show up till you click donate on the amazon form and actually pay money [16:28:41] well, I guess it won't cost money if we can't make the sdk call [16:28:43] oh okay, i was just looking at the screenshot [16:29:04] nothing should have changed about ssl on payments [16:30:09] are the donations going through ejegg? or are they getting this error before then? [16:34:23] mepps they're not - the error comes on the first API call [16:34:32] ah okay ejegg [16:40:32] Fundraising-Backlog, FR-Amazon, MediaWiki-extensions-DonationInterface: Catch Amazon SDK connection errors - https://phabricator.wikimedia.org/T186537#3946575 (Ejegg) [16:47:21] fr-tech added a section in CN doc for quick development setup: https://www.mediawiki.org/wiki/Extension:CentralNotice#Quick_developer_setup [16:47:34] just some lines to add to LocalSettings.php [16:47:35] nice AndyRussG! [16:47:43] good stuff [16:48:37] ejegg have you seen the error message for amazon? i was looking at the adapter just now coincidentally [16:48:44] still looking into this incrementing issue [16:50:36] :) [16:51:19] Pls lmk of any problems.... thx!!!!!!!! [16:59:34] jgleeson_: to answer your earlier question, I do see two failures in vagrant unit tests, both having to do with protocol-relative urls [17:10:37] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408306 [17:11:28] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408306 (owner: Ejegg) [17:12:36] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408306 (owner: Ejegg) [17:12:42] grr [17:13:45] (PS1) Ejegg: Update Amazon SDK for proxy [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/408307 [17:13:54] (CR) Ejegg: [C: 2] Update Amazon SDK for proxy [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/408307 (owner: Ejegg) [17:14:27] (CR) Ejegg: [V: 2 C: 2] Update Amazon SDK for proxy [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/408307 (owner: Ejegg) [17:15:07] (PS2) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408306 [17:25:22] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408306 (owner: Ejegg) [17:26:41] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408306 (owner: Ejegg) [17:26:54] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/408309 [17:27:09] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/408309 (owner: Ejegg) [17:36:22] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/408309 (owner: Ejegg) [17:52:24] !log updated payments-wiki from 341cb573a1 to 39a7ef32e5 [17:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:53:22] ejegg, thanks. I see a few more errors so I'm gonna put it down to environmental differences and see if I can work out what's up. [17:53:50] I'm also gonna remove the WIP from my vagrant branches and submit them to be merged so they don't get lost [18:04:00] great! [18:04:04] I think I have +2 there [18:21:36] Heheh I almost created an eventlogging schema called CentralNoticeImpession [18:21:49] At least the missing "r" would have saved us a few much-needed bits [18:21:56] hehe [18:22:37] I wonder, are bits indivisible? Can an information unit smaller than a bit exist? [18:30:41] Also (on a more work-y note) I wonder if now is the time to start deprecating some beacon/impression parameters that we don't care much about [18:31:19] For example, db, which is just whatever is in $wgDBname [18:31:45] I think I'm gonna do it [18:34:03] is db different from lang? [18:37:56] ejegg: ooooooooooooooh wait a siec [18:37:57] sec [18:38:16] Hmmm [18:38:18] No scratch that [18:38:59] I was thinking it was a way for us to fill in some missing data when we don't really know what lang is gonna be for logged-in users, but I was confusing impressions logs with pageview logs [18:39:13] On pageview logs is where we can only tell what the URL gives us [18:39:42] On impressions, it should be all in the URL [18:39:51] Though I guess sometimes db might be more convenient than URL parsing [18:40:14] Yeah db should be combination of WMF project and the project's default language, most of the time, something like enwiki [18:40:18] I guess I'll leave it then [18:41:24] ^ (I mean, on impressions, what's in DB should be already in the URL, and language as used by CN targeting is in the uselang param that we get back.) [18:41:29] arrrrrrg [18:41:37] Beep, beep, I'm a sheep [18:43:02] heh [18:43:10] ok, I think I get it [18:44:27] arrrg sorry not very clear [18:46:41] Wikimedia-Fundraising: Create translated versions of new Thank You page - https://phabricator.wikimedia.org/T185333#3947079 (TSkaff) @schoenbaechler +1 template to rule them all. [18:50:37] hrmmm anyway it's an opportunity to document these params, since the schema calls for a description field [18:55:33] dstrine: is there an actual tech priorities meeting in 5 min? [18:55:43] hey ejegg, i'm looking at the ingenico integration, and i'm curious, where do we actually call do_transaction('createhostedcheckout')-- i can't find it or how the adapter would know that's the way to start the process [18:56:49] mepps looks like we call it from the donation.api.php [18:57:00] since we're doing an ajax submit [18:58:48] oh yeah i see it now, thanks [19:08:52] ejegg how come globalcollect and paypal express don't use the increment function? [19:09:43] mepps not sure about pp express [19:10:04] but globalcollect (old integration) doesn't use the ct.seq order ID format [19:10:10] it just generates some random numbers [19:10:29] ahh sorry ejegg, brain mush, i know we already talked about that [19:10:44] good point on pp express tho [19:11:03] might need to update that [19:11:21] in case of users canceling and starting over [19:26:31] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: investigate auto updater features for new ingenico - https://phabricator.wikimedia.org/T186554#3947191 (DStrine) [19:32:35] https://meta.wikimedia.org/wiki/Schema:CentralNoticeImpression [19:50:05] jgleeson_: did you figure out anything about the 'sudo' issue bd808 pointed out in that vagrant patch to update_composer? [19:50:21] other than that i'm ready to +2 the whole thing [19:58:07] (PS1) Ejegg: Remove stray semicolons from yaml [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408346 [19:58:46] (PS1) Ejegg: WIP enable Adyen for CC in Netherlands [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408347 [20:05:54] (CR) jerkins-bot: [V: -1] Remove stray semicolons from yaml [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408346 (owner: Ejegg) [20:06:09] wait, really? [20:07:05] ugh, weird packagist thing [20:11:22] ejegg, no not yet. I'm gonna have another look and see if I can work it out [20:41:11] Fundraising-Backlog: Create Phabricator tag for Fundraising analytics tasks - https://phabricator.wikimedia.org/T186236#3947419 (AndyRussG) [20:42:26] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog: Create Phabricator tag for Fundraising analytics tasks - https://phabricator.wikimedia.org/T186236#3947423 (DStrine) a:DStrine [20:43:08] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: centralnotice_analytics: tests and sample data - https://phabricator.wikimedia.org/T186234#3938039 (DStrine) [20:43:32] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: centralnotice_analytics: simple Hive integration - https://phabricator.wikimedia.org/T186233#3947428 (DStrine) [20:44:53] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: centralnotice_analytics: Complete inline documentation - https://phabricator.wikimedia.org/T186232#3947430 (DStrine) [20:45:51] ejegg: amazon proxy is working? [20:46:45] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Add yearly totals to WMF donor table - https://phabricator.wikimedia.org/T186165#3947438 (DStrine) [20:48:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Process greetings for Major Gifts contacts - https://phabricator.wikimedia.org/T186162#3947446 (DStrine) [20:48:54] cwd well... it's working for the payments front-end and report downloader [20:49:00] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Process greetings for Major Gifts contacts - https://phabricator.wikimedia.org/T186162#3935425 (DStrine) [20:49:23] the IPN listener just sent out a couple of failmails though :( [20:49:33] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, MediaWiki-Vagrant: Fix vagrant bugs affecting fundraising tech role - https://phabricator.wikimedia.org/T186119#3947464 (DStrine) [20:49:34] so I rolled it back [20:49:40] harder to test that bit locally [20:49:58] aah right on [20:50:01] well progress anyway [21:31:23] catch you all tomorrow o/ [21:41:59] mepps: re upstreaming (activity & others) - I'm hitting enough issues around 4.7.31 (the DAO->free & tightened option value validation) that I think we should just go with our patches over 4.7.31 without too much more upstreaming (I've done some) and then visit that issue again [21:49:11] eileen, i'm paused on that right now while i work on ingenico, but when are you thinking of revisiting? [22:00:04] mepps: hmm after we are on 4.7.31 [22:00:30] perfect time would be at a UK sprint - but not sure when that will happen as the people are there [22:00:52] there are a few patches that need revisiting [22:05:39] trying to deal with this one ATM - https://github.com/civicrm/civicrm-core/pull/11633/commits [22:06:11] oh huh [22:06:44] so we need to have some language entries with the same code but different labels? [22:07:14] (PS1) Eileen: CRM-21737 loosen option value validation to support language option group [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/408432 [22:07:18] oh hey, my fix_address pr got merged :) [22:07:24] ejegg: :-) [22:07:41] The rc for 4.7.31 is cut on Wed & that's what I want to deploy [22:07:51] so I'm trying to get things merged for Wed where I can [22:08:24] (I'm OK going with an rc to deploy as I don't think much changes between rc & go live & we will test a little on staging [22:08:37] & our tests are picking up things we care about [22:08:37] eileen so what's the use case for the language group needing duplicate values? [22:08:50] ejegg: crazy historical idiocy [22:09:08] sounds bad [22:09:14] they used value for 'en' & name for 'en_AU' [22:09:37] oh yuck [22:09:40] this is the geocoding one I want to get in https://github.com/civicrm/civicrm-core/pull/11540 [22:10:52] & this one https://github.com/civicrm/civicrm-core/pull/11615 I doubt anyone will be brave enough to merge for the rc - but I want to put it in ours & get merged for the next roung [22:11:08] (being able to say 'yes we have deployed it') [22:11:30] This one - https://github.com/civicrm/civicrm-core/pull/11621 might let us drop one patch [22:11:54] wow, you've been busy! [22:12:00] :-) [22:12:11] this one is for a dedupe issue & trying to resolve this one https://github.com/civicrm/civicrm-core/pull/11519 - but guzzle issues [22:12:36] basically core doesn't ship with guzzle but as of Mar it *could* ship with latest guzzle [22:12:43] & I want to make the case it should [22:13:02] Mar is last release with < 5.5 php support [22:14:12] From what I can tell the only risk with Guzzle is if people are already using a different version in their CMS. THere are only 4 extensions with guzzle …. & 3 are by me :-) [22:14:44] heh, need to get civi core using composer merge [22:16:00] yep - for D8 I think the whole install is composer [22:16:01] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [22:16:11] D? [22:16:16] drupal [22:16:32] I meant the phab bot... [22:16:41] oh right, Differential [22:16:46] the gerrit replacement [22:17:13] ah - I'm just thinking about the broader implication for CiviCRM users of adding Guzzle 6 to the package [22:17:42] not how / whether we can manage it - I think the mere fact of 'require_once' means we don't care if Civi loads a version of guzzle in it's vendor or if we do [22:18:04] but if someone had some obscure Joomla! module loading Guzzle 5 it could be an issue [22:18:11] ah yeah [22:18:41] ejegg i'm about to jump off, but can you take a look at: https://gerrit.wikimedia.org/r/#/c/407738/ [22:18:43] k, i'm going to head back to the neighborhood. back in a bit [22:18:48] sure mepps! [22:18:50] (CR) jerkins-bot: [V: -1] CRM-21737 loosen option value validation to support language option group [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/408432 (owner: Eileen) [22:18:58] no rush, just wanted to make sure you saw it [22:19:15] oh btw - don't worry about reviewing any of my patches against Civicrm - the imports against CRM we should complete but the civicrm patches are all wip [22:36:48] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: Add history link for banners - https://phabricator.wikimedia.org/T132732#2208618 (MarcoAurelio) Could this work and be added to [[ https://phabricator.wikimedia.org/diffusion/ECNO/browse/master/special/SpecialCentralNoticeBanners.php;81d1... [22:47:08] (Abandoned) Eileen: Temp hack fix [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/407196 (owner: Eileen) [22:47:32] (PS1) Eileen: CRM-21737 loosen option value validation to support language option group [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/408444 [22:48:17] (Abandoned) Eileen: CRM-21737 loosen option value validation to support language option group [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/408432 (owner: Eileen) [22:52:57] (CR) jerkins-bot: [V: -1] CRM-21737 loosen option value validation to support language option group [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/408444 (owner: Eileen) [23:31:53] (PS3) Ejegg: Update return from timeout [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407738 (https://phabricator.wikimedia.org/T184284) (owner: Mepps) [23:32:00] (CR) Ejegg: [C: 2] Update return from timeout [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407738 (https://phabricator.wikimedia.org/T184284) (owner: Mepps) [23:35:53] (PS9) Ejegg: Allow Benevity currencies [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/399252 (https://phabricator.wikimedia.org/T182808) (owner: Eileen) [23:36:03] (CR) Ejegg: [C: 2] "Lookin good!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/399252 (https://phabricator.wikimedia.org/T182808) (owner: Eileen) [23:39:38] (Merged) jenkins-bot: Update return from timeout [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407738 (https://phabricator.wikimedia.org/T184284) (owner: Mepps) [23:41:30] (Merged) jenkins-bot: Allow Benevity currencies [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/399252 (https://phabricator.wikimedia.org/T182808) (owner: Eileen)