[00:14:44] (PS4) Ejegg: WIP MariaDB strict mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) [00:16:39] (CR) jenkins-bot: [V: -1] WIP MariaDB strict mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) (owner: Ejegg) [00:17:58] (PS5) Ejegg: WIP MariaDB strict mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) [00:19:29] (CR) jenkins-bot: [V: -1] WIP MariaDB strict mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) (owner: Ejegg) [00:22:40] (CR) Ejegg: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) (owner: Ejegg) [00:25:37] Ah yeah that qunit failure is a CI issue [00:25:42] I think.. [00:33:42] (PS6) Ejegg: MariaDB strict mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/310457 (https://phabricator.wikimedia.org/T145591) [00:36:35] AndyRussG: that /might/ have been all we needed ^^^ [00:37:18] ejegg: woohoo nice stuff! [00:37:43] At the very least we can get our phpunit on CI back :) [00:37:47] thx much!! [00:39:52] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: CentralNotice: Support MariaDB Strict Mode - https://phabricator.wikimedia.org/T145591#2635164 (Ejegg) a:Ejegg [00:40:05] yw! [00:51:32] (CR) Ejegg: [C: -1] "PaymentsInitQueueTest needs to set up the pending db config. Might want to rebase this on top of I8a482363442c2" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310439 (https://phabricator.wikimedia.org/T143945) (owner: Awight) [00:51:53] Well, guess I'll take off. Have a good evening all! [01:00:04] ejegg|away: cya! [04:16:49] Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2635674 [05:36:48] awight: pageviews vs impressions for ko wlm checks out, all clean [05:36:57] (in case u were worrying :) ) [05:36:59] https://phabricator.wikimedia.org/P4040 [05:37:03] (just commenting now...) [05:38:59] AndyRussG: hi! [05:39:30] Hey! [05:39:33] excellent news [05:39:38] indeed :) [05:39:39] So it was the sample rate? [05:40:18] We gotta get off of the self-titled db one day soon, anyway [05:40:28] Ah I guess... I just went straight to Hive and the ratio Special:BannerLoader:/beacon/impression is consistent [05:40:36] Yeah for sures! [05:40:55] Ah I see, so I manufactured a huge incident just by using a bad data source ;) [05:41:25] not at all, we totally had to check this out due to the bugstorm of related [05:41:31] Looks like ecom is the last barrier to jumping ship and living in a brave new hivey world [05:41:48] We could clone that fossil in a second [05:42:02] ecom... don't even know that one [05:42:18] the-wub has some fancy scripts for the DB but maybe he'd like to learn ipython notebook :) [05:42:21] take a look in /srv/br on the staging server [05:42:43] It's not even in source control... and is the main analytical tool used by -creative, to this day [05:43:13] Also the main consumer of pgehres misinformation ;) [05:43:26] (ah, the dangers of naming a database after oneself) [05:43:47] do u mean the statler tool? [05:43:58] umm, remind me what that tool is? [05:44:05] * awight references your docs [05:46:22] I just heard about it from spatton! [05:46:44] That might be what they call "it" [05:47:00] Anyway, that's the dir it's in, I'll add to https://collab.wikimedia.org/w/index.php?title=Draft:Fundraising_Interaction_and_Activity_Diagrams&diff=50165&oldid=50164 [05:47:11] err. you know what I mean :) [05:47:51] Ah yep cool thx! [05:53:18] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2635717 (AndyRussG... [05:55:52] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2635718 (AndyRussG... [06:02:32] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2635720 (AndyRussG... [06:18:51] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2635725 (awight)... [06:23:54] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate the discrepancy between pageview counts and banner impressions for WLM 2016 - https://phabricator.wikimedia.org/T145565#2635729 (awight)... [06:33:43] (PS1) Awight: Fix typo in error handler [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310491 [06:34:40] (CR) jenkins-bot: [V: -1] Fix typo in error handler [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310491 (owner: Awight) [06:34:48] (PS2) Awight: Fix typo in error handler [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310491 [07:41:04] (PS1) Awight: [WIP] Encapsulate country validation in a helper class [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310495 (https://phabricator.wikimedia.org/T145608) [07:43:03] (CR) jenkins-bot: [V: -1] [WIP] Encapsulate country validation in a helper class [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310495 (https://phabricator.wikimedia.org/T145608) (owner: Awight) [10:07:04] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: CentralNotice: Support MariaDB Strict Mode - https://phabricator.wikimedia.org/T145591#2635164 (jcrespo) Please do not view this as a criticism, but as an encouragement (thanks for w... [12:18:09] fundraising-tech-ops: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2636545 (Jgreen) [12:18:23] fundraising-tech-ops: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2636559 (Jgreen) p:Triage>Unbreak! [12:26:25] fundraising-tech-ops: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2636591 (Jgreen) a:Jgreen>None [13:36:43] fundraising-tech-ops: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2636704 (Jgreen) current stats from activemq: %mem queue name 18 149011 payments-init 10 81365 payments-antifraud 6 48961 pending 4 45089 banner-history 3 23415 pending_paypal_re... [16:03:00] awight: morning! [16:03:16] how come we need a job producer beyond multiqueuewriter? [16:08:15] cwd|brb: Just to have a nice interface, right? [16:10:00] fundraising-tech-ops: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2637268 (awight) Ah, okay thanks. We need to start shutting down queue producers once we've moved the consumer to Redis. Thanks for getting the stats! [16:10:28] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2637269 (awight) [16:13:27] cwd|brb: Just checking, the way you wrote DonationQueue::mirror will allow us to configure queues to the new queue and remove the mirroring, one queue at a time right? It looks that way. [16:15:33] hrm did i write that? [16:15:45] hahaha [16:16:17] cwd: maybe take a peek at branch #un_activemq in the deployment config repo [16:21:17] #un_activemq2 [16:22:43] just changing back to the old names? [16:23:28] I *think* that'll stop mirroring but cause messages to only be pushed to Redis now. [16:23:47] * cwd looks again [16:24:00] (updated the branch) [16:24:09] mebbe I should try this in vagrant... [16:25:58] oops, another tweak pushed to the branch [16:29:18] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, fundraising-tech-ops, FR-ActiveMQ, Unplanned-Sprint-Work: fundraising activemq queue/memory bloat - https://phabricator.wikimedia.org/T145634#2637327 (awight) [16:30:17] awight: yeah i don't think i had anything to do with the mirroring set up but... this does look like it would do what you say [16:30:34] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Login for Engage User Chris - https://phabricator.wikimedia.org/T145577#2637332 (Jgreen) [16:31:23] * awight eyes https://gerrit.wikimedia.org/r/#/c/299797/7/gateway_common/DonationQueue.php warily [16:33:20] sigh [16:33:31] i'm brain dead [16:33:53] baaha I do exactly that move at least three times per day [16:34:11] I should have dropped it at "did I write that" cos that's all I need to know :p [16:35:45] my understanding of the project has shifted greatly since i did this [16:38:53] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Stop producing to ActiveMQ - https://phabricator.wikimedia.org/T145660#2637359 (awight) [16:39:27] awight: ok what comes to mind for me is changing the mirroring config and/or undoing that patch [16:39:30] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Stop producing to ActiveMQ - https://phabricator.wikimedia.org/T145660#2637375 (awight) Turning off producers to payments-init, payments-antifraud, pending, and banner-history with co... [16:40:42] cwd: Cool. We're not entirely out of the woods, so need to leave the mirroring function in there for a bit longer just on account of globalcollect-cc-limbo (orphan rectifier related) [16:41:06] ah yes of course [16:41:23] lmk if it's useful to fon about the config [16:43:23] yeah i will hop on the tech call in a couple minutes, just gonna make more coffee, clearly need it [16:53:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Login for Engage User Chris - https://phabricator.wikimedia.org/T145577#2637474 (Jgreen) Open>Resolved cert sent, so closing [17:00:54] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Stop producing to ActiveMQ - https://phabricator.wikimedia.org/T145660#2637529 (awight) Once the config is deployed, I'll purge the queues so it's obvious if we have straggling producers [17:55:10] (CR) Ejegg: [C: 2] "nice catch" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310491 (owner: Awight) [17:56:51] (Merged) jenkins-bot: Fix typo in error handler [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310491 (owner: Awight) [18:31:24] (PS21) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) [18:32:26] (CR) jenkins-bot: [V: -1] WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) (owner: Cdentinger) [18:54:52] dunno what that -1 is about [19:00:25] doesn't blow up locally, too weird [19:04:55] (PS22) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) [19:06:24] (CR) jenkins-bot: [V: -1] WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) (owner: Cdentinger) [19:08:38] Call to undefined method SmashPig\PaymentProviders\PayPal\Tests\CaptureIncomingMessageTest::getDefaultSearchPath() in /mnt/jenkins-workspace/workspace/composer-php53/Core/Configuration.php on line 85 [19:08:48] i do not get why it is calling against that class [19:08:55] !log update paymentswiki config to 9919bad6897f2eb7199dd076f6fc04a60c713cf8 [19:08:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:09:40] $this->config = PayPalTestConfiguration::get(); [19:10:16] but only on CI [19:10:37] looking... [19:11:11] (PS23) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) [19:11:24] weird that it doesn't fail in -hhvm [19:11:48] cwd: damn. Okay it's a late static binding issue. [19:11:50] if this is some inheritance gremlin i'm goingn to puke [19:11:59] but why only on CI?? [19:12:13] only in zend, maybe? [19:12:21] must be [19:12:22] you're running hhvm locally, right [19:12:23] yeah cos hhvm and php53 and php55 all have different broken ideas [19:12:24] ? [19:12:35] omg diff the last 2 patch sets [19:13:00] hehe [19:13:17] wait [19:13:30] I was looking at PS21-22 [19:13:35] self:: vs Configuration:: [19:13:38] yeah [19:13:45] * awight mopes [19:13:48] self was resolving to the Test somehow [19:14:00] cos late static binding doesn't actually work [19:14:16] this is the problem with language features [19:14:22] in general [19:14:25] hehe [19:14:35] give me a blunt rock [19:14:37] * cwd opens emacs [19:14:47] wait, it's working for AdyenTestConfiguration... [19:14:57] yeah [19:14:59] what's the difference? [19:15:10] cwd: that needs to be a static function, for one [19:15:17] oho [19:15:37] oh damn [19:15:45] thanks [19:15:57] but i still think it's indefensible language behavior [19:16:53] (PS24) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) [19:17:32] agreed. Thing::nonStaticFunction should throw a handful of poop back over the fence [19:18:30] i mean in this instance i can't see why it should really matter if it's called statically [19:18:44] self should mean the same thing in either context shouldn't it? [19:18:50] yeah but what kind of a language lets you can a non-static function without an instance [19:19:12] solipsism everywhere [19:19:42] christ well it works when self:: when static func sig [19:20:11] hehe nice [19:22:00] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Stop producing to ActiveMQ - https://phabricator.wikimedia.org/T145660#2637950 (awight) Deployed and merged those changes. I'll kick this task out of the sprint, assuming the donatio... [19:22:08] Fundraising-Backlog, FR-ActiveMQ, MediaWiki-extensions-DonationInterface: Stop producing to ActiveMQ - https://phabricator.wikimedia.org/T145660#2637951 (awight) [19:56:06] (CR) Ejegg: "This makes some sense, but I don't know that we need to worry too much about interoperability. The only things we're actually rehydrating " (3 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310163 (owner: Awight) [20:03:36] (PS7) Ejegg: Defaults for more queues [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/309181 [20:11:11] ack! sorry I totally missed standup [20:11:27] (PS25) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) [20:11:33] We're into BS time, but we're all still on if you want to jump in. [20:37:01] (PS26) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) [20:37:17] (PS1) Ejegg: Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) [20:37:59] (CR) jenkins-bot: [V: -1] Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg) [20:39:39] awight, ejegg - last one pushes the Job (what should I name that class? JobJob?) [20:47:14] (PS2) Ejegg: Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) [20:48:02] (CR) jenkins-bot: [V: -1] Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg) [20:48:10] Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2638349 [20:49:16] eileen: Err sorry, I just responded to your first bullet point before finishing reading your essay. Please ignore me :) [20:49:31] (PS3) Ejegg: Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) [20:50:22] (CR) jenkins-bot: [V: -1] Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg) [20:55:22] essay… [20:55:33] (PS4) Ejegg: Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) [20:56:32] hehe the Phabricator comment, not yr blog post [20:58:32] awight: I know it's a worry when my comments are called an essay.... [21:01:04] hehe it's great reading though [21:01:13] locale adventure novel [21:02:36] Fundraising Sprint Octopus Untangling, Fundraising Sprint Pretending This Isn't Happening, Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, and 3 others: Silverpop export not capturing language changes in Civi (not sure if... - https://phabricator.wikimedia.org/T96410#2638429 [21:04:36] eileen: Having second thoughts about the "mutually intelligible language" I was mentioning in https://phabricator.wikimedia.org/T96410#2638349 , because of your final point (3). [21:05:28] about people moving? [21:05:29] If we used BCP-47 language tags, we could merge languages and even sort by specificity or something... preferred_languages="en_NL, en" [21:05:32] yeah [21:05:56] If you look in the civicrm_option_value table the 2 digit codes ARE stored [21:06:00] so, we can use that [21:06:08] even where we have more detail [21:06:32] huh. We could even introduce BCP-47 as a new field, and leave preferred_language as the degraded 2-character form [21:06:47] but, how do we know the BCP-47 [21:07:15] I mean, aren't we largely guessing it in the first place [21:07:38] we can make one up as we go along. See that lib I linked to... like pt_BR -> pt-br, and zh_TW -> zh-hant or whathaveu [21:08:20] yeah but still - we are looking for really granular ways for storing data we don't actually know is correct [21:08:36] We keep that field, for sure [21:08:40] if you look in the linked google doc you see all the variants of 'English' [21:09:01] And what are storing en_PK for? [21:09:22] I know... I'm totally behind your first point (4), let's reduce those to just the real-world variants [21:09:33] I mean we only know the language the form was in & the country they were in don't we [21:09:35] I do think number locale needs to be addressed [21:09:44] yeah, nothing beyond that [21:09:50] unless they were using MediaWiki as pt_BR [21:09:54] uselang=pt_BR [21:10:15] huh. That one https://en.wikipedia.org/wiki/Main_Page?uselang=pt-BR [21:10:19] OK - so where we actually 'know' the language variant that seems useful [21:10:22] We can glean that from logs [21:10:35] yeah totally, let's even try to backfill that data into contribution_tracking [21:10:45] but what worries me is complexity around storing & managing data that was actually quite poor quality [21:10:53] +1 [21:11:40] (PS1) Ejegg: Clean up antifraud queue usage [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310670 (https://phabricator.wikimedia.org/T131273) [21:11:48] So, will you have a special case to preserve pt_BR + pt ? [21:11:59] * awight re-reads (4) [21:12:09] If we went for #4 we would basically delete a tonne of possible variants from the option_value table. And change others to be just 'co' [21:13:56] So we would do something like UPDATE civicrm_option_value SET name = value WHERE name NOT IN ('pt_BR', 'en_GB', 'fr_CA') [21:15:24] eileen: Take a peek at wmf_communication/twigext_l10n/CldrData.php [21:15:58] I think I was wrong about number locale per country, at least in the library we're using. [21:16:47] I'm sort of confused about the relationship between these variants and the unix locale style we'll have [21:17:29] i.e., are those also the written language variants, and they happen to be equal to the global number formatting variations? [21:18:20] What is es_419 ? [21:19:39] well we seem to have started from the CiviCRM list but added a tonne of our 'own' [21:20:06] ones like this 'Do_US' seem to come from an outright error [21:20:33] Do is 'short for Donation [21:20:39] hehe, that's a nice one [21:20:47] (I found something to support that but don't ask me where) [21:21:39] I meant in here, https://github.com/wikimedia/wikimedia-fundraising-crm/blob/master/sites/all/modules/wmf_communication/twigext_l10n/CldrData.php [21:22:10] how does that data get used? [21:22:29] That's reference data for https://github.com/wikimedia/wikimedia-fundraising-crm/blob/master/sites/all/modules/wmf_communication/twigext_l10n/TwigLocalization.php [21:22:41] which builds templated thank-you letters for each donor [21:23:20] https://github.com/wikimedia/wikimedia-fundraising-crm/blob/master/sites/all/modules/thank_you/templates/html/thank_you.ja.html [21:23:36] ah - so that list is the 'meaningful list' [21:24:00] exactly [21:24:09] We're abusing the mapping silly [21:24:20] :-) [21:24:30] currently feeding those la_CO unix locales in, I'm sure [21:25:20] I'd be happy to spruce up the LanguageTag repo if it would be helpful to map between all these funky variants [21:25:34] hadn't seen this list either myself, until now [21:25:46] At least not in recent memory [21:25:58] So, we are handling Belgian English as a legitimate variant with it's own currency formatting… [21:26:25] wow, so it is [21:26:40] not to be confused with how we format currency for Belize English [21:26:46] haha [21:26:50] * awight nudges again [21:26:57] or English from district 150 [21:27:00] With BCP-47, we could have all the crappy variants we're fed [21:27:05] what *is* that [21:28:01] right - but my worry is we are going to some effort to store data that may not actually be meaningful [21:28:43] yah I'm still on board with that sentiment [21:28:51] Somewhere in-between would be nice [21:29:45] e.g. your 1) we strip down all languages to the bare (exhaustively validated with cldr locales) valid locale code [21:30:41] but also 2) change the dedupe matcher logic to only conflict when a match does not have compatible base 2-char language [21:31:57] then a cool, nice-to-have, 3) dedupe merge logic saves original (only the valid ones) language variants as a preferred languages list [21:32:52] whew [21:33:05] Another question, how should we deal with variants which aren't represented by Civi, e.g. az_Cyrl [21:33:18] awight: what do you think about moving the data from cldr to a DB table? [21:33:34] it feels like being able to compare would be good [21:33:45] eileen: then we have to deal with merging new cldr data in, but no biggie [21:34:24] it hasn't changed in git since 2013 [21:34:58] Re az_Cyrl - that is a variant of Azerbaijani? [21:35:08] Fetching MediaWiki uselang locales from contribution_tracking feels a bit cleanupy--do you think it should happen now, or in the unplanned future? [21:35:19] eileen: Just cos we haven't bothered [21:35:55] There has been lots of upstream work, http://cldr.unicode.org/ [21:36:10] cldr btw supports bcp-47 :D [21:36:22] I think MediaWiki does now, at some level [21:38:52] hmm - but when people fill in Donate forms what do we know - language + country or if Brazilian we know a bit more [21:39:48] We do know if they read Serbian in Cyrillic, for example [21:39:55] what does "Fetching MediaWiki uselang locales" mean [21:40:30] A backfill task of trying to improve language quality by scraping payments logs or some other data source [21:41:00] Ah… I suspect we should focus on the more recent ones? [21:41:15] would that take ages to run through…. server time I mean [21:41:36] Re CLDR - so that is a library which has updates which we have not grabbed? [21:42:36] yep. And probably a code generation step too [21:43:00] There's a MediaWiki cldr extension... maybe that's where this data came from? [21:43:24] https://github.com/wikimedia/wikimedia-fundraising-tools/blob/master/CldrParser/CldrParser.py [21:43:28] ^ actually [21:44:22] (CR) Ejegg: "Looks like totally safe normalization, but it would be overwritten by I099efa8aaddc7, right?" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [21:45:11] eileen: You know what... I've made another mountainous molehill. There's only one language with a variant that we respect in thank-you letters: https://github.com/wikimedia/wikimedia-fundraising-crm/tree/master/sites/all/modules/thank_you/templates/html [21:45:41] eh, no we still format numbers according to locale... [21:45:52] & currency? [21:46:00] (CR) Ejegg: "So we're fine clobbering any language changes in Civi to date? Also, I'm testing locally to figure out why that 3-way join gives such low " (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [21:46:32] currency should come from the donation itself, so that's covered. [21:47:06] our forms now do cool stuff with currency localization [21:47:34] e.g. in the US, 1 USD is $1, but in Canada it's 1 USD [21:47:46] nice! [21:47:54] but none of our civi stuff uses that yet [21:47:57] on this occassion I agree with the US :-) [21:48:03] There's an argument for separating country from locale variant, I guess [21:48:18] right - I don't want to throw away meaningful data [21:48:38] my worry is just that a lot of it isn't - ie. en_NL [21:49:07] which is a case for only storing a language variant if it 'exists' [21:49:16] (& gettin rid of a bunch) [21:49:30] the issue comes when we have en_NZ come in [21:49:40] we can then say it's not valid (ahem) [21:49:46] & match to 'en' [21:50:11] ehhehehe [21:50:13] but for, say Corsican we don't have a 'co' [21:50:19] we only have 'co_FR' [21:50:28] and that is the ONLY Corsican variant [21:50:47] * awight squints [21:50:48] which makes me think we should set 'random corsican of any flavour' to 'co [21:51:16] (actually I was looking locally to get a single Corsican [21:51:23] probably we have many…. [21:51:31] I don't see 'co' in our cldr list [21:53:12] I'm 100% for stripping preferred_language down to exhaustive, existing locales. The question I'm still pondering is where to store the communications-preferred country value for a contact, and whether we marry these back into a language tag in order to call out to cldr et al [21:53:56] I don't think we get any complaints about our amount formatting... i.e. it's pretty good [21:54:13] awight: ie you are rejecting Caitlins suggestion we use the address country [21:54:43] (which of course can change - but we don't know how to interpret that change) [21:54:46] I think that's a great idea, if we're repairing crappy old data that's more correct than anything, I'm sure. [21:54:59] (CR) Ejegg: Update preferred language based on contribution_tracking. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/307689 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [21:55:01] s/data that's/data, that seems/ [21:55:54] hmm - why don't I now write some sql statements so we can see if we really ARE converging or just discussing different plans :-) [21:56:53] ejegg: about adding that index - I'm inclined to think we don't want it long term - but I suspect that adding it short-term might make sense [21:57:13] as tidy up query speed is going to change a lot [21:57:38] (CR) Ejegg: "Oops, I'm looking at the old version... abandon?" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/307689 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [21:57:41] awight: i feel like the tests here cover the same cases as the ones in the "make tests real" patch: https://gerrit.wikimedia.org/r/#/c/305761/26/PaymentProviders/PayPal/Tests/phpunit/CaptureIncomingMessageTest.php # it is less unit-y to have them in the same sequence, but also less code repeated. if you find that offensive i will split them up! or anything else i missed/is annoying [21:58:44] (CR) Eileen: "Ah yeah - I created a new one to fix the merge conflict - not sure why it wound up being new but..." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/307689 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [21:58:52] (Abandoned) Eileen: Update preferred language based on contribution_tracking. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/307689 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [21:59:15] cwd: yah I don't see any reason not to split it up, not even much test code to support it [21:59:37] It could be in the same class though, whatev about that [22:00:32] (CR) Eileen: "Re clobbering changes 2 thoughts" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [22:02:16] ah good call, then i can still use the ephemeral queue [22:02:34] anything jump out at you that i missed from your tests? [22:02:50] cwd: uh yeah but purge the queue obviously [22:03:23] cwd: there's the mock paypal api... unless you're just trying to get rid of the one follow patch? [22:03:57] awight: yep just that one, want to merge in the stuff from the api one [22:06:08] (PS1) Awight: Include contribution_tracking_id in test message; set minimal $wgDonationInterface globals [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310677 [22:14:29] awight: sorry dumb question, will the sqlite pdo stores be recreated in between tests? [22:16:43] cwd as long as the config is reset, yes [22:17:08] they only persist during a test because we get them with ->object( path, true ) [22:17:31] where the 'true' means to return the same instance, based on a reference stored in the config class [22:17:43] (CR) Awight: "This is great stuff! +1 on dropping the next test patch, it's covered enough by the tests here." (7 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) (owner: Cdentinger) [22:18:05] cwd: My fault, yeah the sqlite:memory dbs totally are reset for every test [22:18:20] ok right on, i just split those two pieces up, thanks! [22:18:40] dang, that thing looks deployable [22:19:13] If I get out from under this DI-CRM clustereff, I'll try to jump on more nuanced testing [22:19:38] i think it's getting close! [22:21:32] ejegg: Hey that was unexpected... CI is passing on https://gerrit.wikimedia.org/r/#/c/310677/ [22:21:38] (PS27) Cdentinger: WIP smashpig paypal listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/305761 (https://phabricator.wikimedia.org/T141654) [22:22:01] ejegg: I still get one test glitch on my local though, about the DO_PAYMENT.testresponse [22:22:04] weird. [22:22:10] that was just splitting the test up and a couple fixes from ejegg's comments, most notably verified queue instead of completed [22:22:15] and sort of unacceptable ;) [22:22:33] What steps do we take to test drive it? [22:23:08] Using a hacked backend, you can pass PP a custom IPN url which points to somewhere you can capture... [22:23:14] awight: if you want to abandon that test patch i'll rebase the api stuff and move it into a testable module so we can uncomment the business code :) [22:23:21] I've done that once or twice, not too much legwork involved [22:23:29] totes [22:24:02] (Abandoned) Awight: Make tests real [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306599 (https://phabricator.wikimedia.org/T141654) (owner: Awight) [22:24:08] ty! [22:24:16] awight: nice, just adding the ct_id did it? [22:24:45] mind if I squash your fix in to the test enablement? [22:25:06] (PS3) Cdentinger: [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight) [22:25:16] (CR) jenkins-bot: [V: -1] [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight) [22:26:32] ejegg: I guess so! [22:26:40] Run it locally though... [22:27:03] I get .File <>crm/vendor/wikimedia/donation-interface/tests/phpunit/includes/test_gateway/../Responses/globalcollect/DO_PAYMENT.testresponse does not exist. [22:27:06] ... [22:27:30] I can debug, but creeped out by CI not hitting this path [22:28:30] XenoRyet: Does that Configuration situation look at all promising? [22:28:49] I can't quite tell if we need to do that or not. [22:29:19] Still in the poking around phase really, but yea, there might be something here. [22:29:19] awight: fixing the leniency now [22:29:26] Maybe there's a workaround like a DonationInterface::setDefaultGlobals [22:29:50] we're echoing something in the testing adapter when the file's not found, expecting the echo to make tests fail [22:29:57] I'm changing that to an exception [22:30:45] Yea, maybe. [22:31:53] Fundraising-Backlog, FR-Adyen, MediaWiki-extensions-DonationInterface: Orphan rectifier for Adyen - https://phabricator.wikimedia.org/T145708#2638776 (awight) [22:32:08] ejegg: oh huh [22:32:19] (PS1) Ejegg: Make sure tests fail on missing response file [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/310684 [22:32:23] Right, I'm getting "OK" [22:32:24] thx [22:32:47] (PS4) Cdentinger: [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight) [22:33:14] hmmmk got that on top of the current listener junk [22:34:04] eileen: Should we be thinking about optimizations to the dedupe processor? Or do you expect it to catch up by December? [22:34:23] (PS4) Ejegg: Re-enable recurring GC tests, add more [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306806 (https://phabricator.wikimedia.org/T141487) [22:34:33] awight: well it should catch up by then - will it keep yp? [22:34:49] one option is to set a second job going [22:35:39] eileen: Do you happen to have a ballpark estimate on the catch-up date? [22:36:31] I believe ccogdill will want to use the deduped data to build mass mailings for the first week of Big English, if I have the timeline right [22:37:31] awight: do you know the circumstances in which we create synthetic contribution_tracking rows for things that don't come in via the payments frontend? [22:37:45] do we ever add dummy language data to that table? [22:37:54] Ooh good question [22:38:03] Perhaps anything from offline2civicrm [22:38:11] you could join on that table... [22:38:14] hmm - well let me check what number it's up to - actually maybe I should prioritise exposing that data in a dashlet rather than checking [22:38:31] Looking at that now, not finding where we insert to ct [22:38:42] but awight maybe I should start a second job - I don't think they would clash [22:38:52] which would basically double the speed [22:39:03] eileen: if you wish--I guess that will become relevant again when Big English happens and we drop behind again? [22:39:22] awight big english starts much earlier for email. the first big week of US email is the 2nd week in Nov [22:39:29] eileen: Could mean 2.5x the database timeouts, though... [22:39:32] we’ll be emailing the other countries sooner than that [22:39:49] just as an FYI. but if I can get a time estimate from you all on dedupe, I may move things around [22:40:09] ccogdill: and how valuable is the deduped db for that mailing spree, vs. the silverpop dirty-deduped export? [22:40:34] well it’s hard to say without having the good deduped db to compare :) [22:40:38] hahaha [22:40:41] darn [22:41:03] we know that using the donor’s past donation data improved all our emails by ~25% [22:41:17] oof [22:41:19] so my hope is segmenting better across those lines with better data will give us another 10% bump [22:43:37] so if that was 10% bump to the entire US list, that would be $1,000,000 :) [22:43:56] * awight relaxes in a WMF-provided hot tub [22:44:02] a big one. [22:44:20] j/k, for the record no such thing has ever occured [22:44:22] So, over a 24 hour period it went from contacts matching id 2,268,500 to 2,399,905 [22:44:43] eileen: We probably want the longest average, to account for downtime [22:44:45] that should speed up over time in the sense that there are many more deleted ones to skip in the higher numbers [22:44:50] ah hrm [22:45:19] That could fold into the estimate: (remaining - deleted) [22:45:52] so it might get to 150,000 per day as an average [22:46:13] 10 days 1.5 million [22:46:18] 20 days 3 milion [22:46:24] 40 days 6 million [22:46:45] Cool and we have about 60 days [22:46:52] * awight puzzles [22:47:00] We'd better fire up the second boiler, eh? [22:47:27] yeah I think it's good to try that out [22:47:33] And maybe tune "duty cycle" of how many we process in each batch [22:48:54] If you're interested, we're using a demonstrably useful batch concept in the CRM jobs, where we limit the jobs to a certain length of time, i.e. 80% of the time allowed for each run [22:49:17] We can also limit by number of records per batch, but that's only used to slow-start jobs with small pilot runs [22:50:50] It gives us nicely tuned batches regardless of system load and other stuff. [22:50:54] awight: I've tinkered a bit - I think larger batches are more efficient but feel more comfortable with them smaller when they crash [22:51:19] oh? Ah right you're actually loading the entire batch all at once. OK scratch that then [22:53:34] yeah cos it has to do sql to find matches [22:54:00] & then it processes however many matches it finds for the batch of contacts [22:55:55] That's a much more legitimate way to do batching than the batch of isolated transactions we do for the other jobs [22:58:04] well the fact it does fairly intense sql to generate the list & it's a core process not one of ours somewhat dictates that [22:58:51] So, I set the high one going at 11 m [22:58:56] (PS5) Cdentinger: [WIP] Payments API for PayPal [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/306601 (owner: Awight) [22:58:56] If you look you see [22:58:57] Deduping contact range: 11000000 to 11001004 [22:59:16] which means the range of 1000 has been extended by 4 due to deletions in the middle [22:59:23] (which is not many!) [22:59:39] Oh hey, you already have a second job running! [23:00:17] The low numbers job has a great duty cycle anyway, ~80% [23:00:47] hah okay you just created the second job [23:01:05] * awight marvels [23:01:43] maybe I should go higher actually [23:01:55] maybe up to 15 million to be really recent on that one [23:01:58] You're doing that with just a rule_group_id? Like picking contact_id > N? [23:02:09] that's a bit of a hack [23:02:21] the id is tracked through a drupal var [23:02:30] the var name includes the rule_group_id [23:02:35] 13 is the default ... [23:02:45] so adding it just generates a second var [23:02:48] I guess the low job will duplicate that work when it catches up to the midpoint [23:02:53] oh wow that *is* a hack :) [23:02:58] :-) [23:03:05] strange class not found error when i try to use the live api error. gotta run for a while but will investigate later! [23:03:13] eileen: But how did you point it at high ids? [23:03:20] cwd|afk: sounds fun! [23:03:29] I changed the variable with drush vset [23:03:36] ooooh [23:03:53] not cool. My iron maiden playlist was hijacked by metallica [23:04:06] :-) [23:04:16] * awight scratches the needle [23:06:12] So far I can't see us inserting fake language values into contribution tracking - looks like the rows from non-payments sources should all be null [23:07:49] * awight laments any code that calls "new" [23:14:24] (PS3) Eileen: Add Index to contact.preferred_language [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310192 (https://phabricator.wikimedia.org/T96410) [23:14:26] (PS2) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) [23:14:28] (PS2) Eileen: Update preferred language based on contribution_tracking. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410) [23:15:06] (CR) Eileen: "Just fixed that errant comma in the create table - I guess I deleted a row from that statement & left the comma there" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [23:15:56] (Abandoned) Eileen: [WIP] attempt test for mirror queue change. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306069 (https://phabricator.wikimedia.org/T141484) (owner: Eileen) [23:16:55] (PS1) Awight: Hair-raising hack to pass dummy gateway response code to DonationInterface [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310694 [23:16:58] ejegg|afk: :D ^ [23:16:59] (CR) Eileen: "Yeah - I think I should re-order it to be after than..." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [23:20:15] cwd|afk: Wheeling my gurney over to your cage, now... [23:27:12] awight: aha, naughty [23:27:42] I was considering making that thing static in the DI testing adapters [23:29:46] (PS2) Ejegg: Hair-raising hack to pass dummy gateway response code to DonationInterface [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310694 (owner: Awight) [23:30:54] (CR) Awight: [C: 2] "Looks like it'll work!" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306450 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [23:32:24] (CR) Awight: "PS7 is just a rebase? Strange cos it seems to go backwards..." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/309181 (owner: Ejegg) [23:32:52] awight: just looking at this http://unicode.org/cldr/trac/browser/trunk/common/main/fr.xml I can't see any sign of our numeric or currency translations coming from there [23:33:40] (CR) Ejegg: "oops... how did that happen?" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/309181 (owner: Ejegg) [23:36:02] (Merged) jenkins-bot: Reject isRequeue messages after expiry [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/306450 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [23:37:13] eileen: hehe. Wat [23:37:41] eileen: MAybe here? supData = ET.parse(cldr_path + 'supplemental/supplementalData.xml').getroot() [23:39:30] huh. http://unicode.org/cldr/trac/browser/trunk/common/supplemental/supplementalData.xml [23:39:35] such a cool project! [23:39:56] (PS8) Ejegg: Defaults for more queues [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/309181 [23:40:20] The population percentages are quite an achievement [23:41:12] (PS5) Ejegg: Remove all traces of stomp pending queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) [23:41:37] hmm - so it's definitely more to think about! [23:42:07] (PS2) Ejegg: Clean up antifraud queue usage [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310670 (https://phabricator.wikimedia.org/T131273) [23:42:34] eileen: ack, the tools/CldrParser.py gets its data from all over the place... [23:43:08] sigh [23:43:15] & our repo is not regularly updated [23:47:00] (CR) Ejegg: "ok awight, should be all set now" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/309181 (owner: Ejegg) [23:47:51] eileen: This might be the short answer to which locales we actually want to keep: http://unicode.org/cldr/trac/browser/trunk/common/main [23:48:55] (CR) Awight: [C: 2] Defaults for more queues [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/309181 (owner: Ejegg) [23:54:04] (Merged) jenkins-bot: Defaults for more queues [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/309181 (owner: Ejegg) [23:54:45] (CR) Awight: [C: -1] "Nicely combed! Tiny fixups inline" (6 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg) [23:56:52] oh snap, unsubscribe queue needs migration too! Making a ticket... [23:57:11] ah, already there [23:57:16] pulling into sprint... [23:57:48] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ: Migrate FundraisingEmailUnsubscribe to new queue backend - https://phabricator.wikimedia.org/T145419#2638970 (Ejegg) [23:57:50] (CR) Awight: [C: 2] Clean up antifraud queue usage [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310670 (https://phabricator.wikimedia.org/T131273) (owner: Ejegg) [23:58:47] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ: Migrate FundraisingEmailUnsubscribe to new queue backend - https://phabricator.wikimedia.org/T145419#2629496 (Ejegg) @XenoRyet : If you want to switch the consumer over, I'll work on the producer [23:59:09] (CR) Awight: Remove all traces of stomp pending queue (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310665 (https://phabricator.wikimedia.org/T131274) (owner: Ejegg) [23:59:34] (CR) Awight: Clean up antifraud queue usage (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/310670 (https://phabricator.wikimedia.org/T131273) (owner: Ejegg)