[00:01:26] (PS5) Eileen: Rely on CiviCRM address rather than contribution tracking language. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/311491 (https://phabricator.wikimedia.org/T96410) [00:04:28] (PS6) Eileen: [deploy with crm code] Rely on CiviCRM address rather than contribution tracking language. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/311491 (https://phabricator.wikimedia.org/T96410) [00:05:43] (CR) Ejegg: [C: 2] "Nice simple capstone to this huge pyramid of language cleanup" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/311491 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:12:00] (Merged) jenkins-bot: [deploy with crm code] Rely on CiviCRM address rather than contribution tracking language. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/311491 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:16:43] (CR) Ejegg: [C: 2] "Seems to solve the problem. TODO: figure out where we're creating these duplicate addresses in the first place and cut them off at the so" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311908 (https://phabricator.wikimedia.org/T145873) (owner: Eileen) [00:17:11] (CR) jenkins-bot: [V: -1] Fix to prevent duplicate Home addresses resulting in no primary on merge. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311908 (https://phabricator.wikimedia.org/T145873) (owner: Eileen) [00:17:23] (PS3) Ejegg: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:17:34] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Patch-For-Review: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2657991 (awight) @AndyRussG I believe we have server-side logging that will show us every o... [00:17:41] (CR) jenkins-bot: [V: -1] Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [00:19:42] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311908 (https://phabricator.wikimedia.org/T145873) (owner: Eileen) [00:19:47] ejegg|meet: Mind if I deploy the mirroring phase of FUndraisingEmailUnsubscribe? [00:20:03] awight|brb: sure! [00:20:07] cool [00:20:53] just up through 2aa1f7b48e86226043679f66040694cf7b34fe01, right? [00:21:23] hum, I was headed for 4c35ff3a6cc5b0a968a1f7d846d0256924b87d4d [00:21:25] lemme see... [00:21:52] yeah, what you said [00:22:00] right on [00:22:01] * awight blinks at gerrit [00:22:24] Very strange. https://gerrit.wikimedia.org/r/#/c/311070/3 [00:22:36] the SHA is different there [00:22:53] ugh [00:22:55] ps 4 [00:22:56] nvm [00:29:04] (PS1) Awight: Update FundraisingEmailUnsubscribe to mirror to Redis [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/312150 (https://phabricator.wikimedia.org/T145419) [00:29:15] (CR) Awight: [C: 2] Update FundraisingEmailUnsubscribe to mirror to Redis [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/312150 (https://phabricator.wikimedia.org/T145419) (owner: Awight) [00:34:23] (PS2) Eileen: Fix to prevent duplicate Home addresses resulting in no primary on merge. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/311908 (https://phabricator.wikimedia.org/T145873) [00:34:25] (PS8) Eileen: Update preferred language based on contribution_tracking. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410) [00:37:06] (Merged) jenkins-bot: Update FundraisingEmailUnsubscribe to mirror to Redis [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/312150 (https://phabricator.wikimedia.org/T145419) (owner: Awight) [00:39:50] !log update paymentswiki from d572ee9e3ef0a97c044f02e9866469a8f3fa5858 to cc27f83f31ecc609d4400050e73905b7364f1d42; mirror unsubscribe queue [00:39:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:42:49] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, FR-ActiveMQ, Patch-For-Review: Migrate FundraisingEmailUnsubscribe to new queue backend - https://phabricator.wikimedia.org/T145419#2658008 (awight) Mirroring is deployed, and I verified that an unsubscribe request lands in both stomp a... [00:44:54] woohoo! [00:50:44] yah nice work! [00:51:08] gtg for the night, have fun [00:51:18] cya! [00:54:38] (PS9) Eileen: [deploy with tools] Update preferred language based on contribution_tracking. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310219 (https://phabricator.wikimedia.org/T96410) [00:56:29] (CR) Ejegg: [C: -1] "Needs a few fixes. Adding a test would be awesome!" (6 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) (owner: XenoRyet) [01:08:25] (PS7) Eileen: Clean up legacy junk data from preferred_language field. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310218 (https://phabricator.wikimedia.org/T96410) [01:08:53] (PS4) Eileen: Remove handling for bad data (language = '_' as we are cleaning that up [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/310724 (https://phabricator.wikimedia.org/T96410) [01:10:17] (PS1) Eileen: Fix to prevent duplicate Home addresses resulting in no primary on merge. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312152 (https://phabricator.wikimedia.org/T145873) [01:13:05] (PS1) Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312153 [01:13:47] ok, I'm heading out too. good evening all! [01:18:34] (CR) Eileen: [C: 2] CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312153 (owner: Eileen) [01:18:43] (CR) jenkins-bot: [V: -1] CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312153 (owner: Eileen) [01:18:46] (CR) Eileen: [C: 2] Fix to prevent duplicate Home addresses resulting in no primary on merge. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312152 (https://phabricator.wikimedia.org/T145873) (owner: Eileen) [01:18:53] (Merged) jenkins-bot: Fix to prevent duplicate Home addresses resulting in no primary on merge. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312152 (https://phabricator.wikimedia.org/T145873) (owner: Eileen) [01:20:00] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312153 (owner: Eileen) [01:21:21] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312153 (owner: Eileen) [01:30:09] (PS1) Eileen: Revise update to primaries not set. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312154 (https://phabricator.wikimedia.org/T143062) [01:30:11] (PS1) Eileen: Add Index to contact.preferred_language [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312155 (https://phabricator.wikimedia.org/T96410) [01:30:38] (CR) Eileen: [C: 2] Revise update to primaries not set. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312154 (https://phabricator.wikimedia.org/T143062) (owner: Eileen) [01:30:45] (CR) Eileen: [C: 2] Add Index to contact.preferred_language [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312155 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [01:30:47] (Merged) jenkins-bot: Revise update to primaries not set. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312154 (https://phabricator.wikimedia.org/T143062) (owner: Eileen) [01:31:00] (Merged) jenkins-bot: Add Index to contact.preferred_language [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312155 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [01:34:07] (PS1) Eileen: Only store the full country code if it is on our valid list. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312156 (https://phabricator.wikimedia.org/T96410) [01:34:09] (PS1) Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312157 [01:34:34] (CR) Eileen: [C: 2] Only store the full country code if it is on our valid list. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312156 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [01:34:40] (CR) Eileen: [C: 2] CiviCRM submodule update [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312157 (owner: Eileen) [01:34:42] (Merged) jenkins-bot: Only store the full country code if it is on our valid list. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312156 (https://phabricator.wikimedia.org/T96410) (owner: Eileen) [01:40:19] !log update civicrm from 5393a13727e8dbad05ffef9ddd44e965eb9282d1 to d30a5e454e6614214faef8354a3a23d71d573c7f [01:40:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:50:50] !log turned off jenkins jobs to run dbupdate: dedupe(s) thank-you & donate import [01:50:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:51:34] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Patch-For-Review: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2658041 (AndyRussG) @awight woohoo yeah let's do it! Thanks!!!! [01:52:18] AndyRussG: hi! Hope yr travels went smoothly [01:53:25] So, I came into this with the prejudice that it was a simple race condition between creating the banner on the db master and loading the content on a replica, then caching those results. [01:53:55] However, the-wub's comment that it "seems to vary by site" gives me pause. [01:54:16] The message cache is shared across sites... [01:54:41] ooh! his example link is still failing [01:54:55] gonna try to pull that value using mwrepl [01:57:13] AndyRussG: mwrepl enwiki; print_r(RequestContext::getMain()->msg( 'Centralnotice-template-B1617_0921_en6C_ipd_p2_sm_pos_btm' )->inLanguage( 'en' )->text()); [01:57:50] And on metawiki it succeeds... [01:57:52] wowza [01:58:07] fyi update script running on barium now - I stopped the jobs I thought would failmail due to the sql taking a few mins [01:58:54] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Patch-For-Review: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2658060 (awight) The bug can be reproduced from the backend: ``` mwrepl metawiki print_r(Re... [01:59:02] eileen: nice work! [02:03:14] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Patch-For-Review: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2658061 (awight) Also interesting: ``` $cache = MessageCache::singleton(); print_r($cache->... [02:03:49] !log all back on [02:03:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:04:11] awight: it's not all done - ie the stuff per the email I sent out will wait on a bit more feedback time [02:04:19] but at least some of the stuff is [02:05:14] Sorry I keep asking for spoon-feeding of the things to review... [02:10:41] AndyRussG: It might still be a simple race condition, it seems. [02:11:44] I don't understand how RequestContext::getMain->msg is returning different results depending on which wiki db context I'm in, but I can see that MessageCache->get is returning empty string for the cached value, and the correct content when useDb is enabled. Which sounds exactly like the race condition we were expecting. [02:11:59] awight: boo! [02:12:28] Thanks a ton for working on this! [02:12:52] Where is that exactly? [02:13:02] HEy, um this page output is cached eh? https://en.wikipedia.org/w/index.php?title=Special:BannerLoader&banner=B1617_0921_en6C_ipd_p2_sm_pos_btm&uselang=en&debug=false [02:13:45] It seems to be the same no matter what extra params I add [02:14:02] ... and not being picked up by the new unicode angle brackets test :( [02:14:21] oh. Test is wrong [02:14:28] well darn [02:14:51] wait it is [02:14:52] gah [02:15:19] AndyRussG: sorry, you mean where is mwrepl? You can run that from tin, it seems to be chronically underdocumented though. [02:16:01] I've been doing: "mwrepl enwiki" or metawiki [02:16:49] I mean where is the useDb of which u speak, and where do u think the race condition could be? [02:16:59] Who owns our core i18n code btw? [02:17:20] Sorry I'm just getting brainwise [02:18:17] I haven't found the race condition, but imagining it's something where the banner gets requested before the new banner content propagates to db replicas [02:18:41] $useDb is an argument to MessageCache::get, called internally by the Message class [02:18:57] Donno who owns the i18n code unfortunately, but you could try mediawiki-i18n [02:19:01] #!$ [02:19:15] I see that my unicode literal was not ok [02:24:51] (PS1) Awight: Correct unicode literals [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312160 (https://phabricator.wikimedia.org/T144952) [02:24:53] AndyRussG: sigh... [02:26:14] I could try to deploy that if you have time to CR [02:32:19] (PS2) Awight: Stop the unicode nonsense [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312160 (https://phabricator.wikimedia.org/T144952) [02:32:20] on second thought, ^ [02:32:31] Seems to be equivalent, and less fragile. [02:38:17] awight: hmmm hope I can look in a few aaaarg [02:38:37] hehe no rush, I'm going disappear into real life in a minute [02:39:44] Getting those log messages will be nice, but it looks like the volume is gonna be *huge* [05:13:59] (PS1) Cdentinger: Add recurring PayPal test [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 [05:18:55] (PS2) Cdentinger: Add recurring PayPal test [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 [05:24:32] (PS3) Cdentinger: Add recurring PayPal test [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 [05:28:03] * cwd yawns [05:40:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Help Civi users configure their dashboards - https://phabricator.wikimedia.org/T118345#2658159 (Eileenmcnaughton) [05:40:10] Fundraising Sprint Bloodletting 2016, Fundraising Sprint Cat Herding, Fundraising Sprint Dirt Farming, Fundraising-Backlog, and 2 others: Figure out & fix the certificate issue that's not allowing civi users to have dashlets - https://phabricator.wikimedia.org/T118356#2658157 (Eileenmcnaughton) ... [05:43:03] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: Resolve conflicts on preferred language - https://phabricator.wikimedia.org/T146344#2658163 (Eileenmcnaughton) [05:44:18] (PS1) Eileen: Resolve conflicts on preferred language. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312182 (https://phabricator.wikimedia.org/T146344) [11:24:26] Fundraising-Backlog: Engage import failing to import certain significant fields - https://phabricator.wikimedia.org/T146295#2658636 (CCogdill_WMF) [14:09:51] Hey all! Fr-Tech, thanks for the work on Phab T144952. Posting here to let you know I'm going to enable our Big English campaigns for a one hour test today. [14:09:52] T144952: Banner not showing up on site - https://phabricator.wikimedia.org/T144952 [14:44:32] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Reintegrate Ingenico rtbt_enets payment method - https://phabricator.wikimedia.org/T141989#2659119 (Pcoombe) @MBeat33 [[ https://donate.wikimedia.org/w/index.php?title=Template:2012FR/Form-section/Paymentmethods/Realtime/Enets&diff=27638&oldid=... [14:53:36] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Reintegrate Ingenico rtbt_enets payment method - https://phabricator.wikimedia.org/T141989#2659166 (MBeat33) Thanks, @Pcoombe [15:56:19] spatton: hi! OK, understood [16:01:43] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Spike: SPIKE: investigate level of completeness of civi deduped DB - https://phabricator.wikimedia.org/T146384#2659343 (DStrine) [16:10:27] (PS1) Cdentinger: Add some sparse logging to PayPal listener. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312273 [16:44:49] AndyRussG: we concluded a 1-hr test in US & CA at noon eastern, today. The one issue I'm seeing is w/ impression counts for our mobile test, where there's a 15% difference between the two banners. The one w/ less impressions is one that wasn't showing up in our troubleshooting yesterday :-/ [16:45:31] spatton: yeah that sounds like a significant differnce [16:45:51] yesterday awight made a patch that will throw an error when we get this problem [16:46:12] That will be good, because we'll get more information in a log on the server, and the impression won't get incorrectly counted as shown [16:46:18] I saw that! Any details you need from me? [16:46:39] Right now part of the problem is error handling code that just lets the bad banner content through, and CentralNotice then registers the impression as OK [16:46:49] spatton: thanks, no not for now, I don't think [16:47:03] AndyRussG: thanks back [16:47:09] I'll definitely let you know if there is anything needed! thanks! [16:49:30] I'm going to see if this could be deployed to production around 11 am Pacific Time [17:00:16] AndyRussG: Thanks much. What do you think; my next step was going to be a 1-hr test in New Zealand and Australia when it's their morning, so in about 6 hours. Should I plan to procede? [17:04:24] spatton: I think so. The error shouldn't be noticeable to users and should prevent any unwanted content from showing [17:04:36] AndyRussG: awesome, thanks [17:19:20] (CR) Ejegg: [C: -1] "can't depend on txn_id existing" (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312273 (owner: Cdentinger) [17:22:20] ejegg: thanks, i was wondering about that [17:22:28] i thought i missed something copying the file [17:23:01] oh, has it been mislabeled for a while? [17:23:04] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Patch-For-Review, Spike: Spike: Zendesk migration from Web Portal to Helpdesk - https://phabricator.wikimedia.org/T143142#2659537 (MBeat33) I migrated the store, and it looks like everything's wo... [17:23:15] nah that file was never in use before [17:23:24] gotcha [17:23:29] i scrubbed adam's record and put it in there [17:23:38] but i guess i grabbed the signup one instead of payment [17:23:42] ah right [17:23:48] can do both at some point [17:24:00] so, the existing logger var_dumps everything in the message [17:24:11] spatton: I think I'm mistaken. It looks like actually the most important patch to prevent the banner error was deployed yesterday! [17:24:13] but... are we trying to limit PII in logs? [17:24:14] woohoo [17:24:25] ^ that's the case, right? fr-tech? [17:24:37] ejegg: yeah probably [17:24:42] AndyRussG: which one, the one detecting the new angle brackets? [17:25:00] Yeah [17:25:12] I see it's on the wmf_deploy branch, but I don't see any record on the Deployments page [17:25:26] I +2ed it yesterday, but I wasn't aware of awight or anyone getting it deployed [17:26:23] morning awight [17:26:41] awight: boo! so fancy angle patch +2'd but not yet deployed, is that status? [17:27:11] ejegg: do signup and payment messages both go to recurring queue? [17:27:18] cwd yep! [17:27:23] the consumer for that is a beast [17:27:40] word [17:27:40] especially since it's doing the normalization for who-knows-what-reason [17:28:37] AndyRussG: ah cool, I can sneak that deployment out [17:30:04] ejegg: ah yeah that will be nice to fix [17:30:08] awight: ah it's ok, we I can put it on SWAT in half an hour also :) [17:30:31] Just wondering if it war deployd or nicht [17:30:50] thx much 4 working on that!!!! [17:31:12] also 4 the CR ejegg [17:32:14] fr-tech OK so I'll put it on the SWAT at 11 am PDT [17:32:21] cool [17:32:29] (in 5 I'm relocating and I'll be back online by then...) [17:32:58] cwd besides adding logging what else is left on the listener? [17:33:12] or is the logging just to reassure us that it's all working correctly? [17:33:20] yep [17:33:30] i added another test and fixed a warning last night [17:33:38] but afaik we can switch it over today [17:33:46] nice! I'll prep a config update to enable mirroring of the donations queue [17:33:59] sweet [17:34:03] AndyRussG: good point, I'll prep those patches [17:34:07] i'll have the rest of this fixed up momentarily [17:35:17] (PS2) Cdentinger: Add some sparse logging to PayPal listener. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312273 [17:39:08] awight: K just added us to the morning SWAT [17:40:14] AndyRussG|bassoo: oh, here's the fix we want to deploy: 312160 master Stop the unicode nonsense [17:40:41] AndyRussG|bassoo: The last patch was bogus cos "\u" doesn't exist until PHP 7 [17:41:27] fr-tech: I need a bit of CR on https://gerrit.wikimedia.org/r/#/c/312160/ and its predecessor... [17:41:49] awight: ooh [17:42:02] all these assumptions I make... [17:42:22] I should have tested! [17:45:04] (CR) Ejegg: [C: 2] "Please excuse my dear aunt Sally - she never told me about concatenation!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312139 (owner: Awight) [17:46:25] wat [17:46:37] well, this will be an exciting upgrade. [17:46:44] I'm expecting a relentless flood of fail logs [17:47:00] (Merged) jenkins-bot: Fix bad ternary [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312139 (owner: Awight) [17:49:05] (CR) Ejegg: "Slightly alters the conditions where we throw the error - is that intentional?" (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312160 (https://phabricator.wikimedia.org/T144952) (owner: Awight) [17:49:52] ejegg: eh good point. I was trying to get fancy, but an empty message is not quite a translation error [17:51:27] huh. On third thought... [17:51:46] nah. [17:52:07] (PS3) Awight: Stop the unicode nonsense [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312160 (https://phabricator.wikimedia.org/T144952) [17:52:09] * awight smoke tests though [17:52:13] ejegg: gimme a minute... [17:53:09] cool [17:54:32] As much as I like breaking CentralNotice for readers... [17:57:32] :P [17:57:46] How are you triggering the error locally? [17:57:57] I'm gonna damage the key variable [17:58:30] mw.centralNotice.insertBanner( false /* due to internal exception */ ); [17:58:30] cool [17:58:39] and normal thing works. [17:58:43] nice [17:58:51] I'll just trust that the log message is written [17:59:12] (CR) Ejegg: [C: 2] "Way less flukey!" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312160 (https://phabricator.wikimedia.org/T144952) (owner: Awight) [17:59:30] Can't believe I tried to make the workaround even more kludgey [18:00:44] ejegg|meet: don't sign anything! [18:00:51] (Merged) jenkins-bot: Stop the unicode nonsense [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312160 (https://phabricator.wikimedia.org/T144952) (owner: Awight) [18:01:25] (PS1) Awight: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/312290 [18:01:34] (CR) Awight: [C: 2] Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/312290 (owner: Awight) [18:03:31] (Merged) jenkins-bot: Merge master into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/312290 (owner: Awight) [18:11:18] AndyRussG|bassoo: I've tweaked our SWAT request slightly, fyi [18:13:55] awight: ah OK [18:17:36] ah K I see it's the more recent ones going out [18:18:08] awight: ejegg|meet cool thx for doing that update! [18:34:36] cwd|afk: Anything I can do to help with the listener? [18:34:54] Ah, CR! [18:34:57] awight: there are a couple patches up if'n you wanna... [18:34:59] yeah [18:42:28] (CR) Awight: [C: -1] Add recurring PayPal test (6 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 (owner: Cdentinger) [18:44:20] (CR) Awight: Add some sparse logging to PayPal listener. (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312273 (owner: Cdentinger) [18:48:49] cwd: ok, settings repo has a patch to enable donations mirroring in DI, and there's an updated SmashPig.yaml in my deploy-server that enables mirroring from SmashPig. [18:49:35] ejegg: forgive me, why was this listener blocking donation mirror? [18:49:35] oh, the updated settings also copies the jobs-paypal data-store over to default in case we want to move that key for requeueing purposes [18:50:01] cwd cause we didn't want to hack redis into the standalone piece of ... code [18:50:10] ooh, right right [18:50:12] cool [18:50:15] like half of that logic was stomp-entangled [18:50:43] maybe i should pull the little piece that fixes the warning out of the first patch so we can deploy that while i spruce up the tests [18:50:59] otherwise we'll get a bunch of fail mail [18:51:29] k... i'mma relocate, back shortly [18:55:25] (PS1) Cdentinger: Fix undefined date warning [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312311 [18:56:01] (CR) Awight: [C: 2] Fix undefined date warning [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312311 (owner: Cdentinger) [18:56:54] (Merged) jenkins-bot: Fix undefined date warning [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312311 (owner: Cdentinger) [18:58:24] awight: think that was the only part of those patches that will actually affect prod (besides logging i guess) [18:58:36] so if we wanna move forward we can do that while i address your comments [19:02:31] awight: so that's all the log shows? [19:02:42] Not too surprising, I guess [19:02:48] cwd: I don't know if there's much moving forward to do until we can feel confident about recurring messages [19:02:57] AndyRussG: Want to turn on a campaign and see what happens? [19:04:31] awight: I guess I don't know what you'd hope to discover by finding more details about the scale... Or am I missing something? apologies for the grouch in me... [19:04:54] eh I suppose nothing [19:05:30] I was expecting to see a flurry of WLM failures, too. But disappointed on all fronts [19:06:37] Also on my mind is that I've set up a powderkeg, so would like to see what happens when a campaign is enabled. If it's bad, then we need to disable logging until the underlying content bug is fixed. [19:07:08] Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, and 3 others: Banner not showing up on site - https://phabricator.wikimedia.org/T144952#2660027 (awight) a:awight>None [19:08:33] (PS1) Ejegg: Move jobs-paypal under default queue key [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312316 [19:11:46] awight: right, we want to be able to tell when it's happening, but not cause the alarm itself to blow up the phone lines [19:12:22] I think in terms of scale, it's something that can get huge or not, it's just chance [19:13:00] AndyRussG: We seem to have this turned on: http://www.rsyslog.com/doc/rsconf1_repeatedmsgreduction.html [19:13:20] which could save the butts [19:13:43] Hey cool, we're getting these logs too, now: 2016-09-22 19:08:42 [V@QsOgpAME8AAIFyhDoAAAAR] mw1244 metawiki 1.28.0-wmf.18 CentralNotice INFO: StaleCampaignException while loading banner: 'B1617_0921_en6C_dsk_p1_lg_dsn_cnt'. Campaign: C1617_en6C_dsk_FR [19:15:59] I don't understand why it sez "INFO" if we're throwing a RuntimeException. Is that wrapped in a try/catch somewhere? [19:16:04] yes [19:16:06] I mean for the no banner message thing [19:16:12] it's caught by SpecialBannerLoader [19:16:20] Ah K sneaky [19:16:38] totally... but we need to return valid JS to the client [19:17:59] Ah yeah /me recalls that bit [19:19:24] Some of those stale campaign exceptions are expected, I think, due to JS cache oddities of the universe [19:22:03] Democracy Now is killing me [19:22:28] They need to rename? [19:24:55] nice one [19:26:05] We need to deploy Ignorant Racist Recognition experts to these streets [19:31:24] awight: a little confused about how to make the dataprovider accessible from the mock api [19:31:56] also whether i am actually confirming anything by comparing against the message i just set [19:32:26] awight: its all craziness beyond what one would have imagined [19:32:58] Americans are so full of fear [19:33:50] Stay off the "freedom" fries and drill out your bullets, we're all gonna make it. http://www.cdc.gov/nchs/fastats/leading-causes-of-death.htm [19:34:53] amazing how many of those things can fall under the blanket of malpractice [19:35:28] cwd: Lessee, you can do something like DonationInterface/tests/phpunit/DataValidatorTest.php [19:36:26] ah, have the mock api call a static method on there? [19:36:36] Ah, now I get what you're after with the mock API. You're testing whether the message actually matches. [19:36:40] I don't think there's a need for that [19:36:51] ok, can i just have it return VERIFIED? [19:37:02] yeah, or INVALID depending on configuration. [19:37:14] sounds good [19:37:21] Cos testing that it compares the message correctly is an internal of the real API [19:38:13] yeah [19:38:16] makes sense [19:38:44] AndyRussG: We got a little flurry of Centralnotice-template-B1617_0922_enJP_dsk_p1_lg_txt_nat errors--was that you? [19:38:44] thanks [19:39:13] awight: nope [19:39:28] BTW I'm hunting for who owns or has tinkered with MessageCache [19:39:37] Just asked folks on #wikimedia-perf [19:39:43] cwd: I stumbled across this propaganda, but it's great stuff! http://st.inf.tu-dresden.de/files/teaching/ws11/st2/slides/21-modular-design.pdf [19:41:38] nice [19:41:40] * cwd pinboards [19:44:18] awight: "p":"paypal/listener" is in the request from PP so i thought i'd just leave it in for verisimilitude [19:45:21] p.s. would you mind dialing me in to standup? [19:46:00] k [19:46:14] ty [20:12:53] (CR) Ejegg: Add recurring PayPal test (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 (owner: Cdentinger) [20:13:26] (PS4) Cdentinger: Add recurring PayPal test [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 [20:20:41] AndyRussG: the strict mode patch should be ready for another round of review [20:20:56] I broke out the function signature changes into a follow-on [20:21:05] fr-tech: hey all, throw your scraps my way... I need morsels [20:22:10] ejegg: ah fantastic thx :) [20:22:13] here's a truly bite-sized one: https://gerrit.wikimedia.org/r/312316 [20:23:02] (CR) Awight: [C: 2] "Agreed! I feel weird about everything that's happening here, but it's good for now." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312316 (owner: Ejegg) [20:23:09] awight: just put that test patch up, think i addressed everything but there was some rebase snafu [20:23:28] awight: also, this one makes DI tests strict about mock data existing: https://gerrit.wikimedia.org/r/310684 [20:24:02] ooh, I should make some progress on this damaged message UI WIP: https://gerrit.wikimedia.org/r/305744 [20:24:04] (Merged) jenkins-bot: Move jobs-paypal under default queue key [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312316 (owner: Ejegg) [20:24:10] cwd: base looks fine on that, fwiw [20:24:47] I think I stalled out wondering where to put a common re-queueing method for use by the delayed-requeue job and the UI [20:25:57] praying i didn't accidentally your personal information [20:27:28] (CR) Awight: [C: 2] Add recurring PayPal test (4 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 (owner: Cdentinger) [20:27:45] cwd: oh dear. /me rereads [20:28:05] pretty sure i got it all! [20:28:23] (Merged) jenkins-bot: Add recurring PayPal test [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312175 (owner: Cdentinger) [20:28:24] nothing I recognize as belonging to me :) [20:29:08] * cwd wipes brow [20:32:01] awight: Hey, just seeing your note re: Centralnotice-template-B1617_0922_enJP_dsk_p1_lg_txt_nat errors - I just created that banner, it didn't load on the first few attempts then 'started working' [20:32:19] by "just" I mean - over an hour ago [20:35:38] spatton: cool! awight has been watching the log... [20:43:37] * ejegg confuses self trying to resolve crm merge [20:47:32] that seems like it would be easy to do [20:47:46] i mean getting confused merging, not merging itself [20:50:26] (PS1) Ejegg: Merge commit 'ae5a926512fb07c3a74ed5163e43f83dd93bc917' into HEAD [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312332 [20:50:41] i would definitely say i've learned more about git at this job than any other single thing [20:51:10] i used to be terrified of submodules [20:51:10] hehe, yeah, especially given two+ weeks of cherry-pick commits [20:51:13] i've resigned myself to always feeling like a git noob [20:51:13] spatton: Very interesting, okay thanks. The good news is that I only saw 7 cases where the banner failed to load for people. [20:51:28] even though I'm like 10 years into using it [20:51:28] spatton: I'm curious--does that match the number of times you requested it, by any chance? [20:51:34] I'd love to have someone else bang on https://gerrit.wikimedia.org/r/312332 [20:51:41] ooh, lemme update the commit message [20:51:58] spatton: The translations loaded were en, en, en, en, fr, ru, en in that order [20:52:50] ooh fun [20:53:18] (PS2) Ejegg: Merge commit 'ae5a926512fb07c3a74ed5163e43f83dd93bc917' into HEAD [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/312332 [20:53:27] getting food, back soon [20:53:27] ejegg: Sometimes I sanity check by diffing against master at the same commit. I'll do that for your merge [20:53:39] awight: yep, I was doing that [20:53:42] k [20:53:54] Jeff_Green: https://git-man-page-generator.lokaltog.net/ [20:54:06] so much deleted test stuff to scroll by, though, i'm not sure i caught all the things [20:54:34] and of course it's master-at-that-commit-plus-tons-of-cherry-picks [20:55:12] yeah, gross. There are many diffs [20:56:23] ejegg|food: not sure what you mean about the cherry-picks, though-- git cherry -v b30b34f76f2647cdaac27f9b139978c7390b4313 ae5a926512fb07c3a74ed5163e43f83dd93bc917 is clean. [20:58:32] cwd: yes, that's the best reference in the whole world for git [21:00:33] makes me laugh every time [21:09:44] awight: Yeah, that matches! I loaded in fr and ru to see if switching the language would get the banner to load (seems to have worked at other times) [21:13:11] ejegg|food: Something's wrong with the merge, I think. We lost a few patches, e.g. 293604a7f9dee64f0f282e897224bb2411dfdced [21:13:48] spatton: That's pretty weird! Well, I'm glad to see there wasn't a huge flood of the whole world getting invalid banners, which is what I had expected. [21:14:46] awight: Well, FWIW, when I've had problems loading a banner ... and then it starts loading ... it seems to *keep* loading, and I didn't enable this campaign til the banner was loading for me. [21:17:46] wow, okay thanks for the data points. That seems like a good workaround for now. [21:18:55] ejegg|food: I'm very confused. I see that the "clean up legacy junk" patch I'm following wasn't merged until after ae5a926512fb07c3a74ed5163e43f83dd93bc917 -- still, when I git cherry your new deployment against origin/master, the clean up junk patch is marked with a "-", which means it'll never be merged into our repo, it was somehow marked as already cherry-picked. [21:19:26] awight: I think it /was/ already cherry-picked [21:19:32] ejegg|food: oho! yeah git cherry -v d30a5e4 origin/master [21:19:37] however, it's not present in the code. [21:19:52] Looks like an earlier merge is to blame. [21:20:36] awight: huh? I definitely see it in the code [21:20:50] git log -p sites/all/modules/contribution_tracking/contribution_tracking.install [21:21:06] eileen's patch is the latest to touch it [21:21:25] O_o [21:21:51] you're looking at b30b34f ? [21:22:09] okay sorry I'm reading a diff backwards. Glad I could help :p [21:22:15] hehe [21:23:42] Now I wish we had tests on deployment :) [21:25:35] ejegg|food: Okay the diff looks clean to me, just a few tweaks that were cherry-picked over as you said. [21:26:23] huh. this log is fun. 2016-09-22 21:06:36 [V@RH3ApAADwAAXJ7XiUAAADQ] mw1265 metawiki 1.28.0-wmf.20 CentralNotice INFO: Campaign could not be retrieved from database with id '' or name 'TOU_Notification' [21:30:34] ^ awight wat? [21:31:46] It's requesting future campaigns that don't even exist yet???! [21:34:53] (CR) Krinkle: "Why is this dependant on DB strict mode?" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg) [21:37:38] (CR) Krinkle: [C: 1] Fix typos and PHPdoc hints [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312069 (owner: Ejegg) [21:42:59] (CR) Ejegg: "CI was failing all CN patches on strict mode violations, so I rebased stuff on top of this fix to get accurate test results. So I guess CI" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 (owner: Ejegg) [21:43:28] (PS5) Ejegg: jQuery usage tweaks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/291917 [22:00:17] awight, ejegg - any opinions at this point about rolling the donations queue over? [22:00:30] tomorrow is friday and all... [22:01:57] yeah, if the paypal listener is ready! [22:02:24] I've just asked for a SmashPig.yaml deploy with the new settings [22:02:32] and mirroring turned on [22:03:01] It looks like the paypal audit is already configured to mirror - I'm just looking in the redis queues to see if that's been happening [22:04:36] i *think* it's ready [22:04:59] oh yeah, bunch of paypal stuff from audits already in the redis queues [22:05:38] cwd so now we flip the url rewrite to point to our new guy? [22:06:07] guess we should hustle to do that before Jeff_Green's work day is over [22:08:49] hmmm yes i think that's right [22:08:58] there was some weirdness [22:14:52] (PS2) Ejegg: Remove pre-1.25 API compatibility code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/311780 (owner: Anomie) [22:34:47] (CR) Ejegg: [C: 2] Remove pre-1.25 API compatibility code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/311780 (owner: Anomie) [22:36:54] (Merged) jenkins-bot: Remove pre-1.25 API compatibility code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/311780 (owner: Anomie) [22:40:48] nice! [22:41:29] cwd: Were you able to push a subscr_signup and subscr_payment through the new endpoint? [22:42:34] awight: didn't they both work yesterday? [22:43:03] O uh I thought not. [22:43:11] one sec... [22:43:11] They went through the old endpoint, right? [22:43:32] * awight rubs irclogs [22:44:57] Ah hey, you're right--somehow the charge came through on the new endpoint rather than the old one [22:45:32] notify_url weirdness? [22:47:48] cwd: Thanks, those sure do look good [22:48:04] \o/ [22:48:06] I'm confused now, though--did both the subscr_signup and the subscr_payment create contributions? [22:48:28] hmm [22:48:34] they both get pushed unmodified to the consumer [22:49:00] which i am not familiar with [22:49:09] queue2civicrm? [22:49:43] yah queue2civicrm/recurring [22:50:46] I don't understand, the subscr_payment shows up twice, too. once with a contact_id and once without [22:51:51] ah in the jenkins log? [22:51:53] funky logging, seems to only be in the db once [22:51:54] yeah [22:52:01] yeah i see that [22:52:03] hmmmm [22:52:38] It's only one incoming message, so not the fault of the listener [22:53:54] there are a lot of moving pieces [22:58:25] looks like wmf_civicrm_normalize_msg is called twice on recurring payments [22:59:03] and that's where the log line with Contribution (normalized) is written [22:59:40] any harm in that? [23:01:05] besides filling up the disk with logs [23:01:06] well, it's been going on for a while! [23:01:18] notable that that disk did fill up recently :P [23:02:22] Fixing that can be part of the cleanup where we transmogrify recurring messages in SmashPig [23:02:31] ah yes [23:02:56] ejegg: good find! [23:03:20] Fundraising Sprint Qwerty Thwacking, Fundraising Sprint Rocket Surgery 2016, Fundraising-Backlog, Patch-For-Review, Spike: Spike: Zendesk migration from Web Portal to Helpdesk - https://phabricator.wikimedia.org/T143142#2660921 (MBeat33) Open>Resolved a:MBeat33 [23:03:24] ty [23:04:04] awight: any ideas on that orphan rectifier failure to queue? [23:04:23] oh, that's in done! [23:04:25] Yeah cwd thanks for moving that var_map into SmashPig config for us to experiment with. My current thinking is that we should probably wrap each API dialect in its own namespace, what you think? [23:04:27] I missed the fix... [23:05:29] ejegg: It's a mess. The orphan rectifier is on life support for the moment--I believe it just barely does its job, settles the correct transactions, and after that it just spews garbage at the queues. [23:05:36] (kind of describes me!) [23:06:14] awight: you mean among the providers or the different txn types? [23:07:17] Mostly the different txn types. Like, PP has a legacy name-value dialect, and an "express checkout" dialect. Then, the recurring txns are mostly made up of legacy dialect, but with additional fields. Maybe we use yaml transclusion, or an explicit schema field for inheriting dialects... [23:07:58] There are lots of interesting GlobalCollect cases where the API is slightly different per payment method or country. It's always tricky deciding where to stop being declarative and just have the custom logic to override. [23:08:09] yep [23:08:21] there's already a ton of that shit in Job (sorry about that class name btw) [23:08:42] But I really like the balance we're striking in DI, where even the exceptions are listed in a static file, and we just include a bit of functional glue to pull in the exception [23:08:46] lol [23:09:00] naming things is hard [23:09:01] git mv is no obstacle [23:09:17] SmashPigPayPalListenerJob would have been worse :) [23:09:27] hehe [23:09:28] At least it's erring in the right direction [23:10:05] the stuff i just made that does the var_mapping would be easy to generalize [23:10:17] and add the ability to override [23:11:40] I'm going to run for the day. Talk to you all tomorrow. [23:11:58] cya! [23:12:27] cwd: Yah let's just all talk about it first. I think we should do that as part of merging DI/*_gateway/config into the SmashPig style [23:13:25] gotta do a few dishes and mopping so my family doesn't come home to the wild desolation and filth that I prefer ;) [23:13:35] that'll do [23:14:54] yeah definitely, wasn't trying to set a precedent, and want to be deliberate [23:26:13] (PS3) XenoRyet: Move Unsubscribe Queue Consumer off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) [23:27:26] (CR) XenoRyet: "Fixes in. I was figuring on doing tests in their own patch. That seem ok to you?" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) (owner: XenoRyet) [23:29:28] XenoRyet: cool! will review in one sec [23:29:46] No rush, I've gotta go pick up the boy from school in a few minutes anyway. [23:30:27] Though while you're here, you use PHPStorm, right? You ever get a thing where the code inspection stuff goes wonky on you? [23:31:17] It's almost like the inspector isn't doing new passes when I change stuff. [23:38:26] XenoRyet: yeah, PHPstorm [23:38:36] Hmm, it's still being really helpful for me [23:39:09] It has a little delay between fixing stuff and the warnings going away, but mostly timely [23:39:26] Yea, mine is being weird. I'm missing stuff because I haven't gotten used to it lying to me. [23:40:09] Figured I'd just check to see if you'd seen that before. No worries. [23:40:33] It's probably good practice to not be so reliant on it. [23:43:43] so seductive though! [23:43:53] when it's working it's been a real timesaver [23:44:01] Right? So good when it's working. [23:44:14] missing 'use' statements are so hard to spot otherwise [23:44:24] Yep [23:46:16] (CR) Ejegg: [C: -1] "Few more fixes..." (5 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) (owner: XenoRyet) [23:47:51] Yea, see. The IDE was complaining about it when I just did CRM_CORE_DAO. Weirdness. [23:58:37] (PS4) XenoRyet: Move Unsubscribe Queue Consumer off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/312084 (https://phabricator.wikimedia.org/T145419) [23:58:51] Anyway, off to pick up the kid. Catch you later.