[00:13:03] http://www.mediawiki.org/wiki/Gerrit [00:13:25] This part is important, http://www.mediawiki.org/wiki/Gerrit/Tutorial#Set_Up_SSH_Keys_in_Gerrit [00:13:39] Also, gotcha: http://www.mediawiki.org/wiki/Gerrit/Tutorial#Prepare_to_work_with_gerrit [00:13:58] perfect [00:13:59] ty [00:25:41] Fundraising Tech Backlog: Orphan slayer logging should be bucketed - https://phabricator.wikimedia.org/T99029#1283984 (awight) NEW a:Jgreen [13:29:22] (PS1) AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/210906 [13:33:07] (CR) AndyRussG: [C: 2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/210906 (owner: AndyRussG) [13:33:52] (Merged) jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/210906 (owner: AndyRussG) [13:41:05] power cycling barium now [13:47:28] cmjohnson1: hi! I think it's not a problem, but we're about to do a CentralNotice deploy that touches the code that displays banners. So it's important that we be able to see if anything is amiss due to the deploy [13:48:38] AndyRussG, barium is already back up [13:48:59] cmjohnson1: cool, thanks! :) [13:59:45] Morning ejegg|away! Almost got all the patches ready... Here's the first core one: https://gerrit.wikimedia.org/r/#/c/210909/ [14:02:40] hi AndyRussG ! [14:03:30] ejegg: hi! :) [14:03:42] Ready for some +2 core depoy branch magic? [14:03:52] Is https://www.mediawiki.org/wiki/Extension:ContributionTracking really still used? [14:04:07] Nemo_bis: yeah, sadly [14:04:22] I think it should totally be rolled into DonationInterface [14:04:43] AndyRussG: OK, merged that first submodule bump [14:05:45] ejegg: fantastc! Just lemme get the second [14:06:08] Might want to hop on #operations, too, BTW [14:07:04] Ok thanks [14:08:41] Nemo_bis: yep, we're using a vanishingly small piece of ContributionTracking functionality in DonationInterface to update a row in a contribution_tracking table - I'm pretty sure the ajax call referenced on that page doesn't happen any more [14:11:57] ejegg: thx! [14:19:41] ejegg: this is where the one I'm still nervous about (not too much tho): sync-dir php-1.26wmf5/extensions/CentralNotice/ 'Update CentralNotice' [14:19:48] Looks right? [14:21:02] yeah, looks good! [14:21:12] ejegg: cool, here goes then :) [14:26:34] ejegg: looks fine, just looking for an active campaign to check [14:26:55] cool! [14:28:47] ejegg: yeah all good: https://en.wikipedia.org/wiki/Main_Page?uselang=fr&country=FR&randomcampaign=0.2 [14:28:52] vs [14:29:04] https://en.wikipedia.org/wiki/Main_Page?uselang=fr&country=FR&randomcampaign=0.7 [14:29:14] looks good to me too! [14:29:49] oh, what's supposed to be different? [14:30:29] ejegg: nothing ostensibly for the user [14:30:37] AH, OK [14:30:41] * K4-713 yawns [14:30:41] ah, ok [14:30:58] K4-713: mornin', early bird! [14:31:09] the randomcampaign param is new tho [14:56:29] ejegg: all synced to wmf6 too. The test.wikipedia.org site looks fine (desktop and mobile) though I couldn't see any campaigns on sites with wmf6 to test with... [14:57:56] anyway I'm logging out of tin, a different deploy starts now, but I'll still try to test banners w/ wmf6 [14:58:03] ejegg: thanks 4 ur help! [14:58:32] thanks for a smooth deploy! [14:59:28] heh it hasn't quite gotten programmed into my subconscious as is the case of awight, but at least it's getting easier :) [15:41:10] ejegg: K4-713: tested a banner on wmf6 too, all good [15:41:32] Had some issues because of debug=true URL param not working, apparently it's a cache issue unrelated to our stuff [15:41:49] (see #wikimedia-operations backscroll for details) [15:42:03] odd... reading [15:42:34] yeah! I was freaking out for a sec when I did debug=true and saw the old code! [15:43:17] :) [15:43:49] ah, so resourceloader stuff is now coming through a different varnish or something? [15:50:04] AndyRussG: ah, I was having issues with debug=1 earlier. haven't seen any other problems though :D [15:50:09] thanks for the deploy! [15:50:55] the-wub: np! yeah all smooth except for that... do you know when the debug issue started happening? (might help ops track it down) [15:53:27] think I noticed it around 1500 UTC AndyRussG. but only tried it that one time (was trying to make sure I got an uncached link) [16:20:39] K4-713: ejegg: update about debug=true: It _is_ our code breaking when you invoke debug=true, but the root cause is some cluster config weirdness that is causing a mix and of old and new code when u use that use that URL param [17:27:03] https://wiki.debian.org/DontBreakDebian#Don.27t_.27make_install.27 [17:27:06] wat [17:27:40] funny that the next bullet point is "don't blindly follow bad advice" [17:27:45] http://mashable.com/2015/05/14/cat-purr-world-record/ [17:27:47] That's all I got [17:28:15] lol, "make install" considered evil [17:28:36] well debian i think this is my stop [17:28:42] make install 4 life [17:29:04] Cantchu just set PREFIX=/usr/local and call it a day? [17:29:21] for sure [17:29:29] "Don't have poor etiquette when getting support, motherfucker!" [17:29:32] and anyway it's not like apt cleanly removes everything it installs [17:29:55] hehe [17:29:58] slackware packages (aka tarballs) now come with a bill of materials for easy removal [17:30:09] everything old is new again [17:53:17] ejegg: K4-713: awight: cwdent: (/me pings massively for anyone who might be intersted) https://phabricator.wikimedia.org/p/bd808/ [18:10:09] AndyRussG: what's that? [18:10:17] bd808 is great :) [18:11:21] Is this about https://phabricator.wikimedia.org/T99096 ? [18:28:41] awight: it's cos debug=true stopped working for CN after this morning's deploy [18:28:52] oh dear! [18:29:01] luckily that only affects us [18:29:14] I'm very interested in the solution, cos paymentswiki also fails when debug=true [18:33:45] awight paymentswiki debug fail is whitelisting [18:34:50] oh? thanks! [18:35:00] We need to whitelist another ResourceLoader URL? [18:42:19] ejegg: (or any node people) https://gist.githubusercontent.com/anonymous/c5a3d076dbf6762d89ae/raw/208053ec8dba5c566674f5d9c2dea643b2d99e14/gistfile1.txt [18:42:25] i'm getting that about several packages [18:42:37] sure looks like the requested version is there [18:42:42] 3 versions that would match in fact [18:43:29] wondering if i'm missing something obvious [18:46:28] cwdent - ahh, that's odd. Let me see [18:47:09] cwdent: oh, major gotcha for dash - we're using node 0.8.2 in production, so the packages in the node_modules git submodule are all for that version [18:48:01] you can use node version manager to switch between versions locally [18:49:19] awight: since debug=true doesn't bundle things, I think we'd have to whitelist each and every script and css file to make that work [18:50:39] awight: ejegg: see the e-mail to fr-tech for the fix to CN debug=true [18:51:47] Ahh, more cache fun [18:51:59] That bit me pretty bad on my first CN deploy [18:52:55] AndyRussG: we were riffing on that to think about the fact that debug=true has never worked on paymentswiki [18:53:26] But I'm pretty sure it's giving 403s or 404s on all the unbundled scripts 'cause we lock that whole place down with a whitelist [18:55:15] huh, yeah i'm using 0.8.2... [18:55:51] hmm. You should be able to run just fine if you checkout out the node_modules submodule [18:56:14] We checked all those in since we can't run npm from production during deployment [18:56:39] Similarly to how we check in our vendor directories with composer libs [18:56:41] heh the cn case today also gave s a 40 because of a fun fluke bug-invoke [18:56:51] bassoon! [18:57:58] cwdent: what are you working from as far as documentation? I forget how much we actually wrote down [19:26:35] sorry ejegg, was eating, but what i'm going off is here: https://collab.wikimedia.org/wiki/Fundraising/Engineering/Dash [19:28:11] cwdent: oh hey, that's pretty informative! thanks pizzzacat, wherever you are [19:28:50] cwdent: were you trying to npm install when you got those package errors? [19:31:12] yeah...should just use the submodule instead? [19:32:46] the submodule should get you started, but we should figure out the install issues [19:33:48] maybe some things have conflicting dependencies and we need to add resoltions to package.json [20:18:39] atgo: sorry!!! I didn't mean to exclude you wrt the CentralNotice intro... [20:18:45] haha no worries! [20:18:52] i'm there if you need me, but i totally don't need to me [20:18:53] be* [20:20:23] atgo: how "busy" is your 11:30 - 1 pm "busy" block on your calendar tomorrow? [20:20:30] busy :) [20:20:32] don't worruy about it [20:20:39] i really was just offering to be there if it would be helpful for context [20:22:41] atgo: I think it'd be really great for you to learn more about CN and its current state of affairs... I was planning to talk especially about how the code is organized and where we can go with it... also about requirements [20:23:36] Everyone also has a "brown bag" -type spot after standup tomorrow, that also might work [20:24:39] awight: ejegg: cwdent: XenoRyet ^ for CentralNotice, how is 1:15 (Pacific Time) tomorrow? too lunchy? [20:24:48] Also, there's a talk on right now about the graph extension: http://www.youtube.com/watch?v=j7DTn9jHnI0 [20:25:34] AndyRussG: works for me! [20:25:39] me too! [20:26:04] and me [20:26:06] Yea, looks good for me [20:26:10] AndyRussG: btw there is this tab I just learned about in google calendar, when you create an event, click "find a time" [20:26:15] it's like hotel booking [20:26:28] That seems handy [20:26:56] ah cool! [20:28:31] https://github.com/ingydotnet/git-subrepo [20:29:44] cwdent: hmm [20:30:26] atgo: hrm you also have something after, or actually during and also after, standup tomorrow! [20:31:30] no idea if it's any better than all the other git-sub* [20:39:42] this week is crazypants AndyRussG. you guys should skip me :) [20:59:38] Fundraising-Backlog: Spike: Investigate NULL country values in drupal - https://phabricator.wikimedia.org/T99128#1285944 (CCogdill_WMF) [21:16:01] Fundraising-Backlog: Spike: Investigate NULL country values in drupal - https://phabricator.wikimedia.org/T99128#1286059 (atgo) p:Triage>Normal [21:18:29] Wikimedia-Fundraising-CiviCRM: CiviCRM - how to format text in a text box in the prospect tab? - https://phabricator.wikimedia.org/T98262#1286080 (atgo) p:Triage>Low Triaging as low priority. @lgillis @rlewis @caitvirtue if you need this, let me know and we can discuss where to get it on the list. Than... [21:21:13] (PS1) Awight: cleanup [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211039 [21:23:18] booted the CN meeting to Monday, please lmk if it works 4 u :) thx! [21:23:53] Works for me [21:26:30] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Check Memcache limbo contents for sanity - https://phabricator.wikimedia.org/T99016#1286112 (awight) Successfully tested limbo message destruction after completing the normal donation workflow. I simulated... [21:26:42] Fundraising Sprint House of Pain, Fundraising Sprint Indigo Girls, Fundraising Sprint James Brown, Fundraising Sprint Kraftwerk, and 3 others: Use the new queue wrapper for limbo, antimessage, and orphan slaying - https://phabricator.wikimedia.org/T92922#1286114 (awight) [21:26:45] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Check Memcache limbo contents for sanity - https://phabricator.wikimedia.org/T99016#1286113 (awight) Open>Resolved [21:27:12] Fundraising Sprint House of Pain, Fundraising Sprint Indigo Girls, Fundraising Sprint James Brown, Fundraising Sprint Kraftwerk, and 4 others: Use the new queue wrapper for limbo, antimessage, and orphan slaying - https://phabricator.wikimedia.org/T92922#1123892 (awight) [21:27:21] Fundraising Sprint House of Pain, Fundraising Sprint Indigo Girls, Fundraising Sprint James Brown, Fundraising Tech Backlog, and 3 others: Use the new queue wrapper for limbo, antimessage, and orphan slaying - https://phabricator.wikimedia.org/T92922#1123892 (awight) [21:31:48] (PS1) Ejegg: Make do_transaction return a PaymentTransactionResponse [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211046 (https://phabricator.wikimedia.org/T98063) [21:32:13] (CR) jenkins-bot: [V: -1] Make do_transaction return a PaymentTransactionResponse [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211046 (https://phabricator.wikimedia.org/T98063) (owner: Ejegg) [21:34:06] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Orphan slayer reads from Memcache rather than ActiveMQ - https://phabricator.wikimedia.org/T99017#1286142 (awight) [21:37:27] Jeff_Green: Whaddyu think about putting Redis on the payments cluster? [21:44:35] Jeff_Green: I'm doing the PCI compliance thing with our "limbo" queues, and would like to avoid implementing a queue on top of Memcache. Redis has everything I need built-in... [21:48:01] (PS2) Ejegg: Make do_transaction return a PaymentTransactionResponse [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211046 (https://phabricator.wikimedia.org/T98063) [21:48:28] (CR) jenkins-bot: [V: -1] Make do_transaction return a PaymentTransactionResponse [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211046 (https://phabricator.wikimedia.org/T98063) (owner: Ejegg) [21:55:05] replacing memcache with redis? i think it's fine [21:55:17] i'm not familiar with it, but how hard can it be? [21:59:25] This would be in addition to memcache [21:59:41] (PS3) Ejegg: Make do_transaction return a PaymentTransactionResponse [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211046 (https://phabricator.wikimedia.org/T98063) [22:00:22] Jeff_Green: we wouldn't need to tweak the settings much, cos this is just ephemeral storage. [22:00:55] I do have a long-term plan to replace activemq and the contribution_tracking table with Redis, but this is not the time :) [22:48:13] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Orphan slayer reads from Memcache rather than ActiveMQ - https://phabricator.wikimedia.org/T99017#1283622 (awight) [22:51:44] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Provide access to limbo messages without knowing keys - https://phabricator.wikimedia.org/T99152#1286377 (awight) NEW a:awight [22:51:49] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Provide access to limbo messages without knowing keys - https://phabricator.wikimedia.org/T99152#1286377 (awight) [22:52:32] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Orphan slayer reads from Memcache rather than ActiveMQ - https://phabricator.wikimedia.org/T99017#1286389 (awight) [22:54:42] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Orphan slayer reads from Memcache rather than ActiveMQ - https://phabricator.wikimedia.org/T99017#1286392 (awight) [23:11:29] awight cool. why don't we prototype it in vbox or on lutetium [23:11:37] that would be a great hackathon project actually [23:12:33] Jeff_Green: I worked around just now, by birthing a bastard queue under Memcache... [23:13:11] But yeah, I still think Redis could replace our two worst SPOFs. [23:15:40] Jeff_Green: fwiw, twentyafterfour and bd808 helped me kick the mw-vagrant payments across the endzone... so we have a healthy dev vbox situation if you ever need it. [23:16:05] https://www.mediawiki.org/wiki/Fundraising_tech#Installing_the_software_stack [23:16:22] One day, we should totally collide the prod and dev puppet modules... [23:16:40] (mostly, I hope a puppet gets hurt in the process ;) [23:28:13] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Provide access to limbo messages without knowing keys - https://phabricator.wikimedia.org/T99152#1286504 (awight) [23:33:31] (PS2) Awight: cleanup [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211039 [23:33:33] (PS1) Awight: Orphan slayer reads from Memcache [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211062 (https://phabricator.wikimedia.org/T99017) [23:38:05] awight ok [23:38:31] i use a multi-host vbox cluster b/c I'm usually modelling puppet behavior across nodes [23:40:53] Fancy! [23:41:12] ya, runs good [23:50:28] Fundraising Sprint Kraftwerk, Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review: Orphan slayer reads from Memcache rather than ActiveMQ - https://phabricator.wikimedia.org/T99017#1286724 (awight) Successful local tests, for status 25 and 600, and that time buffer wo... [23:51:53] Jeff_Green: What do you use for that? Production puppet files and stock hieradata, or do you override some of the config data? [23:54:27] I git push the frack puppet repo to the vbox puppetmaster, and puppet-merge just like on boron [23:54:38] awesome [23:54:57] often I do the puppet hacking on the vb puppetmaster directly, and then work it back into git [23:55:46] there's a hook I use, some debian thing I forget, to signal to various things that it's vb and not production [23:56:15] basically if the file exists in the private repo on the puppetmaster it ends up on all the clients, and then you can read it as desired [23:56:58] Sounds like those modules might be almost appropriate to use for provisioning public payments staging boxes... [23:57:12] (PS2) Awight: Orphan slayer reads from Memcache [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/211062 (https://phabricator.wikimedia.org/T99017) [23:57:51] i've thought about putting a vbox host in frack somewhere that we can hack on [23:58:51] i talked to ryan lane long ago about trying to model frack in labs and he thought representing the heavily partitioned network was probably too hard to bother [23:59:30] in virtualbox i use an openbsd host with 8 network interfaces to behave as a router, and also to intercept mail, dns, ntp, etc as desired