[00:00:05] let's see... [00:00:22] i have never built a .deb but that might be the next step [00:00:28] i see adam has a start [00:01:09] (Merged) jenkins-bot: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [00:01:09] then figure out how to deploy the jobs repo/files and automate the crontab rebuild [00:01:17] i think jgreen has some ideas there [00:05:00] (PS5) Ejegg: Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) [00:05:14] (CR) Ejegg: "PS5: rebase" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) (owner: Ejegg) [00:08:11] cwd so that'll be part of the fr deploy script? [00:08:23] ejegg: that is what i've been thinking [00:09:21] k, so for now we go single-machine, and just add all the jobs to the crontab there + sync all the yamls [00:09:55] deploying the process-control script itself is separate from the jobs, which live in localsettings repo [00:10:06] or another repo next to that [00:10:46] yeah [00:10:55] the process control package can get installed by puppet [00:11:54] it will come with the script that generates the cron files [00:21:49] i'm gonna take off for now, later! [10:48:00] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, and 6 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3109339 (ema) >>! In T154954#31086... [13:34:59] fundraising-tech-ops: replace db1025 with new hardware running jessie - https://phabricator.wikimedia.org/T145107#3109639 (mark) [13:35:01] fundraising-tech-ops: replace lutetium with new hardware running debian/jessie - https://phabricator.wikimedia.org/T145110#3109638 (mark) [13:56:12] fundraising-tech-ops: rename backup4001 to frbackup4001 and jessie reimage - https://phabricator.wikimedia.org/T158220#3109693 (Jgreen) [15:25:44] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint Dirt Farming, and 8 others: Store and update list of currently working iDEAL banks - https://phabricator.wikimedia.org/T128692#3109953 (Pcoombe) @DStrin... [16:55:03] hi fr-tech, I'll be 10 min late to -talk [17:00:43] fr-tech: The Ancient Doctrine of Mind Over Matter: [17:00:43] I don't mind... and you don't matter. [17:00:43] -- As revealed to reporter G. Rivera by Swami Havabanana [17:00:43] -- discuss. [17:02:56] i have some chores to finish before i dive in [17:06:38] Yea, that's me too this morning. [17:26:26] k, I'm ready when you are [17:29:00] ejegg: i'm not gonna make it today, my day is going to revolve around packaging process-control and hopefully deploying it [17:31:06] (CR) Ejegg: [C: 2] Comments and todos [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/342547 (owner: Awight) [17:32:15] cwd I'd love to help with that stuff! [17:32:37] gonna read up on deb packaging [17:34:08] cool, i'm learning what to do from jgreen [19:14:30] (PS1) XenoRyet: WIP recurring Message Normalization. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/343324 (https://phabricator.wikimedia.org/T157074) [19:24:03] (CR) XenoRyet: [C: 2] Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) (owner: Ejegg) [19:27:49] (Merged) jenkins-bot: Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) (owner: Ejegg) [19:34:43] (CR) XenoRyet: [C: 2] Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) (owner: Ejegg) [19:36:18] (Merged) jenkins-bot: Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) (owner: Ejegg) [19:41:10] (CR) XenoRyet: [C: 2] Cast a wider net when searching for ct_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342893 (owner: Ejegg) [19:41:51] (Merged) jenkins-bot: Cast a wider net when searching for ct_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342893 (owner: Ejegg) [19:47:40] XenoRyet: meeting? [19:48:00] One second, my headphones just asploded [20:16:11] cwd|brb: looks like this person has the same python / stderr / stdout problem http://stackoverflow.com/questions/31833897/python-read-from-subprocess-stdout-and-stderr-separately-while-preserving-order [20:16:43] ejegg: yep, that's probably the right idea [20:22:58] every time i write python i think, why are the std lib interfaces so confusing? [20:23:18] there are a dozen ways to shell out and they are all weird [20:23:40] argparse is like a strange language within a language [20:24:10] whew [20:25:27] by the looks of that function we will not be able to capture output until the call returns [20:25:38] s/function/stack overflow post/ [20:26:11] not sure if that is a big deal [20:27:13] which solution are you looking at? [20:27:26] I'm hoping to do timestamping too [20:27:54] really wish there was an event-based way to do this! [20:28:20] nodejs is good at this stuff [20:28:22] there's this solution that has a thread polling each stream: https://github.com/rst2pdf/rst2pdf/blob/master/rst2pdf/tests/execmgr.py [20:28:53] err, not that one [20:28:59] dangit, where did that go? [20:29:04] yeah i saw that [20:29:09] but...threads seems like overkill [20:30:17] it would let us timestamp tho [20:31:01] I guess interleaved polling would too [20:40:11] cwd threads are looking like the only way to get around readline() blocking on the end of the output [20:40:26] lemme try something [20:42:35] well one thing is, we have a current theory that we should fail on anything on stderr [20:42:51] but i'm wondering how much we'll actually want that [20:43:10] if it turns out we only want to fail on non-zero exit we could just pipe stderr to stdout [20:43:19] true [20:43:26] be a heckuva lot simpler [20:43:36] timestamping still would be nice [20:43:46] also, writing to the output file in realtime [20:43:53] nice to know how far along a job is! [20:44:21] yeah definitely would [20:44:32] worried that python is the wrong tool if those things are deal breakers [20:44:45] i think the thread will be ok [20:55:32] ejegg: maybe we could just try communicate() for the first edition? [20:56:02] k, if that's not a hard blocker [20:56:23] well, there are only a couple jobs that take a long time to run [20:56:38] silverpop... [20:56:50] yeah [20:56:52] and audit parsing [20:57:05] but i don't find myself at 2 in the morning watching their streaming output very often [20:57:24] k, so what else is missing? [20:58:05] ejegg: tyler suggests looking into the logging module [20:58:12] k [20:58:17] might be a better path to what we want [20:58:40] other than that, we need to make process control send email [20:59:38] ah, word. I guess we're copying stuff in from tools rather than depending on the whole mess [21:00:02] which mess? [21:00:44] rather than having process_control depend on all of tools [21:03:35] well, logging module would give us timestamps and threadsafe writing to file! [21:05:30] yeah, seems like we can replace the open() pretty effectively [21:05:34] and not reinvent the wheel there [21:05:58] still need to interleave the streams [21:06:05] ok, but back to failmail. [21:06:11] think we want that global config now [21:06:36] though different jobs should be able to override destination [21:07:32] can't it just all go to fr-tech? [21:07:46] ehh, sure, for MVP [21:09:11] ejegg: tyler has a lot of good suggestions about this code, i want to find a way to annotate [21:09:28] that dept has a lot of python know how we are missing [21:09:35] cwd put up a patch with comments? [21:09:44] yeah imma do something like that [21:30:44] (PS1) Ejegg: Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 [21:31:25] (CR) jerkins-bot: [V: -1] Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 (owner: Ejegg) [21:44:25] Fundraising-Backlog: error opening Annual Report in Safari Version 9.1.1 (10601.6.17) - https://phabricator.wikimedia.org/T159389#3111149 (MBeat33) I emailed the screenshot Sam & Juliet in Comms; we'll see if they know who would own this. [21:44:52] (PS1) Cdentinger: Comments from thcipriani [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343389 [21:45:14] (CR) jerkins-bot: [V: -1] Comments from thcipriani [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343389 (owner: Cdentinger) [21:47:31] (PS2) Ejegg: Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 [21:48:25] (CR) jerkins-bot: [V: -1] Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 (owner: Ejegg) [21:55:43] (PS3) Ejegg: Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 [21:56:20] (CR) jerkins-bot: [V: -1] Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 (owner: Ejegg) [21:57:44] (PS4) Ejegg: Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 [22:02:55] aw snap, it's pouring out and i didn't bring my raincoat [22:03:20] i still have the little poncho from wikimania in my bag [22:03:35] nice. [22:03:39] i'd upload it if i could [22:03:49] at least I have a plastic bag I can wrap around compy [22:04:09] PS4 of that should get us basic failmailing [22:07:13] excellent [22:07:33] (PS5) Ejegg: Send failmail [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343365 [22:13:29] https://github.com/etsy/phan [22:14:07] that looks handy [22:14:53] oughtta be straightforward to add to tests [22:15:37] might be better than nothing [22:25:36] nice, rain sounds like it's almost stopped before the library closes [22:26:47] relocating & grabbing grub, back in a bit [23:16:56] dang, downpour is back! [23:20:10] cwd any other must-haves for an initial run? [23:26:02] hmmm [23:26:13] * cwd considers [23:29:37] ejegg: there are a couple smallish things from the ops perspective to do with making deployable by frtech [23:29:46] some puppet stuff [23:30:16] but i think we are very close [23:30:27] awesome! [23:30:48] yeah i am encouraged with how this has gone so far [23:31:42] i am thinking a new repo for the config files akin to localsettings [23:31:52] with a deploy time tweeze [23:32:34] my feeling is if twiddling the queues is somewhat labor intensive to begin that is ok cause we can script later [23:33:41] yep, sounds fine [23:34:36] tyler had a few good suggestions, including using context object for the lock file [23:34:43] that is one area i have not tested [23:36:45] i'm not sure how that would work [23:38:07] i uses a file on the filesystem, same idea really just python-y [23:38:43] huh, ok. i'll read up [23:39:11] ejegg: here is an example https://github.com/wikimedia/scap/blob/master/scap/main.py#L56 [23:49:11] ah, k, so it just releases the lock after you leave the context [23:54:41] (PS2) Ejegg: Work around Memcache not expiring some things [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/342266 (https://phabricator.wikimedia.org/T128692) [23:58:29] (PS1) Ejegg: Global config [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343417 [23:58:55] (CR) jerkins-bot: [V: -1] Global config [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343417 (owner: Ejegg)