[00:00:53] right - so it would be a good thing.... [00:02:45] I'd love to move towards more "update"y behavior everywhere [00:05:12] In the benevity case, do we want to add the email as a new secondary, or replace/displace the primary? [00:06:44] hmm - the code already is setting the location to work & I think passing is_primary = 0 [00:06:58] that's for address but I think the same makes sense for email [00:07:38] I wouldn't trust that the precedent makes sense... [00:07:53] But it does sound reasonable [00:13:15] ejegg|afk: I'm poking at your signal handler tests for a few [00:14:56] Fundraising Sprint Far Beer, Fundraising-Backlog: Benevity import not updating emails for existing contacts - https://phabricator.wikimedia.org/T161666#3138993 (Eileenmcnaughton) [00:15:58] awight: i can't seem to deduce at what point this package will suddenly be picked up by puppet [00:16:14] i don't want to sudo anything i'm not 100% on [00:16:32] (PS1) Eileen: Fix Benevity to update existing contact with email. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/345268 (https://phabricator.wikimedia.org/T161666) [00:32:58] Fundraising-Backlog, fundraising-tech-ops: [Epic] Basic process-control good enough to run all CRM jobs - https://phabricator.wikimedia.org/T161569#3139022 (awight) [00:34:43] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops, Performance: Run CRM on HHVM - https://phabricator.wikimedia.org/T91896#3139025 (awight) stalled>declined [00:51:28] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Fundraising is running a duplicate cron job - https://phabricator.wikimedia.org/T161668#3139056 (awight) [00:53:13] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Don't chain donation queue consumer to thank_you job - https://phabricator.wikimedia.org/T161669#3139070 (awight) [00:55:07] Fundraising Sprint Far Beer, Fundraising-Backlog: process-control will need to run some jobs sequentially - https://phabricator.wikimedia.org/T161035#3139084 (awight) [01:05:59] (PS1) Awight: Bump version: 0.0.2 [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345273 [01:06:04] cwd: et voila :) [01:09:15] awight: hilariously the dir is called process-control_1.0.0 [01:09:35] perhaps the debian convention is a mirror image of the version number [01:09:39] nooo [01:09:43] That's gotta be a random typo [01:10:03] the slander deb is the same way [01:11:02] definitely not a convention, the other upstream pkgs are numbered correctly [01:18:29] awight: well i'm guessing build_package.sh generates all the files in the package dir [01:18:38] you think? [01:19:12] so if i just rename all this stuff 0.0.2 and rebuild [01:19:17] what will happen to the old package? [01:19:36] if it is 1.0.0 i wonder if it will favor that one? [01:20:17] ugh yeah you're right [01:20:28] no [01:20:37] cos the deb packaging had the right number [01:21:17] process-control: [01:21:19] Installed: 1.0.0.1~precise [01:21:22] now i understand even less [01:21:39] ayiiii [01:21:51] yup. That happened. [01:22:26] I'd prefer if you could ensure=>absent, but the 1.0.0 will still be in the custom repo, huh? [01:22:30] did the build script add the thousandths place minor version number? [01:22:36] haha [01:22:46] hey this is for our own safety [01:23:30] yeah i'm not sure how to wipe the old version out [01:23:37] (PS9) Awight: WIP still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [01:23:45] (CR) jerkins-bot: [V: -1] WIP still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [01:24:02] I'm loathe to jump to 1.0.1 just b/c I give major version "1" a bit more respect than that [01:24:07] but uh... [01:24:09] here goes [01:24:18] heh [01:24:22] 1.0.1.2 [01:25:41] (PS2) Awight: Bump version: 1.0.1-rc1 [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345273 [01:25:49] I don't wanna talk about it :p [01:31:31] (PS10) Awight: WIP still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [01:31:39] (CR) jerkins-bot: [V: -1] WIP still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [01:31:44] ejegg|afk: ^ That should be testing the... d'oh [01:34:28] (PS6) Awight: Only speak in slugs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345094 [01:34:30] (PS11) Awight: WIP still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [01:35:21] ejegg|afk: I flipped the threads around and it seems to be working, anyway... [01:37:03] Fundraising Sprint Far Beer, Fundraising-Backlog, Patch-For-Review: process-control script to list all jobs and statuses - https://phabricator.wikimedia.org/T161584#3139124 (awight) IMO this is normal priority, and doesn't need to be part of our MVP. [01:37:15] Fundraising Sprint Far Beer, Fundraising-Backlog, Patch-For-Review: process-control script to list all jobs and statuses - https://phabricator.wikimedia.org/T161584#3139126 (awight) p:High>Normal [01:37:55] * awight breaks through 8-hour ribbon [01:38:00] See ya tomorrow! [01:38:30] awight: well if you are not going to be around i'm not going to push my luck building this thing [01:38:51] i have a checklist of steps but it would be prudent to check with jeff first [01:39:28] i haven't even unpacked yet so i'm going to go ahead and wait [01:39:34] later! [15:51:12] (CR) Ejegg: [C: 2] Only speak in slugs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345094 (owner: Awight) [15:52:01] (Merged) jenkins-bot: Only speak in slugs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345094 (owner: Awight) [15:52:41] (PS2) Ejegg: Nicer log capture [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345260 (owner: Awight) [15:53:42] (CR) Ejegg: [C: 2] Nicer log capture [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345260 (owner: Awight) [15:55:28] (Merged) jenkins-bot: Nicer log capture [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345260 (owner: Awight) [15:56:59] (PS2) Ejegg: mini-lock login encapsulation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345265 (owner: Awight) [15:57:20] (CR) Ejegg: [C: 2] mini-lock login encapsulation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345265 (owner: Awight) [16:01:48] (Merged) jenkins-bot: mini-lock login encapsulation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345265 (owner: Awight) [16:03:30] (PS12) Ejegg: WIP still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 [16:03:58] (CR) Ejegg: "PS12: rebase" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [16:09:50] (CR) jerkins-bot: [V: -1] WIP still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [16:37:44] (PS13) Ejegg: Still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 [16:38:38] cwd that one's got both my and awight's fingerprints on it now ^^^^ [16:38:44] want to give it a look? [16:40:01] Also curious if you or Jeff_Green wants to weigh in on the mystery of kill() waiting for sleep to finish when it's a sub-sub-process spawned by sh -c [16:40:40] sure i'll take a look [16:40:45] thanks! [16:40:54] do you think this is a necessary mvp feature? [16:41:23] this one yeah, so we don't ditch all output if we have to kill things [16:41:56] it really seems to me like killing things is an edge case [16:42:28] but like awight says, one we're way more likely to need while testing [16:42:30] how do you intend to send signals to the main process? [16:42:49] cwd while testing, I guess you can do it [16:42:57] eventually, with the --kil-job argument [16:43:53] assuming we can run run-job as jenkins [16:44:03] you can do that right now [16:44:10] cool [16:44:12] ejegg that's interesting re. sleep, breaking it into shorter sleeps actually works? [16:44:19] Jeff_Green: yep! [16:44:42] does it wait for the first 2-second sleep? [16:45:10] with 'sleep 2', it was waiting 2 seconds, then killing the subprocess before the 'echo goodbye' [16:45:59] Jeff_Green: it DOESN'T wait for sleep() to finish if that's the subprocess directly started by python's subprocess.Popen [16:46:09] only when Popen starts sh -c [16:46:15] and sh -c starts sleep [16:46:42] maybe it's effectively killing by process ID? [16:46:56] i.e. it kills "sh -c" which doesn't in turn signal to its subprocess? [16:47:14] seems to be the case [16:48:00] out of curiosity why do the sh at all, instead of just calling sleep directly? [16:48:07] but on the command line, if I [16:48:21] sh -c "sleep 1000" & [16:48:27] then kill the sh process [16:48:33] it exits immediately [16:49:14] hmm, lemme try killing run-job in a non-test environment [16:50:10] this feature seems to have a lot of subtleties [16:52:42] Maybe i should just work on streaming stdout [16:52:54] would make this moot [16:54:18] moving to a non-quiet area of the library for -talk [16:54:19] Fundraising-Backlog, Continuous-Integration-Config, Patch-For-Review: symfony-polyfill54 is breaking CI - https://phabricator.wikimedia.org/T143598#2573044 (Krinkle) Looks like this is still failing on recent commits, . [17:00:44] fr-tech: I love Mickey Mouse more than any woman I've ever known. [17:00:44] -- Walt Disney [17:00:44] -- discuss. [17:02:21] i am in the middle of trying to package p-c but i'll put it on pause if there is stuff to -talk about [17:02:36] cwd nah, that takes precedence [17:02:47] any asks from Scrum of Scrums, btw? [17:03:11] hrm, don't think so ty [17:03:15] k [17:04:09] (CR) XenoRyet: "Wait, I confused myself. Don't merge this yet, another PS incoming." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/345189 (https://phabricator.wikimedia.org/T161121) (owner: XenoRyet) [17:04:30] @seen Jeff_Green a few minutes ago? [17:04:55] awight: missed him by 2 minutes [17:04:56] cwd: I was secretly still harboring wishes for a v0.0.2 [17:05:06] cool, wm-bot tells all [17:05:41] awight: we can ask jeff about that [17:05:53] probably just has to do with nuking the current package from the local repo? [17:05:54] awight: I re-complexified that preserve-output-on-kill patch [17:06:21] and discovered that we don't seem to kill sub-sub-processes [17:06:24] cwd: yeah I think apt purge and clear out of the private repo would do the trick [17:08:04] ejegg: harr I think we can't take responsibility for bad child management? [17:08:10] bad grandchild [17:09:00] rightright [17:09:13] ohey thanks for the CR [17:09:28] thanks for the code! [17:10:36] I'm reading your comment, very strange stuff [17:11:18] I don't understand why the test was passing as I had written it, then [17:11:38] awight: it kills sleep if that's the process directly launched by python [17:12:01] but when I wanted to also check the output, I had to concatenate the echo and sleep with sh -c [17:13:39] (CR) Awight: "--pedantic mode enabled" (4 comments) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [17:13:50] IMO we shouldn't test output [17:14:06] but we *should* test the grandchild bug with a second test. More in the comments... [17:14:55] awight: but the whole point of catching the signal is to preserve output [17:15:22] seems silly just to test that we can kill a thing [17:15:22] * awight sheepens [17:15:28] ok that's so true [17:15:55] coming back from the run_job in the same thread is a good first step, but yeah you're right [17:18:48] (PS5) XenoRyet: Mark refunds with correct gateway [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/345189 (https://phabricator.wikimedia.org/T161121) [17:19:42] ejegg: Supposedly the child processes will inherit a process group ID. Shall we target that/ [17:19:45] ? [17:19:56] oh? [17:20:34] background reading? [17:21:30] Nothing authoritative yet [17:22:01] http://unix.stackexchange.com/questions/149741/why-is-sigint-not-propagated-to-child-process-when-sent-to-its-parent-process [17:22:14] > The POSIX specification says that when INTR is received, it should send a SIGINT to the foreground process group of that terminal [17:22:34] so ^C does kill -INT -PGID [17:22:59] I think PGID is just the PID of the parent tho? Not sure when that tree is rooted [17:23:00] oh, interesting [17:23:33] would be strange if sigint was more immediately effictive than sigkill! [17:24:16] I don't think the choice of sig matters as much, but it's the negative PID term which is interpreted to mean, sig the process group [17:25:13] scrum of scrums coming right up - any news or requests to pass along fr-tech? [17:25:24] Negative [17:25:54] AndyRussG: anything you need from other teams? [17:26:52] fyi, os.getpgid [17:27:00] I'm still learning how a pgid gets set though [17:27:12] os.setpgid(pid, pgrp)¶ [17:29:54] fun: pstree -pg [17:32:48] search term: "process group leader" [17:33:16] sounds like management-speak [17:34:30] whoa. You can't setpgid after a child has exec'ed... https://lists.gt.net/python/python/45738 [17:34:34] Fundraising-Backlog: SV Thank You page to Facebook error - https://phabricator.wikimedia.org/T161727#3141139 (MBeat33) [17:34:36] I don't really understand that [17:34:47] when would you then? [17:34:51] however, starting to think we can set the p-c parent as a process group leader [17:34:59] In fact, it's behaving that way by default [17:35:14] │ │ ├─bash(25843,25843)───python(26255,26255)───sh(29356,26255)───sleep(29357,26255+ [17:35:29] Running this from an interactive shell: [17:35:32] subprocess.call(["/bin/sh", "-c", "/bin/sleep 100"]) [17:36:21] So I'm thinking, --kill-job throws a signal at the process group under the PID already stored in .lock [17:36:41] why is it a process group? [17:36:41] the p-c parent just withstands the signal, and doesn't need to propagate it. (?)) [17:36:45] if php shells out other things? [17:36:57] sure seems like we are reinventing linux in python [17:37:00] drush is a few levels deep [17:37:10] Good point, anything else is probably free to jack the PGID [17:38:14] So... back to the use cases and trying to shave the yak [17:38:49] ...or we could just solve streaming stdout [17:38:50] IMO there are two things we're trying to do, * don't lose logs ever, and * provide a killing facility [17:39:16] I've been thinking that the first thing might not actually be a problem [17:39:32] if the child dies naturally, signal or not, then we correctly capture the logs already. [17:39:55] If we're just working on the second thing, killing facility, let's postpone until after the MVP [17:39:58] ? [17:41:01] I can write a test to kill the job command subprocess and prove whether that's true or not, if it helps [17:41:07] if ops has to send the signal anyway i don't see a compelling reason for all the BDUF [17:41:19] no, devs would send the signal [17:41:26] but we might not need it for MVP [17:41:29] you'd have to own the process [17:41:36] afaik [17:41:43] we'd be running run-job suid [17:41:50] so it has perms [17:42:02] gotcha [17:44:19] every single time i have to run puppet in VB i have to regenerate the certs at least 10 times before it randomly works [17:44:35] yeesh [17:44:42] I've been having a great time with that, for a side project. [17:45:19] I have a vagrant box, and tickle it with the ansible provisioner. ansible copies puppet in and calls puppet apply [17:45:29] But I guess you're all puppetmaster/agent [17:45:39] yeah [17:46:10] ejegg|scrummy: pls take a scroll when you're done with SoS ^ [17:46:50] cwd: lmk what I can do to help package. I'm fine with the 1.0.1 hack or going back to the 0.0.2 patch [17:47:52] awight: well i got the low down on it and there are a dozen steps i didn't deduce last night [17:47:58] haha good thing [17:47:59] but i am 100% blocked on VB puppet [17:48:03] d'oh [17:48:12] for the millionth time [17:48:13] * awight roughs up a puppet on your behalf [17:49:48] i can do the prod repackage if we're ready [17:49:57] cwd: I'm sure you tried all this, http://stackoverflow.com/a/16909050 [17:50:01] grr. sorry about all the auditcronspam [17:50:18] Jeff_Green: sure, I'd be content with any recent master checkout [17:50:26] cool! [17:50:49] ejegg: cool=>we can drop kill for now? [17:50:52] right, so we can just kill the child process for testing [17:51:12] grabbing some lunch, back soon [17:51:14] yeah, and my guess is that we'll be getting the output correctly [17:51:17] kk [17:51:29] awight: yeah i tried all of the 100 or so answers i found for this catch-all error [17:51:46] what has worked so far is regenerating the certs over and over and over again and eventually it works [17:52:23] (CR) Awight: "I don't know why I said "login", nor why I punctuated as I did. tired." [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345265 (owner: Awight) [17:54:13] Fundraising-Backlog, Continuous-Integration-Config, Patch-For-Review: symfony-polyfill54 is breaking CI - https://phabricator.wikimedia.org/T143598#3141208 (awight) @Krinkle Thanks for pointing out that the workaround no longer works :-) Looks like the php5lint job should be disabled for that repo. [17:55:06] * Jeff_Green is puzzled about all whether the paypal audit cronspam means it's still actively failing...or it's all cascading from the brief mysql restart [17:55:06] cwd: you tried --server ? [17:55:14] I'll check [17:56:06] damn emails have PII but no timestamps. useless [17:56:26] Pretty sure I did that. [17:56:59] The failmails are telling an active failure story, that it can't connect to MySQL, but yeah I don't know if we enqueued a million or what [17:58:19] !log disabling PayPal audit parser [17:58:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:59:00] or uh, s/disabled/stopped/ [17:59:31] I'll let the failstream bleed out, then restart the job [18:01:18] awight: kthx [18:02:48] Looks like those were all queued at once, though [18:04:53] oh hey, speaking of killing jobs... [18:05:39] well, isn't this turning a job off? [18:05:44] vs killing an instance [18:06:20] nah, this is killing an instance [18:06:27] This would be "immediately stop all jobs with "db" tag" [18:06:44] it's sending an email per line in the audit file, till we kill it [18:07:03] gotcha [18:07:22] would kill(1) have been insufficient? [18:07:30] so... it needs its own fail logic [18:07:55] On the other hand, it sent all its 10,000 emails in a short period so we might have missed the chance to kill anyway [18:08:12] and yeah, what it really needs is for failmail to turn around and say ENOF [18:08:24] cwd yeah, as long as ops is around to kill the subprocess [18:10:37] we have this sort of long running job to avoid excessive round trips to the db, right? [18:11:37] mm? we make excessive round trips like nobody's business [18:12:07] does this job send emails or does it schedule another job to send them? [18:12:08] That job is parsing a big audit file, so I don't see any natural way to split the job into anything smaller. [18:12:15] no emails. [18:12:23] nor does it schedule [18:12:37] the audit parser writes normalized payment messages to the donations queue [18:12:38] ejegg said above it's sending one email per line... [18:12:45] ah those are failmails [18:12:46] ah [18:12:51] gotcha [18:13:06] so sorry yeah it's producing those and putting them on the mailq directly [18:13:39] did you press the red button in jenkins? why is it still failmailing? [18:14:05] I did kill it in jenkins [18:14:09] but mailq's are queues [18:14:19] gotcha [18:14:19] oh hey I can peek [18:14:32] huh, maybe I can't [18:22:42] This does not include fundraising servers right? https://ganglia.wikimedia.org/latest/graph_all_periods.php?title=mail+delivery&vl=&x=&n=&hreg[]=.&mreg[]=exim.%2B>ype=line&glegend=show&aggregate=1 [18:24:57] Nemo_bis: yes/no. incoming fundraising mail goes via the main MXs, outgoing fundraising mail does not [18:26:08] actually, even that's not quite correct. some of the outgoing mail goes via the production MXs too... [18:27:35] I feel less guilty for being confused about the status :) [18:28:14] status: "its complicated" :-P [18:28:31] I suppose we only care about sending fundraising campaign emails from a different IP than the MediaWiki email notifications [18:28:35] ;) [18:29:58] ha, even that is complicated [18:30:14] Nemo_bis: cool, thanks--I hadn't noticed the mail monitoring, that's fun [18:31:06] It still bugs me when ganglia axes have no labels [18:31:35] 1k mail delivery failures per hour? per second? Per graph line? [18:32:11] probably per sample interval [18:33:08] ... which also isn't indicated [18:34:54] awight: agreed, terrible. take heart though, ganglia is at the chopping block [18:35:35] It's getting to be like the Terror around here! [18:36:43] huh, that doesn't disambiguate well. Anyway, that one: https://en.wikipedia.org/wiki/Reign_of_Terror [18:41:38] process-control feature request: Show which scheduled jobs failed recently and suggest re-run. Bulk re-run (needs manual confirmation). [19:02:12] awight: how about notifying via IRC bot when a job fails? [19:02:34] you can do that now easily if you manipulate the syslog ident string [19:11:10] oh neat [19:11:41] are you suggesting that a flood of billions of unstoppable, redundant emails is not useful? [19:12:29] haha [19:13:48] want some fillet-o-fail with that lunch? [19:14:15] How about we just feed into icinga? [19:15:09] !log pick at paypal scab: re-run audit parser [19:15:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:16:07] !log re-run today's ingenico audit job [19:16:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:17:41] Jeff_Green: Should we reconsider piping errors into icinga? [19:19:18] awight: maybe for a later revision [19:20:45] for sure, these would also come straight out of application logic [19:20:50] like, replace failmail [19:21:13] then we pipe icinga to IRC [19:21:20] and email [19:26:15] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 2 others: [Tech Debt] Consolidate DonationInterface validation - https://phabricator.wikimedia.org/T160385#3141520 (awight) Aww, this work is at risk of going stale.... [19:26:50] cwd: Once you've deployed the deb, lmk if I can abandon https://gerrit.wikimedia.org/r/#/c/345273/ [19:29:04] awight: are we gonna go for 0.0.2 ? [19:32:37] cwd: Do we need any version bump on my side? [19:33:44] i'm not sure [19:33:55] i just got it to build in vb [19:34:01] ejegg: Hey, did you see this comment about sequential jobs? > Donations queue consume -> Thank you mail send (shouldn't do this) [19:34:09] overjoy! [19:36:33] Fundraising-Backlog: process-control: change failmail address per job - https://phabricator.wikimedia.org/T161741#3141566 (awight) [19:49:32] (PS1) Ejegg: WIP stream subprocess stdout to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 [19:49:52] awight: looking! [19:50:12] also, stab at streaming ^^^ (will fail tests) [19:52:24] oh dang, he's going for it [19:52:34] :) [19:52:46] not trying to interleave stderr [19:53:10] oh hey, but using the logging facilities for output should make that easy! [19:55:46] Fundraising-Backlog: process-control: change failmail address per job - https://phabricator.wikimedia.org/T161741#3141566 (Ejegg) We can already do this in config, since it's in the default_job_config section [19:56:05] (CR) jerkins-bot: [V: -1] WIP stream subprocess stdout to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (owner: Ejegg) [19:56:40] (CR) Awight: [C: 1] "That was *fancy!" (2 comments) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (owner: Ejegg) [19:57:38] ejegg: Anyway, I made T161669, if you agree that the TY job schedule is inexplicably regressed [19:57:38] T161669: Don't chain donation queue consumer to thank_you job - https://phabricator.wikimedia.org/T161669 [19:57:45] way less fancy than I thought it would have to be, thanks to thcipriani's suggestion to use logging for output [19:58:06] That was an amazing touch [19:58:19] Does that even give the lines templated timestamps?! [19:58:29] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Don't chain donation queue consumer to thank_you job - https://phabricator.wikimedia.org/T161669#3139070 (Ejegg) Ah, I thought we were trying to avoid deadlocks by alternating big reads and writes [19:58:36] timestamps indeed! [20:03:21] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Don't chain donation queue consumer to thank_you job - https://phabricator.wikimedia.org/T161669#3141655 (awight) oh nice! Feel free to invalidate this task, seems like a good idea when you put it like that. [20:07:14] ejegg: btw, I like the config vs global_config paradigm better than SmashPig's beatiful mush. You? [20:07:27] yep, definitely! [20:07:40] rad. [20:07:44] It's more explicit. [20:07:56] and leaves open the possibility of using two different job configs in the same process for e.g. listing [20:08:24] ah yeah [20:08:26] or in the DI world, picking a gateway based on params [20:08:45] yesabsolutely [20:10:10] (PS1) Awight: Run commands in sequence [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345429 (https://phabricator.wikimedia.org/T161035) [20:19:56] (PS2) Awight: [WIP] Run commands in sequence [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345429 (https://phabricator.wikimedia.org/T161035) [20:21:07] (PS2) Ejegg: WIP stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 [20:28:51] (CR) jerkins-bot: [V: -1] [WIP] Run commands in sequence [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345429 (https://phabricator.wikimedia.org/T161035) (owner: Awight) [20:30:23] (CR) jerkins-bot: [V: -1] WIP stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (owner: Ejegg) [20:44:13] (PS3) Awight: Run commands in sequence [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345429 (https://phabricator.wikimedia.org/T161035) [20:44:44] (CR) Ejegg: WIP stream subprocess stdout and stderr to log file (2 comments) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (owner: Ejegg) [20:46:10] ejegg: The command sequence patch is looking good but I'm going to rebase it after your streaming work. There's some stdout capture I don't want to clean up twice... [20:49:24] (PS4) Awight: [WIP] Run commands in sequence [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345429 (https://phabricator.wikimedia.org/T161035) [20:50:31] awight: k, one more PS about to land for streaming [20:51:22] * awight leaps to avoid the flying stream of PS [20:51:41] (PS3) Ejegg: WIP stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 [20:52:20] oh man, that sounded dirty [20:52:39] do clue what you mean :p [20:52:41] *no [20:52:44] * ejegg plays some R. Kelly [20:53:34] * ejegg reviews MI-5's Trump dossier [20:53:54] *shudder [20:53:54] still need to fix the tests there [21:01:31] (CR) jerkins-bot: [V: -1] [WIP] Run commands in sequence [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345429 (https://phabricator.wikimedia.org/T161035) (owner: Awight) [21:01:53] (CR) jerkins-bot: [V: -1] WIP stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (owner: Ejegg) [21:17:36] uggg, where the heck do I call nose.config.configure() to change the logging-filter before tests are collected? [21:17:42] d'argh [21:18:01] setup_module perhaps? [21:18:17] Is the problem that the global logging filter is applied to the logfile stream? [21:19:30] ejegg: ohey. You can pass Popen a file object in lieu of .PIPE [21:19:46] Maybe we can wire it up without Queue.Queue? [21:19:59] if we provide file objects that log-wrap the output log [21:20:23] or just pass the logfile object for the MVP [21:21:21] hmmm [21:21:49] Fundraising-Backlog, Continuous-Integration-Config, Patch-For-Review: symfony-polyfill54 is breaking CI on wikimedia/fundraising/crm/vendor - https://phabricator.wikimedia.org/T143598#3142071 (hashar) [21:21:56] awight: yeah, the tests were capturing the 'logging' to the output file [21:22:28] aha [21:22:41] you can run nosetests --logging-filter=root [21:22:52] (still failing, need to strip timestamps) [21:23:46] (CR) Awight: WIP stream subprocess stdout and stderr to log file (4 comments) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (owner: Ejegg) [21:24:17] ejegg: I don't think that'll do it, cos p-c isn't using the root logger [21:25:58] Fundraising-Backlog, Continuous-Integration-Config, Patch-For-Review: symfony-polyfill54 is breaking CI on wikimedia/fundraising/crm/vendor - https://phabricator.wikimedia.org/T143598#3142093 (hashar) That looks like the same problem we had on mediawiki/vendor . composer 1.1 generates a file meant fo... [21:26:29] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: EPIC: Add key silverpop data into Civi - https://phabricator.wikimedia.org/T161757#3142095 (DStrine) [21:28:04] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: retrieve the text/ html and statistics data for mail we have sent - https://phabricator.wikimedia.org/T161758#3142109 (DStrine) [21:28:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: EPIC: Add key silverpop data into Civi - https://phabricator.wikimedia.org/T161757#3142121 (DStrine) [21:28:19] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: retrieve the text/ html and statistics data for mail we have sent - https://phabricator.wikimedia.org/T161758#3142122 (DStrine) [21:28:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: EPIC: Add key silverpop data into Civi - https://phabricator.wikimedia.org/T161757#3142095 (DStrine) [21:28:48] (PS4) Ejegg: WIP stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 [21:29:05] awight: I think the root logger is the default [21:29:21] all those captured log lines seem to be tagged with "root" [21:29:30] (CR) jerkins-bot: [V: -1] WIP stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (owner: Ejegg) [21:29:42] huh, true but we're doing config.log = logging.getLogger("process-control") [21:29:46] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: retrieve the contacts who have signed up for email without going through our donor system - https://phabricator.wikimedia.org/T161759#3142127 (DStrine) [21:30:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, FR-Email: Reflect all unsubscribes via Silverpop in CiviCRM - https://phabricator.wikimedia.org/T161760#3142140 (DStrine) [21:31:10] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, FR-Email: EPIC: Add key silverpop data into Civi - https://phabricator.wikimedia.org/T161757#3142156 (DStrine) [21:32:18] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, FR-Email: Reflect all undeliverables via Silverpop in CiviCRM - https://phabricator.wikimedia.org/T161761#3142157 (DStrine) [21:33:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, FR-Email: retrieve lists of contacts who received a particular mailing - https://phabricator.wikimedia.org/T161762#3142174 (DStrine) [21:34:20] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, FR-Email: EPIC: Add key silverpop data into Civi - https://phabricator.wikimedia.org/T161757#3142095 (DStrine) [21:35:13] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: retrieve the text/ html and statistics data for mail we have sent - https://phabricator.wikimedia.org/T161758#3142193 (DStrine) [21:36:54] (PS5) Ejegg: Stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) [21:37:16] (CR) jerkins-bot: [V: -1] Stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) (owner: Ejegg) [21:39:19] (PS6) Ejegg: Stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) [21:39:48] (CR) jerkins-bot: [V: -1] Stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) (owner: Ejegg) [21:45:26] (PS7) Ejegg: Stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) [21:46:35] (Abandoned) Ejegg: Still output when killed [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345082 (owner: Ejegg) [21:49:24] awight: re doing without Queues - I guess we can just append to an array of lines as long as we only want to read them after the process is dead and the output threads have exited [21:50:38] (CR) Awight: Stream subprocess stdout and stderr to log file (1 comment) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) (owner: Ejegg) [21:50:55] ejegg: Your note in the commit message makes me think, we don't need to buffer either stream. [21:51:19] just a flag to say something came out stderr? [21:51:26] exactly [21:51:39] and all failmails include the logfile path. [21:51:59] k, and it'll be in the output file, so no need to stderr outselves [21:52:10] hrm, I don't even know that it's helpful to flag has_stderr. [21:52:31] syslog-capable commands won't emit stderr [21:52:47] hmm [21:53:00] so I guess we're only catching some small minority of print 2> cruft [21:53:04] ooh, also, are random drupal warnings barfed out to stderr? [21:53:04] and probably a bunch of spam [21:53:27] that's what I'm thinking, like random sh*t we shouldn't pay attention to [21:54:05] Having a record of whether a message came over stderr is real nice, but we can probably leave it at that. [21:54:21] Anyway, the tests are passing--should we just merge what you have now? [21:54:42] I'm nervous about Queue.Queue cos I think we're sending the data around through some unnecessary IPC [21:54:49] but nothing I can't ignore for now :-) [21:57:00] awight: sure, let's start with this and see what the bloat/slowdown is? [21:57:38] I'll prep a paredown patch just in case [22:00:17] awight: oh hey, !override_config.has("logging") [22:00:25] is why tests see "root" [22:01:03] ooh nice [22:03:06] (CR) Awight: [C: 2] "Merging this as a savepoint, it gets us most of what we need." [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) (owner: Ejegg) [22:03:38] (Merged) jenkins-bot: Stream subprocess stdout and stderr to log file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345424 (https://phabricator.wikimedia.org/T161571) (owner: Ejegg) [22:08:31] Fundraising Sprint Far Beer, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: process-control streams to log - https://phabricator.wikimedia.org/T161571#3135678 (Ejegg) a:Ejegg [22:10:16] Fundraising-Backlog: Send interview answers to LegoKTM - https://phabricator.wikimedia.org/T161770#3142349 (Ejegg) [22:11:11] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Don't chain donation queue consumer to thank_you job - https://phabricator.wikimedia.org/T161669#3142362 (Ejegg) Open>Invalid [22:35:51] Fundraising-Backlog: process-control: change failmail address per job - https://phabricator.wikimedia.org/T161741#3142390 (Ejegg) Open>Resolved [22:38:58] Fundraising Sprint Far Beer, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: CentralNotice: for JS linting, switch from jshint+jscs to eslint - https://phabricator.wikimedia.org/T161142#3122784 (DStrine) [22:39:08] Fundraising Sprint Far Beer, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review, Unplanned-Sprint-Work: CentralNotice: for JS linting, switch from jshint+jscs to eslint - https://phabricator.wikimedia.org/T161142#3122784 (DStrine) [22:40:29] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising-Backlog, Patch-For-Review: Upgrade to drupal 7.5 - https://phabricator.wikimedia.org/T143268#3142404 (Eileenmcnaughton) Open>Resolved Did this a while back... [22:42:30] Fundraising-Backlog: process-control should have a mode to display all the things it's currently running - https://phabricator.wikimedia.org/T160700#3142411 (Ejegg) @awight this is solved by --list-jobs showing the status, right? [22:43:17] Fundraising-Backlog: process-control should have a mode to display all the things it's currently running - https://phabricator.wikimedia.org/T160700#3142413 (awight) Open>Resolved a:awight I think so! [22:43:51] Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal, Epic: Paypay Express checkout internal test - https://phabricator.wikimedia.org/T131815#3142417 (DStrine) Open>Resolved [22:43:53] Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal, Epic: [epic] PayPal upgrade - https://phabricator.wikimedia.org/T87621#3142418 (DStrine) [22:47:48] awight: sprint G name input? [22:48:38] Fundraising Sprint G 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: retrieve the text/ html and statistics data for mail we have sent - https://phabricator.wikimedia.org/T161758#3142420 (DStrine) [22:48:38] Gadget belt [22:48:41] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, Patch-For-Review: Benevity import not updating emails for existing contacts - https://phabricator.wikimedia.org/T161666#3142421 (DStrine) [22:48:43] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, Patch-For-Review: process-control script to list all jobs and statuses - https://phabricator.wikimedia.org/T161584#3142422 (DStrine) [22:48:46] Fundraising Sprint G 2017, Fundraising-Backlog, FR-Amazon: NULL referrers - https://phabricator.wikimedia.org/T161539#3142424 (DStrine) [22:48:48] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: process-control streams to log - https://phabricator.wikimedia.org/T161571#3142423 (DStrine) [22:48:50] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, Patch-For-Review: process-control will need to run some jobs sequentially - https://phabricator.wikimedia.org/T161035#3142427 (DStrine) [22:48:52] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Errors in CiviCRM dedupe scrren - https://phabricator.wikimedia.org/T160571#3142428 (DStrine) [22:48:54] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, FR-PayPal-ExpressCheckout, and 2 others: Need to distinguish between paypal and paypal_ec for refund IPN messages - https://phabricator.wikimedia.org/T161121#3142426 (DStrine) [22:48:56] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, FR-Smashpig, and 2 others: Handle iDEAL push notifications - https://phabricator.wikimedia.org/T161153#3142425 (DStrine) [22:48:58] Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Spike: investigate getting silverpop data into civi - https://phabricator.wikimedia.org/T159767#3142431 (DStrine) [22:49:00] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, and 2 others: Paypal audit - stop undoing normalization for recurring messages - https://phabricator.wikimedia.org/T160138#3142430 (DStrine) [22:49:02] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, and 3 others: [Tech Debt] Consolidate DonationInterface validation - https://phabricator.wikimedia.org/T160385#3142429 (DStrine) [22:49:04] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, and 3 others: Enable PayPal EC everywhere it's available - https://phabricator.wikimedia.org/T159755#3142432 (DStrine) [22:49:06] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, and 2 others: PHP-Queue: remove IndexedFifo and KeyValue interfaces - https://phabricator.wikimedia.org/T159175#3142433 (DStrine) [22:49:08] Fundraising Sprint G 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Upgrade our CiviCRM server php version to 5.5+ - https://phabricator.wikimedia.org/T158717#3142434 (DStrine) [22:49:10] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, and 3 others: Review fr-tech data retention guidelines - https://phabricator.wikimedia.org/T156317#3142437 (DStrine) [22:49:12] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, and 2 others: IPN listener should normalize new PayPal recurring messages - https://phabricator.wikimedia.org/T157074#3142436 (DStrine) [22:49:15] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, and 6 others: Mediawiki namespace pages, including CentralNotice banners, are slow to save - https://phabricator.wikimedia.org/T158084#3142435 (DStrine) [22:49:16] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 8 others: Fix Coinbase file to support importing UTM fields - https://phabricator.wikimedia.org/T153791#3142441 (DStrine) [22:49:18] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, and 5 others: CentralNotice banner sequence: implement feature for MVP - https://phabricator.wikimedia.org/T144453#3142443 (DStrine) [22:49:23] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, and 7 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3142439 (DStrine) [22:49:23] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 8 others: [Spike] investigate contribution tracking data (was Engage import fail... - https://phabricator.wikimedia.org/T146295#3142442 [22:49:25] Fundraising Sprint G 2017, Fundraising-Backlog: WMF rebranding, logos and font - https://phabricator.wikimedia.org/T141921#3142444 (DStrine) [22:49:26] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, and 5 others: Clean up confusing and repeated code in the PayPal express adapter - https://phabricator.wikimedia.org/T134445#3142445 (DStrine) [22:49:28] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint Dirt Farming, and 9 others: Store and update list of currently working iDEAL banks - https://phabricator.wikimedia.org/T128692#3142446 (DStrine) [22:49:30] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, and 4 others: Move gateway-specific normalizations out of recurring.module - https://phabricator.wikimedia.org/T107372#3142447 (DStrine) [22:49:33] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising Sprint G 2017, Fundraising-Backlog, and 5 others: Assisted currency conversion for PayPal is broken again - https://phabricator.wikimedia.org/T98447#3142448 (DStrine) [23:33:46] fundraising-tech-ops, Patch-For-Review: rename backup4001 to frbackup4001 and jessie reimage - https://phabricator.wikimedia.org/T158220#3030204 (Dzahn) @Jgreen Thanks for your comment on the gerrit change above. I amended again to remove the host from DHCP. So this task is still in progress and not reso... [23:37:07] So, I'm just trying to get my head around tying donations back to bulk-mails [23:37:32] looking at the query ccodgill put up it looks like she is looking at date ranges [23:38:02] to tie emails to donations - ie. did they open an email & donate in a similar period of time [23:38:12] but, also drawing on the utm_medium [23:38:31] which, is ? specific to each email ? - perhaps I need to ask Caitlin [23:39:53] eileen: yep, it's the utm_medium that she looks at [23:43:21] she also puts contact_id in a lot of the links, though - could potentially use that to not-dupe [23:44:22] ok - I just checked & she really needs the utm_source data back in civi for queries [23:44:29] (PS1) Awight: Force a pythonic build, ignoring the Makefile [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345479 [23:44:32] utm_medium & campaign not so much [23:44:46] the only existing Civi field that we could use is the campaign_id [23:44:51] cwd: ^ But that should fix it [23:45:00] (CR) jerkins-bot: [V: -1] Force a pythonic build, ignoring the Makefile [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345479 (owner: Awight) [23:45:07] which we are not using as yet [23:45:48] (PS1) Awight: Short -l flag for --list-jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345480 [23:46:03] the campaign id field is kind of special because it exists in multiple tables (contributions, recurring contributions, mailings, events) [23:47:25] (CR) Awight: "recheck" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345479 (owner: Awight) [23:47:52] cwd: Just grab the debian/ changes regardless of the V-1, that can't be related. [23:48:13] * awight notes that "recheck" is not a real word. [23:50:56] (CR) Awight: "FIXME: could be that we need to relax test timing constraints a bit" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345479 (owner: Awight) [23:52:34] awight: i guess i can just make those changes and try to package [23:52:44] since the git info doesn't matter anyway [23:53:24] That should do it... [23:53:50] can't really get any less working than currently [23:53:53] It will all be here tomorrow, too, if you've had enuf [23:54:15] i am not good at putting things down :P