[00:05:14] (CR) jerkins-bot: [V: -1] Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) (owner: Ejegg) [00:20:50] (PS1) Eileen: Benevity loosening, handle multiple contact use in the past. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342776 (https://phabricator.wikimedia.org/T115044) [00:40:59] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 5 others: Create an import method for matching gifts and payroll deductions - https://phabricator.wikimedia.org/T115044#3100637 (... [00:52:18] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342781 [00:53:26] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342781 (owner: Eileen) [00:53:59] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342781 (owner: Eileen) [00:55:51] !log update civicrm from 31f19d6901fc6e4bf3b8a57ecde0ea62c1256c95 to 639eb688db02911948814fe8749523145d1a2df6 [00:55:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:05:09] Fundraising Sprint English Cuisine, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we make the nickname field searchable? - https://phabricator.wikimedia.org/T158784#3100662 (Eileenmcnaughton) Open>Resolved [01:06:06] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Create table to track blank addresses created of these on live. As we resolve them we will delete... - https://phabricator.wikimedia.org/T159396#3100665 [01:06:09] Fundraising Sprint English Cuisine, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Delete blank addresses which have been created but have no other actions against them - https://phabricator.wikimedia.org/T159402#3100663 (Eileenmcnaughton) Open>Resolved This step is... [01:28:50] (PS1) Eileen: Set street address to NULL where placeholder info has been used. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342783 (https://phabricator.wikimedia.org/T158268) [01:37:23] (PS5) Ejegg: Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) [01:38:18] (CR) jerkins-bot: [V: -1] Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) (owner: Ejegg) [01:38:44] (PS6) Ejegg: Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) [01:39:35] (CR) jerkins-bot: [V: -1] Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) (owner: Ejegg) [01:40:00] (PS7) Ejegg: Stop denormalizing paypal recurring messages in audit [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342693 (https://phabricator.wikimedia.org/T160138) [01:52:27] Fundraising Sprint English Cuisine, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Delete blank addresses which have been created but have no other actions against them - https://phabricator.wikimedia.org/T159402#3100693 (Pppery) [02:56:44] (CR) Ejegg: "Very careful, and well commented! Just one question on the check for all-done-ness." (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [03:02:19] ejegg: I'm so sorry I had to put you through reading all that - brain-hurty huh? [03:02:29] (CR) Eileen: Restore blanked addresses overwritten on merge. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [03:05:13] (PS14) Eileen: Restore blanked addresses overwritten on merge. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) [03:05:48] (CR) Eileen: "ejegg I've pushed up a change in response to your comment." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [03:06:03] (CR) Ejegg: [C: -1] "This seems unnecessary. wfMessage is totally fine getting the params as an array in the second argument. I agree with the FIXME, but this " [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/342548 (owner: Awight) [03:08:33] (CR) Ejegg: [C: 2] Comments [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/341477 (owner: Awight) [03:10:57] (Merged) jenkins-bot: Comments [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/341477 (owner: Awight) [03:17:11] eileen: oof! [03:17:32] eileen: also, very tough to replicate relevant data at home [03:20:55] ejegg|away: yep I did some through tests but some is just so complicated to reproduct [03:21:02] reproduce [03:28:45] (CR) Ejegg: Restore blanked addresses overwritten on merge. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [03:30:12] (PS15) Eileen: Restore blanked addresses overwritten on merge. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) [03:30:22] (CR) Eileen: "so true, fixed" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [12:38:34] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: BUG: Campaign date fields cannot be edited as text - https://phabricator.wikimedia.org/T97159#3101673 (Aklapper) @amritsreekumar: This issue has been assigned to you in October 2016. Could you please share a status update? Are you still... [12:45:15] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: BUG: Campaign date fields cannot be edited as text - https://phabricator.wikimedia.org/T97159#3101727 (amritsreekumar) a:amritsreekumar>None [12:52:08] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Easy: BUG: Campaign date fields cannot be edited as text - https://phabricator.wikimedia.org/T97159#3101902 (amritsreekumar) @Aklapper: Sorry I forgot about de-assigning myself, as this bug had already disappeared due to a regression. [14:20:00] fundraising-tech-ops, Operations, netops: set up firewall policies for barium, lutetium, db1025, and indium replacement servers - https://phabricator.wikimedia.org/T159336#3102151 (Jgreen) p:Normal>Unbreak! Raising priority because this blocks deprecating Precise in fundraising. [14:21:50] fundraising-tech-ops: replace db1008 with new hardware running jessie - https://phabricator.wikimedia.org/T145107#3102170 (Jgreen) [14:21:52] fundraising-tech-ops, Operations, netops: set up firewall policies for barium, lutetium, db1025, and indium replacement servers - https://phabricator.wikimedia.org/T159336#3102171 (Jgreen) [14:22:05] fundraising-tech-ops: replace db1025 with new hardware running jessie - https://phabricator.wikimedia.org/T145107#2620249 (Jgreen) [14:22:48] fundraising-tech-ops: rename backup4001 to frbackup4001 and jessie reimage - https://phabricator.wikimedia.org/T158220#3102176 (Jgreen) p:Triage>High [14:26:58] fundraising-tech-ops, Operations, netops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102204 (Jgreen) [14:27:39] fundraising-tech-ops, Operations, ops-eqiad: rack and cable frdev1001 - https://phabricator.wikimedia.org/T159887#3102205 (Jgreen) p:Normal>High This blocks deprecating Precise in Fundraising, thus raising priority to high. [14:28:42] fundraising-tech-ops, Operations, netops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3064590 (Jgreen) [14:28:45] fundraising-tech-ops: replace lutetium with new hardware running debian/jessie - https://phabricator.wikimedia.org/T145110#3102209 (Jgreen) [14:30:07] fundraising-tech-ops, Operations, netops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102213 (Jgreen) [14:30:10] fundraising-tech-ops: replace indium (eqiad fundraising logger) with new hardware running jessie - https://phabricator.wikimedia.org/T145116#3102212 (Jgreen) [14:30:51] fundraising-tech-ops, Operations, netops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3064590 (Jgreen) [14:30:54] fundraising-tech-ops, Operations, netops: reassign wmf7010/frpm1001 to host "civi1001.frack.eqiad.wmnet" - https://phabricator.wikimedia.org/T159342#3102217 (Jgreen) [14:37:47] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, and 5 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3102252 (Pcoombe) @AndyRussG That... [15:16:25] fundraising-tech-ops: build fundraising civicrm (barium) replacement server on Debian Jessie, with HHVM or PHP5.5 - https://phabricator.wikimedia.org/T136959#3102392 (Jgreen) [15:20:35] fundraising-tech-ops: replace boron (fundraising build server) with new hardware running jessie - https://phabricator.wikimedia.org/T145117#3102423 (Jgreen) [15:24:39] Fundraising Tech Backlog, Fundraising-Backlog, fundraising-tech-ops, HHVM, Performance: Run Payments wiki on HHVM - https://phabricator.wikimedia.org/T91899#3102435 (Jgreen) [15:24:41] Fundraising-Backlog, fundraising-tech-ops, Epic: [Epic] Upgrade paymentswiki to MediaWiki REL_* release branch or follow the WMF train. Upgrage PHP to HHVM or PHP 5.5 - https://phabricator.wikimedia.org/T130130#3102432 (Jgreen) Open>Resolved Closing task because AFAICT it was completed by 9/2... [15:26:59] fundraising-tech-ops, Operations, ops-eqiad: Rack and setup Fundraising DB frdb1001 - https://phabricator.wikimedia.org/T136200#3102455 (Jgreen) [15:27:01] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#3102456 (Jgreen) [15:29:11] fundraising-tech-ops, Operations, netops: Cleanup layer2 firewall config from pfw-eqiad - https://phabricator.wikimedia.org/T111463#3102464 (Jgreen) [15:34:41] fundraising-tech-ops, Operations, ops-eqiad: decommission the old pay-lvs1001/pay-lvs1002 boxes - https://phabricator.wikimedia.org/T156284#3102475 (Jgreen) a:Cmjohnson [16:00:28] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, and 5 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3102614 (DStrine) I'm cool with th... [16:20:20] Fundraising-Backlog, MediaWiki-extensions-ContributionTracking, Scap (Scap3-MediaWiki-MVP): Clean up Contribution Tracking settings in main wmf config repo - https://phabricator.wikimedia.org/T147479#3102766 (demon) We should get this done :) [16:36:09] Fundraising-Backlog, MediaWiki-extensions-ContributionTracking, Scap (Scap3-MediaWiki-MVP): Clean up Contribution Tracking settings in main wmf config repo - https://phabricator.wikimedia.org/T147479#2693840 (awight) @demon I don't understand how the current config works, it seems to include a file w... [16:52:20] fundraising-tech-ops, Operations, netops: reassign wmf7010/frpm1001 to host "civi1001.frack.eqiad.wmnet" - https://phabricator.wikimedia.org/T159342#3102920 (faidon) [16:52:24] fundraising-tech-ops: replace indium (eqiad fundraising logger) with new hardware running jessie - https://phabricator.wikimedia.org/T145116#3102921 (faidon) [16:52:26] fundraising-tech-ops: replace db1025 with new hardware running jessie - https://phabricator.wikimedia.org/T145107#3102923 (faidon) [16:52:28] fundraising-tech-ops: build fundraising civicrm (barium) replacement server on Debian Jessie, with HHVM or PHP5.5 - https://phabricator.wikimedia.org/T136959#3102924 (faidon) [16:52:31] fundraising-tech-ops, Operations, netops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102917 (faidon) Open>Resolved a:faidon That's done now :) [17:04:49] Fundraising-Backlog, MediaWiki-extensions-ContributionTracking, Scap (Scap3-MediaWiki-MVP): Clean up Contribution Tracking settings in main wmf config repo - https://phabricator.wikimedia.org/T147479#3102966 (demon) >>! In T147479#3102879, @awight wrote: > @demon I don't understand how the current co... [17:08:23] fundraising-tech-ops, Operations, netops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102993 (Jgreen) Great, thank you!!! [17:09:11] hi fr-tech! [17:09:32] how's the skiing, Jeff_Green ? [17:13:26] hey ejegg. skiing was fun, although it was windy as hell and there were points where we had to stop and face the other way for several minutes because it was impossible to keep our eyes open [17:13:47] and the entire town was closed, ended up having to ski back home for lunch [17:13:58] oh man! [17:30:22] fr-tech any news for scrum of scrums? [17:30:32] Negative [17:43:22] fundraising-tech-ops, Operations, netops, Patch-For-Review: reassign wmf7010/frpm1001 to host "civi1001.frack.eqiad.wmnet" - https://phabricator.wikimedia.org/T159342#3103222 (Jgreen) p:Normal>High [17:43:33] cwd: whew! [17:43:54] What a waste of a perfectly good technology. [17:44:07] heh [17:44:23] Job name is not entirely trivial, I'm thinking [17:44:49] Should we keep the super verbose naming, e.g. "Ingenico Orphan Rectifier" [17:44:51] fundraising-tech-ops: frack eqiad hardware refresh - https://phabricator.wikimedia.org/T133524#3103232 (Jgreen) [17:44:52] ? [17:44:53] fundraising-tech-ops: build fundraising civicrm (barium) replacement server on Debian Jessie, with HHVM or PHP5.5 - https://phabricator.wikimedia.org/T136959#3103231 (Jgreen) [17:45:10] fundraising-tech-ops: build fundraising civicrm (barium) replacement server on Debian Jessie, with HHVM or PHP5.5 - https://phabricator.wikimedia.org/T136959#2353563 (Jgreen) [17:45:14] fundraising-tech-ops: upgrade all frack servers to debian/jessie - https://phabricator.wikimedia.org/T146479#3103236 (Jgreen) [17:45:14] awight: can we just be diligent and put them in the crontab? [17:45:26] Good idea! [17:45:29] * * * * * wrapper.sh -n 'Job Name' cmd [17:45:33] +1 [17:45:34] *wrapper.pl [17:45:38] bahaha [17:45:48] **wrapper.py [17:45:48] capitulation [17:50:01] cwd: I dislike what's happened with wikimedia-fundraising-tools [17:50:11] but it has the lock.py class I was going to pull in. [17:50:25] Seems like we don't want to install any such thing on the deployment server. [17:50:32] New repo? [17:50:51] awight: yeah i vote for new repo [17:50:57] metoo [17:54:00] wheee! [17:54:35] wikimedia/fundraising/process-control, unless someone stops me [17:55:03] crash override [17:55:26] awight: slight shift in wrapper responsibility, it should run a list of commands [17:55:33] described in a config file [17:55:34] eh? [17:55:47] humm [17:55:56] how's that? [17:56:56] like, show me an example of what the config file would have for the orphan job [17:57:05] awight: * * * * * wrapper.py /srv/jobs/job.conf [17:57:09] fr-tech any questions for the new Senior Quality Analyst? [17:57:16] cool! [17:57:46] file could say things like, list of commands, job name, description, email recips, halt on failure... [17:58:39] crash-override.py [17:59:19] impulse-control.py [18:00:10] * awight is concerned we're reinventing Zuul now [18:00:21] fr-tech: So now [18:00:21] that you have- [18:00:21] you know, whoever [18:00:21] you're trying [18:00:21] to do [18:00:22] a favor [18:00:22] for [18:00:23] -you've done it- [18:00:23] and I'm sure [18:00:24] you had [18:00:24] a smirk [18:00:25] on your mouth [18:00:26] as you got me [18:00:26] into this. [18:00:49] hahaha slander doesn't know about daylight savings. [18:02:54] cwd: I'm mostly convinced that the config file is nicer than a slalom course of commandline flags [18:03:09] awight: agreed - thank Jeff_Green :) [18:03:22] I'm tempted to implement so they can be passed either way... [18:03:49] generally a good practice imo [18:05:20] awight: should we do json for the config file? [18:05:36] seems like that's what fr-tech is most acclimated to [18:06:29] Jeff_Green: I was thinking yaml... [18:06:33] yaml might suffice [18:07:09] * awight kicks XML skeleton back into the closet [18:08:15] yaml is fine by me [18:08:49] groovy [18:09:03] i'll start writing some files for the jobs and see what the sharp edges are [18:09:26] We might need a global config as well, for things like log archive path [18:09:43] although I'd love to not need global config [18:10:20] could just have a block of defaults... [18:10:23] wrapper should feed to syslog, with an appropriate tag [18:10:43] awight I have some well-developed syslog foo for python you can steal [18:11:09] ooh yes pls [18:11:16] so rather than putting the logfile in the config, i think you'd put the syslog ident tag [18:11:24] finding.... [18:11:33] Jeff_Green: syslogging is perfect, but there's one catch. [18:11:48] We currently syslog directly from some of the jobs [18:11:55] that's fine [18:12:05] so we'll be duplicating lines if we pipe all of stdout+err to syslog [18:12:38] I would strip out the direct syslogging and just printf everywhere, but I think there might be some benefit to at least having the structured logging levels. [18:12:41] I suppose you could control that by loglevel [18:13:02] How do you suggest? [18:13:42] the python syslog framework allows you to send things internally with a low level of priority (i.e. debug, whatever), and at the end only output stuff above a given threshold to syslog [18:13:48] evolving "spec": https://github.com/adamwight/process-control [18:14:11] so for example you could always collect whatever the job spews to stdout, but set a higher threshold for output from the wrapper to syslog [18:14:23] and that output threshold could be per-job [18:14:30] Jeff_Green: That's sensible, but I don't see how it helps cos I think we always want to capture all the output. [18:14:32] loglevel = 'err" [18:15:01] awight: the wrapper needs to watch both stderr and stdout in realtime right? [18:15:06] yeah [18:15:21] (and somehow deal with preserving the interleaving) [18:15:35] for drush jobs that do their own direct sysloggging, you could have the wrapper filter away everything that gets spewed to stdout [18:15:45] but log everything that gets spewed to stderr [18:16:01] that way you filter away some of the double logging [18:16:25] does that help? [18:16:31] I don't think so. [18:17:02] what's still unsolved? [18:19:28] We need to capture all output, which for our best-maintained jobs currently consists of * random printf'd things on stdout, * loglines that get printed to stdout and also syslogged, * loglines that print to stderr and also syslog, and * fatal stuff on stderr [18:20:51] i guess short term I would opt for double-logging, and long term I would adapt the drush jobs to be more intentional re. what they send to stdout vs syslog? [18:21:39] right now what happens? you end up with jenkins collecting a bunch of STDOUT and sending it in the email at the end of the job? [18:21:53] Well, I guess this whole cronification will help us clean up lots of things. [18:22:04] Currently, it collects stdout+stderr and saves to a file. [18:22:22] Email only happens after failure, or when the job directly failmails [18:22:46] Meanwhile, most of the important lines are also syslogged [18:22:59] It's horrific and I'm happy to move towards the sunlight [18:23:15] Just trying to figure out what that looks like now that I've been in the cave for almost 5 years. [18:24:15] awight: ok, i have no objection to having collect-to-file being a feature, but my hope is long term we can slay it [18:24:31] because it's hard to centralize collected files [18:24:47] Yeah I hate it [18:25:16] awight: how much content do we typically collect in the outputfiles? an emailable quantity or beyond lots? [18:25:26] And the application code shouldn't have to know about syslog-and-echo-to-stdout [18:25:37] I don't think we ever want to email the job output [18:25:39] Too much PII [18:25:40] those output files are something we should stop emailing [18:25:41] yeah [18:25:44] ok [18:25:45] just send a notice [18:25:48] hey look at this file [18:25:52] exactly. [18:25:57] cool [18:26:06] agreed that "look at this syslog tag" would be cleaner [18:26:12] Centralized files are fine, too [18:26:34] But I wonder how to address a location in that [18:26:44] in syslog? [18:26:46] yeah [18:26:59] PID tag? [18:27:07] I had a thought a while ago that we should have a tool that can "show all logs for order ID NNN" [18:27:12] right now nearly all of our syslog filtering is based on "tag' which is really just a prefix that's part of the message [18:27:25] yeah it might also be cool to have "show logs for job NAME pid PPP" [18:27:28] awight: if only we had a more consistent order number [18:27:42] CTID seems like the most uniquest [18:27:55] wow it didn't spell correct uniquest [18:28:03] you could do what MTAs tend to do, which is generate a semi-UUID and add it to every log line [18:28:10] true, but it's easy to have log viewer lookup correlated IDs [18:28:14] how about somethign like this: [18:28:17] sec [18:28:39] for example: [18:28:40] Mar 15 18:28:28 boron puppet-master[7393]: Compiled catalog for barium.frack.eqiad.wmnet in environment production in 1.57 seconds [18:28:44] that's a typical log line [18:28:51] puppet-master is the tag [18:28:51] d'oh [18:28:56] 7393 is the PID [18:29:02] right, that would be nice [18:29:04] what if you did something like: [18:29:22] currently most of our jobs are drupal[PID] [18:29:25] which is terrible [18:29:27] happy_fun_job-X7fh3I828[1198]: an error has occured [18:29:38] thank you and okay! [18:30:02] I like it, but not sure we need the complexity. [18:30:15] How often do PIDs cycle on our prod host? [18:30:23] well [18:30:58] in this case unless you do something interesting it's going to be the wrapper's PID that gets logged, so it will be new every time cron invokes the wrapper [18:31:20] I was indeed planning on something interesting ;) [18:31:48] you could try to have the wrapper fetch the PID of the subprocess, in which case I guess it would get drush's PID, so again, new every invocation [18:32:31] But imagining that PIDs overrun 32768 a few times a day? [18:33:09] i don't think they just march up like a counter [18:33:16] i think they get reused regularly [18:34:00] at least based on my experience parsing mail logs, PID is only useful when you also take into account the timeframe [18:34:20] right [18:34:54] so another way to think about this.... [18:35:04] uh apparently we can set /proc/sys/kernel/pid_max to 64 bits if we want [18:35:13] yeah let's not [18:35:22] :-{ [18:35:34] I think PIDs weren't really intended for long term job identification [18:35:39] You don't want the newest features? :p [18:36:00] this strikes me as being off in the weeds with hatchet :-) [18:36:14] so how I've used cron wrappers in the past.... [18:36:41] IMO the wrapper should be responsible for identifying and notifying re. errors, and for optionally sending job reports [18:37:05] routine stuff should be handled by the subtask directly (hence drush jobs already writing to syslog) [18:37:30] and imo it's appropriate for drush jobs to respond with a summary report at the end, to stderr [18:37:37] err sorry to stdout! [18:37:51] hmm. That sounds happy [18:38:16] if we can work toward that as a design goal, it makes it pretty easy to keep things sane [18:38:46] and that way, since the drush job is writing its own syslogs, the wrapper doesn't have to be all intelligent about which PID was running and whatnot [18:38:58] Perfect. [18:39:31] So you're imagining that we throw out the summary report when running under cron? [18:39:48] yeah [18:40:09] in some cases I've hacked an autodetect into the subtask so it spews differently if it's run by cron vs interactively [18:40:21] Yeah I did that with our python jobs [18:40:37] & we can always add a "verbose" flag [18:40:44] sure [18:41:28] in the interest of not forcing a whole lot of rewrites, I like your idea of the output text files to start [18:41:32] i' [18:41:42] I like the design and will work towards it, but during the transition we'll still want to collect the output somewhere, in case it contains forensic things we needed but weren't syslogged. [18:41:47] Right. what you were saying [18:42:36] we could make it configurable: stdout_destination = "/tmp/jobname_${timestamp}.txt" [18:42:51] or: stdout_destination = "email body" [18:43:30] defaulting to nowhere if unconfigured [18:43:52] Maybe default to pass-through if unconfigured [18:43:56] detail tho [18:44:10] pass through meaning to stdout, so cron itself sends mail? [18:44:22] yah [18:45:12] my personal preference would be to have cron mail directly to root@ and that only happens if the wrapper fails to compile [18:45:44] +1 okay [18:45:47] that way you keep all your config, including mail destinations, in the /srv/cronwhatever/job.conf files [18:46:21] Jeff_Green: fwiw you can stick MAILTO=addr in the cron file [18:46:36] if we are going to be generating them anyway [18:47:11] it's true however the final output all looks the same (same from/to/subject), which is terrible when you have 400 of them [18:47:30] if you take control of the entire mailflow in the wrapper, you can set useful subjectlines like [18:47:49] Subject: SUCCESS: job blarney ran happily in 3:39:29! [18:48:19] Subject: FAILURE: job blarney crapped out at subtask 'fetch_files'! [18:48:45] yep that's true [18:49:01] i actually filter mail differently based on SUCCESS/FAILURE, so I see the failures quickly without being distracted by the tremendous success [18:49:21] so nothing should make it to stderr from the wrapper unless it's a catastrophic failure? [18:49:31] that's my vote anyway yeah [18:49:36] sounds good to me [18:49:51] stdanything i guess [18:51:44] awight: frack-puppet modules/apache2/templates/mods/security/modsec_log_alerter.py.erb has usable syslog code, also mailing code [18:52:36] also modules/aide/files/aiderator [18:58:51] wonderful, thanks! [18:59:29] np [19:01:42] cwd: Oops, you're right that I should be using the subprocess module rather than multiprocessing. [19:13:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-DonationInterface, Patch-For-Review: Latest DonationInterface unusable under CRM - https://phabricator.wikimedia.org/T131798#3103618 (Ejegg) Open>Resolved a:Ejegg This is OK now [19:25:42] cwd: It runs a job now... https://github.com/adamwight/process-control [19:25:54] I'll work on failure detection [19:26:53] holler! [19:27:16] lol crash override [19:27:20] i was joking, but ok [19:32:40] It might get confusing. But we can rename later [19:32:54] * awight puts on most literal face [19:34:04] awight: nice! [19:36:47] (PS1) Ejegg: Cast a wider net when searching for ct_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/342893 [19:41:47] awight: maybe instead of having crash_override as a separate script, could it just be a flag for the main wrapper script? [19:42:53] like --force or --retry or similar? [19:43:26] Jeff_Green: All I did there was define the job wrapper as a class for testability and stuff, but IMO the frontend to run it should be in a different file. [19:43:38] Maybe "crash-override" is too confusing of a name [19:43:42] started as a joke [19:43:46] took on a life of its own [19:43:48] uncrash [19:43:54] ok [19:44:09] (PS1) Ejegg: WIP detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 [19:44:26] Meme from my childhood that will not die, http://autoguide.com.vsassets.com/blog/wp-content/uploads/2010/07/Vince_and_Larry.jpg [19:44:45] don't be a dummy! [19:44:50] yass [19:45:00] * Jeff_Green goes looking for the famous Chinese truck crash test footage [19:45:37] https://www.youtube.com/watch?v=D827IxEJVS4 [19:46:02] it's gotta be fake [19:46:50] whoa [19:47:00] comments say it's a VW [19:47:35] whatever it is, apparently everything forward of the rear bumper is the crumple zone [19:53:13] cwd: docs updated to explain the file format [19:53:42] Jeff_Green: It's amazing. The video quality does make it look a bit fake, but still believable. [19:56:32] awight: excellent [19:58:13] Gotta lunch [19:59:00] me too [20:04:03] Jeff_Green: that is horrifying [20:06:46] this one always cheers me up http://www.wimp.com/bill-dances-classic-bloopers/ [20:08:09] ha [20:08:48] my favorite is the detached electric outboard :-) [20:09:06] dad gum it! [20:09:52] when the dog jumps in you can clearly hear that they bleeped the word heck [20:11:40] biab! [20:24:30] (PS2) XenoRyet: PayPal subscription message normalization [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/341595 (https://phabricator.wikimedia.org/T107372) [20:31:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Errors in CiviCRM dedupe scrren - https://phabricator.wikimedia.org/T160571#3103876 (Eileenmcnaughton) [20:33:34] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Errors in CiviCRM dedupe scrren - https://phabricator.wikimedia.org/T160571#3103900 (Eileenmcnaughton) note the email only rule works [20:34:21] (CR) Awight: [C: 2] "Thanks!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/341595 (https://phabricator.wikimedia.org/T107372) (owner: XenoRyet) [20:36:49] (Merged) jenkins-bot: PayPal subscription message normalization [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/341595 (https://phabricator.wikimedia.org/T107372) (owner: XenoRyet) [20:42:03] ejegg: OK I'm going to reread through this one now…. https://gerrit.wikimedia.org/r/#/c/342554/ & see if I still think I knew what I was doing when I wrote it… [20:42:34] comments seem lucid enough... [20:42:41] I'll scan it again too [20:43:03] (PS2) Ejegg: WIP Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) [20:43:44] ejegg: new coding goal, lucid comments :-) [20:44:12] :) [20:46:24] library i'm in is going to close halfway through sprint planning. relocating to a cafe that'll stay open! [20:49:25] (CR) jerkins-bot: [V: -1] WIP Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) (owner: Ejegg) [20:57:17] (CR) Eileen: "OK, I read this through & I still think it makes sense & I can't spot any danger points... phew" (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [21:27:37] Fundraising Sprint English Cuisine, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Trilogy import with foreign checks not working - https://phabricator.wikimedia.org/T159882#3104091 (Eileenmcnaughton) Open>Resolved Leanne confirmed ok to close [21:28:40] (CR) Ejegg: [C: 2] "Yep, still looks good to me too." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [21:28:41] dstrine: there are a couple of tickets in 'done' which were waiting input from ccogdill - I think you were going to discuss in some meeting you had lined up with her? [21:29:17] (Abandoned) Eileen: E-notice fix. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342555 (owner: Eileen) [21:29:22] cwd: Now with tests :-) [21:29:44] eileen: I thought is was only one? https://phabricator.wikimedia.org/T153791 [21:29:54] let me try and ping her in gchat [21:32:53] (Merged) jenkins-bot: Restore blanked addresses overwritten on merge. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342554 (https://phabricator.wikimedia.org/T159408) (owner: Eileen) [21:37:53] dstrine: and this https://phabricator.wikimedia.org/T146295 [21:38:30] ejegg: should I start deploying & setting that up now or should I wait for you to look at that last benevity one? [21:43:55] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 6 others: Fix Coinbase file to support importing UTM fields - https://phabricator.wikimedia.org/T153791#2891044 (DStrine) @Leann... [21:44:12] eileen: I'm trying to coordinate a bit more on T153791 [21:44:12] T153791: Fix Coinbase file to support importing UTM fields - https://phabricator.wikimedia.org/T153791 [21:44:37] eileen: Caitlin will respond to T146295 shortly [21:44:37] T146295: [Spike] investigate contribution tracking data (was Engage import failing to import certain significant fields) - https://phabricator.wikimedia.org/T146295 [21:45:06] dstrine: thanks [21:46:08] Jeff_Green, please update the channel Topic to link the new log location: https://wm-bot.wmflabs.org/logs/%23wikimedia-fundraising/ [21:46:11] thanks! [21:48:07] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342943 [21:49:11] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342943 (owner: Eileen) [21:49:25] eileen oops, sorry, I'll get to the benevity one by end of day! [21:49:31] ejegg: that's fine [21:49:50] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342943 (owner: Eileen) [21:49:52] I just wanted to know whether it was going to be so soon I should wait to deploy - but have started deploy now [21:51:58] woo! [21:52:26] !log civicrm update from 639eb688db02911948814fe8749523145d1a2df6 to f1a3d647bc7d7fa4d7bbbff291d4a160ac8777c0 [21:52:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:42:43] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 2 others: [Tech Debt] Consolidate DonationInterface validation - https://phabricator.wikimedia.org/T160385#3104289 (DStrine) [22:42:46] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, FR-Paypal, Patch-For-Review: Paypal audit - stop undoing normalization for recurring messages - https://phabricator.wikimedia.org/T160138#3104290 (DStrine) [22:42:48] Fundraising Sprint F 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Spike: investigate getting silverpop data into civi - https://phabricator.wikimedia.org/T159767#3104291 (DStrine) [22:42:50] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, FR-PayPal-ExpressCheckout, and 2 others: Enable PayPal EC everywhere it's available - https://phabricator.wikimedia.org/T159755#3104292 (DStrine) [22:42:52] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Revert updates that set valid addresses to blank, and have no further actions against them - https://phabricator.wikimedia.org/T159408#3104293 (DStrine) [22:42:56] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog: Create fr-tech deadlines calendar - https://phabricator.wikimedia.org/T159177#3104295 (DStrine) [22:42:58] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, Patch-For-Review: PHP-Queue: remove IndexedFifo and KeyValue interfaces - https://phabricator.wikimedia.org/T159175#3104296 (DStrine) [22:43:00] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog: Update fr-tech job roles on wiki - https://phabricator.wikimedia.org/T158710#3104297 (DStrine) [22:43:03] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, and 2 others: We still have contacts whose merge is blocked on mis-rounded geocodes - https://phabricator.wikimedia.org/T158271#3104298 (DStrine) [22:43:08] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, and 3 others: Stop creating addresses N0NE Provided & delete them where we have them - https://phabricator.wikimedia.org/T158268#3104301 (DStrine) [22:43:12] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, and 5 others: Mediawiki namespace pages, including CentralNotice banners, are slow to save - https://phabricator.wikimedia.org/T158084#3104302 (DStrine) [22:43:14] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, FR-PayPal-ExpressCheckout: IPN listener should normalize new PayPal recurring messages - https://phabricator.wikimedia.org/T157074#3104303 (DStrine) [22:43:16] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, and 2 others: Review fr-tech data retention guidelines - https://phabricator.wikimedia.org/T156317#3104304 (DStrine) [22:43:18] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog: fill out PCI SAQ-A form for 2017 - https://phabricator.wikimedia.org/T155779#3104305 (DStrine) [22:43:22] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, and 6 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3104307 (DStrine) [22:43:34] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 7 others: Fix Coinbase file to support importing UTM fields - https://phabricator.wikimedia.org/T153791#3104311 (DStrine) [22:43:39] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 7 others: [Spike] investigate contribution tracking data (was Engage import fail... - https://phabricator.wikimedia.org/T146295#3104314 [22:43:42] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, and 4 others: CentralNotice banner sequence: implement feature for MVP - https://phabricator.wikimedia.org/T144453#3104315 (DStrine) [22:43:47] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, FR-PayPal-ExpressCheckout, and 4 others: Clean up confusing and repeated code in the PayPal express adapter - https://phabricator.wikimedia.org/T134445#3104316 (DStrine) [22:43:51] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint Dirt Farming, and 8 others: Store and update list of currently working iDEAL banks - https://phabricator.wikimedia.org/T128692#3104317 (DStrine) [22:43:55] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 6 others: Create an import method for matching gifts and payroll deductions - https://phabricator.wikimedia.org/T115044#3104318 (... [22:44:00] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, and 3 others: Move gateway-specific normalizations out of recurring.module - https://phabricator.wikimedia.org/T107372#3104320 (DStrine) [22:44:03] Fundraising Sprint English Cuisine, Fundraising Sprint F 2017, Fundraising-Backlog, FR-Amazon, and 4 others: Assisted currency conversion for PayPal is broken again - https://phabricator.wikimedia.org/T98447#3104321 (DStrine) [22:44:15] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) (owner: Ejegg) [22:50:16] (CR) jerkins-bot: [V: -1] WIP Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) (owner: Ejegg) [22:53:19] Fundraising Sprint F 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Spike: investigate getting silverpop data into civi - https://phabricator.wikimedia.org/T159767#3104358 (Eileenmcnaughton) a:Eileenmcnaughton [22:54:58] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 6 others: Benevity: Create an import method for matching gifts and payroll deduc... - https://phabricator.wikimedia.org/T115044#1726852 [23:25:35] ejegg: my thinking on the blank addresses is once the script has run maybe we should book a hangout next week just to try to look at what remains & decide how we want to deal with what is left [23:25:53] yep, sounds good! [23:29:48] (CR) Ejegg: [C: -1] "This is only passing because there's another error. AMD is at 486 to the dollar in CurrencyRates.php, so you need an amount much higher th" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/341478 (owner: Awight) [23:31:34] (CR) Ejegg: [C: 2] Remove deprecated function [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/341721 (owner: Awight) [23:35:16] (CR) Ejegg: "Looks like it should work, just a couple questions." (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342776 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [23:39:18] (CR) Eileen: Benevity loosening, handle multiple contact use in the past. (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342776 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [23:41:40] (PS2) Ejegg: Benevity loosening, handle multiple contact use in the past. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342776 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [23:42:35] (CR) Ejegg: [C: 2] "Yep, this is correct. Sorry, read the diff wrong and thought an extra line was deleted!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342776 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [23:47:19] (Merged) jenkins-bot: Benevity loosening, handle multiple contact use in the past. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342776 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [23:51:32] (PS3) Ejegg: Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) [23:55:51] ejegg: awight just talking silver pop with Caitlin & she is talking about doing a quick tour - do you guys want to see that, in which case maybe I should schedule a time next week (cwd, XenoRyet AndyRussG may also be interested) [23:56:12] (CR) jerkins-bot: [V: -1] Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) (owner: Ejegg) [23:56:23] eileen: I'd love to sit in, but optionally so don't schedule around me! [23:56:54] Same boat here, I'd be interested, but don't hold up on my account. [23:57:41] eileen: I had an account a long time ago, but I'd be up for a refresher! [23:58:25] (PS1) Awight: Add .gitreview [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342967 [23:58:27] (PS1) Awight: Tests for the lock module [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342968 [23:58:29] (PS1) Awight: test failopen; py3 [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342969 [23:58:31] (PS1) Awight: Correct failopen test [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342970 [23:58:33] (PS1) Awight: 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 [23:58:34] cwd|afk: enjoy ;) ^ [23:59:27] nicenice!