[00:01:22] (PS4) Ejegg: Detect normalized paypal messages in queue consumer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342896 (https://phabricator.wikimedia.org/T107372) [00:02:15] ejegg: XenoRyet|afk I've put it in the calendar for next week - optional [00:02:31] thanks eileen [00:03:35] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342972 [00:04:44] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342972 (owner: Eileen) [00:12:56] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/342972 (owner: Eileen) [00:14:14] !log updated civicrm from f1a3d647bc7d7fa4d7bbbff291d4a160ac8777c0 to cca5921a2a63346bb0d98dbaa5710952dafd1b21 [00:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:57:32] (CR) Ejegg: [C: 2] "Seems to be a safe cleanup! Might be a couple stray bits left to zap, but this gets most of it. Now to upstream!" [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/340157 (owner: Awight) [00:58:35] (Merged) jenkins-bot: Remove unused key-value dead-end [wikimedia/fundraising/php-queue] - https://gerrit.wikimedia.org/r/340157 (owner: Awight) [01:00:36] (CR) Ejegg: "Just needs the corresponding tests deleted, I think" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/340924 (https://phabricator.wikimedia.org/T159175) (owner: Awight) [01:01:26] k, I'm out. see ya! [01:02:50] Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising-Backlog: Create fr-tech deadlines calendar - https://phabricator.wikimedia.org/T159177#3059222 (Ejegg) Open>Resolved [01:03:22] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, Fundraising-Backlog: fill out PCI SAQ-A form for 2017 - https://phabricator.wikimedia.org/T155779#2954361 (Ejegg) Open>Resolved [11:01:49] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, and 6 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3105544 (ema) If I understand the... [15:59:29] Fundraising-Backlog, Wikimedia-Fundraising, MediaWiki-extensions-DonationInterface: Donation form extends off the right side of the page in some mobile browsers - https://phabricator.wikimedia.org/T149247#3106556 (Pcoombe) [17:00:43] fr-tech: Jogger, n.: [17:00:43] An odd sort of person with a thing for pain. [17:00:43] -- discuss. [17:14:16] ghi fr-tech! [17:14:23] s/g// [17:16:42] hi awight [17:16:57] ejegg: oh hey! [17:17:04] I was just about to ping you wrt GatewayAdapter [17:17:12] -talk ? [17:17:16] Thanks for all the preparatory CR [17:17:19] sure [17:17:22] I only have 10 min tho [17:17:40] ok, later is good too [17:20:18] awight: i have been reviewing the process control code and it looks good, about to start actually running it [17:20:24] btw i don't have +2 on that repo [17:21:10] cwd: We're in fr-tech-talk, as you wish [17:21:19] I'll add you to the repo, thx for the nudge [17:21:26] Fundraising-Analysis, Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#3107015 (Nuria) ping @awight @AndyRussG I think our job on pivot and banners takes care of this item, le... [17:21:35] sure, be there in a few [17:30:28] ejegg: i'm in the workshop down the hill [17:30:30] a little far from the wifi [17:30:40] ah, gotcha [17:32:14] anyway i'm going to be building something to take these job config files and turn them into cron files [17:32:26] and you can update them with a deploy [17:33:59] oh, spiffy [17:34:27] oh look, mw-vagrant made the switch to jessie! [17:34:30] cron just scans the dir so no need to restart anything [17:34:35] yeah! [17:35:30] ah, I see your name on a dash task [17:36:17] but I guess the rest of our stuff is working? [17:37:14] i guess [17:37:18] can you link me that task? [17:37:25] https://phabricator.wikimedia.org/T154264 [17:50:59] XenoRyet: I'm available for fr-tech-talk if you want to discuss anything! [17:55:01] ejegg: do you know of anything about our code that would prevent us from running php fpm? [17:55:23] cwd nope! That's what I'm running locally [17:56:14] actually, we should start thinking about php7 or hhvm before big english next year [17:56:48] i feel like hhvm has a pretty bad bug history [17:57:02] but php7 has (probably) more breaking changes [17:57:34] there are certainly some warning when I run crm on it, but everything seems to work OK [17:57:43] http://php.net/supported-versions.php [17:57:54] 5.6 loses security patches at the end of 2018 [17:58:44] but 7.1 only gets 'em up through Dec 1, 2019 [17:58:53] What's the plan for Debian Jessie? [18:00:44] oh wow, jessie LTS version is supposed to live on through 2023 or 2025? [18:01:30] +1 php5-fpm [18:05:13] i don't know of any reason we couldn't just swap it out [18:11:21] https://issues.civicrm.org/jira/browse/CRM-17789 [18:11:29] Support PHP 7 -> completed [18:12:18] noice [18:12:33] +1 to what cwd and sometimes Jeff_Green said, hhvm has not impressed me. I was excited at first by what seemed like snappiness, before discovering that the boot time is often crazy long, and php 5.6 has very similar performance [18:12:48] (zend php 5.6) [18:13:25] cwd: What do you mean by, > anyway i'm going to be building something to take these job config files and turn them into cron files [18:13:36] For real? [18:14:03] awight: you don't like that idea? [18:14:13] I'm not sure [18:14:26] wait, you mean the jenkins job config files? [18:14:36] no [18:14:43] ejegg I guess as soon as we're fully on Jessie we start testing with the next LTS [18:14:59] I think for example https://github.com/adamwight/process-control/blob/master/tests/data/timeout.yaml [18:15:16] we could also investigate in backports for php I guess, if it got to that point [18:15:29] sounds good Jeff_Green [18:15:53] cwd: It seems like a good idea in theory, but for example, are you going to specify recurrence in vixiecron format? [18:16:03] cwd so you're thinking add a schedule line to the yaml? [18:16:07] In which case, maybe that doesn't belong in a random yaml file but in crontab [18:16:12] jinx [18:16:32] It also feels like we can get there incrementally, if we decide it's a good idea. [18:16:45] Writing cron glue would be a nearer next step. [18:17:21] awight: you mean the .conf file? [18:17:38] i mean, yes you could just write those files directly [18:17:49] it would just generate a bunch of flags for the wrapper etc [18:18:08] no flags [18:18:11] the wrapper script just takes the one arg [18:18:12] a bunch of the features we want are in the wrapper's domain [18:18:14] the conf file gets read in directly [18:18:14] awight: what is timeout for? to delay the start of the job? [18:18:25] Jeff_Green: nah, to kill overlong stuff [18:18:29] ^ [18:18:51] Jeff_Green: fwiw, the conf file format as it is now is documented in the readme: https://github.com/adamwight/process-control [18:20:12] oic. wouldn't it be some form of alarm() (in perl terms anyway) in the python wrapper and not /bin/sleep? [18:20:26] it is. [18:20:26] awight: looking [18:20:33] ah ok, I didn't understand [18:20:36] Jeff_Green: Sorry, that link was incredibly confusing. [18:20:43] You're looking at a unit test resource [18:20:44] ha [18:20:47] awight: well, we can either put the frequency in the conf file or we will have to make some other path to writing the crontabs [18:20:48] It's a job which sleeps too long. [18:21:11] oh! now I get it, duhhh. [18:21:12] i agree that it's a little funny to paste 1 * * * * in the yaml but jenkins does the same thing [18:21:21] jenkins is a bad role model :p [18:21:39] i vote to stick with cron syntax, because it works and people already understand it [18:22:10] Jeff_Green: +1 to cron syntax, but I'm wondering whether using the .conf files as input to some text processing which then creates templates is prudent [18:22:17] It seems like we're reinventing puppet [18:22:36] awight: what's the best alternative? [18:22:49] donno, I mean I like your idea but at the same time it feels weird [18:22:52] you can write the cron files manually, but they can be easily generated [18:23:01] awight: puppet runs every 10-15 minutes, do you want that kind of delay? [18:23:03] they're gonna be basic af [18:23:05] I trust that you'll do whatever makes sense, so feel free to ignore me at any point [18:23:22] Jeff_Green: uhno :) [18:23:22] i mean i see what you are saying and i agree in principle [18:23:32] but it feels like either this concession or some other one [18:23:36] kk [18:23:42] but maybe... [18:23:43] no [18:23:50] right, we can't write hiera or anything cos root [18:23:53] we have to hack in some way to have the deploy script upload cron.d files [18:23:55] also the tweezing function should really be at most a couple dozen lines of code [18:24:08] which means making teh deploy script open a shell on the remote machine [18:24:11] and run some command [18:24:14] which is slightly fug [18:24:19] it's totally fine [18:24:46] we already have that feature in frdeploy, which we use in dash [18:24:52] ohya [18:24:57] I'm queasy about having two different codebases parse these .confs [18:25:05] awight: it should be the same codebase [18:25:14] process-control writes crontab? k [18:25:18] err [18:25:34] We'll also need scripts for useful things like * killing live processes [18:25:36] make parsing the the config modular, use it from both the wrapper and the cron-job-deployer [18:25:37] it can seriously be sed [18:25:42] * awight does not deliver littany [18:25:49] let it be sed :p [18:25:50] don't ever let it be sed... [18:25:52] heh [18:25:53] jinx [18:25:56] ha no yeah no [18:26:05] Jeff_Green: yep, parsing is reusable [18:26:10] awight: if it were me, the wrapper would have a --kill flag :-) [18:26:21] You can JobWrapper(config_path=FILE) and then do other things with that object [18:26:24] other than just run() [18:26:47] and I would do human interaction with the wrapper all on the machine in question [18:26:57] deployment box? [18:27:16] oh sorry, I meant i.e. on barium [18:27:43] hmm. so the process-control suite lives both on barium and on the deploy box? [18:27:45] ssh to barium, wrapper --kill civijobXYZ [18:27:56] nope, everything but the config lives on barium [18:28:04] humh [18:28:14] k gotcha [18:28:18] config would live there too, right? [18:28:33] but that means the deploy box will spray config to barium, then run a script on barium which produces crontabsen? [18:28:35] since jobwrapper reads it [18:28:37] config being /srv/jobs ? [18:28:56] awight: that is one option [18:28:56] +1 for /etc [18:28:59] ok here's what I was thinking in long form: [18:29:18] config goes in localsettings repo, as a new project, and gets deployed using the frdeploy tools [18:29:39] frdeploy has the restarter hook like dash does now [18:29:48] config contains: [18:29:54] cron_host = barium [18:30:04] cron_timing = 0 1 * * * * [18:30:13] and the rest of the job specs [18:30:23] iinteresting, I like the _host touch [18:30:36] all the python code gets shoved in a .deb which is deployed to every host where we might want to run cron jobs this way [18:30:55] (Do we .deb anything else?) [18:31:03] there's a script in the python code that knows how to parse the configs, and tweeze to /etc/cron.d/job_name with the proper timing & user [18:31:31] (python running as root?) [18:31:51] and there's the wrapper script that can be invoked by cron or by a person, and does everything referenced by job_name, getting its configs from i.e. barium:/srv/cronstuff/jobname.conf [18:32:12] the cron tweezer only would run as root [18:32:38] the /etc/cron.d/jobname files would always get the same user in them, like "www-data" or whatever [18:32:55] if we don't have to write command line flags etc to cron, the tweezer just needs to know the schedule and the conf file path, right? [18:33:15] (remember /etc/cron.d files are always processed by root, and there's an extra field for user) [18:33:20] ejegg: IMO yes [18:33:51] Jeff_Green: There's a user-permissioned crontab thing too, if we want that [18:34:30] ejegg yeah, but if it were me I'd prefer simple job names like "civi_job5" or whatever, which match the /srv/cronstuff/* filenames so you don't have to remember that path all the time [18:34:41] /var/spool/cron/crontabs/ [18:35:00] Jeff_Green: makes sense [18:35:08] +1 to the Vision [18:35:27] awight: yeah, but crontabs are harder to manipulate, and require crond HUP [18:35:28] I really like that we could use this on other hosts as well. [18:35:43] Jeff_Green: ah. /etc/cron.d doesn't? [18:35:46] so yeah, that kind of super-simple tweezing wouldn't even need python [18:35:50] awight: right [18:35:53] kk [18:36:08] ejegg: What are you imagining instead? [18:36:38] ejegg its true, but i don't see any reason not to use python since we're already using it to parse the config files [18:36:38] just a couple lines in the deploy script using e.g. sed [18:36:53] sedding .yaml does not sound like a win tho [18:37:06] eh, nbd [18:37:07] the tweezer script has to sanity check all the fields so we don't build a root escalation [18:37:32] which fields? [18:38:13] awight: si cron daemon will slurp up cron.d [18:38:23] so no need for aservice restart [18:39:30] ejegg for example a normal cron.d line looks like this: [18:39:49] 5-55/10 * * * * jgreen /usr/local/bin/partybus [18:39:56] whoohHHOOO [18:40:01] ding, ding [18:40:14] ah k, so just need to make sure the schedule doesn't have a trailing username [18:40:22] crontime = "5-55/10 * * * * root /do/something/bad;" [18:40:45] ya, also we should probably sanity check the syntax and email if a bad config goes out [18:41:16] true [18:41:55] yep [18:42:14] we ~could~ allow some degree of user flexibility too I guess [18:42:30] i.e. have a predetermined list of non-root users that can handle scheduled cron jobs [18:42:47] ehh, no need for that in first draft [18:42:48] but we don't need that now so i should keep my mouth shut :-P [18:42:57] I've been avoiding global config for the moment, yeah I'll probably just hardcode "jenkins" [18:43:19] we can just have the tweezer do that for now [18:43:20] awight: who killed the butler with the candle stick? that's who it should be [18:43:33] So many possibilities... [18:43:36] mean mr mustard [18:43:43] I haven't a clue [18:43:44] I never trusted the coloner [18:43:50] coronel [18:43:51] kernel [18:44:10] i like where this is going [18:44:17] user kernel_sanders [18:44:20] ha [18:46:04] https://en.wikipedia.org/wiki/Stephen_Root [18:46:29] awight: you asked earlier if we deb anything else, we do a few things and actually I just converted slander deployment to a deb [18:46:41] ohwow [18:46:44] How? [18:46:56] Did you write the debian packaging? [18:46:58] i built a deb :-P and told puppet to deploy it [18:47:00] yeah [18:47:06] kk. I'll do that for process-control [18:48:12] how does this sound: localsettings/process-control/*.conf [18:48:35] sounds good to me [18:49:43] awight: is it a pita to get new things into gerrit? [18:50:13] It's quite pleasant. [18:50:36] that was not the answer I was anticipating! [18:51:01] I was going to say something snarky about "after almost 5 years" but it actually is objectively easy. [18:51:06] nice [18:51:12] at some point can we move this project there? [18:51:21] If you'd ever like to do it yourself: https://www.mediawiki.org/wiki/Git/New_repositories/Requests [18:51:23] i understand not wanting the overhead early on [18:51:27] But I did already get process-control into gerrit. [18:51:34] https://gerrit.wikimedia.org/r/#/admin/projects/wikimedia/fundraising/process-control [18:51:37] oh! i missed that, that's excellent [18:51:42] :D [18:51:43] there is a gyro truck in denver called wiki pita [18:51:45] WINNING [18:52:00] now I'm hungry dammit [18:52:19] wiki as in "fast"? [18:52:32] it's got a few clevers going on [18:53:43] We need a wiki gyro now more than ever [18:53:58] street meat [18:55:29] oh also I forgot something sort of important [18:55:44] speaking of which... now that i've got my vagrantbox swapped out, i'mma venture forth from 'FAST INTERNET tranquil room' [18:55:48] back soon! [18:55:49] don't feed them after midnight [18:56:01] the process-control tweezer script should syslog changes, and possibly email about them, for better visibility [18:57:17] that's something that bugs me about jenkins now, it's not easy to figure out what happened when [18:57:41] ooh [18:57:43] ty [18:58:50] Jeff_Green: we will have it all in git too now [18:59:14] many improvements [18:59:31] yeah [18:59:46] but agreed that the transform should log too [19:00:23] While we're at it, might as well log each invocation of the wrapper, too. [19:04:09] awight: absolutely [19:10:37] cwd: Are you still unable to +2 in the new repo? [19:10:46] I don't understand why perms are not working for you [19:11:21] ah i haven't checked yet [19:11:45] (CR) Awight: [C: 2] "Thanks!" (1 comment) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342967 (owner: Awight) [19:12:17] cwd: I haven't changed anything, just read through the permissions [19:12:42] awight: i see v+2...maybe because you already c+2 [19:12:46] logged out and in again [19:13:12] awight: why does it say Author [19:13:15] Christian Aistleitner ? [19:13:16] You don't see CR+2? [19:13:31] That's cos he wrote that patch. QChris is the person who created the repo for us. [19:13:48] (CR) Cdentinger: [V: 2] Add .gitreview [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342967 (owner: Awight) [19:14:08] working on integration-config to automatically merge.. [19:14:17] oh! well thanks to him [19:15:11] pushed, remote: https://gerrit.wikimedia.org/r/343113 Run tox for new repo [19:15:40] yeah i don't see c+2 there [19:15:55] cwd: What patch are you looking at? [19:16:02] the one you just pasted [19:16:10] oh derp [19:16:15] integration [19:18:44] You could +1 that to raise eyebrows if you want :-) [19:28:06] ok nice, jenkins now shows which patch sets are closed in the 'Related Changes' list [19:28:11] *gerrit [19:53:05] (CR) Ejegg: [C: 2] Set street address to NULL where placeholder info has been used. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342783 (https://phabricator.wikimedia.org/T158268) (owner: Eileen) [19:57:26] (Merged) jenkins-bot: Set street address to NULL where placeholder info has been used. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/342783 (https://phabricator.wikimedia.org/T158268) (owner: Eileen) [20:06:02] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Slow query on live triggered by user dedupe - https://phabricator.wikimedia.org/T159752#3107743 (MBeat33) I spoke with Natalie about the more recent instance of this, and the only unusual things she may have done when merging were using the //back //but... [20:21:26] guys - am just on a call & being told to promote CiviCon St Louis - so consider this a promote! [20:27:12] (PS1) Ejegg: Restore autoload_static [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/343118 [20:27:33] cwd / awight ^^^ [20:29:57] (CR) Awight: [C: 2] "Do we also need one in crm/vendor ?" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/343118 (owner: Ejegg) [20:30:36] awight: I've been updating crm/vendor with oldcomposer that didn't make a _static file [20:30:54] but I guess it's a good time to update that! [20:31:03] ah cool, so it wouldn't cause an error [20:31:07] yep [20:31:31] I guess _static should be slightly more performant, though [20:32:55] (PS1) Ejegg: Update to using autoload_static [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/343119 [20:33:17] don't we not want that on master yet? [20:33:32] cwd should be fine [20:33:41] it was just the php53lint that failed [20:33:41] wouldn't it break if we deployed? [20:33:47] oh ok [20:33:59] nah, composer checks php version [20:34:10] and only uses _static on 5.6+ [20:34:55] (PS2) Awight: Tests for the lock module [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342968 [20:34:57] (PS2) Awight: test failopen; py2 [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342969 [20:34:59] (PS2) Awight: Correct failopen test [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342970 [20:35:01] (PS2) Awight: 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 [20:35:03] (PS1) Awight: Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 [20:35:05] (PS1) Awight: Fix flake8 typos [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343121 [20:35:37] (Merged) jenkins-bot: Restore autoload_static [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/343118 (owner: Ejegg) [20:36:21] cwd: [20:36:27] cwd: Do you have CR+2 on those? [20:36:32] (CR) Ejegg: "Looks pretty good! One question about test_live_lock" (1 comment) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342968 (owner: Awight) [20:37:04] awight: I see +/- 2 in that repo now [20:37:32] (CR) jerkins-bot: [V: -1] Update to using autoload_static [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/343119 (owner: Ejegg) [20:39:15] dumb thing, dying on polyfill [20:39:42] (PS3) Awight: Tests for the lock module [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342968 [20:39:44] (PS3) Awight: test failopen; py2 [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342969 [20:39:46] (PS3) Awight: Correct failopen test [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342970 [20:39:48] (PS3) Awight: 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 [20:39:51] (PS2) Awight: Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 [20:39:52] (PS2) Awight: Fix flake8 typos [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343121 [20:39:54] (CR) Ejegg: [V: 2 C: 2] Update to using autoload_static [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/343119 (owner: Ejegg) [20:40:02] cwd: If you happen to chat with Tyler... [20:40:14] We could use those jobs to merge process-control patches [20:40:49] (CR) Awight: Tests for the lock module (1 comment) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342968 (owner: Awight) [20:41:51] cwd: So let's design the tweezing stuff that belong in this repo, when you have a minute. [20:43:04] Are you thinking the CLI signature is like, covert-to-crontab.py /srv/jobstuff/*.yaml --destination-dir=/etc/cron.d ? [20:43:53] (CR) jerkins-bot: [V: -1] Update to using autoload_static [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/343119 (owner: Ejegg) [20:43:54] it could even just output to stdout, no? [20:44:12] if you wanted to minimize the code running as root [20:44:24] Maybe not, cos one of the options is that it crashes with an exit code when the input is invalid. [20:44:32] ah, k [20:44:58] and if you want to bake a batch of cron.d files at once, you need file output [20:45:58] (CR) Awight: "Ugh, we really need an ignore list for the linter." [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/343119 (owner: Ejegg) [20:46:22] I'm not too attached to the globbing tho [20:47:15] (CR) Ejegg: "yep. unfortunately, the linter is something like 'find -iname *php . | xargs php --lint'" [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/343119 (owner: Ejegg) [20:50:18] (PS1) Ejegg: Update civicrm subrepo [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/343123 [20:50:56] (CR) Ejegg: [C: 2] Update civicrm subrepo [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/343123 (owner: Ejegg) [20:56:00] (CR) Ejegg: [C: -1] "missing arg on second call" (1 comment) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342969 (owner: Awight) [20:56:25] (Merged) jenkins-bot: Update civicrm subrepo [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/343123 (owner: Ejegg) [20:56:39] (CR) Ejegg: "Oop, nvm, just saw the next patch" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342969 (owner: Awight) [20:57:54] ejegg: Shall I squash the fix in? [20:58:12] awight: no need, I guess, since the tests aren't blocking merge yet [20:58:23] kk [20:59:20] (PS1) Awight: More docs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343124 [21:00:11] awight: sorry my neighbor showed up [21:00:15] oh hey [21:00:18] and he's a talker [21:00:30] no worries, I can do .deb packaging if you're still busy [21:00:45] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/343125 [21:00:51] Did you tell him about the unfortunate thing that happened to his chihuahua? [21:01:13] heheh, i think he took her home [21:01:21] too little, too late... [21:01:51] haha [21:02:20] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/343125 (owner: Ejegg) [21:02:28] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/343125 (owner: Ejegg) [21:03:32] (CR) Ejegg: [C: 2] "Looks good!" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342968 (owner: Awight) [21:03:45] cwd ok, autoload_static should be all set to go. I'll just deploy it to the new target [21:04:15] ty! [21:06:20] !log updated new CiviCRM from cca5921a2a63346bb0d98dbaa5710952dafd1b21 to e058e8cf2377d6964eec909990e9f592275822c6 [21:06:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:07:11] getting a log in screen! [21:07:35] css is all messed up, but that's just baseurl [21:09:08] cwd looks like a winner [21:09:33] (CR) Ejegg: [V: 2 C: 2] Tests for the lock module [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342968 (owner: Awight) [21:09:40] sweet [21:09:48] (CR) Ejegg: [V: 2 C: 2] test failopen; py2 [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342969 (owner: Awight) [21:09:57] (CR) Ejegg: [V: 2 C: 2] Correct failopen test [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342970 (owner: Awight) [21:11:36] cwd: When you have a minute, can we talk about the tweeze-to-cron step? [21:12:50] awight: sure, my though was a small script included in the .deb [21:19:27] cwd / awight, so these jobs will output to remote logger dir & get logrotated like the syslog stuff does now? [21:19:41] cwd: yeah, I'm planning to do a script parallel to crash-override [21:19:49] what should the CLI signature look like? [21:20:19] awight: it would almost be nice to have a global config [21:20:27] cwd: We can do that [21:22:46] (CR) Ejegg: "(not this patch) If lock.end() is in the finally section, will failopen actually work?" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 (owner: Awight) [21:23:04] awight: what about a big old 'enabled' flag in that file as well? [21:23:18] nice, soft shutdown [21:23:28] yea [21:24:21] cwd: enabled, defaults to true? [21:24:27] also nice, but not MVP: enabled flag for job groups, so we can for example just shut off things that touch civi DB [21:24:47] woot! multiple tags per job [21:25:24] awight: in that particular case i feel like it should be off by default and only on if the job itself is on and the global flag [21:25:54] i'mma make phab tasks for these non-MVP features [21:25:57] like an additional circuit breaker [21:28:19] That sounds good for us, but incredibly annoying in the unlikely event anyone else wants to use this. [21:28:35] Fundraising-Backlog: process-control should be able to disable groups of jobs - https://phabricator.wikimedia.org/T160699#3108169 (Ejegg) [21:28:35] why do you mean? [21:28:45] like, a quick way to turn off all the jobs [21:28:49] But I like that you might have solved the problem of, turning a job off by removing the conf file, which then leaves a cron entry behind. [21:29:02] Oh the global thing is good [21:29:12] but remembering to add "enabled: true" to everything feels like fluff [21:29:48] (PS1) Awight: [WIP] Debian packaging [wikimedia/fundraising/process-control] (debian-packaging) - https://gerrit.wikimedia.org/r/343129 [21:30:07] how else would you turn off one job? delete the cron.d file? [21:30:15] chmod 000 [21:30:28] Maybe "disabled: true" or better yet, by commenting out the schedule. [21:30:28] Fundraising-Backlog: process-control should have a mode to display all the things it's currently running - https://phabricator.wikimedia.org/T160700#3108184 (Ejegg) [21:30:43] w/o a schedule, a job can only be run one-off [21:31:26] i like the 'disabled' flag [21:31:41] hehe I like commenting out the schedule [21:31:52] works better for the groups, too [21:31:58] i mean it's the same fact as the enabled flag [21:32:10] but semantically better [21:32:16] cwd: hehe chmod. you're in full devops mode now [21:32:34] k I'll do disabled them [21:32:36] *then [21:32:56] i thought devops meant you had several corporate backed software stacks between you and chmod [21:33:04] haha [21:33:13] each with its own conference [21:33:28] I'm thinking we remove cron.d/job if disabled:true or !schedule [21:33:43] cwd: But tell me more about what you want the tweezer CLI to look like [21:33:52] sorry to repeat myslef [21:34:47] pc-cron-generate /srv/jobs [21:35:16] my thought was it gets installed by the deb and called by the deploy script as root [21:35:34] if the server has the process-control project [21:35:59] cwd: When you say installed by the deb, I'm planning to just plop it right into the repo unless you thinking something else... [21:36:10] yeah totally [21:36:20] so you give pc-cron-generate a source directory but not the dest directory? [21:36:20] sounds good [21:36:36] or it could have a config file [21:36:57] i mean it's probably safe to assume /etc/cron.d for our purposes [21:37:01] but feel free to future proof [21:37:15] Can we do /etc/cron.d/fundraising? [21:37:24] fine with me [21:37:34] I mean, does that work for crond discovery? [21:37:47] Cos I worry about having a job with the same name as a system thing. [21:38:10] awight: i don't think there's any system stuff in there [21:38:39] eh maybe there is [21:38:49] cwd: hey. How about we pipe everything into a single crontab under cron.d? [21:39:02] that's fine too [21:39:02] That moots dealing with stale jobs. [21:39:05] ok [21:39:26] yeah the cron tab entries are really simple now b/c of the config file [21:39:37] excellent. [21:39:44] And sometimes, it's nice to see them all at once. [21:39:50] yeah [21:39:50] yeah, and helps keep track of which ones are controlled by crash-override [21:40:15] yeah agreed, much better than a bunch of one line files [21:41:51] well it looks like i pretty much delegated myself out of a job [21:41:54] * cwd kicks feet back [21:42:02] cwd: what do you think about a signature of "pc-cron-generate /srv/jobdir > /etc/cron.d/process-control" ? [21:42:19] Not necessary, I'm fine hardcoding the dir in there [21:42:38] ah yeah that's nice and unixy [21:42:42] :) [21:43:05] awight: integration thing gon merge shortly [21:43:34] whee [21:43:38] thanks [21:53:24] (CR) Ejegg: "tests look good, nitpick about file naming" (1 comment) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 (owner: Awight) [21:59:19] (PS2) Ejegg: WIP Test more fields on import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 [22:05:20] (CR) jerkins-bot: [V: -1] WIP Test more fields on import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [22:21:27] (PS1) Awight: Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 [22:21:29] cwd: ^ [22:23:05] great [22:23:07] ty [22:24:43] (CR) jerkins-bot: [V: -1] Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 (owner: Awight) [22:26:01] awight: i noticed crash-override prints to stderr [22:26:10] i thought it was going to eat all that and do its own emailing? [22:28:01] I could do that [22:28:14] I'd like to get this pilot run going though [22:28:20] so we might want to limit scope somewhere [22:28:34] sure [22:28:44] so fail mail on failure [22:28:45] I mean, I'm just stating the fact of my own laziness [22:28:51] cron will email on failure, right? [22:28:54] nah time is a concern [22:29:01] yeah but the cron fail formatting is garbage [22:29:03] ejegg: cron will email on any output whatsoever [22:29:07] esp the subject line [22:29:24] we really only want to to email on non-zero exit right? [22:29:29] something we can start with, though? [22:30:51] We should have a more definitive talk about failure conditions, IMO [22:31:22] my concern is that there may be a whole ton of stuff going to stderr right now but still exiting 0 [22:31:25] I'm leaning towards, non-zero exit means that the job failed in a unhandled and unrecoverable way and we should disable the job until future intervention [22:31:34] cwd: maybe--let's find out ;) [22:31:40] hrm, but we don't do that now [22:31:44] right [22:31:46] because a lot of them recover [22:32:15] i wouldn't want to disable the donation consumer over one failure in dec [22:32:19] and we don't do a great job of catching all recoverable errors [22:32:20] at 2am [22:32:24] yup [22:32:44] we can code ease-off or whatever we need [22:32:49] but i would want to disable it over 50 in a row [22:32:56] yeah [22:33:15] but since it's not dec maybe that is a fine way to start [22:33:49] awight I think failopen needs the lock.end() call someplace that's not the finally: block [22:34:20] ejegg: I hadn't looked at your comment yet... lemme see [22:35:59] (PS1) Awight: "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 [22:36:48] (CR) jerkins-bot: [V: -1] "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 (owner: Awight) [22:36:58] oh boy, CI! [22:37:41] :D [22:37:53] nice [22:39:48] ejegg: I don't think there's a problem with failopen cos * try is not around the lock.begin, and * we don't actually use failopen [22:40:09] (PS4) Awight: 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 [22:40:11] (PS3) Awight: Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 [22:40:13] (PS3) Awight: Fix flake8 typos [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343121 [22:40:15] (PS2) Awight: More docs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343124 [22:40:17] (PS2) Awight: Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 [22:40:19] (PS2) Awight: "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 [22:40:55] awight but doesn't the finally: block mean we'll delete the lockfile however the job exits? [22:41:25] so, nothing will be around to indicate previous failure if we want that to halt the job? [22:41:33] (CR) jerkins-bot: [V: -1] 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 (owner: Awight) [22:41:54] (CR) jerkins-bot: [V: -1] Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 (owner: Awight) [22:42:06] ejegg: I don't understand. failopen just means, if there's a stale lockfile (owning PID is dead), don't remove it and throw an error [22:42:22] (CR) jerkins-bot: [V: -1] Fix flake8 typos [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343121 (owner: Awight) [22:42:29] (CR) jerkins-bot: [V: -1] Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 (owner: Awight) [22:42:33] the default, failopen=False, is to remove the stale lockfile and otherwise behave normally. [22:42:34] (CR) jerkins-bot: [V: -1] More docs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343124 (owner: Awight) [22:42:37] (CR) jerkins-bot: [V: -1] "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 (owner: Awight) [22:42:50] nice. failed assertions with no content [22:44:27] awight: oh, failopen is only for when the wrapper itself crashes somehow? [22:44:58] failopen is an extra-cautious mode that we might sometimes want [22:45:18] yeah exactly--so there are probably several ways to end up with a stale lockfile [22:45:23] but most of them are Bad [22:45:26] k [22:45:50] my thought (many years ago) was, we might sometimes want to use a mode where a stale file forced a manual witchhunt [22:46:08] I should probably remove it cos it's unused complexity... [22:46:09] try/finally - the finally will run even if there's an uncaught exception? [22:46:21] cwd: yep [22:46:32] but IMO that finally clause is not really involved in what we're discussing [22:46:49] oh yeah i've just never seen that [22:47:05] it makes me wonder how exceptional exceptions really are [22:47:26] haha [22:47:34] This one's a *really* special exception [22:48:34] i can see the usefulness [22:48:36] How do I ssh to the CI nodes, now... [22:48:42] cleaning up state before exit [22:48:53] but it's a pretty mushy call stack [22:48:53] exactly [22:49:05] yeah, you gotta be careful in the finally [22:49:16] vars from inside the try might not be defined, and so on [22:49:32] and you don't want to error out of the finally :) [22:50:13] i bet there's a lot of people who regret writing code that uses finally to retry the whole thing [22:53:50] (PS5) Awight: 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 [22:53:52] (PS4) Awight: Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 [22:53:54] (PS4) Awight: Fix flake8 typos [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343121 [22:53:56] (PS3) Awight: More docs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343124 [22:53:58] (PS3) Awight: Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 [22:54:00] (PS3) Awight: "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 [22:58:22] (CR) jerkins-bot: [V: -1] Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 (owner: Awight) [22:58:37] (CR) jerkins-bot: [V: -1] Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 (owner: Awight) [22:59:08] (CR) jerkins-bot: [V: -1] "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 (owner: Awight) [22:59:19] * awight does a flying high-five due to one of the patches passing tests [23:00:17] (PS3) Ejegg: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 [23:00:52] those import tests need a bunch of cleaning up, but that patch should catch things like the blank name regression ^^^ [23:01:04] awight: i get a yaml error on the cron schedule format [23:01:24] cwd: U gotta quote it [23:01:27] cramming it in quotes [23:01:29] yeah [23:01:36] * awight fixes REAME [23:01:40] ahem [23:01:42] :) [23:01:44] ty [23:02:08] Are we still using the orphan rectifier as our pilot job? [23:02:09] Neal Stephenson's REAMDE was a fun read [23:02:24] sounds like a good test to me! [23:02:44] looks like a fun book! [23:03:35] (CR) Ejegg: [C: 2] 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 (owner: Awight) [23:03:50] thanks [23:04:07] cool, gate-and-submit job kicked off even [23:04:14] thinking again that it might be easier to have the jobs be in their own repo [23:04:20] (Merged) jenkins-bot: 100% test coverage [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/342971 (owner: Awight) [23:04:26] but a private one on the puppet master [23:04:42] cwd wait, fr-tech needs access [23:04:51] yeah, like localsettings [23:04:54] (PS5) Awight: Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 [23:04:56] (PS4) Awight: More docs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343124 [23:04:57] ah, k [23:04:58] (PS4) Awight: Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 [23:05:00] (PS4) Awight: "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 [23:05:02] (PS1) Awight: Note that you have to quote crontab due to its punctuation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343212 [23:05:16] just the rsync stuff is wired to tweeze one filename [23:05:27] not a while dir, plus run this command [23:05:28] * ejegg just submitted a gargantuan pull request to php-queue [23:05:33] but it should be pretty trivial in puppet [23:05:38] ejegg: ooh [23:05:49] heh, I really should break it up [23:06:07] (CR) jerkins-bot: [V: -1] Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [23:06:08] just a little hard with the commits from our repo interleaved without respect to topic [23:07:09] You could wait for feedback from coderkungfu, I guess [23:07:11] grr, assertArraySubset output is way less useful than assertEquals for arrays [23:07:40] awight: yeah, I just put a note in the PR saying I could break it up if needed [23:09:05] XenoRyet: how does that CRM patch look? think isset($msg['payer_email']) is ok for old-message-detection? [23:09:20] (PS1) Awight: DO NOT MERGE: instrument the damn test [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343213 [23:09:53] Ah, right. I was futzing with other stuff. Let me take a look. [23:10:05] ah, I see it doesn't have you on the list. link: https://gerrit.wikimedia.org/r/342896 [23:10:25] there's also this one for the audit parser: https://gerrit.wikimedia.org/r/342693 [23:10:41] Cool. I'll take a look at that too. [23:11:39] cwd: ejegg: oh hey, we got V+2 for the remainder of the patch chain [23:11:56] woot! [23:13:35] (PS4) Ejegg: WILLFAIL: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 [23:16:29] (Abandoned) Awight: DO NOT MERGE: instrument the damn test [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343213 (owner: Awight) [23:17:12] cwd: What am I doing now... [23:17:38] (CR) jerkins-bot: [V: -1] WILLFAIL: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [23:17:43] Should we try this thing? [23:17:54] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, and 6 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#2929493 (Ejegg) Wouldn't Vary: Coo... [23:19:05] (PS5) Ejegg: WILLFAIL: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 [23:22:15] (CR) Ejegg: [C: 2] Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 (owner: Awight) [23:23:10] letsgoletsgoletsgo! [23:23:15] hehe [23:23:48] (CR) jerkins-bot: [V: -1] WILLFAIL: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [23:24:51] (CR) Ejegg: [C: 2] More docs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343124 (owner: Awight) [23:24:53] (Merged) jenkins-bot: Config validation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343120 (owner: Awight) [23:25:57] (Merged) jenkins-bot: More docs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343124 (owner: Awight) [23:26:33] (PS6) Ejegg: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 [23:32:15] (CR) Ejegg: [C: 2] Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 (owner: Awight) [23:32:24] * awight does a jig [23:32:58] (Merged) jenkins-bot: Rough pass at crontab tweezer [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343171 (owner: Awight) [23:34:17] (CR) Ejegg: [C: 2] "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 (owner: Awight) [23:34:27] (CR) Ejegg: [C: 2] Note that you have to quote crontab due to its punctuation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343212 (owner: Awight) [23:35:57] Whew, extra fields tests for import finally passes: https://gerrit.wikimedia.org/r/340885 [23:36:12] (Merged) jenkins-bot: "disabled" flag [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343195 (owner: Awight) [23:36:30] (Merged) jenkins-bot: Note that you have to quote crontab due to its punctuation [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343212 (owner: Awight) [23:37:18] ejegg: Want some CR? [23:37:37] sure :) [23:38:47] once that merges, i'mma rebase https://gerrit.wikimedia.org/r/342896 on top of it, then make a new patch to split the import tests up by consumer & clean up better [23:43:00] (CR) Awight: Test more fields on donation import (4 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [23:51:23] (PS7) Ejegg: Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 [23:51:35] (CR) Ejegg: Test more fields on donation import (4 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [23:56:14] (CR) Awight: [C: 2] Test more fields on donation import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [23:57:17] thanks awight ! [23:57:34] cwd are we set to try out crash-override? [23:57:45] I have to run but lmk how that goes! [23:58:17] ejegg: well there is packaging and deployment to figure out still [23:58:20] (PS1) Awight: [WIP] Debian packaging [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343218 [23:58:44] (PS2) Awight: [WIP] Debian packaging [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/343218 [23:59:44] ah, word [23:59:56] anything I can help with?