[00:09:05] (PS1) Ejegg: WIP cache directory lookup 404s [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/345790 (https://phabricator.wikimedia.org/T161072) [02:20:24] (PS2) Ejegg: Cache directory lookup 404s [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/345790 (https://phabricator.wikimedia.org/T161072) [02:23:57] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, FR-Smashpig, MediaWiki-extensions-DonationInterface, and 2 others: iDEAL bank lookup should also cache failures - https://phabricator.wikimedia.org/T161072#3145657 (Ejegg) a:Ejegg [03:31:10] (PS3) Ejegg: Merge environment, don't clobber [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345768 [03:31:39] (CR) jerkins-bot: [V: -1] Merge environment, don't clobber [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345768 (owner: Ejegg) [04:36:40] (PS4) AndyRussG: Remove "too many banners" warning [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/247785 (https://phabricator.wikimedia.org/T109714) (owner: Awight) [15:52:03] fundraising-tech-ops, Patch-For-Review: rename backup4001 to frbackup4001 and jessie reimage - https://phabricator.wikimedia.org/T158220#3146838 (Jgreen) [15:52:06] fundraising-tech-ops, Operations, ops-ulsfo: upgrade backup4001 hard disk array - https://phabricator.wikimedia.org/T157473#3146836 (Jgreen) Open>Resolved resolving this task because the hardware is full of fail, we're procuring a new system for codfw instead [15:54:10] fundraising-tech-ops: replace backup4001 with a new machine at codfw - https://phabricator.wikimedia.org/T161903#3146847 (Jgreen) [15:54:42] fundraising-tech-ops: replace backup4001 with a new machine at codfw - https://phabricator.wikimedia.org/T161903#3146859 (Jgreen) [15:55:31] fundraising-tech-ops: replace backup4001 with a new machine at codfw - https://phabricator.wikimedia.org/T161903#3146847 (Jgreen) [15:57:48] fundraising-tech-ops, Patch-For-Review: rename backup4001 to frbackup4001 and jessie reimage - https://phabricator.wikimedia.org/T158220#3146868 (Jgreen) >>! In T158220#3142515, @Dzahn wrote: > @Jgreen Thanks for your comment on the gerrit change above. I amended again to remove the host from DHCP. So th... [15:58:10] fundraising-tech-ops, Patch-For-Review: rename backup4001 to frbackup4001 and jessie reimage - https://phabricator.wikimedia.org/T158220#3146869 (Jgreen) Open>Resolved [15:59:52] fundraising-tech-ops: replace backup4001 with a new machine at codfw named frbackup2001 - https://phabricator.wikimedia.org/T161903#3146847 (Jgreen) [16:09:14] https://github.com/antirez/redis/issues/3909 [16:13:43] Jeff_Green: hi! not sure if you pulled code from puppet yet but i made a bunch of changes in there [16:14:06] yep! [16:14:18] did you figure out the packaging snafu? [16:14:36] yep, got it all working and ran a couple successful jobs! [16:14:39] nice [16:14:48] i was just looking at T161569 [16:14:48] T161569: [Epic] Basic process-control good enough to run all CRM jobs - https://phabricator.wikimedia.org/T161569 [16:15:24] ah, i reset perms on run-job so that last one should be good too [16:15:37] in the package? [16:15:40] it's 550 and group jenkins [16:15:44] Jeff_Green: in puppet [16:15:50] i wasn't sure which would be better [16:15:57] it requires the package [16:16:00] right [16:16:56] this is an interesting question [16:17:19] i usually try to avoid having puppet undo things that packages do [16:17:54] thinking [16:18:07] yeah i was torn [16:18:27] cause chgrp to a specific user doesn't feel like package scope [16:18:36] yeah [16:18:44] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice: Remove unused code for banner preview in banner editor - https://phabricator.wikimedia.org/T161906#3146928 (AndyRussG) [16:19:00] it's not that uncommon to have a package make something only root executable, but doing that for an arbitrary user is a little weird [16:19:16] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice: Remove unused code for banner preview in banner editor - https://phabricator.wikimedia.org/T161907#3146941 (AndyRussG) [16:19:31] yeah, cause then should the package also add the user? [16:19:36] right [16:19:43] it's a gray area for sure [16:20:28] it's also absurd that we're going with user jenkins :-P [16:20:39] understandable, but absurd [16:20:58] totally [16:21:07] hysterical raisins [16:21:22] in other cases I've moved things out of the path, and made puppet drop a wrapper script to handle the sudo part [16:21:31] that's how the frdeploy stuff works [16:21:44] ah yes [16:21:49] that might be a saner way to do this [16:24:44] the one that generates /etc/cron.d -- seems appropriate in /usr/sbin since it's intended to be run as root [16:25:30] the other one we actually don't want run as root... [16:27:27] what about making that one read global conf, and bail if it's not being run as the intended user? [16:28:01] yep that would work too [16:28:48] yeah agreed about cron.d, one thing that came up is rssh is restricted to only acouple commands [16:28:54] (PS1) AndyRussG: Comments flagging possibly unused code for inline banner previews [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345862 (https://phabricator.wikimedia.org/T161907) [16:28:59] so i ran the generator manually yesterday [16:29:05] wanted to see how you wanted to handle that [16:29:59] take a look at puppet/modules/role/manifests/fundraising_qa.pp -- search for frdeploy [16:30:24] frdeploy::mirror [16:30:29] service_restart => [ [16:31:05] if that class variable is present, the frdeploy user gets a real shell [16:31:11] ah right on [16:31:28] gotta hang laundry, brb... [16:31:29] and it also sets up sudo access for the commands listed there, so the nonroot remote user can run those commands [16:31:32] k [16:41:18] back [16:41:23] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: rack and cable frdev1001 - https://phabricator.wikimedia.org/T159887#3147024 (faidon) [16:44:04] Jeff_Green: if service_restart is set that's when it gets a shell? [16:44:21] yes [16:44:36] so we could hijack that for cron-generate? [16:44:43] it's more or less a service restart [16:44:49] yes, it is [16:44:57] it's not even a hijack imo :-P [16:45:28] yeah that's fair [16:45:56] so i see that set in frdeploy init.pp [16:46:06] service-restart [16:46:41] yes, that's the other end of it [16:47:01] so here's what I suggest [16:47:52] 1) fix the package to move cron-generate to /usr/sbin/cron-generate [16:48:37] 2) fix the package to move run-job somewhere TBD... [16:49:34] 3) create wrappers for both in puppet/module/process_control/templates/*erb (see frdeploy/templates for examples) and make those handle the sudo to root or jenkins as appropriate [16:49:48] yeah i like the sound of that [16:49:51] 4) have puppet install #3 in /usr/local/bin [16:51:19] 5) fix frdeploy/manifests/init.pp from 'service-restart' => '/usr/bin/sudo /usr/bin/cron-generate', to 'service-restart' => '/usr/local/bin/cron-generate', [16:52:11] 6) add '/usr/local/bin/cron-generate' to that frdeploy restart class variable in modules/role/manifests/civicrm.pp [16:53:08] oh right and 0) fix run-job in python to bail if it's called as a user other than the one listed in the global config [16:53:31] anything missing? [16:54:36] sounding pretty good from the ops perspective [16:55:06] i think there is still a bunch of hacking necessary for the logging to work right [16:55:25] sad to say i think we should have written the wrapper in bash [16:56:21] i can't believe how much nastiness has been required to get 2>&1 working [16:56:34] it didn't look to me as though they really figured out the open3 style handling [16:56:39] i have a familiar feeling that it has everything to do with windows compat [16:57:20] Jeff_Green: yeah it seems like the only way to do this in python is to open several new threads [16:57:27] which seems like insane overcomplication to me [16:58:40] maybe we should have stuck with perl :-) [16:58:57] yeah [16:59:02] (PS1) AndyRussG: Add FIXME comments in centralnotice.js [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345864 [16:59:10] I already have that figured out in perl, thanks to help from someone much smarter than me, long ago... [17:00:19] fr-tech: Most people deserve each other. [17:00:19] -- Shirley [17:00:19] -- discuss. [17:00:26] it sounded like adam had some good leads [17:02:30] but i have found so many insane solutions to variations of this question [17:02:44] Jeff_Green: http://pythonhosted.org/sarge/overview.html#what-is-sarge-for [17:31:32] (CR) Cdentinger: Cache directory lookup 404s (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/345790 (https://phabricator.wikimedia.org/T161072) (owner: Ejegg) [18:23:25] (PS6) XenoRyet: Mark refunds with correct gateway [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/345189 (https://phabricator.wikimedia.org/T161121) [18:28:32] Wikimedia-Fundraising, Technical-Debt: Refactor fundraising banner code - https://phabricator.wikimedia.org/T161608#3147290 (Pcoombe) Progress made, these are being checked now and should hopefully be rolled out soon. - Desktop large: https://en.wikipedia.org/wiki/Albert_Einstein?banner=peter_refactor... [18:55:04] (PS1) AndyRussG: Admin UI: Consolidate and refactor client-side bucket change handler [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345872 [18:57:24] (PS2) AndyRussG: Admin UI: Consolidate and refactor client-side bucket change handler [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345872 [19:18:25] (PS1) AndyRussG: Admin UI: delay setup until document ready in campaignManager.js [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345873 [20:01:35] (PS14) AndyRussG: [WIP] Custom mixin param handlers [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/343953 [20:24:15] (PS2) AndyRussG: [WIP] Banner sequence campaign mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/344988 [20:27:50] (CR) jerkins-bot: [V: -1] [WIP] Banner sequence campaign mixin [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/344988 (owner: AndyRussG) [20:36:44] Fundraising-Backlog: process-conotrol jobs should target a host - https://phabricator.wikimedia.org/T161931#3147574 (cwdent) [21:21:11] (PS1) Umherirrender: Use tabs to indent in json [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345899 [21:25:16] (PS1) Umherirrender: Use tabs to indent in json [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/345907