[01:13:52] Fundraising-Backlog, fr-donorservices: 11/20 Email donation link pointed to “foundation.wikimedia.org/wiki/Home" - https://phabricator.wikimedia.org/T268502 (MBeat33) p:Medium→Low [01:17:26] Fundraising-Backlog: JCB card not an option for recurring donation in Japanese form - https://phabricator.wikimedia.org/T126268 (MBeat33) Open→Resolved a:MBeat33 Old and resolved - JCB on form now [07:01:49] fundraising-tech-ops, Operations, netops: Manage frack switches with Netbox - https://phabricator.wikimedia.org/T268802 (ayounsi) [09:01:18] (PS1) QChris: Allow “Gerrit Managers” to import history [wikimedia/fundraising/dev] (refs/meta/config) - https://gerrit.wikimedia.org/r/643684 [09:01:23] (CR) QChris: [V: +2 C: +2] Allow “Gerrit Managers” to import history [wikimedia/fundraising/dev] (refs/meta/config) - https://gerrit.wikimedia.org/r/643684 (owner: QChris) [09:01:25] (PS1) QChris: Import done. Revoke import grants [wikimedia/fundraising/dev] (refs/meta/config) - https://gerrit.wikimedia.org/r/643685 [09:01:29] (CR) QChris: [V: +2 C: +2] Import done. Revoke import grants [wikimedia/fundraising/dev] (refs/meta/config) - https://gerrit.wikimedia.org/r/643685 (owner: QChris) [16:45:20] any fr-techers around today? [18:42:02] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Wiki-Loves-Monuments, Wikimedia-production-error: It should not be possible to run a banner campaign with a syntax error - https://phabricator.wikimedia.org/T268792 (AndyRussG) [19:43:25] AndyRussG: do you take today off for thanksgiving? [19:49:03] hey eileen [19:49:13] hey jgleeson [19:49:35] how goes [19:50:13] good good, you? [19:51:27] I had a frustrating day with the docker stuff yesterday & now I'm struggling with my touchpad (I need the latest version of 'disable touchpad when typing but all posts seem old) [19:51:59] oh that sucks [19:52:00] I tried to integrate the stuff I had with Andy's docker yesterday but didn't go forwards at all [19:52:43] ah, tricky isn't it. I'm currently dockerizing process-control [19:53:48] getting there slowly [19:55:26] (PS1) Jgleeson: WIP: Dockerize process-control [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/643767 (https://phabricator.wikimedia.org/T268685) [19:55:30] yeah - there are some different assumptions in AndyRussG's to the other - I don't want to fight the civi tools so I like the way the civi one creates a user with the same UID & a group/user with the same GID [19:55:56] (CR) jerkins-bot: [V: -1] WIP: Dockerize process-control [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/643767 (https://phabricator.wikimedia.org/T268685) (owner: Jgleeson) [19:56:09] eileen: jgleeson: hi! [19:56:17] AndyRussG: hey [19:56:27] I feel like I just went in circles yesterday. [19:56:39] I am here-ish... Not fully taking the day off, certainly available to chat or even do a call if useful [19:56:45] eileen: oh sorry to hear that! [19:57:03] well I don't need to work on it today - but I just felt that I didn't get anywhere [19:58:03] eileen: hmmm sorry that happened... in what way was it frustrating? (sorry also don't mean to interrupt if you're working on something else) [19:58:05] hi AndyRussG :) [19:58:15] jgleeson :) [19:59:11] yeah I'll probably "officially" take today and tomorrow off, but in practice work some to make up for not having gotten enough hours in earlier in the week [20:00:57] eileen jgleeson I could try to do a bit on the Civi Docker task at least to maybe experiment in trying to get it working in a way that's parallel to what's already there for Payments [20:01:03] yeah I feel like I missed a few hours so I was weighing up working on it today - but also doubting I'd do more then go in the same circues [20:01:58] https://www.irccloud.com/pastebin/ADXQCPVD/ [20:02:01] definitely there are different assumptions from what's in the existing Civi docker image... so maybe if I try to build something at least it'll be a way of seeing if it makes any sense at all to try to apply those assumptions to the civi side of things [20:02:32] I think the main differences between the cividocker are [20:02:32] 1) some extra extensions & modules - one is sudo which I realise might be controversial but I suspect it will be otherwise uphill [20:02:39] (I also added less) [20:03:26] I AM running cividocker more as a virtual box in several ways - but civibuild comes with all sorts of tools & bits & pieces [20:06:10] eileen: hmm right... do you know what sudo is actually used for? [20:06:16] maybe for setting Apache config? [20:06:42] yeah definitely that - I'm not too sure beyond that [20:06:44] In the Payments stuff, there are a few things that are normally sudo-needing that we get around by setting permissions differently i nthe images [20:06:54] so it might be avoidable I guess [20:07:35] I like the way it creates a user with the same UID & GID as local - since I normally do do stuff from the box it 'feels better' [20:09:12] AndyRussG: I think we have an issue with the perms as they are at current that we'll have to look at. We currently can't use volume mounts as we're running fundraising-dev as a non-privileged user (which comes up quite a bit in the docs). I ran into an issue on this on this PR https://gitlab.com/andyrussg/fundraising-dev/-/merge_requests/4 [20:09:35] hmmm right... definitely user rights have been a topic [20:12:15] jgleeson eileen so here we change permissions on stuff when we create the image, which does run stuff as root, so that when it's containerized it doesn't have to do anything as root: https://gerrit.wikimedia.org/r/plugins/gitiles/releng/dev-images/+/027ba17fd8354a108cf5679211fd128260063f80/dockerfiles/fundraising-buster-php73-apache2-xdebug/Dockerfile.template#85 [20:12:29] If you search the Dockerfile.template for "chmod" you'll find a bunch more [20:12:47] several of which were copied from the general MediaWiki dev image setups [20:13:27] jgleeson wrt to the volumes vs bind mounts, what was the advantage of volumes again? [20:13:40] yeah - I think one think the buildkit script does is restart apache with sudo [20:14:18] eileen K hmmm in Payments we have apache running as non-root, so no need for sudo for that [20:14:29] the only requirement was to use a high port number [20:15:01] the only time we really couldn't get around root was for rsyslog to bind to a Unix socket in order to receive syslog messages sent by DonationInterface and Smashpig [20:15:14] yeah - I expect we can find a way to get it out of the buildkit command [20:15:18] In that case, however, rsyslog has a feature to reduce priviliges after startup [20:15:26] so we set the suid bit on rsyslog [20:15:33] and it starts up as root, binds to the socket [20:16:15] and then reduces privileges to the "normal" user, which is set via the .env file to be the same user as the one who initially ran the setup.sh script [20:16:59] I know it's right - it just feels hard when we are dealing with a dev environment [20:17:34] the main reason for not running things are root, other than security (i.e. if we're downloading random composer packages or something inot the box) is so that the files created on the host won't be owned by root on the host [20:18:14] eileen I think also probably as a first step having sudo in there is fine, just to get things initially up and running [20:18:32] AndyRussG: OK - well that's nice :-) [20:18:38] ;) [20:18:51] I can't recall where I fell apart yesterday - I didn't fight that battle [20:19:06] but, I also didn't get it working & wound up going back to what was working [20:19:14] jgleeson hey also looking at the pull request, did you check that the directories for the volumes on the host didn't somehow become owned by root on the host? [20:20:11] (I did push up what I had to my branch) [20:20:12] eileen I should probably try to dig in more today... I have a couple hours I could use on this now, actually, since I did some of the cooking for today yesterday evening;p [20:20:14] AndyRussG: i guess the reason we'd use volume mounts for internal data of things like mysql and redis is that it remove a couple of folders in the project of files we don't really need to see or edit. there are actual dockerish benefits outlined here https://docs.docker.com/storage/volumes/ [20:20:29] AndyRussG: well I'm happy to work on it with you if you want [20:21:12] eileen cool yeah! [20:21:15] As an aside - I rather liked exposing the mariadb container on port 3306 so we can connect to it as if it's local - although I'm using 33306 atm [20:21:41] eileen ah right I was thinking of making that configurable [20:21:43] in our use-case, I feel like bind mounts would apply to files we want to change during development and internal data of services stored in their own obscure format doesn't fit into that category [20:21:54] jgleeson so then they wouldn't be accessible at all on the host? [20:21:57] does that make sense? [20:22:09] I kinda liked having them on the host so you could like erase them easily [20:22:16] oh yeah andy [20:22:22] you can manage them with `docker volume` [20:23:54] jgleeson ah hmm interesting [20:24:52] AndyRussG: but erase easily is a bit double edges on mysql (& to a lesser extent redis) [20:25:25] eileen well it'd just be for when you really want to start things from scratch [20:26:19] eileen: AndyRussG it's pretty easy to back up volumes although I guess the idea is with docker that we can throw away stuff pretty easily in the confidence that we can rebuild it in a flash if needed [20:26:58] eileen jgleeson what would the standard procedure be for checking out all the civi code and extensions via git? Akin to lines 89 to 108 in setup.sh: https://gitlab.com/andyrussg/fundraising-dev/-/blob/master/setup.sh [20:26:59] the worlflow is almost build a new stack every time you're testing a new feature [20:27:34] jgleeson: could be, or maybe reset all your db... Had the idea we should have a standard set of test data somewhere for... everything [20:28:19] jgleeson eileen also, what is the standard place logs are sent in our Civi setup? Is it also syslog, like on Payments? [20:28:38] hmm they end up in tmp [20:28:47] I think they can be configured to write to syslog [20:29:05] jgleeson hmmm but in current setup they go to tmp? [20:29:22] I think SmashPig always sends stuff to syslog and we found that hard to change on Payments [20:29:37] which is why we also had to run rsyslogd in the Payments container [20:30:05] I imagine that SmashPig library that is used by Civi would do the same, since it actually shares config IIRC [20:31:03] jgleeson though also the idea of being able to easily switch in and out different DB contents, or share them around the team, sounds appealing! [20:31:11] AndyRussG: so basically it is [20:31:11] git clone "https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm/civicrm-buildkit" && (cd "civicrm-buildkit" && mkdir -p .git/hooks && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit.wikimedia.org/r/tools/hooks/commit-msg; chmod +x `git rev-parse --git-dir`/hooks/commit-msg) [20:31:11] From within buildkit/bin [20:31:11] civi-download-tools [20:31:11] civibuild cache-warmup [20:31:12] From anywhere (once it's in the path) [20:31:13] civibuild create wmff [20:32:21] eileen: ah I see, so basically check buildkit out from Gerrit and then use that to fetch the rest? [20:32:26] yep [20:32:34] ahhh yay [20:32:57] there is some config we copy over to help in the docker image but it's not core to it [20:33:23] eileen which config is that? [20:33:26] AndyRussG: re the logs - there is a civicrm specific logs directory quite deep in - under our crm repo sites/default/files/civicrm/ConfigAndLog [20:34:37] I'm could try to quickly adapt the Payments image quickly to add the packages needed by Civi as per the image, and move the apache config to the host, so we can look into re-using that image as a generic php apache image [20:34:58] for both Civi Web interface and Payments [20:35:50] so the config to support buildkit - this file controls the url & also file location [20:35:50] https://gitlab.com/eileenmcnaughton/fundraising-dev/-/blob/master/civicrm/civibuild.conf [20:36:02] & gives us a nice simple admin password [20:36:27] eileen: okok... so civibuildkit needs that from the get-go, I guess? [20:37:08] It needs it before civibuild create is run - ditto [20:37:08] this one forces the database names [20:37:08] https://gitlab.com/eileenmcnaughton/fundraising-dev/-/blob/master/civicrm/instances.yml [20:38:51] the top 3 folders here https://gitlab.com/eileenmcnaughton/fundraising-dev/-/tree/master/civicrm are all candidates for config / symlinks & that's most of it really [20:39:05] eileen: ah okok cool beans! [20:39:08] We could use the mail catcher container - seems like a nice idea [20:39:23] jgleeson wrt volumes: https://github.com/moby/moby/issues/2259 [20:40:47] the other think to support buildkit is that it creates vhosts files in it's own directory - & this includes that https://gitlab.com/eileenmcnaughton/fundraising-dev/-/blob/master/civicrm/apache.conf [20:41:10] hmm yeah AndyRussG I guess that's changing the perms of the /var/lib/docker contents on the host [20:44:02] AndyRussG: the docker deamon is running as root right? [20:45:11] i guess I'm wondering if there are benefits in us using the host user to run the containers [20:45:37] I can understand why we wouldn't want that on prod [20:46:37] AndyRussG: that ticket is actually a diffrent thing I think [20:47:10] that's trying to set the ownership of a binded mount on the container after it's created [20:47:47] in our case we wanna use docker managed volume mounts [20:48:28] jgleeson: ah hmmm [20:49:10] jgleeson so we've been really avoiding running things as root... when we run composer, it creates files inside the source tree, which are then be owned by root on the host if the container runs as root [20:49:38] eileen ah okok thanks! [20:49:59] AndyRussG: if you are tinkering on it now how can I best help? [20:51:24] eileen maybe in about 30 min I might have a possible generic image to try things out with? [20:51:40] AndyRussG: ok cool - I'll check in then [20:53:53] eileen: thx!!! :) [20:54:34] AndyRussG: we could chown those files to ${USER}:${GROUP} or prefix the command with sudo -u maybe? [20:56:52] or I wonder if we could get around this differently [20:57:10] by adding the user to a group that can modify /var/lib/docker/volumes [20:57:34] I guess I like the idea of not running as root in general [20:58:32] jgleeson: yeah wrt root we've done a fair amount of work to avoid it so far [20:58:50] jgleeson: ejegg was pretty emphatic about not running containers as root generally [21:00:13] I can understand that [21:02:37] jgleeson: for the general MediaWiki dev setup, here is how they suggested running as non-root.. so that's the same pattern we followed, except we scripted it up instead of having folks copy and paste stuff: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/DEVELOPERS.md#linux-users [21:10:08] I feel like there must be a way to use docker volumes in a non-root use-case. it's a core feature of docker storage so someone must have figured it out [21:11:34] AndyRussG: I figured out an easy way to use xdebug without the need to work out host ips and stuff [21:12:26] you can just pass the '--network host' arg which then makes the host available on 127.0.0.1 instead of the usual bridge network [21:13:34] although that might run into issues when we have two containers exposing the same port [21:19:37] jgleeson: right I think it's best to keep the network isolation, no? I mean, currently it does work [21:20:15] jgleeson: so the idea is that on the internal network amongst the containers the ports are all standardized and hard-coded [21:20:33] so each container definitely knows how to access each others' services on that internal network [21:20:48] and then for exposing the ports on the host network, you can set whichever ports you want [21:21:00] that way you can have any number of local setups running at the same time [21:21:10] and not run into anything else you may hav eoging on [21:35:41] eileen: ahh sorry I've had about a million non-work interruptions all at once [21:36:02] I decided to just try running things and installing things manually inside a container as a first step [21:36:09] almost got something [21:44:04] cool [21:55:51] yeah AndyRussG I'll let you focus on that sorry for distracting you, I was just talking about debugging single containers [21:56:09] might be tricky to do more than one [21:58:21] jgleeson: ah no it's not a distraction, heheh what was a distraction was my ex writing me to send by Uber Ceci's cellphone that she forgot, then the neighbour calling to insist on selling beauty products for my kids, and a door-to-door ISP salesperson ringing the doorbell, among other thingies that occurred...... [21:58:46] so in my case now, I don't have anything sophisticated setup, I just run `docker run --rm -it -v $(pwd):/srv/process-control --network host process-control` and then from the container I can hit my docker host on 127.0.0.1 and then send debug traffic via that. might be helpful when testing stuff quick and dirty! [21:59:22] jgleeson: ahhh I see cool beans! [22:06:24] jgleeson: what base image are you using? [22:08:32] for process-control? [22:08:47] lemme find the patch [22:09:34] debian:buster AndyRussG [22:09:36] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/process-control/+/643767/1/Dockerfile [22:09:57] although it might make sense to use the slim-buster to save the OS overhead [22:10:54] this is just to run process-control [22:10:56] btw [22:11:34] which I'm kinda starting at the first part of the job to then work out how to use it within the fundraising-dev project, which will be part two [22:11:45] jgleeson: cool beans!!! [22:11:50] it's not clear which is the best way yet based on our chat [22:12:40] jgleeson: when cron runs things set up by process-control, does it somehow run directly the bash commands set by the process-control setup, or does it run process control somehow which then runs the commands? [22:14:29] eileen: so I got now "docker-php-ext-install: command not found" [22:14:40] I wonder what we should use as a substitute for that? [22:15:05] do you know how those things are installed on production? via composer, or debian package, or pear or something? [22:15:29] AndyRussG: process control has a script that generates a process-control crontab full of scheduled jobs which then call back to run-job [22:15:36] I feel we should make it load stuff up in as similar a way to production as possible, so that it's easy to keep things in sync [22:16:00] which acts as a wrapper for the commands run so it can do the logging and set locks and stuff [22:16:22] jgleeson: ah ok so cron calls job-run (Python) which then calls drush and whatnot? [22:16:33] yep! [22:18:00] jgleeson: okok so the image will need cron, Python and php [22:18:34] hmm [22:18:47] this is the bit I'm not clear on [22:18:51] because if not how will it run all those things? they'll all run in the same container [22:19:15] so you have one container with all those things, with cron as the main process of the container [22:19:20] so I think at the moment I'm looking at this as the task being in two part [22:19:22] s [22:19:35] first task is dockerize the process-control app/repo [22:19:58] jgleeson: yeah that sounds like the right first step [22:20:02] this is so anyone who checks it out can develop on it and test jobs and the process-control source (outside of civi or anything else that uses it) [22:20:21] okok right [22:20:31] but the setup inside fundraising-dev will need all those things [22:21:04] and the second part is then taking that dockeized project and looking at how we wire it into the fundraising-dev project [22:22:51] I've been thinking about cron as a service [22:22:56] much like mysql and redis [22:23:10] jgleeson yis agreed :) [22:23:15] which the other containers use [22:23:24] well [22:23:26] here's the issue [22:23:39] it's tough to make a thing in one container initiate a command in another container [22:23:56] other containers could set up the cron processes I suppose [22:24:08] and the process-control scripts that set the cron tab could be run in a different container [22:24:32] but any commands that are executed by cron will need to have in the same container the libraries and scripting languages needed to execute [22:25:43] hmmm [22:26:29] eileen so all the debian packages specified in your Dockerfile installed fine manually in the image build via dev-images [22:26:38] but I got stuck with the line 54 here: https://gitlab.com/andyrussg/fundraising-dev/-/blob/aa779112d7b16feaedfacc3495f32904469c29fb/civicrm/Dockerfile#L54 [22:27:21] eileen I tried looking in the production puppet library to see how they're installed, but grepping for the names of the PHP extensions didn't immediately show how that works [22:27:31] hmm [22:27:33] AndyRussG: I have an idea how we can do this without creating a monolith container which houses everything but need to test it first before I can explain the merits of it [22:28:09] ultimately process-control just generates a cronfile, so that can easily be shared via a mount point across all containers [22:28:18] AndyRussG: yeah I got stuck on those - but I figured we could try without those extra php extensions - they might be there / not required on the base image anyway [22:28:33] the job files are just text so they don't need to live anywhere but on the process-control container [22:29:17] eileen some of them are set up on prod in .ini files, so they're definitely installed... if you're able to find out how they're installed on prod (what packagey managery thing) and what versions we have on prod, that'd be fantastic [22:30:18] AndyRussG: those docker-php-ext-install thingies will only work on images which extend from an official php image [22:30:23] they are helpers [22:30:23] jgleeson: ok but then the other containers would have to run the cron daemon and also have process control installed to be able to do anything with the cronfile, no? [22:30:52] provided in the official php docker image [22:30:54] jgleeson: ahh right... so I think we need to imitate prod here ^ regarding those extensions? [22:31:40] jgleeson eileen I do now need to take a break to finish making supper and stuff ;) I'll be around later, though I think you'll be asleep jgleeson ..... [22:31:51] jgleeson: thanks for taking time so late in your day for all of this!! [22:32:15] jgleeson eileen I'll see backscroll in any case, thanks again!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [22:32:23] np I'm not actually working now, I'm doing a udemy course :) enjoy supper! [22:32:37] :) thx!! [22:33:02] cool catch you later