[00:03:35] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) I've just deployed this on staging The upgrade script has triggered /ALTER... [00:07:14] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) It's doing activity too */ALTER TABLE `dev_civicrm`.log_civicrm_activity MOD... [00:42:24] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) Now I'm seeing ALTER TABLE `dev_civicrm`.log_civicrm_activity_contact MODIF... [00:43:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) except it won't run on live... [00:46:33] (PS3) Eileen: Add post-check to make sure all locations are valid [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/629216 (https://phabricator.wikimedia.org/T262883) [00:53:24] (CR) jerkins-bot: [V: -1] Add post-check to make sure all locations are valid [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/629216 (https://phabricator.wikimedia.org/T262883) (owner: Eileen) [00:57:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) and ` ALTER TABLE `dev_civicrm`.log_civicrm_activity_contact MODIFY `log_i... [01:04:52] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) ALTER TABLE `dev_civicrm`.log_civicrm_address MODIFY `log_id` bigint(20) un... [02:02:27] (PS1) Eileen: Make upgrade script time sane [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/633597 (https://phabricator.wikimedia.org/T264124) [02:23:06] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) OK - I found a fix for this - https://lab.civicrm.org/dev/core/-/issues/2120 [03:26:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Contributions in Civi currently not saving - https://phabricator.wikimedia.org/T264222 (Eileenmcnaughton) Open→Resolved [03:48:49] (PS2) Eileen: Adjust our code for possibility ssl var willl be in the DNS string [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633588 (https://phabricator.wikimedia.org/T264124) [03:48:51] (PS4) Eileen: Add post-check to make sure all locations are valid [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/629216 (https://phabricator.wikimedia.org/T262883) [03:48:53] (PS1) Eileen: Enable optional on-way-in contact matching [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633602 [03:57:47] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Queue speed - the new frontier - https://phabricator.wikimedia.org/T264061 (Eileenmcnaughton) I did find a few memory leaks which I fixed in core - including one in our main code path & one... [03:58:50] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Queue speed - the new frontier - https://phabricator.wikimedia.org/T264061 (Eileenmcnaughton) I also tried symfony 4 vs symfony 3 but locally could not measure a consistent difference [04:08:59] (CR) jerkins-bot: [V: -1] Add post-check to make sure all locations are valid [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/629216 (https://phabricator.wikimedia.org/T262883) (owner: Eileen) [10:34:22] Wikimedia-Fundraising-Banners, JavaScript, Wikimedia-production-error: Banner code faulty when no disk space available - https://phabricator.wikimedia.org/T264952 (Pcoombe) Open→Resolved a:Pcoombe Confirmed not seeing any more of these errors since the fix {F32383167} [12:40:09] Fundraising-Backlog, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) [12:40:47] Fundraising-Backlog, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) p:Triage→High [12:45:58] Fundraising-Backlog, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (jgleeson) hey @KHaggard, I will kick off the job to export them and look into why last night's job failed. [12:56:13] Fundraising-Backlog, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (jgleeson) Today's export did fail but I've just kicked off the job again to see if it completes this time. The failure was due to data integrity issues (something important... [13:10:57] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (jgleeson) Thanks for handling this @Eileenmcnaughton , it would have definitely been tricky to... [13:16:07] Fundraising-Backlog, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) Thanks @jgleeson ! Will be around to process the new files if the job completes. [13:43:13] Fundraising-Backlog, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (jgleeson) @KHaggard the export and upload ran successfully. You should be able to see the files at your end now. [14:44:13] Fundraising-Backlog, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) Thanks @jgleeson! I just ran them and they look good, so we're back on track. Were you able to find the source of the data integrity issues? [15:06:18] fr-tech I'm going to re-queue the paypal recurrings from the damaged table [15:07:23] ejegg: hi! :) ehhhh thx for doing that! [15:10:00] some of them might fail again, so don't be too surprised if we see a bit of failmail [15:11:02] Fundraising-Backlog, Thank-You-Page, Wikipedia-iOS-App-Backlog, fr-donorservices: Donation redirect to app shows broken inset - https://phabricator.wikimedia.org/T264259 (Pcoombe) Open→Resolved a:Pcoombe Made some more changes to the CSS to account for other themes. This looks okay no... [15:25:42] cool, looks like it's inserting the recur records [15:27:59] yay [15:28:36] ejegg: btw did u see my eod e-mail yesterday? by chance do you have any thoughts about why that logging config doesn't seem to have an effect? [15:28:48] if not, it's ok... [15:28:59] in that case it'll just be time to try out Xdebug with docker... [15:29:02] thx in advance also! [15:37:42] oops, not yet [15:37:46] will look! [15:41:05] ejegg: thanks much! yeah mainly my thought was if if u maybe see something off the bat (I know there's other stuff going on...) :) thx again [15:41:56] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (DStrine) [15:51:53] oh darn AndyRussG! I'd just been blindly trusting those logging settings ever since awight figured them out several years back [15:52:19] ah heheh okok all good [15:52:28] I've got xdebug working so I can try a quick dive into core [15:52:48] yeah looking at the ticket (https://phabricator.wikimedia.org/T107918) did bring back memories [15:54:10] ejegg: hmnmm more than core, I'm trying to understand if the logger requested by DonationInterface in GatewayPage.php is really the one we think it is [15:54:27] as in, maybe there's also some DI settings I'm missing? [15:54:46] oh, in GatewayPage.php? Looking [15:54:52] also, getting Xdebug working under Docker is also a thing to do in any case [15:55:14] ejegg: well that's one of the place that sends a log that doesn't get sent via UDP [15:55:21] but really it's all of the logging I get [15:55:25] so far anyway [15:55:34] also stuff from SmashPig [15:55:36] ohhhhhh [15:55:46] it's doing some logging via SmashPig?? [15:55:58] hang on, lemme paste the log I have [15:55:59] weird, why did we decide to do that for gatewayPage? [15:56:11] ok, so in that case there IS a separate logging config [15:56:14] a bit simpler [15:56:32] Oct 13 04:18:43 1b18ff2f96e9 SmashPig: | Entering logging context 'ingenico'. | | [15:56:34] Oct 13 04:18:43 1b18ff2f96e9 ingenico_gateway: Constructing! Creating a new adapter of type: [Ingenico] [15:56:46] Hmmm there's more [15:57:25] ejegg: https://paste.toolforge.org/view/d6814730 [15:57:46] So, the first line says SmashPig: [15:57:58] and the next two say ingenico_gateway: [15:58:00] Fundraising-Backlog: New engage endowment import - https://phabricator.wikimedia.org/T265369 (DStrine) [15:58:13] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: New engage endowment import - https://phabricator.wikimedia.org/T265369 (DStrine) [15:58:55] the last line seems to be coming from GatewayPage.php line 126 [15:59:17] and none of them are sent to syslog via UDP, it's all over the Unix socket, as far as I can tell, anyway [16:00:03] Things that say SmashPig: will be configured in a yaml file that by default lives under /etc/smashpig/ [16:00:12] (i.e., when I run rsyslogd to only listen on UDP, I don't get them in the log, but when I tell it to listen on the Unix socket, I do get those things. I also tested that when rsyslogd is only listening on UDP, it does log things I send manually via that means) [16:01:15] I'm looking at the DonationLoggerFactory class [16:01:16] ejegg: humm so that's standalone SmashPig running under DI? [16:01:25] yeah I was poking around there... [16:01:44] AndyRussG: even SmashPig-as-a-library uses its own logging config, which is a bit wack [16:01:55] ...mmmm so on the payments box we also need a separate SP config? [16:02:17] no, we use the same SP config everywhere [16:02:34] ejegg: right, but I mean, it needs to be on the payments box too, then? [16:02:43] it just feels like libraries should be happy to use whatever logger the application wants to use [16:02:57] remember all the services in Docker run in separate containers [16:03:04] yes, we need to put the SmashPig config anywhere that SmashPig is used [16:03:32] ejegg: ok, and how does SP under DI know where to find its config? What are the options? and what config should I use? [16:03:58] SmashPig root config is always at /etc/smashpig/main.yaml [16:04:30] ok gotcha, so it hast to be there... K that's fine [16:04:33] that mostly just defines the data sources (db and queue connections) [16:05:07] the default location for the logging and email stuffs, plus all the gateway-specific config is also under /etc/smashpig [16:05:20] but that main.yaml file can specify a different location if there's need to [16:06:01] in prod we've got all the dev-configurable stuff under /srv/SmashPig [16:06:28] because /etc can only be updated by ops [16:07:27] anyway, provider-defaults.yaml in (default) /etc/smashpig has the syslog config [16:07:39] argh [16:08:04] not seeing any port numbers in there [16:08:07] let's see [16:08:34] ejegg: hmmm... so I have no SP config on this Docker box [16:08:52] it's easy to make /etc/smashpig writable by non-root in this case, fortunately [16:08:56] so that's not an issue [16:09:13] huh, though it's just using the php function syslog($priority, $message) [16:09:17] no port numbers there [16:09:25] maybe syslog is configured via php.ini? [16:09:40] ejegg: noup, not ever that I could find [16:09:49] I did see that call there in SP [16:10:10] That's also why I was trying to hack a way to just get syslog instead of reconfiguring logging [16:10:37] Maybe somehow the lack of a SP config is going wonky? [16:10:53] Also, if it's just that one call to syslog, maybe that itself is not too hard to change in the SP source [16:11:26] ejegg: so what should I use a base config to start for SP on Docker? Should I look at prod? Vagrant? [16:11:32] I don't see anything on-wiki [16:11:43] https://www.mediawiki.org/wiki/Fundraising_Tech/Donation_Pipeline_Setup [16:11:46] Thing is, I don't see how to actually syslog to a different port under php [16:12:02] https://www.mediawiki.org/wiki/Fundraising_tech/donation_pipeline_setup/settings#Payments-Wiki_LocalSettings.php [16:12:05] ejegg: correct [16:12:09] vagrant has a template at least for main.yaml [16:12:29] it doesn't even have to be a different port. It just hast to be UDP instead of /dev/log Unix socket [16:12:34] ah right [16:13:03] Hmmm maybe if we somehow disable /dev/log the php will default to UDP [16:13:10] or I mean failover, rather [16:13:17] mebbe? [16:13:21] heheheh [16:13:28] K wrt config I'll check out vagrant [16:13:39] so we can look at what core is doing with those settings too [16:14:43] aaargh... I feel at least they should be impacting on what's being logged from DI [16:15:28] aha, the DonationLoggerFactory from DonationInterface will need a code change [16:15:38] it's instantiating a SyslogHandler directly [16:15:53] and it'll need the option to instantiate a SyslogUdpHandler [16:15:59] https://github.com/Seldaek/monolog/blob/master/src/Monolog/Handler/SyslogUdpHandler.php [16:16:28] ejegg: hmmm? But isn't the syslog handler what's configured in the logging settings I'm using [16:16:48] that's for MW core :( [16:16:53] I did see it doing that, but I was assuming that's what the settings were configuring [16:17:02] only messages from MW core [16:17:12] oh hmmmm [16:17:24] so maybe I can make core send a message and see if it gets to me via UDP [16:17:25] we have our own line formatting and such [16:17:51] * AndyRussG 's head swims [16:18:01] yeah, it's overcomplicated [16:19:05] fundraising-tech-ops, fr-tech-ops-okr: Upgrade payments servers to Debian Buster - https://phabricator.wikimedia.org/T256146 (Dwisehaupt) Open→Resolved a:Dwisehaupt payments1xxx upgraded on 2020-10-09 [16:19:07] fundraising-tech-ops, FR-Q2-FY2020-21-cleanup-list, Patch-For-Review: Upgrade payments to MediaWiki 1.35 - https://phabricator.wikimedia.org/T254298 (Dwisehaupt) [16:19:09] fundraising-tech-ops: Epic: Upgrade fundraising servers to buster - https://phabricator.wikimedia.org/T254198 (Dwisehaupt) [16:19:11] Fundraising-Backlog, fundraising-tech-ops, fr-tech-ops-okr: OKR 2019-2020 Q3: Support fundraising through maintaining infrastructure for payments and donor data - https://phabricator.wikimedia.org/T243019 (Dwisehaupt) [16:20:14] ejegg: eh don't worry... mmm we can make it work, one step at a time [16:20:19] fundraising-tech-ops: Configure payments2xxx for MediaWiki 1.35 upgrade testing - https://phabricator.wikimedia.org/T256153 (Dwisehaupt) [16:21:39] fundraising-tech-ops: Configure payments2xxx for MediaWiki 1.35 upgrade testing - https://phabricator.wikimedia.org/T256153 (Dwisehaupt) Stalled→Resolved a:Jgreen This is completed and payments2xxx have been serving payments in a multiday test. It's looking good to roll wide. [16:21:41] fundraising-tech-ops, FR-Q2-FY2020-21-cleanup-list, Patch-For-Review: Upgrade payments to MediaWiki 1.35 - https://phabricator.wikimedia.org/T254298 (Dwisehaupt) [16:25:02] fundraising-tech-ops, FR-Q2-FY2020-21-cleanup-list: Configure payments1xxx for MediaWiki 1.35 - https://phabricator.wikimedia.org/T265375 (Dwisehaupt) [16:25:05] AndyRussG: so the DonationLoggerFactory part seems like a smaller fix than the SmashPig logging [16:25:22] since DonationLoggerFactory is already creating a monolog-compatible thing [16:25:46] Fundraising-Backlog, fundraising-tech-ops, FR-Q2-FY2020-21-cleanup-list: Configure payments1xxx for MediaWiki 1.35 - https://phabricator.wikimedia.org/T265375 (Dwisehaupt) [16:25:51] it's just a few more settings and a little switch [16:26:16] whereas in SmashPig we're using a custom logging framework that calls syslog() directly [16:26:44] looks like when monolog wants to write to syslog over UDP it's got to do the low-level formatting at socket communication directly [16:27:00] so to get that in SmashPig we can either copy a whole bunch of monolog code [16:27:18] or finally get around to just tearing out the custom framework and replacing it with monolog [16:27:36] fundraising-tech-ops: Fundraising access request for Randy from Trilogy - https://phabricator.wikimedia.org/T264645 (Dwisehaupt) a:Dwisehaupt [16:27:55] fundraising-tech-ops: Fundraising access request for Eric from Trilogy - https://phabricator.wikimedia.org/T264644 (Dwisehaupt) a:Dwisehaupt [16:28:22] (and in the process allowing SmashPig-as-a-library to accept any monolog-compatible logging object from the app it's running under) https://phabricator.wikimedia.org/T87065 [16:30:18] oops, a few lines back I wrote 'at socket...' when I mean 'and socket...' [16:33:52] ejegg: hmmmmm [16:34:41] ejegg: K so this is good to know to think about possible solutions [16:35:09] I'm digging back into now ways to hack a non-root user to be able to read from that Unix socket [16:35:40] hmm, so to relay that to a udp port? [16:35:52] err, wait, what about how we do things on prod? [16:36:17] where everyone writes to their local syslog and there's a central logging server that gets copies of all of it? [16:36:27] ejegg: if we can run any kind of syslog daemon as non-root and read from that socket, we're good to go. The rest is just arranging the already-working fiddly bits [16:37:16] dwisehaupt Jeff_Green do you have any thoughts on this? I don't know much about Unix sockets. Summary: for Docker, trying to make it possible to run rsyslogd or syslog-ng as non-root and make them still able to get syslog messages over /dev/log... thanks in advance! [16:37:30] ejegg: I'm sure in prod we have a normal rsyslogd running as root [16:38:19] dwisehaupt Jeff_Green I've been flailing about Google and Stack Overflow and such, but not much solutions so far [16:38:23] AndyRussG: yes that's true re prod. why does it need to run as non-root? [16:38:32] and also why syslog-ng instead of rsyslog? [16:38:57] (in other words, why would you not want it to look like prod) [16:39:09] Jeff_Green: it can be either syslog-ng or rsyslog, it doesn't matter [16:39:21] Jeff_Green: non-root: because in docker you don't want stuff running as root [16:39:38] because if it's root inside your container, it's also root on your host machine's kernel, apparently [16:39:48] though yes, that is also still an option [16:40:07] This is what I've found for syslog-ng: Can syslog-ng run as a user other than root? [16:40:09] Yes. Start syslog-ng using the ''--user'' command-line parameter. For example: '''syslog-ng --user syslog-ng'''. Note that it is not possible to reload the syslog-ng configuration if the specified user has no privilege to create the ''/dev/log'' file. [16:40:24] (quoted from https://www.syslog-ng.com/syslog-ng-faq/) [16:40:46] so if the non-root user could just create /dev/log socket, we'd be good to go [16:41:36] without really digging into it I don't know. you may also have to contend with systemd around that socket [16:42:01] Jeff_Green: we don't have systemd or any system init running in docker ;p [16:42:15] or any services! [16:42:25] that's good :-) systemd is evil [16:42:26] I mean, we can start them, and I think we have to, in this case [16:42:40] Jeff_Green: we don't have the other system init thing either [16:43:55] in the docker manifest, can you have it execute a command. ie, could it `sudo chmod` to loosen up the permissions on /dev/log to allow a non-root logger to access it? i honestly haven't dealt with logging at this level in docker. [16:44:26] dwisehaupt: I did try to do that manually and it didn't work, but maybe I was doing it wrong? [16:44:40] also not sure what other effects that might have. [16:44:56] Wikimedia-Fundraising-Banners: QA for en6C pre-test on October 15 - https://phabricator.wikimedia.org/T265282 (jbolorinos-ctr) Alright getting started on these now [16:46:08] and to get this clear in my head, it defaults loggint to the socket with no way to change either what that socket is or to switch to a different mode (aka udp/tcp). correct? [16:46:25] without running as root that is. [16:47:37] dwisehaupt: so the base issue of that part is that we have a complex logging setup in our PHP code that ends up in several cases just calling PHP's syslog() function, it seems [16:47:52] another option has been to fix config/code so it doesn't do that [16:48:13] but as far as I can tell, you can't configure how PHP's syslog() function works. It just goes to /dev/log socket, end of story [16:48:16] I may be wrong, though [16:48:33] it's hard to imagine they don't support tcp/udp logging [16:48:51] Jeff_Green: if you can find the way to do that, that'd be wonderful [16:49:10] Jeff_Green: the php built-in function syslog() seems to be unconfigurable and always use the socket [16:49:27] lame [16:49:36] the php libraries which log to syslog over UDP do their own low-level udp socket opening/closing [16:49:53] and line formatting [16:49:59] AndyRussG: ok. cool. i'll have a look to see if i can get any ideas. [16:50:02] Wikimedia-Fundraising-Banners: 2020-10-15 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T265283 (jbolorinos-ctr) Screenshot Test Results - Desktop: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zfeeeda9f62d467d9faf - Reversed button order variant:... [16:51:44] Jeff_Green: ....and the normal way to get PHP's syslog() messages elsewhere is through a running (as root) syslog daemon [16:52:00] yeah, I see what you mean [16:52:24] ok so what about the must-not-run-as-root thing, what is the downside of it running as root in docker? [16:52:54] https://docs.docker.com/config/containers/logging/syslog/ [16:53:01] AndyRussG: oh. just one quick question. when testing with syslog, have you been testing this with the `--log-driver syslog` option or some other method? [16:53:27] Jeff_Green: well, it's doable, but it's not recommended, and any files that the root process creates in volumes mounted in the host have root as owner [16:53:29] I don't know anything at all about docker, but saw that ^^^ and it makes me wonder if docker has hooks to intervene re. syslog [16:54:21] oh this appears to be the same as what Dallas is talking about [16:54:23] dwisehaupt: Jeff_Green regarding the --log-driver, this is basically a different side of a logging pipeline. It's for when your docker service already is spitting logs out via the standard Docker means, and you want Docker to redirect that to your host syslog, or some other machine's syslog [16:54:58] So it's not about when you have sysloging inside your Docker container that you can't get at [16:55:15] i see [16:55:28] Basically what you do inside your container is your business, and your problem, as far as Docker is concerned [16:55:44] Though there are ways to set different kernel permissions [16:56:47] AndyRussG: ok. cool. just wanted to know where you were coming from so i don't look for things you've already tried. :) [16:57:06] So, when I thought I could just use iptables to forward UDP 514 to a different logging container (which I ended up doing successfully, though now it's not useful) I added this to my docker-compose.yml file: [16:57:07] # Capacity and sysctl for forwarding localhost UDP 514 for syslog [16:57:09] cap_add: [16:57:11] - NET_ADMIN [16:57:13] sysctls: [16:57:15] - net.ipv4.conf.all.route_localnet=1 [16:57:17] Wikimedia-Fundraising-Banners: 2020-10-15 en6C pre-test: mobile large batch - https://phabricator.wikimedia.org/T265285 (jbolorinos-ctr) Screenshot Test Results - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zc40de877373091047f8 - Reverse button order variant: ht... [16:57:58] so you can set namespaced sysctl stuff for your docker container and it's set on the kernel just for the docker container processes [16:58:26] if there were a kernel sysctl to let the non-root rsyslogd create the /dev/log socket, that'd also be noice [16:58:55] dwisehaupt: heheh yeah I did dive into that for a while in the hopes it'd fix things... [16:59:07] dwisehaupt: Jeff_Green: hugely appreciate the help here, btw! :D [16:59:30] my vote is to let it run as root [17:00:18] hmmm [17:00:25] yeah probably the simplest option for now [17:00:28] the only reason I can think of not to do that is if you were going to expose the service running in docker to outside traffic, otherwise you're just protecting yourself from yourself, unless I'm missing something [17:01:24] Jeff_Green: hmmm right [17:01:30] ejegg: ^ thoughts? [17:01:51] wellllll [17:01:51] tl;dr: Jeff_Green's suggestion on just letting the rsyslogd daemon in the container run as root? [17:02:04] I guess it's ok [17:02:21] ejegg: actually I think there is a way to make it write files as a different user [17:02:24] Wikimedia-Fundraising-Banners: 2020-10-15 en6C pre-test: mobile small batch - https://phabricator.wikimedia.org/T265286 (jbolorinos-ctr) Screenshot Test Results - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zf977343f74644250b33 - Reversed button order: https://a... [17:02:41] and we could just set the suid bit on that, so the apache/php stuff in the container runs as your local user [17:02:47] as long as we're not running arbitrary stuff pulled down from git as root [17:03:24] rsyslog can drop privileges ~after~ startup, that's the normal way to do it, but startup would run as root to bind that socket I think [17:03:25] ejegg: no, just rsyslogd pulled down as a debian package from the WMF's own debian buster repo [17:03:36] right, seems safe enough [17:03:42] okok sounds good [17:03:58] Jeff_Green: ahhh right, got a quick link on how to configure that? [17:04:03] can you run the stock debian version, or does it have to be a docker-specific repackage? [17:06:33] Jeff_Green: it's stock debian! our base image is a WMF version of buster on Docker, but for all the packages it just points to our own Debian mirror, which AFIK just has stock .deb packages for stuff [17:08:47] AndyRussG: I don't have any quick links other than the rsyslog documentation [17:09:37] you're looking for $PrivDropToUser and $PrivDropToGroup [17:12:11] Jeff_Green: okok cool beans :) [17:12:33] dwisehaupt: Jeff_Green: ejegg: thx so much!!! [17:16:46] Jeff_Green: dwisehaupt: any quick-and-easy tricks you know offhand for rsyslogd to limit log size, or rotate? [17:17:03] (so we don't have to run a logrotate cron job under Docker) [17:17:47] logrotate ~is~ the quick easy trick :-P [17:20:12] Jeff_Green: hmmm okok ;p [17:20:33] AndyRussG: I guess you could configure rsyslog not to log to files at all and just forward somewhere? [17:22:23] Jeff_Green: yes, if it can just push them out to stdout, then the Docker logging facilities can get them. Not sure if that's easiest though. Other option is to have a special Docker container that concentrates, manages and exposes logs from all the other Docker containers in the swarm [17:22:31] (of which there well be a bunch) [17:25:46] Wikimedia-Fundraising-Banners, Instrument-ClientError, Patch-For-Review: Tag errors where a banner is being displayed - https://phabricator.wikimedia.org/T265096 (Jdlrobson) p:Triage→High [17:25:55] Wikimedia-Fundraising-Banners, Instrument-ClientError, Patch-For-Review: Tag errors where a banner is being displayed - https://phabricator.wikimedia.org/T265096 (Jdlrobson) (blocker for rolling out to enwiki) [17:27:23] this is all for docker running on your laptop? how about feeding them into the laptop's syslog, and then filtering them there to a separate file? [17:29:17] * Jeff_Green is very troubled by the image of a 'swarm' of php services [17:33:23] Jeff_Green: can we make the socket inside the container forward to the socket on the laptop? [17:33:56] ejegg: that's a good question, i have no idea, never used docker [17:34:23] but that sounds much simpler [17:35:45] not quite the same thing, but similar https://jpetazzo.github.io/2014/08/24/syslog-docker/ [17:36:27] https://www.projectatomic.io/blog/2016/10/playing-with-docker-logging/ [17:38:16] "Getting messages out of the container to the host logging system." sounds about right, apologies in advance if it turns out to be a terrible rabbithole [17:39:13] Jeff_Green: ejegg: so this is for everyone's development setup, so it should be independent of how things are on your local laptop [17:39:27] However I think the "where to send the logs" is easy to solve [17:40:06] We can just mount a directory inside the docker app's directory [17:41:15] so maybe the mode where one instance collects logs for all the others? [17:41:51] Jeff_Green: yeah could be [17:42:07] Jeff_Green: dwisehaupt: fwiw (doesn't have the latest stuff): https://gitlab.com/andyrussg/fundraising-dev [17:43:24] neat. unfortunately I don't think it will work for our purposes, since what we're developing is all the stuff you need to strip away to make sane docker images [17:47:41] fundraising-tech-ops: Fundraising access request for Randy from Trilogy - https://phabricator.wikimedia.org/T264645 (Dwisehaupt) SSL client certificate created and sent via email. Password sent via sms. Superset account created with ChangeOwnPasswd, Data Consumer rights. Password sent via sms. Welcome email... [17:48:01] fundraising-tech-ops: Fundraising access request for Eric from Trilogy - https://phabricator.wikimedia.org/T264644 (Dwisehaupt) SSL client certificate created and sent via email. Password sent via sms. Superset account created with ChangeOwnPasswd, Data Consumer rights. Password sent via sms. Welcome email s... [17:48:37] fundraising-tech-ops: Fundraising access request for Randy from Trilogy - https://phabricator.wikimedia.org/T264645 (Dwisehaupt) [17:49:06] fundraising-tech-ops: Fundraising access request for Eric from Trilogy - https://phabricator.wikimedia.org/T264644 (Dwisehaupt) [17:51:51] Jeff_Green: you mean docker images for prod? or for fr-operations development? Yeah I think those would have to be pretty different [17:52:19] Docker (or generally container) principles for a dev setup are apparently pretty different than those for prod deployment [17:52:47] for example, for development you want the source code outside the image, not included in it [17:53:07] whereas for a quick deployable image, you want your source code already inside the image [17:53:11] fundraising-tech-ops: Fundraising access request for Randy from Trilogy - https://phabricator.wikimedia.org/T264645 (Dwisehaupt) Notified eyener of superset account setup so she can modify permissions as needed if more access is required. [17:53:17] fundraising-tech-ops: Fundraising access request for Eric from Trilogy - https://phabricator.wikimedia.org/T264644 (Dwisehaupt) Notified eyener of superset account setup so she can modify permissions as needed if more access is required. [17:53:19] anyway that's one difference I learned about, still got a lot to learn [17:53:57] yeah, for fr-dev-ops [18:06:02] Jeff_Green: yeah surely a different universe [18:23:01] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (jgleeson) @KHaggard, unfortunately, the error message is pretty vague and doesn't tell us much other than //an//... [18:24:33] fundraising-tech-ops: Fundraising access request for Randy from Trilogy - https://phabricator.wikimedia.org/T264645 (Dwisehaupt) [18:45:34] fundraising-tech-ops: Fundraising access request for Randy from Trilogy - https://phabricator.wikimedia.org/T264645 (Dwisehaupt) Open→Resolved Verified in logs that Randy has access to superset. Closing. [19:05:42] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (Eileenmcnaughton) @jgleeson yeah - in fact that issue hit us last time too I think - but didn't... [19:06:31] Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, Patch-For-Review: Grafana graph of wmf_civicrm_contribution_message_import - https://phabricator.wikimedia.org/T261689 (DSt... [19:06:46] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Error message when importing a Benevity file - https://phabricator.wikimedia.org/T263744 (DStrine) Open→Resolved [19:06:57] Fundraising-Backlog, fundraising sprint Theme songs for programming languages: new endowment TY email - https://phabricator.wikimedia.org/T263877 (DStrine) Open→Resolved [19:10:09] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, FR-Email: Matching Gifts database data cleanup - https://phabricator.wikimedia.org/T260935 (Ejegg) I went ahead and deleted the current blank rows. Here's the task to make sure new ones d... [19:11:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, FR-Email: Matching Gifts database data cleanup - https://phabricator.wikimedia.org/T260935 (DStrine) Open→Resolved @MNoorWMF we think this is good to close for now. Could you conf... [19:11:48] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (DStrine) Open→Resolved [19:13:20] Fundraising-Backlog, fundraising sprint Theme songs for programming languages: Rounding issue causing ingenico INVALID_VALUE errors - https://phabricator.wikimedia.org/T265279 (DStrine) Open→Resolved [19:14:25] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Ingenico, Recurring-Donations, fr-donorservices: Oct. 2020 Ingenico recurrings failing to attempt new donations - https://phabricator.wikimedia.org/T264954 (DStrine) Open→Resolved [19:16:14] Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Adyen: reimplement adyen iframe in a variant - https://phabricator.wikimedia.org/T261590 (DStrine) Open→Resolved [19:17:24] fundraising-tech-ops: Fundraising access request for Eric from Trilogy - https://phabricator.wikimedia.org/T264644 (Dwisehaupt) [19:17:35] (CR) Hashar: "check experimental" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/632951 (owner: Ejegg) [19:17:58] Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, fr-donorservices: French TY email sending donors to unsubscribe page in English - https://phabricator.wikimedia.org/T260814 (DStrine) Open→Resolved [19:18:23] fundraising-tech-ops: Fundraising access request for Eric from Trilogy - https://phabricator.wikimedia.org/T264644 (Dwisehaupt) Open→Resolved Verified Eric has access from log files. Closing. [19:19:25] Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Review activity related queries in contribution processing - https://phabricator.wikimedia.org/T262884 (DStrine) Open→Resolved [19:19:34] Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Address extraneous location related queries in queue processing - https://phabricator.wikimedia.org/T262883 (DStrine) Open→Resolved [19:19:36] Fundraising Sprint Raw data never hurt anyone, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Analyse queries running during queue processing - https://phabricator.wikimedia.org/T261688 (DStrine) [19:20:24] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages: Queue speed - the new frontier - https://phabricator.wikimedia.org/T264061 (Eileenmcnaughton) Open→Resolved [19:31:55] Wikimedia-Fundraising-Banners, Wikimedia-production-error: Object doesn't support this action at frb.formatCurrency Unknown script code:1240:13 - https://phabricator.wikimedia.org/T265396 (Jdlrobson) [19:35:18] Wikimedia-Fundraising-Banners, JavaScript, Wikimedia-production-error: Object doesn't support this action at frb.formatCurrency Unknown script code:1240:13 - https://phabricator.wikimedia.org/T265396 (Umherirrender) [19:53:03] Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Email: Civi to IBM export: Yesterday's files were sent today - https://phabricator.wikimedia.org/T265352 (KHaggard) Ah I see, got it :) thanks for explaining and fixing the issue so quickly @jgleeson ! Thanks for resolv... [19:56:08] Wikimedia-Fundraising-Banners, Wikimedia-production-error: Banner error: Uncaught TypeError: event.target.getAttribute is not a function - https://phabricator.wikimedia.org/T265401 (Jdlrobson) [20:01:38] Wikimedia-Fundraising-Banners, JavaScript, Wikimedia-production-error: Intl is not available., Object doesn't support this action at frb.formatCurrency Unknown script code:1240:13 - https://phabricator.wikimedia.org/T265396 (Jdlrobson) [20:03:48] Fundraising-Backlog, FR-Smashpig: Make cURL debug logging useful again - https://phabricator.wikimedia.org/T264931 (DStrine) [20:07:37] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Investigate impact of buster upgrade on queue processing - https://phabricator.wikimedia.org/T264819 (DStrine) [20:08:18] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Investigate impact of buster upgrade on queue processing - https://phabricator.wikimedia.org/T264819 (Eileenmcnaughton) https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?viewPanel=22&orgId=1&from=now-120d&to=no... [20:11:28] Wikimedia-Fundraising-Banners, JavaScript, Wikimedia-production-error: Banner error: Uncaught TypeError: event.target.getAttribute is not a function - https://phabricator.wikimedia.org/T265401 (Umherirrender) [20:14:33] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: fr-tech to look into supporting mail fail over for civi - https://phabricator.wikimedia.org/T264663 (DStrine) [20:20:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: New engage endowment import - https://phabricator.wikimedia.org/T265369 (DStrine) [20:23:10] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: New engage endowment import - https://phabricator.wikimedia.org/T265369 (DStrine) [20:27:41] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: New engage endowment import - https://phabricator.wikimedia.org/T265369 (DStrine) [20:33:20] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Q2-FY2020-21-cleanup-list: Automatic TY for Stock Gifts - https://phabricator.wikimedia.org/T259173 (DStrine) [20:37:01] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Changes to Citibank International import - https://phabricator.wikimedia.org/T261795 (DStrine) [20:39:13] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Q2-FY2020-21-cleanup-list: Dedupe donors before creating new contact - https://phabricator.wikimedia.org/T254270 (DStrine) [20:44:21] Fundraising Sprint U 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: New engage endowment import - https://phabricator.wikimedia.org/T265369 (DStrine) [20:44:23] Fundraising Sprint U 2020, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Adyen visa debit users see error coming back from card processor - https://phabricator.wikimedia.org/T264941 (DStrine) [20:44:25] Fundraising Sprint U 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: fr-tech to look into supporting mail fail over for civi - https://phabricator.wikimedia.org/T264663 (DStrine) [20:44:27] Fundraising Sprint U 2020, Fundraising-Backlog, FR-Smashpig: Make cURL debug logging useful again - https://phabricator.wikimedia.org/T264931 (DStrine) [20:44:29] Fundraising Sprint U 2020, Fundraising-Backlog, fundraising sprint Theme songs for programming languages: Donations queue consumer failed - Invalid character input in postal code - https://phabricator.wikimedia.org/T262232 (DStrine) [20:44:31] Fundraising Sprint U 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Changes to Citibank International import - https://phabricator.wikimedia.org/T261795 (DStrine) [20:44:35] Fundraising Sprint U 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Q2-FY2020-21-cleanup-list: Dedupe donors before creating new contact - https://phabricator.wikimedia.org/T254270 (DStrine) [20:44:37] Fundraising Sprint U 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Q2-FY2020-21-cleanup-list: Automatic TY for Stock Gifts - https://phabricator.wikimedia.org/T259173 (DStrine) [20:44:39] Fundraising Sprint U 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (DStrine) [20:44:42] Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint U 2020, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, and 4 others: Test payments-wiki extensions on MediaWiki 1.35 - https://phabricator.wikimedia.org/T260621 (DStrine) [20:44:44] Fundraising Sprint U 2020, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Q2-FY2020-21-cleanup-list, Patch-For-Review: Make easy on/off switch for Monthly covert - https://phabricator.wikimedia.org/T250918 (DStrine) [20:44:46] Fundraising Sprint U 2020, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising sprint Theme songs for programming languages, Patch-For-Review: one last civi point release? - https://phabricator.wikimedia.org/T264124 (DStrine) [20:44:48] Fundraising Sprint U 2020, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages: Get queues working on docker - https://phabricator.wikimedia.org/T262977 (DStrine) [20:44:52] Fundraising Sprint U 2020, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, FR-Q2-FY2020-21-cleanup-list: Paypal recurring payments missing predecessor - https://phabricator.wikimedia.org/T248420 (DStrine) [20:44:54] Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint U 2020, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages: As a email manager, I'd like to see which contacts are ... - https://phabricator.wikimedia.org/T261705 [20:44:57] Fundraising Sprint U 2020, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, and 2 others: Get payments wiki working on docker - https://phabricator.wikimedia.org/T262976 (DStrine) [20:44:59] Fundraising Sprint U 2020, Fundraising-Backlog, fundraising sprint Theme songs for programming languages, fr-donorservices, Patch-For-Review: Add translations for failed recurring email for 7 additional languages - https://phabricator.wikimedia.org/T260703 (DStrine) [20:45:01] Fundraising Sprint Octopus hugs, Fundraising Sprint Pseudopretzels, Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint U 2020, and 4 others: Fr-tech chores list - https://phabricator.wikimedia.org/T258527 (DStrine) [22:09:19] (Abandoned) Eileen: temp debug [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/633584 (owner: Eileen) [22:33:28] (PS1) Eileen: remove lightning bolt as it was messing with layout [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633841 [22:47:01] (CR) Ejegg: [C: +2] remove lightning bolt as it was messing with layout [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633841 (owner: Eileen) [22:48:19] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: New engage endowment import - https://phabricator.wikimedia.org/T265369 (DStrine) p:Triage→Medium [22:48:29] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Adyen visa debit users see error coming back from card processor - https://phabricator.wikimedia.org/T264941 (DStrine) p:Triage→Medium [22:48:37] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, FR-Smashpig: Make cURL debug logging useful again - https://phabricator.wikimedia.org/T264931 (DStrine) p:Triage→Medium [22:48:45] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: fr-tech to look into supporting mail fail over for civi - https://phabricator.wikimedia.org/T264663 (DStrine) p:Triage→Medium [22:49:04] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, fundraising sprint Theme songs for programming languages: Donations queue consumer failed - Invalid character input in postal code - https://phabricator.wikimedia.org/T262232 (DStrine) p:Triage→Medium [22:49:14] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Changes to Citibank International import - https://phabricator.wikimedia.org/T261795 (DStrine) p:Triage→High [22:49:19] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Q2-FY2020-21-cleanup-list: Automatic TY for Stock Gifts - https://phabricator.wikimedia.org/T259173 (DStrine) p:Triage→Medium [22:49:30] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Q2-FY2020-21-cleanup-list: Dedupe donors before creating new contact - https://phabricator.wikimedia.org/T254270 (DStrine) p:Triage→Medium [22:49:38] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: make sure Forget Me requests reach IBM - https://phabricator.wikimedia.org/T222287 (DStrine) p:Triage→High [22:53:46] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Changes to Citibank International import - https://phabricator.wikimedia.org/T261795 (Eileenmcnaughton) a:Eileenmcnaughton [22:57:47] (PS1) Eileen: Typo fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633844 [22:59:12] (CR) jerkins-bot: [V: -1] Typo fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633844 (owner: Eileen) [23:02:40] (PS2) Eileen: Enable optional on-way-in contact matching [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633602 [23:03:28] (PS2) Eileen: Typo fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633844 [23:04:09] (Merged) jenkins-bot: remove lightning bolt as it was messing with layout [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633841 (owner: Eileen) [23:07:19] (PS3) Ejegg: Enable optional on-way-in contact matching [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633602 (owner: Eileen) [23:07:31] (CR) Ejegg: [C: +2] Enable optional on-way-in contact matching [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633602 (owner: Eileen) [23:11:20] (Abandoned) Ejegg: Update CiviCRM to 5.31 beta1 [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/633212 (owner: Ejegg) [23:11:59] (Abandoned) Ejegg: re-add remaining wmf hacks. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/633213 (owner: Ejegg) [23:14:34] (Merged) jenkins-bot: Enable optional on-way-in contact matching [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633602 (owner: Eileen) [23:40:10] fundraising-tech-ops, DC-Ops, Operations, ops-eqiad: (Need By: 2020-09-30) rack/setup/install frdb1004.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T260379 (Dwisehaupt) Invalid→Resolved [23:43:17] fundraising-tech-ops, DC-Ops, Operations, ops-eqiad: (Need By: 2020-09-30) rack/setup/install frdb1004.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T260379 (Dwisehaupt) Resolved→Invalid [23:50:00] (PS1) Eileen: IDE formatting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633848 (https://phabricator.wikimedia.org/T261795) [23:50:30] (PS3) Eileen: Adjust our code for possibility ssl var willl be in the DNS string [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/633588 (https://phabricator.wikimedia.org/T264124) [23:58:47] Fundraising Sprint Upbeat Upholstery, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Changes to Citibank International import - https://phabricator.wikimedia.org/T261795 (Eileenmcnaughton) @LeanneS I just realised there is a question here - we have a citibank import & my in...