[00:10:57] thanks dwisehaupt ! [00:15:07] (PS1) Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/685146 [00:15:18] (CR) Eileen: [C: +2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/685146 (owner: Eileen) [00:15:49] !log updated payments-wiki from 44570561f2 to d449599540 [00:15:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:17:42] hmmm... i'm imagining that this redis instance will be minimal in size. [00:20:21] dwisehaupt: you mean in typical queue size? yeah I imagine so [00:22:01] yeah. ie: we increase the max memory footprint for the queue instance, but i'm thinking we may not need that here. [00:23:07] getting the second redis instance up isn't too difficult. getting it monitored may require some more in depth work unless i'm missing something. [00:24:28] ah hmmm [00:25:49] it delves into the fun of nagios/icinga and its plugins not generally designed for more than a single instance of a service on a host. but we'll find a way to get it done. [00:27:11] not seeing any reason for it to get in the way of testing. [00:29:15] yeah for sure [00:29:24] thanks much for all the work on this dwisehaupt!! :) [00:29:30] !log civicrm revision changed from 94e321dbe0 to e7c610fd87, config revision is 189788d452 [00:29:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:32:03] fundraising-tech-ops, fr-email-preference-center: Add Apache/Nginx config to serve Civiproxy - https://phabricator.wikimedia.org/T281321 (Dwisehaupt) a:Dwisehaupt [00:32:17] fundraising-tech-ops, fr-email-preference-center: Network configuration for Civiproxy - https://phabricator.wikimedia.org/T281320 (Dwisehaupt) a:Dwisehaupt [00:33:19] ok, that payments deploy looks good. signing off for now! [00:34:40] (PS2) Eileen: Catch and handle exceptions in Monolog [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684664 (https://phabricator.wikimedia.org/T281647) [00:35:08] I just re-updated the monolog patch with the try-catch error handling & also a fix for something I spotted [00:39:31] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) [00:47:54] ok. staged a commit for redis but not going to roll it and leave. i'll pick it up tomorrow. have a good one. [03:09:20] eileen: looks like CI was happy with enabling the syslog module on install: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/685137 [03:09:33] ejegg|away: yay - let's merge it then! [03:09:52] (PS2) Ejegg: Enable drupal syslog module on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685137 [03:09:52] (PS3) Ejegg: Enable drupal syslog module on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685137 [03:09:52] (CR) Eileen: [C: +2] Enable drupal syslog module on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685137 (owner: Ejegg) [03:10:00] :) [03:11:08] eileen: the $this->enabled check was the bug causing reinstatiation? [03:11:39] ah, no, the stuff that you moved out of the loop [03:11:43] ejegg: no I dropped it because I figured the try-catch would do what I had obviously hoped it would do [03:13:25] yep, those 3 lines do look like they belong outside the loop in any case [03:22:40] (Merged) jenkins-bot: Enable drupal syslog module on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685137 (owner: Ejegg) [03:26:47] (PS1) Ejegg: Update drupal to 7.80 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685197 [03:29:30] Fundraising Sprint Git Rebase Jump, Fundraising Sprint Humongous bacteria petting zoo, Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, and 3 others: Figure out how to replace watchdog - https://phabricator.wikimedia.org/T279983 (Eileenmcnaughton) We should... [03:30:25] (CR) Eileen: [C: +2] Update drupal to 7.80 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685197 (owner: Ejegg) [03:30:40] (PS1) Ejegg: Re-apply WMF patches [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685198 [03:31:49] ok, manual patch file tweaking made that pretty painless to reapply ^^ [03:31:54] Fundraising Sprint Humongous bacteria petting zoo, Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: New Org Contact Fields in Engage Import not Importing to Record - https://phabricator.wikimedia.org/T278892 (... [03:32:16] (CR) Eileen: [C: +2] Re-apply WMF patches [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685198 (owner: Ejegg) [03:32:28] cool looks good [03:32:44] cool cool, guess it fixes some XSS vulns [03:33:05] ejegg: I decided that if we merge this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/682797 - I'll push that task off to 'done' for this round [03:34:19] (PS2) Ejegg: Remove more unused code [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682797 (https://phabricator.wikimedia.org/T270678) (owner: Eileen) [03:34:24] (CR) Ejegg: [C: +2] Remove more unused code [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682797 (https://phabricator.wikimedia.org/T270678) (owner: Eileen) [03:39:48] (CR) jerkins-bot: [V: -1] Update drupal to 7.80 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685197 (owner: Ejegg) [03:43:04] (CR) jerkins-bot: [V: -1] Re-apply WMF patches [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685198 (owner: Ejegg) [03:43:06] (CR) jerkins-bot: [V: -1] Remove more unused code [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682797 (https://phabricator.wikimedia.org/T270678) (owner: Eileen) [03:49:39] (PS4) Eileen: Keep imported org details on update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682793 (https://phabricator.wikimedia.org/T278892) [03:54:19] (PS5) Eileen: Keep imported org details on update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682793 (https://phabricator.wikimedia.org/T278892) [03:54:45] Fundraising Sprint Humongous bacteria petting zoo, Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Remove unused code from wmf_civicrm - https://phabricator.wikimedia.org/T270678 (Eileenmcnaughton) a:Eileenmcn... [03:58:42] (CR) Ejegg: [V: +2 C: +2] Update drupal to 7.80 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685197 (owner: Ejegg) [03:59:58] (PS1) Ejegg: Update phpseclib [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685207 [04:03:59] (PS2) Ejegg: Update phpseclib [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684053 [04:06:41] (PS1) Eileen: Remove direct phpseclib require from omnimail extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685208 (https://phabricator.wikimedia.org/T281269) [04:08:18] (CR) jerkins-bot: [V: -1] Keep imported org details on update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682793 (https://phabricator.wikimedia.org/T278892) (owner: Eileen) [04:18:43] i'mma just self-merge those seclib bits [04:18:57] (CR) Ejegg: [C: +2] Update phpseclib [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684053 (owner: Ejegg) [04:19:29] (CR) Ejegg: [C: +2] Update phpseclib [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685207 (owner: Ejegg) [04:20:55] (PS2) Ejegg: Remove direct phpseclib require from omnimail extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685208 (https://phabricator.wikimedia.org/T281269) (owner: Eileen) [04:21:00] (CR) Ejegg: [C: +2] Remove direct phpseclib require from omnimail extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685208 (https://phabricator.wikimedia.org/T281269) (owner: Eileen) [04:23:47] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/685211 [04:28:39] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/685211 (owner: Ejegg) [04:29:42] oh interesting, there's some work to prep for php 8.0 support in drupal 7 [04:29:45] (PS6) Eileen: Keep imported org details on update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682793 (https://phabricator.wikimedia.org/T278892) [04:29:52] I guess drupal 7's EOL got extended [04:30:36] https://www.drupal.org/psa-2020-06-24 [04:30:40] oh really? I missed the memo [04:30:55] ^^^ "Given the impact of COVID-19 on budgets and businesses, we will be extending the end of life until November 28, 2022" [04:31:01] yah, first i'd heard of it [04:31:25] so that gives our schedule a bit of wiggle room [04:31:52] though I did kind of like the extra impetus to get stuff into proper Civi modules and shed cruft [04:32:00] err, extensions rather [04:32:24] phew, I think I better turn in soon... [04:33:04] eileen if you feel like deploying that stuff I just merged to deploy please go ahead, otherwise I'll do it first thing tomorrow [04:33:31] ejegg: ok - I might as well deploy [04:34:13] not quite merged yet [04:35:46] (CR) Eileen: [C: +2] "I find the match with the code comment quite compelling" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684054 (https://phabricator.wikimedia.org/T242452) (owner: Ejegg) [04:37:52] thanks eileen [04:37:56] have a good rest of your day [04:37:59] you too [04:45:33] (Merged) jenkins-bot: Update phpseclib [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685207 (owner: Ejegg) [04:45:36] (Merged) jenkins-bot: Remove direct phpseclib require from omnimail extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/685208 (https://phabricator.wikimedia.org/T281269) (owner: Eileen) [04:50:32] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/685211 (owner: Ejegg) [04:52:04] (PS2) Eileen: Copy differences as commented out [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/683114 [04:52:19] (CR) Eileen: "Alright ejegg - changed the commenting" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/683114 (owner: Eileen) [04:53:28] !log civicrm revision changed from e7c610fd87 to 8034e47008, config revision is 189788d452 [04:53:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:59:13] (Merged) jenkins-bot: Don't try to delete pending messages with no order ID [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684054 (https://phabricator.wikimedia.org/T242452) (owner: Ejegg) [05:24:54] Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Error when searching Wikimedia Recipients SmartGroup in Civi - https://phabricator.wikimedia.org/T272415 (Eileenmcnaughton) @RLewis I took a look at this and I think the cleaning fai... [05:28:04] Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Error when searching Wikimedia Recipients SmartGroup in Civi - https://phabricator.wikimedia.org/T272415 (Eileenmcnaughton) a:Eileenmcnaughton [06:09:59] AndyRussG: I don't know if you read the comment I made on Rosie's saved search ticket - but the extra informaiton I found was in the log_ tables - which show who deleted one of the groups & when [06:10:11] ie log_civicrm_group [06:20:03] eileen: ahh ok cool thanks for the heads-up! [06:20:23] I can actually get to it tomorrow, but that helps! :) [07:30:38] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1321 [07:37:32] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 2 [=1] [07:42:32] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 1 [=1] [07:45:38] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 3419348 Threads: 17 Questions: 90302114 Slow queries: 2807046 Opens: 710377908 Flush tables: 1 Open tables: 298 Queries per second avg: 26.409 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61 [07:47:32] RECOVERY - check_log_messages on frav1002 is OK: OK [09:36:32] Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Error when searching Wikimedia Recipients SmartGroup in Civi - https://phabricator.wikimedia.org/T272415 (RLewis) @Eileenmcnaughton oh yikes the cleaning fairy again!! I have been us... [15:48:19] (PS3) Ejegg: SmashPig: connect to fredge as 'drupal' [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/685121 [15:48:22] (CR) Ejegg: [V: +2] SmashPig: connect to fredge as 'drupal' [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/685121 (owner: Ejegg) [16:24:01] (PS2) Ejegg: Delete log in link on payments + e-p-c wikis [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/685120 [16:38:38] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) Redis config rolled out. iptables rules updated and pushed. eq... [16:39:44] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) [16:59:36] (PS1) Ejegg: Use upstream civicrm-buildkit [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/685528 (https://phabricator.wikimedia.org/T277500) [17:00:30] (CR) Ejegg: [V: +2 C: +2] Update PHPMailer [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/685135 (owner: Ejegg) [17:01:10] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/685134 (owner: Ejegg) [17:03:35] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/685134 (owner: Ejegg) [17:10:40] !log updated standalone SmashPig deploy from 250a8570d1 to be272c02ce [17:10:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:10:54] ok, that was the last of the phpmailer security updates ^^^ [17:15:33] fundraising-tech-ops, fr-email-preference-center: Network configuration for Civiproxy - https://phabricator.wikimedia.org/T281320 (Ejegg) [17:16:15] fundraising-tech-ops, fr-email-preference-center: Network configuration for Civiproxy - https://phabricator.wikimedia.org/T281320 (Ejegg) >>! In T281320#7045675, @Jgreen wrote: > @ejegg As far as I know the only server/service planned to access civiproxy is donorwiki, which will run on the frdata servers... [17:24:59] fundraising-tech-ops, fr-email-preference-center: Create fundraising_code_update project for civiproxy - https://phabricator.wikimedia.org/T281311 (Dwisehaupt) Open→Resolved This has been verified. Closing. [17:25:01] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, Fr-planning-ahead, Epic: Epic: Create Preference Center for donors to manage email subscription preferences - https://phabricator.wikimedia.org/T125272 (Dwisehaupt) [17:56:57] (PS1) Jgleeson: Merge branch 'fundraising-dev' [wikimedia/fundraising/crm/civiproxy] (deployment) - https://gerrit.wikimedia.org/r/685542 [17:57:05] (PS1) Jgleeson: Merge branch 'fundraising-dev' [wikimedia/fundraising/crm/civiproxy] - https://gerrit.wikimedia.org/r/685543 [17:57:38] (CR) Jgleeson: [V: +2 C: +2] Merge branch 'fundraising-dev' [wikimedia/fundraising/crm/civiproxy] - https://gerrit.wikimedia.org/r/685543 (owner: Jgleeson) [17:58:11] (CR) Jgleeson: [V: +2 C: +2] Clear out unused files and update Dockerfile [wikimedia/fundraising/crm/civiproxy] - https://gerrit.wikimedia.org/r/664901 (https://phabricator.wikimedia.org/T268495) (owner: Jgleeson) [18:02:47] (PS1) Jgleeson: Update CiviProxy checkout to no longer pull down fundraising_dev branch and just use master [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/685546 [18:03:30] (PS2) Jgleeson: Update CiviProxy checkout to no longer pull down fundraising_dev branch and just use master [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/685546 [18:03:45] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (ayounsi) [18:12:07] Jeff_Green / dwisehaupt the queue servers are not responding to payments [18:12:19] all donors are getting errors [18:13:08] tskaff: pcoombe: might want to take banners down for now [18:14:58] ejegg: ok. looking. [18:15:04] ok, pcoombe says he's taking banners down [18:15:29] seems like it will probably be pfw related. [18:16:02] that change was just pushed. (sorry, was out on a dog walk) [18:16:27] yep yep, suspected as much [18:16:37] is netops available for a quick fix? [18:16:51] banners are down [18:16:57] thanks pcoombe [18:17:12] dwisehaupt: looks like the failures started at 18:03:30 or so [18:20:24] yeah. looking at the diffs and there is definitely something that didn't go right with what i'm seeing in the config. [18:20:48] dwisehaupt Jeff_Green fr-tech we're setting payments to maintenance mode via localsettings [18:20:49] dwisehaupt and Jeff_Green we're setting the maintenance flag in localsettings [18:20:54] jinx! [18:21:09] you owe me a candy bar [18:21:26] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) [18:22:33] ah. i see what it was. [18:27:43] ok. change pushed. contacting XioNoX to see about a repush [18:34:17] fr-tech: ok pfw policy pushed and i verified from payments1001 that i can get to the redis queue on 6379 again. [18:35:07] root cause: when adding the new donor_prefs rule, it was given the same name as the existing one. the last one in the file wins and thus it overwrite the payments rule with the newer donor prefs one. [18:35:45] dwisehaupt: ahhh nice catch! [18:35:55] also verified the new donor_prefs redis rule works as expected. [18:36:00] sorry about that. [18:36:24] thanks! fr-tech about to take payments out of maintenance mode [18:37:22] payments is back [18:38:27] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) [18:38:31] dwisehaupt: all good, thanks so much for the quick fix!!! :) [18:39:08] np. would have been quicker if the dog hadn't dragged me out to do his errands. [18:39:19] ha! [18:39:45] Ok, so where are we overall? What's left on EPC? [18:40:39] haha AndyRussG I can message them I see you typing [18:40:45] oh there you go [18:41:12] (PS1) Ejegg: Update monolog + php-queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/685555 [18:41:29] (CR) Ejegg: [V: +2 C: +2] Update monolog + php-queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/685555 (owner: Ejegg) [18:42:49] (Merged) jenkins-bot: Update monolog + php-queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/685555 (owner: Ejegg) [18:43:12] XenoRyet: we were just handling an emergency outage, seems to be fixed now [18:43:51] Yea, I was following that. I was wondering how far EPC got before the outage got in the way [18:43:53] XenoRyet: on EPC also digging into logging, seems there are some details there. We can reach the new EPC page as planned using the client cert, but so far errors [18:44:16] banners are back up [18:44:22] Cool, progress. [18:44:52] XenoRyet: we have dispersed from tech-talk but I think digging in to those EPC errors and logging stuff will be top priority for today [18:45:00] pcoombe: thanks! :) [18:45:39] Yea, we need everyone on it for as much as we can to get it over the line today. [18:46:11] XenoRyet: k sounds good! [18:46:25] Just saw this and it's already over. That was fast. Thanks for the fast work all! [18:46:50] ok. there appears to be some blockage on getting from frdata->civi on the civiproxy port. trying to figure out where the packets are getting stopped. [18:50:28] tskaff: :) [18:50:54] dwisehaupt: ah cool thx! that could be why the e-mail pref page is erroring out [18:51:07] dwisehaupt: we were also trying to find where we could get the logs from that error btw [18:52:41] pfw and iptables look correct, so not sure why it isn't making it through. i'll get back to that and look for your errors. [18:55:40] sorry, just getting to this as jeff was looking at it earlier. these are the errors of type: Fatal exception of type "RuntimeException" ? [19:00:25] dwisehaupt: yes, for eample: [b75ce2cd7362dcbe5ca04297] 2021-05-05 19:00:03: Fatal exception of type "RuntimeException" [19:01:07] ok cool. let me dig a quick second, then i'll look at getting fr-tech access to this role. [19:02:12] yeah. i'm not seeing anything matching those types of errors in the syslog or /srv dirs at all [19:03:10] dwisehaupt: it seems also that we might have some errors in our monolog setup to capture Mediawiki errors since we merged an update from core [19:07:33] dwisehaupt: also to note, I get the same error regardless of whether it's a real or made up hash/contact_id [19:11:53] ok. i have added the fr-tech group to the frdata build. you should have perms to log in and view logs in /var/log and what you need in /srv. [19:12:00] thanks dwisehaupt [19:12:34] noice! [19:12:50] i need to get some food before i can get my head around why these frdata->civi connections aren't working as expected. [19:13:55] :) sounds important hehe I'm in a food-requiring situation [19:35:09] it's the darndest thing, I can debug all the way down to where the monolog SyslogUdpHandler writes to the socket [19:35:16] but I see nothing in my /var/log/syslog [19:35:28] so... trying to understand rsyslogd configuration :P [19:36:26] ejegg: did you update your mw? [19:36:53] yep yep, using mw and vendor exactly as on production [19:37:14] your error looked like an older version of monolog lib was hanging around somewhere [19:37:58] hmmm that also could be [19:38:05] I didn't run composer update on DI [19:38:18] oh, DI shouldn't have its own vendor directory [19:38:31] just one top-level vendor directory under mediawiki [19:40:04] ahhh right [19:42:21] ejegg: so where else could an old version of monolog come from? [19:42:54] hmm hmm, I noticed I hadn't updated it in the standalone SmashPig, so I've just pushed that fix [19:43:08] but it would be really weird for MW to be autoloading out of that far-off dir [19:47:59] ejegg: K I'll try that in a bit (for now got the non-work stuff flying about...) [19:56:24] ejegg: I guess I can debug in and look at the xdebug log to see for sure which files on the server are being executed [20:01:54] AndyRussG: found the logging issue! production shouldn't be using the UDP logger at all [20:02:04] so payments is now logging the mediawiki problems [20:02:11] and... there are a good number [20:03:48] Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog: Fix payments-wiki DependencyStore config - https://phabricator.wikimedia.org/T282051 (Ejegg) [20:04:04] so we should fix that soon ^^^^ [20:04:29] Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog: Fix payments-wiki DependencyStore config - https://phabricator.wikimedia.org/T282051 (Ejegg) p:Triage→High [20:20:47] !log updated email preferences wiki (donorwiki) from a232fc3438 to d449599540 [20:20:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:23:29] wat [20:23:39] how is that template still not there? [20:25:28] ahhh, needs a submodule update [20:25:49] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/685574 [20:25:53] (CR) Ejegg: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/685574 (owner: Ejegg) [20:35:01] ejegg: ooooh fantastic! [20:35:07] way to go :) [20:35:30] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/685574 (owner: Ejegg) [20:36:48] !log updated payments-wiki from d449599540 to 9f51ace546 [20:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:37:33] !log updated email preferences wiki (donorwiki) from d449599540 to 9f51ace546 [20:37:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:38:42] cool cool, now the template is there [20:38:59] and we're just getting the (still expected) problem connecting to civiproxy [20:39:18] ok, now let's see what's up with that DependencyStore config [20:44:46] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) [20:46:01] fundraising-tech-ops, DC-Ops, ops-codfw: (Need By: TBD) rack/setup/install fran2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T282056 (RobH) [20:46:30] fundraising-tech-ops, DC-Ops, ops-codfw: (Need By: TBD) rack/setup/install fran2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T282056 (RobH) [21:02:07] ok, looks like we need [21:02:10] ResourceLoaderUseObjectCacheForDeps [21:02:15] to be set to true [21:03:49] hmm, it says warning: experimental here: https://www.mediawiki.org/wiki/Manual:$wgResourceLoaderUseObjectCacheForDeps [21:04:04] but... I don't see how else to keep resourceloader from trying to write to the db [21:04:26] testing locally [21:12:14] seems to work fine [21:12:26] well, better than constant errors in the logs [21:12:31] let's put it on payments [21:13:26] ok, deployed [21:13:36] and it seems to have quieted down the payments.error log [21:14:07] seems to be loading all the resourceloader modules just fine [21:14:24] will make one more update in the e-p-c localsettings [21:17:23] Jeff_Green or dwisehaupt - could you please do one more puppet deploy of the donorwiki localsettings from my frpm1001 homedir? [21:17:37] just fixing T282051 [21:17:38] T282051: Fix payments-wiki DependencyStore config - https://phabricator.wikimedia.org/T282051 [21:18:19] ejegg: sure [21:26:51] ejegg: done! [21:31:33] thanks again [21:49:47] fundraising-tech-ops: Fundraising access request for Julia Brungs - https://phabricator.wikimedia.org/T281111 (Dwisehaupt) @MeganHernandez_WMF ping to check on the access request. Did you need for me to submit it to Lisa? [21:56:08] Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog: Fix payments-wiki DependencyStore config - https://phabricator.wikimedia.org/T282051 (Ejegg) Open→Resolved a:Ejegg OK, turned out to be easy to fix by setting $wgResourceLoaderUseObjectCacheForDeps = true;