[00:02:16] cstone: ejegg AndyRussG I pushed up the desymlink commit in local settings [00:02:37] if someone can check [00:03:53] Wikimedia-Fundraising, donate.wikimedia.org: Translate donation support pages into Latin American Spanish and Portuguese - https://phabricator.wikimedia.org/T199685 (CDenes_WMF) Copying over Tomasz' email to centralize: Hi Peter, Thank you, 1. [[ https://docs.google.com/spreadsheets/d/14ccDo8dX0A-YPD... [00:05:20] hmm - I just tried on staging & it is a little scary as it seems we need to move the civicrm.settings.php & settings.php files [00:06:14] eileen: yep yep [00:06:27] actually, THAT might need an update to the deploy tool [00:06:32] dwisehaupt: ^^^ [00:07:05] I think the f_c_u settings might need a little tweak for the civicrm project [00:07:07] Also I had to do ln -s /srv/org.wikimedia.civicrm-files files [00:07:30] ahhh yeah, that's still a symlink, huh? [00:07:51] yep - which is probably an ok use [00:07:51] I guess they wanted to keep any writeable dir outside the tree entirely? [00:07:58] yeah [00:07:58] yeah. we have sites/default/civicrm.settings.php specified as a file in the frdeploy config. [00:08:22] dwisehaupt: ok, we'll want to prepend a 'drupal/' to that [00:08:41] yep drupal/sites/default/civicrm.settings.php - that should work before AND after merge [00:09:04] as we're going to get rid of the drupal/sites symlink and move the sites folder physically into drupal [00:10:28] how is the drupal tree updated? [00:11:02] https://www.irccloud.com/pastebin/OUI1eNFb/ [00:11:17] ok - staging is DOWN after I [00:11:17] - checked out the repo [00:11:17] - copied civicrm.settings.php & settings.php to the new location (I couldn't move) [00:11:17] - recreated the files symlink [00:11:17] - updated settings.php with the new path to vendor [00:11:27] it works via drush not via the UI [00:12:18] I'm inclined to delay this & plan it a bit more [00:12:29] but just thinking through how to manage the git stuff [00:12:52] (CR) Eileen: [C: -1] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/668206 (owner: Eileen) [00:13:13] since I ah that was easy - I'd forgotten how CI never gets round to merging :-) [00:14:10] the reason i ask, if the drupal dir is part of the repo that is updated, files will be overwritten/removed with the checkout and then reupdated at the end of the f_c_u process. [00:14:27] as long as that's a-ok, it's an easy update [00:16:02] eileen is it just the extra ../ needed at the start of a vendor include in the settings file? [00:16:31] I don't think so - because I did that & then ran drush cc all & it worked [00:16:32] (PS4) Ejegg: Move Endowment-related MonthlyConvert logic into hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/647276 (https://phabricator.wikimedia.org/T269717) [00:16:43] oh right, that would block drush too [00:18:13] looks like some civicrm extensions folder are left behind eileen [00:18:31] ejegg:yep - I had no permission to remove sites folder on staging [00:18:51] are all those things that were just plopped onto staging without being checked in? [00:19:19] well mostly the settings files & I guess logs & stuff [00:21:08] dwisehaupt: are the apache / civicrm error logs on frdev somewhere special? [00:21:30] I don't see a civicrm-error.log and error.log is empty despite the access-log having some 500s [00:21:51] they should be there. [00:22:21] ejegg: I have updated localsettings on frpm with changes that should work both before & after the deploy [00:23:48] there is no specific errorlog definition just the combined accesslog [00:24:40] errorlog should go to syslog based on the conf structure [00:25:00] ok, in syslog i see a failed include - drupal looking for redis.inc in the old location [00:28:12] ok - I can see it [00:29:43] kinda wonder if these drupal logs should be grouped somewhere more obvious than just syslog [00:33:16] well what I tryied changing wasn't broken until I touched it - now ok again [00:34:28] ejegg: do you want to check what I have in frpm & if that is good I'll deploy & we won't deploy the crm repo change today (focus on making sense of staging) [00:35:08] sure, I'll take a look [00:36:31] oh right, that extensionsDir setting - I guess that's OK? [00:36:54] there was something bizarre with substring searches and one of those paths, but I can't remember exactly what [00:37:45] ejegg: yeah - just got rid of the relative directories altogether [00:38:11] ok, that change looks worth a try in any case [00:38:51] aka +2 [00:41:31] ok - those 2 went out with no adverse change [00:42:36] so we still have to 1) deploy, 2) move civicrm.settings.php & settings.php , 3) recreate the symlink when we do it on live + we need to figure out what is broken on staging! [00:42:56] dwisehaupt: on staging can you move the old sites directory to my home directory? [00:42:57] yeah, i'm a bit confused by the lack of more errors in syslog [00:43:07] could it be filtering out repeated errors? [00:43:15] not sure [00:43:33] I'm tailing the syslog and reloading, and i see the access line with the 500 error but I no longer see any actual php error messages [00:43:50] eileen: to verify /srv/org.wikimedia.civicrm/sites to your homedir? [00:43:58] yes please [00:45:01] ok. it is moved to /home/eileen/old_sites_dir [00:45:11] thanks [00:45:35] ok that was part of the problem - now it wants to re-install a site :-) [00:51:41] it nginx pointing to the drupal folder as the root? [00:54:53] no. it has /var/ww as the serverroot but proxies all requests to apache [00:55:56] /var/www even. [00:58:35] i need to head off but won't be far from the computer. ping me on slack or sms/whatsapp/signal/(pretty much anything but irc) and i'll hop on. [01:05:54] I think we can leave this for now [01:06:07] no breaking change on live & staging broken is not urgent [02:13:14] Fundraising Sprint Corrugated super slide, Fundraising Sprint Downed power line jump rope, Fundraising Sprint Esperantoland, Fundraising-Backlog: Adding 2 new columns to Engage Reconciliation reports - https://phabricator.wikimedia.org/T271992 (Eileenmcnaughton) There is an inbetween stage called... [02:18:45] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Updating multiple contribution - increase limit - https://phabricator.wikimedia.org/T276083 (Eileenmcnaughton) Sure - I get 14 with this criteria https://civicrm.wikimedia.org/civicrm/admin/search#/create/Contribution?params=%7B%22version%22:4,%22selec... [04:08:23] Fundraising Sprint Corrugated super slide, Fundraising Sprint Esperantoland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: De-symlink our repos - https://phabricator.wikimedia.org/T273822 (Eileenmcnaughton) [04:12:10] Fundraising Sprint Corrugated super slide, Fundraising Sprint Esperantoland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: De-symlink our repos - https://phabricator.wikimedia.org/T273822 (Eileenmcnaughton) I just added this into the sprint as we had un-reviewed work from when... [04:36:24] Fundraising Sprint Corrugated super slide, Fundraising Sprint Esperantoland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: De-symlink our repos - https://phabricator.wikimedia.org/T273822 (Eileenmcnaughton) I wound up getting it going again on staging by 1) fully removing sit... [10:07:07] Fundraising-Backlog, Analytics, FR-Tech-Analytics: Whitelist Portal and WikipediaApp event data for (sanitized) long-term storage - https://phabricator.wikimedia.org/T273246 (Jdrewniak) @mpopov that's right, the client-side instrumentation will have to be updated for the Event Platform. @EYener Th... [12:04:30] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Updating multiple contribution - increase limit - https://phabricator.wikimedia.org/T276083 (RLewis) @Eileenmcnaughton this is great I just amended the search criteria for the $500 to $4,999 donor group and 439 were updated in one go. This is such a tim... [14:27:57] (PS1) Jgleeson: Updating composer dependencies [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/668447 [14:32:29] (CR) jerkins-bot: [V: -1] Updating composer dependencies [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/668447 (owner: Jgleeson) [15:00:04] (Abandoned) Jgleeson: Updating composer dependencies [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/668447 (owner: Jgleeson) [15:00:21] (PS1) Jgleeson: Updating composer dependencies [extensions/FundraisingEmailUnsubscribe] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/668455 [15:11:34] (CR) Jgleeson: [V: +2 C: +2] Updating composer dependencies [extensions/FundraisingEmailUnsubscribe] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/668455 (owner: Jgleeson) [15:31:54] (PS1) Jgleeson: Update twig/twig [core] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/668463 (https://phabricator.wikimedia.org/T271212) [17:18:45] (Abandoned) Ejegg: Deleted non-custom es-419 messages [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/668197 (https://phabricator.wikimedia.org/T276367) (owner: Ejegg) [17:52:25] Fundraising-Backlog, FR-LATAM: Mexico payment form error - https://phabricator.wikimedia.org/T275156 (DStrine) [17:52:44] Fundraising-Backlog, FR-LATAM: TY page for failed BT transaction in Brazil - https://phabricator.wikimedia.org/T275577 (DStrine) [17:52:46] Fundraising-Backlog, FR-LATAM: Brazil TY page in English and not Portuguese - https://phabricator.wikimedia.org/T275576 (DStrine) [17:52:54] Fundraising-Backlog, FR-LATAM: DLocal not loading on IE 11 - https://phabricator.wikimedia.org/T276333 (DStrine) [17:53:19] Fundraising-Backlog, FR-LATAM, FR-dlocal: DLocal not loading on IE 11 - https://phabricator.wikimedia.org/T276333 (DStrine) [17:53:53] Fundraising-Backlog, FR-LATAM, FR-dlocal: Mexico payment form error - https://phabricator.wikimedia.org/T275156 (DStrine) [17:54:23] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-drupal-upgrade-2021: Migrate dash to use new oauth - civicrm or drupal extension/module - https://phabricator.wikimedia.org/T276394 (DStrine) [17:55:17] Fundraising-Backlog, FR-Italy, fr-donorservices: Italian mobile form obscures the card number field - https://phabricator.wikimedia.org/T275458 (DStrine) [17:55:59] Fundraising-Backlog, FR-Sweden, Patch-For-Review: Dynamic 3ds for Sweden? - https://phabricator.wikimedia.org/T220427 (DStrine) [17:56:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-India: CiviCRM queue fail due to emojis in address or an Indian script being used - https://phabricator.wikimedia.org/T261713 (DStrine) [17:56:49] Fundraising-Backlog, FR-India: Delays in transaction completion for India - https://phabricator.wikimedia.org/T259195 (DStrine) [17:57:16] Fundraising-Backlog, FR-India, fr-donorservices: Make panExplain form variant the default in India - https://phabricator.wikimedia.org/T259146 (DStrine) [17:57:26] Fundraising-Backlog, FR-India, MW-1.36-notes (1.36.0-wmf.2; 2020-07-28): AMEX cards not working for india donors - https://phabricator.wikimedia.org/T258509 (DStrine) [17:57:52] Fundraising-Backlog, FR-India, FR-dlocal, MW-1.35-notes (1.35.0-wmf.27; 2020-04-07): PAN Verification issue during India Test campaign - https://phabricator.wikimedia.org/T234342 (DStrine) [18:14:06] Fundraising Sprint Corrugated super slide, Fundraising Sprint Downed power line jump rope, Fundraising Sprint Esperantoland, Fundraising-Backlog: Adding 2 new columns to Engage Reconciliation reports - https://phabricator.wikimedia.org/T271992 (MDemosWMF) @Eileenmcnaughton Ahh I see that now. Tha... [18:16:16] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Updating multiple contribution - increase limit - https://phabricator.wikimedia.org/T276083 (LeanneS) Amazing! Thank you, @Eileenmcnaughton! It's also super helpful to see how you built this search as this kind of searching happens often. [19:12:34] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Astropay-DLocal_endpoint_critical 2 [=1] [19:17:34] RECOVERY - check_log_messages on frav1002 is OK: OK [19:18:13] AndyRussG: how would you feel if we didn't start in the 0.0.*'s on the fundraising dev docker image versions? [19:18:44] I can see it's a convention presumably from debian used on the other images but it feels unusual when tagging/versioning our imags [19:18:57] hmm sure! [19:19:12] I was thinking maybe we start from 1.0.0 [19:19:12] !log replication restarted on frdb1004 after utf8mb4 conversion completed. [19:19:16] if we wanted the semantic versioning [19:19:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:19:23] as opposed to 0.0.1 [19:20:24] ok cool thanks [19:30:53] fr-tech another fun link: https://www.docker.com/blog/experimenting-with-rootless-docker/ [19:30:54] AndyRussG: so I've updated the fundraising-buster-73 image to try out the slightly different ssl certs to confirm whether or not the commonname fix works [19:31:05] jgleeson ah cool! [19:31:11] and after running docker-pkg [19:31:16] it's rebuilding that one image [19:31:24] but I think I need to rebuild the other right [19:31:48] so if I clear those out and rebuild them all (for the hierarchy to be up-to-date) [19:31:53] is that right? [19:32:12] jgleeson: depends on how recently you built the others [19:32:24] but to be sure you could certainly clear them all out [19:33:13] hmm [19:33:42] I'm assuming I need to at least rebuild the civicrm images in order to pull in the new ssl cert (from the base buster-73 image) [19:33:56] jgleeson: ah right, yes if you modified that one [19:34:35] docker-pkg would recognize the need to rebuild when versions are bumped, which you should probably do also, I think [19:35:07] how about setup.sh again? the ssl certs are baked into the image so i shouldn't need to rerun setup I think [19:36:07] ah AndyRussG docker-pkg didn't detect the changes to the other images [19:36:27] I was expecting it would [19:36:31] or had a memory of it doing that [19:36:37] instead this is all i saw [19:36:56] https://phabricator.wikimedia.org/P14637 [19:37:13] I'm gonna be offline today but I got staging up again & made some notes on the phab - I think maybe we put the sites back as a symlink as an interim since there seem to be a few paths that are cached in various places. Anyway the challenge is that it will be hard to deploy other stuff until that commit goes out [19:37:32] because you can't really cherry-pick passt it [19:37:41] hey eileen [19:37:42] so the sites back as a symlink is the reverse of existing [19:37:45] hey [19:37:52] (I'm out the door pretty soon) [19:38:18] jgleeson: did you also update the version that the last image depends on? [19:38:53] I added a new changelog item to the fundraising-buster-php73 image [19:39:09] which is based from by the other images [19:39:27] that's it [19:39:31] hmmm [19:39:37] is that right? [19:39:38] eileen: oh dang, that's a bummer to have to go back on that patch [19:39:55] jgleeson: sounds right, I guess, since the "FROM" tags in the Dockerfile.template are supposed to pull the most recent version [19:39:56] AndyRussG: let me get some useful links [19:39:58] is staging back to symlinked then? [19:40:25] ejegg: so it's not back [19:40:27] jgleeson: https://doc.wikimedia.org/docker-pkg/ this might help? I guess no harm in also forcing it to rebuild the parent image by just removing it [19:40:39] the sites folder is still moved [19:40:56] but I think having a symlink from the old location to the new short-term might be prudent [19:42:04] my guess is I need to remove the descendants also (e.g. fundraising-civicrm-buster-php73 *_ [19:42:20] the parent image is rebuilt [19:42:24] ok I have to shoot - but basically the drupal/sites is still the real folder - we wouldn't reverse that change - but shortterm have a symlink where the old real folder was [19:42:26] byeee [19:42:28] that's the fundraising-buster-php73 one [19:42:31] bye eileen ! [19:43:01] AndyRussG: I'll remove all images and rebuld, that should do it [19:43:21] sometimes docker complains when I do that [19:43:31] about dependency issues [19:43:48] jgleeson: hmmm yeah sometimes it doesn't get the order right indeed [19:48:10] Wikimedia-Fundraising-Banners: Update ptBR desktop banners to fit on modern vector - https://phabricator.wikimedia.org/T276254 (Pcoombe) Open→Resolved Made these changes on desktop large, started out following France but ended up tweaking it quite a bit. The Portuguese text seems to be quite a bit sh... [19:48:11] jgleeson: just thinking about the verisonning stuff, if the rest of the docker images in dev-images start with 0.0.1... is there a specific advantage to doing it differently? [19:49:05] I guess as 0.0.1 means a patch release it just feels a bit weird [19:49:20] feels like the initial version of the imageis a major release right? [19:49:26] image is* [19:50:00] yeah I did mention other images are using the 0.0.1 versioning but wondering how you'd feel us starting from 1.0.0 [19:51:17] AndyRussG: some info on versioning here https://semver.org/#how-do-i-know-when-to-release-100 [19:54:41] fundraising-buster-php73 (0.0.1-1) wikimedia; urgency=medium [19:54:52] that version means a pre release of 0.0.1 [19:55:13] semantic recommends starting at 0.1.0 [19:55:18] during the development stages [19:55:46] but I guess it's up to us. I personally think if it works and we're happy to build on it, it feels like a 1.0.0 [20:00:44] jgleeson: k yeah that makes sense [20:13:47] hmm that didn't fix it [20:14:00] actually it looks like the civicrm site is still using the old cert [20:14:54] which is odd cause they both point to the same docker-ssl cert in the apache config [20:15:54] argh I'll have to pick it up tomorrow [20:16:00] catch you tomorrow fr-tech! [20:19:43] fr-tech sorry, i'm getting pulled away for more trip prep. I'll use a bit more PTO [20:26:25] (CR) Jgleeson: [C: +2] "Looks good, thanks for the update!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/647266 (https://phabricator.wikimedia.org/T269717) (owner: Ejegg) [20:42:38] (Merged) jenkins-bot: Change logo in a hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/647266 (https://phabricator.wikimedia.org/T269717) (owner: Ejegg) [21:54:58] just back briefly to say we are not in the tsunami zone (we turned back from where we were going & now are just having lunch) [21:57:58] * dwisehaupt hits the news site to find out what eileen is talking about. [21:59:18] yeowch. [22:04:18] Fundraising-Backlog, fundraising-tech-ops: Fix our database to support emojis - convert tables to utf8mb4 - https://phabricator.wikimedia.org/T270218 (Dwisehaupt) frdb1004 complete. Timing was 25.7 hours. Good and consistent with the similar footprint hosts. [22:04:36] Fundraising-Backlog, fundraising-tech-ops: Fix our database to support emojis - convert tables to utf8mb4 - https://phabricator.wikimedia.org/T270218 (Dwisehaupt)