[01:03:37] Wikimedia-Fundraising-Banners: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (spatton) [02:09:03] (PS1) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683765 (https://phabricator.wikimedia.org/T281441) [02:09:06] (PS1) Ejegg: WIP Adyen: send drop-in web config to front-end [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683766 (https://phabricator.wikimedia.org/T281509) [02:09:08] (PS1) Ejegg: WIP Adyen show drop-in web payment fields [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683767 (https://phabricator.wikimedia.org/T281516) [02:11:35] (CR) jerkins-bot: [V: -1] Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683765 (https://phabricator.wikimedia.org/T281441) (owner: Ejegg) [02:11:38] (CR) jerkins-bot: [V: -1] WIP Adyen show drop-in web payment fields [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683767 (https://phabricator.wikimedia.org/T281516) (owner: Ejegg) [02:11:47] (CR) jerkins-bot: [V: -1] WIP Adyen: send drop-in web config to front-end [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683766 (https://phabricator.wikimedia.org/T281509) (owner: Ejegg) [02:12:33] oh right, new smashpig version [02:37:49] (Abandoned) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683765 (https://phabricator.wikimedia.org/T281441) (owner: Ejegg) [02:38:06] (Abandoned) Ejegg: WIP Adyen: send drop-in web config to front-end [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683766 (https://phabricator.wikimedia.org/T281509) (owner: Ejegg) [02:38:16] (Abandoned) Ejegg: WIP Adyen show drop-in web payment fields [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/683767 (https://phabricator.wikimedia.org/T281516) (owner: Ejegg) [02:39:22] (PS1) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683770 (https://phabricator.wikimedia.org/T281441) [02:39:24] (PS1) Ejegg: WIP Adyen: send drop-in web config to front-end [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683771 (https://phabricator.wikimedia.org/T281509) [02:39:26] (PS1) Ejegg: WIP Adyen show drop-in web payment fields [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683772 (https://phabricator.wikimedia.org/T281516) [02:40:50] (PS1) Ejegg: Update SmashPig and PHPMailer [core] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/683773 [02:41:02] (CR) Ejegg: [C: +2] Update SmashPig and PHPMailer [core] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/683773 (owner: Ejegg) [02:43:14] (CR) jerkins-bot: [V: -1] WIP Adyen show drop-in web payment fields [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683772 (https://phabricator.wikimedia.org/T281516) (owner: Ejegg) [02:51:20] (Merged) jenkins-bot: Update SmashPig and PHPMailer [core] (fundraising/REL1_35) - https://gerrit.wikimedia.org/r/683773 (owner: Ejegg) [03:29:27] Wikimedia-Fundraising-Banners, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) [03:31:02] Wikimedia-Fundraising-Banners, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) Thanks for raising the ticket! https://grafana.wikimedia.org/d/000000566/overview?viewPanel=16&orgId=1 shows the... [04:17:22] Fundraising-Backlog, FR-Adyen, MediaWiki-extensions-DonationInterface: Make errors in drop-in card fields look like other errors - https://phabricator.wikimedia.org/T281550 (Ejegg) [04:29:23] eileen: hi! if you're around... wondering if you have any thoughts on the remaining two manual steps for Docker civiproxy: https://etherpad.wikimedia.org/p/adfadf-civiproxy [04:30:01] so (1) being setting CIVICRM_SITE_KEY to a known value as part of our wmff setup script [04:30:30] and (2) being automating setting the admin user's API key as a post-install step [04:30:36] thx in advance! [04:31:58] oh right - - I think there is a file in build kit - I'll check [04:32:23] eileen: thanks! [04:33:00] Fundraising-Backlog, FR-Adyen, MediaWiki-extensions-DonationInterface: Handle payment submission in Adyen Drop-In web - https://phabricator.wikimedia.org/T281528 (Ejegg) [04:33:23] Fundraising-Backlog, FR-Adyen, MediaWiki-extensions-DonationInterface: Handle payment submission in Adyen Drop-In web - https://phabricator.wikimedia.org/T281528 (Ejegg) [04:42:44] Wikimedia-Fundraising-Banners, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) I think it might be this one ? https://meta.wikimedia.org/wiki/MediaWiki:Centralnotice-template-B2021_120500_en6C... [04:52:36] AndyRussG: so there is some config but I'm not sure it's 100% working to add a civicrm.settings.d folder in our docker set up https://docs.civicrm.org/dev/en/latest/tools/civibuild/#settings-civicrm [04:53:05] (we seem to have a folder called settings & one called drupal settings but not civicrm.settings.d) [04:54:21] if that folder is in one of the specified places & has a 'pre' folder with a php file in it that defines the site key that will load first & be 'the one' [04:56:20] oh interesting [04:56:21] actually we do have a folder under our crm dir - drupal/sites/default/civicrm.settings.d - the stuff in there would only be loaded on buildkit sites (ie not production) [04:56:40] hmmmm [04:59:41] eileen: so in dmaster.sh in civi-sites I see a line that sets CIVI_SITE_KEY [05:00:10] though that is different from the variable in settings, which is CIVICRM_SITE_KEY [05:01:41] yeah that DOES look like it should be pass-in-able [05:01:41] [ -z "$CIVI_SITE_KEY" ] && CIVI_SITE_KEY=$(cvutil_makepasswd 16) [05:03:26] (CR) AndyRussG: E-mail pref ctr queue consumer using api (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [05:03:37] (PS12) AndyRussG: E-mail pref ctr queue consumer using api [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [05:03:45] AndyRussG: have you tried in civibuild.conf [05:04:12] eileen: no haven't but I will! [05:04:37] actually wasn't planning to try it this evening, but I wanted to get your feedback on it while you're about :) [05:04:47] maybe what was tripping Jack up was the slightly different variable names [05:04:58] eileen: also btw https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/682349 is now rebased, and works for me :) [05:05:04] oh yeah [05:05:28] the thing is I think there are some different keys with similar names.... [05:05:40] hmmm right [05:06:02] I think I checked though and the dmaster site key that gets into the config is indeed the one set in that file [05:06:09] heheh tonight I have to sleep at a decent-ish hour, since Sofi has Zoom class at 8 am [05:07:16] yeah sleep is a good thing to do :-) [05:12:14] hmmm however Sofi does not seem to think so... ahhh [05:12:21] eileen: cya around, thanks! [05:16:11] night [07:28:11] Wikimedia-Fundraising-Banners, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (FriedrickMILBarbarossa) [08:02:39] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 1 [=1] [08:07:35] RECOVERY - check_log_messages on frav1002 is OK: OK [09:52:10] (PS2) Thiemo Kreuz (WMDE): Optimize regular expression code in PHP and JavaScript [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/683619 [09:57:59] (CR) jerkins-bot: [V: -1] Optimize regular expression code in PHP and JavaScript [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/683619 (owner: Thiemo Kreuz (WMDE)) [10:06:58] (PS3) Thiemo Kreuz (WMDE): Optimize regular expression code in PHP and JavaScript [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/683619 [12:34:56] Wikimedia-Fundraising-Banners, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Pcoombe) Huh, I thought we'd fixed this at some point but I guess not. It only happens for desktop large... [14:13:01] Wikimedia-Fundraising-Banners, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Pcoombe) Open→Resolved a:Pcoombe Okay, this is fixed in current best ([diff](https://meta.wik... [14:40:18] Fundraising Sprint Humongous bacteria petting zoo, Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, FR-Email: Civi export data for 2 fields are preventing accurate data validation reports in Acoustic - https://phabricator.wikimedia.org/T270731 (KHaggard) ... [15:49:11] (PS14) AndyRussG: Add CiviProxy setup [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/673094 (https://phabricator.wikimedia.org/T268683) (owner: Jgleeson) [15:51:55] (CR) AndyRussG: [C: +1] "Thanks for this!!" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/673094 (https://phabricator.wikimedia.org/T268683) (owner: Jgleeson) [15:53:06] (PS2) AndyRussG: Set up SmashPig database [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/682190 (https://phabricator.wikimedia.org/T279872) [15:57:43] (PS15) AndyRussG: Add CiviProxy setup [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/673094 (https://phabricator.wikimedia.org/T268683) (owner: Jgleeson) [16:00:38] (CR) AndyRussG: Pare down email pref ctr config (1 comment) [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (owner: Ejegg) [16:03:38] (CR) AndyRussG: "Looks good, other than inline comment (singular--there's just one that I put on the wrong line by mistake). Thanks!!" (2 comments) [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (owner: Ejegg) [16:04:03] Fundraising Sprint Corrugated super slide, Fundraising Sprint Downed power line jump rope, Fundraising Sprint Esperantoland, Fundraising Sprint File Systems Stage Show, and 6 others: Create civiproxy on docker - https://phabricator.wikimedia.org/T268683 (jgleeson) I added some docs around the out... [16:24:38] (CR) XenoRyet: [C: +2] Update 2021 thank you emails [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/683450 (https://phabricator.wikimedia.org/T278363) (owner: Cstone) [16:47:08] (CR) Ejegg: "Thanks for the CR - pretty sure we're not using that config outside of the legacy Unsubscribe extension." (1 comment) [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (owner: Ejegg) [16:48:03] AndyRussG: for the SmashPig database - two of the scripts in that Schema dir are actually for fredge tables [16:48:36] I keep meaning to make a maintenance script in SmashPig to create all tables with the connections specified in the SmashPig config [16:49:48] But for the patch at hand, maybe just explicitly list the 'damaged' and the 'pending' table SQL files [16:50:16] (Merged) jenkins-bot: Update 2021 thank you emails [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/683450 (https://phabricator.wikimedia.org/T278363) (owner: Cstone) [16:52:50] ah, and I'll make another patch to create the fredge db + all those tables [16:54:30] ejegg: so this isn't sufficient? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/682190 [16:54:36] it works for me to create the damaged table [16:55:30] ejegg: see line 649 here: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/682190/2/setup.sh [16:55:59] ejegg: also there were some questions in standup about where things were currently at wrt deploying e-mail pref ctr [16:56:27] I didn't find the file you mentioned in frpm1001:/tmp but the updates to civiproxy config in localsettings look fine [16:58:09] oh hmm, one sec and I'll check on the tmp file [16:58:57] or wrt smashpig setup, you're saying we should make it not run the fraud table create [17:02:33] Wikimedia-Fundraising-Banners, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) Thanks @pcoombe the errors seem to have stopped now! [17:17:44] AndyRussG: yep yep, exactly, better not to copy those tables to a different database than production [17:28:44] ejegg: K I mean no problem, but I guess I don't understand? [17:28:57] like why don't we want in local setup equivalent tables to those in prod? [17:29:34] (CR) AndyRussG: [C: -1] "Apparently still some details to work out here, so setting CR to -1 meanwhile..." [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/682190 (https://phabricator.wikimedia.org/T279872) (owner: AndyRussG) [17:29:59] ejegg: can you post an explanation and details of suggested changes on the Gerrit change? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/682190 thx much! [17:52:33] (CR) Ejegg: "The Schema folder has two scripts for the fredge DB - let's only create the pending and damaged tables in the smashpig db, to match produc" (2 comments) [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/682190 (https://phabricator.wikimedia.org/T279872) (owner: AndyRussG) [17:52:42] AndyRussG: ^^ [18:13:24] ahh yeah i remember being confused by those before [18:13:34] as ejegg|food rightly points out those tables belong to a separate 'fredge' db that we've already created (and added those tbls to) in our Docker setups as part of the earlier CiviCRM/Drupal build. [18:18:30] So my understanding is that 'fredge' first comes into play at the frontend. DonationInterface pushes messages to payments-antifraud and payments-initial queues. Those messages are then consumed and processed by the AntiFraudQueueConsumer & PaymentsInitQueueConsumer over in Drupal-land and saved to the respective db tables (named the same as the queues). That data is then reported on and use in [18:18:32] CiviCRM [18:19:17] jgleeson ejegg cstone ah okok thx! [18:22:36] there is another point of confusion [18:22:51] there is an additional tbl called '`payments_fraud_breakdown`' [18:23:12] which currently only gets created in the soon-to-be-retired fundraising-dash codebase [18:23:45] that's where we go threaten the fraudsters to return the money they stole? [18:23:56] ah no, that's fraud_shakedown sorry [18:23:58] the shakedown tbl [18:24:00] :) [18:24:12] haha [18:25:42] we're probably gonna have to put the 'payments_fraud_breakdown' SQL somewhere else going forward because that tbl is used in Civi/Drupal [18:26:01] ejegg|food: is this on your radar? and why are you working [18:26:19] bad tech-lead [18:31:10] AndyRussG: just scanning over the email-pref-queue-consumer stuff [18:31:25] is there an associated smashpig/other patch that this one depends on https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/682349 ? [18:33:07] haha [18:33:22] just dropping in for a bit today [18:33:25] :) [18:33:33] since the city is chaos outside :S [18:33:39] oh no! [18:33:47] covid scares? [18:34:32] huge protests against new taxes on basic necessities started yesterday, lots of desperate folks taking advantage of the situation to loot [18:34:59] banks and bus stations on fire, chain stores stripped to the walls [18:35:26] ah shit, that sounds pretty chaotic alright [18:35:56] hows you and your neighbours holding up? [18:36:59] we're fine - the neighborhood whatsapp groups have some panicky folks, but so far nobody's attacking houses around here [18:37:11] good to hear [18:37:22] so about that fredge schema [18:38:05] I think the full set of tables is described in our crm repo still [18:38:07] lemme find that [18:40:29] ah, yeah, wmf_fredge_qc.install describes them all [18:40:40] but it's in drupal table definition format, not SQL [18:41:34] OK, so I think we just need to create the fredge db before we do the WMFF install [18:41:47] and make sure the WMFF database settings include fredge [18:42:48] lessee, I think eileen did something tricksy to get those tables prefixed with fredge. [18:44:30] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/625770 [18:45:25] jgleeson: i think you were looking for this smashpig patch earlier if you hadnt found it https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/682190 [18:46:33] so I'm guessing that without a ['fredge' => [ ... block in the drupal settings, it's currently creating those in the drupal db? [18:46:56] yep, those three tables are all in the drupal db [18:48:35] I'm hopping off so will pick up on the review Monday. Also dstrine & XenoRyet were asking about progress on getting email-preferences to production. Looks like we're going to try to do it Monday/Tuesday? I think there's one patch waiting on review and the nginx/apache setup to do from Jeff_Green & dwisehaupt outstanding but once we get those we could talk about the end-to-end deploy test [18:48:40] fr-tech^ [18:48:49] thanks cstone ! [18:52:42] AndyRussG: does that suggestion make sense, just changing the glob to create the two tables we need? [18:55:21] ejegg: sounds decent, I'm just about to properly sit down to do some more work, after having been a bit distrcted [18:55:25] AndyRussG: oh interesting, I see no create scripts for the wmff and drupal dbs [18:55:26] just gonna make moar coffee [18:55:34] I guess buildkit even does the db creation [18:55:48] ejegg: there is a wmff.sh somewhere [18:56:23] got it [18:57:23] ejegg: I don't recall where it lives repo-wise but it does eventually make its way to /src/civi-sites [18:57:27] along with dbmaster.sh [18:57:35] (not a src controlled location, tho) [18:57:41] right [18:58:56] ok, so wmff/install.sh has a commented out bit to create the fredge db [18:59:05] hmmm [18:59:21] I wonder why [19:00:01] I wonder how [19:00:30] ok, and there's also some settings for the fredge db in the drupal db settings [19:00:44] let's see if those made it to civi-sites [19:01:42] nope, they didn't [19:03:04] so... how is the stuff in buildkit's app/config/wmff/drupal.settings.d supposed to get copied into the final settings file? [19:03:16] or should we just copy that whole directory in? [19:05:37] hmm, i don't see anything about using a settings.d folder outside of civibuild, so I guess buildkit is responsible for concatenating those into the final sites/default/settings.php file [19:09:58] ok, so civibuild scans that dir inside the drupal7_install function [19:10:13] quite the puzzle, eh? [19:10:31] could we be skipping that? [19:11:14] nope, not skipping it afaict [19:11:27] I wonder if you can debug into bash scripts [19:11:44] calling drupal_install right on line 21 of wmff/install.sh [19:11:54] and then that calls drupal7_install [19:17:03] hmm hmm, hard to see exactly how that cvutil_inject_settings actually adds them [19:18:19] sortof looks like they need to be in a further subfolder pre.d or post.d [19:18:31] lemme try moving them [19:18:49] hmm, I should disable the civimodule installs for quicker testing [19:48:18] fundraising-tech-ops, SRE, Patch-For-Review: (Need By: TBD) rack/setup/install payments100[5-8] - https://phabricator.wikimedia.org/T266481 (Jclark-ctr) [19:48:26] fundraising-tech-ops, DC-Ops, SRE, ops-eqiad: (Need By: TBD) install second SSD into payments100[5-8] - https://phabricator.wikimedia.org/T278250 (Jclark-ctr) Open→Resolved [19:52:22] dang, that didn't work [19:58:24] ejegg: just saw your notes above about the chaos there--stay safe! [19:58:40] so in my local Docker setup I don't see a fredge table [19:58:55] is that what you mentioned above ^ about adding a pre-civi-install step to create it? [19:59:29] I do however see payments_initial and payments_fraud in my local drupal database [20:00:03] right AndyRussG [20:00:09] they should be in the fredge database instead [20:00:22] ejegg: ok... and so how did they get in the drupal db? [20:01:14] btw I've been destroying and rebuilding the db regularly [20:01:42] so these settings need to be included in the drupal settings file in order to get them into fredge: https://github.com/civicrm/civicrm-buildkit/blob/master/app/config/wmff/drupal.settings.d/50-wmff.php [20:02:32] and this line needs to be uncommented (and the db name put after IF NOT EXISTS): https://github.com/civicrm/civicrm-buildkit/blob/master/app/config/wmff/install.sh#L41 [20:02:52] so I uncommented and fixed that line locally, and the fredge db is created [20:03:25] but the settings are included via a somewhat circuitous path [20:03:35] and I haven't been able to debug through it yet [20:04:16] AndyRussG: do you have debugging working on docker / wmff site? [20:04:43] I'd love to step into a thing in sites/default/settings.php [20:04:49] ejegg: yep works fine :) [20:05:00] want to hop on a video call for a sec? [20:05:06] ok sure! [20:05:14] today's standup? [20:05:22] okok [20:17:16] ejegg: lost you on the call there [20:17:23] ah dang [21:02:14] Fundraising-Backlog, FR-Docker: Docker dev setup: fix fredge database setup - https://phabricator.wikimedia.org/T281609 (AndyRussG) [21:26:04] AndyRussG: aha, it's trying to create it now, it's just failing because the drupal user doesn't have table create rights in the fredge db [21:26:32] should be surmountable [21:27:00] ejegg: hmmm there are settings we have somewhere or other for civicrm to get the right db passwords [21:27:17] and the root db password in docker is blank [21:29:29] ejegg: just smoke testing the fix for the sp db patch [21:32:27] (PS3) AndyRussG: Set up SmashPig database [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/682190 (https://phabricator.wikimedia.org/T279872) [21:34:07] (CR) AndyRussG: "Thanks!!" (2 comments) [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/682190 (https://phabricator.wikimedia.org/T279872) (owner: AndyRussG) [21:34:37] ejegg: ^ k that's fixed there [21:36:41] (CR) Ejegg: [C: +2] "Looks good!" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/682190 (https://phabricator.wikimedia.org/T279872) (owner: AndyRussG) [21:43:32] i think i just need a grant statement, trying again [21:46:09] ah cool ^ thx! [21:47:42] (CR) AndyRussG: [V: +2] Set up SmashPig database [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/682190 (https://phabricator.wikimedia.org/T279872) (owner: AndyRussG) [21:48:31] heh still needs manual submit [21:53:25] oh ya [21:54:56] Wikimedia-Fundraising-Banners, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) Resolved→Open p:Triage→Unbreak! I spoke too soon. This just triggered our aler... [21:57:10] (PS1) Ejegg: Restore fredge db, move drupal extra settings [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/683994 [21:57:27] so that's the fix for buildkit ^^^ [21:57:33] which I see is still being cloned from gerrit [21:57:48] anyway, I'll put in a PR for upstream too, to keep 'em in sync [22:00:25] (PS1) Ejegg: Delete drupal settings override [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683995 [22:00:37] (PS2) Ejegg: Delete drupal settings override [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683995 [22:14:17] ok AndyRussG those two patches should do it [22:14:37] and the matching PR upstream should keep it in sync for when we switch the builkdit source [22:22:04] Wikimedia-Fundraising-Banners, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Pcoombe) Open→Resolved @Jdlrobson This isn't banners now. There are lots of errors on zh.m.wikipe... [22:23:30] anyone curious about the situation in Cali can follow here: https://twitter.com/EnterateCali [22:27:25] Wikimedia-Fundraising-Banners, RelatedArticles, Revision-Slider, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) Resolved→Open I don't thin we can discount b... [22:28:17] Then there's this account for the rest of the country: https://twitter.com/Col_Informa [22:31:59] ejegg: thanks for the patches and the links! [22:42:18] Wikimedia-Fundraising-Banners, RelatedArticles, Revision-Slider, Spanish-Sites, Wikimedia-production-error: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) @Jdforrester-WMF could your changes as part of T2500... [22:43:09] Wikimedia-Fundraising-Banners, RelatedArticles, Revision-Slider, Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: Error with frb.initNag in some banner(s) on Spanish and Portugese Wikipedias - https://phabricator.wikimedia.org/T281547 (Jdlrobson) [22:45:51] Wikimedia-Fundraising-Banners, RelatedArticles, Revision-Slider, Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: TypeError: Cannot read property 'top' of undefined - https://phabricator.wikimedia.org/T281547 (Jdlrobson) [22:46:17] Wikimedia-Fundraising-Banners, RelatedArticles, Revision-Slider, Readers-Web-Backlog (Kanbanana-FY-2020-21), and 2 others: TypeError: Cannot read property 'top' of undefined - https://phabricator.wikimedia.org/T281547 (Pcoombe) There's no banner changes that line up with that time. Since 16:00 UT... [23:20:51] AndyRussG: was there some issue with civiproxy pulling in the dmaster api key instead of the wmff one, I just rebuilt civiproxy and it errors on the one from wmff