[01:06:30] (CR) Ejegg: [C: +2] "> Patch Set 2:" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/683114 (owner: Eileen) [01:08:30] (Merged) jenkins-bot: Copy differences as commented out [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/683114 (owner: Eileen) [01:34:05] ejegg: so I shouldn't try debugging into the sqlite code path like u did to see if there's an easy code-y way to fix the error? [01:34:37] AndyRussG: If you look here: https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/db/MWLBFactory.php$203 [01:35:14] it seems like we'd need to hack core to allow a POST to not use IMMEDIATE [01:35:25] and I'm pretty sure we don't want to submit the form with 'GET' [01:41:57] if you can find otherwise, it would be great! [01:44:21] ejegg: right I saw that, thanks [01:44:58] ejegg: K I'll dig a bit but assume not too much assuming the mariadb option is probably ok to then [01:45:20] ejegg: can you think of anything else I can look at now for e-p-c deploy? [01:45:28] So since that query is in the context of looking up DB blocks, I had hopes of getting around it by setting BlockDisablesLogin to false: https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/Permissions/PermissionManager.php$713 [01:47:45] but then I hit another place that uses the block logic: https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/MediaWiki.php$779 [01:49:05] hmm, maybe we could get couldBePublicCached to return true somehow? [01:54:05] AndyRussG: If you want to test the consumer, you can use the PopulateQueueFromDump smashpig maintenance script to inject a message into the queue [01:56:31] oh, but $output->haveCacheVaryCookies() is always true right there, so yah, I think there's no real way around that [02:04:04] ejegg: ah okok yeah the queue consumer sounds like a fun test! [02:57:13] Fundraising-Backlog, FR-Docker: Docker dev setup: follow-on Civiproxy setup - https://phabricator.wikimedia.org/T281651 (AndyRussG) [02:58:03] Fundraising-Backlog, FR-Docker, Patch-For-Review: Docker dev setup: fix fredge database setup - https://phabricator.wikimedia.org/T281609 (AndyRussG) I think this may be all done? [07:30:36] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1213 [07:35:36] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 3591547 Threads: 17 Questions: 95144885 Slow queries: 2946910 Opens: 759394455 Flush tables: 1 Open tables: 236 Queries per second avg: 26.491 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [13:52:36] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 5 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/mysql/cacert.pem],File[/etc/mysql/server-cert.pem],File[/etc/mysql/server-key.pem] [13:57:36] RECOVERY - check_puppetrun on frdata2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [14:55:48] howdy fr-tech! [14:56:59] jgleeson fr-tech hi! [14:57:55] hi hi. AndyRussG do you know if anyones managed to replicate the sqlite-readonly-POST bug locally, that ejegg|away put in his EOD email? [14:58:13] jgleeson: just replying, yes I did [14:58:17] or I think it was the same bug [14:58:27] however it works fine if you just make the sqlite file read-write! [14:58:39] oh nice [14:59:02] so we might not need another mariadb instance afteall [14:59:08] after all * [14:59:21] well it depends I guess on how hard we want to lock down the db [15:00:06] I don't think we're putting anything sensitive in it [15:02:23] yeah [15:02:41] I think maybe the idea was to have something parallel to the payments setup, where I think somehow the dbs are in read-only mode? [15:07:56] jgleeson: there is already $wgReadOnly set to true in LocalSettings btw [15:11:06] ah cool [15:11:58] Roxy's grandparents just left so I've been a little distracted. My road has been closed at the top end so the cars were crawling along. Rush hour fun [15:12:52] jgleeson oh hey grandparents' visit is important! [15:13:02] they're all vaccinated I imagine? [15:13:06] morn [15:13:17] dwisehaupt hi!! [15:13:33] dwisehaupt just a thought here, how do you feel about temporarily making the sqlite file writable? [15:13:44] at least as a temporary measure to get this out the door [15:14:00] instead of spinning up the whole mariadb thing? [15:15:07] (sorry to hit you with this first thing in the morning hehe) [15:15:29] i'm actually ok with that in the short term. not sure what the downside to that would be. other than possible stray edits. [15:15:52] i'm just getting to email but it looks like jeff may have rolled the changes to install mariadb on that role. [15:15:52] dwisehaupt ok that sounds great! yeah we'd keep $wgReadOnly set to true [15:16:02] ahhh nocie! [15:16:04] noice* [15:16:09] yep AndyRussG vaccinated [15:16:18] jgleeson: fantastic :) [15:16:22] they've had two [15:16:28] I haven't had mine yet [15:16:31] ah great [15:16:36] yeah me neither [15:16:45] i'd like to at least check in a see his thoughts and what he completed since i was last on. [15:17:35] I signed up on a web site and they're supposed to call me sometime for the actual appointment, but I'm actually at this point one month younger than the cutoff age [15:17:43] dwisehaupt cool thanks much! :) [15:20:01] AndyRussG: maybe you'll get a call next month! [15:20:11] I think I've gotta wait about a month also [15:20:34] Roxy has her second booked in for sunday. She has asthma tho [15:20:43] ah glad she can get hers! [15:25:16] AndyRussG: I'm curious why we didn't run into that same issue locally. Is our setup locally different in some way? [15:25:26] the sqlite readonly POST thing [15:25:47] jgleeson: on fundraising-dev e-p-c uses mariadb currently [15:26:10] I had to switch to sqlite and run the mw setup web wizard to get it to work actually [15:26:25] btw where to we have smashpig ipn listeners installed on the cluster? [15:26:33] oh I see are we using mysql locally? [15:26:47] for EPC [15:26:48] yep! [15:26:51] gotcha [15:27:05] hmm [15:27:07] re:smashpig [15:27:18] the lie on frpig* I think [15:27:23] live and lie [15:27:58] ahh right makes sense :) [15:35:43] jgleeson dwisehaupt so I'm about to test out the queue consumer by injecting a fake message into the queue [15:35:58] cool [15:36:03] frpig1001:~$ php /srv/www/org/wikimedia/listeners/SmashPig/Maintenance/PopulateQueueFromDump.php --queue email-preferences --file test_e-p-c_msg.json [15:36:11] just attempting the sqlite setup locally also [15:36:20] ah cool [15:37:24] ah updated command attempt [15:37:26] php /srv/www/org/wikimedia/listeners/SmashPig/Maintenance/PopulateQueueFromDump.php --queue email-preferences test_e-p-c_msg.json [15:37:31] that ran but spat out an error [15:37:45] looks good [15:37:47] 2021-05-07T15:36:46+00:00 [ERROR ] {SmashPig-PopulateQueueFromDump::SPCID-0955108927} (2) stream_socket_client(): unable to connect to tcp://127.0.0.1:6379 (Connection refused) @ /srv/www/org/wikimedia/listeners/SmashPig/vendor/predis/predis/src/Connection/StreamConnection.php:127 [15:37:53] or maybe not [15:38:02] also I think now it's a different port for the e-p-c queue [15:38:02] oh I know why [15:38:12] yeah. it should be 6380 for that port [15:38:26] hmmm the config should handle that [15:38:41] yeah I was thinking that of the config [15:39:01] I guess I don't know why [15:39:03] lol [15:39:10] but also if the other one is on that other port i don't know why it'd give me connection refused [15:40:23] ah hmm we don't have that queue in the config on frpig [15:40:26] maybe the config hasn't been deployed to that box [15:40:34] I can try from civi, running the code in the vendor repo [15:40:40] +1 [15:40:43] it has been deployed jgleeson we just didn't include that code there [15:40:50] I mean that queue there [15:44:03] Hmmm on civi1001 it does get the right port [15:44:09] and the config is all fine [15:44:14] but still connection refused [15:44:15] 2021-05-07T15:42:48+00:00 [ERROR ] {SmashPig-PopulateQueueFromDump::SPCID-0170945433} (2) stream_socket_client(): unable to connect to tcp://frqueue1003.frack.eqiad.wmnet:6380 (Connection timed out) @ /srv/www/org/wikimedia/listeners/SmashPig/vendor/predis/predis/src/Connection/StreamConnection.php:127 [15:46:45] yeah looks like frpig1001 config doesn't have the email-prefs queue config deployment to it [15:48:16] dwisehaupt: Jeff_Green where's the repo which the puppet job configs again? [15:49:21] it's frpm1001:/var/lib/git/puppet.git [15:49:35] gotcha [15:49:46] fundraising-tech-ops: Fundraising access request for Julia Brungs - https://phabricator.wikimedia.org/T281111 (MeganHernandez_WMF) @Dwisehaupt I Just started a thread with you & Lisa. Thanks! [15:49:57] AndyRussG: that is strange [15:50:14] AndyRussG: you could try connecting to the redis-cli on live [15:50:24] and rpushing a message to the queue [15:50:48] oh wait [15:50:49] oh. the frpigs don't have firewall access to that redis instance. that wasn't in the list i had. [15:50:56] ^^ was thinking that [15:51:03] it's a separate instance isn't it [15:51:19] hmm civi1001 should do though dwisehaupt [15:51:25] AndyRussG: that also looks like the wrong IP address [15:51:33] oh that second one is better [15:51:59] nope. i only had from frdata to frqueue. i think that was an oversight. [15:52:44] dwisehaupt: the frpigs don't need it [15:52:45] ahhh [15:52:48] i'm in the iptables and pfw area right now, so which roles need access. frdata, frqueue, civi ? [15:52:50] civi1001 will need it [15:52:53] to read the messages [15:52:59] yah, civi + frdata [15:53:01] jgleeson: it's on frpm [15:53:12] got it AndyRussG thanks [15:53:17] ok. i'll get to updating those now. [15:53:53] funnily, i have held off updating the rig map until this all settled. in hindsight it may have been better to map it all out so all the interconnects were clear. [15:57:00] fr-tech also run-job for queue consumer fails on civi1001 [15:57:13] also connection timed out, then another error [15:57:41] AndyRussG: to be expected if the connectivity wasn't allowed [15:57:45] AndyRussG: looks like civi1001 can't connect to frqueue1003 yet [15:57:50] sounds like dwisehaupt is opening it up [15:58:05] https://paste.toolforge.org/view/b1d3ff83 [15:58:12] i'll be able to get the iptables done soon, and then will get the pfw ticket queued. [15:58:17] hmmm ok [15:58:24] dwisehaupt: thanks much! [15:58:32] isn't that a problem for like other stuff too? [15:58:40] no cause the others don't use that redis instance [15:58:52] just EPC's queue [16:02:31] AndyRussG: standup [16:03:20] talking about your readwrite fix [16:24:49] wow, there was a broadway version of One Flew Over the Cuckoo's Nest, with Kirk Douglas as the character Nicholson played in the movie [16:25:11] https://en.wikipedia.org/wiki/One_Flew_Over_the_Cuckoo%27s_Nest_(play) [16:25:51] like 12 years before the movie [16:34:25] ejegg: hmmmm [16:34:48] ok, sounds like I can do the meeting in 2.5 hrs from now [16:34:58] will put up an invite [16:35:10] ejegg: that works great for me yeah :) [16:35:12] thx! [16:35:30] (heheh I was just about to pester u about it :) ) [16:40:17] oh. the civi servers already have access to 6380 in the pfw. [16:40:23] i just missed the iptables bit. [16:40:34] we should be able to do the queue test now. [16:54:13] dwisehaupt: ah fantastic thanks, trying! [16:55:54] dwisehaupt: ok I was able to push a message! [16:55:56] andyrussg@civi1001:~$ php /srv/www/org/wikimedia/listeners/SmashPig/Maintenance/PopulateQueueFromDump.php --queue email-preferences test_e-p-c_msg.json [16:55:58] \n2021-05-07T16:55:31+00:00 [INFO ] {SmashPig-PopulateQueueFromDump::SPCID-0949031995} Imported 1 messages from test_e-p-c_msg.json in 0 seconds. [16:59:39] dwisehaupt fr-tech and it looks like I was able to consume the message to! [17:00:01] dwisehaupt I did something wrong on logging in to check the db to see if it updated, though, and now frbast doesn't like me it seems [17:01:16] fr-tech here's the queue consumption job command-line output: https://paste.toolforge.org/view/3f9224c9 [17:28:09] i've gotta run the dog to the vet. will have my phone with me. [17:33:13] fr-tech confirmed queue consumer works on prod, db correctly updated!! [17:37:49] great news AndyRussG [17:41:25] yeee :) [17:52:42] nice AndyRussG [17:52:52] I managed to recreate the SQLite issue also [17:53:01] looks like my sqlite db file was already rw [17:53:11] but setting it to read-only triggered the noisy error [17:53:27] Error 8: attempt to write a readonly database [17:53:28] Function: Wikimedia\Rdbms\Database::beginIfImplied (MediaWiki\Block\DatabaseBlock::newLoad) [17:53:30] Query: BEGIN IMMEDIATE [17:53:50] i guess we can run the rw sqlite for now though until we get the new mariadb instance fr-tech? [17:54:11] would this take us technically into a "deployed" state? [17:54:59] where does the sqlite file live on the email prefs box? dwisehaupt can you set that to rw? [17:55:26] ah nvm looks like Jeff_Green has mysql ready [17:58:37] Fundraising-Backlog: Add dynamic content to some TY emails, to target by country - https://phabricator.wikimedia.org/T282266 (TSkaff) [19:23:23] Fundraising-Backlog: Japan banners default to One-time with Adyen, monthly with Ingenico - https://phabricator.wikimedia.org/T282274 (HNordeenWMF) [19:52:02] Fundraising-Backlog: Japan banners default to One-time with Adyen, monthly with Ingenico - https://phabricator.wikimedia.org/T282274 (DStrine) @EMartin as FYI Both adyen and ingenico are available but this is all part of the banner logic. So ya'll can set it up however you want. [19:52:32] Fundraising-Backlog: Add dynamic content to some TY emails, to target by country - https://phabricator.wikimedia.org/T282266 (DStrine) a:DStrine→None [20:17:31] (Abandoned) Umherirrender: build: Updating dependencies [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686741 (owner: Libraryupgrader) [21:19:23] fundraising-tech-ops, SRE, fr-email-preference-center, netops, WMF-NDA: Deploy pfw policy1620422079 for T268501 and T281320 - https://phabricator.wikimedia.org/T282286 (Dwisehaupt) [21:42:40] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) Open→Resolved a:Dwisehaupt [21:42:43] Fundraising Sprint Princess Mongodb, Fundraising-Backlog, Fr-planning-ahead, Epic: Epic: Create Preference Center for donors to manage email subscription preferences - https://phabricator.wikimedia.org/T125272 (Dwisehaupt) [21:47:06] fundraising-tech-ops: Fundraising access request for Julia Brungs - https://phabricator.wikimedia.org/T281111 (Dwisehaupt) ` Date: Fri, 7 May 2021 08:53:34 From: Lisa Gruwell To: Megan Hernandez Cc: Dallas Wisehaupt Subject: Re: Please approve Julia Brungs superset access ------------------------------------... [21:47:38] fundraising-tech-ops: Fundraising access request for Julia Brungs - https://phabricator.wikimedia.org/T281111 (Dwisehaupt) [22:26:34] (PS1) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429) [22:27:49] Fundraising-Backlog, fundraising-tech-ops, fr-email-preference-center: Set up mysql replication for donorwiki/frdata hosts - https://phabricator.wikimedia.org/T282289 (Dwisehaupt) [22:32:28] (PS2) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429) [22:35:21] (CR) jerkins-bot: [V: -1] Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429) (owner: Ejegg) [22:41:03] (PS3) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429) [22:54:49] (PS1) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686793 (https://phabricator.wikimedia.org/T281441) [22:56:58] (CR) jerkins-bot: [V: -1] Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686793 (https://phabricator.wikimedia.org/T281441) (owner: Ejegg) [23:11:00] Fundraising-Backlog: Ingenico new API key as of Jul 8, 2021 - https://phabricator.wikimedia.org/T282294 (EMartin) [23:14:49] Fundraising-Backlog: Japan banners default to One-time with Adyen, monthly with Ingenico - https://phabricator.wikimedia.org/T282274 (EMartin) Just to confirm , 'available' means Monthly Convert is available on Adyen? [23:56:41] (PS2) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683770 (https://phabricator.wikimedia.org/T281441) [23:59:12] (PS3) Ejegg: Add Checkout scripts to Adyen gateway [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683770 (https://phabricator.wikimedia.org/T281441) [23:59:14] (PS1) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/686808 (https://phabricator.wikimedia.org/T281429) [23:59:49] (Abandoned) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429) (owner: Ejegg) [23:59:59] (Abandoned) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686793 (https://phabricator.wikimedia.org/T281441) (owner: Ejegg)