[01:06:30] <wikibugs>	 (CR) Ejegg: [C: +2] "> Patch Set 2:" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/683114 (owner: Eileen)
[01:08:30] <wikibugs>	 (Merged) jenkins-bot: Copy differences as commented out [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/683114 (owner: Eileen)
[01:34:05] <AndyRussG>	 ejegg: so I shouldn't try debugging into the sqlite  code path like u did to see if there's an easy code-y way to fix the error?
[01:34:37] <ejegg>	 AndyRussG: If you look here: https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/db/MWLBFactory.php$203
[01:35:14] <ejegg>	 it seems like we'd need to hack core to allow a POST to not use IMMEDIATE
[01:35:25] <ejegg>	 and I'm pretty sure we don't want to submit the form with 'GET'
[01:41:57] <ejegg>	 if you can find otherwise, it would be great!
[01:44:21] <AndyRussG>	 ejegg: right I saw that, thanks
[01:44:58] <AndyRussG>	 ejegg: K I'll dig a bit but assume not too much assuming the mariadb option is probably ok to then
[01:45:20] <AndyRussG>	 ejegg: can you think of anything else I can look at now for e-p-c deploy?
[01:45:28] <ejegg>	 So since that query is in the context of looking up DB blocks, I had hopes of getting around it by setting BlockDisablesLogin to false: https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/Permissions/PermissionManager.php$713
[01:47:45] <ejegg>	 but then I hit another place that uses the block logic: https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/MediaWiki.php$779
[01:49:05] <ejegg>	 hmm, maybe we could get couldBePublicCached to return true somehow?
[01:54:05] <ejegg>	 AndyRussG: If you want to test the consumer, you can use the PopulateQueueFromDump smashpig maintenance script to inject a message into the queue
[01:56:31] <ejegg>	 oh, but $output->haveCacheVaryCookies() is always true right there, so yah, I think there's no real way around that
[02:04:04] <AndyRussG>	 ejegg: ah okok yeah the queue consumer sounds like a fun test!
[02:57:13] <wikibugs>	 Fundraising-Backlog, FR-Docker: Docker dev setup: follow-on Civiproxy setup - https://phabricator.wikimedia.org/T281651 (AndyRussG)
[02:58:03] <wikibugs>	 Fundraising-Backlog, FR-Docker, Patch-For-Review: Docker dev setup: fix fredge database setup - https://phabricator.wikimedia.org/T281609 (AndyRussG) I think this may be all done?
[07:30:36] <icinga-wm>	 PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1213
[07:35:36] <icinga-wm>	 RECOVERY - check_mysql on frdev1001 is OK: Uptime: 3591547 Threads: 17 Questions: 95144885 Slow queries: 2946910 Opens: 759394455 Flush tables: 1 Open tables: 236 Queries per second avg: 26.491 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0
[13:52:36] <icinga-wm>	 PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 5 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/mysql/cacert.pem],File[/etc/mysql/server-cert.pem],File[/etc/mysql/server-key.pem]
[13:57:36] <icinga-wm>	 RECOVERY - check_puppetrun on frdata2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[14:55:48] <jgleeson>	 howdy fr-tech!
[14:56:59] <AndyRussG>	 jgleeson fr-tech hi!
[14:57:55] <jgleeson>	 hi hi. AndyRussG do you know if anyones managed to replicate the sqlite-readonly-POST bug locally, that ejegg|away put in his EOD email?
[14:58:13] <AndyRussG>	 jgleeson: just replying, yes I did
[14:58:17] <AndyRussG>	 or I think it was the same bug
[14:58:27] <AndyRussG>	 however it works fine if you just make the sqlite file read-write!
[14:58:39] <jgleeson>	 oh nice
[14:59:02] <jgleeson>	 so we might not need another mariadb instance afteall
[14:59:08] <jgleeson>	 after all *
[14:59:21] <AndyRussG>	 well it depends I guess on how hard we want to lock down the db
[15:00:06] <jgleeson>	 I don't think we're putting anything sensitive in it
[15:02:23] <AndyRussG>	 yeah
[15:02:41] <AndyRussG>	 I think maybe the idea was to have something parallel to the payments setup, where I think somehow the dbs are in read-only mode?
[15:07:56] <AndyRussG>	 jgleeson: there is already $wgReadOnly set to true in LocalSettings btw
[15:11:06] <jgleeson>	 ah cool
[15:11:58] <jgleeson>	 Roxy's grandparents just left so I've been a little distracted. My road has been closed at the top end so the cars were crawling along. Rush hour fun
[15:12:52] <AndyRussG>	 jgleeson oh hey grandparents' visit is important!
[15:13:02] <AndyRussG>	 they're all vaccinated I imagine?
[15:13:06] <dwisehaupt>	 morn
[15:13:17] <AndyRussG>	 dwisehaupt hi!!
[15:13:33] <AndyRussG>	 dwisehaupt just a thought here, how do you feel about temporarily making the sqlite file writable?
[15:13:44] <AndyRussG>	 at least as a temporary measure to get this out the door
[15:14:00] <AndyRussG>	 instead of spinning up the whole mariadb thing?
[15:15:07] <AndyRussG>	 (sorry to hit you with this first thing in the morning hehe)
[15:15:29] <dwisehaupt>	 i'm actually ok with that in the short term. not sure what the downside to that would be. other than possible stray edits.
[15:15:52] <dwisehaupt>	 i'm just getting to email but it looks like jeff may have rolled the changes to install mariadb on that role.
[15:15:52] <AndyRussG>	 dwisehaupt ok that sounds great! yeah we'd keep $wgReadOnly set to true
[15:16:02] <AndyRussG>	 ahhh nocie!
[15:16:04] <AndyRussG>	 noice*
[15:16:09] <jgleeson>	 yep AndyRussG vaccinated
[15:16:18] <AndyRussG>	 jgleeson: fantastic :)
[15:16:22] <jgleeson>	 they've had two
[15:16:28] <jgleeson>	 I haven't had mine yet
[15:16:31] <AndyRussG>	 ah great
[15:16:36] <AndyRussG>	 yeah me neither
[15:16:45] <dwisehaupt>	 i'd like to at least check in a see his thoughts and what he completed since i was last on.
[15:17:35] <AndyRussG>	 I signed up on a web site and they're supposed to call me sometime for the actual appointment, but I'm actually at this point one month younger than the cutoff age
[15:17:43] <AndyRussG>	 dwisehaupt cool thanks much! :)
[15:20:01] <jgleeson>	 AndyRussG: maybe you'll get a call next month!
[15:20:11] <jgleeson>	 I think I've gotta wait about a month also
[15:20:34] <jgleeson>	 Roxy has her second booked in for sunday. She has asthma tho
[15:20:43] <AndyRussG>	 ah glad she can get hers!
[15:25:16] <jgleeson>	 AndyRussG: I'm curious why we didn't run into that same issue locally. Is our setup locally different in some way?
[15:25:26] <jgleeson>	 the sqlite readonly POST thing
[15:25:47] <AndyRussG>	 jgleeson: on fundraising-dev e-p-c uses mariadb currently
[15:26:10] <AndyRussG>	 I had to switch to sqlite and run the mw setup web wizard to get it to work actually
[15:26:25] <AndyRussG>	 btw where to we have smashpig ipn listeners installed on the cluster?
[15:26:33] <jgleeson>	 oh I see are we using mysql locally?
[15:26:47] <jgleeson>	 for EPC
[15:26:48] <AndyRussG>	 yep!
[15:26:51] <jgleeson>	 gotcha
[15:27:05] <jgleeson>	 hmm
[15:27:07] <jgleeson>	 re:smashpig
[15:27:18] <jgleeson>	 the lie on frpig* I think
[15:27:23] <jgleeson>	 live and lie
[15:27:58] <AndyRussG>	 ahh right makes sense :)
[15:35:43] <AndyRussG>	 jgleeson dwisehaupt so I'm about to test out the queue consumer by injecting a fake message into the queue
[15:35:58] <jgleeson>	 cool
[15:36:03] <AndyRussG>	 frpig1001:~$ php /srv/www/org/wikimedia/listeners/SmashPig/Maintenance/PopulateQueueFromDump.php --queue email-preferences --file test_e-p-c_msg.json
[15:36:11] <jgleeson>	 just attempting the sqlite setup locally also
[15:36:20] <AndyRussG>	 ah cool
[15:37:24] <AndyRussG>	 ah updated command attempt
[15:37:26] <AndyRussG>	 php /srv/www/org/wikimedia/listeners/SmashPig/Maintenance/PopulateQueueFromDump.php --queue email-preferences test_e-p-c_msg.json
[15:37:31] <AndyRussG>	 that ran but spat out an error
[15:37:45] <jgleeson>	 looks good
[15:37:47] <AndyRussG>	 2021-05-07T15:36:46+00:00 [ERROR  ] {SmashPig-PopulateQueueFromDump::SPCID-0955108927} (2) stream_socket_client(): unable to connect to tcp://127.0.0.1:6379 (Connection refused) @ /srv/www/org/wikimedia/listeners/SmashPig/vendor/predis/predis/src/Connection/StreamConnection.php:127
[15:37:53] <jgleeson>	 or maybe not
[15:38:02] <AndyRussG>	 also I think now it's a different port for the e-p-c queue
[15:38:02] <jgleeson>	 oh I know why
[15:38:12] <dwisehaupt>	 yeah. it should be 6380 for that port
[15:38:26] <jgleeson>	 hmmm the config should handle that
[15:38:41] <AndyRussG>	 yeah I was thinking that of the config
[15:39:01] <jgleeson>	 I guess I don't know why
[15:39:03] <jgleeson>	 lol
[15:39:10] <AndyRussG>	 but also if the other one is on that other port i don't know why it'd give me connection refused
[15:40:23] <AndyRussG>	 ah hmm we don't have that queue in the config on frpig
[15:40:26] <jgleeson>	 maybe the config hasn't been deployed to that box
[15:40:34] <AndyRussG>	 I can try from civi, running the code in the vendor repo
[15:40:40] <jgleeson>	 +1
[15:40:43] <AndyRussG>	 it has been deployed jgleeson we just didn't include that code there
[15:40:50] <AndyRussG>	 I mean that queue there
[15:44:03] <AndyRussG>	 Hmmm on civi1001 it does get the right port
[15:44:09] <AndyRussG>	 and the config is all fine
[15:44:14] <AndyRussG>	 but still connection refused
[15:44:15] <AndyRussG>	 2021-05-07T15:42:48+00:00 [ERROR  ] {SmashPig-PopulateQueueFromDump::SPCID-0170945433} (2) stream_socket_client(): unable to connect to tcp://frqueue1003.frack.eqiad.wmnet:6380 (Connection timed out) @ /srv/www/org/wikimedia/listeners/SmashPig/vendor/predis/predis/src/Connection/StreamConnection.php:127
[15:46:45] <jgleeson>	 yeah looks like frpig1001 config doesn't have the email-prefs queue config deployment to it
[15:48:16] <jgleeson>	 dwisehaupt: Jeff_Green where's the repo which the puppet job configs again?
[15:49:21] <Jeff_Green>	 it's frpm1001:/var/lib/git/puppet.git
[15:49:35] <jgleeson>	 gotcha
[15:49:46] <wikibugs>	 fundraising-tech-ops: Fundraising access request for Julia Brungs - https://phabricator.wikimedia.org/T281111 (MeganHernandez_WMF) @Dwisehaupt  I Just started a thread with you & Lisa. Thanks!
[15:49:57] <jgleeson>	 AndyRussG: that is strange
[15:50:14] <jgleeson>	 AndyRussG: you could try connecting to the redis-cli on live
[15:50:24] <jgleeson>	 and rpushing a message to the queue
[15:50:48] <jgleeson>	 oh wait
[15:50:49] <dwisehaupt>	 oh. the frpigs don't have firewall access to that redis instance. that wasn't in the list i had.
[15:50:56] <jgleeson>	 ^^ was thinking that
[15:51:03] <jgleeson>	 it's a separate instance isn't it
[15:51:19] <jgleeson>	 hmm civi1001 should do though dwisehaupt
[15:51:25] <ejegg>	 AndyRussG: that also looks like the wrong IP address
[15:51:33] <ejegg>	 oh that second one is better
[15:51:59] <dwisehaupt>	 nope. i only had from frdata to frqueue. i think that was an oversight.
[15:52:44] <ejegg>	 dwisehaupt: the frpigs don't need it
[15:52:45] <jgleeson>	 ahhh
[15:52:48] <dwisehaupt>	 i'm in the iptables and pfw area right now, so which roles need access. frdata, frqueue, civi ?
[15:52:50] <jgleeson>	 civi1001 will need it
[15:52:53] <jgleeson>	 to read the messages
[15:52:59] <ejegg>	 yah, civi + frdata
[15:53:01] <AndyRussG>	 jgleeson: it's on frpm
[15:53:12] <jgleeson>	 got it AndyRussG thanks
[15:53:17] <dwisehaupt>	 ok. i'll get to updating those now.
[15:53:53] <dwisehaupt>	 funnily, i have held off updating the rig map until this all settled. in hindsight it may have been better to map it all out so all the interconnects were clear.
[15:57:00] <AndyRussG>	 fr-tech also run-job for queue consumer fails on civi1001
[15:57:13] <AndyRussG>	 also connection timed out, then another error
[15:57:41] <ejegg>	 AndyRussG: to be expected if the connectivity wasn't allowed
[15:57:45] <jgleeson>	 AndyRussG: looks like civi1001 can't connect to frqueue1003 yet
[15:57:50] <ejegg>	 sounds like dwisehaupt is opening it up
[15:58:05] <AndyRussG>	 https://paste.toolforge.org/view/b1d3ff83
[15:58:12] <dwisehaupt>	 i'll be able to get the iptables done soon, and then will get the pfw ticket queued.
[15:58:17] <AndyRussG>	 hmmm ok
[15:58:24] <AndyRussG>	 dwisehaupt: thanks much!
[15:58:32] <AndyRussG>	 isn't that a problem for like other stuff too?
[15:58:40] <jgleeson>	 no cause the others don't use that redis instance
[15:58:52] <jgleeson>	 just EPC's queue
[16:02:31] <jgleeson>	 AndyRussG: standup
[16:03:20] <jgleeson>	 talking about your readwrite fix
[16:24:49] <ejegg>	 wow, there was a broadway version of One Flew Over the Cuckoo's Nest, with Kirk Douglas as the character Nicholson played in the movie
[16:25:11] <ejegg>	 https://en.wikipedia.org/wiki/One_Flew_Over_the_Cuckoo%27s_Nest_(play)
[16:25:51] <ejegg>	 like 12 years before the movie
[16:34:25] <AndyRussG>	 ejegg: hmmmm
[16:34:48] <ejegg>	 ok, sounds like I can do the meeting in 2.5 hrs from now
[16:34:58] <ejegg>	 will put up an invite
[16:35:10] <AndyRussG>	 ejegg: that works great for me yeah :)
[16:35:12] <AndyRussG>	 thx!
[16:35:30] <AndyRussG>	 (heheh I was just about to pester u about it :) )
[16:40:17] <dwisehaupt>	 oh. the civi servers already have access to 6380 in the pfw.
[16:40:23] <dwisehaupt>	 i just missed the iptables bit.
[16:40:34] <dwisehaupt>	 we should be able to do the queue test now.
[16:54:13] <AndyRussG>	 dwisehaupt: ah fantastic thanks, trying!
[16:55:54] <AndyRussG>	 dwisehaupt: ok I was able to push a message!
[16:55:56] <AndyRussG>	 andyrussg@civi1001:~$ php /srv/www/org/wikimedia/listeners/SmashPig/Maintenance/PopulateQueueFromDump.php --queue email-preferences test_e-p-c_msg.json
[16:55:58] <AndyRussG>	 \n2021-05-07T16:55:31+00:00 [INFO   ] {SmashPig-PopulateQueueFromDump::SPCID-0949031995} Imported 1 messages from test_e-p-c_msg.json in 0 seconds.
[16:59:39] <AndyRussG>	 dwisehaupt fr-tech and it looks like I was able to consume the message to!
[17:00:01] <AndyRussG>	 dwisehaupt I did something wrong on logging in to check the db to see if it updated, though, and now frbast doesn't like me it seems
[17:01:16] <AndyRussG>	 fr-tech here's the queue consumption job command-line output: https://paste.toolforge.org/view/3f9224c9
[17:28:09] <dwisehaupt>	 i've gotta run the dog to the vet. will have my phone with me.
[17:33:13] <AndyRussG>	 fr-tech confirmed queue consumer works on prod, db correctly updated!!
[17:37:49] <ejegg>	 great news AndyRussG
[17:41:25] <AndyRussG>	 yeee :)
[17:52:42] <jgleeson>	 nice AndyRussG
[17:52:52] <jgleeson>	 I managed to recreate the SQLite issue also
[17:53:01] <jgleeson>	 looks like my sqlite db file was already rw
[17:53:11] <jgleeson>	 but setting it to read-only triggered the noisy error
[17:53:27] <jgleeson>	 Error 8: attempt to write a readonly database
[17:53:28] <jgleeson>	 Function: Wikimedia\Rdbms\Database::beginIfImplied (MediaWiki\Block\DatabaseBlock::newLoad)
[17:53:30] <jgleeson>	 Query: BEGIN IMMEDIATE
[17:53:50] <jgleeson>	 i guess we can run the rw sqlite for now though until we get the new mariadb instance fr-tech?
[17:54:11] <jgleeson>	 would this take us technically into a "deployed" state?
[17:54:59] <jgleeson>	 where does the sqlite file live on the email prefs box? dwisehaupt can you set that to rw?
[17:55:26] <jgleeson>	 ah nvm looks like Jeff_Green has mysql ready
[17:58:37] <wikibugs>	 Fundraising-Backlog: Add dynamic content to some TY emails, to target by country - https://phabricator.wikimedia.org/T282266 (TSkaff)
[19:23:23] <wikibugs>	 Fundraising-Backlog: Japan banners default to One-time with Adyen, monthly with Ingenico - https://phabricator.wikimedia.org/T282274 (HNordeenWMF)
[19:52:02] <wikibugs>	 Fundraising-Backlog: Japan banners default to One-time with Adyen, monthly with Ingenico - https://phabricator.wikimedia.org/T282274 (DStrine) @EMartin as FYI  Both adyen and ingenico are available but this is all part of the banner logic. So ya'll can set it up however you want.
[19:52:32] <wikibugs>	 Fundraising-Backlog: Add dynamic content to some TY emails, to target by country - https://phabricator.wikimedia.org/T282266 (DStrine) a:DStrine→None
[20:17:31] <wikibugs>	 (Abandoned) Umherirrender: build: Updating dependencies [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686741 (owner: Libraryupgrader)
[21:19:23] <wikibugs>	 fundraising-tech-ops, SRE, fr-email-preference-center, netops, WMF-NDA: Deploy pfw policy1620422079 for T268501 and T281320 - https://phabricator.wikimedia.org/T282286 (Dwisehaupt)
[21:42:40] <wikibugs>	 Fundraising Sprint Princess Mongodb, Fundraising-Backlog, fundraising-tech-ops, Fr-planning-ahead, and 2 others: Set up and configure donor_prefs redis instance on FR queue role - https://phabricator.wikimedia.org/T281955 (Dwisehaupt) Open→Resolved a:Dwisehaupt
[21:42:43] <wikibugs>	 Fundraising Sprint Princess Mongodb, Fundraising-Backlog, Fr-planning-ahead, Epic: Epic: Create Preference Center for donors to manage email subscription preferences - https://phabricator.wikimedia.org/T125272 (Dwisehaupt)
[21:47:06] <wikibugs>	 fundraising-tech-ops: Fundraising access request for Julia Brungs - https://phabricator.wikimedia.org/T281111 (Dwisehaupt) ` Date: Fri, 7 May 2021 08:53:34 From: Lisa Gruwell To: Megan Hernandez Cc: Dallas Wisehaupt Subject: Re: Please approve Julia Brungs superset access ------------------------------------...
[21:47:38] <wikibugs>	 fundraising-tech-ops: Fundraising access request for Julia Brungs - https://phabricator.wikimedia.org/T281111 (Dwisehaupt)
[22:26:34] <wikibugs>	 (PS1) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429)
[22:27:49] <wikibugs>	 Fundraising-Backlog, fundraising-tech-ops, fr-email-preference-center: Set up mysql replication for donorwiki/frdata hosts - https://phabricator.wikimedia.org/T282289 (Dwisehaupt)
[22:32:28] <wikibugs>	 (PS2) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429)
[22:35:21] <wikibugs>	 (CR) jerkins-bot: [V: -1] Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429) (owner: Ejegg)
[22:41:03] <wikibugs>	 (PS3) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429)
[22:54:49] <wikibugs>	 (PS1) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686793 (https://phabricator.wikimedia.org/T281441)
[22:56:58] <wikibugs>	 (CR) jerkins-bot: [V: -1] Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686793 (https://phabricator.wikimedia.org/T281441) (owner: Ejegg)
[23:11:00] <wikibugs>	 Fundraising-Backlog: Ingenico new API key as of Jul 8, 2021 - https://phabricator.wikimedia.org/T282294 (EMartin)
[23:14:49] <wikibugs>	 Fundraising-Backlog: Japan banners default to One-time with Adyen, monthly with Ingenico - https://phabricator.wikimedia.org/T282274 (EMartin) Just to confirm , 'available' means Monthly Convert is available on Adyen?
[23:56:41] <wikibugs>	 (PS2) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683770 (https://phabricator.wikimedia.org/T281441)
[23:59:12] <wikibugs>	 (PS3) Ejegg: Add Checkout scripts to Adyen gateway [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/683770 (https://phabricator.wikimedia.org/T281441)
[23:59:14] <wikibugs>	 (PS1) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] (adyen) - https://gerrit.wikimedia.org/r/686808 (https://phabricator.wikimedia.org/T281429)
[23:59:49] <wikibugs>	 (Abandoned) Ejegg: Skeleton files for Adyen Checkout integration [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686789 (https://phabricator.wikimedia.org/T281429) (owner: Ejegg)
[23:59:59] <wikibugs>	 (Abandoned) Ejegg: Add Drop-In web scripts to Adyen gateway [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/686793 (https://phabricator.wikimedia.org/T281441) (owner: Ejegg)