[00:01:04] <wikibugs>	 (03Merged) 10jenkins-bot: i18n: Remove 4 messages, now moved to DonationInterface [extensions/FundraisingEmailUnsubscribe] - 10https://gerrit.wikimedia.org/r/1097498 (https://phabricator.wikimedia.org/T376514) (owner: 10Ejegg)
[00:09:00] <wikibugs>	 (03PS8) 10Ejegg: Add missing apihelp param messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117622
[00:11:33] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add missing apihelp param messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117622 (owner: 10Ejegg)
[00:17:37] <wikibugs>	 (03PS9) 10Ejegg: Add missing apihelp param messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117622
[00:20:29] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add missing apihelp param messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117622 (owner: 10Ejegg)
[00:22:20] <ejegg>	 wohoo, that's all the missing params!
[00:22:33] <ejegg>	 lemme see if i really can 'git push'
[00:30:36] <wikibugs>	 (03PS10) 10Ejegg: Add missing apihelp param messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117622
[00:30:45] <ejegg>	 yep, it let me
[00:30:59] <cstone>	 woo ejegg  im back now to look at that other test
[00:31:24] <ejegg>	 testAdyenFormLoad ?
[00:31:30] <cstone>	 yeah
[00:31:36] <ejegg>	 cool, I'll peek at testResultSwitcherRepeat
[00:31:57] <ejegg>	 I feel like we're doing horrible things to fake MW page loads in those tests
[00:32:06] <ejegg>	 and they totally are fragile across versions
[00:32:18] <ejegg>	 and maybe we can just disable the tests if it's not a quick fix
[00:32:26] <ejegg>	 s/disable/delete
[00:46:51] <cstone>	 hmm yeah i cant even get this one to run locally
[00:50:14] <cstone>	 ok i was the problem
[00:52:30] <cstone>	 why is it just canada hmmm
[00:56:50] <cstone>	 is it just because there are two
[00:56:56] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for sinaba - https://phabricator.wikimedia.org/T385759 (10Dwisehaupt) 03NEW
[00:56:57] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for iackerman - https://phabricator.wikimedia.org/T385760 (10Dwisehaupt) 03NEW
[00:56:58] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for mbeat - https://phabricator.wikimedia.org/T385761 (10Dwisehaupt) 03NEW
[00:58:49] <wikibugs>	 (03PS1) 10Cstone: Remove duplicate line [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117642 (https://phabricator.wikimedia.org/T384656)
[01:01:06] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Remove duplicate line [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117642 (https://phabricator.wikimedia.org/T384656) (owner: 10Cstone)
[01:35:28] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Remove duplicate line [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117642 (https://phabricator.wikimedia.org/T384656) (owner: 10Cstone)
[01:35:38] <ejegg|food>	 wanna do a 'git push' for that one cstone?
[01:37:54] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Remove duplicate line [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117642 (https://phabricator.wikimedia.org/T384656) (owner: 10Cstone)
[02:10:26] <ejegg>	 ahh, looks like the ObjectCache::getLocalClusterInstance() returns a different object inside the verifyFormOutput fn than outside
[02:12:32] <wikibugs>	 (03PS2) 10Cstone: Remove duplicate line [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117642 (https://phabricator.wikimedia.org/T384656)
[02:13:17] <ejegg>	 OK, I force-pushed that one cstone. Nice catch!
[02:34:56] <wikibugs>	 (03PS1) 10Ejegg: Skip failing test [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117647
[02:37:51] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Skip failing test [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117647 (owner: 10Ejegg)
[02:38:33] <wikibugs>	 (03PS7) 10Ejegg: Bump up minimum PHP and SmashPig versions [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1115152
[02:38:55] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Bump up minimum PHP and SmashPig versions [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1115152 (owner: 10Ejegg)
[02:40:26] <wikibugs>	 (03Merged) 10jenkins-bot: Skip failing test [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1117647 (owner: 10Ejegg)
[02:41:46] <wikibugs>	 (03Merged) 10jenkins-bot: Bump up minimum PHP and SmashPig versions [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1115152 (owner: 10Ejegg)
[02:41:59] <wikibugs>	 (03PS2) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1117569
[02:42:25] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1117569 (owner: 10Ejegg)
[02:43:37] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1117569 (owner: 10Ejegg)
[02:45:40] <wikibugs>	 (03PS1) 10Ejegg: Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648
[02:45:43] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[02:46:58] <ejegg>	 dwisehaupt: do you want to try the MW 1.43 upgrade at this late hour?
[02:50:01] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[02:51:27] <ejegg>	 well that's a flaky fail
[02:51:35] <ejegg>	 21:49:53 1) LanguageIntegrationTest::testGetLanguageNames_withHook with data set "Hooks don't make a language supported" ('', 'es-419', 'en', 'mwfile')
[02:54:27] <ejegg>	 ok, it fails locally too
[02:54:36] <ejegg>	 but does it fail because of something on our branch?
[03:01:37] <ejegg>	 ohhh right we do try to make es-419 supported with this: https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/DonationInterface.class.php$107
[03:02:10] <ejegg>	 so... is that broken in 1.43
[03:02:22] <ejegg>	 ?
[03:03:53] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] "recheck" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:07:45] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:11:20] <ejegg>	 it's not broken - I'm seeing the right 'problemsdonating' link text at https://localhost:9001/index.php?title=Special:DlocalGateway&appeal=JimmyQuote&payment_method=cc&recurring=0&uselang=es-419&language=es-419&currency=COP&amount=6000&country=CO&first_name=Jimmy&last_name=Wales&fiscal_number=9.999.999.999&email=jwales%40example.com&debug=true
[03:13:26] <ejegg>	 ugh i'mma neuter that test too
[03:15:19] <wikibugs>	 (03PS2) 10Ejegg: Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648
[03:16:54] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:19:08] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:21:34] <wikibugs>	 (03PS3) 10Ejegg: Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648
[03:21:44] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:31:28] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:34:31] <ejegg>	 whaaat now MediaWiki\Tests\Maintenance\WikiBirthdayTest failing???
[03:35:35] <ejegg>	 grr
[03:40:24] <wikibugs>	 (03PS1) 10Ejegg: Only call setContributionTrackingID with int or NULL [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1117652
[03:45:08] <dwisehaupt>	 ejegg: howdy. sorry, was getting food.
[03:45:17] <cstone>	 ejegg: im also back was also getting food
[03:45:32] <dwisehaupt>	 i'm around for another 1.5-2 hours, so if you want to, i think we can.
[03:45:54] <ejegg>	 woohoo
[03:46:20] <ejegg>	 so I remember there's alway a little hiccup with the git checkout when we change the deploy branch
[03:46:28] <ejegg>	 do you remember the way around that?
[03:46:54] <dwisehaupt>	 i think so. it's been quite a while since we've done this.
[03:47:21] <wikibugs>	 (03CR) 10Ejegg: [V:03+2 C:03+2] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:48:14] <ejegg>	 oh hah, i have V+2 privs but not the privs to remove the V-1 from that flakey birthday test
[03:48:20] <ejegg>	 need to make a trivial change
[03:48:39] <wikibugs>	 (03PS4) 10Ejegg: Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648
[03:48:42] <wikibugs>	 (03CR) 10Ejegg: [V:03+2 C:03+2] Update fundraising extensions [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117648 (owner: 10Ejegg)
[03:49:25] <dwisehaupt>	 we will be updating this for payments-wiki, payments-wiki-staging, and donorwiki, yes?
[03:49:34] <dwisehaupt>	 or just a subset?
[03:50:31] <ejegg>	 dwisehaupt: yep, all of 'em
[03:52:16] <dwisehaupt>	 ok. that's all in one file. iirc, we _may_ have to change the branch by hand on the frpm host, but that will be easy to test with payments-wiki-staging first.
[03:52:25] <ejegg>	 ok, great
[03:53:15] <dwisehaupt>	 yeah, last time we did this was jan 2023. so been a while. :)
[03:53:47] <ejegg>	 so i'm running fundraising_code_update -p payments-wiki-staging
[03:54:02] <ejegg>	 ahh project is locked
[03:54:17] <ejegg>	 adding =head
[03:54:38] <dwisehaupt>	 yeah, i'll have to update the branch in frdeploy.yaml also.
[03:55:23] <ejegg>	 ah yeah, it's still pulling REL1_39 there
[03:57:13] <dwisehaupt>	 ok. try it again. should pull REL1_43 now
[03:57:28] <dwisehaupt>	 but like i said, we may need to change the branch by hand. i can't remember 100%
[03:58:47] <dwisehaupt>	 should probably document this after we do it so we'll remember in 2 years when we do it again. :)
[03:59:13] <ejegg>	 looks like it pulled 1_43!
[03:59:24] <ejegg>	 ok, I'll sync it out
[03:59:38] <dwisehaupt>	 sweet.
[04:01:05] <ejegg>	 !log upgraded payments-wiki-staging from 7eeb643bb146edfc7cc to 4cdd67b99e399f5cc9
[04:01:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:02:29] <ejegg>	 so we get to staging by messing with the hosts file, right?
[04:02:40] <dwisehaupt>	 should be able to hit payments-staging.wikimedia.org but it didn't like my client cert on first load.
[04:02:49] <ejegg>	 oh right
[04:03:38] <dwisehaupt>	 ha. i updated it but forgot to install it in the browser and it just expired. silly me. :)
[04:03:52] <ejegg>	 heh
[04:04:12] <ejegg>	 ok, I'm getting the expected error from a MW update with no database update
[04:04:44] <ejegg>	 so can we unlock and run maintenance/update.php ?
[04:07:14] <dwisehaupt>	 ok yeah. expected that. 
[04:14:12] <dwisehaupt>	 ok. so since it's a staging one, it is a little different. don't think we planned for this when we broke it out.
[04:15:09] <dwisehaupt>	 ejegg: ok. is that update.php something i will be running or you will?
[04:17:51] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Only call setContributionTrackingID with int or NULL [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1117652 (owner: 10Ejegg)
[04:18:28] <ejegg>	 dwisehaupt: I think you need to
[04:18:44] <ejegg>	 It has to be run on the payments box with the db unlocked
[04:18:56] <dwisehaupt>	 yeah. seeing the UPGRADE doc i see what i need.
[04:19:00] <ejegg>	 oh, do I have ssh access to that box?
[04:19:00] <dwisehaupt>	 ok. doing that.
[04:19:06] <dwisehaupt>	 i don't think you do.
[04:19:15] <dwisehaupt>	 oh. actually, payments1008, maybe
[04:19:22] <dwisehaupt>	 but i'll run it.
[04:19:55] <ejegg>	 cool cool
[04:20:35] <ejegg>	 so at the payments-staging URL i expect some 'bad JS origin' issues with a few payments methods, but we'll at least be able to see some pages render
[04:24:17] <dwisehaupt>	 ok. i think i have it. needed to hand twiddle the db in addition to the php readonly settings.
[04:24:38] <ejegg>	 yep, renders the page now
[04:24:43] <dwisehaupt>	 unique to the staging server bits.
[04:25:24] <ejegg>	 right, adyen secure fields don't want to render on that URL
[04:25:48] <ejegg>	 (or we don't have all credentials there)
[04:26:16] <dwisehaupt>	 oh. yeah. possibly. depends on the state of localsettings for staging.
[04:26:23] <dwisehaupt>	 not sure when that was last updated.
[04:26:47] <ejegg>	 well, I'm satisfied that we can go ahead with the non-staging one
[04:27:09] <ejegg>	 we have a backup of the db?
[04:27:55] <dwisehaupt>	 let me check the latest.
[04:28:35] <dwisehaupt>	 yes. as of 2 hours ago.
[04:28:41] <ejegg>	 great!
[04:28:53] <ejegg>	 want to update the payments-wiki branch ?
[04:31:28] <dwisehaupt>	 ok. all set when you are.
[04:36:47] <ejegg>	 ah, it's having the problem
[04:36:48] <ejegg>	 git_checkout(payments-wiki) update failed!
[04:36:51] <ejegg>	 you could try forcing a git fetch (-f) or rerun in debug mode (-d) get more info
[04:37:00] <ejegg>	 trying with -f
[04:37:25] <ejegg>	 still failed
[04:37:34] <dwisehaupt>	 ok. let me check.
[04:37:59] <ejegg>	 exec_git stderr: error: Your local changes to the following files would be overwritten by checkout:
[04:38:03] <ejegg>	 maintenance/update.php
[04:38:18] <ejegg>	 modechange?
[04:38:39] <ejegg>	 git diff doesn't show anything
[04:39:41] <ejegg>	 oh but there were other errors above that one in the -d run
[04:39:55] <ejegg>	 /usr/bin/git -C /srv/www/org/wikimedia/payments/vendor show 14271e93976f61912a0e0dd6eb30b624efa362bd:.gitmodules
[04:40:01] <ejegg>	 fatal: path '.gitmodules' does not exist in '14271e93976f61912a0e0dd6eb30b624efa362bd'
[04:42:15] <dwisehaupt>	 i'm going to reset to git to 1_39 and then try changing branches.
[04:43:35] <ejegg>	 ohh i was doing git diff in the staging dir
[04:43:52] <wikibugs>	 (03Merged) 10jenkins-bot: Only call setContributionTrackingID with int or NULL [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1117652 (owner: 10Ejegg)
[04:44:13] <dwisehaupt>	 mwdeploy@frpm1002:/srv/www/org/wikimedia/payments$ git status
[04:44:13] <dwisehaupt>	 On branch fundraising/REL1_43
[04:44:13] <dwisehaupt>	 Your branch is up to date with 'origin/fundraising/REL1_43'.
[04:44:28] <ejegg>	 oh nice
[04:44:46] <dwisehaupt>	 try the depoly bits now.
[04:45:20] <ejegg>	 oh hmm, submodules are not updated tho
[04:45:50] <dwisehaupt>	 oh. yeah. i didn't do that. you know the git command for that? i don't usually deal with submodules
[04:46:37] <ejegg>	 git submodule update
[04:47:17] <dwisehaupt>	 that went really fast. 
[04:47:29] <dwisehaupt>	 check if it dtrt
[04:47:45] <ejegg>	 vendor looks good!
[04:48:27] <ejegg>	 hmm, DonationInterface is missing some stuff
[04:48:30] <dwisehaupt>	 it was deceptively fast.
[04:48:37] <ejegg>	 like, lots of stuff
[04:49:10] <dwisehaupt>	 ok. let me try another fetch and pull
[04:49:55] <dwisehaupt>	 check again.
[04:50:21] <ejegg>	 weird, git status in DonationInterface shows lots of stuff deleted
[04:50:37] <ejegg>	 maybe a git reset --hard in each submodule directory?
[04:51:27] <dwisehaupt>	 sure.
[04:53:23] <dwisehaupt>	 ok. check now.
[04:55:06] <ejegg>	 ok, looks better
[04:55:10] <ejegg>	 gonna rsync
[04:55:55] <ejegg>	 !log payments-wiki upgraded from MW 1.39 to MW 1.43 (needs db update)
[04:55:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:56:24] <dwisehaupt>	 cool. just let me know when you are ready for the db update.
[04:56:55] <ejegg>	 ready for the update!
[04:57:13] <dwisehaupt>	 done.
[04:57:19] <dwisehaupt>	 Done in 4.0 s.
[04:59:07] <dwisehaupt>	 i'll need to clean up the db on payments1008 since it essentially got the update run twice, but that's easy enough.
[05:02:26] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on payments1008 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt known - need cleanup after upgrade replication https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[05:03:22] <ejegg>	 looking good so far
[05:06:21] <cstone>	 ejegg: is the part you did for the upgrade documented anywhere?
[05:06:28] <cstone>	 i was trying to see if we had anything but couldnt find it
[05:07:10] <ejegg>	 cstone you mean getting the repos ready?
[05:07:21] <ejegg>	 or just now getting the new code onto production?
[05:07:30] <cstone>	 the new code onto production
[05:09:05] <ejegg>	 after dwisehaupt updated the frdeploy config to point to the new branch, i just ran the usual fundraising_code_update 
[05:09:10] <ejegg>	 and rsync_blaster
[05:09:17] <ejegg>	 but it seemed to be messed up
[05:09:41] <cstone>	 ah okay there wasn't anything special going on
[05:09:48] <dwisehaupt>	 yeah. git can go wonky when changing branches with untracked files or changes.
[05:10:23] <ejegg>	 k, i'm overdue for bed.
[05:10:40] <dwisehaupt>	 cool. i'll fix up the temp read only bits i did.
[05:10:42] <ejegg>	 i started a testing etherpad here: https://etherpad.wikimedia.org/p/payments-1.43-test
[05:12:32] <dwisehaupt>	 oh. just so it's clear, donorwiki is still on 1_39, but i don't know if we need to do that one tonight.
[05:12:41] <dwisehaupt>	 may be a good one for someone else to do tomorrow.
[05:12:50] <dwisehaupt>	 low stakes and all
[05:13:18] <cstone>	 good point dwisehaupt 
[05:16:15] <ejegg>	 ah yeah, thanks for the reminder
[05:16:27] <ejegg>	 ok, good night!
[05:16:40] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for jbrungs - https://phabricator.wikimedia.org/T385651#10527626 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[05:17:27] <dwisehaupt>	 ok. i think it would be unwise to do the upgrade to frbast and frlog tonight given how i didn't sleep last night. i'll send the end of day email and then do them tomorrow.
[05:17:45] <ejegg|away>	 thanks dwisehaupt and cstone 
[06:13:45] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_42) - 10https://gerrit.wikimedia.org/r/1117692 (owner: 10L10n-bot)
[07:20:52] <ppenloglou>	 Hey all! Just sharing here that payments.wiki seems to be down at the moment. The error displayed is "MediaWiki internal error.Original exception: [Z6Rh496TU5qrm_bAax5KAwAAAAA] 2025-02-06 07:16:51: Fatal exception of type "TypeError"Exception caught inside exception handler.Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information."
[08:31:48] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 06Data-Engineering, 06Data-Engineering-Radar, and 2 others: TLS connection for hive-standalone-metaserver with minio - https://phabricator.wikimedia.org/T385031#10527838 (10BTullis)
[08:40:12] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.092 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki
[08:40:16] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2006.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.064 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_payments_wiki
[08:40:20] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2005.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.063 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_payments_wiki
[08:41:12] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1006.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.076 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki
[08:42:14] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1005.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.087 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki
[08:42:18] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2004.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.065 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_payments_wiki
[08:45:14] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.075 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki
[08:45:18] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2006.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.059 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_payments_wiki
[08:45:22] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2005.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.062 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_payments_wiki
[08:46:12] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1006.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.090 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki
[08:47:16] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1005.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.093 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki
[08:47:20] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2004.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.059 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_payments_wiki
[08:50:16] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2006.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.069 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_payments_wiki
[08:50:16] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.073 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki
[08:50:22] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2005.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.066 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_payments_wiki
[08:51:14] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1006.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.074 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki
[08:52:14] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1005.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.088 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki
[08:52:18] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2004.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.065 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_payments_wiki
[08:55:12] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.068 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki
[08:55:16] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2006.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.060 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_payments_wiki
[08:55:20] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2005.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.065 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_payments_wiki
[08:56:12] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1006.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.073 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki
[08:57:14] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1005.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.086 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki
[08:57:18] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2004.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.062 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_payments_wiki
[09:00:14] <icinga-wm>	 PROBLEM - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.089 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki
[09:00:18] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2006.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.067 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_payments_wiki
[09:00:22] <icinga-wm>	 PROBLEM - check_payments_wiki on payments2005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2005.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.065 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_payments_wiki
[09:01:00] <icinga-wm>	 ACKNOWLEDGEMENT - check_payments_wiki on payments1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1005.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.086 second response time Dwisehaupt rollback caused issues. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki
[09:01:00] <icinga-wm>	 ACKNOWLEDGEMENT - check_payments_wiki on payments1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1006.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.073 second response time Dwisehaupt rollback caused issues. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki
[09:01:00] <icinga-wm>	 ACKNOWLEDGEMENT - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 15148 bytes in 0.089 second response time Dwisehaupt rollback caused issues. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki
[09:01:00] <icinga-wm>	 ACKNOWLEDGEMENT - check_payments_wiki on payments2004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2004.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.062 second response time Dwisehaupt rollback caused issues. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_payments_wiki
[09:01:00] <icinga-wm>	 ACKNOWLEDGEMENT - check_payments_wiki on payments2005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2005.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.065 second response time Dwisehaupt rollback caused issues. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_payments_wiki
[09:01:00] <icinga-wm>	 ACKNOWLEDGEMENT - check_payments_wiki on payments2006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2006.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 11637 bytes in 0.067 second response time Dwisehaupt rollback caused issues. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_payments_wiki
[09:45:12] <icinga-wm>	 RECOVERY - check_payments_wiki on payments1007 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.047 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki
[09:45:16] <icinga-wm>	 RECOVERY - check_payments_wiki on payments2006 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.044 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_payments_wiki
[09:45:20] <icinga-wm>	 RECOVERY - check_payments_wiki on payments2005 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.044 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_payments_wiki
[09:46:10] <icinga-wm>	 RECOVERY - check_payments_wiki on payments1006 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.053 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki
[09:47:12] <icinga-wm>	 RECOVERY - check_payments_wiki on payments1005 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.052 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki
[09:47:18] <icinga-wm>	 RECOVERY - check_payments_wiki on payments2004 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.045 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_payments_wiki
[11:15:23] <wikibugs>	 (03PS1) 10Jgleeson: Hotfix to unbreak payments [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117870
[11:16:06] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] Hotfix to unbreak payments [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117870 (owner: 10Jgleeson)
[11:18:47] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2] Hotfix to unbreak payments [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117870 (owner: 10Jgleeson)
[11:45:20] <wikibugs>	 (03PS1) 10Jgleeson: Add guard clause to to requiresChargeback fn [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1117878
[11:46:28] <jgleeson>	 damilare: that should fix those recent failmails ^
[11:47:57] <jgleeson>	 it might be worth a chaos ticket for digging into to see what type of other unsuccessful trxn messages we get 
[11:48:31] <damilare>	 yea that makes sense
[11:48:42] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] Add guard clause to to requiresChargeback fn [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1117878 (owner: 10Jgleeson)
[11:48:53] <damilare>	 thanks for the fix!
[11:48:59] <jgleeson>	 np
[11:49:08] <wikibugs>	 (03Merged) 10jenkins-bot: Add guard clause to to requiresChargeback fn [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1117878 (owner: 10Jgleeson)
[11:50:20] <wikibugs>	 (03PS1) 10Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1117879
[11:52:05] <wikibugs>	 (03CR) 10Jgleeson: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1117879 (owner: 10Jgleeson)
[11:52:52] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1117879 (owner: 10Jgleeson)
[11:56:05] <wikibugs>	 (03PS1) 10Damilare Adedoyin: Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880
[11:57:10] <damilare>	 jgleeson this should also fix the original issue and still allow the fallback where necessary. I'd test on staging also
[11:57:39] <damilare>	 still not feeling settled about not knowing where the config is being set
[11:58:58] <jgleeson>	 hmm I think it would be better to do a not empty check
[11:59:13] <jgleeson>	 as we're still potentially risking sending in empty arrays 
[11:59:24] <jgleeson>	 if they are both empty, I'd say we want an exception
[11:59:37] <jgleeson>	 feels exceptional circumstances 
[12:00:03] <jgleeson>	 so at the top, expand that ternary out and have a !empty check before assignnment
[12:00:16] <jgleeson>	 and then if both are not set call it out
[12:00:35] <jgleeson>	 damilare: ^
[12:02:42] <damilare>	 seems the issue is for null, I tested with empty arrays and it works. 
[12:03:11] <damilare>	 good call to do the expansion of the ternary too
[12:03:42] <jgleeson>	 yeah that will work but you only have to change it in one place and also I think until we know why it's broken we should guard against both being empty just to be more defensive 
[12:08:10] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[12:08:43] <wikibugs>	 (03PS2) 10Damilare Adedoyin: Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880
[12:09:32] <wikibugs>	 (03CR) 10Jgleeson: "As discussed, here's a slightly different approach which expands it out and adds a guard against it not being set in both." [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[12:11:29] <damilare>	 jgleeson: oh I think the ternary works. During the test in the staging env, AuthManagerConfig was set but with only primaryauth in the values
[12:12:00] <damilare>	 that's why it was failing because it couldn't read the preauth
[12:13:10] <damilare>	 so if we go with your suggestion, we would hit the error page again
[12:13:29] <jgleeson>	 ah ok so let's add a guard before each of those assignments
[12:13:44] <damilare>	 the underlying issue is knowing where that config is being set
[12:13:50] <jgleeson>	 I don't think we can send in an empty array into providerArrayFromSpecs() without side effects 
[12:14:26] <jgleeson>	 so I'd say we wanna have a complete config or if not call it out
[12:15:17] <damilare>	 as in throw an exception when one part is missing?
[12:16:32] <damilare>	 we probably need to understand where this code is useful in our flow
[12:17:08] <jgleeson>	 I'm just looking at it now and the wider code
[12:18:01] <damilare>	 as it is sending empty array for each provider works, so we need to be sure how it is useful. 
[12:18:18] <damilare>	 as it is, sending empty array for each provider works, so we need to be sure how it is useful. 
[12:18:46] <jgleeson>	 it's probably never used by our code as we don't require logins
[12:22:07] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[12:22:15] <jgleeson>	 so I'm thinking it might be best to leave it as null if it's not set
[12:22:34] <jgleeson>	 	private $preAuthenticationProviders = null;
[12:22:37] <jgleeson>	 this is the default
[12:23:06] <jgleeson>	 so in-keeping with your targetted fix, could we check if they are set before sending them to that schema method 
[12:23:55] <jgleeson>	 spec* method
[12:25:50] <damilare>	 that works too
[12:27:44] <wikibugs>	 (03PS3) 10Jgleeson: Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[12:27:50] <jgleeson>	 damilare: I just pushed up a suggestion 
[12:27:51] <wikibugs>	 (03PS4) 10Damilare Adedoyin: Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880
[12:28:42] <damilare>	 ahh looks our patches jammed
[12:28:43] <jgleeson>	 ok you have pretty much the same thing
[12:28:50] <jgleeson>	 I'll +2 that
[12:29:18] <jgleeson>	 jammed?
[12:29:30] <jgleeson>	 your last one LGTM
[12:29:31] <damilare>	 sorry I meant hit each other
[12:29:36] <jgleeson>	 ah lol
[12:29:38] <damilare>	 went in at the same time
[12:29:57] <jgleeson>	 that way, we keep those vars null and avoid that empty call to the spec method
[12:30:02] <jgleeson>	 I think that's best we can do for now
[12:30:10] <jgleeson>	 and then as you say, investigate the underlying
[12:30:24] <damilare>	 makes sense
[12:31:13] <jgleeson>	 cs is complaining about spaces
[12:31:17] <jgleeson>	 CI*
[12:32:02] <damilare>	 ok I'd fix that
[12:32:10] <jgleeson>	 gonna go grab lunch but I'll +2 that when I get back. 
[12:33:50] <wikibugs>	 (03PS5) 10Damilare Adedoyin: Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880
[12:42:52] <wikibugs>	 (03PS1) 10Damilare Adedoyin: Load payments on MW 1_43 and bring mac users in on the bookworm PHP 8.2 fun [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1117897
[12:44:36] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[13:13:51] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] "LGTM!" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1117897 (owner: 10Damilare Adedoyin)
[13:46:38] <wikibugs>	 (03PS4) 10Jgleeson: Verify sending recurring Paypal payments to the donations queue. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1113582 (https://phabricator.wikimedia.org/T366526)
[13:55:12] <wikibugs>	 03Fundraising Sprint Bibimbap, 14Fundraising Sprint: Abula, 06Fundraising-Backlog, 10FR-Smashpig, and 2 others: PayPal audit processor and IPN listener should send payments to donations queue - https://phabricator.wikimedia.org/T366526#10528721 (10jgleeson) I've added a new test suite [[ https://gerrit.wik...
[14:05:24] <wikibugs>	 03Fundraising Sprint Bibimbap, 14Fundraising Sprint: Abula, 06Fundraising-Backlog, 10FR-Smashpig, 10FR-WMF-Audit: Handle PayPal recurring charges that don't include a subscription ID - https://phabricator.wikimedia.org/T385801 (10jgleeson) 03NEW
[14:05:33] <wikibugs>	 03Fundraising Sprint Bibimbap, 14Fundraising Sprint: Abula, 06Fundraising-Backlog, 10FR-Smashpig, 10FR-WMF-Audit: Handle PayPal recurring charges that don't include a subscription ID - https://phabricator.wikimedia.org/T385801#10528757 (10jgleeson) a:05jgleeson→03None
[14:09:05] <wikibugs>	 (03PS2) 10Jgleeson: Add some test links for Gravy dLocal [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1115017 (https://phabricator.wikimedia.org/T385008)
[14:09:12] <wikibugs>	 (03PS2) 10Jgleeson: Break out Gravy Main Page donation links by backend processor/method. [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1115018 (https://phabricator.wikimedia.org/T385008)
[14:09:17] <wikibugs>	 (03PS2) 10Jgleeson: Add full list of Gravy dLocal test payment links [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1115856 (https://phabricator.wikimedia.org/T384967)
[14:09:22] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] Add some test links for Gravy dLocal [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1115017 (https://phabricator.wikimedia.org/T385008) (owner: 10Jgleeson)
[14:09:31] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] Break out Gravy Main Page donation links by backend processor/method. [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1115018 (https://phabricator.wikimedia.org/T385008) (owner: 10Jgleeson)
[14:09:39] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] Add full list of Gravy dLocal test payment links [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1115856 (https://phabricator.wikimedia.org/T384967) (owner: 10Jgleeson)
[14:11:27] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] Update libraries [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1117159 (owner: 10Jgleeson)
[14:11:46] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] "Looks like we have to self-merge these" [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1117159 (owner: 10Jgleeson)
[14:12:20] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Verify sending recurring Paypal payments to the donations queue. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1113582 (https://phabricator.wikimedia.org/T366526) (owner: 10Jgleeson)
[14:21:12] <wikibugs>	 (03PS1) 10Jgleeson: Remove old build script. [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1117913
[14:21:26] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] Remove old build script. [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1117913 (owner: 10Jgleeson)
[14:30:55] <jgleeson>	 right im gonna take a stab at fixing those broken tests on the dlocal chain
[14:31:42] <jgleeson>	 I updated the country field validation here https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1115372/10 and restacked the patch chain so make life easier for us damilare and then a bunch of tests decided the fail :) 
[14:32:08] <jgleeson>	 I figured out a way to get rid of the need to add 'fiscal_number' for each country it applies on
[14:32:24] <jgleeson>	 but CI doesn't like it
[14:34:28] <damilare>	 nice, looks like it's failing in the createPayment block. The catch block in createPayments makes failed test return an error that shows up as a failed asserrtion
[14:34:58] <damilare>	 I wonder how we can handle that differently
[14:46:38] <damilare>	 jgleeson, I'd self +2 that payments fix just so we can test on staging
[14:47:23] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] Add fallbacks to the Authentication Providers config (031 comment) [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[14:47:28] <wikibugs>	 (03CR) 10Damilare Adedoyin: [V:03+2 C:03+2] Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[14:58:59] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add fallbacks to the Authentication Providers config [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117880 (owner: 10Damilare Adedoyin)
[15:08:23] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 06Data-Engineering, 06Data-Engineering-Radar, and 2 others: TLS connection for hive-standalone-metaserver with minio - https://phabricator.wikimedia.org/T385031#10528960 (10Jgreen)
[15:19:10] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] Refactor: small tweaks to RequestMapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1116805 (https://phabricator.wikimedia.org/T385014) (owner: 10Jgleeson)
[15:19:53] <wikibugs>	 (03Merged) 10jenkins-bot: Refactor: small tweaks to RequestMapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1116805 (https://phabricator.wikimedia.org/T385014) (owner: 10Jgleeson)
[15:23:30] <wikibugs>	 (03CR) 10Jforrester: Hotfix to unbreak payments (031 comment) [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117870 (owner: 10Jgleeson)
[15:25:32] <wikibugs>	 (03CR) 10Damilare Adedoyin: "Thanks for putting this up, just had a question about countries with existing integrations inline." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1115372 (https://phabricator.wikimedia.org/T385008) (owner: 10Jgleeson)
[15:43:48] <wikibugs>	 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install fransc1001 - https://phabricator.wikimedia.org/T367814#10529181 (10VRiley-WMF)
[15:50:28] <wikibugs>	 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install fransc1001 - https://phabricator.wikimedia.org/T367814#10529213 (10VRiley-WMF) a:03Jgreen
[15:56:19] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 07Epic: [Metabase] Enable TLS for mariadb connection. - https://phabricator.wikimedia.org/T385813 (10Jgreen) 03NEW
[17:00:07] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2] Hotfix to unbreak payments (031 comment) [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117870 (owner: 10Jgleeson)
[17:25:54] <wikibugs>	 (03CR) 10Jforrester: Hotfix to unbreak payments (031 comment) [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117870 (owner: 10Jgleeson)
[17:26:02] <wikibugs>	 06Fundraising-Backlog, 10FR-payments-general, 06translatewiki.net: Issue with updating es-419 translations for the payments form on translatewiki - https://phabricator.wikimedia.org/T385816#10529708 (10Pcoombe)
[17:31:55] <wikibugs>	 06Fundraising-Backlog, 10FR-payments-general, 06translatewiki.net: Issue with updating es-419 translations for the payments form on translatewiki - https://phabricator.wikimedia.org/T385816#10529726 (10Ejegg) @Nikerabbit I wanted to ask you about this as well - When we updated payments to MW 1.43 I had to di...
[17:37:14] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 07Epic: [Metabase] Enable TLS for mariadb connection. - https://phabricator.wikimedia.org/T385813#10529728 (10Jgreen) For data source connections, adding the CA certificate to "Server SSL certificate chain" in the UI database setup works. It would be better to re...
[17:38:33] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 07Epic: EPIC: Metabase Implementation - https://phabricator.wikimedia.org/T377363#10529730 (10Jgreen)
[19:34:44] <wikibugs>	 (03PS2) 10Umherirrender: Replace call_user_func with dynamic function call [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1115946
[19:57:43] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] Update thank you unsubscribe url to epc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1115952 (https://phabricator.wikimedia.org/T359445) (owner: 10Wfan)
[19:57:58] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] "Thanks for all the work on this. LGTM!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1115952 (https://phabricator.wikimedia.org/T359445) (owner: 10Wfan)
[19:58:21] <wfan>	 thanks dami~
[19:58:39] <damilare>	 np, thanks for all the work on that and the demo too
[19:59:22] <damilare>	 I'd have to drop off now, I'd continue the review on that chain tomorrow.
[20:00:55] <wfan>	 good night
[20:15:59] <cstone>	 wfan: I wonder if we sould just prevemtively put a -1 on that link change just so it doesnt get accidentally deployed
[20:16:12] <cstone>	 or i guess its already merged now
[20:16:45] <cstone>	 ah okay its ontop the other patch
[20:16:48] <wfan>	 Let me update the patch you comment
[20:16:57] <wfan>	 yeah, it's on top
[20:17:02] <cstone>	 so not merged right now
[20:17:10] <wfan>	 yep
[20:17:41] <cstone>	 okie cool
[20:23:39] <cstone>	 ill -1 that other one just as a precaution too just so we dont repeat last time
[20:24:03] <wikibugs>	 (03CR) 10Cstone: [C:04-1] "-1 just because we don't want this to merge until stakeholders are ready for it" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1115835 (https://phabricator.wikimedia.org/T384412) (owner: 10Wfan)
[20:25:04] <wfan>	 haha I will update that based on ur comments then reapply -1 
[20:40:07] <wikibugs>	 (03PS1) 10Jgleeson: Revert "Add fallbacks to the Authentication Providers config" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117965
[20:41:26] <wikibugs>	 (03PS1) 10Jgleeson: Revert "Hotfix to unbreak payments" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117966
[20:41:44] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Revert "Hotfix to unbreak payments" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117966 (owner: 10Jgleeson)
[20:41:58] <jgleeson>	 I thought that might happen
[20:42:21] <jgleeson>	 I'll let CI finish on the revert for damilare|away's followup and then I'll revert out mine
[20:42:49] <cstone>	 ah okay not a 43 issue a merge conflct issue?
[20:43:01] <jgleeson>	 ha no not yet
[20:43:04] <jgleeson>	 that's the dlocal fun
[20:43:14] <jgleeson>	 or did you have your own?
[20:43:38] <cstone>	 hah no just that most -1s this week were from 43 issues
[20:44:46] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Revert "Add fallbacks to the Authentication Providers config" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117965 (owner: 10Jgleeson)
[20:46:37] <jgleeson>	 ah lol
[20:47:24] <jgleeson>	 wth - how can we have phpcs violations in a revert ?!?!?
[20:47:52] <wikibugs>	 (03Abandoned) 10Jgleeson: Revert "Hotfix to unbreak payments" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117966 (owner: 10Jgleeson)
[20:48:06] <wikibugs>	 (03CR) 10Jgleeson: "recheck" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117965 (owner: 10Jgleeson)
[21:00:56] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for mritch - https://phabricator.wikimedia.org/T385136#10530243 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[21:02:03] <jgleeson>	 ok I guess I'll just try and fix those CI failures on the revert and give up on the gods helping us out on that one by reminding CI that the code is being reverted back to a previously merged commit
[21:08:21] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] "force-merging. the phpcs CI failures are due to a comment length exceeding 148 chars. The comment will be gone in the next revert, and fix" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117965 (owner: 10Jgleeson)
[21:10:01] <wikibugs>	 (03PS1) 10Jgleeson: Revert "Hotfix to unbreak payments" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117976
[21:15:08] <wikibugs>	 (03CR) 10Wfan: "recheck" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117966 (owner: 10Jgleeson)
[21:16:04] <wfan>	 this https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1117976 revert seems pass
[21:16:51] <jgleeson>	 wfan: I think we're ok to leave that one in the graveyard - I pushed up a new revert earlier which is just about to pass https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1117976?tab=checks
[21:18:00] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 07Epic: [Metabase] Enable TLS for mariadb connection. - https://phabricator.wikimedia.org/T385813#10530282 (10Jgreen) For the application database:  `MB_DB_CONNECTION_URI=jdbc:mysql://<%= @db_user -%>:<%= @db_password -%>@<%= @db_host -%>:<%= @db_port -%>/<%= @db...
[21:18:02] <wfan>	 yeah I see it's 100% almost there
[21:18:21] <jgleeson>	 yeah I'm watching that pending status in suspense 
[21:19:05] <cstone>	 gogo
[21:19:10] <jgleeson>	 EXACTLY!
[21:19:25] <jgleeson>	 "in your own time CI"
[21:20:02] <jgleeson>	 stewie griffin asking Brian about his novel popped into my head
[21:20:15] <jgleeson>	 "so how's that pending patch coming along..."
[21:21:01] <jgleeson>	 https://www.youtube.com/watch?v=XisVNZ2Xom8
[21:21:03] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Revert "Hotfix to unbreak payments" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117976 (owner: 10Jgleeson)
[21:21:09] <jgleeson>	 omg
[21:21:18] <cstone>	 nuu
[21:21:24] <jgleeson>	 I swear CI is human
[21:22:25] <jgleeson>	 is this for real
[21:22:28] <jgleeson>	 21:20:53 1) MediaWiki\Tests\Maintenance\WikiBirthdayTest::testExecuteForOneRevision
[21:22:28] <jgleeson>	 21:20:53 Failed asserting that 'Wiki was created on: 1 year ago <age: 1 yr(s), 0 month(s), 0 day(s) old>.\n
[21:22:28] <jgleeson>	 21:20:53 ' matches PCRE pattern "/Wiki was created on: 5 April 2023.*age: 1 yr.*0 month.*0 day.*old/".
[21:22:57] <jgleeson>	 these are the important tests right here
[21:23:42] <jgleeson>	 oh wait, was ejegg|away force merging the patches before ours to get past this also?
[21:24:06] <jgleeson>	 let's see if we can work out why the birthday test is failing
[21:24:38] <cstone>	 oh yeah birthday was failing last night
[21:24:41] <cstone>	 i think he was jgleeson 
[21:24:48] <jgleeson>	 lol
[21:24:54] <jgleeson>	 it doesn't want to age
[21:25:11] <cstone>	 i went to go get food inbetween but i think we force merged
[21:25:33] <jgleeson>	 ok great I'll avoid going down that rabbit hole and just force the revert
[21:25:58] <cstone>	 trying to see if we put anything in the backscroll
[21:26:24] <wikibugs>	 (03CR) 10Jgleeson: [V:03+2 C:03+2] "Force merging. We can sing happy birthday later." [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117976 (owner: 10Jgleeson)
[21:33:35] <cstone>	 yeah birthday was failing last night that was a hunt
[21:34:21] <cstone>	 and he forced it
[21:35:05] <jgleeson>	 ok cool well that's back to what it was prior to our fixes earlier
[21:35:22] <jgleeson>	 I'll push that to staging
[21:37:04] <jgleeson>	 fr-tech, just a heads up, once I run the fundraising_code_update on payments (to make it deployable to staging), it will pull in breaking changes to the main copy of the code so we shouldn't deploy that to payments until we get the green light from dwisehaupt that the config fix is deployed 
[21:37:20] <cstone>	 +1 jgleeson 
[21:38:40] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Revert "Hotfix to unbreak payments" [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117976 (owner: 10Jgleeson)
[21:39:28] <dwisehaupt>	 jgleeson: the config fix is out.
[21:40:24] <jgleeson>	 ah thanks dwisehaupt 
[21:40:31] <jgleeson>	 hmm that force-merge got rejected also
[21:41:46] <jgleeson>	 ok fixed
[21:42:01] <cstone>	 ok i had hunted down the old birthday force to see if anything was different
[21:42:11] <cstone>	 woo for merge
[22:03:24] <wikibugs>	 06Fundraising-Backlog: Logo missing on payments wiki 1_43 - https://phabricator.wikimedia.org/T385838 (10Cstone) 03NEW
[22:19:17] <wikibugs>	 (03PS3) 10Wfan: For email-preferences queue do not force country lang as params [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1115835 (https://phabricator.wikimedia.org/T384412)
[22:20:23] <wikibugs>	 (03CR) 10Wfan: For email-preferences queue do not force country lang as params (035 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1115835 (https://phabricator.wikimedia.org/T384412) (owner: 10Wfan)
[22:27:40] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 07Epic: [Metabase] Enable TLS for mariadb connection. - https://phabricator.wikimedia.org/T385813#10530503 (10Jgreen) 05Open→03Resolved This is done. Note that Metabase logs the entire connection string if there's connection error, which is problematic an...
[22:30:30] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 07Epic: [Metabase] Enable TLS for mariadb connection. - https://phabricator.wikimedia.org/T385813#10530564 (10Jgreen) 05Resolved→03Open p:05Triage→03Medium
[22:32:07] <wikibugs>	 10fundraising-tech-ops, 10BDC-Implementation, 07Epic: [Metabase] Enable TLS for mariadb connection. - https://phabricator.wikimedia.org/T385813#10530591 (10Jgreen) Reopening because it looks like we can move the password out of MB_DB_CONNECTION_URI so we don't log it on a connection error: https://github.com...
[22:53:35] <wikibugs>	 06Fundraising-Backlog: Logo missing on payments wiki 1_43 - https://phabricator.wikimedia.org/T385838#10530697 (10Cstone)
[23:07:29] <wikibugs>	 (03PS4) 10Wfan: For email-preferences queue do not force country lang as params [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1115835 (https://phabricator.wikimedia.org/T384412)
[23:19:24] <wikibugs>	 (03PS1) 10Cstone: WIP: Add logos [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117986
[23:20:57] <wikibugs>	 (03PS2) 10Cstone: WIP: Add logos [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117986
[23:21:56] <wikibugs>	 (03PS3) 10Cstone: Add logos [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117986
[23:26:13] <cstone>	 i hope these dont fail the birthday test hah
[23:28:41] <wikibugs>	 06Fundraising-Backlog, 13Patch-For-Review: Logo missing on payments wiki 1_43 - https://phabricator.wikimedia.org/T385838#10530774 (10greg) Aren't those urls (eg `images/e/eb/`) autogenerated by MW?
[23:32:17] <wikibugs>	 06Fundraising-Backlog, 13Patch-For-Review: Logo missing on payments wiki 1_43 - https://phabricator.wikimedia.org/T385838#10530781 (10Cstone) yeah I think from the first upload @greg and then we need to copy them over to the new branches
[23:32:59] <cstone>	 we did it for the previous branches greg-g 
[23:33:24] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add logos [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117986 (owner: 10Cstone)
[23:33:28] <cstone>	 birthday test too strong
[23:33:34] <greg-g>	 cstone: gotcha!
[23:33:36] <wikibugs>	 (03CR) 10Cstone: [V:03+2 C:03+2] Add logos [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117986 (owner: 10Cstone)
[23:45:23] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add logos [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1117986 (owner: 10Cstone)
[23:46:06] <cstone>	 hmm can i not force merge that one what did the others do
[23:46:32] <cstone>	 hah
[23:48:20] <cstone>	 !log payments-wiki upgraded from d266fdf9 to 793998c0
[23:48:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:48:27] <cstone>	 ok are there logos
[23:48:37] <cstone>	 woo hoo!
[23:49:11] <wikibugs>	 06Fundraising-Backlog, 13Patch-For-Review: Logo missing on payments wiki 1_43 - https://phabricator.wikimedia.org/T385838#10530802 (10Cstone) logos are there!
[23:51:56] <dwisehaupt>	 thanks cstone for noticing and fixing that.