[00:01:11] Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint The Pogues, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Recent Engage Import - Contribution Type = Cash when it should be Engage - https://phabricator.wikimedia.org/T100853#1473614 (cwdent) Ok, this has bee... [00:12:00] Fundraising Sprint The Pogues, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Determine how to load CentralNotice RL modules and when to execute campaign and banner selection logic - https://phabricator.wikimedia.org/T106577#1473640 (AndyRussG) [00:31:24] Fundraising Sprint The Pogues, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Spike: Determine how to load CentralNotice RL modules and when to execute campaign and banner selection logic - https://phabricator.wikimedia.org/T106577#1473704 (AndyRussG) [00:33:43] Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, Epic, Patch-For-Review, and 2 others: Special:RecordImpression should die in a fire - https://phabricator.wikimedia.org/T45250#1473707 (AndyRussG) >>! In T45250#1473036, @ori wrote: > What's the status of this? Are we any closer to the comp... [00:36:57] (CR) AndyRussG: "Task: Performance analysis of this and other possible approaches. :)" [extensions/CentralNotice] (campaign_mixins) - https://gerrit.wikimedia.org/r/221759 (https://phabricator.wikimedia.org/T100686) (owner: AndyRussG) [01:46:35] (CR) AndyRussG: "> Neither the current situation nor anything else on the table delays main content," [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/213990 (https://phabricator.wikimedia.org/T100372) (owner: Jdlrobson) [06:44:20] (PS2) Awight: Better success check for recurring globalcollect [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/225261 (https://phabricator.wikimedia.org/T105848) (owner: Ejegg) [06:44:29] (CR) Awight: [C: 2] Better success check for recurring globalcollect [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/225261 (https://phabricator.wikimedia.org/T105848) (owner: Ejegg) [06:52:23] Fundraising Sprint The Pogues, Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Epic: Modify DonationInterface limbo code for high availability deployment - https://phabricator.wikimedia.org/T104499#1474064 (awight) [07:15:09] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Iframe spinner below the fold is bad UX - https://phabricator.wikimedia.org/T106648#1474093 (awight) [07:15:41] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Design: Iframe spinner below the fold is bad UX - https://phabricator.wikimedia.org/T106648#1474086 (awight) [07:15:54] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Design: Iframe spinner below the fold is bad UX - https://phabricator.wikimedia.org/T106648#1474086 (awight) [07:16:26] (CR) Awight: "Screwy submit... hrm" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/225261 (https://phabricator.wikimedia.org/T105848) (owner: Ejegg) [07:28:36] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, and 6 others: Upgrade payments mw-core to REL1_25 - https://phabricator.wikimedia.org/T94528#1474113 (awight) Status: Newly merged fundraising/REL1_25 code passed another smoke t... [16:05:19] Fundraising Tech Backlog: Choose a new templating engine for Donation Interface forms - https://phabricator.wikimedia.org/T86250#1475473 (Ejegg) Mustache! (...with some handlebars style helpers for l10n) [16:05:26] Fundraising Tech Backlog: Choose a new templating engine for Donation Interface forms - https://phabricator.wikimedia.org/T86250#1475474 (Ejegg) Open>Resolved [16:08:54] Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Use a mockable request wrapper rather than curl wrapper methods - https://phabricator.wikimedia.org/T106702#1475479 (Ejegg) NEW [16:11:09] ejegg: you closed this as resolved but PPena is not seeing it. https://phabricator.wikimedia.org/T102496 [16:11:24] sorry PPena: ^^ [16:11:59] dstrine: I think you have to change the ffname in the link to get the alternate ordering. Let me see... [16:13:20] OK, I just went to 'ways to give', selected japan, and chose 1500 yen. It redirected me to this link: https://payments.wikimedia.org/index.php?title=Special:GlobalCollectGateway&appeal=JimmyQuote&ffname=cc-jvma&recurring=&payment_method=cc&utm_source=Waystogive.default~default~default~default~control.cc&utm_medium=Waystogive&utm_campaign=C11_Waystogive&utm_key=&referrer=https%3A%2F%2Fwikimediafoundation.org%2Fwiki%2FWays_to_Give&langu [16:13:30] which has the JCB card first [16:14:31] how do you select japan on that page? [16:14:39] but if I manually change the ffname in the link to cc-vjma I get the JCB card second [16:14:46] https://payments.wikimedia.org/index.php?title=Special:GlobalCollectGateway&appeal=JimmyQuote&ffname=cc-vjma&recurring=&payment_method=cc&utm_source=Waystogive.default~default~default~default~control.cc&utm_medium=Waystogive&utm_campaign=C11_Waystogive&utm_key=&referrer=https%3A%2F%2Fwikimediafoundation.org%2Fwiki%2FWays_to_Give&language=en&country=JP&returnto=Thank_You%2Fen&amountGiven=¤cy_code=JPY&frequency=onetime&amount=1500& [16:14:58] dstrine: on https://wikimediafoundation.org/wiki/Ways_to_Give/en [16:15:14] under credit/debit card, first expand Asia [16:15:46] oook yeah I see it when selecting japan [16:15:56] I see JCB first [16:16:25] PPena: how did you get your screenshot? [16:16:43] ohhhh [16:16:48] I think I see the problem [16:17:01] the vmaj form is still configured as valid for JP also [16:17:15] and none of them is weighted any higher than the rest [16:17:30] what's vmaj? [16:17:39] so i think you have an equal chance of getting any of the three forms [16:17:54] dstrine: visa, mc, amex, jcb order = vmaj [16:18:06] ah ok [16:18:22] so we should take JP out of the list for the old form [16:18:59] and weight the JCB-in-second form over the JCB-first form [16:19:02] easy patch [16:20:44] ok cool so this is a bug? and the current chance of getting the JCB in first position is 1 in 3? [16:25:16] dstrine sorry just saw this now [16:25:36] dstrine incognito window [16:25:44] then went to our doaniton page in JP [16:25:53] (PS1) Ejegg: Give Japan forms correct selection weight [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/226550 (https://phabricator.wikimedia.org/T102496) [16:26:05] good morning! Do you see my last comment? [16:26:14] PPena, dstrine yep, it's a bug, and right now 1 in 3 chance of each order [16:26:39] so we are almost there [16:26:45] ejegg dstrine cool, thanks for adding this to the fix list! :) [16:27:26] and it's technically slightly improved ... just not 100% awsome [16:27:27] that patch ^^^ should make it show JCB 2nd by default and let you show JCB first as an alternate by setting ffname=cc-JVMA in the link [16:27:45] err, cc-jvma [16:27:51] might be case-sensitive [16:30:42] Fundraising Sprint The Pogues: Bug: JCB logo not always appearing in first position in Japan - https://phabricator.wikimedia.org/T106705#1475568 (DStrine) NEW [16:30:52] bugged^^^ [16:34:58] (PS2) Ejegg: Give Japan forms correct selection weight [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/226550 (https://phabricator.wikimedia.org/T106705) [16:35:38] Fundraising Sprint The Pogues, Patch-For-Review: Bug: JCB logo not always appearing in first position in Japan - https://phabricator.wikimedia.org/T106705#1475610 (Ejegg) a:Ejegg [16:42:51] (Abandoned) Awight: WIP Load Stomp via Composer [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/209846 (owner: Awight) [16:50:20] hi awight ! [16:51:24] top o' the marnin to ye [16:54:21] meganhernandez: the-wub: holler whenever you start disabling campaigns! [16:55:24] unrelated uh, oh: payments1004 orphans: globalcollect_gateway_tr 27059622:8322365987 processResponse Error 21000050 : Blocking validation problems with this payment. Investigation required! [16:55:42] oh really? [16:56:08] 99% sure nothing I did in the last week could have done that... [16:56:31] will en-task it [16:57:06] case 21000050 : //REQUEST {0} VALUE {2} OF FIELD {1} IS NOT A NUMBER WITH MINLENGTH {3}, MAXLENGTH {4} AND PRECISION {5} : More validation pain. [16:57:31] Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Orphan slaying bug - https://phabricator.wikimedia.org/T106714#1475742 (awight) NEW [16:58:00] Fundraising Tech Backlog, MediaWiki-extensions-DonationInterface: Orphan slaying bug - https://phabricator.wikimedia.org/T106714#1475749 (awight) [16:58:23] meganhernandez: the-wub: ready to go [16:58:24] ?? [16:58:32] sorry, meant to be less emphatic: "?" [16:58:43] hi awight sure can take campaigns down [16:58:50] hehe thank you! [17:02:49] ok you should be set awight [17:03:23] confirmed, all our campaigns are down [17:04:45] Great. Waiting for the logs to settle... [17:04:54] awight: how do you plan to do the LocalSettings changes? patch file? [17:05:39] ejegg: edit on boron and rsync out [17:05:48] ok enjoy, ping if you need anything [17:05:50] little loophole in our security... [17:05:54] meganhernandez: thank you! [17:06:05] Jeff_Green: just checking that you're around to potentially roll back the DB? [17:06:20] yawp [17:06:31] Jeff_Green: aah, and I just realized I don't know how to do a schema migration! [17:06:44] Can I run maintenance/update.php on payments1004 and that will do it? [17:06:45] Jeff_Green: barbaric! [17:07:09] awight: yeah it should, but beware payments1004 doesn't have the same checkout as production [17:07:10] Side note: lots of these, Jul 23 17:05:16 payments1003 globalcollect_gateway: 27060516:8608658601 SET_PAYMENT couldn't communicate properly! [17:07:16] Jeff_Green: right, thx [17:07:17] at least...hmm. lemme doublecheck that [17:07:45] C13_en.wikipedia.org ?? [17:08:21] I hate that I only look at the logs at moments like this. [17:08:26] fundraising/REL1_23 [17:08:50] Jeff_Green: that also! [17:08:55] Can you change that to fundraising/REL1_25 [17:08:57] oof [17:09:08] yup. sec [17:09:18] both payments-wiki and payments-wiki-staging [17:09:22] awight: that looks like someone clicking on the sidebar link. I can block people at donatewiki too if you need it? [17:09:29] awight: anything you can offload my way? [17:10:17] awight how about payments-wiki-testing, iirc that's a half-done spin-up-payments-test-instance-on-lutetium thing [17:10:22] ejegg: Erm, could you document all the things I've overlooked in the Phabricator task? [17:10:38] Jeff_Green: I'm ready to go straight to production, unless you object? [17:10:54] i mean, should we update that too? [17:10:58] oh heh [17:11:07] sure, maybe we'll just resync the whole db later [17:11:10] sure. so far I've got "change the branch in the deploy script config" [17:12:10] ejegg: that was a big one :) [17:12:21] also: where we run update.php [17:12:38] ok [17:12:41] the-wub: looks pretty well settled, going into maintenance mode now. [17:12:47] ok the script config is updated [17:13:08] shall I flip payments to maintenance mode in puppet? [17:13:16] or do you need to access it? [17:13:48] Jeff_Green: oh, yeah if puppet is the best practice, please do. [17:14:00] that will stop the crons too [17:14:04] doing... [17:14:05] I was about to rsync_blast Localsettings with $wgContributionTrackingFundraiserMaintenance = true; [17:14:14] perfect, that's better. [17:15:20] Okay, upgrading boron code to 1.25 [17:15:52] puppet is propagating the maintenance mode flip [17:16:02] cool, I'll wait before rsyncing [17:16:04] Jeff_Green: error: Cloning into 'skins/Vector'... [17:16:15] in /srv/www-staging... [17:16:28] ooh, old skins are still in that dir [17:16:31] ouch [17:16:54] do we need to force a git pull or whatever? [17:17:22] I think, just rm -rf skins/Vector [17:17:34] are you working in www-staging now? [17:17:34] maybe creepier: error: fatal: unable to access 'https://git.wikimedia.org/git/mediawiki/skins/Vector.git/': Failed to connect to git.wikimedia.org port 443: Connection timed out [17:17:38] yeah [17:17:45] err, there's no skins/Vector, just skins/vector [17:17:49] oh? [17:17:51] trying a second checkout [17:17:55] errors again [17:18:06] we must have changed the IP for git.wm.o [17:18:07] sec [17:18:17] wrong submodule remote, maybe! [17:18:33] same as I've got locally, and I just fetched ok [17:18:35] totally wrong. eff [17:18:53] ejegg: there's a more restrictive firewall around the FR cluster [17:18:58] ohh [17:19:44] git.wikimedia.org not gerrit.wikimedia.org ? [17:19:50] (PS1) Awight: hack the submodule remote [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226565 [17:19:53] ^^ [17:19:58] That will require a "git submodule sync" [17:20:04] looking [17:20:07] thx [17:20:14] Jeff_Green: exactly. fubar [17:20:32] awight: ok are you able to fix it in the git config or do we need fw holes? [17:20:49] The patch above will fix it, but we might need a "sudo git submodule sync" [17:21:02] donno which user needs to run that to keep the repo healthy [17:21:15] (CR) Ejegg: [C: 2 V: 2] hack the submodule remote [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226565 (owner: Awight) [17:21:27] I'll try a simple update, first [17:21:35] FYI: maintenance mode has propagated [17:21:55] awight the local user is mwdeploy [17:22:11] let me know if you need me to do anything [17:22:58] Jeff_Green: yes, please: cd /srv/www-staging/org/wikimedia/payments; sudo -u mwdeploy git submodule sync [17:23:03] also in the non-staging dir [17:23:16] ok, sec [17:24:12] done [17:24:32] Ok? Enter the following 4-digit key if you're sure: iyes / [no]: [17:24:39] healthy. [17:24:46] hehe [17:24:50] nice [17:25:02] ha. never yet have I seen it swear, you'd think that would happen now and again [17:25:23] I believe K4-713 got "soxy" once [17:25:36] not bad [17:25:40] It was "rsxy". [17:26:06] I fell over laughing. [17:26:50] ...I miss deploying. [17:26:51] * dstrine lulz [17:26:57] following 4-digit key if you're sure: kafk [17:27:12] o_O [17:27:24] I couldn't invent this stuff [17:27:29] The deploy machine is going all skynet. [17:27:32] "pseudorandom", indeed [17:27:39] I didn't think it would be us. [17:27:42] two for you--one, two for me [17:28:05] status: fully rsync'd, now I'm doing the db migration [17:28:33] Oh: In case it wasn't totally apparent, I'm WFH today. [17:28:48] Waiting for a new power supply to show up so I can put my desktop back together. [17:29:20] old school [17:29:22] Jeff_Green: my user can't run anything on pay1004, cos I can't read localsettings. [17:29:33] glorious [17:29:35] aren't we supposed to throw our devices away as soon as they get slow or malfunction now? [17:29:46] Jeff_Green: I need payments1004: cd /srv/www/org/wikimedia/payments ; php maintenance/update.php --quick [17:29:49] wheee [17:30:14] Sorry to have you do my dirty work... [17:30:25] cwdent: Mostly if you blast the six years of warehouse dust out of the heat sink, they're just like new. [17:30:29] Mostly. [17:30:44] cwdent: I burn them in the back yard every 2 months or so [17:31:07] regardless of performance [17:31:13] awight that took zero time [17:31:19] Grr, I got a new CPU fan to breathe the life back into my favorite old laptop--the new one came from Shenzhen with the last winner's heat sink goop all dried up in my business [17:31:24] (Why is nobody saying "Hit old toys repeatedly with a hammer".) [17:31:25] refurb [17:31:41] Jeff_Green: did it say anything? it should have been fast, but actually done a few things. [17:31:42] Jeff_Green: that one's a pretty tiny db, right? [17:31:48] dstrine: you should be sending them to china where small children can use hazardous chemicals to separate small amounts of precious metal! [17:32:02] Jeff_Green: whatev, let's turn maintenance mode off and see what we got [17:32:16] and build toxic dystopian homes from the boards [17:32:25] ok [17:32:26] cwdent: why should they have all the fun? [17:32:37] awight yeah tiny db [17:32:43] cwdent: Does hitting things with hammers make that whole process easier, or more difficult? [17:32:47] Also, now I'm sad. [17:32:53] cwdent: I hear that silicon valley was built on piecemeal home soldering and defluxxor, why go to another continent? [17:33:15] we're number one in density of abandoned Superfund sites! [17:33:25] Or should I say, new soccer fields... [17:33:58] Wow, we're running this deploy right up to the team health check? [17:34:07] That's going to be distracting for some people. [17:34:07] It's a test, you see... [17:34:08] propagating [17:34:21] yeah I was a little concerned by this timing [17:34:50] do we think this is going to be complete by 11? [17:35:10] Will overall distraction of the team actively tailing the logs for an hour be taken into consideration when evaluating our overall health? [17:35:10] Fundraising Tech Backlog, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Moar s3xy payments maintenance mode - https://phabricator.wikimedia.org/T106719#1475881 (awight) NEW [17:35:12] ..:p [17:35:56] Ow! File does not exist: /srv/www/org/wikimedia/payments/images/9, [17:36:03] dstrine: For context, it is FORBIDDEN to walk away from a done deploy until an hour passes with no hiccups. [17:36:05] ??? [17:36:29] the-wub: meganhernandez: please try donating via a few different methods... [17:36:40] We're not ready for campaigns yet, just smoke testing. [17:36:46] wait, all I get is whitescreen still... [17:37:08] awight: on it [17:37:12] awight you did the non-staging propagation? [17:37:16] Jeff_Green: yep [17:37:28] well then. what now? [17:37:36] ejegg: what URL? https://payments.wikimedia.org/index.php?title=Special:GlobalCollectGateway&appeal=JimmyQuote&ffname=cc-vmad&recurring=&uselang=en&language=en¤cy=EUR&country=US&amount=2 [17:37:49] lemme know if you want maint mode back, you can still get in if you tunnel instead of going through the front end [17:37:49] Jeff_Green: smoke test, perhaps announce success [17:38:07] aha, good to know. I just made a task related to that. [17:38:21] oh I thought the hitescreen comment meant explode [17:38:22] awight: whitescreen there, also at https://payments.wikimedia.org/index.php?title=Special:AstropayGateway&appeal=JimmyQuote&ffname=astropay&recurring=&payment_method=cc&utm_source=fr-redir.default~default~default~default~control.cc&utm_medium=spontaneous&utm_campaign=spontaneous&utm_key=&referrer=&language=en&country=BR&returnto=Thank_You%2Fen&amountGiven=¤cy_code=BRL&frequency=onetime&amount=20&uselang=en [17:38:25] I thought we had a slightly classier "we're totally screwed" notice [17:38:35] *that* is whitescreen for me [17:38:37] in different private mode windows [17:38:46] now at my URL as well. [17:39:08] awight: problems with GC credit card. only get visa/mc in the US. and then iframe doesn't load: get an alert with "There was an error processing your request." [17:39:19] https://payments.wikimedia.org/index.php?title=Special:GlobalCollectGateway&appeal=JimmyQuote&ffname=cc-vmad&recurring=&payment_method=cc&utm_source=test.default~default~default~default~control.cc&utm_medium=test&utm_campaign=test&utm_key=&referrer=&language=en&country=US&returnto=Thank_You%2Fen&amountGiven=¤cy_code=USD&frequency=onetime&amount=3&uselang=en [17:39:22] paypal doesn’t go to paypal [17:39:36] okay, too many problems... rolling back. [17:40:08] Jeff_Green: you good to do maintenance mode; the DB rollback; and set the frdeploy needle back to fundraising/REL1_23 [17:40:16] sure [17:40:51] Wow. That's a new one. [17:40:56] logs have squat, fwiw [17:41:27] PHP Catchable fatal error: Argument 1 passed to DatabaseBase::__construct() must be an array, string given [17:41:37] wat. [17:41:47] hold on, that would be something we can fix [17:41:49] called in /srv/www/org/wikimedia/payments/extensions/ContributionTracking/ContributionTracking.processor.php on line 40 [17:42:18] we should be in maintenance mode-ish now [17:42:33] Jeff_Green: maybe give one more minute before hitting with the other hammers [17:42:38] ejegg: where did you find that? [17:42:52] payments-globalcollect and -astropay [17:42:53] Jeff_Green: If this is just a DB connection string thing, I'd be happy to try fixing [17:42:56] k [17:43:01] awight: ok [17:43:15] why don't you try tunneling and we can stay in maint mode while you poke at it [17:45:51] (PS1) Awight: Update ContributionTracking submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226575 [17:45:52] ejegg: that one ^ [17:46:05] oho! [17:46:23] yep, essential feb 23 fix [17:48:00] (CR) Ejegg: [C: 2] Update ContributionTracking submodule [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226575 (owner: Awight) [17:48:21] no deploy branch for ContributionTracking, huh? [17:48:27] not yet... [17:48:49] Jeff_Green: I'm not getting tunelling working properly. please remind me of the ports? [17:49:03] looking [17:49:34] I'm ready to retry the frontend [17:49:56] 8080 [17:50:02] ok [17:50:02] that's non-ssl [17:50:09] oh! [17:50:27] ssl-->nginx-->plaintext-->apache ya [17:51:00] shit, can't get resources. [17:51:06] ejegg: Really, we should just have DonationInterface suck all the goodies out of ContributionTracking and be done with it. [17:51:07] Jeff_Green: can we try public-facing again? [17:51:13] K4-713: +1 [17:51:16] ejegg: I mean, probably. [17:51:27] The only issue would be deploying on donate. [17:51:34] we have replacements for the interstitial pages... [17:51:34] awight sec, lemme try here [17:51:37] thx [17:51:49] K4-713: oh, so donate has CT but not DI? [17:51:54] Yep. [17:51:57] Well... sort of. [17:52:05] There's that Language-only mutant. [17:52:17] 'fortunately', CT doesn't seem to be working on donatewiki any more [17:52:25] Also, contribution tracking on donate is... yeah, that. [17:52:28] Broken. [17:52:32] MBeat: hi. Anyone come by for me with torches and pitchforks yet? [17:52:33] :/ [17:52:40] heh, in that case definitely squash 'em together [17:52:46] The really weird thing is that nobody seems to care. [17:52:48] They hella should. [17:52:50] nope, all quiet awight [17:53:21] Usually with things like that, they start caring very suddenly in the middle of a campaign. [17:54:41] sorry, I think I’m late to the party here… is JP processing down? [17:54:52] blarg it's much harder with the magical enforced https->http thing [17:54:58] sigh ok. going nonmaint [17:55:12] ccogdill: yeah, campaigns are off for another few minutes. [17:55:25] We're deploying a paymentswiki upgrade, experiencing turbulence... [17:55:41] okay, I think I missed the memo :D I thought I’d broken something in an email test [17:55:45] thanks! [17:55:49] Fundraising Sprint Kraftwerk, Fundraising Sprint Lou Reed, Fundraising Sprint Miles Davis, Fundraising Sprint N*E*R*D, and 6 others: Upgrade payments mw-core to REL1_25 - https://phabricator.wikimedia.org/T94528#1476006 (Ejegg) [17:55:51] ccogdill: so far, campaigns have been down for an hour. We hope to turn it back on at 11 PDT, one way or another. [17:56:00] sounds good [17:56:34] I like how we all unconsciously assign ouselves blame first. [17:56:42] woot I've got it [17:56:44] mines! [17:56:58] ccogdill: sorry we discuss in standup yesterday and had separate emails with Peter and Megan yesterday. I wasn't sure who else to loop in [17:57:08] My daughter by the way says "mines" about everything. It's chilling [17:57:29] Minesfield. [17:57:34] Jeff_Green: we're just waiting for the puppet? [17:57:53] sec [17:58:14] give it 1 minute [17:58:23] so what we can do in the future.... [17:58:27] ^R^R^R^R^R :p [17:58:36] put payments100[1-3] in maint mode [17:58:39] then [17:58:49] (PS1) Ejegg: Restore favicon to fundraising/REL1_25 [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226579 (https://phabricator.wikimedia.org/T94528) [17:58:54] Get that tunnel working smoothly... then we can just stage it before even touching 1-3 [17:59:06] [as root on your local machine] ssh -L 443:payments1004:443 tellurium.wikimedia.org [17:59:14] (CR) Awight: [C: 2] Restore favicon to fundraising/REL1_25 [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226579 (https://phabricator.wikimedia.org/T94528) (owner: Ejegg) [17:59:25] and change your hosts file to stick payments.wm.o on 127.0.0.1 [17:59:39] ok. [17:59:42] that worked for me [18:00:14] dstrine no worries, please just loop me in in the future. I almost emailed ~80k people last night and would have been a bummer if our forms went down in the first 24 hours of the send [18:00:38] HSTS makes use of the plaintext connection difficult [18:00:39] meganhernandez: the-wub: we're ready for more smoke testing [18:00:42] woohoo! Forms still look good [18:00:48] dstrine: I think Major Gifts wanted to know about these things, too. [18:01:09] So... emailing fr-all would not be weird. [18:01:22] Definitely covers everything that way. [18:01:52] awight we should be back by now [18:02:02] AstroPay redirect looks healthy [18:02:07] the-wub: meganhernandez: I made a successful CC payments. [18:02:22] paypal went to paypal [18:03:06] amazon went to amazon [18:03:12] wowza. [18:03:34] I'd say we're ready for campaigns to go back on. Nothing fishy in the logs. [18:03:40] meganhernandez: ^ [18:04:06] are the non-DI bits of mediawiki logging just like before? [18:04:35] ejegg: and K4-713 are you available for team health check? [18:04:39] ejegg: Umm... I don't even know where those would go [18:04:47] K4-713: ^ [18:04:52] i know we made a card for this before, just double checking this is an old issue and not new from this deployment. the legal text on donate wiki is in centered at the bottom of the page. it’s usually on the right. that’s old, right? [18:05:06] Ah. I thought you told me it was cancelled. [18:05:17] not yet [18:05:21] maybe we should discuss? [18:05:25] awight: recurring GC donation worked :) Amazon and BPay looked okay too (although didn't make test donations) [18:05:28] meganhernandez: I can't say. Mine is left-justified. [18:06:26] Jeff_Green: Thanks again! Things to learn from, but I think we got liftoff this time! [18:06:26] meganhernandez: I think it's always been centered? at least since we implemented Trilogy's redesign [18:06:45] awight: ya. I think we should make the staging instance more useful [18:06:55] no it’s still right on the credit card form, it’s just on donate wiki [18:06:56] ejegg: you around? [18:07:03] we should be able to run a second site on payments1004, on different ports, using a different local db instance [18:07:06] Jeff_Green: I love the idea of a fully functional staging thing. [18:07:10] ya [18:07:16] That sounds even better, to split staging from orphan slaying. [18:07:21] right [18:07:29] Cos orphan slaying has a tight time window in which it operates. [18:07:41] dstrine: yep, i'm right here [18:07:48] yup, and you don't want it surprise using a different broken version of the code [18:07:56] in call in 1 sec [18:08:07] awight: want me to put campaigns back up? [18:08:09] awight I'll poke in a phab ticket for posterity [18:08:19] meganhernandez: yes please! [18:08:24] I think we're good. [18:08:47] ok [18:11:00] back up! [18:11:28] cool [18:11:42] I checked iDeal, paypal in usd, and paypal recurring. no issues [18:13:00] haven't noticed any layout issues either. tried right-to-left language too [18:13:13] hey the-wub want to double check i didn’t miss enabling any campaigns? [18:13:19] should be good [18:18:11] brb [18:18:58] awight ejegg: just noticed something, footer images are 404 (https://payments.wikimedia.org/resources/assets/poweredby_mediawiki_176x62.png, https://payments.wikimedia.org/images/9/9a/Wikimediafoundation-logo.png) [18:19:28] the-wub: ok, let me see if I can restore those from the older branch [18:19:46] aha [18:19:52] the-wub: Thanks for the spelunking [18:19:53] not a big deal though. I can make a phab task for it if you want to tackle it later [18:21:02] np [18:21:10] meganhernandez: campaigns all look good to me [18:32:33] thanks [18:34:49] Fundraising-Backlog: Footer images on payments missing - https://phabricator.wikimedia.org/T106728#1476129 (Pcoombe) NEW [19:28:18] (PS1) Awight: Copy images from fundraising/REL1_23 [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226609 [19:29:50] (CR) Ejegg: [C: 2 V: 2] "Thank goodness you rescued Jimmy!" [core] (fundraising/REL1_25) - https://gerrit.wikimedia.org/r/226609 (owner: Awight) [19:30:14] lol [19:30:20] poor jimmy fell down the well again! [19:30:36] hehe [19:31:35] Jeff_Green: c'mon, tell me you added AI. Ok? Enter the following 4-digit key if you're sure: efud / [no]: [19:32:09] it's trying to tell you to eat lunch! [19:32:09] sorry i was out of irc for a while and didn’t realize. all good with the deployment? need anything? [19:32:14] ino! [19:32:24] meganhernandez: I think the dust finally settled, thanks for bearing with us! [19:32:44] thank you! [19:32:45] Payments seem to be going through normally, the proof of the pudding will be conversion and complaint rates, of course. [19:33:18] do you need pats to check conversion on her side for the different methods? [19:33:21]