[00:01:03] K4-713: whenever I look closely, it seems like there's an Opsen in there actually stabbing and rebooting [00:01:18] But it can sure play a mean game of chess :p [00:04:03] awight: I understand it's being rebooted. [00:05:17] I think it's actually ops doing the builds, too... they just like to call it "automated" [00:05:54] hahaha [00:06:08] They ran out of goldfish crackers and passed out. [00:08:26] brrrp! [00:08:38] awight: Are you sure we don't want to do contribution tracking things in batch mode? [00:09:03] hurm. [00:09:17] rats! I thought that was our consensus from... Friday [00:09:49] oof, well CT is only to track user interaction, but.. in the orphan case... [00:10:07] I guess we've already done CT in the frontend, so there's nothing to add during slaying? [00:10:10] So, orphans I can probably be talked in to. [00:10:31] Because: Yes, they got far enough through DI to get a limbo transaction. [00:10:36] for RGC we definitely don't want to mess with CT [00:10:43] are there other batch operations? [00:10:45] ...yeah, okay. I relent. [00:11:31] donno! [00:11:50] thanks for talking it through. more documentation! [00:12:14] I feel... gross. And I'm going home now. [00:12:34] Largely unrelated to documentation. [00:13:14] I really don't know how zuul schedules things... [00:13:26] 43 mintues. I guess I don't either. [00:13:48] Also minutes. [00:15:07] ...kbye. [00:16:52] awight: -operations may be notable [00:16:57] Krinkle says Jenkins is stuck. [00:17:29] Yeah, it's not slow [00:17:34] if anything, it's faster than ever [00:17:51] Krinkle: I didn't complain because I've noticed slightly slower merge times for a while now [00:17:58] Progressively [00:20:14] For some reason, we stop paying Jenkins and he's working harder than ever! [00:34:09] awight: Would it be feasible to fix https://gerrit.wikimedia.org/r/#/c/166889/ wihtin a few minutes [00:34:15] it has a merge conflict [00:34:28] either that, or I restart zuul and we lose the last hour of queue. [00:35:05] (CR) Krinkle: [C: -1] "Doesn't merge. Removing K4's +2 to see if this unstucks Jenkins." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 (owner: Ejegg) [00:37:51] I'm rebasing it [00:40:40] (PS2) Krinkle: Orphan adapter tests for MC retry [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166803 (owner: Ejegg) [00:40:42] (PS3) Krinkle: Add IP spanking for shady cards to Worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 (owner: Ejegg) [00:40:44] (PS4) Krinkle: Penalize toxic card IPs in velocity filter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166678 (owner: Ejegg) [00:40:47] (CR) Krinkle: [C: 2] "Rebased to resolve conflict with 84f603b3fded27f145239e2a50818413d9c833fb which added a return statement in the same place." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 (owner: Ejegg) [00:41:09] (CR) jenkins-bot: [V: -1] Add IP spanking for shady cards to Worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 (owner: Ejegg) [00:50:28] Krinkle: oof, just saw this. Looking at the patch now... [00:50:54] Krinkle: ah, so it's not blocking you any longer? [00:51:06] nope [00:51:12] it's now back into gerrit with -1 [00:54:22] oh good :) [00:54:27] cya [01:51:48] (CR) Ejegg: "Whoops, halfway between isset and array_key_exists. Sorry to waste everyone's time!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 (owner: Ejegg) [01:58:00] (CR) Ejegg: [C: 2] "Kicking Jenkins per Katie's earlier +2" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166678 (owner: Ejegg) [01:58:28] (Merged) jenkins-bot: Penalize toxic card IPs in velocity filter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166678 (owner: Ejegg) [01:58:30] (Merged) jenkins-bot: Orphan adapter tests for MC retry [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166803 (owner: Ejegg) [02:00:42] (PS4) Ejegg: Add IP spanking for shady cards to Worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 [15:46:14] (PS1) Ejegg: Merge remote-tracking branch 'remotes/origin/master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/167009 [16:01:41] hi AndyRussG ! [16:01:47] Hi ejegg!! :) [16:01:54] Sorry to just be joining here... [16:02:19] So, we are going to have to push our update to both of the wmf versions in prod right now [16:02:22] How long of a slot do we have? [16:02:25] 2 hours [16:02:40] last couple of times we had 1 hour slots and it felt a little rushed [16:03:05] OK [16:03:17] I've got a merge commit in the wmf_deploy branch of CentralNotice that needs a +2 [16:03:29] I mean, I could do that myself [16:03:39] I'm not at home so I'm not sure I,ll be able to do the screenshare [16:03:51] I can plus 2 if you like [16:03:53] Oh, ok [16:04:25] I could probably get home within about 40 minutes tho... dunno what is best [16:04:35] Then with that merged, we have to do the submodule bump in core for wmf/1.25wmf2 and wmf/1.25wmf3 branches [16:05:07] (I went for an exam to get a driver's license at 8:30 this morning, didn't think it would take so long, so I just stopped at this store on the way home...) [16:06:09] Right I remember readig something like that and seeing that when I watched the tutorial... [16:06:40] I'm just following this basically: https://wikitech.wikimedia.org/wiki/How_to_deploy_code [16:09:16] So if you want to help do the submodule update in 1.25wmf2, I'll do it in 1.25wmf3 [16:09:55] Looks like I can do the screenshare!! :) [16:10:02] then we can push wmf3 first and verify on mediawiki.org before pushing wmf2 to all the wikipedias [16:10:16] Oh, cool! [16:10:32] google hangout OK? [16:11:08] Yeah! I don't see any web cams here but I should be able to see your screenshare... [16:11:15] OK, great [16:11:16] one sec [16:12:09] ok, think i sent an invite to your gmail [16:12:30] OK I think I see it :) [16:13:18] I see you! So screenshare will work :) [16:13:40] no mic / headphones though? [16:14:01] Lemme ask at the desk here... don't think so... [16:15:08] No, sorry no mike or headphones [16:15:24] ok, no worries [16:16:25] so I have one concern - the 1.25wmf3 and 1.25wmf2 branches have the vendor submodule in an odd state [16:16:54] Let's ask over in #wm-tech [16:17:03] (Sadly also no system to do anything on also) [16:17:04] OK [16:17:26] Where do you see the odd state? [16:22:07] ejegg: maybe after fetch --all you need to do git pull? [16:23:00] fetch doesn-t update your local tree, just info about remote branches I think [16:32:31] (CR) Ejegg: [C: 2] Merge remote-tracking branch 'remotes/origin/master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/167009 (owner: Ejegg) [16:33:42] AndyRussG: mostly 1c (extension update), then 1b (submodule update), then step 2 [16:34:05] ejegg: OK thx! [16:35:27] Ah cool the JS curiosities patch got in :) [16:36:32] Oh, let's review/push 1.25wmf3 first [16:38:44] K... yeah I've not worked much with submodules befor [16:39:17] you basically just cd into the submodule and update it to the revision you want core to refer to [16:39:33] right... [16:39:36] then back into core and git add the submodule, then commit [16:39:42] and it just updates the reference [16:40:00] yeah interesting... [16:40:58] Hmm I always do git review -R, to prevent any automagic rebasing, mabye it's not needed here... [16:41:02] aargh, I thought I had +2 in core gerrit [16:41:15] Oh really? That doesn't come up for me [16:41:19] Paste the URL? [16:41:36] I think I may [16:41:58] https://gerrit.wikimedia.org/r/167017 [16:43:31] Hmmm no I'm only +1 too, funny [16:44:09] crud... let me check my email for when I tried to get that [16:47:23] Or we could ask someone... [16:48:51] gonna ask over in wm-operations [16:49:02] OK! [17:04:40] ejegg: still following here, looks good [17:05:02] yay for ssh-add [17:07:09] I like the rain better [17:08:12] I see the core patch didn't get merged [17:09:13] Anything I can do? [17:09:21] I just got my +2s back [17:09:48] weee! [17:10:03] \o/ [17:11:11] OK, on to tin! [17:11:29] rin-tin-tin-tin! [17:12:23] Cool, just the one change [17:13:32] no security updates to work around [17:15:12] let's dump it onto testwiki [17:15:12] Ah right [17:16:41] K it's doin' stuff... [17:18:13] well, definitely has the new code [17:18:49] not seeing any js errors [17:18:58] U said that was testwiki? or is it meta? [17:19:34] just testwiki [17:19:40] we'll push to meta in a second [17:19:44] https://test.wikipedia.org/wiki/Main_Page?debug=true [17:20:30] ejegg: I'm getting erros on testwiki [17:20:38] oh? What? [17:21:12] http://test.wikipedia.org/w/index.php?title=Special:CentralNotice&method=listNoticeDetail¬ice=msgFallback [17:21:54] oh dang [17:22:06] Code expecting a schema change? [17:22:17] Not here, no [17:22:33] This does work fine, and shows the new functionality: http://test.wikipedia.org/wiki/Special:CentralNoticeBanners/edit/fallbackBanner [17:22:47] That not_throttle column should have been there for a while, right? [17:23:20] Lemme see... [17:25:04] ejegg: yeah I think so [17:25:24] So I'm willing to bet testwiki's schema is out of date [17:25:36] silly testwiki [17:25:46] Let's try it on meta and roll back if we have to [17:25:48] The question is how much you'll bet [17:25:52] OK yeah that sounds find [17:25:58] fine [17:27:11] down in step 4 [17:27:24] command look right? [17:28:54] I'm afraid I don't know enough to say 8p Don't see the second argument there in the instructions... [17:29:15] "To sync a directory, run sync-dir [path to directory] [summary]" [17:30:10] Well, that was quick! Let's check out meta [17:30:19] OK! [17:31:01] I'll look at the js side of it if you want to look at the admin stuff [17:32:08] ejegg: so far so good... [17:33:12] ejegg: I don't have admin rights to fully test the new features there [17:33:48] But it looks fine [17:33:59] Just a minor bug, but nothing to worry about for now [17:34:05] ok, let's get on to 1.25wmf2 [17:34:09] K! [17:34:10] minor bug? [17:34:18] Erm, nothing wrong wtih the deploy [17:34:21] ok [17:35:02] If you go here. http://meta.wikimedia.org/wiki/Special:CentralNoticeBanners/edit/B14WMDE_141016_ctrl [17:35:47] You get a list of campaigns the banner is used on at the bottom. That's what's new. The info is fine, no existing functionality seems broken, but the link is wrong [17:36:59] oh well, can fix that later [17:37:31] just waiting for verified + merged on the 1.25wmf2 patch [17:37:46] ah jenkins [17:40:29] Hmm I think there was some slowness on Jenkins yesterday [17:41:21] the last one went in really quickly! [17:41:30] yeah [17:41:44] Must be some really pokey ones ahead in the queue [17:42:14] it's being processed now, just the mediawiki-vendor-integration task is taking like 5 min [17:42:54] there are two test queue jobs running the same task - maybe something in that is not happy running in parallel with itself [17:44:00] merged! [17:44:14] yup! [17:46:39] look good? [17:47:12] syncing [17:47:28] ok, let's test enwiki [17:48:25] Yeah! [17:49:05] ooh weird, alterImpressionData hid that one. Uh... [17:49:59] Really? I'm getting cookie data as expected.. [17:50:04] Maybe clear your cache? [17:50:13] I'm going in anonymously [17:50:41] me too [17:50:45] hmmm though [17:51:02] ok, that time was 'empty' [17:51:09] Ah OK [17:51:51] I say +1! [17:52:04] hmm, i really want to see a banner work [17:52:13] Hmm [17:53:11] ejegg: I can test by simulating a France geolocated cookie [17:53:25] ooh, good call [17:54:23] ejegg: confirmed a banner shows nicely [17:54:40] whew! [17:54:42] try chrome's edit this cookie extension [17:54:44] Yeah [17:54:47] Just in time! [17:54:49] OK, guess we're good [17:54:51] yep! [17:54:56] ejegg: :D [17:55:12] K I'm gonna bolt... and will miss standup 'cause I'll be in local transit [17:55:15] OK [17:55:37] THanks much!! cheers :) [17:55:48] ty too [17:57:29] :) [18:32:52] ejegg: Hangout? [18:32:56] It's that time. [18:48:34] (CR) Katie Horn: [C: 2] Add IP spanking for shady cards to Worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 (owner: Ejegg) [18:49:06] (Merged) jenkins-bot: Add IP spanking for shady cards to Worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/166889 (owner: Ejegg) [18:51:02] yo everybody: I'm about to start rolling up another payments deploy so we can get the MC changes out before UTC midnight. [18:51:17] Then probably tail some logs looking for alarming new lines. [18:54:23] harumph. awight isn't here... [18:55:15] * ejegg can't find a cartoon of two gumshoes surreptitiously following a logging sledge, but knows it must exist [19:00:08] (PS1) Katie Horn: Merge branch 'master' into deploy-payments_1.22 [extensions/DonationInterface] (deploy-payments_1.22) - https://gerrit.wikimedia.org/r/167045 [19:00:31] ejegg: That seems like the kind of thing google would be good at finding, too. [19:01:03] (CR) Katie Horn: [C: 2] "+2 for deploy" [extensions/DonationInterface] (deploy-payments_1.22) - https://gerrit.wikimedia.org/r/167045 (owner: Katie Horn) [19:01:12] right? image search with type=line drawing has nothing! And so many unix ppl are fond of puns [19:01:29] maybe we all can't draw? Except Randall Munroe... [19:01:36] I'll draw that. [19:01:42] awesome [19:04:21] Jeff_Green: Looks like the underlying oauth node module has support for different hostnames for user / backend requests, but the drupal wrapper doesn't expose 'em. Drupal wrapper doesn't do a lot though, so I can just use the oauth one and steal a function or two from the wrapper. [19:04:44] ah cool [19:09:06] !log updated payments to 14c415fcfc3cade9a1 [19:09:12] Logged the message, Master [19:09:19] Well, there it goes. [19:10:23] ejegg: I guess I should add the toxic card timeout var, too, huh? [19:10:35] Ooh, good call! [19:12:51] hmm, those are some unfamiliar errors in payments.error - puppet and ntpd stuff, but starting before your deploy [19:14:12] I just said that elsewhere. [19:14:13] oh, #mw-ops is flooded with puppet failure notifications. Must be something wider than just us [19:14:32] ...so it is. [19:25:00] OH, FLIPPING HELL. [19:25:04] I just noticed something. [19:25:05] what? [19:25:22] So... you know how we have the IP velocity filter for live payments happening? [19:25:27] right [19:25:49] It's kind of important that the box that does orphan slaying doesn't have that enabled, because duh. [19:26:14] At some point, somebody seems to have copied the settings from the live boxes, to the orphan slayer. [19:26:21] Oh damn [19:26:24] * K4-713 facepalms [19:26:33] Should code check for batchmode too? [19:26:53] Yeah, that would probably be a better solution. But I guarantee that's not happening now. [19:27:12] Blarf. [19:27:28] ...I'll fix that. [19:27:43] you fix config, I'll put up a patch to check batchmode [19:27:56] ? [19:28:00] Eh, this probably isn't as critical as... a bunch of other things. [19:28:16] Definitely add a card to mingle, though. [19:29:02] it does have that IP whitelist at least. Guess we don't have 127.0.0.1 in there? [19:31:27] Well, here's the thing. It should be going off the client IP, which we still have. [19:31:43] oh, ok [19:36:09] !log payments localsettings updates - supported countries and fraud filter settings [19:36:16] Logged the message, Master [19:39:01] ejegg: Okay, I'm out. Everything is deployed and set... can you tail the logs for a few minutes while I warm some food? [19:39:09] sure thing! [19:39:26] I've seen enough go through that I'm confident that if we're broken, it's an edgecase. :) [19:39:32] brb [19:39:32] nice [20:38:22] hummm. That's weird. [20:38:27] * K4|fooding is still fooding [20:38:31] which one? [20:38:38] The blocking validation errors. [20:38:52] It's because currency = null. [20:38:57] ohhh [20:38:59] ...but how did they get that far? [20:39:07] yah, that's weird [20:39:34] What do you bet we're supernormalizing the currency data? [20:39:35] is it that one banner deep linking? [20:39:46] nope. That's the orphan slayer box. [20:40:06] Oh man, what would we be doing to normalize that into nothing? [20:40:17] Soemthing. [20:40:18] Er. [20:40:26] I know for a fact we don't validate it some places we should [20:40:47] orly [20:40:53] Like... when somebody first loads the page? [20:40:53] if you look at original_currency in wmf_contribution_extra, you see a bunch of things that are not currency codes [20:41:02] been meaning to make a card for that [20:41:02] er [20:41:10] Yeah, that seems Bad. [20:41:25] Though, it might be useful to see when those were inserted. [20:41:38] I don't care so much if they were all last year. [20:41:45] at least a few of them were in the recalc range [20:41:55] d'oh. [20:42:13] Cards for Big EN cleanup, sounds like. [20:42:16] um, which i've also been meaning to finish. I just did the stuff for last FY [20:42:29] At least for the spike into what's happening. [20:42:42] See anything else that looked strange in there, that I should know about? [20:42:58] nope, maybe a missing ew-yandex form [20:43:10] Missing form, eh? [20:43:30] 19816071:9478136236 [20:44:08] Ah, that might be a bit deceptive. [20:44:34] payment method: ew. [20:44:42] payment submethod: visa? [20:44:54] That's not going to work. [20:44:58] yeah, that's odd [20:45:25] supposedly referred from the IT landing page [20:45:28] That also implies that the fix to wipe session things is not as inclusive as it might have been. [20:45:38] ...they probably hit cc first, and backed out and changed their minds. [20:46:03] ahh, since they're both GC [20:46:09] Exactly. [20:46:10] it wouldn't have hit that reset [20:46:21] Want to add payment method switch to the list of things that trigger the wipe? [20:46:30] Or... add a card for big en cleanup. :) [20:46:34] sure, will do [20:58:19] ooh, hope I didn't cause problem #2047 with my anti-double-submit js! [21:10:10] ejegg: Okay, I guess we're going to try to run something through WP's QA servers. [21:10:19] Yeah? [21:10:35] via our own staging site? [21:10:37] ...which... I don't know off the top of my head how to switch to. So, that's cool. [21:12:54] Yeah, that account setup doc is a little vague on live vs test [21:12:56] * K4-713 headscratches [21:13:37] 'URL to post Live and Test transaction' - https://trx3.blah/blah.dll or https://trx9.blah/blah.dll [21:14:33] then: PT front-end Test URL (View Payment Transactions ONLY) - https://admin1.blah/blah/Test/, with a 'Front End' username and password [21:14:42] what? [21:16:37] It looks like the URL doesn't change, but mwalker has helpfully built in a Test array key in the account info. [21:16:40] Wonder what that does. [21:16:44] oh, nice! [21:17:41] Just looks like it sets a test param in the XML. [21:18:02] oh, simple as that, huh? [21:18:13] It can't be. [21:18:16] I disbelieve. [21:45:29] wow, I made my drush commands really user-unfriendly [21:45:45] hey ccogdill [21:45:55] hey atgo [21:45:57] do you know if the WP vs GC banner is running in Belgium right now? [21:46:02] I don’t think so [21:46:22] I never got notified of it [21:46:30] and I’d be surprised if they didn’t tell *either* of us [21:46:43] are you seeing new WP donations? [21:46:47] ok.. could you log into the console really quickly and check to see if there were transactions? we are looking at shutting it down for a bit on the server side to test something [21:46:52] sure [21:47:01] thank you [21:48:05] atgo all I see today is one attempt from Megan [21:48:18] cool thank you [21:49:11] no problem [21:51:33] hey atgo is this work affecting civi too? [21:51:38] nope :) [21:51:54] katie's trying to wrangle WP [21:51:59] oookay. civi’s being a diva, just wanted to make sure I shouldn’t give up [21:52:00] thanks! [22:09:40] !log payments settings updates [22:09:47] Logged the message, Master [22:09:50] * K4-713 shakes head [22:12:59] Invalid merchant account number [22:13:07] Super thumbs up. [22:13:09] well then [22:13:35] glad i didn't bet anything on it really being that simple! [22:13:52] Nobody would see my bet. [22:14:05] atgo: ^^ [22:14:22] I need a minute to undo my bad behavior, and then I'll send that email. [22:14:22] this means... no? [22:14:37] This means absolutely nothing changed as far as we can see. [22:14:43] Same. Exact. Error. [22:14:45] Same place. [22:15:51] !log undo last payments settings change [22:15:57] Logged the message, Master [23:21:34] ejegg: I'll do the summary of the CN deploy in about an hour and a half, + the CN admins list, eh? [23:21:47] AndyRussG: awesome, thanks! [23:21:48] * AndyRussG waves at wm-fundraising [23:21:57] np, likewise... :)