[00:37:39] (PS2) AndyRussG: WIP: Test fixtures: expand to cover more scenarios [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188687 [00:38:46] (CR) jenkins-bot: [V: -1] WIP: Test fixtures: expand to cover more scenarios [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188687 (owner: AndyRussG) [00:39:39] (CR) AndyRussG: "The main thing visible in this WIP so far is just the new structure for fixtures data" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/188687 (owner: AndyRussG) [01:34:16] hey pizzzacat1 K4-713 i think i'm going to head out soonish [01:34:22] woot. [01:37:19] oh! ok! [01:37:25] are we all going together? [07:24:35] (PS11) Awight: Preload JS comes with no baggage [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/181244 [07:27:01] (PS3) Awight: Document BannerRenderer [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/182897 [07:27:07] (PS12) Awight: Preload JS comes with no baggage [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/181244 [07:49:54] Wikimedia-Fundraising: Tracking: Fundraising Tech Staging - https://phabricator.wikimedia.org/T88657#1017254 (awight) NEW [16:36:25] Wikimedia-Fundraising-CiviCRM, Labs, Wikimedia-Fundraising: Create new labs project: fundraising-integration - https://phabricator.wikimedia.org/T88599#1018187 (awight) Please hold one moment... @hashar is suggesting we create this instance as integration-civicrm01.eqiad.wmflabs, possibly under the integratio... [17:19:38] meganhernandez, jessicarobell, ccogdill: in case you missed it all the sites are down at the moment. ops are looking into it [17:20:14] yowza [17:32:15] whoaoaoaoa [17:33:04] crazy [17:33:12] olala. Thanks for the heads up the-wub. I was;t aware. [17:33:45] * AndyRussG waves at jessicarobell the-wub ejegg awight [17:33:47] Thanks the-wub [17:34:06] hi AndyRussG [17:34:14] hey AndyRussG [17:35:30] Hi K4-713, atgo [17:35:35] Seen the meltdown? [17:35:36] hello! [17:35:40] Er, no? [17:35:43] :) [17:35:49] I mean, I see something about... donate? [17:35:50] All sites are down :( [17:35:59] ah [17:36:14] Yeah, it's really odd for donate to go away all by itself. [17:36:14] Yeah it's not us, thankfully [17:36:20] a switch went down in the datacenter I think [17:36:27] oh, f. [17:36:56] yeah donate.wikimedia.org is down, but payments.wmo still seems to be up [17:36:58] Payments is fine. :p [17:37:13] Typical. [17:37:27] Beta cluster is also alive... [17:37:29] Oh, but wait. [17:38:06] I guess they didn't get the link to payments in that problem page. [17:38:09] are campaigns down? [17:38:18] Wikipedia is down. [17:38:22] So: Sure. [17:38:42] ah i mean do we need to turn stuff off on our side [17:38:55] Those buttons, are also down. [17:39:00] sorry you probably already figured this out, im on calls [17:39:23] Oh, actually: Meta seems okay. [17:39:45] enwiki is back actually! [17:40:02] Donate is still out. [17:40:07] all still down for me :( [17:40:13] K4-713: enwiki is browsable but not editable [17:40:28] Probably trying to show banners, though. [17:40:37] the-wub: Can you not get to meta? [17:40:49] K4-713: no [17:41:02] Hard reload? [17:41:10] Oh, ha. I can't log in there. [17:41:13] I heard the words "the cable was loose" ;) [17:41:13] nvm. [17:41:45] awight: that was the speaker system on 6th [17:41:51] Or does it share the same cable? [17:41:57] That wasn't a cable. [17:42:02] :D [17:42:09] Quick, tell ops! [17:42:27] the encyclopedia is not delivering my karaoke! [17:42:29] "Someone brought down enwiki by using it for karaoke" [17:42:41] SORRY. Geeze. :p [17:42:43] it was really screechy before all hell broke loose [17:42:46] (bahaha) [17:43:35] it made people sing about lists of lists of lists to the tune of the Beatly Boys [17:43:55] Also: I totally thought they were going to put a donate link on all the outage pages. [17:43:59] I am not seeing that. [17:44:28] ...sure seemed critical in December. [17:44:35] * K4-713 shrugs [17:44:38] I saw it on some for a brief while. think it depends on the type of error. [17:44:51] orly [17:45:33] https://www.dropbox.com/s/98kf0labe9ithm3/Screenshot%202015-02-05%2017.45.19.png?dl=0 [17:45:45] Doesn't look like anybody actually clicked on it. [17:45:45] ah yeah I remember that goal now... is there a Phab task for that BTW? [17:45:50] BSC is so disorganized... I had them cancel baby insurance in December, and meanwhile they charged me twice already. Apparently they thought I said "cancel her $4/mo dental insurance and leave the rest to be duplicated by group plan" [17:45:54] riiiight. [17:46:04] actually it just links to https://wikimediafoundation.org/wiki/Fundraising. we should change that. [17:46:11] * K4-713 facepalms [17:46:15] That's not what I gave them. [17:46:29] wait. but they can't donate if donate is down. [17:46:44] n e way [17:46:49] Yes they can... if you assume an annoying number of variables. [17:46:55] Which is what we decided to do. [17:47:03] Straight link to US paypal. [17:47:07] $5 usd. [17:47:17] ah right, I even made that URL [17:47:24] >o< [17:47:29] almost all of the errors I've seen today are this though, with no donate link https://www.dropbox.com/s/4ups7c8haylrmdr/Screenshot%202015-02-05%2017.46.31.png?dl=0 [17:47:47] yep [17:47:50] I didn't get anything else. [17:47:59] the-wub: yeah same here [17:48:15] oooh, maybe over now. [17:48:51] yeah, things are back for me [17:48:55] The bigger goal for the error pages was to get country-specific links in there, right? [17:49:04] Nah, we can't do that. [17:49:17] Presumably, everything is out if you're seeing that page. [17:49:22] So, there are no "smart" options. [17:49:40] the-wub: wasn't that something you mentioned ^ ? apologies if I should have... er... notes... [17:50:05] also things seem back online here, was able to go to an edit page, too [17:50:06] could still do something in javascript [17:50:17] based on the top-level country code in the url [17:50:40] K4-713: any need for me to pull campaigns now I can get to CN admin? or are we okay? [17:50:44] ejegg: you'd need at least geoip lookup... Or I guess you could grab it from the cookie, actually [17:50:46] I think we're fine. [17:51:11] AndyRussG: don't think it's something I'd looked at [17:51:18] ...except I can't log in to meta. [17:51:23] Hmmm, who was mentioning that to me [17:51:24] ? [17:52:24] we should include a video ad in banners, about a site outage happens to someone cos they didn't donate [17:52:29] ejegg: Uh, country code in the URL? [17:52:49] derp, nvm [17:52:53] :) [17:53:07] ...failover server in a bunker which only exists to gather country and money :) [17:53:32] Please donate $5 or the encyclopedia gets it... [17:53:41] heheh [17:54:43] * AndyRussG turns back to work... [17:55:35] On the topic of outages, anyone feel like helping me try to deploy CN updates again soon? [17:55:43] * K4-713 giggles [17:55:46] huh, I think banners aren't showing up https://en.wikipedia.org/?banner=B15_0202_enHU_dsk_lg_ask_ctrl&force=1 [17:56:07] the-wub: Total lack of moving things on payments would seem to support that premise. [17:57:15] meta isn't returning a banner on that request.. [17:58:23] K4-713: the-wub: maybe it's a caching thing? [17:58:54] Always a fine choice to explain... anything. [17:59:06] Ghosts, total lack of ghosts... [17:59:27] * K4-713 sets a clock for 15 minutes [17:59:31] This banner was previewable: http://meta.wikimedia.org/w/index.php?title=%E8%A1%9D%E7%AA%81%E7%9A%84%E7%B6%AD%E5%9F%BA%E7%99%BE%E7%A7%91%E7%90%86%E5%BF%B5&banner=B15_0205_enHU_mob_sm_tw1&uselang=en&force=1 [18:00:30] This one was not, https://meta.wikimedia.org/w/index.php?title=Wikimedia_press_releases/Facts_%26_Figures/Wikisource&banner=B15_0204_heIL_dsk_lg_18&uselang=en&force=1 [18:00:49] Yeah... weird [18:01:59] awight|afk: ^ though that one does send banner content, maybe that's due to in-banner js? [18:03:01] hi there [18:03:06] i have heard of the meltdown but not seen [18:03:13] Not a lot to see. [18:03:16] It happens. [18:03:28] what happened? [18:03:37] Wikipedia went down. [18:03:40] oh. that. [18:03:48] I think there was an issue with MediaWiki namespace messages not loading. which would have affected banners. seems to be resolving now [18:03:48] K4-713: awight|afk: the-wub: atgo: Here's a wp with a live banner that's working: http://kk.wikipedia.org/wiki/II_%D0%95%D0%BB%D0%B8%D0%B7%D0%B0%D0%B2%D0%B5%D1%82%D0%B0 [18:03:50] And, you know, basically everything else. [18:04:03] Except for payments. [18:04:16] invincible! [18:04:17] Because we're in a special hole in the ground, with a special connection. [18:04:22] ...to the hole. [18:04:24] yaaay [18:04:51] hmm, I was getting connection timeouts to payments while the rest were down [18:04:56] There does seem to be some activity on payments coming from banners again. [18:05:14] ejegg: Really? I didn't get any of those. Just to donate. [18:05:34] ah, must have been a fluke then [18:05:48] Where you getting connection timeouts to payments? [18:05:52] In-browser? [18:06:10] None of our alerts got tripped for that. [18:07:17] And, if you just go to payments.wikimedia.org with no params, you always get a whitescreen. [18:07:36] Am I... getting anywhere? [18:07:39] * ejegg checks hosts file [18:07:45] d'oh [18:07:53] K4-713: ejegg: the-wub: atgo: awight|afk: the above link to a working banner was a lie. It's not a CentralNotice banner, it's some other weird kind of banner [18:08:20] You mean... somebody is running a *notice*? [18:09:10] oh yeah AndyRussG, that looks like a sitenotice [18:09:24] I think our banners are fine now. e.g. https://en.wikipedia.org/?banner=B15_0202_enIL_dsk_lg&force=1&country=IL [18:10:16] * K4-713 ducks reflexively [18:10:26] I forgot how big those were. [18:10:32] the-wub: thanks! good to see [18:10:43] don't blame me! :P [18:10:47] Oh, I wasn't. [18:10:59] K4-713: cute alert [18:11:02] K4-713: ehh, 2 out of 3 compys are getting the expected whitescreen, just this little netbook is still timing out [18:11:22] https? [18:12:01] if anyone's interested, I found the phabricator task for better error pages https://phabricator.wikimedia.org/T76560 [18:12:19] oh, that hsts deploy did away with the http->s redirect, huh? [18:12:32] I don't know if that ever worked for me properly. [18:12:43] http just sort of dumped you on your head the whole time. [18:13:02] ah, https works fine. sorry for the noise [18:13:07] No worries. [18:13:15] This is precisely the kind of thing we should document. [18:13:31] Also, precisely the kind of thing that I would not remember to document. [18:13:40] Useful. [18:14:41] wikitech fr page? [18:15:31] ejegg: I've been strongly encouraged to land these things on mediawiki.org. [18:15:51] I expect if we start doing that, other people will strongly encourage us to land them on wikitech. [18:16:08] Got any flipping coins handy? [18:16:48] heh. mw.o it is [20:50:51] hey ejegg [20:51:08] hi atgo [20:52:08] do you want to jump in on PPena's call with worldpay about the contrib id in the transaction statement? [20:52:24] oh sure [20:52:39] videochat or conf call? [21:34:27] ejegg conf ccall [21:35:17] hi PPena [21:44:08] Analytics, Wikimedia-Fundraising: Public dashboards for CentralNotice and Fundraising - https://phabricator.wikimedia.org/T88744#1019258 (awight) NEW [21:53:49] Wikibugs, Wikimedia-Fundraising: Wikibugs bot is skipping many notifications - https://phabricator.wikimedia.org/T88747#1019330 (awight) NEW [21:59:48] awight: hm, odd. It also didn't appear on #wikimedia-dev, which suggests it's a deeper issue [21:59:57] hi! [22:00:35] valhallasw`cloud: interesting--I saw similar squirrely behavior from gerritbot, but I guess that's in the other direction. New and merged patches were not being announced on the associated task. [22:02:27] awight: ok, I can reproduce it and I get an error in the logs. That's good (sort of ;-)) [22:02:52] awesome, that does sound like most of the solution :) [22:16:09] Wikibugs, Wikimedia-Fundraising: Wikibugs bot is skipping many notifications - https://phabricator.wikimedia.org/T88747#1019435 (valhallasw) §! [22:23:27] awight: okay, now the issue is at least reported in #wikimedia-dev. Next question is why it isn't reported here [22:23:55] even though it's clearly in Wikimedia-Fundraising :| [22:24:09] I was hoping it was something about the way project lists are parsed... It seems like some tasks always cause notifications, and others never do. [22:24:16] We tag tasks with many, many projects... [22:25:09] awight: well, the issue /was/ the sprints because of the paragraph symbol. Now there's another issue. [22:25:44] hah! [22:25:49] UTF-7 [22:26:38] Wikibugs, Wikimedia-Fundraising: Wikibugs bot is skipping many notifications - https://phabricator.wikimedia.org/T88747#1019498 (valhallasw) Unfortunately, this still doesn't fix the issue, as the bug still doesn't show up in the correct channel. The bug itself reports Sprints: § Fundraising Tech Backlog (A... [22:27:33] awight: the gateway reconciliation report is still timing out for PPena [22:27:35] awight: one quick hack would be to add "§ Fundraising .*" to go to this channel [22:28:25] PPena: what URL are you visiting? [22:28:40] The one I sent should take a few seconds to load, but that's it. [22:28:48] valhallasw`cloud: hehehe [22:29:45] PPena: ok, I see you're going to the correct URL [22:29:54] Are you in the office? [22:30:21] atgo: would you mind trying that link, too? [22:30:23] awight: thanks 4 doing those EP e-mails! yeah that Gather thing looks really interesting [22:31:05] AndyRussG: and it's a good time to jump in and push the tiller a bit :) [22:31:15] fantastic! [22:31:34] do i hit 'preview report' awight? [22:31:37] AndyRussG: btw, do you have a good dump of SWalling's final will + testament ;)? like road maps for what he was planning to do? [22:32:13] I was thinking of doing like a draft for a requirement-gathering RFC, also to clarify that it isn't an RFC about starting from scratch but where to go assuming that what was done so far is a good starting point [22:32:19] Yeah got that somehwere [22:32:24] Sort of anyway [22:32:37] PPena: can you visit that URL again? [22:32:51] PPena: I think this is related to logging in as you hit the URL. [22:33:24] awight: Here's one thing https://www.mediawiki.org/wiki/Editor_campaigns [22:33:46] More: https://trello.com/b/8wr09hXx/editor-campaigns-groups [22:34:53] A bunch of stuff is here too: https://www.mediawiki.org/w/index.php?title=Special%3APrefixIndex&prefix=Editor_campaigns&namespace=0 [22:35:07] (though some of it is from the original old Campaigns extension) [22:36:47] awight: There was also some synergy with stuff the Multimedia team wanted to do, I think even YuviPanda had done some preliminary code or at least technical design [22:44:43] awight: ready for more gateway controller talk when you are [22:55:12] § Fundraising Tech Backlog, § Fundraising Sprint Devo: Gateway Reconciliation report times out - https://phabricator.wikimedia.org/T87834#1019587 (valhallasw) And sorry again for spamming your bug, but I think the IRC report should work now :-) [22:55:19] \o/ [22:58:54] PPena: sorry, I think you missed the astonishing punchline. U there? [22:59:04] Try logging into civi, *then* going to that URL [23:00:04] AndyRussG: rad, thanks I'll send those links to Anna also [23:00:30] Definitely trying to get the higher-level view, to see what other components he was planning aside from Campaigns. [23:00:43] awight: cool thanks! [23:01:05] § Fundraising Tech Backlog, § Fundraising Sprint Devo: Gateway Reconciliation report times out - https://phabricator.wikimedia.org/T87834#1019603 (awight) @PPena, I'm pretty sure this is a ridiculous bug having to do with login. I'm updating the task description with what I believe are the steps to work arou... [23:01:41] The Phab task there might also be nice [23:01:57] I don't see Yuvi's proposal right off the bat, I can look more [23:02:01] Thanks! [23:02:33] AndyRussG: or on second thought, maybe you can send the email with whatever cookie crumbs SW left for us? [23:02:51] awight: sure! [23:03:11] § Fundraising Tech Backlog, § Fundraising Sprint Devo: Gateway Reconciliation report times out - https://phabricator.wikimedia.org/T87834#1019614 (awight) [23:03:57] ejegg: good time 4 a call? [23:04:58] sure [23:11:03] awight: well, enjoy the bugspam! thanks for the report :-) [23:18:46] awight thats what i did [23:18:53] so I got to the url [23:19:10] and got to "run report", and tries 'saving' the second time [23:19:16] and THEN it gave me the error [23:19:30] awight sorry I should have been more specific) [23:37:17] PPena: are u here? [23:37:29] PPena: I want to walk through w/ you to see where this is crashing. [23:37:56] awight yep upstairs [23:38:07] be right there [23:53:37] § Fundraising Tech Backlog, § Fundraising Sprint Devo: Set CiviCRM timeout to 30 min - https://phabricator.wikimedia.org/T88759#1019749 (awight) NEW [23:55:48] § Fundraising Tech Backlog, § Fundraising Sprint Devo: Set CiviCRM timeout to 30 min - https://phabricator.wikimedia.org/T88759#1019760 (awight) a:Jgreen