[18:57:50] (CR) Mwalker: [C: 2 V: 2] Fix Amazon breakage [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/96290 (owner: Adamw) [19:55:59] (PS1) Adamw: (FR #1183) Allow arbitrarily negative refunds [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/96572 [19:59:01] (PS1) Adamw: some testing data [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/96574 [20:11:32] @rss-on [20:11:32] Rss feed has been enabled on channel [20:11:36] @rss-on [20:11:36] Rss feed is enabled on this channel [20:13:39] False [20:13:52] False [20:14:21] eh? [20:14:29] @rss+ mingle https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/feeds/gUz0EGVfIspv3gfHFQ9Qcg.atom [20:14:29] Item was inserted to feed [20:14:35] False [20:15:10] [mingle] Tech Task #1185 Recurring Amazon are not recurring changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1185?version=10 [20:15:11] [mingle] Tech Task #1183 Some paypal refunds come in with negative amounts changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1183?version=6 [20:15:11] [mingle] Tech Task #1179 Security update to Civi: -> 4.2.13 changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1179?version=7 [20:15:12] [mingle] Tech Task #1113 Spike: Broken Thank-you pages changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1113?version=15 [20:15:13] [mingle] Tech Task #1030 Globalcollect listener we can actually use! changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1030?version=14 [20:15:14] [mingle] Tech Task #988 thank_you (email) is missing base translations when locales are present: e.g. "es" and "zh" changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/988?version=24 [20:15:15] [mingle] Tech Task #917 DB transaction for queue2civicrm changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/917?version=27 [20:15:15] [mingle] Tech Task #662 VERIFY FIXED: investigate BannerRandom vs RecordImpression die-off changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/662?version=18 [20:15:16] [mingle] Tech Task #770 Spike: which fields are required when creating a Civi contact? changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/770?version=17 [20:15:23] False [20:15:24] False [20:15:25] False [20:15:26] Hehe-- just in time for the flood [20:15:26] False [20:15:27] False [20:15:27] False [20:15:28] False [20:15:29] False [20:15:30] False [20:15:31] False [20:15:31] [mingle] Tech Task #917 DB transaction for queue2civicrm changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/917?version=27 [20:15:32] WTF is False... [20:15:32] [mingle] Tech Task #770 Spike: which fields are required when creating a Civi contact? changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/770?version=17 [20:15:33] [mingle] Tech Task #662 VERIFY FIXED: investigate BannerRandom vs RecordImpression die-off changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/662?version=18 [20:15:34] [mingle] Tech Task #1000 Recording GC DD Recurring Success when Payment Not Successful changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1000?version=26 [20:17:07] [mingle] Tech Task #1030 Globalcollect listener we can actually use! changed by Adam Wight: Planning - Sprint changed from #1191 Sprint 36 - Nearly Go Time to #1137 Sprint 34 - End of Octob... https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1030?version=15 [20:17:22] False [20:18:40] (PS2) Adamw: WIP GlobalCollect listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/94086 [20:21:21] False [20:21:22] False [20:21:23] False [20:21:24] False [20:21:25] [mingle] Tech Task #1144 Rejected message queue overview changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1144?version=4 [20:21:25] [mingle] Tech Task #807 recurring globalcollect does not preserve gateway account name changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/807?version=15 [20:21:26] [mingle] Tech Task #632 Recurring code has some local vs UTC time errors changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/632?version=16 [20:21:27] [mingle] Tech Task #631 globalcollect audit script has local vs UTC time errors changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/631?version=20 [20:21:39] False [20:21:40] False [20:21:40] False [20:21:41] False [20:21:42] [mingle] Tech Task #1144 Rejected message queue overview changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1144?version=4 [20:21:43] [mingle] Tech Task #807 recurring globalcollect does not preserve gateway account name changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/807?version=15 [20:21:44] [mingle] Tech Task #632 Recurring code has some local vs UTC time errors changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/632?version=16 [20:21:44] [mingle] Tech Task #631 globalcollect audit script has local vs UTC time errors changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/631?version=20 [20:21:56] False [20:21:57] False [20:21:58] [mingle] Tech Task #1030 Globalcollect listener we can actually use! changed by Katie: Planning - Sprint changed from #1137 Sprint 34 - End of October to #1191 Sprint 36 - Nearly Go Time https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1030?version=16 [20:21:59] [mingle] Tech Task #961 CentralNotice: add Device filtering to Allocations and GlobalAllocations changed by Katie: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/961?version=15 [20:22:13] False [20:22:15] [mingle] Tech Task #961 CentralNotice: add Device filtering to Allocations and GlobalAllocations changed by Katie: Owner changed from Adam Wight to (not set),Planning - Sprint changed from #1137 Sprint 34 - End o... https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/961?version=15 [21:21:17] hey folks [21:22:03] quick recap: /wiki/Special:HideBanners?duration=1209600&category=fundraising is by far (50x) the most-requested URL on mobile right now [21:22:27] it's uncacheable and it contributes to -or, very likely is the cause of- an overload of varnish caches [21:22:46] there's tons of /wiki/Special:RecordImpression?result=hide&reason=empty&country=DE&uselang=de&project=wikipedia&db=dewiki&bucket=0&anonymous=true&device=android for all languages/wikis/devices too [21:23:17] site issues started approximately 08:20 UTC [21:23:56] paravoid: the RecordImpression call should be cacheable-- we're returning an empty file for everything... [21:24:00] yes, that is cacheable [21:24:19] HideBanners is trickier... why it is being called so often seems like the big question [21:24:36] HideBanners will be called anytime someone clicks a close button [21:24:42] are we running a mobile campaign right now? [21:24:45] * mwalker looks [21:24:45] Isn't this supposed to just be people... yes, clicking a close button. [21:24:46] on one of the european boxes, hidebanners is 4700 req/min, / is 1700, /wiki/Main_Page is 1000 [21:24:56] so maybe hiding is not working, and everyone has to click the close box? [21:25:04] Nope. [21:25:18] well, works for you, but... [21:25:20] We haven't run anything for mobile since... [21:25:38] November 6th, says meganhernandez. [21:25:43] sorry, cross-threaded that grokking [21:26:12] ok; paravoid; so this is all definitely for the mobile domains we're seeing this traffic? [21:26:49] let me cross check that to be sure [21:27:22] 2016 RxURL c /wiki/Special:HideBanners?duration=1209600&category=fundraising [21:27:25] 2016 RxProtocol c HTTP/1.1 [21:27:28] 2016 RxHeader c Host: en.m.wikipedia.org [21:27:30] 2016 RxHeader c Connection: keep-alive [21:27:33] 2016 RxHeader c Accept: image/webp,*/*;q=0.8 [21:27:35] 2016 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36 [21:27:39] 2016 RxHeader c Referer: http://de.wikipedia.org/wiki/Cascading_Style_Sheets [21:27:41] uhm [21:27:42] what on earth? [21:27:44] referer de.wikipedia.org, but host: en.m? [21:27:50] hmmm [21:28:04] I was about to ask if there was any chance desktop traffic is being sent to the mobile gateway... [21:28:10] * mwalker looks to see that desktop domains are not using the mobile domains for hiding and what not [21:28:12] it gets better [21:28:13] oh! [21:28:14] 119 RxHeader c Host: en.m.wikinews.org [21:28:14] 119 RxHeader c Referer: http://de.wikipedia.org/wiki/Kernkraftwerk_Olkiluoto [21:28:30] actually; any time someone closes a banner; it hits all domains [21:28:38] because we wanted cross wiki hiding [21:28:44] wait, what? :) [21:28:58] How new could that be, though? [21:29:12] just like centralauth; we request a cookie from every domain to 'hide' a banner [21:29:15] I thought we froze everything interesting weeks ago. [21:29:20] we did; this isn't new [21:29:32] you're requesting en.m.wikisource.org when someone clicks to close a banner on de.wikipedia? [21:29:40] yes [21:30:29] we can immediately disable this if you want to test to see if this is the problem [21:30:35] * mwalker looks up the config variable [21:30:57] uhm, I must be missing something [21:31:01] we have hundreds of wikis [21:31:25] we use one wiki from each group [21:31:33] see $wgNoticeHideUrls in CommonsSettings.php [21:32:35] each of those URLs gets called; as well as the mobile equivalent [21:32:46] *mobile equiv is automatically calculated [21:33:00] has anything changed since yesterday, e.g. are you running a campaign? [21:33:15] meganhernandez: ping [21:33:23] We're basically always running a campaign these days. [21:33:38] Something at 13% has been normal since the beginning of October. [21:33:44] but didn't we start running at 100% or something today? [21:33:52] Tomorrow, I thought. [21:33:55] And Friday. [21:34:26] paravoid: I don't see any immediate suspects in the CentralNotice logs [21:34:31] https://meta.wikimedia.org/wiki/Special:CentralNoticeLogs [21:34:47] why is this request uncacheable? [21:35:22] hello, need something? [21:36:19] meganhernandez: have you changed anything with respect to banners/campaigns? [21:36:31] a site issue, seemingly related to CentralNotice has been ongoing since 0820 [21:36:35] Jeff_Green says there's a not-fundraising banner running now. [21:36:45] Could be misrepresenting itself. [21:36:47] https://meta.wikimedia.org/wiki/Special:CentralNotice [21:36:55] paravoid: it is cacheable; expect for cases where the user does not already have a login session on the wiki [21:36:58] "Trademark policy discussion" [21:37:02] i change the banners everyday, but just colors ,text & ask amounts lately [21:37:15] paravoid: and by that I mean; if a user is logged in on any wiki; we cannot cache the request [21:37:24] I'm a noob to that tool, but it looks like that's up everywhere and went up in the past 24H [21:37:48] mwalker: nope, that's not it [21:39:08] mwalker: I see cache-control: no-cache even for requests that have no session token [21:39:45] also, it's 91% dewiki as far as I can see [21:40:31] damn; ok for Special:HideBanners you're right -- we, for some reason, do not set a CC header [21:41:04] 77% de.wikipedia.org Referers for the whole globe, not just esams [21:42:59] folks, this is causing a mobile site outage in europe, so I'm going to merge https://gerrit.wikimedia.org/r/96641 for now [21:43:06] and until you fix the cache-control headers [21:43:58] nods [21:44:39] writing patch now; but push that and let me know if it helps [21:44:41] meganhernandez: Did you do anything new in DE this morning? [21:44:48] no [21:44:53] but the chapter is running fundraising banners [21:45:17] Oh right: Duh, that's a chapter country. [21:45:21] Like, one of three remaining. [21:45:26] 2 remaining [21:45:28] DE & CH [21:45:45] heh, and I thought of that [21:45:56] but I was thinking that you'd have risen it already if that was the case :) [21:46:29] I didn't see any changes from any DE targeted banners at first glance [21:47:14] they had one that started at 19:00 UTC yesterday [21:47:37] Yeah, what I could see looked relatively reasonable to me, but maybe they just upped their percentages like crazy or something. [21:48:50] (PS1) Mwalker: Cache Hide Requests [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/96643 [21:49:32] awight: ^ if you would please [21:49:56] mediawiki has a special function to set max-age iirc [21:50:01] setSquidMaxage or something [21:50:24] mwalker: k [21:50:25] there is a reason I don't typically use that function... looking into why right now [21:50:43] dunno, not a mediawiki developer, don't trust me [21:51:12] ahhh; it's the $this->getOutput()->disable(); [21:51:12] wfResetOutputBuffers(); [21:51:15] bit above it [21:51:17] mwalker: how is it OK to cache the set-cookies for a day? [21:51:23] how about an hour? [21:51:31] hey guys i have to run to a meeting, need anything from me? [21:51:40] nope, thanks! [21:52:09] awight: ah; good call [21:52:11] * mwalker hates cookies [21:52:20] mwalker: sorry, i think it was fine [21:52:27] this is the long expiration... duh [21:52:32] well; long and short [21:52:39] but that's still two weeks or a year [21:52:42] cookies right now expire on Dec 4th [21:52:45] 1649 ObjHeader c Set-Cookie: centralnotice_hide_fundraising=hide; expires=Wed, 04-Dec-2013 21:51:06 GMT; path=/; domain=.wikisource.org [21:52:48] fwiw [21:53:01] cool, +/- 1 day is fine [21:53:07] so I don't think a day would matter [21:53:09] right [21:55:03] (PS1) Mwalker: Do not mobilify hide links [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/96646 [21:55:21] mwalker: I'm also trying to grok max-age=0, this is a recommendation to the browser cache, meaning never refresh? [21:55:40] awight: we can also push ^ out as well -- I had a more comprehensive patch waiting in the wings for mobile fixes; but this is useful now [21:55:48] awight: correct [21:56:16] how will that behave when the short expiry expires, and they ask to Hide again? [21:56:35] won't that prevent a browser request to the URL... [21:56:36] browser does not have cached content; a request goes up to the server [21:56:54] no max-age=0 prevents browser and midstream proxies from caching [21:56:59] so it'll always request new contnet [21:57:15] ah ok that's the opposite of what I thought [21:58:44] (CR) Adamw: [C: 2 V: 2] Cache Hide Requests [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/96643 (owner: Mwalker) [21:59:02] paravoid: now that the resourceloader 5 minutes of doom has expired; how are we looking? [21:59:25] (CR) Adamw: [C: 2 V: 2] Do not mobilify hide links [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/96646 (owner: Mwalker) [21:59:46] I'm still investigating the situation, please do not reenable that feature until you get an ack from me [22:00:10] *nods* do you want me to push the two patches I just wrote [22:00:18] or just hold off completely on new changes [22:00:23] these will be noops aiui [22:00:27] so I don't mind either way [22:01:17] *nods* I'm going to wait then until you finish looking into it in case I have more patches to write [22:01:27] I missed what color bandaid we just put on-- CN is disabled? [22:01:35] no, cross wiki hiding is disabled [22:01:39] oic [22:01:41] cool [22:01:44] good [22:01:49] down with that feature ;) [22:03:02] and I guess those .m. Special:BannerRandom / Special:RecordImpression are also WMDE? [22:03:47] Special:BannerRandom and Special:RecordImpression are just always there [22:03:57] it's how we deliver banners [22:04:08] so even if there is null content they still get called on all content namespace pages [22:05:29] Special:BannerRandom gets cached for 15 minutes for logged out users [22:05:39] Special:RecordImpression gets cached for a day [23:09:39] (PS3) Adamw: GlobalCollect listener [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/94086 [23:10:28] False [23:10:29] [mingle] Tech Task #1030 Globalcollect listener we can actually use! changed by Adam Wight: Tech Status changed from In Development to Pending Code Review,Moved to Pending Code Review on se... https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1030?version=17 [23:11:17] [mingle] Tech Task #807 recurring globalcollect does not preserve gateway account name changed by Adam Wight: Tech Status changed from Backlog to Dev Ready,Moved to Dev Ready on set to 2013/11/20 https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/807?version=16 [23:11:30] False [23:11:31] False [23:11:32] [mingle] Tech Task #807 recurring globalcollect does not preserve gateway account name changed by Adam Wight: Assigned To set to Adam Wight,Tech Status changed from Dev Ready to In Development,Moved to In De... https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/807?version=17 [23:12:00] @rss-list [23:12:05] lots of false... [23:12:05] @rss [23:12:11] Eff. What is it. [23:13:27] I guess there's no way to list. [23:21:49] marktraceur: thanks, I was trying to figure that out earlier. [23:21:57] weirder yes, what the hell is this? [23:21:58] @help [23:21:59] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.2.0 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [23:22:01] oh [23:22:04] @info [23:22:04] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-fundraising.htm [23:22:06] that one [23:22:21] wouldn't you want that to show like rss settings [23:23:39] oh cool, the code is modular [23:30:11] damn. The trail went cold somewhere in here: https://github.com/benapetr/wikimedia-bot/blob/e8610addf23724bb2dce44fbd3048181fde1ce97/plugins/wmib_rafeed/wmib_rafeed/Module.cs [23:30:30] I think Extension_DumpHtml is supposed to fire on that /dump URL above. [23:30:58] and, I don't think our GetConfig is loading, for that module at least. [23:34:09] marktraceur: oh. So, I think the rafeed module only pulls from the textfile config, rather than the database. [23:34:31] Ah. [23:34:32] u happen to know where that file is / who enabled wm-bot here? [23:34:36] Hah, no [23:34:41] lol ok [23:34:47] It's petan's baby, but I dunno who enabled it here [23:35:01] cool I will polish a trout, meanwhile.