[20:38:25] tgr: o/ could https://github.com/wikimedia/mediawiki/commit/d511626236f9eeeb6a55c97c3c0d74d30150dc4f#diff-21f6ec59fbbe16aa314597b2ae23eed9 have caused a revival of T165115? [20:38:26] T165115: action=mobileview sections delivering a div of class "mw-parser-output" without a matching closing div - https://phabricator.wikimedia.org/T165115 [20:38:36] ...because that's happening again [20:39:24] textextracts broken too [20:39:39] https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&formatversion=2&exsentences=5&exintro=true&titles=Cragside [20:45:50] cc jdlrobson ^ [20:46:12] Now I really wish we would have done the summary switchover, too. https://en.wikipedia.org/api/rest_v1/page/summary/2018_Winter_Olympics [20:52:21] mhurd: what's the impact for the iOS app? [20:55:02] bearND: broken extracts and same broken transforms as last time as seen on these screenshots: https://phabricator.wikimedia.org/T129717#2119336 - suspect same is true for android [20:55:50] on Android it still shows the link previews but they don't have any text, just pictures [20:56:40] mhurd: I think, at least for the pages I looked at, on iOS it skips the previews [20:57:37] mdholloway: we are still on wmf.17 as far as I know [20:58:03] or was there an extra train slot today? [20:58:22] tgr: hmm, yeah, i wasn't quite clear on what the current state of mw deployment is [20:58:55] there's some mediawiki activity today on https://wikitech.wikimedia.org/wiki/Server_Admin_Log but it doesn't look like a full train [20:59:29] the train was aborted yesterday, but seems like wmf.20 was rolled out successfully four hours ago [20:59:37] ah: 18:31 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.20 [21:00:51] in theory https://gerrit.wikimedia.org/r/#/c/399910/ should have fixed MF [21:01:22] and this is now an unwrap transform so caching should not be an issue [21:03:31] I can roll back the unwrapping patch on mwdebug, see if that fixes things [21:03:49] can you test that via MW apis? [21:04:04] as I doubt restbase honors X-Wikimedia-Debug [21:04:16] this doesn't involve restbase, just mobileview [21:04:26] yeah, we can test via mw api [21:05:08] TextExtracts will need to be fixed as well [21:08:32] at a glance TextExtracts doesn't really care about wrapping divs [21:14:06] uh, yeah, unwrap => true seems broken [21:18:22] tgr: possible to roll back? or is it an easy fix? [21:20:51] seems easy [21:21:43] the unwrap code assumes that is at the end of the document but some debug comments are added after it [21:21:53] I wonder why this didn't happen in vagrant [21:22:03] i guess we could just keep using the deprecated setWrapOutputClass( false ) in MF if we wanted to be super quick and dirty about it, but probably better to fix the unwrap code itself... [21:22:42] setWrapOutputClass would not work anymore [21:22:54] a couple more patches would have to be rolled back for that [21:23:10] ah, ok. [21:23:13] as the option was removed from the parser cache key generation logic [21:23:53] waiting for chad to finish deploying then I can text a fix [21:24:03] tgr: sounds good, thanks. [21:30:42] mdholloway: is there a bug number for this? [21:31:11] no new bug filed yet afaik // (mhurd coreyfloyd ?) [21:31:42] i can open one if no one else has yet [21:32:05] mhurd: would know… I did not file [21:32:10] @mdholloway no i haven't filed yet [21:34:23] @mdholloway should i file or do you got it? [21:34:32] mhurd: writing it up now [21:34:34] i'll cc you [21:34:44] @mdholloway ah cool thx! [21:37:08] tgr: mhurd: coreyfloyd: T186297 [21:37:17] er, T186927 [21:37:20] T186927: mw-parser-output divs leaking into mobileview output again - https://phabricator.wikimedia.org/T186927 [21:39:59] should we call it a UBN? [21:45:16] yeah [21:45:25] patch is on mwdebug1002 [21:46:00] aand, still doesn't work [21:46:09] I suppose TextExtracts is cached [21:50:28] seems to work in a console, at least [21:50:44] mdholloway: do you know how to break cache for textextracts / mf api? [21:51:41] i do not [21:56:50] tgr: the only people i can think of offhand who might know about that (and are likely to be around right now) are jdlrobson and MaxSem [22:04:33] huh? [22:05:41] tgr, bump ApiQueryExtracts::CACHE_VERSION [22:05:51] MaxSem: we're trying to debug T186927 and i understand there's some caching around the mobileview api that might mean that our fix might not show up right away [22:05:51] T186927: mw-parser-output divs leaking into mobileview output again - https://phabricator.wikimedia.org/T186927 [22:08:45] right, that fixed it [22:09:21] mdholloway: do you have a test url for the mf api? [22:09:46] MaxSem: do you know the ttl for that cache? [22:09:58] ParserCacheExpireTime apparently [22:10:19] $wg ;) [22:10:54] so I gues bump cache version as a security patch, drop it in a month? [22:11:13] yep, that looks better [22:12:04] for TextExtracts anyway [22:12:06] tgr: https://en.wikipedia.org/w/api.php?action=mobileview&format=json&page=Cragside§ions=all&prop=text|sections [22:12:08] same for mobileview, it has a cache key [22:13:09] tgr, commit for third parties? [22:13:34] bumped that too, seems like it did the trick [22:13:42] OK, I'll just make it into a normal commit then [22:14:04] tgr: mobileview.sections[0].text should not begin with '
' [22:14:27] yep, better [22:18:07] mdholloway: https://gerrit.wikimedia.org/r/#/q/topic:T186927+(status:open+OR+status:merged) [22:18:07] T186927: mw-parser-output divs leaking into mobileview output again - https://phabricator.wikimedia.org/T186927 [22:34:58] eh, probably not a good time to wait for CI [22:37:24] tgr: yeah, the MF tests are quite slow [22:44:45] mdholloway: do we need to purge MCS or RESTBase as well? [22:45:31] the fix should be live btw [22:45:35] tgr: checking now [22:48:06] mhurd: is the iOS app getting extracts from RB these days or still from MW API? [22:50:54] we may want to consider purging/regenerating /page/summary in RB as well /cc bearND Pchelolo [22:51:08] https://en.wikipedia.org/api/rest_v1/page/summary/Cragside is still giving bad extracts after a varnish purge [22:54:26] ah, looking better now [23:01:06] mdholloway: I seem to recall RB getting a few minutes expiration time after the problems with featured X vandalism [23:03:31] it actually seems to be taking a minute for the TextExtract fix to come through, even from MW API, though I'm not sure why that would be [23:03:33] e.g., https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&formatversion=2&exsentences=5&exintro=true&titles=2018_Winter_Paralympics [23:03:55] (i just purged that specific URL, and still not fixed) [23:04:03] same with Cragside though that's better now [23:08:37] weird [23:09:22] these are not cached in varnish, and the code change should have been immediate [23:09:29] I don't see any other caching layer [23:09:45] tgr: and now that's suddenly fixed [23:10:01] tgr: oh, those aren't cached even for GETs? [23:10:25] no, the MW API is not, there would be no way to invalidate it [23:10:37] you can request caching manually via &smaxage=... [23:13:53] so, do you think it is fixed for good? [23:14:02] i'm just clicking around in the Android app. every now and then i find an article with an extract that hasn't been regenerated, and it's coming from the MW API. when it finally comes through RB seems to update immediately fwiw [23:14:30] tgr: still finding faulty extracts: https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&formatversion=2&exsentences=5&exintro=true&titles=Seoul_Capital_Area [23:14:59] the good news is that the extracts are coming back, albeit gradually... [23:17:14] tgr: interesting... 2018 Winter Paralympics was edited right around the time the extract suddenly came back https://en.wikipedia.org/w/index.php?title=2018_Winter_Paralympics&action=history [23:18:15] not true for Cragside, though... so much for that theory [23:20:16] the cache version is still 1 on that box [23:20:24] did something go wrong with the deploy? [23:21:22] hmm :[ [23:23:21] oh duh [23:23:30] I forgot the submodule update didn't I [23:28:19] tgr: that's better! [23:28:30] sorry about that :/ [23:29:29] ha, no worries [23:29:56] as long as the underlying extracts are back we can probably wait for stuff to age out of RB IMO [23:30:08] there don't seem to be a large number of articles affected [23:34:37] tgr: thanks again for fixing this up for us. time for me to wander away from the computer. [23:34:39] have a good weekend! [23:34:57] you too [23:35:28] @mdholloway sorry i missed your question - ya we're using RB extracts [23:36:30] mhurd: that's ok! seems like RB is still catching up on a few summaries but is everything looking better overall? [23:37:40] huh, Crust_(geology) is semi-protected for some reason [23:37:48] very controversial apparently [23:38:45] @mdholloway ya the article i 1st noticed the issue on looks good! others too! [23:38:49] the android app seems in pretty good shape [23:39:25] @mdholloway @tgr thx so much for such quick fixes! :) [23:40:17] mhurd: awesome. tgr gets the credit for this one. i'm off, have a good weekend! [23:40:22] sorry for breeking it in the first place :) [23:40:57] @mdholloway thx you too! [23:41:34] @tgr hehe no worries! [23:44:02] Thanks tgr! [23:51:30] everything seems to be not on fire so I'm off to find some food [23:51:38] I'll see IRC pings