[00:02:11] 10Traffic, 10ArticlePlaceholder, 06Operations, 10Wikidata: Performance and caching considerations for article placeholders accesses - https://phabricator.wikimedia.org/T142944#2566366 (10hoo) >>! In T142944#2566353, @DaBPunkt wrote: >>>! In T142944#2560827, @hoo wrote: >> For this, we also desire to get th... [00:20:56] bblack: left another comment on https://gerrit.wikimedia.org/r/305419 -- just flagging it in case it's missed for my first -1 [00:21:36] bblack: my ping earlier was because I wanted to discuss a separate entirely different matter with you (re: lvs/pybal/router config) but I'll draft a task up tomorrow my morning hopefully [06:42:44] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#1323201 (10Nikerabbit) >>! In T100902#2566120, @Platonides wrote: > I think a new ULS version not relying on that should be released before t... [06:51:03] 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations, 13Patch-For-Review: ULS GeoIP should only use the Cookie - https://phabricator.wikimedia.org/T143270#2566808 (10Nemo_bis) [07:31:56] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2566816 (10Nemo_bis) If all else fails, could the WMF consider serving something like https://freegeoip.net/ (which uses the free downloads h... [07:57:08] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2566841 (10faidon) We're not, cannot and will not be a general purpose GeoIP provider, for various and diverse reasons. You can use freegeoi... [13:08:13] paravoid: lvs/pybal/router config? :) [13:11:23] 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations, 13Patch-For-Review: ULS GeoIP should only use the Cookie - https://phabricator.wikimedia.org/T143270#2567345 (10BBlack) Ok, I guess I missed the cookie-parsing code in ULS.... or is it relying on the fact that CN has already parsed... [13:39:21] bblack: I was speaking with a former colleague (from China) and he mentioned that all traffic from China towards the rest of the world is going through 2 IXPs, Shanghai and Guangzhou [13:39:22] 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations, 13Patch-For-Review: ULS GeoIP should not use meta.wm.o/geoiplookup - https://phabricator.wikimedia.org/T143270#2567423 (10BBlack) [13:39:39] which are both closer to ulsfo then to esams geographically speaking I think [13:40:03] so the ulsfo choice for China in config-geo is probably quite appropriate :) [13:40:31] paravoid: ^ [13:50:56] 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations, 13Patch-For-Review: ULS GeoIP should not use meta.wm.o/geoiplookup - https://phabricator.wikimedia.org/T143270#2567456 (10Nikerabbit) >>! In T143270#2567345, @BBlack wrote: > Ok, I guess I missed the cookie-parsing code in ULS.... o... [14:06:25] 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations, 13Patch-For-Review: ULS GeoIP should not use meta.wm.o/geoiplookup - https://phabricator.wikimedia.org/T143270#2567499 (10BBlack) So, the obvious options are: http://freegeoip.net/json/8.8.8.8 http://geoip.nekudo.com/api/8.8.8.8 B... [14:49:56] 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations, 13Patch-For-Review: ULS GeoIP should not use meta.wm.o/geoiplookup - https://phabricator.wikimedia.org/T143270#2567611 (10Nikerabbit) Simplest thing would be to switch the default freegeoip.net which is already supported and give so... [14:56:08] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2567659 (10Nikerabbit) >>! In T100902#2566841, @faidon wrote: > You can use freegeoip, or another free or paid-for service — or set up your o... [14:57:53] 10Traffic, 10MediaWiki-extensions-UniversalLanguageSelector, 06Operations, 13Patch-For-Review: ULS GeoIP should not use meta.wm.o/geoiplookup - https://phabricator.wikimedia.org/T143270#2567661 (10BBlack) ULS already supports the freegeoip format? [14:58:29] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2567664 (10BBlack) >>! In T100902#2567659, @Nikerabbit wrote: >Non-WMF MediaWiki installs? That [14:59:12] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2567671 (10faidon) Sorry, my bad. I meant that non-WMF MediaWiki installs could use any of these. Someone (the ULS maintainers/contributors)... [15:01:53] oh the dead code paths in varnishmedia were most likely copy-pasted from varnishrls [15:01:56] https://github.com/wikimedia/operations-puppet/blob/production/modules/varnish/files/varnishrls#L46 [15:02:46] but they do make sense there, see the regexp passed to varnishlog.varnishlog towards the bottom of the file [15:08:12] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2567686 (10Nikerabbit) Location based language suggestions is a feature in ULS that is maintained by the Language team. It would have been ni... [15:25:38] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2567717 (10BBlack) We'll be talking to the relevant teams next week about making new releases ahead of completely disabling any service, that... [16:08:23] ema: cool. sorry I didn't think of these scripts before when we were discussing upload timeline. [16:08:36] ema: but sounds like they're fairly straightforward anyways, all things considered. [16:09:06] I've also fallen behind on doing the frontend memory restarts, I'm going to clean that up today for rest of upload + text + maps [16:12:04] bblack: varnishmedia4 is merged and seems to work on my labs instance, so we should be good in that regard [16:12:21] awesome [16:12:59] unless something else pops up I guess we can start the upgrade soonish? [16:13:34] yeah I think so [16:13:39] nice [16:14:42] maybe start early-mid next week, with 1x cache, then either 1x dc or 1x cache per dc, etc [16:14:52] and assuming no major issues, we can be done before the first [16:15:42] with deprecated_persistent or with file? :) [16:16:08] with file I think [16:16:22] I should link that up harder in phab and press for more protest responses before we do it [16:16:41] yeah given the bugs we've seen on persistent I guess it makes sense [16:16:48] (among other reasons) [16:17:16] I really think there's probably a lot of subtle things that get better on 'file' [16:17:31] things like how eviction really works and how/when things stall out a bit on some operations, etc [16:17:48] even if it weren't for the outright bugs and deprecation/complexity in general [16:19:02] in phab terms there's also https://phabricator.wikimedia.org/T131761 in the way, but that's also mostly a matter of phab cleanup [16:19:18] we know how it works and we know there isn't a major issue there in practice thanks to swift always providing Content-Length [16:19:30] will clean up that now too [16:19:58] cool, thanks [16:20:57] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502#2567932 (10BBlack) [16:21:00] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Solve large-object/stream/pass/chunked in upload cluster better - https://phabricator.wikimedia.org/T131761#2567929 (10BBlack) 05Open>03Resolved a:03BBlack So, we've reviewed the VCL and the Swift output, and the bottom line is this is a non-iss... [16:25:14] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502#2567938 (10BBlack) [16:25:17] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Varnish 4 stalls with two consecutive Range requests using HTTP persistent connections - https://phabricator.wikimedia.org/T142233#2567936 (10BBlack) 05Open>03Resolved The commit merged above gives us two behaviors (on all cache layers) for Range... [16:27:20] ema: are we leaving varnishrls for later? if so can split the task for the scripts [16:27:27] xcps was already done [16:27:46] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2567939 (10faidon) >>! In T100902#2567686, @Nikerabbit wrote: > Location based language suggestions is a feature in ULS that is maintained by... [16:31:16] bblack: I was looking into varnishrls today, it's not much work but we need it only for text actually [16:31:24] so yeah we can surely leave it for later [16:32:37] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502#2567949 (10BBlack) The remaining blockers aren't full blockers and aren't linked into here, but basically they're these: 1. T131353 - Just the cache_upload -related scrip... [16:33:50] 10Traffic, 06Operations: Stop using persistent storage in our backend varnish layers. - https://phabricator.wikimedia.org/T142848#2548194 (10BBlack) Note we'd like to make a call on this before converting cache_upload to v4 in T131502 (and presumably switching to `file` as part of the process). If anyone has... [16:35:00] 10Traffic, 10ArticlePlaceholder, 06Operations, 10Wikidata: Performance and caching considerations for article placeholders accesses - https://phabricator.wikimedia.org/T142944#2567964 (10hoo) [17:53:18] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#1323201 (10jeblad) Talk to the language team. Please? [17:56:37] 10Traffic, 06MediaWiki-Stakeholders-Group, 06Operations, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#1323201 (10greg) >>! In T100902#2568267, @jeblad wrote: > Talk to the language team. That would be what the conversation with @Nikerabbit is... [18:10:26] hi, 8 differents users report http 400 error on edits. https://fr.wikipedia.org/wiki/Sujet:T9wtwe84pz3n18fy [18:10:29] thanks [19:15:53] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: make blog links from wmfwiki front page use HTTPS links - https://phabricator.wikimedia.org/T104728#2568496 (10EdErhart-WMF) [19:16:04] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: make blog links from wmfwiki front page use HTTPS links - https://phabricator.wikimedia.org/T104728#1425614 (10EdErhart-WMF) 05Open>03Resolved [19:41:12] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Blog: make blog links from wmfwiki front page use HTTPS links - https://phabricator.wikimedia.org/T104728#2568535 (10jeremyb)