[01:24:07] 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3257455 (10kaldari) @MusikAnimal: Does the memory usage go back down when it starts on a new project (within the same bot run)? [07:58:16] not sure if everybody has access but https://logstash.wikimedia.org/app/kibana#/dashboard/Varnish-Webrequest-50X is finally there :) [08:03:37] I think it is a very nice and useful tool for ops coming from the analytics pipeline :) [08:42:40] elukey: Heya ! And finally it uses kafkatee? [08:43:33] joal: yep! [08:43:37] elukey: :( [08:44:00] ?? [08:44:03] it is cool! [08:44:43] elukey: result is, but I don't like kafkatee for these usage, I'd have prefered a streaming job (even if a bit overkill_ [08:44:59] * joal is a streaming integrist [08:45:41] elukey: don't bother, as you said, it's awesome to be able to monitor 5XX in realtime :) [08:51:16] elukey: I'm sorry if I said something wrong :( [09:00:08] joal: ahhhhh got you! I proposed to use Spark for this use case so I was in favor, maybe in the future we'll convince Andrew :) [09:00:44] * elukey plays Stereophonics - Maybe Tomorrow [09:01:01] :) [09:02:55] * joal has found the uniques redirects pattern that mess with global uniques computation - YES [09:03:06] \o/ [09:03:17] BUT - It's not easily fixable :( [09:03:24] * joal cries in a corner [09:03:39] * elukey hugs joal [09:04:24] * joal plays Queen - It's a hard life [09:05:10] At least it's half of the problem found :) [09:12:13] :) [09:12:39] joal: totally unrelated - have you already checked the trip Airport -> accomodation? [09:13:08] gmaps seems to suggest two routes with two buses each [09:15:00] elukey: Nope I didn't [09:16:49] ahh okok [09:16:56] elukey: to me it says: Bus 119, then tube A [09:17:07] Then walk or tram q13 [09:19:03] ah ok sorry didn't see that the second was the tube [09:19:14] are we going together with Marcel? [09:19:38] elukey: I wonder about waiting or not - It'll be like a 2 hours wait [09:19:51] yeah I think it might be too much [09:20:16] I can definitely wait marcel [09:21:00] (Just noticed that in the manifest I am listed as Tuscano, that's a new one, usually it is Tascano) [09:21:08] hehehe [09:36:37] (03PS2) 10Joal: Provide RedirectToPageview function and UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353310 (https://phabricator.wikimedia.org/T143928) [09:38:32] (03CR) 10Joal: [V: 032] Add druid hourly loading of pageviews [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352784 (https://phabricator.wikimedia.org/T164730) (owner: 10Joal) [09:41:22] (03Abandoned) 10Joal: Add uniques global jobs and correct uniques [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352099 (https://phabricator.wikimedia.org/T143928) (owner: 10Joal) [10:01:13] elukey: Do you recall by any chance if something got done on traffic on 2016-11-10?n [10:01:34] elukey: I found a funny artifact in uniques: [10:03:17] need to check but nothing that I recall on top of my head [10:03:19] goo.gl/7iv65dcontent_copyCopy short URL [10:03:27] https://goo.gl/7iv65d [10:03:29] sorry [10:04:00] elukey: --^ [10:06:43] joal: so for https://tools.wmflabs.org/sal/production?p=0&q=&d=2016-11-09 I can see "upgrading varnish to varnish 4" [10:08:23] and it went up again feb 13/14 [10:08:36] elukey: I wnoder if, when moving to v4, we removed some cookies [10:08:59] elukey: look at the uniques_offset metric: almost exact counter-part [10:09:29] elukey: this tells us two things: Uniques underestimate + Unique offsets are actually a good way to count ! [10:09:32] on the 13th I can see a deployment for mobile apps https://tools.wmflabs.org/sal/production?p=0&q=&d=2017-02-13 [10:09:44] elukey: And, some strange things happen with cookies [10:10:03] elukey: What happend on Feb13 was that we merged Last-access-Global [10:10:20] elukey: I'd love to brainbounce on that with you and nuria :) [10:10:45] elukey: need to go for lunch (last lunch home before a week away, going to have with my wife) [10:10:53] :) [10:11:00] we can chat later on or in Prague! [10:11:06] Sounds good [10:11:14] Later elukey ;) [10:42:01] 10Analytics, 10Analytics-EventLogging, 06Collaboration-Team-Triage, 10MediaWiki-ContentHandler, and 5 others: Multiple MediaWiki hooks are not documented on mediawiki.org - https://phabricator.wikimedia.org/T157757#3015493 (10Tgr) TBH this feels like a waste of manpower to me. hooks.txt is easily parsable... [10:45:01] 10Analytics, 10Analytics-EventLogging, 06Collaboration-Team-Triage, 10MediaWiki-ContentHandler, and 5 others: Multiple MediaWiki hooks are not documented on mediawiki.org - https://phabricator.wikimedia.org/T157757#3258147 (10Mainframe98) Well, we should update https://www.mediawiki.org/wiki/Manual:Hooks r... [11:16:11] (03CR) 10Elukey: [C: 031] "Filippo explained the change to me and it seems sound and correct, LGTM" [analytics/kafkatee] - 10https://gerrit.wikimedia.org/r/352591 (owner: 10Filippo Giunchedi) [11:23:25] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] Reset signal disposition and unblock signals for children [analytics/kafkatee] - 10https://gerrit.wikimedia.org/r/352591 (owner: 10Filippo Giunchedi) [13:13:40] (03PS3) 10Matthias Mullie: Add dewiki to illustrations query config [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/268206 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:15:42] (03CR) 10Matthias Mullie: [C: 032] Add dewiki to illustrations query config [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/268206 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:16:18] (03CR) 10Matthias Mullie: [V: 032 C: 032] Add dewiki to illustrations query config [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/268206 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:17:04] (03CR) 10Matthias Mullie: [V: 032 C: 032] Add illustration queries for enwiki [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/267722 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:18:49] (03PS4) 10Matthias Mullie: Add illustration queries for enwiki [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/267722 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:19:06] (03CR) 10Matthias Mullie: [V: 032 C: 032] Add illustration queries for enwiki [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/267722 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:22:38] (03PS4) 10Matthias Mullie: Add dewiki to illustrations query config [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/268206 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:23:35] (03CR) 10Matthias Mullie: [V: 032 C: 032] Add dewiki to illustrations query config [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/268206 (https://phabricator.wikimedia.org/T111793) (owner: 10MarkTraceur) [13:29:01] 06Analytics-Kanban, 13Patch-For-Review: Count global unique devices per top domain (like *.wikipedia.org) - https://phabricator.wikimedia.org/T143928#3258549 (10JAllemandou) So here's where we are so far: **Successes**: - `Uniques project-wide` (by opposition to `uniques per-domain`) have been computed for 2 m... [13:56:17] joal: shall we skip the standup? I think that it will probably me and you (maybe fdans?) [13:56:24] not sure if Marcel is online [13:56:33] I'm here but started like an hour ago [13:56:37] so not much to report [13:56:40] elukey: as you wish :) [13:57:23] don't have a strong opinion, if you guys want we can meet otherwise get some time back [13:57:36] we can just say hello :) [13:57:54] Sounds good, let's say hello :) [13:58:22] i'm in [15:27:39] 10Analytics, 06Operations, 10ops-eqiad: SATA errors for stat1004 in the dmesg - https://phabricator.wikimedia.org/T162770#3258807 (10Cmjohnson) [16:06:56] 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3258911 (10MusikAnimal) >>! In T164178#3257455, @kaldari wrote: > @MusikAnimal: Does the memory usage go back down when it starts on a new project (within the s... [16:13:26] 10Quarry: Some querries cannot be 'unstarred' - https://phabricator.wikimedia.org/T165169#3258930 (10XXN) [16:45:39] 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3259016 (10MusikAnimal) Here's the same run with the memory output, except with the promises removed – so all it does is fetch redirects (and store 0 as the num... [19:41:23] 10Analytics, 10Reading Epics, 06Wikipedia-iOS-App-Backlog, 07Spike, 05iOS-app-v5.5.0-Snake-On-A-Magic-Towel: Research and define initial technical requirements for app analytics - https://phabricator.wikimedia.org/T164801#3259558 (10AMroczkowski) a:03AMroczkowski [20:22:50] Hey, do we have a list of browsers+versions our traffic comes through? I remember seeing it listed somewhere but I can't find it. :( [20:24:01] Niharika: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os :) [20:24:17] Thanks neilpquinn! [20:25:35] milimetric: is the 2017-04 snapshot of the edit data incomplete? I ran `select * from mediawiki_history where event_entity = "revision" and event_type = "create" and snapshot = "2017-04" and wiki_db = "enwiki" and revision_parent_id is null and revision_is_deleted = 0 limit 10` and got no results. [20:28:24] Hmm, no, must not be that because `snapshot = "2017-02"` also got no results [20:32:43] aha, that's because page creations get `revision_parent_id = 0` [21:06:36] 10Analytics, 06Editing-Analysis: Old deleted pages have empty fields in Analytics Cluster edit data - https://phabricator.wikimedia.org/T165201#3259757 (10Neil_P._Quinn_WMF) [23:11:00] elukey: that kibana site is nice (and slow... 4 sec paint time for me right now, maybe is my machine)