[02:08:19] !log stat1002: disabled puppet, umounted /tmp, /home and /a, poweroff [02:08:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [02:10:21] 10Analytics, 10Analytics-Dashiki: Convert Extension:Dashiki to use extension registration - https://phabricator.wikimedia.org/T171884#3480317 (10Legoktm) 05Open>03Invalid mediawiki/extensions/Dashiki in Gerrit has an extension.json file. If there's confusion about git repositories or 404 links, please file... [02:10:38] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3480320 (10Ottomata) Done. /a/$USER directories from stat1002 are in /srv/stat1002-a/user_dirs_from_stat1002. stat1002 has been powered off. [02:16:37] ottomata: i have updated https://wikitech.wikimedia.org/wiki/Stat1002 [02:17:58] 10Analytics, 10Analytics-Cluster, 10Operations: thorium - failed git clone of geowiki-data-private - https://phabricator.wikimedia.org/T171923#3480324 (10Dzahn) [03:26:59] thanks HaeB [07:54:15] !log re-run webrequest-load-wf-upload-2017-7-28-6 from Hue (was playing with eth0 issues on an1034) [07:54:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:55:28] !log update nodejs to 6.11 on aqs1004 (testing prod node after beta qa) [07:55:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:56:22] 10Analytics, 10User-Elukey: Upgrade AQS to node 6.11 - https://phabricator.wikimedia.org/T170790#3480588 (10elukey) a:03elukey [07:56:43] 10Analytics-Kanban, 10User-Elukey: Upgrade AQS to node 6.11 - https://phabricator.wikimedia.org/T170790#3442777 (10elukey) [08:11:40] 10Analytics, 10Analytics-Cluster, 10Operations: thorium - failed git clone of geowiki-data-private - https://phabricator.wikimedia.org/T171923#3480643 (10elukey) This issue has already happened in the past, this brutal sequence of commands fixed it: ``` root@thorium:/srv/geowiki# rm -rf data-private root@th... [08:12:06] 10Analytics, 10Analytics-Cluster, 10Operations, 10User-Elukey: thorium - failed git clone of geowiki-data-private - https://phabricator.wikimedia.org/T171923#3480644 (10elukey) p:05Triage>03Normal [09:39:47] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3480794 (10elukey) Since the kafka1012->kafka1022 are going to be decommed and kafka-jumbo is a complete new cluster from our... [09:46:48] 10Analytics-Kanban, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 1st December - https://phabricator.wikimedia.org/T152193#2841163 (10Ewaszwed) Hello, we see the same issue staring tonight around 1 AM. Is this being looked at? [10:12:49] mmm --^ is weird [10:14:53] I don't see jobs failing [10:15:24] BUT stat1002 was turned off yesterday [10:15:32] * elukey suspects that a rsync si not working [10:17:03] 10Analytics-Kanban, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 1st December - https://phabricator.wikimedia.org/T152193#3480839 (10elukey) Thanks for the ping, we are taking care of the issue as we speak :) [10:17:08] yes [10:17:09] :) [10:19:13] 10Analytics, 10Analytics-Dashiki: Convert Extension:Dashiki to use extension registration - https://phabricator.wikimedia.org/T171884#3480840 (10MarcoAurelio) @Legoktm so, to clarify, what is the right extension we're using on Wikimedia: rANAD or the Gerrit one you linked? I can file a task but I need to know... [10:20:27] 10Analytics-Kanban, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 1st December - https://phabricator.wikimedia.org/T152193#3480841 (10elukey) https://gerrit.wikimedia.org/r/#/c/368383 should fix the issue, it will take a bit of time but the new files will be created soon. [10:21:33] * elukey lunch! [10:21:57] fdans: https://en.wikipedia.org/wiki/Umarell - some interesting knowledge from Bologna [10:23:06] elukey: ooooohhhh I like this term [10:23:22] there's no word for that in Spanish [10:23:41] 10Analytics-Kanban, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 1st December - https://phabricator.wikimedia.org/T152193#3480845 (10ArielGlenn) Please open a new ticket in the future if the old one has been resolved for so long, as likely the issue will be different. It seems that... [11:56:04] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3426541 (10JGonsior) Hey, could you please install pip on stat1005? [12:01:11] 10Analytics, 10Analytics-Dashiki: Convert Extension:Dashiki to use extension registration - https://phabricator.wikimedia.org/T171884#3479265 (10Milimetric) @MarcoAurelio: rANAD is not an extension, it's only related to mediawiki-Dashiki by the word "Dashiki". Which, in this context, means "dashboards on wiki... [13:18:26] (03CR) 10Mforns: [C: 031] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/367940 (https://phabricator.wikimedia.org/T171760) (owner: 10Nuria) [13:46:52] fdans: had an idea [13:47:14] imagine a button on the graphs that said "Explain This Graph" [13:47:34] when you click it, it renders the graph piece by piece, starting with the axes [13:47:58] oh like a tutorial? [13:48:22] so it renders the Y axis, and says Pageviews: how many times human users look at articles [13:48:33] and then that text shrinks into just the label Pageviews [13:48:44] and then it renders the X axis and says Time: from X to Y [13:48:50] and then it renders the plot inside [13:49:09] it occurred to me that's what we all do anyway when we look at a new graph we don't understand [13:52:12] (03PS4) 10Milimetric: Implement Wikistats metrics as Druid queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365806 (https://phabricator.wikimedia.org/T170882) [13:56:13] milimetric: that sounds very cool, although I don't completely visualise it right now :) [13:57:48] :) well, then we make a good team 'cause I can visualize it perfectly but I don't know how it sounds [13:58:21] Hello, anyone knows how to make a list of most UNVIEWED articles? Articles with minimum of views per last...month? [14:00:07] Urbanecm: there are millions of articles that receive almost no views [14:00:25] you can figure out which these are by looking at the dumps of pageview data, one sec I'll link you [14:00:32] * elukey brb [14:00:36] I'll know where they are :) [14:00:43] Urbanecm: [14:00:44] https://dumps.wikimedia.org/other/pagecounts-ez/ [14:00:55] that dataset is a lot smaller and has all the data you need [14:01:02] Thank you! [14:01:05] np [14:51:04] 10Analytics, 10Operations, 10netops, 10User-Elukey: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3481411 (10elukey) [14:56:57] 10Analytics, 10Analytics-EventLogging, 10Community-Tech, 10DBA, 10User-Elukey: Drop CookieBlock* tables from EventLogging DB - https://phabricator.wikimedia.org/T171883#3481417 (10elukey) [14:57:23] 10Analytics-Cluster, 10Analytics-Kanban, 10User-Elukey: Perf test RAID vs JBOD with new hardware and kafka versions - https://phabricator.wikimedia.org/T168538#3481418 (10elukey) [15:00:26] elukey: o/ [15:00:57] elukey: re: page cache, have you had a look at the rate of page faults? [15:01:15] elukey: pidstat -r -p 1 [15:01:18] (03PS5) 10Milimetric: Implement Wikistats metrics as Druid queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365806 (https://phabricator.wikimedia.org/T170882) [15:04:07] urandom: I didn't! [15:04:12] 10Analytics-Kanban, 10Analytics-Wikistats: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3481435 (10fdans) Hi @hashar I suspect this is happening because the npm version installed in the Jenkins machine is npm2, and I believe we require npm3 for webpack, and peer dependencies to wo... [15:05:30] elukey: not sure how definitive that is, but if it is utilizing page cache, and the rate of major faults is low, then it seems reasonable to assume that it's "mostly reading from page cache" [15:05:40] ¯\_(ツ)_/¯ [15:06:30] urandom: thanks a lot for following up :) [15:06:40] there always seems to be lots of magic when it comes to this though [15:06:49] elukey: sure sure [15:07:36] really nice, majflt/s is flat zero [15:08:07] yeah, and i wouldn't think that even a handful would be indicative of a problem [15:08:25] not if that machine is doing any kind of a volume [15:08:36] but yeah, zero is definitely a good number there :) [15:09:31] Average: 114 37472 11.19 0.12 19486996 1361656 2.76 java [15:10:01] that's pretty much zero, yeah :) [15:11:05] elukey: i concur with paravoid there, it doesn't sound like io is going to be an issue [15:11:35] maybe you can just do RAID1? [15:11:48] i guess that'd technically be 1e [15:11:57] or hell, maybe 5? [15:12:13] this might be one of those times when a RAID-5 actually makes sense [15:13:42] elukey: it also occurs to me that by tossing out ideas like that, i might be moving you further from the prize, not closer to it, so feel free to ignore :) [15:20:57] urandom: all ideas are well accepted :) [15:21:19] I am wondering if with bcc/bpf something better could be done [15:24:08] mforns__: I've gotta do lunch now 'cause I have two meetings later, but take a look at the latest patch and if you want we can pair again after 17 UTC [15:24:34] milimetric, sure :] [15:24:37] nick mforns [15:24:42] hehe [15:25:03] elukey: you mean this? https://github.com/iovisor/bcc [15:25:44] yep! [15:25:53] i've had that open in a browser tag for *months*, with the intention to make some time to look at it [15:26:01] s/tag/tab/ [15:26:02] same thing in here [15:26:26] even systemtap, e*ma is the expert in here and he did really good analysis for Varnish internals [15:26:58] yeah, systemtap is insane [15:27:27] i'd definitely have to put in some time to know what to do with that, though [15:33:41] milimetric, HaeB : meeting? [15:33:49] milimetric: wait , let me log in again [15:34:14] yes, will be there [15:34:15] milimetric, HaeB : ah sorry, it is in half an hour, my mistake [15:35:36] University of Washington released a tool for automatically checking semantic equivalences of SQL queries https://medium.com/@uwdb/introducing-cosette-527898504bd6 and it's interesting [15:49:02] * elukey off earlier to get back his car (hopefully) [15:49:04] o/ [17:00:03] 10Analytics-Kanban: Oxygen transient logs recovery - https://phabricator.wikimedia.org/T171972#3481824 (10Nuria) [17:36:55] ottomata, o/ [17:37:21] I could use "virtualenv" on stat machines. Seems like that got squashed in the new hosts [17:38:42] * halfak works on puppet stuff [17:42:03] I'm struggling to find the right place to add the require_packages() [17:45:39] haflak i gotcha [17:46:10] halfak: its there, no? [17:46:19] oh, missing python3-virtual env on stat1006 [17:46:33] Aha! https://github.com/wikimedia/puppet/blob/7dfe90c0d494999e2cfc05b12169401d40d54c99/modules/statistics/manifests/packages.pp [17:46:46] Right. [17:46:55] Is this the right place to make an edit to include it? [17:47:11] https://github.com/wikimedia/puppet/blob/7dfe90c0d494999e2cfc05b12169401d40d54c99/modules/statistics/manifests/packages.pp#L95 [17:47:16] Looks like it should be there :\ [17:47:55] that is the spot :) [17:48:03] you making patch halfak? [17:48:14] Looks like it's already there [17:48:20] Line 95 [17:48:20] python3-venv isn't [17:48:24] you want python3? [17:48:27] Oh... is that what we want? [17:48:28] otherwise ya its there [17:48:30] i dunno [17:48:30] do you? [17:48:34] haha [17:48:34] virtualenv isn't version specific [17:48:48] on stat1006 i see both packages avail [17:48:52] but the python3 one isn't installed [17:49:00] I'm looking for the "virtualenv" binary in path [17:49:11] Maybe it got renamed? [17:49:13] on 1006? [17:49:29] yeah [17:49:38] maybe i need p virtualenv - Python virtual environment creator [17:49:42] maybe the python- one is just al ib [17:49:44] lib [17:49:46] adding. [17:49:56] https://packages.debian.org/stretch/python-virtualenv [17:50:05] yeah, that's what we want [17:50:07] its on 1006 now [17:50:08] puppetizing.. [17:50:16] virtual env installs both py3 and py libs [17:51:01] there is a weird python3-venv thing which isn't really what I want. ... I think [17:53:20] it is a dep of just 'virtualenv' [17:53:23] which has the bin [17:54:58] halfak: https://gerrit.wikimedia.org/r/#/c/368461/ [17:55:30] Oh! Gotcha! Thanks for working that out. [17:56:06] works now! [17:56:08] gr8 :) [17:56:12] <3 ottomata :D [18:06:23] stat1006 starts it's first bigass halfak processing job. [18:06:24] :) [18:14:50] :D [18:27:13] halfak, how much popcorn do you think we'll need? [18:28:44] Depends on how the first part of the processing job goes. If we pass the first 30 minutes successfully, then the fireworks start. [18:28:56] If that fails, then we get a frustrated halfak and no show [19:29:26] * halfak continues to wait on the index building. [19:29:44] I guess scanning a 500m row dataset and building a hash table of it takes a long time. Who knew? ;) [19:30:05] I should do a grep first next time. I don't need to scan the *WHOLE THING* [19:34:54] (03PS6) 10Milimetric: Implement Wikistats metrics as Druid queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365806 (https://phabricator.wikimedia.org/T170882) [20:44:52] 10Analytics-Kanban, 10Analytics-Wikistats, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3482606 (10hashar) Indeed we are still using npm 2 and havent bothered yet to upgrade to 3 or later. That is the subject... [21:17:46] 10Analytics, 10Discovery, 10Discovery-Analysis: Add referer to WebrequestData - https://phabricator.wikimedia.org/T172009#3482778 (10mpopov) [22:16:21] I just checked back and THE FIREWORKS ARE HAPPENING [22:16:27] Sweet sweet multiprocessing [22:16:31] MWAHAHAHA! [22:35:43] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Pagecounts-ez_dataset_hasn.27t_generated_since_JUL-23 [22:36:58] 10Analytics, 10Analytics-Cluster: Cannot request more than 4 cores per spark executor - https://phabricator.wikimedia.org/T172018#3482974 (10EBernhardson) [23:23:23] 10Analytics, 10DBA, 10Chinese-Sites: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3483134 (10Neil_P._Quinn_WMF) >>! In T165233#3292656, @Nuria wrote: > Ok, so we can plan on this data being available in September, correct? Until then we will continue taking snapshots... [23:23:37] 10Analytics, 10Contributors-Analysis, 10DBA, 10Chinese-Sites: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3483140 (10Neil_P._Quinn_WMF)