[00:20:37] Never mind. I think I figure it out :) [01:38:17] 10Analytics-Kanban, 10Analytics-Wikimetrics, 10Software-Licensing: Add a license file to wikimetrics - https://phabricator.wikimedia.org/T60753#644117 (10Legoktm) WTFPL is not an appropriate license, it's not OSI approved and suffers from the s ame problems as "public domain" in that it doesn't work in all c... [03:00:34] chelsyx: some docs [03:01:38] chelsyx: we just switched boxes [03:01:43] chelsyx: for EL database [03:02:15] chelsyx: but the tables you are interested on do not exist on that db or host [03:02:21] chelsyx: ping us tomorrow, we can help [03:05:42] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3829966 (10Nuria) Both actually, i think we could use this space to organize many upcoming tasks [03:06:27] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3829967 (10Nuria) But I tthink @Aklapper will get to it now that ticket is not in "blocked" ( i think I should have changed that earlier and I forgot) [03:52:20] (03CR) 10Nuria: "I think i truly do not understand why is the fork needed, the scala job produces 1 file per wiki using the list of passed wikis, right?" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) (owner: 10Joal) [04:05:02] (03CR) 10Nuria: "We can catch up on irc, answer is probably obvious." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) (owner: 10Joal) [04:10:20] nuria_: Thanks Nuria! I've got it work! :) [10:10:32] Hi fdans :) Would you have a minute to share with me your thoughts on the doc page I wrote? [10:13:08] 10Analytics-Kanban, 10Patch-For-Review: Productionize Superset - https://phabricator.wikimedia.org/T166689#3830383 (10JAllemandou) @Ottomata : Super cool ! Many thanks :) [10:21:53] (03CR) 10Joal: "@Nuria: Scala job produces one FOLDER per wiki, with one file inside each. While folder name is usable, file name is hadoop-style (part-00" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) (owner: 10Joal) [10:22:41] elukey: we have 3 curves !!! https://grafana.wikimedia.org/dashboard/db/prometheus-druid?panelId=41&fullscreen&orgId=1&from=now-1h&to=now [10:22:47] Thanks a lot mate :) [10:25:56] yes!!! \o/ [10:26:19] elukey: ig you teach me, I can ry to add alerts for realtime events not flowing :) [10:27:38] joal: still need to figure out how since the prometheus alarming is new, but andrew added some stuff for kafka [10:27:56] np elukey :) [10:28:21] elukey: I have bandwidth now that the WKS2 rush is over - I can help for anything you need me on (java 8, alarms, stream, or other) [10:29:13] all right it should be 'monitoring::check_prometheus' on puppet [10:29:43] joal: java8 would definitely be a good thing to do [10:30:26] * elukey afk for a bit [10:48:49] heyaaa [10:48:53] Hi mforns [10:52:25] joal - so sorry, I haven't been getting notifications from irccloud since yesterday [10:52:29] do you want to batcave? [10:52:50] maybe mforns can join us too [10:53:13] fdans, sure! 'bout what? [10:53:36] feedback for joal's document on data quality [10:53:37] http://spin.js.org/ [10:53:41] no not that [10:53:45] https://wikitech.wikimedia.org/wiki/Analytics/AQS/Wikistats/Data_Quality [10:54:48] * joal like spinning wheel feedback :) [10:55:16] So here it is again: https://www.youtube.com/watch?v=kK62tfoCmuQ [10:56:02] Ho fdans - Sorry I didn't get you wished me to batcave ! [10:56:05] OMW fdans ! [11:11:00] joal, I like the general structure: 1) introducing the different comparisons made, 2) showing the results. I also like the 3 cases you presented: tiny wiki, big wikis, special wikis. I agree with Fran that maybe tabular layout would help catch it quicker. [11:11:24] mforns: tables in each section, right? [11:11:46] joal yes [11:12:05] cool mforns :) [11:12:07] Got it [11:12:09] Will do that [11:14:56] joal, also, there's a sentence in "8 most viewed Wikipedias: Very-active editors" that I'm not sure I understand [11:15:51] you mean that the difference is due to those wikis having more edits (per pageview) than the other wikis? [11:17:35] hm, no, mforns [11:20:01] another question for fdans: Will we keep the /v2 url for anouncement, or will we use another one? I'd love to link some new UI charts to that report :) [11:20:40] joal, ah, it points to the same problem in the "edits" study? [11:20:49] correct mforns :) [11:20:52] I seeeeee, my bad [11:20:54] I could be clearer [11:21:15] no no, it's clear [11:21:24] ok then :) [11:21:28] :] [11:33:02] * elukey lunch [11:59:48] here I am [12:00:27] so I'd need to run errand for ~1.30h later on, will work a bit later this evening :) [12:17:30] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: kafka1018 fails to boot - https://phabricator.wikimedia.org/T181518#3830640 (10elukey) a:03elukey [12:19:42] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: kafka1018 fails to boot - https://phabricator.wikimedia.org/T181518#3830645 (10elukey) notebook1002 is now PXE installing fine, I removed the previous hw config and created 12 1 disk RAID0 virtual devices with PERC. The oth... [12:20:58] who's on the oncall rotation this week? [12:21:48] there are some data loss alarms that are probably false positives, but I'd like to check with other people and possibly document a procedure :) [12:34:20] It;s me elukey [12:35:59] ahhh okok I'll do the check then, update the docs and submit to your mercy :D [12:37:36] elukey: I can do it [12:37:59] elukey: If you can send me your existing queries, I'll triple check and document [12:42:45] joal: I basically use the script to find the holes, check the big ones and see if the missing sequence numbers are in the next hour bucket [12:42:57] and see their timings [12:46:41] elukey: that's what I'd do :) [12:47:03] elukey: should we document that in a procedure on a page? [12:47:15] elukey: I'm happy to do it [12:47:59] that would be great! maybe https://wikitech.wikimedia.org/wiki/Analytics/Team/Oncall ? [12:48:25] Dan asked me to do it last time but if you have time I'd be really glad [12:48:36] elukey: you did already !! [12:49:15] Ah, not exactly :) [12:49:25] |Ok, will document more [12:51:16] \o/ [12:51:29] all right going afk for 1h people, ttl! [13:13:19] updated data quality doc team :) [13:13:38] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3830789 (10Aklapper) As per T174675#3577182: Could someone please provide a description of that Space, and a name for that Space? Also, I assume that the existing public #wmf-leg... [13:15:29] Taking a break a-team - will investigate data-loss warnings and update docs when back [13:59:06] hiya [13:59:18] looking at siege now [14:43:00] 10Analytics: Make superset more scalable - https://phabricator.wikimedia.org/T182688#3831095 (10Ottomata) [14:43:10] 10Analytics: Make superset more scalable - https://phabricator.wikimedia.org/T182688#3831106 (10Ottomata) [14:43:39] 10Analytics-Kanban, 10Patch-For-Review: Productionize Superset - https://phabricator.wikimedia.org/T166689#3831110 (10Ottomata) Made a new task for some of the above points: T182688 [14:43:56] 10Analytics-Kanban, 10Patch-For-Review: Productionize Superset - https://phabricator.wikimedia.org/T166689#3304286 (10Ottomata) [14:51:09] 10Analytics, 10Operations, 10ops-eqiad: Decomission eventlog2001 - https://phabricator.wikimedia.org/T182397#3831144 (10Cmjohnson) p:05Normal>03Low [14:52:05] 10Analytics, 10DC-Ops, 10Operations, 10ops-codfw: Decomission eventlog2001 - https://phabricator.wikimedia.org/T182397#3822380 (10Cmjohnson) a:03Papaul assigning to @papaul and correct data center [15:01:03] 10Analytics-Kanban, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db104[67] - https://phabricator.wikimedia.org/T181784#3831180 (10Cmjohnson) p:05Triage>03Low a:03Cmjohnson [15:01:38] elukey: are there any non varnish 4 instances left? [15:01:49] shoudl we make the varnishkafka/varnishkafka_v4.conf.erb template the default one in the instance define? [15:02:10] only v4 left, we should yes [15:05:16] elukey: another q [15:05:29] do you think the varnishkafka profiles should be included from the cache roles, or the other cache proifles? [15:05:34] e.g., i'm looking at statsv now [15:05:37] and moving it to a profile [15:05:46] currently it is included from profile::cache::text [15:06:01] which makes sense, as probably all cache::text should include it [15:06:06] but i could also include it from role::cache::text [15:06:10] which woudl effectively be the same... [15:06:13] ? [15:06:35] perhaps profile::cache::text shoudl handle only setting up the varnish instance [15:06:47] yep I'd prefer to put it in the role if I had to choose [15:06:48] the the role should include other profiles that also shoudl be for our text caches? that sounds right, right? [15:06:50] ok cool [15:06:53] i think so too [15:06:54] will do that. [15:07:09] ottomata: I am currently renaming notebook1002 to kafka1023 [15:07:24] +1 cool! [15:07:36] new notebook servers order has been approved by mark [15:08:59] yep saw it! [15:14:12] (03CR) 10Fdans: [C: 032] Fix bar chart not re-rendering [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/396537 (https://phabricator.wikimedia.org/T182461) (owner: 10Milimetric) [15:14:32] ottomata: paranoid review - https://gerrit.wikimedia.org/r/#/c/397534/ [15:14:41] I don't see any issue in introducing kafka1023 with role spare [15:14:53] (03CR) 10Fdans: [C: 032] Fix loading sparse data into widgets [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/396469 (https://phabricator.wikimedia.org/T182224) (owner: 10Milimetric) [15:15:01] anything that might auto-pick it up? [15:15:14] (99% sure no but better to brain bounce) [15:15:56] (03CR) 10Fdans: [V: 032 C: 032] "Not sure what you mean with "data more than one year ago", but this looks good to me!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/396469 (https://phabricator.wikimedia.org/T182224) (owner: 10Milimetric) [15:16:00] no [15:16:03] +1 elukey [15:16:07] (03CR) 10Fdans: [V: 032 C: 032] Fix bar chart not re-rendering [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/396537 (https://phabricator.wikimedia.org/T182461) (owner: 10Milimetric) [15:17:18] fdans: the more than a year ago thing, maybe it's important, let's talk it over [15:17:42] so first, did you see how lastIndex - 12 was wrong with non-continuous data? [15:20:46] yeah that made sense milimetric [15:21:18] elukey: I see that nuria had installed siege on aqs1004 and ran it from there. She mentioned installing it on restbase yesterday, but aqs seems like a better fit, unless you all found it problematic last time you ran it [15:21:32] so I'm just checking, is it ok to run it on aqs1004? [15:22:02] ok, fdans, then the fix now is, when there's fewer than 12 months, it goes backwards until it finds the first one that's > 1 year ago, right? [15:22:19] so, that could be a year ago or it could be *more* than a year ago [15:22:39] it could be two years ago [15:22:50] so YoY would show change of the first month that's at least a year ago [15:23:12] which is not exactly YoY, but it's the only available stat [15:23:19] ohhhh I see, that makes sense [15:23:21] milimetric: sure, no issue [15:23:23] k, thx elukey [15:23:38] gotcha milimetric [15:23:53] k, fdans, similar logic goes with MoM, I noticed these problems when I was looking at some tiny wikis like oromo [15:23:59] oromo wiktionary I think [15:24:26] ah oromo wiktionary is the one that has way more edits than pageviews [15:28:59] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: kafka1018 fails to boot - https://phabricator.wikimedia.org/T181518#3792843 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` notebook1002.eqiad.wmnet ``` The log... [15:32:07] yooo elukey lemme know if you have a min for a brainbounce about statsv stuff [15:32:28] ottomata: sure! [15:32:47] gimme 5 min [15:39:10] ottomata: irc/bc? [15:39:33] ya in bc [15:41:44] 10Analytics, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: Decomission eventlog2001 - https://phabricator.wikimedia.org/T182397#3831340 (10Cmjohnson) [15:42:17] 10Analytics, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: Decomission eventlog2001 - https://phabricator.wikimedia.org/T182397#3822380 (10Cmjohnson) Switch port is ge-5/0/9 labeled eventlog2001-decommed @papaul all yours [15:54:23] !log sieging aqs1004 with 100.000 transactions [15:54:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:54:39] milimetric: sieging aqs or druid? [15:55:01] well, hitting aqs urls from aqs via localhost joal [15:55:07] k [15:55:22] milimetric: not sure if you track that: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&orgId=1&var-server=druid1004&var-network=eth0 [15:55:25] joal: i was trying to ask luca but do you know if that distributes work evenly now, after you changed that config? [15:55:53] heh, yeah, I'm definitely hurting it a little [15:56:23] milimetric: I can't recall - I think so yes, I think we have a loadbalancer before broker - but elukey is the one to confirm [15:56:43] yeah, he's busy, we can talk in standup [15:58:02] is there variability in the requests you sent milimetric ? Looks done [15:58:23] joal: yeah, it's done, I'll report at standup and what I'm thinking next [15:58:32] (just 'cause it's in a minute) [15:58:38] :D [16:06:07] milimetric: aqs calls druid via druid-public-broker.svc.eqiad.wmnet, so it should be distributed evenly [16:10:19] thanks for confirmaion elukey :) [17:09:49] 10Analytics-Kanban: Geowiki stopped updating on October 24th - DATA LOSS (read comments) - https://phabricator.wikimedia.org/T179952#3831635 (10Nuria) 05Open>03Resolved [17:10:12] 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Add action api counts to graphite-restbase job - https://phabricator.wikimedia.org/T176785#3831636 (10Nuria) 05Open>03Resolved [17:10:26] 10Analytics-Kanban: Rename historical fields in mediawiki-history - https://phabricator.wikimedia.org/T179689#3831639 (10Nuria) 05Open>03Resolved [17:11:02] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Test and possibly raise the Xmx/Xms settings for the Hadoop Yarn Namenode and HDFS datanode daemons - https://phabricator.wikimedia.org/T178876#3831640 (10Nuria) 05stalled>03Resolved [17:11:24] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Provision new Kafka cluster(s) with security features - https://phabricator.wikimedia.org/T152015#3831643 (10Nuria) [17:11:32] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Write generic certificate management software for use with Puppet and Self Signing CAs. - https://phabricator.wikimedia.org/T166167#3831642 (10Nuria) 05Open>03Resolved [17:12:11] 10Analytics-EventLogging, 10Analytics-Kanban: Find an alternative query interface for eventlogging on analytics cluster that can replace MariaDB - https://phabricator.wikimedia.org/T159170#3831650 (10Nuria) [17:12:13] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Implement EventLogging Hive refinement - https://phabricator.wikimedia.org/T162610#3831649 (10Nuria) 05Open>03Resolved [17:12:26] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Alpha release: Wikistats 2 UI feedback From Erik Z - https://phabricator.wikimedia.org/T178084#3831652 (10Nuria) 05Open>03Resolved [17:12:39] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Alpha Release: Breakdowns don't work in Firefox - https://phabricator.wikimedia.org/T180556#3831653 (10Nuria) 05Open>03Resolved [17:18:51] (03PS2) 10Jforrester: [WIP] Analyze external link insertion and deletion [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/301432 (https://phabricator.wikimedia.org/T115119) (owner: 10Milimetric) [17:19:18] (03CR) 10Jforrester: "Is this something that I can help with?" [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/301432 (https://phabricator.wikimedia.org/T115119) (owner: 10Milimetric) [17:31:26] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: kafka1018 fails to boot - https://phabricator.wikimedia.org/T181518#3831700 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` kafka1023.eqiad.wmnet ``` The log can... [17:31:54] 10Analytics-Kanban, 10Analytics-Wikistats: Add link to new wikistats 2.0 to wikistats 1.0 pages - https://phabricator.wikimedia.org/T182001#3831701 (10Nuria) Ping @Erik_Zachte could we change text by this thursday (actually wednesday so it is done by Thursday morning?) cc @JAllemandou @Milimetric @fdans [17:39:02] 10Analytics-Kanban, 10Analytics-Wikistats: The laptop attempts a vertical take-off when loading ii.wikipedia.org - https://phabricator.wikimedia.org/T182700#3831713 (10Milimetric) [17:48:35] 10Analytics-Kanban, 10Analytics-Wikistats: Privacy pageview threshold for map report - https://phabricator.wikimedia.org/T181508#3831742 (10fdans) @Erik_Zachte is this a fair summary of the restrictions in WiViVI? - For a Wikipedia to be shown, it has to have a minimum of 0.1% of all traffic in pageviews. - D... [17:50:27] 10Analytics-Kanban, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db104[67] - https://phabricator.wikimedia.org/T181784#3802296 (10Cmjohnson) [18:02:36] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: kafka1018 fails to boot - https://phabricator.wikimedia.org/T181518#3831798 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['kafka1023.eqiad.wmnet'] ``` Of which those **FAILED**: ``` ['kafka1023.eqiad.wmnet']... [18:06:23] ottomata: there are some issues with jessie reimages atm, so kafka1023 will probably be completed tomorrow [18:06:35] hopefully it will be ready to become a broker when you'll be online [18:10:08] ok cool [18:19:48] joal: are you up for writing a wikistats blogpost for next week? [18:20:12] joal: ok to say no, if we aim to have it next week we should probably have some copy by Friday [18:23:48] * elukey off! [18:32:13] milimetric: OMG ii.wikipedia.org [18:32:19] milimetric: we shall never know [18:34:02] nuria_: oh no, I think I know what's wrong, I'll find it and fix it. I think the project is closed, but it still shouldn't kill your computer like that. [18:36:46] milimetric: ok, https://stats.wikimedia.org/v2/#/garbage.wikimedia.org doesn't kill anything [18:37:07] milimetric: 3G of memeory at the time of me closing out the tab [18:37:47] milimetric: but also garbage wikipedia has that problem too, GOOD find [18:38:04] yeah, it’s an infinite loop in wiki selector or something. Interesting idea, garbage, thanks! [18:41:24] milimetric: i think you need to disable some widgets to even load devtools [18:52:36] Hey nuria_ - I'll happily try to write for next friday [18:52:53] joal: please let it not be a strech eh? [18:52:57] it can wait [18:53:20] nuria_: no prob, I have nothing urgent now, so that will be good :) [19:03:41] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Support multi DC statsv - https://phabricator.wikimedia.org/T179093#3831963 (10Ottomata) [19:57:27] have a dentist appointment, will be back to solve this loop [19:57:58] nuria_ / fdans: I found the bug, it's an infinite loop between initWithCurrentProject and close on WikiSelector (in case I get hit by a bus) [19:58:25] milimetric: OK, we will KEEP THIS INFO SAFE! [19:59:01] lol [21:22:35] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Support multi DC statsv - https://phabricator.wikimedia.org/T179093#3832479 (10Ottomata) Wow ok, had so much discussion in IRC today with folks, especially with @BBlack https://gerrit.wikimedia.org/r/#/c/391705/ is updated with the results of tha... [21:49:34] tgr: for some reason this ticket is closed to comments [21:49:40] https://phabricator.wikimedia.org/T181952 [21:57:48] 10Analytics, 10Analytics-Wikistats: SEO-friendly HTML titles for Wikistats 2.0 - https://phabricator.wikimedia.org/T182718#3832561 (10DarTar) [22:51:38] 10Analytics-Kanban, 10Analytics-Wikistats: Add link to new wikistats 2.0 to wikistats 1.0 pages - https://phabricator.wikimedia.org/T182001#3832722 (10Erik_Zachte) Yes I can update the text tomorrow. One comment on the new landing page: The team is currently updating Wikistats 1 - dump based info (edits,cont... [22:56:22] 10Analytics-Kanban, 10Analytics-Wikistats: Add link to new wikistats 2.0 to wikistats 1.0 pages - https://phabricator.wikimedia.org/T182001#3832733 (10Nuria) Ayayaya I do not understand [22:59:48] 10Analytics-Kanban, 10Analytics-Wikistats: Add link to new wikistats 2.0 to wikistats 1.0 pages - https://phabricator.wikimedia.org/T182001#3832748 (10Erik_Zachte) https://wikitech.wikimedia.org/wiki/Analytics/Systems/Wikistats briefly mentions the data lake and edits, but 90% of the page is about pageviews an...