[00:02:25] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2266204 (DannyH) @Majora, @Josve05a and @tom29739: Don't worry, the Fundraising team has... [03:00:42] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2266371 (Earwig) Agreed, we can handle this without panic. [05:43:59] Analytics-Kanban, Operations, Patch-For-Review: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#2266526 (elukey) [06:18:08] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Use MySQL as Hue data backend store - https://phabricator.wikimedia.org/T127990#2266542 (elukey) [06:23:39] Analytics-Cluster, EventBus, Operations, Services: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#2266566 (elukey) [06:23:48] Analytics-Kanban, DC-Ops, EventBus, MediaWiki-Cache, and 5 others: setup kafka2001 & kafka2002 - https://phabricator.wikimedia.org/T121558#2266565 (elukey) Open>Resolved [06:59:28] Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: access for amire80 to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T122524#2266608 (Amire80) [08:26:36] Yay Andrew! new domain :) [08:31:00] Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2266733 (JAllemandou) Good for me! @Krenair: Thanks for caring about the definition :) For more details on the precise domain topic, see https://meta.wikimedia.org/wiki/Research:Page_... [08:31:13] Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2266734 (JAllemandou) Open>Resolved [09:21:18] (CR) Joal: "Added doc to wikitech" [analytics/refinery] - https://gerrit.wikimedia.org/r/286471 (owner: Joal) [09:22:06] elukey: here ? [09:22:13] o/ [09:22:18] Hey ! Hi mate :) [09:23:02] elukey: Do you mind merging some changes for me to deploy them (don't like self-merging too much ;) [09:23:53] which ones? [09:24:01] first: https://gerrit.wikimedia.org/r/#/c/285400/ [09:24:15] second: https://gerrit.wikimedia.org/r/#/c/285998/ [09:24:30] third: https://gerrit.wikimedia.org/r/#/c/286471/ [09:31:38] joal: I think that you are free to self merge! you got ottomata's +1 and everything looks good [09:31:42] what is your concern? [09:31:51] no concern, just don't like to do it :) [09:32:01] I mean, you are not pushing code like crazy without any review :) [09:32:11] True ! [09:32:18] if you got already the mmmmmmm ok [09:32:21] Ok, I'll go for it then [09:32:23] you are fine :) [09:32:29] hehehe [09:40:45] (CR) Joal: [C: 2 V: 2] "Self-merging for deploy." [analytics/refinery] - https://gerrit.wikimedia.org/r/285400 (https://phabricator.wikimedia.org/T130732) (owner: Joal) [09:41:05] (CR) Joal: [C: 2 V: 2] "Self-merging for deploy." [analytics/refinery] - https://gerrit.wikimedia.org/r/285998 (https://phabricator.wikimedia.org/T130731) (owner: Joal) [09:41:35] (CR) Joal: [C: 2 V: 2] "Self-merging for deploy." [analytics/refinery] - https://gerrit.wikimedia.org/r/286471 (owner: Joal) [09:43:59] (PS1) Joal: Update oozie diagram removing refine step [analytics/refinery] - https://gerrit.wikimedia.org/r/287067 [09:48:17] (PS2) Joal: Update oozie diagram [analytics/refinery] - https://gerrit.wikimedia.org/r/287067 [09:48:44] !log deploying refinery from tin [09:48:56] eqi tin [09:49:03] oops :) [09:51:16] !log deploy refinery onto hdfs [09:53:38] elukey: If you have a minute, would you mind douvle checking that this makes sense: https://wikitech.wikimedia.org/w/index.php?title=Analytics%2FCluster%2FOozie&type=revision&diff=486127&oldid=474887 [09:55:49] joal: reading :) [09:59:11] hi team! [10:00:19] looks good! [10:07:12] Cool :) Thanks elukey ! [10:07:16] Hi mforns :) [10:20:23] * elukey lunch! [10:28:57] !log Pause hadoop job to restart in clean mode [10:40:12] (PS7) Mforns: Fix issues in metrics-by-project breakdown patterns [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria) [10:48:52] (CR) Mforns: "I managed to remove the dictionary and use an array instead. Thus, the breakdownId is not needed any more, and the code is a lot cleaner i" (3 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria) [11:33:13] !log restarting every analytics oozie job [11:39:05] \o/ [12:23:36] (PS1) Joal: Correct bug in oozie rerun script [analytics/refinery] - https://gerrit.wikimedia.org/r/287075 [12:52:09] (CR) Elukey: [C: 1] "Looks good from the python and logic side, but I have to admit that my lack of experience with Oozie prevents me to understand the whole d" [analytics/refinery] - https://gerrit.wikimedia.org/r/287075 (owner: Joal) [12:53:18] (CR) Elukey: [C: 1] Update oozie diagram [analytics/refinery] - https://gerrit.wikimedia.org/r/287067 (owner: Joal) [13:56:05] hey!! [13:56:26] I'm around, catching up [13:56:40] but lemme know if there's anything urgent I should look at first [13:56:53] cc: a-team [13:57:35] hey heyyYy [13:57:35] :) [13:57:40] welcome back, what did you learn? [14:03:29] milimetric: helloooo [14:03:40] hi! :) [14:03:51] I learned... ummm [14:04:03] how to relax again [14:04:48] I was wound up a lot tighter than I thought [14:26:12] yooo elukey gonna pick a day for analytics kafka cluster upgrade next week [14:26:31] mayybee wednesday my morning? [14:26:32] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2267140 (Josve05a) >>! In T125459#2266204, @DannyH wrote: > @Majora, @Josve05a and @tom29... [14:28:21] Hi milimetric ! [14:28:25] Welcome back :) [14:28:51] thanks :) [14:29:04] ottomata: +1 :) [14:29:06] it's great to be back [14:29:09] Good to see you around :) [14:30:26] ok cool, elukey if that works for you i'll email and schedule it then [14:30:31] 9am my time wed morning [14:32:07] hola milimetric ! [14:32:16] hi nuria :) [14:32:31] nothing like the fresh smell of the 500+ e-mails expecting to be read ... [14:32:38] I'm all caught up on emails!! [14:32:48] but I have 200+ phabricator notifications [14:33:03] and reviews and stuff like that [14:33:03] wow milimetric, impressive ! [14:33:14] Hi nuria_ [14:33:19] hola [14:33:22] well, I've been reading every day, otherwise it would take me literally a month :) [14:33:40] nuria_: Do we take a few minutes on whitelist stuff ? [14:33:46] Seems still wrong to me :( [14:33:46] ya you are right, you will never be able to work again [14:34:17] milimetric: I'm between liking your way, and prefering to leave it behind [14:34:19] joal: sure, let me look at it with a different editor [14:36:19] elukey: forgot to congratulate you on the hue-mysql thing :) [14:36:35] joal: resubmitting [14:36:37] elukey: So here it is ! [14:36:45] elukey: holy moly that is fast now!!! [14:36:46] nice job [14:38:09] (PS5) Nuria: Adding jam.wikipedia to domains for which we count pageviews [analytics/refinery] - https://gerrit.wikimedia.org/r/286672 (https://phabricator.wikimedia.org/T134279) [14:39:25] (PS6) Joal: Adding jam.wikipedia to domains for which we count pageviews [analytics/refinery] - https://gerrit.wikimedia.org/r/286672 (https://phabricator.wikimedia.org/T134279) (owner: Nuria) [14:39:42] joal: thanks! all credits to the usual mmmmmmmmmm suggestions from ottomata :) [14:39:45] (CR) Joal: [C: 2 V: 2] "Yay ! Works for me." [analytics/refinery] - https://gerrit.wikimedia.org/r/286672 (https://phabricator.wikimedia.org/T134279) (owner: Nuria) [14:39:49] huhu :) [14:39:55] The hmmmm power ! [14:40:03] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2267191 (Compassionate727) >>! In T125459#2267140, @Josve05a wrote: >>>! In T125459#22662... [14:40:38] huh the hue on mysql made hue faster?! [14:40:43] cool! that is an unexpected bonus! [14:42:07] (CR) Joal: [C: 2 V: 2] "Self merging for deploy." [analytics/refinery] - https://gerrit.wikimedia.org/r/287075 (owner: Joal) [14:44:17] (CR) Joal: [C: 2 V: 2] "Self merging for deploy" [analytics/refinery] - https://gerrit.wikimedia.org/r/287067 (owner: Joal) [14:44:49] !log deploying refinery from tin [14:47:16] !log deploying refinery on hdfs [14:53:57] git log [14:54:00] oops [14:55:11] !log restarted pageview job [14:55:52] Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2267245 (Cmjohnson) [14:59:25] joal: did you just restart it with your script? [14:59:33] ottomata: YESIR ! [14:59:48] awesome! [15:00:14] ottomata: I didn't use the full restart in real mode, only in dry-run mode, but seems to work fine :) [15:00:35] * joal is happy to have stopped procrastinating on that matter [15:00:49] * joal will NEVER AGAIN restart all jobs manually :) [15:00:50] joal: milimetric, weigh in here please [15:00:50] https://phabricator.wikimedia.org/T133785 [15:00:55] i think they might be waiting on us [15:01:37] nice, that knocked off 30 phab notifications :) [15:05:03] Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2267254 (JAllemandou) @ottomata: 6 instances with 4 disks each in RAID 0 works for me. As you said, 1 lost over 6 is acceptable, and having 6.5Tb per instance seems fine about empty spac... [15:05:07] ottomata: --^ [15:05:54] joal: so we're not getting SSDs any more and the other three machines are sticking around? [15:05:58] this is all news to me [15:06:21] milimetric: I'll fast track you: 6 * 1.6Tb SSDs per machine [15:06:54] oh wow! [15:07:15] +1 then, but how come we get to keep an100[1-3]? [15:07:30] The plan was originally to have them in RAID10 (6.5 Tb usable, and redundancy), and currently leading toward using only RAID0, and have 2 cassandra instances per node [15:07:37] joal: 8 * 1.6 [15:07:46] milimetric: you don't get to keep them :p [15:07:50] ottomata, milimetric Yes, sorry [15:08:06] I am disturb by the 6.4Tb and the 8 SSDs ;) [15:08:09] oh, so why even talk about 6 instances [15:08:20] milimetric: RAM limitation [15:08:30] it'll be 3 instances after the upgrade, right? [15:08:46] milimetric: 2 casandra instances per node :) [15:08:52] gotcha [15:08:53] ok [15:08:56] Sorry, this one was the last bit [15:09:25] sounds good [15:09:52] We even wondered with elukey if we might get to 4 instances per node (reducing the amount of data to be handled by a single node), but there are RAM limitations, and 2 seems better says our cassandra expert [15:11:18] milimetric: another thing: we'll take advantage of having to copy all of our data onto new instances to double check our compaction settings [15:11:36] milimetric: The first stages of the migration is about testing some settings on "fake data", then migrate with the best setting found [15:11:54] milimetric: elukey has a plan :) [15:12:00] cool, great [15:12:21] hi milimetric :] [15:12:25] milimetric: details on the plan : https://etherpad.wikimedia.org/p/analytics-aqs-cassandra [15:12:31] milimetric: I stop flooding you ;) [15:12:35] konichiwa! [15:13:00] haha, hi mforns [15:13:12] :] [15:16:16] I'm gonna go grab some lunch before meetings start [15:16:29] are people going to the monthly lunch thing? Or are we just doing our regular meetings? [15:18:15] Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2267331 (Ottomata) Ok then, unless there are objections, let's go with that. Since they are mounting cassandra stuff under `/srv` elsewhere, let's do that here too. |mount|disks|raid l... [15:18:41] I was thinking to do the regular meetings, but we didn't talk about this.. [15:19:05] me too, was aiming for standup [15:19:13] oh and goal checkup i think is important [15:19:56] a-team, was about to post that I'll miss second part of tasking (after gaols checkup) for caring Lino [15:20:42] np joal [15:45:37] cool, I'd rather regular meetings too [15:47:23] Analytics-Cluster, Operations, ops-eqiad: Analytics hosts showed high temperature alarms - https://phabricator.wikimedia.org/T132256#2267422 (Cmjohnson) @elukey: re-applying thermal paste is needed. There has been several servers that have required lately and it appears to have fixed the issue. [15:57:08] milimetric: regular MEETINGS [15:57:16] sorry, argh, "meetings" [15:57:51] Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2267453 (Cmjohnson) [16:02:29] Analytics-Kanban, Patch-For-Review: Make webrequest load and refine jobs a single bundle - https://phabricator.wikimedia.org/T130731#2267476 (JAllemandou) [16:02:32] Analytics-Kanban: Standardise naming in oozie jobs (particularly for top level ones) - https://phabricator.wikimedia.org/T130732#2267477 (JAllemandou) [16:02:34] Analytics-Cluster, Analytics-Kanban: Standardize use of refinery_path over oozie_path in all refinery oozie property files - https://phabricator.wikimedia.org/T133206#2267479 (JAllemandou) [16:02:45] Analytics-Kanban: Ease restarting and backfilling of jobs in cluster {hawk} - https://phabricator.wikimedia.org/T115985#2267480 (JAllemandou) [16:05:41] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Puppetize and make useable confluent kafka packages - https://phabricator.wikimedia.org/T132631#2267488 (Ottomata) [16:06:53] Analytics-Kanban, Patch-For-Review: Count jam.wikipedia pageviews - https://phabricator.wikimedia.org/T134279#2267490 (JAllemandou) [16:07:05] Analytics-Kanban, Patch-For-Review: Count jam.wikipedia pageviews - https://phabricator.wikimedia.org/T134279#2260635 (JAllemandou) a:Nuria [16:27:19] Analytics-Kanban: Propose evolution of Mediawiki EventBus schemas to match needed data for Analytics need - https://phabricator.wikimedia.org/T134502#2267543 (JAllemandou) [16:27:35] Analytics-Kanban: Propose evolution of Mediawiki EventBus schemas to match needed data for Analytics need - https://phabricator.wikimedia.org/T134502#2267556 (JAllemandou) p:Triage>Normal [16:27:46] Analytics-Kanban: Propose evolution of Mediawiki EventBus schemas to match needed data for Analytics need - https://phabricator.wikimedia.org/T134502#2267543 (JAllemandou) [16:27:53] Analytics-Kanban: Examine wikistats reports, make a summary of the most granular data needed that would serve all reports - https://phabricator.wikimedia.org/T131783#2267560 (JAllemandou) [16:30:24] a-team: let's met in batcave [16:31:13] elukey: goals /tasking? [16:31:17] elukey: in batcave [16:31:51] yepp [17:14:20] Analytics: Productionitize druid - https://phabricator.wikimedia.org/T131974#2267626 (Nuria) [17:14:22] Analytics: Prototype Data Pipeline on Druid - https://phabricator.wikimedia.org/T130258#2267625 (Nuria) [17:15:05] Analytics: Create debian packages for druid - https://phabricator.wikimedia.org/T134503#2267631 (Nuria) [17:15:23] Analytics: Prototype Data Pipeline on Druid - https://phabricator.wikimedia.org/T130258#2131128 (Nuria) [17:15:25] Analytics: Create debian packages for druid - https://phabricator.wikimedia.org/T134503#2267644 (Nuria) [17:16:05] Analytics-Kanban: Create debian packages for druid - https://phabricator.wikimedia.org/T134503#2267631 (Nuria) [17:23:29] Analytics-Kanban: Create debian packages for druid - https://phabricator.wikimedia.org/T134503#2267668 (Nuria) Dependencies: - Some of them already have debian packages, use those when existing (and versions match) - For deps that do not have debian packages we can either build the packages or copy them to... [17:24:29] Analytics: Puppetize druid - https://phabricator.wikimedia.org/T131974#2267671 (Nuria) [17:25:43] https://phabricator.wikimedia.org/T134426 [17:32:04] Analytics: Puppetize druid - https://phabricator.wikimedia.org/T131974#2267693 (Nuria) This can be done even if the absence of new hardware. How is the documentation? This would take more or less time depending on the amount of processes that we have to set up and daemons that are running. For example:... [17:34:53] Analytics-Kanban: Puppetize druid - https://phabricator.wikimedia.org/T131974#2184741 (Nuria) [17:36:04] Analytics: Having index page on analytics.wikimedia.org - https://phabricator.wikimedia.org/T134506#2267717 (Nuria) [17:40:21] Analytics: Having index page on analytics.wikimedia.org - https://phabricator.wikimedia.org/T134506#2267751 (Nuria) Index page: * will have links like: https://analytics.wikimedia.org/reports/browsers * will have a notice about wikistats 2.0 The index page lives in puppet for now. [17:53:20] Analytics: Deploy browsers reports to https://analytics.wikimedia.org/reports/browsers - https://phabricator.wikimedia.org/T134510#2267805 (Nuria) [17:55:49] Analytics: create repo analytics.wikimedia.org with index and build of browser reports - https://phabricator.wikimedia.org/T134506#2267824 (Nuria) [17:56:24] Analytics: Create repo analytics.wikimedia.org with index and build of browser reports for puppet to source - https://phabricator.wikimedia.org/T134506#2267717 (Nuria) [17:56:51] Analytics: Deploy browsers reports to analytics.wikimedia.org using fab - https://phabricator.wikimedia.org/T134510#2267839 (Nuria) [17:57:15] Analytics: Create repo analytics.wikimedia.org with index and build of browser reports for puppet to source and deploy to analytics.wikimedia.org - https://phabricator.wikimedia.org/T134506#2267717 (Nuria) [18:03:39] going offline a-team! byeee [18:04:21] https://analytics.wikimedia.org/ bah hhhahhaha [18:04:23] DONUT [18:05:09] if no one objects, i'm going to bounce cassandra-metrics-collector in the AQS cluster to apply a new version [18:06:45] it should be a no-op, but i'm never super keen on having a new version of software, waiting to be applied on an unattended restart at some point in the future, when everyone has forgotten :) [18:11:22] or maybe not... i don't have the karma [18:16:41] Analytics, cassandra: AQS Cassandra cluster: Restart cassandra-metrics-collector - https://phabricator.wikimedia.org/T134513#2267883 (Eevans) [18:18:24] Analytics, cassandra: AQS Cassandra cluster: Restart cassandra-metrics-collector - https://phabricator.wikimedia.org/T134513#2267910 (Eevans) [18:18:36] Analytics, cassandra: AQS Cassandra cluster: Restart cassandra-metrics-collector - https://phabricator.wikimedia.org/T134513#2267883 (Eevans) p:Triage>High [18:37:01] Analytics, Commons, Multimedia, Wikidata, and 3 others: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2268041 (Yurik) [18:38:28] Analytics, Commons, Multimedia, Wikidata, and 3 others: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#1860168 (Yurik) I removed T124569 because with T134426 this task could be marked as done (unless we want... [18:47:58] ottomata, xD [18:51:48] Analytics: Pageview API: Limit (and document) size of data you can request - https://phabricator.wikimedia.org/T134524#2268111 (Nuria) [18:55:50] mforns: ? [18:57:44] Analytics, EventBus, Wikimedia-Stream: Public Event Streams - https://phabricator.wikimedia.org/T130651#2268152 (Ottomata) @Krinkle FYI we are slightly deprioritizing this for this quarter in favor of some other goals. I will probably still work on it some, and would like to sync up with you at some... [19:11:37] (CR) Nuria: [V: 2] "Looks good, I think we should merge let me know otherwise." [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria) [19:12:43] (CR) Milimetric: [C: 2] "good to merge. I have a few small style improvements but I'll save those for later." [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria) [19:17:07] mforns, milimetric : will be merging last patch for the breakdown issue and deploying to vital signs [19:17:17] cool [19:18:20] nuria_, milimetric ok [19:29:50] !log deployed latest master candidate to vital-signs on labs [19:38:19] thx nuria_1 [19:38:20] ! [20:47:35] Analytics, Operations, Traffic, Privacy: Connect Hadoop records of the same request coming via different channels - https://phabricator.wikimedia.org/T113817#2268432 (chasemp) p:Triage>Normal [21:14:01] milimetric: Hiii sorry today has been crazy - I'm happy to have you back :) we all missed you! [21:52:37] Analytics, Operations, Traffic, Privacy: Connect Hadoop records of the same request coming via different channels - https://phabricator.wikimedia.org/T113817#2268606 (BBlack) IMHO, we should define better what we need. There's a lot of grey area and disagreement in the discussion so far. Genera... [22:16:58] Analytics-Kanban: Create debian packages for druid - https://phabricator.wikimedia.org/T134503#2268643 (Ottomata) [22:17:29] Analytics-Kanban: Create debian packages for druid - https://phabricator.wikimedia.org/T134503#2267631 (Ottomata) WIP here: https://github.com/ottomata/druid-debian I decided it'd be much easier to start with the prebuilt tarball. Not sure if Ops will let me get away with this, but it will be WAY easier.... [22:17:34] madhuvishy: Hey, you're in the office today right? Could I get your input on something quickly, in 5 minutes or so? [22:17:41] Analytics-Kanban: Create debian packages for druid - https://phabricator.wikimedia.org/T134503#2268649 (Ottomata) p:Triage>Normal a:Ottomata [22:18:38] Deskana: aaah I had to leave early. Will irc work? [22:21:33] madhuvishy: Sure, I'll ping you about it in a minute. :-)