[06:51:08] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3750027 (10Marostegui) After the BBU re-learn: ``` ˜/icinga-wm 7:50> RECOVERY - MegaRAID on db1046 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy ``` [07:02:29] 10Quarry, 10DBA, 10Data-Services, 10Patch-For-Review: Raise concurrent mysql connection limit for Quarry (or throttle application concurrency) - https://phabricator.wikimedia.org/T180141#3750037 (10Marostegui) 05Open>03Resolved a:03Marostegui Done. [10:45:49] the druid prometheus exporter is almost read [10:45:51] *ready [10:45:55] testing it now in labs [10:46:02] It sounds like the [kids] prefix/notice is being widely adopted so I will set a filter accordingly, and I look forward to the day when I'm comfortable deleting it. [10:46:08] oops [10:46:16] hello joal :) [10:46:19] man - that middle click thing ... [10:46:22] Hi elukey o/ [10:47:15] That thread initiated from Dan was really touching - I love that part of the WMF culture [10:48:19] yes definitely, me too [11:49:58] (03PS2) 10Joal: Add clickstream oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) [12:26:39] * elukey lunch! [13:21:08] taking a break a-team [14:13:45] joal: whenever you have time I'd need to figure out how to generate broker/coordinator metrics in druid :) [14:13:49] (in labs to test it out) [15:28:01] hi elukey yt? :] can you remind me of the date the EL purging finished, to compare purged data with not purged? [15:28:45] mforns: o/ [15:28:54] up to 2016-06-21 [15:29:03] don't know exactly the hour though [15:29:09] is it ok? [15:29:38] elukey, thanks a lot! [15:29:50] mforns: if you want to pair and/or have any info ping me! [15:29:55] k [15:36:26] 10Analytics-Kanban: Geowiki stopped updating on October 24th - DATA LOSS (read comments) - https://phabricator.wikimedia.org/T179952#3751240 (10Milimetric) [15:39:38] joal: do yo happen to know: where is the yarn spark distributed cache (per job?) [15:39:53] i'm trying to see what spark2 uploads to yarn when it doesn't have spark.yarn.jars set [15:41:45] OK NM I FOUND IT [15:41:48] google did not help [15:41:56] but ls -a in my /user/otto dir did! [15:42:02] .sparkStaging/ :) [15:45:55] ottomata: hiiiiiiiiii [15:46:12] (whenever you have time) - would something like https://gerrit.wikimedia.org/r/#/c/390419 be ok? (also repeated for the other daemons [15:46:28] Heya ottomata [15:46:39] Thanks for having found- I actually never checked :) [15:46:45] pcc https://puppet-compiler.wmflabs.org/compiler02/8733/ [15:47:05] Heya elukey - sorry I'm late, do you want us to have a look at metrics after standup? [15:47:39] joal: sure, atm they seems good but I haven't got any broker/coordinator metric (probably because of inactivity) [15:48:28] elukey: I can simulate broker activity (do some queries [15:48:42] elukey: for coordinator, I think it'll involve indexing new segments [15:50:23] yeah, it would be great to learn some stuff about that :) [15:51:25] elukey: My post standup time is for you ;) [15:52:46] * elukey imagines Joseph singing "Get up, stand-up.. stand-up for your rights.." [15:57:33] 10Analytics-Kanban: Geowiki stopped updating on October 24th - DATA LOSS (read comments) - https://phabricator.wikimedia.org/T179952#3751304 (10Milimetric) TL;DR; We lost 11 days of data, and if we wait until the recentchanges data is deleted, the loss will be permanent. We may want to migrate to Hadoop before... [15:59:06] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), and 2 others: Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#3751306 (10mforns) @phuedx No, all data will be purged the same. A thing we could do, if you want... [16:18:08] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), and 2 others: Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#3751414 (10phuedx) @mforns: That sounds good. Any objections @Tbayer? [16:21:16] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), and 2 others: Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#3751418 (10mforns) @phuedx and @Tbayer But consider this only for the sake of consistency. Regardi... [16:22:40] elukey: I forgot we have a meeting now - I'll ping you when its finished [16:23:23] ack! [16:38:54] Ok elukey - nobody in meeting [16:38:58] elukey: batcave? [16:40:50] joal: sure! [17:01:09] (03PS3) 10Joal: Add clickstream oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) [17:01:54] (03CR) 10Joal: "Tested as is - works for the 5 choosen wikis :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/390226 (https://phabricator.wikimedia.org/T175844) (owner: 10Joal) [17:02:07] Hey nuria_ -- If you don't know what to do today ;) --^ [17:19:22] joal: the json field is dataSource not datasource, this was the issue [17:19:27] * elukey cries in a corner [17:19:43] :( [17:19:53] it is also interesting that druid sends alerts via json posts! [17:21:54] {u'feed': u'alerts', u'severity': u'component-failure', u'service': u'druid/overlord', u'timestamp': u'2017-11-10T17:21:33.423Z', u'host': u'druid-test02.analytics.eqiad.wmflabs:8090', u'data': {u'exceptionMessage': u'java.sql.SQLException: Cannot create PoolableConnectionFactory (Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago etc.. [17:25:05] joal: i have those on my queue, i think i will not get to them until monday [17:25:16] no prob nuria_ [17:25:24] just leting you know [17:29:53] ping fdans mforns map meeting on batcave or ,eeting link? [17:29:58] *meeting link? [17:30:15] I'm in the caviar [17:30:20] cave? although not sure dan is coming [18:02:28] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3033748 (10Ottomata) [18:30:48] * elukey off! [19:24:09] 10Analytics, 10Analytics-Wikistats: Wikistats2: The granularity selector does not work for tops metrics - https://phabricator.wikimedia.org/T180266#3751801 (10mforns) [20:51:43] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Make Qgil a fallback for Bitergia access (lock-in) - https://phabricator.wikimedia.org/T178381#3751895 (10Qgil) a:05Aklapper>03Qgil [21:03:13] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Advertise wikimedia.biterg.io more widely in the Wikimedia community - https://phabricator.wikimedia.org/T179820#3737130 (10Qgil) > ...once our worst data quality issues like T157898 are sorted out. Should those tasks be actual blockers... [21:36:48] 10Analytics: RStudio web version on SWAP - https://phabricator.wikimedia.org/T180270#3751938 (10mpopov)