[07:21:42] (03PS1) 10Joal: Update MWH-reduced to parquet storage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/441341 (https://phabricator.wikimedia.org/T192483) [07:36:07] (03PS2) 10Joal: Update sqoop script to include jar generation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/440382 (https://phabricator.wikimedia.org/T196912) [07:36:27] (03PS4) 10Joal: Add oozie jobs loading druid daily uniques monthly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/436826 [07:38:03] (03PS6) 10Joal: Add MediawikiHistoryChecker spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/439869 (https://phabricator.wikimedia.org/T192481) [07:42:09] (03PS4) 10Joal: Add validation step in mediawiki-history jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/440005 (https://phabricator.wikimedia.org/T192481) [11:12:00] PROBLEM - Number of segments reported as unavailable by the Druid Coordinators of the Analytics cluster on einsteinium is CRITICAL: 857 gt 200 https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&panelId=46&fullscreen&orgId=1&var-cluster=druid_analytics&var-druid_datasource=All [11:19:06] this is joal afaics :) [11:19:16] I need to fix this alarm now [11:24:46] elukey: indeed it was me [11:24:50] elukey: sorry about that :( [11:25:27] nope! It is my fault, the alert is not good [11:25:29] I just fixed it [11:25:54] it alarms only if after one hour the overall segments unavail are more than 200 [11:26:14] Thanks elukey - Sorry for the false alarm :( [11:26:54] (03CR) 10Joal: [V: 031] "Tested on cluster" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/441341 (https://phabricator.wikimedia.org/T192483) (owner: 10Joal) [11:27:01] feel free to test anything joal, I see that you are playing with parquet :D [11:27:06] \o/ [11:27:20] :) [11:37:43] RECOVERY - Number of segments reported as unavailable by the Druid Coordinators of the Analytics cluster on einsteinium is OK: (C)200 gt (W)180 gt 27 https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&panelId=46&fullscreen&orgId=1&var-cluster=druid_analytics&var-druid_datasource=All [11:47:22] (03PS1) 10Joal: Updating MediawikiHistoryChecker for reduced [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/441378 (https://phabricator.wikimedia.org/T192483) [11:58:14] (03PS2) 10Joal: Update MWH-reduced to parquet storage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/441341 (https://phabricator.wikimedia.org/T192483) [14:41:08] (03CR) 10Nuria: [C: 031] "Have we tested that the parquet data indexantion works as advertised?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/441341 (https://phabricator.wikimedia.org/T192483) (owner: 10Joal) [15:44:39] (03CR) 10Joal: "> Patch Set 2: Code-Review+1" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/441341 (https://phabricator.wikimedia.org/T192483) (owner: 10Joal) [16:00:59] ping ottomata [16:04:58] 10Analytics-Kanban: Unique druid segment compaction - https://phabricator.wikimedia.org/T197885#4305708 (10JAllemandou) [16:05:11] 10Analytics-Kanban: Unique druid segment compaction - https://phabricator.wikimedia.org/T197885#4305720 (10JAllemandou) [16:06:33] (03PS5) 10Joal: Add oozie jobs loading druid daily uniques monthly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/436826 (https://phabricator.wikimedia.org/T197885) [16:34:53] (03CR) 10Sahil505: "@Nuria: It works with npm run build and with the changes that I've made." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/441037 (https://phabricator.wikimedia.org/T197482) (owner: 10Sahil505) [16:36:56] 10Analytics, 10Pageviews-API, 10Tool-Pageviews: Pageviews agent=bot is always 0 - https://phabricator.wikimedia.org/T197277#4305773 (10Ottomata) p:05Triage>03Normal [16:38:41] 10Analytics: turnilo x axis improperly labeled - https://phabricator.wikimedia.org/T197276#4284396 (10Ottomata) HMmm https://github.com/allegro/turnilo/issues/105 [16:39:03] 10Analytics: turnilo x axis improperly labeled - https://phabricator.wikimedia.org/T197276#4305778 (10Ottomata) p:05Triage>03Normal [16:40:40] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4305786 (10Ottomata) p:05High>03Triage [16:40:43] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4300251 (10Ottomata) p:05Triage>03High [16:41:56] 10Analytics, 10Analytics-Dashiki, 10Story: EEVSUser selects ALL wikis - https://phabricator.wikimedia.org/T70478#4305798 (10Ottomata) 05Open>03Invalid [16:41:58] 10Analytics, 10Analytics-Wikimetrics, 10Story: Story: WikimetricsUser runs report against all wikis - https://phabricator.wikimedia.org/T70477#4305799 (10Ottomata) [16:42:23] 10Analytics, 10Analytics-Wikimetrics, 10Story: Story: WikimetricsUser runs report against all wikis - https://phabricator.wikimedia.org/T70477#4305802 (10Ottomata) 05Open>03Invalid [16:42:33] 10Analytics, 10Analytics-Wikimetrics, 10Story: Story: WikimetricsUser runs report against all wikis - https://phabricator.wikimedia.org/T70477#4305803 (10Ottomata) 05Invalid>03Resolved [16:46:00] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4305808 (10elukey) a:05elukey>03None [16:47:00] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T197707#4300251 (10elukey) I had a chat with Chris today and the 2T disk should do just fine. Removing myself as assignee to let DC-Ops handling the hw swap. Thanks! [16:52:18] joal: https://gist.github.com/ottomata/6b0af13856383acb55214c5ddaef80e3 [17:23:44] 10Analytics: Drop mediawiki history old snapshots from druid public cluster - https://phabricator.wikimedia.org/T197889#4305928 (10Nuria) [17:23:46] 10Analytics: Drop old mediawiki_history_reduced snapshots - https://phabricator.wikimedia.org/T197888#4305918 (10JAllemandou) [17:24:25] 10Analytics: Drop mediawiki history old snapshots from druid public cluster - https://phabricator.wikimedia.org/T197889#4305941 (10Nuria) [17:33:01] 10Analytics, 10Analytics-Wikistats: When cursor is out of graph overlay should not display - https://phabricator.wikimedia.org/T192416#4305958 (10sahil505) [17:33:38] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add data-quality check on mediawiki-history-reduced before druid indexation - https://phabricator.wikimedia.org/T192483#4305960 (10Nuria) Operational changes that go with this change: - convert existing data (json) into parquet -... [17:34:16] (03CR) 10Nuria: [V: 032 C: 032] "Merging this one, please see operational changes needed for this work to take effect:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/441341 (https://phabricator.wikimedia.org/T192483) (owner: 10Joal) [17:44:03] (03CR) 10Nuria: [V: 032] Update sqoop script to include jar generation (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/440382 (https://phabricator.wikimedia.org/T196912) (owner: 10Joal) [17:44:55] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4305974 (10Jdlrobson) [17:45:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Scoop jars , automate generation at the beginning of job - https://phabricator.wikimedia.org/T196912#4305975 (10Nuria) This needs a puppet companion change Ping @JAllemandou so he remembers to submit that one [17:47:05] 10Analytics, 10Research: Provide data dumps in the Analytics Data Lake - https://phabricator.wikimedia.org/T186559#4305976 (10Ottomata) Q: How does ElasticSearch get the text for indexing? [17:51:47] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4305993 (10ovasileva) [18:05:40] 10Analytics, 10Research: Provide data dumps in the Analytics Data Lake - https://phabricator.wikimedia.org/T186559#3947335 (10EBernhardson) Essentially we get it from the mediawiki object `ParserOutput`. For the literal text the rendered html is run through a process to remove some specific css selectors and t... [18:39:29] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4306060 (10RobH) Please note that next Monday's SRE team meeting has been canceled, as the SRE off-site is occurring this week. If this access needs to be approved before Monday, July... [18:39:43] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access for mbsantos - https://phabricator.wikimedia.org/T197237#4306062 (10RobH) [18:59:38] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4306109 (10Jdlrobson) @mforns running on the latest dump. > Looking at the errors i... [19:09:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10SEO: Make Google API Python Client Library available on stat* machines - https://phabricator.wikimedia.org/T190767#4306125 (10mpopov) >>! In T190767#4092654, @Ottomata wrote: > Done! Thank you for this! Much appreciated :) [19:39:13] 10Analytics, 10Product-Analytics, 10SEO: Make various auth libraries available on stat* machines - https://phabricator.wikimedia.org/T197896#4306180 (10mpopov) p:05Triage>03Normal [19:40:37] (03CR) 10Mforns: [C: 032] "LGTM! Makes sense :]" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/440005 (https://phabricator.wikimedia.org/T192481) (owner: 10Joal) [20:12:45] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4306269 (10mforns) @Jdlrobson Looks good to me overall. Now, could it be that the m... [21:00:30] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4306428 (10Jdlrobson) > Now, could it be that the maximum length of the source_url i... [21:35:53] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904#4306495 (10mforns) > I'm still a little cautious about adding additional logic to th... [21:40:48] bd808: Is wmf_raw.ApiAction still a thing? [21:52:16] bd808: never mind me. I've been matching the date partition columns by string rather than int.