[06:18:09] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3583116 (10Yurik) I would like to solicit more community feedback on how useful this would be. Perhaps this is not needed at all, or not worth the hassle As an a... [08:19:27] Good morning elukey :) [08:27:30] o/ [08:41:24] joal: ping! [08:41:36] OMW ! [09:31:59] 10Analytics, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3583490 (10JAllemandou) @GoranSMilovanovic : You're very welcome :) Thanks for the informative analysis ! Also a quick note on... [09:59:46] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3583648 (10elukey) @Tbayer let's see how it goes with... [10:11:28] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3583884 (10elukey) Some numbers: ``` elukey@db1046:~$... [10:11:32] joal: --^ [10:14:53] the eventlogging_sync.sh script doesn't keep up with busy schemas :( [10:27:47] * elukey lunch! [10:29:35] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Document SQL queries I should run regularly to ensure data quality in the account database behind wikimedia.biterg.io - https://phabricator.wikimedia.org/T175143#3583913 (10Aklapper) [10:54:56] (03PS1) 10GoranSMilovanovic: 09/06/2017 Tobi, Adam, Goran [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/376217 [11:00:49] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] 09/06/2017 Tobi, Adam, Goran [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/376217 (owner: 10GoranSMilovanovic) [11:02:02] (03CR) 10Tobias Gritschacher: [V: 032 C: 032] 09/06/2017 Tobi, Adam, Goran [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/376217 (owner: 10GoranSMilovanovic) [11:02:52] (03CR) 10Addshore: [V: 031 C: 031] 09/06/2017 Tobi, Adam, Goran [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/376217 (owner: 10GoranSMilovanovic) [11:12:42] (03PS2) 10GoranSMilovanovic: 09/06/2017 Tobi, Adam, Goran [correction] [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/376217 [11:13:07] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] 09/06/2017 Tobi, Adam, Goran [correction] [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/376217 (owner: 10GoranSMilovanovic) [11:14:34] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Document SQL queries I should run regularly to ensure data quality in the account database behind wikimedia.biterg.io - https://phabricator.wikimedia.org/T175143#3583951 (10Aklapper) 05Open>03Resolved Dumped into https://www.mediawik... [11:50:35] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3583985 (10Esc3300) There is https://tools.wmflabs.org/glamtools/treeviews/ [12:03:53] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3584033 (10mobrovac) [12:29:46] 10Analytics-Kanban, 10Operations, 10hardware-requests: Decommission stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T173097#3584079 (10elukey) [12:29:49] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3584078 (10elukey) [12:33:06] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Decommission stat1003.eqiad.wmnet - https://phabricator.wikimedia.org/T175150#3584101 (10elukey) [12:35:39] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3584116 (10elukey) [12:35:42] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Decommission stat1003.eqiad.wmnet - https://phabricator.wikimedia.org/T175150#3584115 (10elukey) [12:39:02] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Decommission stat1003.eqiad.wmnet - https://phabricator.wikimedia.org/T175150#3584146 (10elukey) [12:49:48] (03PS1) 10Joal: Add oozie mediawiki-history-reduced job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) [12:53:09] (03PS2) 10Joal: Update mediawiki history oozie denormalize job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/374987 (https://phabricator.wikimedia.org/T174484) [13:04:43] (03PS2) 10Joal: Add oozie mediawiki-history-reduced job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) [13:16:38] Taking a reak a-team [13:44:41] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3584342 (10Tbayer) Thanks @elukey - right now, while t... [13:51:55] (03CR) 10Ottomata: [C: 031] "One nit, +1 tho." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) (owner: 10Joal) [14:07:28] kafka-jumbo100[123] up and running with puppet/salt configured, the rest is still showing up issues with PXE :( [14:08:08] Hahaha, kafka-jumbo [14:08:29] Please tell me they are jumbo simply because they are more powerful machines? [14:08:59] they'll become the new big and general purpose kafka cluster, so we chose "jumbo" as name [14:09:06] :D [14:17:44] * elukey brb [14:30:20] 10Analytics-Kanban, 10Analytics-Wikistats: Use daily granularity for 1-month time ranges - https://phabricator.wikimedia.org/T173372#3584440 (10fdans) [14:30:22] 10Analytics-Kanban, 10Analytics-Wikistats: Productionise line graph - https://phabricator.wikimedia.org/T171766#3584441 (10fdans) [14:30:24] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3584442 (10fdans) [14:30:26] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (1/4) - Dashboard and general UI - https://phabricator.wikimedia.org/T170933#3584444 (10fdans) [14:30:28] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (2/4) - Wiki selector - https://phabricator.wikimedia.org/T170936#3584443 (10fdans) [14:31:53] ottomata,joal - ops sync? [14:33:19] * elukey feels alone [14:33:36] AH! [14:33:37] ya [14:52:40] Arf, forgot ops-sync elukey :( excuse me [14:52:50] * elukey missed Joseph [14:54:01] holaaa a-team [14:54:16] Hi nuria_ :) [14:55:40] o/ [14:55:40] ottomata: VERY relevant comment on naming convention for duid datasources [14:55:53] ottomata: I didn't realize we were so unconventional :) [14:58:28] :) [15:29:27] 10Analytics: Rename datasources and fields in Druid to use underscores instead of hyphens - https://phabricator.wikimedia.org/T175162#3584677 (10JAllemandou) [15:31:08] 10Analytics-Kanban: Move GraphiteClient from refinery-core to refinery-job module - https://phabricator.wikimedia.org/T175163#3584688 (10JAllemandou) [15:31:30] 10Analytics-Kanban: Move GraphiteClient from refinery-core to refinery-job module - https://phabricator.wikimedia.org/T175163#3584700 (10JAllemandou) a:03JAllemandou [15:33:50] oozie restart templates: https://gist.github.com/anonymous/6a1517bbb9c1c089f53d22b538f87573 [15:34:59] kafka-jumbo100[4,6] ready [15:35:02] 1005 the only one left [15:40:06] joal I have some questions about the Scala Clickstream job. Do you have a minute for them? [15:40:26] hi Shilad - In a meeting now, after? [15:40:51] Sure! I'm mostly around for the next hour or so. [16:03:47] Hey Shilad -- I have 1/2 hour now - would that work? [16:04:14] elukey: Can you please cover for me in cassandra standup? [16:04:37] joal: I am not joining (and Filippo too) because there is a problem with esams now [16:04:48] urandom: --^ [16:04:54] gotcha! [16:04:57] no worries [16:04:58] sorry :( [16:05:01] no prob elukey -- urandom -- I'm sorry I think we can cancel this wek [16:05:04] :( [16:05:17] joal Yes! Can I send you a hangout req on your wmf address? [16:05:24] Sure ! [16:05:27] Shilad: --^ [16:07:50] Just sent. See you soon! [16:08:00] Joining ! [16:10:32] hey ottomata, what is the right to ask for a hue - password sync for Shilad ? [16:10:53] you just did it! [16:10:55] ottomata: all jumbos are up and running, cluster ready for kafka [16:10:59] !!!:o [16:11:49] Shilad: done. [16:12:09] elukey: hoooly crap cool ok [16:12:23] will finish up a few things with certpy, eat some lunch, then see what i can do [16:13:39] Awesome ottomata :) Many thanks [16:23:50] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3584905 (10elukey) All hosts up with OS installed and puppet/salt running. [16:24:38] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3584927 (10elukey) [16:25:05] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3352337 (10elukey) [16:28:35] taking a break for diner a-team - will be back after [16:47:04] * elukey off! [17:23:26] 10Analytics, 10EventBus, 10ORES, 10Reading-Infrastructure-Team-Backlog, and 3 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#3585295 (10Halfak) [17:23:34] 10Analytics, 10Scoring-platform-team-Backlog, 10articlequality-modeling, 10Spike, 10artificial-intelligence: [Spike] Store article quality data inside hadoop and make AQS outputs a public API - https://phabricator.wikimedia.org/T164377#3585300 (10Halfak) [17:24:29] 10Analytics, 10Contributors-Analysis: Bring the Editor Engagement Dashboard back - https://phabricator.wikimedia.org/T166877#3585329 (10Nuria) @Halibutt: performance metrics of the editor you would need to request via @Neil_P._Quinn_WMF to the edit team. The only similar metrics we have are for the backend of... [18:11:49] (03CR) 10Nuria: [V: 032 C: 032] "Let's merge and restart jobs" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/375766 (https://phabricator.wikimedia.org/T161824) (owner: 10Joal) [18:20:14] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3585778 (10Nuria) NDA holders is quite a different set of permits. It is mostly geared towards research collaborators access to data so actually, there are quite a bit of extern... [18:22:14] 10Analytics-Kanban, 10RESTBase-API, 10Services (later), 10User-mobrovac: Expose pageview data in each project's REST API - https://phabricator.wikimedia.org/T119094#3585790 (10Nuria) @movroac: again, we do not feel it is worth to do these change, to be honest the benefit is small for the amount of work it... [18:24:35] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3585810 (10Nuria) @kaldari: this is odd, I am not aware of any permits requirements to edit these pages before. Maybe you as an ad... [18:51:22] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3585937 (10Aklapper) The argumentation makes sense I guess, I'm just not sure if it's technically possible what you have in mind. :) > similar to the one we use for hardware requ... [19:13:07] (03PS3) 10Joal: Add oozie mediawiki-history-reduced job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) [19:13:13] fdans: --^ [19:13:25] fdans: You'll find the md file with the queries in there [19:13:43] fdans: I realise there is one I forgot, but should be complete enough for a start :) [19:13:47] joal: you hero [19:13:56] merci mon ami [19:16:25] 10Analytics, 10Contributors-Analysis: Bring the Editor Engagement Dashboard back - https://phabricator.wikimedia.org/T166877#3310516 (10Jdforrester-WMF) >>! In T166877#3585329, @Nuria wrote: >>Does the second link show VE edits only, or is it both editors combined? > This link [1] shows metrics from both edito... [19:17:50] (03PS4) 10Joal: Add oozie mediawiki-history-reduced job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) [19:18:02] fdans: --^ With the previously forgotten metric ;) [19:19:08] (03PS5) 10Joal: Add oozie mediawiki-history-reduced job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) [19:19:42] (03CR) 10Joal: "Tested, with some docs on how to query druid." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) (owner: 10Joal) [19:20:13] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3586047 (10Nuria) I guess I was thinking of s4 as it is the only other space i see when I go to: https://phabricator.wikimedia.org/spaces/, I am not sure what ACL here refers to,... [19:23:54] 10Analytics-Kanban: Move GraphiteClient from refinery-core to refinery-job module - https://phabricator.wikimedia.org/T175163#3586056 (10JAllemandou) [19:27:42] (03PS1) 10Joal: Move GraphiteClient from core to job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/376333 (https://phabricator.wikimedia.org/T175163) [19:28:58] (03PS1) 10Ottomata: Add webrequest sample bundle for creating a sampled webrequest table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376334 [19:30:03] (03PS6) 10Joal: Add oozie mediawiki-history-reduced job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) [19:30:04] (03PS2) 10Ottomata: Add webrequest sample bundle for creating a sampled webrequest table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376334 [19:32:20] (03CR) 10Ottomata: [C: 031] Move GraphiteClient from core to job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/376333 (https://phabricator.wikimedia.org/T175163) (owner: 10Joal) [19:35:07] (03CR) 10Joal: [C: 031] "Did a quick pass - Seems good." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376334 (owner: 10Ottomata) [19:37:42] Leaving for tonight a-team - See you ! [19:40:32] laters! [19:50:43] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3569915 (10mmodell) Ideally we'd designate one or two people who will decide on the membership of the acl group. Then we delegate control to them so that phabricator admins don't... [19:54:28] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3352337 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by otto on neodymium.eqiad.wmnet for hosts: ``` ['kafka-jumbo1001... [19:58:16] 10Analytics-Cluster, 10Analytics-Kanban: Provision new Kafka cluster(s) with security features - https://phabricator.wikimedia.org/T152015#3586233 (10Ottomata) Hm, there is a 0.11.0.1 RC out, with some bugfixes for the 0.11. http://home.apache.org/~damianguy/kafka-0.11.0.1-rc0/RELEASE_NOTES.html I think it w... [20:16:47] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3586375 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['kafka-jumbo1001.eqiad.wmnet', 'kafka-jumbo1002.eqiad.wmnet', 'k... [20:23:03] 10Analytics-Kanban, 10Patch-For-Review: Create purging script for mediawiki-history data - https://phabricator.wikimedia.org/T162034#3586391 (10Nuria) a:05mforns>03Nuria [20:33:43] channel: does anyone knows how long takes the Cloaks to be assigned? I've open my request around 4 weeks ago [20:34:54] dsaez: better to ask in #wikimedia-operations about that, don't know much around here :) [20:34:59] not sure who usually handles that [20:35:25] ottomata: will do, thanks [20:47:56] 10Analytics, 10Contributors-Analysis, 10DBA, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3586457 (10Nuria) Looks like it is all good: select wiki_db, count(*) from wmf.mediawiki_history where snapshot = "2017-08" and wiki_d... [20:48:12] 10Analytics, 10Contributors-Analysis, 10DBA, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3586458 (10Nuria) 05Open>03Resolved [21:04:23] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3586517 (10Nettrom) From what I can tell after digging around a bit, the configuration of the Dashiki extension limits the creatio... [21:26:37] 10Analytics, 10Contributors-Analysis, 10DBA, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3586571 (10Neil_P._Quinn_WMF) @Nuria, yes, the queries where I originally discovered this work now. Thank you! [22:11:29] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3586778 (10kaldari) @Nettrom: According to [[ https://wikitech.wikimedia.org/wiki/Analytics/Systems/Dashiki#On-wiki_configuration... [22:15:43] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3586801 (10Nettrom) Ah, I see! [[ https://wikitech.wikimedia.org/wiki/Analytics/Tutorials/Dashboards#Write_the_configuration_for_y... [22:20:13] 10Analytics, 10Operations, 10Traffic: Invalid "wikimedia" family in unique devices data due to misplaced WMF-Last-Access-Global cookie - https://phabricator.wikimedia.org/T174640#3586807 (10Dzahn)