[00:05:27] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Investigate use-cases for delayed job executions - https://phabricator.wikimedia.org/T172832#3590203 (10Pchelolo) I wrote a little script to run through a sample of events in the job topics we have in prod right now and here's.a lis... [00:05:43] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3590204 (10Tbayer) >>! In T174815#3589983, @Nuria wrot... [00:47:33] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3590294 (10Tbayer) >>! In T174815#3589327, @Nuria wrot... [06:03:24] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3590510 (10elukey) ``` elukey@kafka-jumbo1001:/usr/share/jmxtrans$ source /etc/default/jmxtrans elukey@kafka-jumbo1001:/usr/share/jmx... [07:24:12] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3590584 (10elukey) Applied the following to all the nodes to remove the placeholder logical volumes: ``` root@kafka-jumbo1001:/home/... [07:42:54] 10Analytics, 10Operations, 10netops, 10User-Elukey: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#3590602 (10elukey) The next step is to design and add the `analytics-in6` filter to cr1/cr2 eqiad, but I would wait for kafka1012-1022 to be decommissioned before that. Those h... [08:20:16] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3590648 (10elukey) I could be wrong but from cr1/cr2 eqiad the hosts seem to be in the Analytics VLAN, and they shouldn't be: ``` el... [08:23:44] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3590651 (10elukey) @Ottomata: let's also remember to whitelist the jumbo IPs in the Analytics VLAN firewall rules, otherwise hosts li... [08:55:09] joal: o/ [08:55:20] Hi elukey [08:57:08] I was checking why on druid100[456] we don't have jmxtrans because I thought that I had mistakenly removed it when doing some refactoring [08:57:27] but now I realized that we have it on drud100[123] because of zookeeper (that is not deployed on 456) [08:57:40] Ahhhhh ! [08:57:49] Am I correct? Do you know about any druid metric ? [08:57:57] or dashboard [08:59:39] elukey: I don't - So far we've not been running into problems with Druid, so I have not even looked after metrics (how bad ...) [09:03:37] checking now with Jconsole what MBeans are available [09:04:42] so I can only see JVM metrics and log4j ones, nothing related to Druid itself (at least for the broker) [09:05:46] ahh http://druid.io/docs/latest/operations/metrics.html [09:05:53] so we'd need to use something like logster [09:06:06] or a new prometheus agent [09:06:13] (to poll the http api) [09:06:34] joal: I guess I found another ops task for the next quarter :P [09:06:38] :D [09:08:36] Doctor appointment - Will be back after [09:55:06] Updated https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Ports#JMX with the Druid jmx ports [10:02:17] 10Analytics: Export Druid metrics and build a grafana dashboard - https://phabricator.wikimedia.org/T175343#3590873 (10elukey) [10:07:34] 10Analytics: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344#3590888 (10elukey) [10:09:29] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3590908 (10elukey) @Ottomata: I merged https://gerrit.wikimedia.org/r/#/c/376663 but I then realized that master/debian branches are... [10:31:21] * elukey off! [10:55:06] 10Analytics: Provide top domain and data to truly test superset - https://phabricator.wikimedia.org/T166689#3591026 (10JAllemandou) I updated the datasources to contain interesting metrics and created a new user able to use and create visualisation, but not mess with the config: tester / tester [11:01:10] 10Analytics-Tech-community-metrics: Zero results shown for certain repositories on Git dashboard (though there has been Git activity) - https://phabricator.wikimedia.org/T175351#3591043 (10Aklapper) [11:01:17] 10Analytics-Tech-community-metrics: Zero results shown for certain repositories on Git dashboard (though there has been Git activity) - https://phabricator.wikimedia.org/T175351#3591043 (10Aklapper) p:05Triage>03High [12:08:04] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#3591270 (10mobrovac) [12:09:29] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3591273 (10mobrovac) [14:05:04] (03PS3) 10Joal: Add mediawiki history edits metrics endpoint [analytics/aqs] - 10https://gerrit.wikimedia.org/r/373961 (https://phabricator.wikimedia.org/T174174) [14:05:18] fdans, nuria_ : --^ When you have a moment :) [14:05:27] taking a break now :) [14:07:26] 10Analytics: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344#3591560 (10Suhadakashter) [14:08:19] 10Analytics: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344#3590888 (10Suhadakashter) [14:12:28] 10Analytics: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344#3591641 (10Reedy) 05duplicate>03Open [14:50:23] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Separate off ChangePropagation for JobQueue as a new deployment - https://phabricator.wikimedia.org/T175281#3591719 (10mobrovac) [14:55:04] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Separate off ChangePropagation for JobQueue as a new deployment - https://phabricator.wikimedia.org/T175281#3591738 (10mobrovac) The repo has been set up and cloned on `tin` and the `ops/puppet` profile created and merged. Left to d... [15:17:45] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3591819 (10elukey) >>! In T174815#3590294, @Tbayer wro... [15:31:11] fdans: I confirm, project_family can easily be done in druid [15:36:58] nuria_, fdans: if we wanted to have project-family cross-metrics, it would mean adding per-project-family pageviews - feasible but not free [15:43:04] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3591889 (10GWicke) @pchelolo, based on our previous conversation about this I am assuming that the bulk of the task is a very large list of pages. Is this correct? [15:58:24] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3591923 (10Nettrom) @Nuria: I've tested our dashboard locally here and everything seemed to be working just fine. How do we go abo... [16:59:49] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3592020 (10Pchelolo) >>! In T175316#3591889, @GWicke wrote: > @pchelolo, based on our previous conversation about this I am assuming that the bulk of the task is a ver... [17:01:33] 10Analytics-Kanban, 10Contributors-Analysis, 10Patch-For-Review: Provide cumulative edit count in Data Lake edit data - https://phabricator.wikimedia.org/T161147#3122928 (10ksmith) @Neil_P._Quinn_WMF : This task is part of a quarterly goal, so we would like it to be fully resolved before the end of September... [17:07:48] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3592059 (10GWicke) [17:07:49] dsaez: heyas [17:07:55] can you log out of stat1005 and kill your screen sessions please? [17:08:02] im trying to fix your user id login issue =] [17:08:10] and i have to rm 'diego' to allow 'dsaez' to replace it [17:08:30] they share the same uid after all [17:12:36] Does anyone know if dsaez is around today? [17:12:46] or if I can kill his screen sessions on stat1005 to fix his user [17:12:51] (basically puppet is broken due to it on that host) [17:13:42] robh: please kill the session, i can't connect right noe [17:13:51] cool, will do! [17:13:57] and itll be all fixed shortly! [17:14:01] the other hosts are fixed already [17:14:11] (the ones i know you can login to so far that is) [17:15:06] ok, they are killed and puppet is rerunning. [17:15:11] cool, thx [17:15:34] you should be able to log back in now. also now that username in shell and ldap match other issues should clear up on ldap login stuff (if that was the cause) [17:15:44] and i can make sure you are now flagged into the wmf group in ldap [17:15:50] in fact, i shall do so right now [17:17:44] you were already in there, so it should work now [17:17:53] dsaez: if you have issues with logging in today due to thsi change, let me konw =] [17:18:13] sorry for having to kill your screen sessions, i didnt realize they were running befor emy uid patchset [17:18:19] or i would have held off for you to check data =[ [17:20:33] fyi : https://wikitech.wikimedia.org/wiki/Analytics/Data_access is now outdated [17:21:05] since it seems one of your groups gave you access to analytics1002, stat100[456] [17:21:38] but the analytcs1002 isnt covered on the dataaccess graph [17:21:44] grid, chart, whatever =] [17:22:04] ok, I'll connect la [17:22:08] ter [17:22:14] thanks [17:22:45] welcome, sorry you had the issues =] [17:25:40] robh: does my wikitech username also changed? [17:26:14] Your wikitech account for work was dsaez? [17:26:28] thats the one i pulled the uid from originally since it ties to your @wikiemdia.org email address [17:26:34] so nothing should have changed on wikitech afaik [17:26:44] nop, Diego [17:26:47] ok [17:26:55] diego on wikitech is your volunteer account though right? [17:26:59] that isnt tied to @wikiemdia.org [17:27:11] you'll login for work metrics and the like with 'dsaez' not diego [17:30:14] diego is tied to @wikimedia [17:31:31] =/ [17:31:59] dsaez: when i look up 'diego' on ldap/wikitech [17:32:11] it shows someone called : Diego Queiroz [17:32:28] oh, caps matter [17:32:37] wait, sorry nope, same guy... [17:32:46] dsaez: so im not sure what wikitech account you mean, sorry [17:33:06] oh, its diego(wmf) [17:33:10] thats totally a different username ;D [17:34:17] if i go here https://wikitech.wikimedia.org/w/index.php?title=Main_Page&welcome=yes [17:34:27] and go to reset password [17:35:10] i put username diego, and i receive a email on @wikimedia for resetting :/ [17:35:15] hrmm [17:35:22] well, im officially confused then [17:35:35] can you login to swap with 'dsaez' now? [17:36:56] I'm not with me computer now, i can try later or Monday [17:37:26] yeah im not 100% sure what the correct fix is so lets try that later or monday and work on it then [17:37:36] im around all next week (except friday) so happy toassist [17:42:07] oh... i see [17:42:09] i read ldap wrong [17:42:15] uid: dsaez [17:42:15] sn: Diego [17:42:15] cn: Diego [17:42:18] do yeah, its right. [17:42:58] ok [17:43:29] dsaez: if you have a phone you can check on yarn.wikimedia.org [17:43:46] if that's fine, you should be set for SWAP [17:45:59] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3592255 (10Nuria) > Could I add a couple of requirements to the tutorial? Please do, thank you. >How do we go about getting it de... [17:57:24] robh, madhuvishy, in yarn.wikimedia.org works with user Diego :S [17:57:46] yeah! its a uid in wikitech being dsaez but your sn being Diego [17:57:49] so all is now well [17:59:54] good :) Thanks [18:01:06] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#3592304 (10EBernhardson) cirrusSearchCheckerJob - basically idempotent. It verifies data in elasticsearch matches mysql, cre... [18:01:27] which will be my home folder at stats machines? diego or dsaez? [18:03:32] (03CR) 10Zhuyifei1999: [C: 032] "I'm merging this. I don't have a replica to test against so hopefully it won't break anything." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/376561 (https://phabricator.wikimedia.org/T175285) (owner: 10Zhuyifei1999) [18:03:51] (03Merged) 10jenkins-bot: output: Fallback to write_string when some error occur in write for xlsx [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/376561 (https://phabricator.wikimedia.org/T175285) (owner: 10Zhuyifei1999) [18:04:43] hi analytics folks. [18:05:13] question: is pageviews-daily not updated in pivot? joal et al. :) [18:09:54] 10Quarry, 10Patch-For-Review: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3592323 (10zhuyifei1999) 05Open>03Resolved Should be resolved. Such text are no longer written as a link but as basic string. Please reopen if there are issues. [18:12:08] 10Quarry, 10Patch-For-Review: Quarry XLSX cells for long urls are wrongly empty - https://phabricator.wikimedia.org/T175285#3592341 (10IKhitron) Checked. Works great. Thank you, @zhuyifei1999. [18:17:09] leila: pageviews-daily? [18:17:22] leila: yes, it should be but not with data from yesterday necessarily [18:17:41] leila: pageviews-hourly has most recently but only going back 3 months [18:20:08] nuria_: I only see it until Aug 31. [18:20:15] it seems the past week is for some reason not there, nuria_ [18:20:33] leila: it might be updated weekly or monthly, let me see [18:22:03] leila: ah no, it is updated daily but i bet joseph stop jobs to add zero carrier [18:24:49] dsaez: home folder should be dsaez [18:25:44] leila: mmmm... no hue said jobs run , let me see [18:25:50] hmm [18:27:48] hi leila - pageview-daily is loaded monthly (see https://pivot.wikimedia.org/#) [18:28:23] ah! thanks, nuria_! [18:29:18] leila: no, sorry, it looks like it is loaded once a month [18:29:33] leila: my mistake, so it is up to date [18:29:43] Shilad: following up here: leila: no, sorry, it looks like it is loaded once a month [18:29:46] oops [18:29:59] Great. Thanks! [18:30:02] wrong copy paste [18:30:04] About how to handle the job, I'd say let's do it in a branch, adding a new Scala class to the refinery-job module of the refinery-souce repo [18:30:07] Shilad: --^ [18:30:09] nuria_: got it. Then I look at the hourly data. I was interested in numbers in early September which are not in monthly yet. thanks. [18:30:27] leila: pageviews-hourly is loaded about once an hour so it should be two hours behind [18:30:37] leila: pageviews-daily is loaded monthly [18:31:00] nuria_: great. thank you! (and welcome back!) [18:31:13] joal: I presume I should do this through gerrit? [18:31:27] Shilad: Yes, that's our usual tool for CR [18:32:11] Also Shilad, I had a look at how was done Wikidata entities loading - It's a spark job that parses a wikidata json dump [18:32:49] thanks joal! Is that in the same repo? [18:32:55] And finally Shilad, we miss one table from the ones used in current ETL (page_props) - I'll be posting some CRs on that [18:33:39] Shilad: Nope - it's a script named wikidata_utils.py, in wmf repo [18:34:50] Thanks. I'll find it. Also, a workflow question. I presume you use an IDE for dev. Since there's no local dev environment, how do you sync to test code on the server? [18:35:02] Via git, or rsync, or something else? [18:35:19] I've done this all sorts of ways in the past and am wondering what you have found easiest here. [18:35:29] Shilad: I use gerrit - I push my CR to gerrit, then download that version onto stat1004 and mvn [18:35:54] joal: That makes sense. Thanks! [18:36:20] no prob Shilad :) [18:38:01] (03CR) 10Nuria: [C: 032] Move GraphiteClient from core to job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/376333 (https://phabricator.wikimedia.org/T175163) (owner: 10Joal) [19:49:16] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3592763 (10Nuria) >Regarding these many MySQL issues:... [19:52:35] (03PS15) 10Joal: Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) [19:52:54] (03PS7) 10Joal: Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) [20:06:23] (03CR) 10Nuria: [V: 031 C: 031] Add script to purge old mediawiki data snapshots (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355601 (https://phabricator.wikimedia.org/T162034) (owner: 10Mforns) [21:11:13] Hi all. I'm trying to push a new branch to analytics-refinery-source but am getting a 403. Is there a possibility that I don't have permissions? [21:19:43] 10Analytics, 10Analytics-Cluster, 10Analytics-Wikistats: Create Daily & Monthly pageview dump with country data and Visualize on UI - https://phabricator.wikimedia.org/T90759#3592986 (10Nuria) [21:20:16] 10Analytics, 10Analytics-Cluster, 10Analytics-Wikistats: Design map visualization on UI - https://phabricator.wikimedia.org/T175422#3592991 (10Nuria) [21:23:47] (03PS1) 10Shilad Sen: Placeholder for job to create page ids viewed in each session. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/376797 (https://phabricator.wikimedia.org/T174796) [21:25:04] I figured it out! Realized I need to use git review instead of git push [21:38:57] 10Analytics-Kanban, 10Contributors-Analysis, 10Patch-For-Review: Provide cumulative edit count in Data Lake edit data - https://phabricator.wikimedia.org/T161147#3593030 (10Neil_P._Quinn_WMF) 05Open>03Resolved I actually started working on the systematic random checks today, but it looks like that will b... [21:44:18] joal or somebody with admin access on the repo: Would you please create a nav-vectors branch of analytics/refinery/source for me? [21:57:23] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#3593074 (10Pchelolo) [21:58:00] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#3586259 (10Pchelolo) Thank you @EBernhardson, updated the task with your info. Now we've got a complete list of jobs execute... [22:41:34] (03PS1) 10Catrope: Add codemirror-syntax-highlight beta feature [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/376849 [22:51:43] (03CR) 10Niharika29: [C: 032] "Thanks for the patch." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/376849 (owner: 10Catrope) [22:51:52] (03Merged) 10jenkins-bot: Add codemirror-syntax-highlight beta feature [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/376849 (owner: 10Catrope)