[00:00:11] <wikibugs>	 10Analytics, 10EventBus, 06Services (done): Create mediawiki.page-restrictions-change event - https://phabricator.wikimedia.org/T160942#3138967 (10mobrovac) Merged. To be deployed on the next MW train.
[00:35:14] <wikibugs>	 06Analytics-Kanban: Pageview Spike in Tagalog Wikipedia  mid-June 2016 - https://phabricator.wikimedia.org/T144635#3139028 (10Nuria) a:03Nuria
[00:46:50] <wikibugs>	 06Analytics-Kanban: Pageview Spike in Tagalog Wikipedia  mid-June 2016 - https://phabricator.wikimedia.org/T144635#3139048 (10Nuria) {F7065290}  {F7065293}
[00:48:08] <wikibugs>	 06Analytics-Kanban: Pageview Spike in Tagalog Wikipedia  mid-June 2016 - https://phabricator.wikimedia.org/T144635#3139051 (10Nuria) {F7065290}  {F7065293}
[00:50:47] <wikibugs>	 06Analytics-Kanban: Pageview Spike in Tagalog Wikipedia  mid-June 2016 - https://phabricator.wikimedia.org/T144635#3139054 (10Nuria)
[00:55:52] <wikibugs__>	 06Analytics-Kanban: Pageview Spike in Tagalog Wikipedia  mid-June 2016 - https://phabricator.wikimedia.org/T144635#3139088 (10Nuria) Will be closing ticket as I cannot really find anything that might point to a bug, from my brief analysis traffic seems organic.
[03:47:00] <wikibugs>	 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3139172 (10Beetstra) @Samwalton9, what do you mean with 'at some point'?  Do you mean that this has an...
[06:38:18] <wikibugs__>	 (03PS3) 10Mforns: Use both projectcounts raw and all sites to load cassandra [analytics/refinery] - 10https://gerrit.wikimedia.org/r/345144 (https://phabricator.wikimedia.org/T161494)
[06:49:39] <mforns>	 heloooo
[07:54:12] <wikibugs__>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3139486 (10Nemo_bis)
[07:57:40] <wikibugs>	 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Deployment of Maniphest panel - https://phabricator.wikimedia.org/T138002#3139504 (10Nemo_bis) I'd have multiple feature requests depending on what's the expected purpose of this statistical tool (I think the main purpose should be to co...
[09:18:49] <GoranSM>	 Hi there. I have started working as a Data Analyst for WMDE. Does anyone here know, by any chance, do we have RStudio Server running on stat1002 or stat1003? Thx.
[09:35:22] <moritzm>	 GoranSM: no, doesn't seem so
[09:36:57] <GoranSM>	 moritzm: Thanks. So, one question, just to check if get the workflow correctly: we mainly use production machines like stat1002/1003 to gather data from the various sources that are hosted there, then move the data to some other place (Labs, for example, or our own locals) to analyze there?
[09:40:45] <moritzm>	 no, I'm not using R on those machines, better wait until someone else is around
[09:41:01] <GoranSM>	 moritzm: Thanks.
[09:44:14] <wikibugs>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3139639 (10Aklapper)
[09:46:06] <wikibugs__>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3139486 (10Aklapper) 05Open>03Invalid For future reference, please use [[ https://www.mediawiki.org/wiki/Phabricator/Help#Formatting | table markup ]] for tables...
[09:52:54] <wikibugs>	 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3139666 (10Samwalton9) @Beetstra Good question. @Milimetric thinks that if/when this is working correc...
[10:05:57] <elukey>	 GoranSM: hi! I am not super familiar with R but I can speak for the stat machines..
[10:07:04] <GoranSM>	 elukey: Support, please: I am trying to connect to Jupyter notebooks with ssh -N notebook1001.eqiad.wmnet -L 8000:127.0.0.1:8000 as explained on https://meta.wikimedia.org/wiki/Discovery/Analytics#Analysis_with_R. I should be able to do this with my LDAP (Wikitech credentials); however, all that I get is username/password incorrect. 
[10:08:23] <elukey>	 GoranSM: you'd need to have a shell account on notebook1001.eqiad.wmnet, LDAP credentials are valid for Wikitech and some other web interfaces only
[10:08:57] <elukey>	 there is a process for this that should be part of your onboarding
[10:09:47] <elukey>	 (also there is a bit of paperwork to do for NDAs between WMDE and WMF to protect sensitive data that we store)
[10:10:44] <GoranSM>	 matthiasmullie: I have LDAP confirmed, NDA signed, and access to stat 1002 and 1003 already.
[10:11:57] <elukey>	 ahhh nice! Didn't know it
[10:12:00] <moritzm>	 elukey: Goran has signed an NDA with Legal
[10:12:08] <GoranSM>	 elukey: Will a Phabricator ticket be enough to ask for a shell account on notebook1001.eqiad.wmnet?
[10:12:24] <elukey>	 GoranSM: let me check what access is needed
[10:12:31] <GoranSM>	 elukey: Thanks a lot.
[10:13:37] <moritzm>	 elukey: researchers or statistics-privatedate-users
[10:13:53] <elukey>	 there you go, but I can see your username on it
[10:14:16] <elukey>	 ah yes you already are in researchers
[10:14:20] <GoranSM>	 elukey: I am in researchers, yes
[10:14:58] <moritzm>	 GoranSM: does a standard SSH login to notebook1001.eqiad.wmnet work for you? (it should)
[10:15:40] <GoranSM>	 elukey: Again, to make my question simpler: (A) My SSH config is set to acess stat 1002 and 1003 via bastion host; (B) I do ssh -N notebook1001.eqiad.wmnet -L 8000:127.0.0.1:8000 as advised on the Wiki; (C) I open localhost:8000 in my browser and enter my Wikitech credentials; output == incorrect username password.
[10:16:23] <GoranSM>	 elukey: Yes the standard shh to notebook1001.eqiad.wmnet works, I'm there right now
[10:17:13] <elukey>	 GoranSM: ahhh sorry I thought that the problem was during the ssh connection
[10:18:03] <GoranSM>	 elukey: Well the problem was during during the ssh connection: once I do ssh -N notebook1001.eqiad.wmnet -L 8000:127.0.0.1:8000, I cannot login to Jupyter notebook from localhost:8000
[10:18:34] <GoranSM>	 elukey: I want to say, the SSH tunneling is opened while I'm trying to connect to localhost:8000 for Jupyter, as it should be opened
[10:20:18] <elukey>	 I get login failures too with my credentials, so probably a specific ldap group is needed? I am in the middle of something but I'll get back to you in a bit!
[10:20:32] <elukey>	 (ops work pending sorry0
[10:20:45] <GoranSM>	 elukey
[10:20:59] <GoranSM>	 elukey: (sorry for a typo) Thanks :)
[10:40:11] <mforns>	 joal hi :] yt?
[10:44:08] <wikibugs>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3139791 (10Nemo_bis) 05Invalid>03Open > Your Phab query is not restricted to the last two years.  Indeed, and a superset of tasks can't have less tasks. Yet 262 <...
[10:44:59] <wikibugs>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3139795 (10Nemo_bis)
[10:58:53] <elukey>	 GoranSM: just managed to get in, have you tried with your Wikitech username lowercase?
[10:59:08] <GoranSM>	 No let me try
[11:00:36] <GoranSM>	 elukey: no change, still incorrect
[11:00:44] <GoranSM>	 Invalid username or password    
[11:01:22] <elukey>	 afaics it needs wikitech credentials.. I went in with 'elukey' and my wikitech pass
[11:02:20] <GoranSM>	 elukey: I am trying with my wikitech username/password
[11:02:56] <GoranSM>	 elukey: let me try to change my wikitech password and then re-try
[11:08:37] <GoranSM>	 elukey: Nothing. Changed password, tried with the new password, username normal and username lower case, nothing. Invalid password username.
[11:09:19] <elukey>	 GoranSM: super weird, will ask to other people in analytics when they'll be online
[11:09:46] <GoranSM>	 elukey: Thanks. This is very important for my work. Can I ask you another (much simpler) question?
[11:10:46] <elukey>	 sure
[11:11:48] <GoranSM>	 Here it goes: when at production, say stat1002 or stat stat1004, do we have any tools like Jupyter, RStudio and similar, or we generally access Hadoop, mySQL and similar via konsole and scripts written out in vim and similar?
[11:12:57] <GoranSM>	 elukey: I simply want to understand the organizational culture and the usual workflow here.  If people do not use such tools to access data and process on production, neither will I do - I don't want to complicate others' (read: sysadmins') lives just in order to be able to use GUIs for dev/analytics.
[11:17:00] <elukey>	 GoranSM: from what I know it is (mostly) the second one that you mentioned, with some efforts like notebooks (and labs tools) to overcome the console limitation
[11:25:26] <GoranSM>	 elukey: Ok, thanks.
[11:26:07] <GoranSM>	 elukey: Thank you for all the support that you have provided. Please, when someone who could know how to resolve this access issue turns online, let me know. Thanks again.
[11:26:47] <elukey>	 GoranSM: sure! I'll try to get back to you with an answer today
[11:27:12] <GoranSM>	 elukey: Thank you so much!
[11:39:05] * elukey lunch!
[11:46:09] <wikibugs>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3139486 (10Albertinisg) This is a clear bug. I've been digging into it and I've found we have an issue with the `[[mw:User:This, that and the other]` identity.   Rega...
[11:58:08] <wikibugs>	 (03PS4) 10Mforns: Use both projectcounts raw and all sites to load cassandra [analytics/refinery] - 10https://gerrit.wikimedia.org/r/345144 (https://phabricator.wikimedia.org/T161494)
[12:14:05] <joal>	 Hi mforns !
[12:14:10] <joal>	 Excuse me I missed your ping !
[12:17:24] <mforns>	 joal, np :]
[12:17:42] <mforns>	 I wanted to present you with a riddle, but I think I got it
[12:18:14] <joal>	 Ah ! What is it?
[12:18:36] <mforns>	 the projectcounts oozie hql query was wrong
[12:18:51] <mforns>	 the resulting data for the mobile-site slice was veeery weird
[12:19:09] <mforns>	 at least at first glance, then actually it made sense
[12:19:28] <mforns>	 if you want I can explain more, but I'd need to share screen
[12:19:50] <joal>	 sure mforns, let's batcave
[12:19:55] <mforns>	 k!
[12:20:27] <mforns>	 joal, having problems with hangouts...
[12:39:56] <wikibugs>	 06Analytics-Kanban, 06Operations, 06WMDE-Analytics-Engineering, 13Patch-For-Review, 15User-Addshore: /a/mw-log/archive/api on stat1002 no longer being populated - https://phabricator.wikimedia.org/T160888#3140077 (10elukey) @Addshore: I am going to close this task but we might want to open another one to...
[12:40:21] <wikibugs__>	 06Analytics-Kanban, 06Operations, 06WMDE-Analytics-Engineering, 13Patch-For-Review, 15User-Addshore: /a/mw-log/archive/api on stat1002 no longer being populated - https://phabricator.wikimedia.org/T160888#3113734 (10elukey) a:03elukey
[12:40:52] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban, 06Operations: Reinstall  Analytics Hadoop Cluster with Debian Jessie - https://phabricator.wikimedia.org/T157807#3140081 (10elukey) a:03elukey
[12:41:09] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban, 06Operations: Reinstall  Analytics Hadoop Cluster with Debian Jessie - https://phabricator.wikimedia.org/T157807#3017036 (10elukey) a:05elukey>03None
[12:41:37] <wikibugs__>	 06Analytics-Kanban, 06Operations, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3140084 (10elukey)
[12:42:04] <elukey>	 joal: reimaging an1045 to debian :)
[12:44:51] <joal>	 k elukey :)
[12:45:25] <joal>	 elukey: so that you know - We've been discussing with mforns, and some data reloading will happen on pagecounts keyspace
[12:46:20] <joal>	 elukey: Given that after that, it shouldn't be touched before some time (no regular insertion), it'd be good to do a keysapce clean-up
[12:51:29] <elukey>	 sure
[12:51:56] <joal>	 elukey: that keysapce is not huge, so ti shouldn't be that long
[12:53:02] <mforns>	 elukey, joal, for now doing some loading to the test keyspace still, will ping you later when I need your help elukey :]
[12:53:32] <elukey>	 sure :)
[12:54:54] <elukey>	 stopped hadoop daemons on 1045, some jobs might fail
[13:03:35] <wikibugs__>	 (03PS1) 10Joal: [WIP] Update sqoop job to better handle failures [analytics/refinery] - 10https://gerrit.wikimedia.org/r/345327
[13:04:05] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 15User-Elukey: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3140160 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['ana...
[13:10:40] <wikibugs>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3140185 (10Aklapper) @Albertinisg: You're fast. :) I hadn't reloaded the page to see your comment before adding mine...
[13:12:21] <halfak>	 o/
[13:12:31] <halfak>	 hey joal & milimetric 
[13:12:41] <halfak>	 no live systems stuff from me today
[13:12:52] <joal>	 Hi halfak 
[13:13:07] <joal>	 I have a question for you on scikit-learn, but that'll be all
[13:13:15] <joal>	 halfak: I can do with IRC
[13:13:23] <halfak>	 sure!
[13:13:44] <joal>	 halfak: Trying to make sklearn with spark, we are encountering dependencies issues
[13:17:12] <joal>	 halfak: It'd be easier for ottomata if we could use scikit-learn 0.18 instead of 0.17
[13:17:21] <joal>	 halfak: Do you think it would be a big change?
[13:17:25] <halfak>	 No problem
[13:18:13] <joal>	 halfak: So I could myself change the deps in requirement.txt?
[13:18:41] <halfak>	 joal, not sure which requirements.txt you are referring to. 
[13:19:13] <joal>	 the one in revscoring forlder, sorry halfak 
[13:20:03] <halfak>	 joal, OK.  So I think that everything will just work, but we might need to rebuild the models (which is not too much trouble)
[13:20:24] <joal>	 halfak: ok, awesome :)
[13:20:48] <joal>	 halfak: It should facilitate us being to test revscoring in spark :)
[13:21:24] <halfak>	 I can rebuild a model with sklearn 0.18 if that'll help
[13:22:36] <joal>	 halfak: I don't need that yet, I'll for sure ask when I will; :)
[13:23:24] <joal>	 ottomata: in case you haven't seen: We can try with scikit-learn 0.18 :)
[13:23:38] <ottomata>	 oh i'm reading :)
[13:23:39] <ottomata>	 haha
[13:23:41] <ottomata>	 hi!
[13:23:48] <ottomata>	 cool!
[13:23:53] <ottomata>	 ok i'll try to backport 
[13:23:55] <ottomata>	 should work
[13:23:58] <joal>	 Hi (sorry, even forgetting to be polite by being happy)
[13:25:30] <halfak>	 \o/
[13:25:31] <halfak>	 :) 
[13:28:35] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 15User-Elukey: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3140234 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1045.eqiad.wmnet'] ```  and were **ALL** suc...
[13:37:01] <wikibugs__>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3140245 (10Nemo_bis) I only checked TTO so far. Feel free to merge, I can reopen if I find other suspicious numbers.
[13:38:42] <wikibugs__>	 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Deployment of Maniphest panel - https://phabricator.wikimedia.org/T138002#3140257 (10Aklapper) >>! In T138002#3139504, @Nemo_bis wrote: > I'd have multiple feature requests depending on what's the expected purpose of this statistical too...
[13:40:02] <elukey>	 1045 is already up and running, fixing perms at the momnet
[13:40:04] <elukey>	 *moment
[13:40:10] <elukey>	 should be up and running before standup
[13:42:11] <ottomata>	 nie
[13:42:13] <ottomata>	 nice
[13:48:19] <elukey>	 ottomata,joal - I'd need a bit of time to complete my work on mediawiki ssl certs (now it is time for the eqiad ones), do you mind if I skip the ops meeting?
[13:49:10] <ottomata>	 elukey:  we can skip it this week i think if you like, i want you to show me this auto-reimage script one day though...:)
[13:49:20] <ottomata>	 also I got a Q for you about RAID in new Kafka brokers, wondering what you think
[13:50:55] <elukey>	 ah yes I saw it but didn't follow up sorry! :(
[13:51:00] <ottomata>	 no hurry!
[13:51:18] <ottomata>	 joal:  looks like this isn't going to be a snap
[13:51:23] <ottomata>	 joblib tests fail when building
[13:51:25] <ottomata>	 grrrr
[13:51:36] <elukey>	 ottomata: I added https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Worker_Reimage_.2812_disk.2C_2_flex_bay_drives_-_analytics1028-analytics1057.29 but I
[13:51:42] <elukey>	 I'll finish it today
[13:51:50] <elukey>	 should be good after that :)
[13:52:20] <ottomata>	 oo
[13:52:45] <ottomata>	 right elukey i'm curious though, you have some auto-reimage script, no?
[13:53:12] <elukey>	 ahhh ok! Yes it is Riccardo's magic, available on neodymium
[13:56:08] <ottomata>	 ha! amazing!
[13:56:10] <ottomata>	 haven't used that
[13:56:13] <ottomata>	 pretty cool
[13:56:28] <ottomata>	 elukey:  are you telling me that you can reimage these hosts without logging into mgmt console?
[13:57:51] <elukey>	 ottomata: oh yes!
[13:57:59] <elukey>	 and it logs in the task
[13:58:02] <elukey>	 reboots the host
[13:58:03] <elukey>	 etc..
[13:58:54] <ottomata>	 amazing
[14:00:56] <wikibugs>	 (03CR) 10Hashar: "recheck" [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/337452 (owner: 10Milimetric)
[14:01:34] <elukey>	 joal, mforns 
[14:01:35] <elukey>	 <icinga-wm> PROBLEM - aqs endpoints health on aqs1004 is CRITICAL: /legacy/pagecounts/aggregate/{project}/{access-site}/{granularity}/{start}/{end} (Get pagecounts) is CRITICAL: Test Get pagecounts returned the unexpected status 404 (expecting: 200)
[14:01:37] <wikibugs>	 (03CR) 10Hashar: "Did a recheck to verify the job pass all fine (for T97514). And it does!" [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/337452 (owner: 10Milimetric)
[14:01:44] <elukey>	 a lot of alarms like these on the ops chan
[14:01:47] <mforns>	 elukey, ops
[14:02:19] <mforns>	 elukey, I truncated the table
[14:02:19] <joal>	 mforns: Have you truncated the table?
[14:02:24] <mforns>	 yes
[14:02:28] <joal>	 mforns: please don't do that :)
[14:02:41] <mforns>	 I have to add the monitoring data
[14:02:46] <joal>	 mforns: correct
[14:02:49] <mforns>	 k
[14:03:01] <joal>	 mforns: I thought we had talked about relaoding, not truncating :)
[14:03:23] <joal>	 mforns: in case of data deletion (truncate), no need to cleanup keyspace
[14:03:30] <mforns>	 joal, I thought the opposite, sorry :[
[14:03:50] <joal>	 mforns: no problem, let's correct that :)
[14:03:54] <mforns>	 I just added the monitoring check fake data
[14:04:02] <mforns>	 alarms should stop in theory now
[14:04:29] <joal>	 mforns: aqs recovered :)
[14:04:34] <mforns>	 my bad!
[14:16:10] <elukey>	 ottomata: on neodym - sudo -E wmf-auto-reimage analytics1046.eqiad.wmnet -p T160333
[14:16:11] <stashbot>	 T160333: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333
[14:16:13] <elukey>	 that's it
[14:16:37] <elukey>	 it takes care of icinga, updating phab, rebooting the host in the end, etc..
[14:18:02] <ottomata>	 amaing
[14:46:15] <joal>	 ottomata: a minute in ops meeting for python deps ?
[14:46:51] <ottomata>	 k
[14:53:16] <elukey>	 a-team: still working on the ssl cert update in eqiad, will probably be a bit late to the standup.. if I don't make it I'll send e-scrum.. 
[14:53:26] <mforns>	 ok
[14:57:11] <wikibugs>	 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3140478 (10Nuria) @Beestra: @samwalton is looking at data in a database replica the data does not appe...
[15:00:24] <nuria>	 a-tem: staddduppppp
[15:01:08] <nuria>	 milimetric, fdans : staddduppp
[15:06:31] <wikibugs__>	 10Analytics-Cluster, 06Analytics-Kanban: Provision new Kafka clusters in eqiad and codfw with security features - https://phabricator.wikimedia.org/T152015#3140502 (10Ottomata)
[15:07:18] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban: Hadoop cluster expansion.  Add Nodes - https://phabricator.wikimedia.org/T152713#3140507 (10Ottomata)
[15:20:44] <mforns_>	 fdans, I have no task, I thought maybe you'd want someone to pair with you? I'd be happy to do so :]
[15:21:22] <fdans>	 mforns: sure! batcave-2?
[15:21:26] <mforns>	 k!
[15:22:40] <milimetric>	 mforns: do you have any old screenshots of ash's visual design of the topic explorer
[15:22:55] <milimetric>	 it's not in her latest and I just noticed
[15:23:04] <mforns>	 milimetric, mmm looking
[15:23:17] <milimetric>	 I remember it was basic, just headings of each area followed by the questions for that area, right?
[15:24:15] <milimetric>	 oh yea, it's only in the wireframes mforns, got it: https://www.dropbox.com/sh/948iggybzgkzrzp/AABVjHg_4MwagVtox0p_L0t0a/Wireframe%20flats%20feedback%20round%201/Research%20Mocks%20Round%201?dl=0&preview=Topic+Selector+with+Subcategories.png
[15:24:31] <mforns>	 milimetric, OK
[15:28:03] <wikibugs>	 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3140604 (10Marostegui) Hi! What's the status of this?
[15:28:30] <elukey>	 an1045 is up and running btw :)
[15:30:28] <wikibugs>	 (03PS6) 10Fdans: [wip] Add legacy pageviews metric [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/344114 (https://phabricator.wikimedia.org/T143906)
[15:33:35] <wikibugs__>	 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3140617 (10Ottomata) Happening today! :)
[15:40:02] <ottomata>	 nuria:  https://etherpad.wikimedia.org/p/analytics-el-table-rename
[15:41:53] <nuria>	 ottomata: updated wiki
[15:41:59] <nuria>	 ottomata: updated etherpad, that is
[15:42:54] <ottomata>	 nuria:  do we need a revert of hte EL revert?
[15:43:14] <ottomata>	 for the ua parser thing?
[15:43:25] <ottomata>	 ah no
[15:43:29] <ottomata>	 its already in
[15:43:29] <ottomata>	 i remember
[15:43:38] <ottomata>	 because i deployed it to beta and tested there
[15:50:37] <nuria>	 ottomata: yes
[15:51:25] <nuria>	 together with bumping len of varchar
[15:51:47] <nuria>	 fdans: i think you had a prior dashiki change you need to abandon that had also changes for AQS
[15:52:10] <fdans>	 nuria: I'm using that one
[15:52:25] <nuria>	 fdans: ah ok, sounds good
[15:52:37] <wikibugs>	 (03Abandoned) 10Nuria: Bump up pageviews.js to version that supports pagecounts [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/345197 (https://phabricator.wikimedia.org/T149358) (owner: 10Nuria)
[15:52:49] <fdans>	 nuria: fyi I'm pairing with marcel and we've got this working, starting work on multiple wikis now
[15:54:02] <nuria>	 ottomata: the ony thing i wonder is about that custom replication thing
[15:54:08] <nuria>	 ottomata: do we need to stop it?
[15:57:06] <ottomata>	 nuria:  right, cause it is not a cron, reading..
[15:57:08] <ottomata>	 uhhhh
[15:57:11] <ottomata>	 how does this script ever work?
[15:57:19] <ottomata>	 it seems to only grab the list of tables to replicate when it starts
[15:57:28] <ottomata>	 how does it know to get a new table if it is created after it starts...????
[15:57:50] <nuria>	 ottomata: it must, right?
[15:58:04] <nuria>	 ottomata: cause new schemas are being added w/o script being restarted
[16:00:51] <ottomata>	 nuria:  yeah i guess so
[16:01:28] <ottomata>	 welp, i dunno how that works, but yeah, let's stop
[16:01:29] <ottomata>	 adding
[16:02:55] <nuria>	 ottomata: be right abck
[16:02:57] <nuria>	 *back
[16:19:47] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban: Provision new Kafka clusters in eqiad and codfw with security features - https://phabricator.wikimedia.org/T152015#3140829 (10elukey) @Ottomata the perfect scenario would be to have a comparative test of performances on production hw but it would be really painful and h...
[16:20:46] <elukey>	 ottomata: --^ not sure if helpful but I tried to give me opinion :)
[16:21:04] <ottomata>	 thanks elukey
[16:21:05] <ottomata>	 yeah, hard to say
[16:37:44] <nuria>	 ottomata: also we need to check perf metrics, let's give a holler to performance
[16:37:49] <nuria>	 ottomata: (added that)
[16:38:45] <ottomata>	 k
[16:40:09] <nuria>	 ottomata: let's get on batcave while we do this 
[16:40:22] <ottomata>	 ja
[16:44:19] <wikibugs>	 (03PS2) 10Joal: Update sqoop script to better handle failures [analytics/refinery] - 10https://gerrit.wikimedia.org/r/345327
[16:44:34] <joal>	 ottomata: --^ it has been tested, let me know what you think of it :)
[16:45:17] <wikibugs__>	 10Analytics-Tech-community-metrics: Author names that include commata or "and" are split into separate identities in the frontend - https://phabricator.wikimedia.org/T161241#3140991 (10Aklapper)
[16:45:19] <wikibugs>	 10Analytics-Tech-community-metrics: Maniphest statistics show implausible numbers of submitted reports - https://phabricator.wikimedia.org/T161682#3140988 (10Aklapper)
[16:46:30] <nuria>	 ottomata: let's be on batcave while we do all this no?
[16:47:59] <ottomata>	 nuria:  ya
[16:48:04] <ottomata>	 oh you there now?
[16:48:05] <ottomata>	 o
[16:48:05] <ottomata>	 k
[16:48:43] <nuria>	 omw
[16:53:51] <wikibugs__>	 10Analytics: Upgrade pivot - https://phabricator.wikimedia.org/T161725#3141015 (10Nuria)
[17:09:07] <wikibugs__>	 10Analytics, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review: Implement server side filtering (if we should) - https://phabricator.wikimedia.org/T152731#2858381 (10Smalyshev) I'd definitely like to have per-wiki filtering for events. More granular would be nice but not a must. Something like `enedit` ma...
[17:13:11] <nuria>	 !log deploying eventlogging latetst: 28740773cea545215ea610c8c3e1a3ba36ef5a6a (UA changes)
[17:13:13] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:19:45] <nuria>	 !log restarted EL on eventlog1001 with new changeset and tables renamed
[17:19:46] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:22:23] <wikibugs__>	 10Analytics, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review: Implement server side filtering (if we should) - https://phabricator.wikimedia.org/T152731#3141090 (10GWicke) I really think we should consider exposing separate streams, one per wiki, ideally (if we can make the technical stuff work), as par...
[17:22:44] <elukey>	 team just completed the ssl upgrade work, all fine :)
[17:23:06] <elukey>	 analytics1045 seems working good, interestingly I noticed DiskChecker$DiskErrorException in there too (gone after a yarn restart)
[17:23:20] <elukey>	 anyhow, ping me if you see job failures etc.
[17:23:28] <elukey>	 hopefully there will be none :)
[17:23:34] <elukey>	 I am heading out for the evening! 
[17:23:38] <elukey>	 talk with you tomorrow! :)
[17:23:40] * elukey off
[17:24:33] <ottomata>	 oh interesting elukey.  cool laters!
[17:26:15] <fdans>	 mforns: 👌🏼👌🏼👌🏼👌🏼
[17:26:53] <wikibugs__>	 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3141118 (10Ottomata) Done! https://etherpad.wikimedia.org/p/analytics-el-table-rename
[17:26:55] <wikibugs>	 (03CR) 10Nuria: [C: 031] Adding renamed tables to sql union statements [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/344049 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria)
[17:27:04] <mforns>	 fdans, what is that :] ?
[17:27:27] <nuria>	 milimetric: someone with +2 needs to merge these changes: https://gerrit.wikimedia.org/r/#/c/344049/1
[17:27:33] <nuria>	 cc bearloga 
[17:27:45] <wikibugs>	 (03CR) 10Nuria: [V: 032 C: 032] Adding renamed tables to sql union statements [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria)
[17:27:56] <wikibugs__>	 (03CR) 10Nuria: [V: 032 C: 032] Adding renamed tables to sql union statements [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/344055 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria)
[17:28:02] <fdans>	 mforns: solved the promises conundrum, data is returning fiiine
[17:28:10] <fdans>	 working now on merging it and we're set :)
[17:28:15] <bearloga>	 nuria: I certainly don't have +2 on that repo
[17:28:51] <nuria>	 bearloga: funny, me neither, milimetric must, i have +2  it on all the other ones for reportupdater
[17:29:06] <wikibugs__>	 (03CR) 10Milimetric: [C: 04-1] Adding renamed tables to sql union statements (031 comment) [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/344049 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria)
[17:29:28] <milimetric>	 I have +2 nuria & bearloga, I'll merge after the small fix (missing a select)
[17:29:41] <milimetric>	 brb though, gotta go grab lunch
[17:30:01] <nuria>	 milimetric: no, will fix, give me a sec
[17:30:27] <nuria>	 milimetric: i had not see your comment, sorry, fixing
[17:32:20] <wikibugs__>	 (03PS2) 10Nuria: Adding renamed tables to sql union statements [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/344049 (https://phabricator.wikimedia.org/T160454)
[17:32:57] <wikibugs>	 (03CR) 10Nuria: Adding renamed tables to sql union statements (031 comment) [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/344049 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria)
[17:33:54] <wikibugs>	 (03CR) 10Milimetric: [V: 032 C: 032] Adding renamed tables to sql union statements [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/344049 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria)
[17:36:32] <wikibugs>	 10Analytics, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review: Implement server side filtering (if we should) - https://phabricator.wikimedia.org/T152731#3141153 (10Ottomata) > Does it mean I could "reconnect" to the stream at any point? Would it be possible to make it so I could choose that point e.g. b...
[17:50:23] <ottomata>	 heading to cafe...
[17:50:47] <mforns>	 fdans, sorry had to restart, what was the news?
[17:51:49] <fdans>	 mforns: problem with promises solved, now merging data :)
[17:51:57] <mforns>	 woohoo!
[17:52:32] <mforns>	 fdans, SoS is over, do you want help, otherwise I'm going to log off for today
[17:52:52] <fdans>	 mforns: I'm going out now to run a couple of errands pre trip
[17:52:55] <fdans>	 will be back later
[17:53:01] <mforns>	 ok, ok
[17:53:06] <fdans>	 let's talk tomorrow mforns :)
[17:53:10] <mforns>	 so, good luck and see you guys tomorrow!
[17:53:22] <mforns>	 k :]
[18:04:29] <wikibugs>	 10Analytics-Tech-community-metrics: Git's "Last Attracted Developers" lists established developers and developers without a First Commit Date - https://phabricator.wikimedia.org/T161309#3141264 (10Aklapper)
[18:16:54] <joal>	 milimetric: quick question
[18:17:01] <milimetric>	 shoot
[18:17:05] <joal>	 milimetric: What should I do of this page: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Mediawiki_history/Presentation
[18:17:27] <milimetric>	 oh, joal those were just brainstorms, you wanna delete it?
[18:17:31] <joal>	 I'm thinking of renaming it to: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/communication/Presentation_DATE
[18:17:42] <milimetric>	 I'll delete it
[18:17:57] <joal>	 as you prefer, I don't mind keeping it
[18:18:55] <milimetric>	 nope, just clutter, forgot about it
[18:18:56] <milimetric>	 gone
[18:18:57] <milimetric>	 thx
[18:19:03] <joal>	 thank you :)
[18:19:23] <joal>	 milimetric: also, provided a new version of CR for sqoop (tested and all)
[18:20:41] <milimetric>	 cool, will take a look
[18:22:57] <wikibugs>	 10Analytics-Tech-community-metrics: Check if we actually index all code repositories - https://phabricator.wikimedia.org/T161211#3141303 (10Aklapper) https://gerrit.wikimedia.org/r/labs/tools/grrrit is listed in https://github.com/Bitergia/mediawiki-repositories/blob/master/git_repositories.conf so I'd expect ac...
[18:25:11] <wikibugs__>	 10Analytics, 06Discovery, 10EventBus, 10MediaWiki-API, and 4 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#3141307 (10Smalyshev)
[18:26:43] <wikibugs>	 10Analytics, 06Discovery, 10EventBus, 10MediaWiki-API, and 4 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#3141325 (10Smalyshev) p:05Triage>03Normal
[18:30:11] <wikibugs__>	 10Analytics, 06Discovery, 10EventBus, 10Wikidata, and 3 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#3141332 (10Anomie) The pony being requested here is not going to happen in the action API, removing tag. This is probably an effective duplicate of {T1...
[18:30:20] <wikibugs>	 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#3141334 (10Aklapper) p:05High>03Low T138002 got fixed so three more dashboard to check, however...
[18:37:04] <wikibugs__>	 10Analytics, 06Discovery, 10EventBus, 10Wikidata, and 3 others: Create reliable change stream for specific wiki - https://phabricator.wikimedia.org/T161731#3141367 (10Smalyshev) I think EventStreams is closest to the goal too, but I want to have a complete description of the pony for the record so that we...
[19:00:05] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Change userAgent field to user_agent_map in EventCapsule - https://phabricator.wikimedia.org/T153207#2872765 (10Nuria) Changes have been deployed to production.
[19:03:46] <nuria>	 milimetric: i am so proud of my bubble plots: https://phabricator.wikimedia.org/F7065293
[19:04:13] <nuria>	 https://phabricator.wikimedia.org/F7065290
[19:05:11] <nuria>	 cc joal, while doing those  i also realized that the number of unidentified pageviews ('-') was akin to Search:Requests
[19:16:27] <milimetric>	 woah cool!
[19:19:43] <joal>	 nuria: Not sure I understand the point about "-" and Search:Request
[19:20:25] <nuria>	 joal, that  the number of unidentified pageviews is ~equal to the number of Search "pageviews"
[19:20:28] <joal>	 ottomata: I think I succesfully ran a spark job with scikit learn
[19:20:51] <joal>	 nuria: I think this is unrelated --> If they both come from pageivew,
[19:21:17] <joal>	 ottomata: Can you explain to me how I could setup a full jessie cluster in labs?
[19:21:43] <joal>	 nuria: When looking at title in pageviews table, there is no duplicate
[19:22:25] <joal>	 nuria: If you took the uri_path, then it makes sense :)
[19:22:49] <nuria>	 joal: ay sorry, no, I mean that numbers "in bulk" are similar as in "the number of pageviews for which we cannot find out the title" and "the number of search requests" is about the same
[19:22:51] <ottomata>	 joal: !!! amaazing!
[19:23:01] <ottomata>	 joal:  ya we just have to put new workers in it
[19:23:07] <ottomata>	 you don't need the master to be jessie, right?
[19:23:20] <nuria>	 joal: but nevermind, it was my own learning
[19:23:21] <ottomata>	 joal:  we can probably jsut delete the trusty workers
[19:23:30] <ottomata>	 then you'd have a single worker node jessie  cluster
[19:23:31] <joal>	 ottomata: only the worlkers, and I need some setup (packages installed
[19:24:00] <joal>	 ottomata: forgot to copy the numpy special package you pasted earlier, can you paste again?
[19:25:09] <ottomata>	 joal:  https://packages.debian.org/jessie-backports/python-numpy
[19:25:15] <ottomata>	 python-numpy=1:1.12.0-2~bpo8+1
[19:25:32] <joal>	 thanks ottomata 
[19:35:44] <joal>	 ottomata: master is cdh3-1, right ?
[19:35:52] <ottomata>	 sounds right :)
[19:36:28] <ottomata>	 double checking
[19:37:10] <ottomata>	 joal:  correct
[19:37:16] <joal>	 k thanks
[19:38:41] <wikibugs__>	 06Analytics-Kanban: Pageview Spike in Tagalog Wikipedia  mid-June 2016 - https://phabricator.wikimedia.org/T144635#3141582 (10Nuria) 05Open>03Resolved
[19:39:02] <wikibugs>	 06Analytics-Kanban, 06Operations, 06WMDE-Analytics-Engineering, 13Patch-For-Review, 15User-Addshore: /a/mw-log/archive/api on stat1002 no longer being populated - https://phabricator.wikimedia.org/T160888#3141583 (10Nuria) 05Open>03Resolved
[19:39:23] <wikibugs__>	 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Review Druid's logging configuration - https://phabricator.wikimedia.org/T155491#3141587 (10Nuria) 05Open>03Resolved
[19:39:35] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Move away Hue and Camus (and other crons) from analytics1027 - https://phabricator.wikimedia.org/T159527#3141589 (10Nuria) 05Open>03Resolved
[19:41:21] <joal>	 ottomata: Trying to add a new node to the cluster - How can I setup conf?
[19:45:35] <joal>	 actually ottomata - I deleted cdh3-[2|3|4], and it seems the cluster didn't like :(
[19:45:37] <wikibugs__>	 10Analytics: Update pivot to latest version - https://phabricator.wikimedia.org/T161630#3141606 (10Nuria)
[19:54:01] <joal>	 mwarf, I have lost ottomata :(
[20:25:37] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3141759 (10Pchelolo)
[20:33:49] <wikibugs>	 06Analytics-Kanban: Check abnormal pageviews for XHamster - https://phabricator.wikimedia.org/T158071#3141805 (10Nuria) Traffic starts getting significantly higher from 1/8/2016 onwards  Xhamster blog has loads of stats as of their traffic: https://xhamster.com/blog but couldn't find anything  that might be rela...
[20:36:41] <wikibugs__>	 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 3 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#3141818 (10Halfak)
[20:36:41] <ottomata>	 sorry joal!
[20:36:49] <ottomata>	 id din't realize i had signed off of irc
[20:36:52] <ottomata>	 cafe here being weird
[20:37:00] <ottomata>	 you can create a new node
[20:37:03] <ottomata>	 and then go to puppet class config
[20:37:09] <ottomata>	 and include role analytics cluster worker
[20:37:10] <ottomata>	 then run puppet
[20:37:11] <ottomata>	 should be it!
[20:38:00] <wikibugs__>	 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 3 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#2730864 (10Halfak)
[20:40:31] <ottomata>	 ah you have to run puppet twice?  hm.  
[20:40:34] <ottomata>	 ok oh well
[20:40:37] <ottomata>	 joal:  am i doing it for you
[20:40:42] <ottomata>	 on cdh3-6
[21:06:43] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Add autoincrement id to EventLogging MySQL tables. {oryx} - https://phabricator.wikimedia.org/T125135#3142024 (10Ottomata) Just a note:  now that EL tables were renamed, and active tables recreated in T160454, they should all have an...