[01:14:35] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Mediawiki History Druid indexing failed - https://phabricator.wikimedia.org/T170493#3468720 (10Nuria) mmm... operator error: format is yyyy-MM-dd HH:mm:ss.S!
[01:16:41] <wikibugs>	 (03PS3) 10Nuria: Modifying ingestion spec after additions to edit history [analytics/refinery] - 10https://gerrit.wikimedia.org/r/366327 (https://phabricator.wikimedia.org/T170493)
[07:18:23] <wikibugs>	 10Analytics, 10EventBus, 10Scap: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3467218 (10Joe) indeed.
[07:41:53] <wikibugs>	 10Analytics, 10EventBus, 10Scap: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3469035 (10elukey) Puppet was broken on kafka2002:  ``` ESC[1;31mError: Could not set home on user[eventlogging]: Execution of '/usr/sbin/usermod -d...
[10:55:19] <wikibugs>	 10Analytics: Upgrade AQS to node 6.11 - https://phabricator.wikimedia.org/T170790#3469578 (10elukey) p:05Normal>03High Installed new debian packages on deployment-aqs01.eqiad.wmflabs / deployment-aqs02.eqiad.wmflabs / deployment-aqs03.eqiad.wmflabs, ready for a test. Since this upgrade contains security upda...
[11:05:30] * elukey lunch
[12:13:25] <elukey>	 !log executed sudo apt-get remove openjdk-8-jre openjdk-8-jre-headless on druid nodes
[12:13:26] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[12:19:05] <mforns>	 hi team :]
[12:19:25] <elukey>	 mforns: o/
[12:19:36] <mforns>	 elukey! :]
[12:19:40] <elukey>	 the script has completed its work on dbstore1002 :)
[12:19:50] <elukey>	 we can check and then restart the full purge
[12:20:37] <mforns>	 elukey, \o/ until end of 2014 right
[12:20:38] <mforns>	 ?
[12:21:48] <elukey>	 yep! actuall let me grab the interval
[12:21:53] <elukey>	 since I've ran it two times
[12:22:03] <mforns>	 k
[12:23:38] <elukey>	 should be 'start_ts': '20090501170620', 'batch_size': 10000, 'end_ts': '20141231170620'}
[12:45:53] <fdans>	 mforns: which aqs url would you use to query uniques for say all of wikiquote?
[12:46:05] <mforns>	 fdans, mmmmm
[12:46:05] <fdans>	 I seem to be only able to query all of wikipedia
[12:48:03] <mforns>	 fdans, how do you get wikipedia family?
[12:48:28] <mforns>	 I can not get data from it
[12:48:37] <fdans>	 mforns: if i do 
[12:48:38] <fdans>	 https://wikimedia.org/api/rest_v1/metrics/unique-devices/wikipedia.org/all-sites/monthly/2016050500/2017050500
[12:48:44] <fdans>	 that works
[12:49:20] <fdans>	 ohhh but I guess that's just for the splash page
[12:49:23] <fdans>	 not the whole family
[12:49:46] <fdans>	 but.... aren't you supposed to be able to query a whole family in uniques mforns ?
[12:50:08] <mforns>	 ah ok, monthly one sec
[12:53:48] <mforns>	 fdans, mmmmm, not sure
[12:54:13] <mforns>	 the numbers are too low to be the whole wikipedia family
[12:54:18] <fdans>	 yeah
[12:54:38] <mforns>	 maybe the unique devices per project family are not yet loaded in cassandra
[12:55:01] <fdans>	 mforns: should they be in theory?
[12:56:02] <mforns>	 fdans, I don't think so: https://phabricator.wikimedia.org/T143927
[12:56:25] <mforns>	 they are still being vetted, and the task for making them public is still untouched
[12:57:25] <fdans>	 oh I misunderstood everything then... I thought I had to deal with a special case in the uniques metric where we can query all uniques for a family
[12:57:35] <fdans>	 (but not for all projects)
[13:06:10] <elukey>	 !log stop cassandra load bundle, restarting AQS for jvm updates
[13:06:10] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:06:32] <elukey>	 a-team: I am restarting cassandra on aqs, let me know if you notice anything weird
[13:41:32] * fdans lunch 🇪🇸
[14:13:05] <elukey>	 ottomata: if you have time after standup I'd need some help in figuring out what's happening on the analytics labs kafka brokers
[14:13:11] <elukey>	 I can see auth going on and happening
[14:13:19] <elukey>	 but can't produce to any topic
[14:13:21] <elukey>	 for weird issues
[14:13:29] <elukey>	 for example, ISR are always 1/3
[14:13:42] <ottomata>	 +1 elukey i'm for you whenever
[14:13:54] <elukey>	 super thanks!
[14:33:22] <leila>	 hey ottomata. I sent Nithum an email about testing TensorFlow on AMD GPU, per the discussion we had in the last chill hour. you are in bcc fyi, I'll let you know if I hear back.
[14:39:27] <ottomata>	 oh great, thank you
[14:46:17] <mforns>	 milimetric, do you have 10 mins before standup?
[14:46:23] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3470314 (10fdans)
[14:46:24] <milimetric>	 yes, cave?
[14:46:28] <mforns>	 yep!
[14:59:12] <wikibugs>	 10Analytics-Kanban: Initial Launch of new  Wikistats 2.0  website - https://phabricator.wikimedia.org/T160370#3470412 (10Nuria)
[14:59:14] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Define, Document  (and test) Desktop and Mobile browser support for wikistats 2.0 - https://phabricator.wikimedia.org/T170457#3470411 (10Nuria) 05Open>03Resolved
[15:00:41] <nuria_>	 fdans: standdduppp
[15:14:27] <elukey>	 ottomata: (ignorant me asking) what did you change for the mysql consumers in EL? I can still see two of them, the m4 and the eventbus one in the logs
[15:15:35] <wikibugs>	 (03PS4) 10Nuria: Modifying ingestion spec after additions to edit history [analytics/refinery] - 10https://gerrit.wikimedia.org/r/366327 (https://phabricator.wikimedia.org/T170493)
[15:24:37] <elukey>	 !log restart cassandra loading after maintenance via hue
[15:24:38] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:31:51] <nuria_>	 milimetric: my druid spec change is reday to merge if data looks good in druid, cc mforns_ :https://gerrit.wikimedia.org/r/#/c/366327/4/oozie/mediawiki/history/druid/load_mediawiki_history.json.template
[15:32:12] <mforns_>	 lookin
[15:33:42] <elukey>	 mforns: let me know when you want to data-vet eventlogging
[15:34:09] <mforns>	 elukey, in 10 minutes?
[15:35:08] <elukey>	 oook!
[15:37:43] <mforns>	 nuria_, change looks good, is the data in pivot? probably in a "hidden" dataset? do you have a link?
[15:38:43] <milimetric>	 mforns: then later we can get back to the router thing
[15:38:54] <mforns>	 milimetric, OK!
[15:42:20] <elukey>	 mforns: you can take a bit more time, I am debugging kafka with Andrew
[15:42:39] <mforns>	 elukey, ok, will ping milimetric first then if he's ok
[15:52:51] <fdans>	 Off to the dentist :(
[15:52:58] <fdans>	 I am the most miserable man on earth
[15:53:14] <harej>	 (Be thankful you have access to a dentist.)
[15:54:33] <fdans>	 harej: not really complaining about the dentist, but my own teeth :)
[15:59:41] <harej>	 Ah, yes.
[16:00:14] * harej is somewhat bitter because a doctor just canceled on him
[16:01:34] <elukey>	 mforns: whenever you are ready, I keep working on kafka in the meantime
[16:01:43] <mforns>	 elukey, ok, let's do it
[16:01:48] <mforns>	 batcave!
[16:38:14] <elukey>	 ottomata: can you ssh to zk2-1.analytics.eqiad.wmflabs ?
[16:41:13] <ottomata>	 grrrr
[16:41:14] <ottomata>	 no
[16:41:33] <nuria_>	 mforns: dataset  should be visible  but maybe snapshot is not  where it should be? https://pivot.wikimedia.org/#mediawiki-history-beta/
[16:41:45] <mforns>	 k
[16:41:58] <nuria_>	 mforns: no, it is, i imported may
[16:41:58] <ottomata>	 elukey: 2017-07-25T16:17:02.940437+00:00 zk2-1 nslcd[487]: [b127f8] <group/member="root"> no available LDAP server found: Server is unavailable
[16:42:01] <ottomata>	 from horizon log
[16:42:13] <ottomata>	 maybe kill it and make a new one :/
[16:42:13] <ottomata>	 dunno
[16:42:31] <ottomata>	 elukey:  or just apply a zk server on one (or all) of the kafka3 nodes
[16:42:34] <nuria_>	 mforns: ... mmmm wait
[16:42:34] <ottomata>	 since you know those work :/
[16:43:59] <elukey>	 :(
[16:51:17] <nuria_>	 mforns: dataset is this one: https://pivot.wikimedia.org/#mediawiki-history-beta
[16:51:24] <mforns>	 nuria_, ok thanks!
[16:51:43] <nuria_>	 mforns: it can be seen cause timstamps are on the new format yyyy:mm:dd hh:mm:ss.S
[16:51:59] <nuria_>	 mforns: it includes up to the month of may
[16:52:05] <mforns>	 k
[17:10:10] <travis-ci>	 wikimedia/mediawiki-extensions-EventLogging#675 (wmf/1.30.0-wmf.11 - 019d339 : Kunal Mehta): The build has errored.
[17:10:10] <travis-ci>	 Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.30.0-wmf.11
[17:10:10] <travis-ci>	 Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/257396082
[17:13:18] <wikibugs>	 10Analytics: Add purge info for Kartographer schema - https://phabricator.wikimedia.org/T171622#3471039 (10mpopov)
[17:14:03] <wikibugs>	 10Analytics, 10Discovery, 10Discovery-Analysis: Add purge info for Kartographer schema - https://phabricator.wikimedia.org/T171622#3471052 (10mpopov)
[17:40:49] <robh>	 Heyas analytics folks, anyone about who can chat with me about the new druid100[456] systems setup?
[17:40:53] <robh>	 ottomata: ^ ?
[17:41:21] <robh>	 I need to make the racking task for chris, and I can see that druid100[123] all exist in different racks and rows.  Are these new systems repalcing the old, or adding to the pool?
[17:43:08] <robh>	 I'm assuming adding to the pool as the existing druid100[123] are fairly new still
[17:43:11] <ottomata>	 robh heya
[17:43:14] <ottomata>	 adding to pool!
[17:43:29] <ottomata>	 so the more diverse the better, but you could pair them in the same racks, so we'd have 2 in each rack, if you like
[17:43:35] <robh>	 ok, so ideally in different rows (so right now we have in a,c,d, toss one of the new in b, and then in different racks in the others
[17:43:41] <ottomata>	 sure ya
[17:43:46] <robh>	 i think we can get by without using the same rack
[17:43:53] <robh>	 but i'll mention its acceptable if nothing else is available
[17:43:59] <ottomata>	 k cool
[17:44:00] <robh>	 jessie or stretch?  (existing are jessie)
[17:44:00] <ottomata>	 danke
[17:44:04] <ottomata>	 jessie
[17:44:09] <ottomata>	 we need java 7
[17:44:11] <robh>	 cool
[17:44:11] <ottomata>	 or it doesn't work :/
[17:44:22] <ottomata>	 gotta upgrade hadoop to java 8 before we can use more stretch
[17:44:27] <elukey>	 ottomata: elukey@zk1-1:~$ \p/
[17:44:31] <elukey>	 \o/
[17:44:35] <ottomata>	 the orig!?
[17:44:37] <mforns>	 elukey, I checked another table, and I found that clientIp values are still there, so we're not updating in vain :]
[17:44:58] <elukey>	 ottomata: yep! It was suffering from the LDAP outage, puppet broken == no updates to the CA
[17:45:09] <mforns>	 elukey, plus the sanitization worked well, because before the timestamp limit, clientIp was NULL and after not.
[17:45:09] * elukey hugs mforns 
[17:45:15] <mforns>	 hehehe
[17:45:33] <elukey>	 mforns: eventlogcleaner /usr/local/bin/eventlogging_cleaner --whitelist /etc/eventlogging/whitelist.tsv --older-than 570 --newer-than 940 --batch-size 100000
[17:45:41] <elukey>	 +1?
[17:45:59] <mforns>	 elukey, why 570 and 940?
[17:46:03] <elukey>	 {'end_ts': '20160102174556', 'batch_size': 10000, 'start_ts': '20141228174556'}
[17:46:14] <mforns>	 +2
[17:46:20] * elukey runs the script
[17:46:48] <robh>	 ottomata: oh, when we finish these installs
[17:46:50] <robh>	 hand off to you?
[17:46:52] <robh>	 or someone else?
[17:47:44] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad: rack/setup/install druid100[456].eqiad.wmnet - https://phabricator.wikimedia.org/T171626#3471295 (10RobH)
[17:47:56] <robh>	 i listed you, but we can change if needed.
[17:48:37] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad: rack/setup/install druid100[456].eqiad.wmnet - https://phabricator.wikimedia.org/T171626#3471329 (10RobH)
[17:58:23] <nuria_>	 fdans: let me know if you need help with npm issue and CI
[18:06:15] <wikibugs>	 10Analytics: Stop collecting Data for pageCreate schema, archive table on hdfs - https://phabricator.wikimedia.org/T171629#3471480 (10Nuria)
[18:06:54] <ottomata>	 robh ya that's fine
[18:07:02] <robh>	 cool
[18:07:11] <ottomata>	 thank you
[18:08:47] <wikibugs>	 10Analytics-Kanban: Archive PageContentSaveComplete in hdfs while we continue collecting data  - https://phabricator.wikimedia.org/T170720#3471501 (10Nuria)
[18:10:39] <wikibugs>	 10Analytics: Stop collecting Data for PageCreation schema, archive table on hdfs  - https://phabricator.wikimedia.org/T171629#3471556 (10Nuria)
[18:16:28] <elukey>	 ottomata: I am able to run the kafka producer without any errors!
[18:16:30] <elukey>	 \o/
[18:16:50] <elukey>	 tomorrow I'll restart doing more careful tests
[18:16:57] <elukey>	 but it looks like I am unblocked
[18:17:17] <nuria_>	 mforns: let me know if you see any  problems on data on mediawiki history
[18:17:30] <elukey>	 I was also using the wrong zk endpoint (without the /kafka/mothership2etc..) for the ACLs
[18:17:45] <elukey>	 feel better now :)
[18:17:46] <elukey>	 thanks ottomata !
[18:17:51] * elukey goes offline for today :)
[18:26:58] <mforns>	 nuria_, data looks good as far as I can see
[18:28:35] <ottomata>	 great elukey :)
[18:28:39] <ottomata>	 laters!
[18:48:40] <nuria_>	 mforns: ok merging new spec
[18:48:51] <mforns>	 k! +2
[18:49:04] <nuria_>	 mforns: https://gerrit.wikimedia.org/r/#/c/366327/
[18:49:12] <nuria_>	 mforns: take a last look, it cannot hurt
[18:49:26] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 032] Modifying ingestion spec after additions to edit history [analytics/refinery] - 10https://gerrit.wikimedia.org/r/366327 (https://phabricator.wikimedia.org/T170493) (owner: 10Nuria)
[18:57:25] <mforns>	 milimetric, I'm looking into using router-view's code, if I get stuck I ping you, ok?
[18:57:39] <milimetric>	 mforns: I looked at it too, it's pretty complicated
[18:57:53] <milimetric>	 so I'm happy to brain bounce
[18:58:04] <milimetric>	 I feel like I'm carelessly sending you down a hard path
[18:58:06] <mforns>	 k milimetric wanna cave?
[18:58:19] <milimetric>	 DISCLAIMER: I have no idea how hard this is, it could be bad :)
[18:58:22] <milimetric>	 k, sure, omw
[19:15:19] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad: rack/setup/install druid100[456].eqiad.wmnet - https://phabricator.wikimedia.org/T171626#3471295 (10Cmjohnson)
[19:26:23] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Add piwik to wikistats 2.0 site - https://phabricator.wikimedia.org/T171642#3471959 (10Nuria)