[03:11:56] <wikibugs>	 Quarry: Option to kill task by myself - https://phabricator.wikimedia.org/T137266#2362997 (Dvorapa)
[03:12:32] <wikibugs>	 Quarry: Option to kill task by myself - https://phabricator.wikimedia.org/T137266#2362997 (yuvipanda) If you fix it and re-execute again the previous task will automatically get killed!
[03:49:14] <wikibugs>	 Quarry: Option to kill task by myself - https://phabricator.wikimedia.org/T137266#2363073 (Dvorapa) Open>Invalid Oh, I didn't realized, thanks
[04:03:16] <wikibugs>	 Quarry: Option to kill task by myself - https://phabricator.wikimedia.org/T137266#2363095 (Dvorapa) Invalid>Open But sometimes you just don't want to run it again. E.g. if you find out the result is too large so you just want to kill it and end working on it
[04:04:05] <wikibugs>	 Quarry: Option to kill task by myself - https://phabricator.wikimedia.org/T137266#2363097 (Dvorapa)
[04:10:54] <wikibugs>	 Quarry: Add an option to export result in Wikilist - https://phabricator.wikimedia.org/T137268#2363098 (Dvorapa)
[06:05:21] <wikibugs>	 Analytics, Commons, Multimedia, Tabular-Data, and 4 others: Review shared data namespace (tabular data) implementation - https://phabricator.wikimedia.org/T134426#2265305 (pwalsh) Hi,  I'm also working on the JSON Table Schema specification (and supporting tooling). It seems quite well suited her...
[06:28:14] <moritzm>	 elukey: there's a failed RAID on analytics1049
[06:32:31] <moritzm>	 ah, just already wrote that in -operations two minutes earlier :-)
[06:33:24] <elukey>	 moritzm: thankssss! Working on it :)
[06:33:42] <elukey>	 the host is a regular hadoop node, I removed it from the cluster and scheduled downtime
[06:56:32] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: analyitics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2363239 (elukey)
[06:56:41] <elukey>	 moritzm: --^ :)
[07:20:37] * elukey commutes to the office
[07:50:12] <elukey>	 joal: aloha!
[07:50:20] <joal>	 Ciao elukey !
[07:50:22] <joal>	 o/
[07:50:31] <elukey>	 Bonjour!
[07:50:37] <joal>	 How are you today?
[07:50:45] <elukey>	 one of the kafka1022 partitions has only 10% of space left :P
[07:51:03] <elukey>	 mmm but it might get purged today, checking
[07:51:06] <elukey>	 ggoooooood
[07:51:13] <joal>	 elukey: great: )
[07:51:17] <elukey>	 really looking forward to deploy vk
[07:51:19] <elukey>	 and you?
[07:51:25] <joal>	 elukey: great you're good, not kafka lack of space;)
[07:51:44] <elukey>	 I have been super distant from aqs load testing sorry, let me know if you need any help or if you are super busy :(
[07:51:47] <joal>	 Good as well, Lino had a difficult night, which means me as well, but the rest ok :)
[07:52:29] <joal>	 elukey: I can give you heads-up on qs if you want, but globally I can handle the rhythm
[07:53:06] <elukey>	 nono I am following what you are doing, just wanted to let you know that I am available if you need to offload things :)
[07:53:55] <joal>	 elukey: Thanks mate :)
[07:54:08] <joal>	 elukey: Currently trying to help others on other fronts as well
[07:54:32] <elukey>	 I know! You always do 100 things at the time, not sure how you do it :)
[07:54:49] <elukey>	 anyhow, bad news.. on kafka1022 the first timestamp is Jun 3
[07:54:54] <joal>	 elukey: I fail at least 30% of them (remember last week)
[07:55:12] <joal>	 elukey: then manual purging will b
[07:55:15] <joal>	 e the thing I guess
[07:56:48] <joal>	 elukey: I have seen an open bug on the issue ion kafka jira, hopefully will be solved soon?
[07:58:03] <elukey>	 joal: I am not that confident, there is no traction from upstream from what I can see.. and we don't have super solid proofs about what is causingit
[07:58:20] <elukey>	 anyhow, to add more joy, /var/spool/kafka/b contains only maps and text partitions
[07:58:34] <elukey>	 so I'd probably need to purge text this time
[07:59:12] <elukey>	 maybe set topic retention days to 4
[07:59:24] <elukey>	 for webrequest_text
[07:59:34] <elukey>	 should do the trick and free a lot of space
[07:59:57] <elukey>	 (brb)
[08:00:11] <joal>	 elukey: I think you're right
[08:00:42] <elukey>	 joal: coffee and then I'll prepare the command, do you mind to review it?
[08:00:50] <joal>	 I don't :)
[08:00:54] <elukey>	 thanks :)
[08:01:00] <elukey>	 four eyes are better than two
[08:01:19] <joal>	 Correct, even I probably count for only one given the night I had ;)
[08:11:26] <elukey>	 joal: kafka configs --alter --entity-type topics --entity-name webrequest_text --add-config retention.ms=345600000
[08:13:12] <joal>	 elukey: except for the command itself (kafka-configs IIRC), the rest seems fine (I double checked the value, correct:)
[08:13:50] <elukey>	 joal: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Kafka/Administration#Temporarily_Modify_Per_Topic_Retention_Settings - I can't remember those commands :P
[08:14:16] <joal>	 k elukey :)
[08:14:27] <elukey>	 super, proceeding :)
[08:17:15] <elukey>	 !log lowering down webrequest_text kafka topic retention time from 7 days to 4 days to free disk space
[08:17:18] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[08:24:06] <elukey>	 ok much better now
[08:26:29] <joal>	 I thanks for taking care of that elukey :)
[08:30:44] <elukey>	 joal: :)
[08:31:16] <elukey>	 I was thinking that we could probably apply a maximum retention bytes set to something like 10TB
[08:31:48] <jand_wmde>	 I wonder if there is analytics on beta (using mock data?). I don't have access to the  webrequestlog (yet) and via beta I could get to know the system already. Also it seems to make sense to be able to test queries before running anyway :-)
[08:32:00] <elukey>	 retention.bytes should correspond to the maximum broker log size afaik
[08:41:41] <elukey>	 joal: woooooo somebody found the buuuuggg https://issues.apache.org/jira/browse/KAFKA-3802
[08:42:35] <joal>	 elukey: YAY !
[08:42:39] <elukey>	 jand_wmde: Hi! We used to have something in beta but as far as I remember the cluster was unstable and not really able to accomodate queries..
[08:43:18] <jand_wmde>	 OK
[08:45:33] <elukey>	 !log removed temporary retention override for kafka webrequest_text topic (T136690)
[08:45:34] <stashbot>	 T136690: Kafka 0.9's partitions rebalance causes data log mtime reset messing up with time based log retention - https://phabricator.wikimedia.org/T136690
[08:45:36] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[08:46:15] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: Kafka 0.9's partitions rebalance causes data log mtime reset messing up with time based log retention - https://phabricator.wikimedia.org/T136690#2363360 (elukey) Somebody was able to narrow down the problem in https://issues.apache.org/jira/browse/KAFKA-3802 finding...
[09:08:42] <elukey>	 joal: I'd propose to set the maximum log size to 10TB with https://gerrit.wikimedia.org/r/#/c/293270/1/modules/role/manifests/kafka/analytics/broker.pp
[09:10:19] <joal>	 elukey: hm, can you remind me how much space there is on the brokers?
[09:15:18] <mforns>	 hi team!
[09:17:57] <joal>	 Hi mforns :)
[09:18:04] <mforns>	 hey joal!
[09:23:22] <joal>	 Hi early ottomata :)
[09:23:37] <ottomata>	 HI!
[09:23:38] <ottomata>	 :)
[09:23:48] <ottomata>	 how goes it?!
[09:23:52] <ottomata>	 you caught me!
[09:24:02] <ottomata>	 heh, i drove my girlfriend to the airport super early
[09:24:06] <ottomata>	 and i kinda felt like just stayin gup
[09:24:09] <ottomata>	 probably will crash later
[09:24:15] <joal>	 indeed very early for you !
[09:24:19] <joal>	 huhu :)
[09:24:42] <joal>	 As you long it's you and not her ;)
[09:26:45] <elukey>	 ottomata: o/
[09:26:53] <elukey>	 what time is it in NYC?
[09:27:06] <elukey>	 5 AM?
[09:27:12] <ottomata>	 yup!
[09:27:19] <ottomata>	 got up 1.5 hours ago though :O
[09:27:34] <elukey>	 :D
[09:27:43] <ottomata>	 we will see how this goes! I am pretty awake right now...
[09:28:09] <elukey>	 if you are super awake, I thought about https://gerrit.wikimedia.org/r/#/c/293270
[09:28:12] <elukey>	 :P
[09:28:25] <elukey>	 CI is down atm so no possibility to test it with pcc etc..
[09:29:01] <elukey>	 joal: re space - it is a bit tricky because we can set up the broker log size but not the partition sizes.. and we store them on disk partitions
[09:29:29] <elukey>	 I put 10TB because from what I can see it seems the threshold that we can sustain..
[09:31:10] <ottomata>	 elukey:  cool, so, ja
[09:31:11] <ottomata>	 hm
[09:31:17] <ottomata>	 just looking at partitions sizes
[09:31:25] <ottomata>	 looks likea webrequest_text is 614G
[09:31:39] <ottomata>	 so 24 of those is about 15T
[09:33:00] <ottomata>	 i guess 10 is good because we can be sure it will prune
[09:33:17] <ottomata>	 but, it sounds like it will probably only prune webrequest_text, right?
[09:33:30] <ottomata>	 an upload partition is 152G
[09:33:40] <ottomata>	 which will be around 3.6T
[09:33:44] <ottomata>	 in a week i guess
[09:33:46] <elukey>	 I brutally checked https://grafana.wikimedia.org/dashboard/db/kafka?panelId=17&fullscreen
[09:34:02] <ottomata>	 oh aye
[09:34:03] <ottomata>	 cool
[09:34:04] <elukey>	 for the past 30 days
[09:34:11] <ottomata>	 hmmm
[09:34:13] <elukey>	 and yeah text will be the target :(
[09:34:22] <elukey>	 but at least it should give us some relief
[09:34:24] <ottomata>	 but, that is for all logs
[09:34:26] <ottomata>	 hm
[09:34:44] <ottomata>	 oh that's per broker ja
[09:34:44] <ottomata>	 hm
[09:34:50] <ottomata>	 hm, i mean
[09:34:54] <ottomata>	 sounds like the best we can do
[09:35:19] <elukey>	 it is not ideal but I wanted to have a chat with you and Jo about a permantent solution
[09:35:40] <ottomata>	 elukey:  since that will really only affect text, that will give us about 4.5 days of text i think
[09:35:58] <ottomata>	 hmm elukey i wonder if instead of setting it globally, if we can get the retention bytes setting to work for upload specificly
[09:36:05] <ottomata>	 and just set it low
[09:36:09] <ottomata>	 would that actually help us?
[09:36:15] <ottomata>	 i guess only if all disks had an upload partition
[09:36:15] <ottomata>	 hm.
[09:36:17] <ottomata>	 and they might now
[09:36:19] <ottomata>	 not
[09:36:28] <ottomata>	 ja they won't
[09:36:28] <ottomata>	 hm
[09:36:29] <ottomata>	 ahhh
[09:36:52] <ottomata>	 oh, if things were distributed 100% evenly
[09:36:53] <ottomata>	 they would
[09:36:58] <ottomata>	 rep factor is 3
[09:36:59] <ottomata>	 24 partitions
[09:37:05] <ottomata>	 so 72 partition replicas
[09:37:11] <ottomata>	 spread across 6 nodes would be 12 per node
[09:37:29] <ottomata>	 buuut, unfortunetly partitions are not distributed evenly
[09:37:34] <ottomata>	 mostly evenly, but not totally
[09:37:41] <elukey>	 I found this morning a disk partition with only text and maps :(
[09:37:46] <elukey>	 so I had to purge text
[09:37:47] <ottomata>	 yeah, makes sense
[09:37:50] <elukey>	 (4 days)
[09:37:52] <ottomata>	 aye
[09:38:08] <ottomata>	 hm, i bet partitions are spread evenly per nodes
[09:38:09] <ottomata>	 just not per disks
[09:38:51] <ottomata>	 elukey: maybe we can be a little more explicit, and instead of setting this globally, just set it on text
[09:38:54] <ottomata>	 and set the retention to 5 days
[09:38:56] <ottomata>	 and just leave it
[09:39:12] <ottomata>	 a 10T retention will only affect text anyway
[09:39:18] <ottomata>	 so it might be nice to be explicit about it
[09:39:43] <ottomata>	 oh, but, hm, would that solve hte problem?  i guess not since our problem is with mitime and time retention
[09:39:56] <elukey>	 yeah...
[09:40:26] <elukey>	 for example, kafka1020 has mtime set to yesterday
[09:40:39] <ottomata>	 jaaa right, yeah
[09:40:40] <ottomata>	 hm ok
[09:40:42] <ottomata>	 ok.
[09:40:54] <ottomata>	 can we set this just on text then instead of changing it globally?
[09:41:00] <ottomata>	 the bytes retention?
[09:41:22] <ottomata>	 hmmmm
[09:41:25] <ottomata>	 hmmm
[09:41:26] <elukey>	 I think that we could apply a kafka override and let it there permanently, but it will not be puppetized..
[09:41:47] <ottomata>	 hmmm, and also, thikning more, this would affect upload too
[09:41:53] <ottomata>	 since in 2 weeks we'd grow over 10tb
[09:41:56] <ottomata>	 even with upload
[09:42:22] <ottomata>	 wait
[09:42:48] <ottomata>	 ok elukey let's do it
[09:42:49] <ottomata>	 yeah
[09:46:57] <elukey>	 all right, I'll wait until CI is up and do a pcc to verify :)
[09:49:12] <ottomata>	 ha ok, i would just do it! :)
[09:49:23] <ottomata>	 it's only going to change the server.properties file
[09:49:29] <ottomata>	 and you can inspect it before you restart any brokers
[09:52:03] <elukey>	 don't feel that confident to merge stuff now :P
[09:54:15] <elukey>	 ah ottomata, joal: analytics1049 is out of the cluster, disk broken
[09:54:49] <ottomata>	 dohhh
[09:55:08] <ottomata>	 elukey:  are we already out another too?
[09:55:10] <ottomata>	 or was that fixed?
[09:56:55] <elukey>	 no no it was fixed (1047 IIRC)
[09:57:08] <elukey>	 we are not lucky with disks :(
[09:57:32] <elukey>	 I think that Chris could replace kafka1018 and analytics1049 together
[09:57:49] <elukey>	 maybe today we can chat with him when he comes online
[09:58:21] <elukey>	 new vk uploaded to carbon and installed via apt on cp1047, all good
[09:58:34] <elukey>	 will install on the whole maps cluster today
[10:01:19] <ottomata>	 ja ok
[10:01:22] <ottomata>	 nice
[10:01:56] <joal>	 elukey: I stepped of that discussion in which I wouldn't have had much value ;)
[10:02:12] <elukey>	 joal: not true! do you like the idea?
[10:02:41] <joal>	 elukey: I think it's an ok temporary solution, but I don't like the idea to keep it this way :)
[10:02:55] <joal>	 elukey: temporal retention makes more sens functionally,
[10:03:10] <ottomata>	 ja especially since it isn't across all topics. kinda funky
[10:03:11] <ottomata>	 but ja
[10:03:11] <joal>	  but since it's buggy now, let's make sure we don't break the things first
[10:03:17] <ottomata>	 joal, at least the bug i made is getting a little attention
[10:03:26] <ottomata>	 https://issues.apache.org/jira/browse/KAFKA-3802
[10:03:35] <joal>	 ottomata: I've seen that yeas, that great: )
[10:03:55] <elukey>	 yep I was about to say that! They have a repro! \o/
[10:04:21] <elukey>	 thing is, the fix will probably be on 0.11 (best case)
[10:04:21] <joal>	 more than that: culprit identified, even if no solution yet
[10:04:37] <ottomata>	 naw, they might do a backport :)
[10:04:38] <ottomata>	 who knows
[10:04:42] <ottomata>	 its kinda critical
[10:04:51] <joal>	 To me this bug is big enough to necesitate a backport yes
[10:04:52] <ottomata>	 we can ask for it at least if someone finds a fix
[10:04:54] <ottomata>	 yeah
[10:04:58] <elukey>	 I was about to say that we could add the patch to the deb but no :P
[10:08:17] <ottomata>	 heh yeah...
[10:22:47] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Operations, ops-eqiad: Smartctl disk defects on kafka1012 - https://phabricator.wikimedia.org/T136933#2363623 (Ottomata) Let’s do today.   Apparently analytics1049 has a bad disk too.  Maybe we can do them together!  Ping either elukey or I when you are online...
[11:29:15] <wikibugs>	 Analytics: Update mediwiki hooks to generate data for new event-bus schemas - https://phabricator.wikimedia.org/T137287#2363796 (JAllemandou)
[11:30:51] * joal is AFK for a whiole
[11:32:03] <wikibugs>	 Analytics: Update mediwiki hooks to generate data for new event-bus schemas - https://phabricator.wikimedia.org/T137287#2363824 (Ottomata) p:Triage>Normal a:Ottomata
[11:53:46] <ottomata>	 heya joal, yt?
[11:56:40] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2363964 (Danny_B)
[11:56:56] <joal>	 ottomata: here but not for long
[11:57:00] <joal>	 what's up ottomata ?
[11:57:22] <ottomata>	 schema qs
[11:57:27] <joal>	 sure
[11:57:28] <ottomata>	 in https://gerrit.wikimedia.org/r/#/c/288210/9/jsonschema/mediawiki/revision_create/1.yaml
[11:57:35] <ottomata>	 rev_bytes
[11:57:42] <ottomata>	 in mw, this looks like it is called rev_len
[11:57:51] <ottomata>	 where did we get rev_bytes from, the db?
[11:57:57] <joal>	 I think so yeah
[11:58:14] <joal>	 halfak names it this way, and ezachte as well
[11:58:22] <ottomata>	 it is rev_len in db too
[11:58:24] <ottomata>	 just chcked
[11:58:24] <ottomata>	 checked
[11:58:46] <ottomata>	 will leave a comment, i guess we ask halfak
[11:58:49] <ottomata>	 rev_len seems more correct
[11:59:27] <joal>	 ottomata: k
[11:59:36] <ottomata>	 also a q about title vs page_title
[11:59:39] <ottomata>	 rev create has page_title
[11:59:43] <ottomata>	 page_delete has title
[11:59:53] <ottomata>	 should we use page_title?
[12:00:02] <joal>	 ottomata: good catch ! We should, yes
[12:01:01] <ottomata>	 same for namespace
[12:01:36] <elukey>	 ottomata: I added a comment to the ZK max-conns but not sure if needed, so I'll leave the final choice to you :)
[12:01:42] <elukey>	 don
[12:01:48] <elukey>	 don't want to block moritzm
[12:01:51] <elukey>	 argh
[12:01:53] <elukey>	 mobrovac:
[12:02:01] <elukey>	 hello Moritz! Wrong ping sorry :)
[12:10:08] <ottomata>	 elukey:  responded inline thanks
[12:10:17] <ottomata>	 joal:  should we do the same for titles on page_move?
[12:13:23] <wikibugs>	 Analytics: Update mediwiki hooks to generate data for new event-bus schemas - https://phabricator.wikimedia.org/T137287#2364032 (Ottomata)
[12:13:25] <wikibugs>	 Analytics-Kanban, EventBus, Patch-For-Review: Propose evolution of Mediawiki EventBus schemas to match needed data for Analytics need - https://phabricator.wikimedia.org/T134502#2364031 (Ottomata)
[12:16:01] * elukey lunch!
[12:24:13] <ottomata>	 joal:  when you come back, another schema q for you about redirect_page_id in page_move
[12:45:14] <wikibugs>	 Analytics: 20160431 produces "end timestamp is invalid, must be a valid date in YYYYMMDD format" - https://phabricator.wikimedia.org/T135812#2364182 (Nemo_bis) >>! In T135812#2318963, @Milimetric wrote: > @Nemo_bis: the query asks for 20160401 through 20160431, and 20160431 is not a valid date, in case that...
[12:46:42] <wikibugs>	 Analytics: 20160431 produces "end timestamp is invalid, must be a valid date in YYYYMMDD format" - https://phabricator.wikimedia.org/T135812#2364185 (Nemo_bis)
[12:50:20] <wikibugs>	 Analytics, Analytics-Cluster, Operations, Services: Better monitoring for Zookeeper - https://phabricator.wikimedia.org/T137302#2364233 (Ottomata)
[12:51:46] <wikibugs>	 Analytics, Analytics-Cluster, EventBus, Operations, Services: Better monitoring for Zookeeper - https://phabricator.wikimedia.org/T137302#2364255 (mobrovac)
[12:58:42] <wikibugs>	 Analytics, Analytics-Cluster, EventBus, Operations, Services: Better monitoring for Zookeeper - https://phabricator.wikimedia.org/T137302#2364285 (Ottomata) The default client connection in our puppet module is no limit.  Once we have alerts we should set a limit, pretty high, maybe 2048.
[13:02:25] <elukey>	 ottomata: you there?
[13:03:27] <ottomata>	 ja
[13:03:33] <ottomata>	 elukey:  fyi i am restarting zks
[13:04:01] <elukey>	 sure :)
[13:04:14] <elukey>	 I was looking for zk metrics aaand... nothing on graphite
[13:04:35] <elukey>	 the jmxtrans module seems in need of love
[13:04:40] <ottomata>	 ayyyye
[13:07:45] <urandom>	 elukey: hi! :)
[13:07:52] <elukey>	 o/
[13:08:18] <urandom>	 i dunno if you have yet seen my latest on https://gerrit.wikimedia.org/r/#/c/290860/2
[13:08:28] <urandom>	 oh
[13:08:30] <urandom>	 you have!
[13:08:45] <urandom>	 elukey: could you merge that for me?
[13:09:17] <urandom>	 as it is, i'm going to need to open an issue for godog to cleanup the metrics introduced when i upgraded the code on the 2.2.6 nodes
[13:09:35] <elukey>	 will do it in ~1hr, is it ok?
[13:09:42] <elukey>	 I mean, in the next hour :)
[13:09:44] <urandom>	 i was supposed to be filtering them out with this gerrit before landing that code, and then all hell broke loose last week and i landed that code :)
[13:09:47] <urandom>	 sure!
[13:09:49] <elukey>	 super!
[13:10:17] <elukey>	 if you have time I'd also like to get your opinion of https://gerrit.wikimedia.org/r/#/c/292568/
[13:10:46] <urandom>	 sure, let me have a look
[14:09:26] <wikibugs>	 Analytics: 20160431 produces "end timestamp is invalid, must be a valid date in YYYYMMDD format" - https://phabricator.wikimedia.org/T135812#2364448 (Danny_B) @Nemo_bis And what would you suggest instead of invalid report?
[14:21:52] <wikibugs>	 Quarry: Add a stop button to halt the query - https://phabricator.wikimedia.org/T71037#2364499 (matej_suchanek)
[14:22:06] <wikibugs>	 Quarry: Option to kill task by myself - https://phabricator.wikimedia.org/T137266#2364500 (matej_suchanek)
[14:22:10] <wikibugs>	 Quarry: Add a stop button to halt the query - https://phabricator.wikimedia.org/T71037#721028 (matej_suchanek)
[14:52:54] <wikibugs>	 Analytics: 20160431 produces "end timestamp is invalid, must be a valid date in YYYYMMDD format" - https://phabricator.wikimedia.org/T135812#2364607 (Nemo_bis) >>! In T135812#2364448, @Danny_B wrote: > @Nemo_bis And what would you suggest instead of invalid report?  One option is to simply perform a numerica...
[15:28:33] <joal>	 ottomata: Heya
[15:29:03] <joal>	 ottomata: From your messages, I understand there is inconsistency in names for page related fields
[15:29:28] <ottomata>	 joal:  seems so, but it  might have been intentional
[15:29:30] <ottomata>	 oh let me save my comments
[15:29:50] <joal>	 ottomata: I tried to be carefull on that aspect, so I'm a bit angry at me, but please let's discuss that, it's an important benefit of new schemas in my opinion
[15:30:10] <ottomata>	 it seems in page contexts, the page_ prefix is dropped
[15:30:21] <ottomata>	 for title
[15:30:32] <ottomata>	 but on revision, it is page_title
[15:30:34] <ottomata>	 maybe that was on purpose
[15:32:29] <joal>	 ottomata: We didn't discuss that, but in my mind page related event didn't need the page_ prefix - But it actually is not true in the case of revisisions which have the prefix ... So let's be verbose and add the page prefix ?
[15:32:54] <ottomata>	 +1
[15:40:54] <joal>	 ok ottomata, I'm gonna get back to modifying my schema :)
[15:44:27] <wikibugs>	 Analytics, Developer-Relations, MediaWiki-API, Reading-Admin, and 4 others: Metrics about the use of the Wikimedia web APIs - https://phabricator.wikimedia.org/T102079#2364746 (bd808)
[15:49:38] <elukey>	 ottomata: I was thinking to try
[15:50:15] <elukey>	 kafka configs --alter --entity-type topics --entity-name webrequest_text --add-config retention.bytes=10000000000000
[15:50:16] <ottomata>	 elukey:  you've installed your new vk on all the varnish4 hosts?
[15:50:21] <elukey>	 yeppa
[15:50:33] <ottomata>	 elukey:  +1 for trying that :)
[15:51:58] <elukey>	 joal: yt?
[15:52:03] <joal>	 I am elukey
[15:52:18] <elukey>	 since you are my official reviewer, would you mind to check the above command?
[15:52:24] <joal>	 :)
[15:54:58] <joal>	 elukey: works for me if we are considering terabytes and not TiB :)
[15:56:54] <wikibugs>	 Analytics, MediaWiki-API, User-bd808: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321#2364785 (bd808)
[15:57:25] <wikibugs>	 Analytics, MediaWiki-API, User-bd808: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321#2364802 (bd808)
[15:57:30] <wikibugs>	 Analytics, Developer-Relations, MediaWiki-API, Reading-Admin, and 4 others: Metrics about the use of the Wikimedia web APIs - https://phabricator.wikimedia.org/T102079#2364801 (bd808)
[15:58:49] <elukey>	 joal: yes :)
[15:59:51] <joal>	 elukey: As an (kind of) old computer scientist, I'm not used to see round numbers when discussing data volumes :)
[16:02:30] <milimetric>	 ottomata: standup
[16:02:38] <ottomata>	 oh thanks
[16:03:09] <elukey>	 !log temporary set a 10TB upperbound to the Kafka webrequest_text topic to free space
[16:03:36] <ottomata>	 trying to join...
[16:10:54] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Update mediwiki hooks to generate data for new event-bus schemas - https://phabricator.wikimedia.org/T137287#2364848 (Ottomata)
[16:25:47] <ottomata>	 joal:  analytisc1001 back as rm active now
[16:25:55] <joal>	 thanks ottomata
[17:06:25] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2363239 (Ottomata) Also:  ``` Jun  8 16:58:34 analytics1049 kernel: [7283582.453037] sd 0:2:2:0: [sdc] Jun  8 16:58:34 analytics1049 kernel: [7283582.453043] Result: hostbyte=D...
[17:09:07] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2365080 (Ottomata) Some output from  ``` sudo megacli -AdpEventLog -GetEvents -f events.log -aALL ```   ``` ...  seqNum: 0x00005441 Time: Wed Jun  8 06:55:17 2016  Code: 0x0000...
[17:11:02] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2363239 (Cmjohnson) Enclosure Device ID: 32 Slot Number: 1 Drive's position: DiskGroup: 2, Span: 0, Arm: 0 Enclosure position: 1 Device Id: 1 WWN: 500003964ba801e7 Sequence Num...
[17:41:41] <wikibugs>	 Analytics, MediaWiki-API, User-bd808: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321#2365196 (bd808) The scripts I have been using are available at https://github.com/bd808/action-api-analytics
[17:48:47] <mforns>	 a-team, logging off, see you tomorrow!
[17:48:53] <mforns>	 :]
[17:49:14] <ottomata>	 laters!
[18:13:23] <elukey>	 !log removed retention.bytes override configuration for kafka webrequest_text (didn't work)
[18:13:32] <ottomata>	 wah wahhh
[18:13:34] <ottomata>	 dunno why that doesn't work
[18:13:44] <elukey>	 :(
[18:20:38] <elukey>	 ottomata: anything that I can do before going offline?
[18:22:41] <ottomata>	 naw its cool elukey i got it
[18:22:44] <ottomata>	 have good eve!
[18:22:45] <ottomata>	 ttyt!
[18:23:18] <elukey>	 you too! byeee! o/
[18:24:16] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2365341 (Cmjohnson) Replaced the disk.  The preserved cache was not able to be cleared using megcli commands.  Had to reboot the server and dicard using the raid bios on-site....
[18:25:59] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2365359 (Cmjohnson) Leave the ticket open until I get the warranty part from Dell.
[21:44:27] <wikibugs>	 Analytics: 20160431 produces "end timestamp is invalid, must be a valid date in YYYYMMDD format" - https://phabricator.wikimedia.org/T135812#2366172 (Milimetric) Oh I see, if the request is to not validate dates, then I would personally decline that.  We validate parameters to make sure the user is asking fo...
[21:52:06] <madhuvishy>	 milimetric: may be there's a bunch of stuff happening in the driver?
[21:52:23] <milimetric>	 driver?
[21:52:24] <milimetric>	 what driver?
[21:52:26] <madhuvishy>	 you can bump driver memory with --driver-memory
[21:52:39] <madhuvishy>	 so there's a driver, and many executors
[21:52:51] <milimetric>	 oh... what's the default driver memory?
[21:53:28] <madhuvishy>	 all distributed operations will happen in the executors, and the driver manages these, and also if you did like a grouping or counting - i believe it will have to be processed on the driver
[21:53:36] <madhuvishy>	 not sure may be 512mb or 1g
[21:54:11] <milimetric>	 hmmm
[21:54:19] <milimetric>	 ok, I'll try, thanks
[21:54:53] <milimetric>	 is 2G too much to ask?
[21:55:23] <madhuvishy>	 https://spark.apache.org/docs/1.5.2/configuration.html says 1G
[21:55:48] <madhuvishy>	 no that sounds fine
[21:55:55] <milimetric>	 aha, ok, thx
[22:13:44] <milimetric>	 indeed it's not dying yet, but it's taking for-ever!! weird
[22:14:52] <madhuvishy>	 aah
[22:18:42] <milimetric>	 ah ok, it died.  Something must be wrong with the logic, back to the drawing board
[22:19:28] <madhuvishy>	 died with OOM?