[01:30:35] <timberlands>	 so, i put up a video by mistake within the last two hours. I just took it down. Can i look use google analytics to see how many people saw it?
[01:54:23] <milimetric>	 hm, wrong analytics :)
[08:13:23] <wikibugs>	 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3414818 (10Marostegui) >>! In T153033#3413793, @Nuria wrote: >> I can write to the typical research lists to ask this question if you like, so this task doesn't stall. But if this situation comes up again we shou...
[10:25:55] <wikibugs>	 10Analytics, 10Patch-For-Review: Eventstreams graphite disk usage - https://phabricator.wikimedia.org/T160644#3415112 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi Resolving this for now, will reopen if necessary
[11:37:21] * elukey lunch!
[12:49:56] <elukey>	 anybody knows what _Edit_11448630_old is for on dbstore1002?
[12:50:11] <elukey>	 it contains ~170GB of data, not on db1047
[12:50:23] <elukey>	 since we are short on disk space on dbstore it might be good if we could drop it
[12:51:15] <elukey>	 and it is not even on db1046
[13:09:29] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3415353 (10elukey) Alters just finished, ran the following two queries from stat1002:  ``` elukey@stat1002:~$ mysql -h analytics-slave.eqiad.wmnet <<< "SELECT d...
[13:09:38] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3415354 (10elukey)
[13:41:46] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3415427 (10elukey) Re-checked for both -slave and -store all the not nullable fields, they are whitelisted.
[13:43:05] <elukey>	 altersss doneeeeee \o/
[13:45:40] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3415444 (10elukey) a:03elukey
[13:47:15] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3360724 (10elukey) There is one table on dbstore1002 that holds ~370M rows (~170GB in size): `_Edit_11448630_old`. This table is not present on db104[67], so it might be a good candidate to drop. Sent an...
[13:55:57] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3415491 (10mforns) Awesooome!
[13:59:36] <milimetric>	 elukey: that sounds like very early days of the Edit schema, some old data that probably didn't get analyzed.  I would check with Neil
[13:59:43] <milimetric>	 neilpquinn: ^
[13:59:56] <milimetric>	 pasting from above: anybody knows what _Edit_11448630_old is for on dbstore1002
[14:00:20] <elukey>	 milimetric: o/
[14:00:28] <milimetric>	 in general though, just because something's not on 1047 it doesn't mean it's droppable, because 1002 is where analytics happens people often have no idea where else that data is
[14:00:48] <milimetric>	 wait, just to make sure, 1002 is what analytics-store points to, right?
[14:00:51] <elukey>	 well it is not on db1046 too
[14:00:55] <elukey>	 yep yep
[14:01:11] <milimetric>	 yeah, then 47/46 don't matter from most people's point of view
[14:01:17] <elukey>	 in any case, I asked because it seems a good candidate to re-gain some space
[14:01:25] <milimetric>	 same with those moodbar tables yesterday
[14:01:32] <milimetric>	 yeah, definitely, that's a lot of space
[14:01:49] <milimetric>	 what about the staging db, is that huge?  That's where people usually forget temp tables
[14:02:04] <milimetric>	 I didn't know we were running out of space, that's bad
[14:02:07] <elukey>	 I didn't check the other dbs, but I can take a look now
[14:03:04] <milimetric>	 staging is the only other one we have some control over
[14:03:16] <milimetric>	 (be careful there, research user has delete permissions :)
[14:03:31] <elukey>	 biggest table is ~89GB, nice to drop :)
[14:03:50] <elukey>	 what is the policy to delete in there?
[14:04:47] <milimetric>	 users are supposed to prefix their tables with <<user>>_ I think
[14:04:59] <milimetric>	 so checking with the users the tables belong to should be easy
[14:05:03] <elukey>	 half of them are not sadly
[14:05:13] <milimetric>	 I'll check it out
[14:05:29] <milimetric>	 how do you check the size?
[14:05:31] <elukey>	 elukey@dbstore1002:~$ sudo mysql -h localhost --skip-ssl < check_size.sql
[14:05:34] <elukey>	 Table	Size in MB
[14:05:36] <elukey>	 mep_word_persistence	89879.65
[14:05:39] <elukey>	 enwiki_monthly_aq	32217.44
[14:05:39] <milimetric>	 :) sudo
[14:05:41] <elukey>	 enwiki_pagelinks_ns0	26128.49
[14:05:44] <elukey>	 th_subst_template_additions	15674.41
[14:05:46] <elukey>	 enwiki_reverted_20140820	12697.46
[14:05:49] <elukey>	 horrible paste
[14:06:02] <milimetric>	 all good - that's why we all use irccloud
[14:06:15] <milimetric>	 those all look like halfak's tables
[14:06:28] <milimetric>	 maybe not
[14:06:38] <milimetric>	 he usually prefixes, but these are things he was looking into
[14:06:57] <milimetric>	 fdans: how goes?
[14:07:54] <fdans>	 milimetric: hey-all right, fixing prod build
[14:08:14] <milimetric>	 cool cool, here if you need me
[14:09:02] <milimetric>	 I got lucky yesterday with the sqoop thing, so it's fixed, but I'd love to brainbounce with mforns when he's around
[14:10:37] <elukey>	 what was the issue?
[14:16:00] <milimetric>	 well, the issue was clear - it was hitting the net_write_timeout limit, but I got lucky that just parallelizing the import by the primary key of each table worked
[14:16:26] <milimetric>	 slight snag that the tables didn't have primary keys, so the first queries might take a bit longer, but I think overall it's faster
[14:17:17] <milimetric>	 (it still works without a primary key, all it needs is a relatively even spread between max(something) and min(something)
[14:18:21] <wikibugs>	 10Analytics-Kanban, 10DBA, 10Operations, 10Patch-For-Review, 10User-Elukey: Puppetize Piwik's Database and set up periodical backups - https://phabricator.wikimedia.org/T164073#3415609 (10elukey) A lot of things changed from my last post, most of them due to the fact that now the apps are not sending any...
[14:22:18] <elukey>	 milimetric: nice!
[14:22:38] <elukey>	 sqoop is one of those things that I should investigate more, still a dark corner of analytics for me :)
[14:23:11] <milimetric>	 oh it's relatively easy, if you want we can brain bounce now because I need someone to think through next steps and I don't see marcel around
[14:23:21] <milimetric>	 oh there he is, mforns! wanna sqoop talk?
[14:23:27] <mforns>	 milimetric, I'm here
[14:23:30] <mforns>	 yes!
[14:23:33] <mforns>	 cave?
[14:23:43] <milimetric>	 yes
[14:23:45] <mforns>	 omw
[14:23:48] <milimetric>	 elukey: you're welcome to join
[14:24:00] <milimetric>	 I don't think Marcel has seen this so I can go over exactly how it works, it's fairly simple
[14:24:26] <elukey>	 milimetric: can you give me 2 mins for a coffee? 
[14:24:37] <milimetric>	 sure
[14:24:46] <elukey>	 Friday afternoon is kicking in for me and my brain is not working properly :)
[15:02:03] <wikibugs>	 10Analytics-Tech-community-metrics: Missing time units for percentile values - https://phabricator.wikimedia.org/T145425#3415725 (10Dicortazar) This is fixed in the current version in production. The fixed is based on the title of the widget where the keyword 'days' is specified.  @Lcanasdiaz your turn to update...
[15:16:29] <wikibugs>	 10Quarry, 10Cloud-Services: Consider moving Quarry to be an installation of Redash - https://phabricator.wikimedia.org/T169452#3415749 (10Halfak) My main concern with this kind of move would be preserving the basic functionality of Quarry in redash.  E.g. permalinks to results, recent queries, user queries, pu...
[15:51:07] * elukey afk! byyyyeee
[16:15:59] <wikibugs>	 (03PS2) 10Milimetric: Implement sqooping with mappers > 1 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/363735 (https://phabricator.wikimedia.org/T169782)
[16:18:51] <milimetric>	 ok, mforns, I added you to the two reviews on puppet and the sqoop script
[16:19:02] <mforns>	 milimetric, thanks :]
[16:19:40] <milimetric>	 I'll go grab lunch, I appreciate a review but maybe it's no rush because we don't have Luca to merge and I'm not sure if I should/can run the job with sudo -u hdfs (it was doing some weird python pathing thing)
[17:35:23] <wikibugs>	 (03CR) 10Mforns: [C: 032] "LGTM!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/363735 (https://phabricator.wikimedia.org/T169782) (owner: 10Milimetric)
[17:39:16] <milimetric>	 aaah, good catch mforns, I would've totally forgotten
[17:39:53] <wikibugs>	 (03PS3) 10Milimetric: Implement sqooping with mappers > 1 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/363735 (https://phabricator.wikimedia.org/T169782)
[17:40:26] <milimetric>	 ok, fixed and now the question is, should we merge and try to run the job over the weekend manually?  Or should we wait for Monday?
[17:40:31] <milimetric>	 nuria_: thoughts on timing ^ ?
[17:47:04] <wikibugs>	 (03CR) 10Mforns: [C: 032] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/363735 (https://phabricator.wikimedia.org/T169782) (owner: 10Milimetric)
[17:49:55] <nuria_>	 milimetric: let 's wait for monday, ibetter oversight and two day difference is really not going to matter that much
[17:51:04] <nuria_>	 milimetric: unless there is something i do not not know here...
[17:52:05] <milimetric>	 no, I mean, that's fine
[17:52:16] <milimetric>	 it's just holding up the edit count field and the new snapshot
[17:52:21] <milimetric>	 so it'll be a few days late
[17:52:46] <milimetric>	 this means that the data will be a little weirder this month because we can only limit what we import from the revision table, the others get imported in full
[17:52:50] <milimetric>	 nuria_: ^
[17:53:32] <nuria_>	 milimetric: i see, cause we run imports at the beggining of the month but this month is 1 week late, is that so?
[17:53:37] <wikibugs>	 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Missing time units for percentile values - https://phabricator.wikimedia.org/T145425#3416134 (10Aklapper)
[17:54:20] <nuria_>	 milimetric: even in that case i think is better to document the fact on teh dataset page and run on MOnday
[17:55:54] <milimetric>	 yep, that's the case, ok, we wait for Monday
[18:00:18] <milimetric>	 nuria_: I updated the documentation too
[18:00:39] <nuria_>	 milimetric: thanks for doing the changes
[19:02:42] <milimetric>	 gotta go get vaccinated to protect bebe
[19:02:55] <milimetric>	 I'll be back in an hour or so
[19:03:08] <mforns>	 k, cya!
[21:04:11] <wikibugs>	 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#3416860 (10Nirmos) Hi. I'm working on https://sv.wikipedia.org/wiki/MediaWiki:Gadget-EventStreams.js  I note that parsedcomment is not present. Is that abse...
[21:27:32] <wikibugs>	 10Analytics: Script that synchronizes EL purging white-list with schema talk pages - https://phabricator.wikimedia.org/T170019#3416943 (10mforns)
[22:17:06] <wikibugs>	 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#3417040 (10Nuria) Comment is available, see: https://github.com/wikimedia/mediawiki-event-schemas/blob/master/jsonschema/mediawiki/recentchange/1.yaml#L69...
[22:20:26] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Reading-Web-Backlog, and 2 others: Duplicate events sent in Firefox after back button press - https://phabricator.wikimedia.org/T170018#3417054 (10Jdlrobson)
[22:25:48] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Reading-Web-Backlog, and 3 others: Duplicate events sent in Firefox after back button press - https://phabricator.wikimedia.org/T170018#3417068 (10Jdlrobson) p:05High>03Unbreak! I have isolated the issue to EventLogging. The fix is simple. The c...
[22:40:56] <wikibugs>	 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#3417129 (10Nirmos) Yes, comment is available. parsedcomment is not.
[23:09:37] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Reading-Web-Backlog, and 3 others: Duplicate events sent in Firefox after back button press - https://phabricator.wikimedia.org/T170018#3417220 (10pmiazga) @Jdlrobson - wait. Firefox sends only a second `load` event (without reloading document) when...
[23:21:44] <wikibugs>	 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#2867045 (10Tbayer) >>! In T153033#3413282, @demon wrote: > A //month//?! I can't imagine there's //any// useful data to be gathered out of this. MoodBar was a complete and absolute failure.  Looks like @nuria alr...
[23:21:49] <wikibugs>	 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#3417250 (10Nuria) @Nimos: was it available in the old RCFeed? I do not see it on the comments but maybe was there under a different name: https://www.mediaw...
[23:29:39] <wikibugs>	 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3417315 (10demon) I understand that not all things are failures. But whatever, moving on.
[23:34:19] * Pchelolo is about to leave till Monday
[23:40:18] <wikibugs>	 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#3417361 (10Nirmos) > was it available in the old RCFeed? I don't know, but it's available in the recentchanges API.  > Can we ask you as to the usage of you...