[01:32:05] lzia: I spelled your name badly in an email :/ Sorry. [01:32:36] bd808: I'm responding to it and I didn't even notice it. /me is curious now. ;) [01:33:14] Its in my second response [01:33:29] quiddity trouted me for it :) [01:38:28] hahaha! bd808, just saw that. [01:39:04] don't worry, it has cost you only $1 so far. ;) [01:39:08] * bd808 bows low and offers a drink of Leila's choice on next meeting [01:39:21] and obviously I hadn't read your second response when I wrote mine, so feel free to use as much as it's relevant bd808. [01:40:13] It seems easy to sign up and I doubt there is much else beyond sending a couple of emails based on what I've seen in the docs [01:40:29] * lzia claps for a drink with bd808. :) [01:40:36] ottomata: is that a no? :D [01:40:48] ok, bd808. :) [01:41:17] I'll let it sit for a while and see if anyone else has an opinion [01:41:29] * bd808 hears the dinner bell and flees [01:42:50] lzia: N 2 IT [01:42:52] that's a yes duh [01:45:18] I need serious education, ottomata. I read that as No to it. :d [01:45:25] read it out loud [01:45:46] do you see an 'o'?! NO [01:45:52] it doesn't work with the Iranian accent, I guess, as it should. :D [01:46:08] its kinda like https://www.youtube.com/watch?v=Eo-KmOd3i7s [01:46:27] :D [01:47:08] thanks for the education, ottomata. :) [01:47:13] haha [02:07:45] 10Analytics, 06Developer-Relations, 10MediaWiki-API, 06Reading-Admin, and 3 others: Is User-Agent data PII when associated with Action API requests? - https://phabricator.wikimedia.org/T154912#3116778 (10Nuria) >so none of these questions can be answered without user agents. Wait, wait.., you need aggregga... [02:15:46] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3116785 (10Nuria) Ping @jcrespo @Jdforrester-WMF will be adding these schemas to blacklist , so it... [02:17:49] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3116787 (10Nuria) cc @Marostegui [02:49:22] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3116826 (10Jdforrester-WMF) Sounds good. Thank you! [04:07:43] 10Analytics-EventLogging, 06Analytics-Kanban, 06Performance-Team, 07Performance, 07Regression: EventLogging schema modules take >1s to build (max: 22s) - https://phabricator.wikimedia.org/T150269#3116870 (10Krinkle) Looking back at this anomaly a few months later: (03PS2) 10Joal: Bump jar version in webrequest load job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/343676 [05:54:09] (03CR) 10Joal: [V: 032 C: 032] "Self merging for deploy." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/343676 (owner: 10Joal) [05:54:59] !log Deploying refinery [05:55:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [05:57:12] !log Restart cassandra-hourly-wf-local_group_default_T_pageviews_per_project-2017-3-20-23 [05:57:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:51:05] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3116946 (10Marostegui) Sounds good! Thank you @Nuria once it is merged I will drop it again and se... [07:33:17] 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3117008 (10Marostegui) >>! In T160454#3114957, @Nuria wrote: > @Marostegui I think @otto can run script in our end, let us know if that is OK with you and we will take a s... [07:52:47] hi team :] [09:02:37] 10Analytics, 10Datasets-General-or-Unknown, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#3117089 (10ArielGlenn) [09:16:26] Good morning a-team! [09:16:30] Hi fdans [09:16:41] I leave for a day and people start having children! [09:16:55] Hi mforns! :D [09:17:07] :) [09:17:16] Sorry, hi joal! [09:17:34] (why did I read you as marcel?) [09:34:57] 10Analytics, 10Analytics-General-or-Unknown, 06WMDE-Analytics-Engineering, 10Wikidata, 15User-Addshore: Grafana: "wikidata-api" doesn't update anymore - https://phabricator.wikimedia.org/T160825#3117198 (10Addshore) p:05Triage>03Normal [09:40:37] 06Analytics-Kanban: Synchronise changes for productionisation of mediawiki history jobs - https://phabricator.wikimedia.org/T160154#3117219 (10JAllemandou) [09:43:03] mforns: quick question [09:44:06] mforns: https://gerrit.wikimedia.org/r/#/c/338786/ has been deployed - Is there anything to be done (like creating tables in prod or something?) [09:46:01] hi joal :] thanks for the deployment, but no, the tables are already there [09:46:13] hi fdans :] [09:47:47] awesome mforns - Shall I move the task to done? [09:52:54] !log Restart webrequest-load bundle to pick up new pageview definition (2017-03-21T09:00Z) [09:52:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:55:13] !log Reset hdfs folders and hive tables and partitions for productionisation of mediawiki history [09:55:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:05:10] sorry joal, missed your ping, I moved the task to done, thanks!! [10:05:18] np mforns :) [10:05:32] :] [10:33:37] joal, what will happen if I launch the cassandra load job for all time, reagarding the existing data? will it be overriden? [10:33:44] or will it exist in parallel? [10:34:12] I guess the primary key is the same right? so it should overwrite [10:34:21] mforns: As long as the primary key is the sae, it'll be overwritten [10:34:42] mforns: this precise eample is the reason for which _tid can cause trouble if not set correctly :) [10:34:51] joal, the primary key is defined as: PRIMARY KEY (("_domain", project, "access-site", granularity), timestamp, "_tid") [10:35:00] I know [10:35:05] all thos fields will be the same, except for _tid [10:35:17] mforns: why would it be different? [10:35:31] I thought it would be the insertion time... [10:36:18] in the oozie job we specify it as 0, but in the table it has an actual value: 13814000-1dd2-11b2-8080-808080808080 [10:37:37] oh, interesting, joal: The UUID 13814000-1dd2-11b2-8080-808080808080 contains a timestamp taken at [10:37:38] Thursday, January 1, 1970 12:00:00 AM GMT [10:38:12] mforns: You have it :) [10:38:17] so yes, it seems the 0 value we pass is translated into 1970-01-01, and thus will be the same [10:38:20] cool, thx [10:38:27] 10Analytics, 10Analytics-Dashiki, 13Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3117414 (10matthiasmullie) Not too urgent, but preferably before July, when most of Multimedia's focus will be shifting towards stuctured data. I guess this falls under reading... [10:38:39] mforns: The inserted value for _tid is the minimal TimestampUID generated by Java for timestamp 0 [10:39:02] mforns: Therefore, overwriting is feasible [10:39:13] aha [10:39:14] mforns: However, something to keep in mind when ovewriting [10:39:19] yes [10:39:42] mforns: Cassandra doesn't delete per say - it flags with a deleteion flag (different from the one restbase creates for us) [10:39:55] And the row really get's deleted only at recompaction time [10:40:03] aha [10:40:18] Knowing you're data will not change after insertion (no addition) [10:40:52] it means, to recover deletion space, that we need to do a clean up of the keyspace <-- elukey [10:41:08] ok [10:41:24] I'll ping him when everything is vetted [10:41:29] So mforns, if it's easy not to load everything, then do that, if not, we'll ask elukey to do a clean up after insertion [10:41:41] makes sense? [10:42:16] joal, I already vetted the 1 year I loaded and looks good, do you think this is enough of a test, and we can skip full load? [10:42:38] If so, I'm all for it [10:42:40] mforns: in testing mode you mean? [10:42:45] yes [10:42:59] mforns: if you've tested with a year and it looks good, just go for full in prod IMO [10:43:39] joal, you mean into the final real keyspace? [10:43:44] correct [10:43:53] I mean, it's been tested, no? [10:43:59] joal, sure [10:44:09] but don't we need to deploy the endpoint first? [10:44:18] mforns: Ahhhh, yes, you do :) [10:44:27] Sorry, thought that was done [10:44:47] OK, joal, I have to leave now, but will continue with the deploy later :] [10:45:05] sure mforns - I'll be away this afternoon, we'll catch up tomorrow morning :) [10:45:17] ok, have a nice class, see you! [11:03:02] Gone for now a-team - Refinery has been deployed succesfully (from what I can see) and mediawiki history code is ready and hive is setup [12:09:30] 10Analytics-Tech-community-metrics: Demographics: Show attracted vs. retained contributors in Git (not Gerrit) in Kibana - https://phabricator.wikimedia.org/T151501#3117590 (10Aklapper) [12:11:20] 10Analytics-Tech-community-metrics: Demographics: Show attracted vs. retained contributors in Git (not Gerrit) in Kibana - https://phabricator.wikimedia.org/T151501#2819403 (10Aklapper) 05Open>03Resolved a:03Aklapper I am going to close this as per last comment as the panel for **Git** exists (though it ha... [12:15:11] 10Analytics-Tech-community-metrics: Have "Last Attracted Developers" information for Gerrit (already exists for Git) - https://phabricator.wikimedia.org/T151161#3117596 (10Aklapper) [12:17:23] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Have "Last Attracted Developers" information for Gerrit (already exists for Git) - https://phabricator.wikimedia.org/T151161#3117599 (10Aklapper) >>! In T151161#3115662, @Dicortazar wrote: > # Add other data sources as Gerrit. You may di... [12:53:20] 10Analytics-Tech-community-metrics: "Patchsets Statistics Per Review" widget on "Gerrit" is incomprehensible - https://phabricator.wikimedia.org/T151218#3117668 (10Aklapper) Looks like this should be trivial to fix by removing the `CustomLabel` "patchsets" on https://wikimedia.biterg.io/edit/app/kibana#/visualiz... [13:37:15] 10Analytics-Tech-community-metrics: "Patchsets Statistics Per Review" widget on "Gerrit" is incomprehensible (due to missing units and misleading custom label) - https://phabricator.wikimedia.org/T151218#3117743 (10Aklapper) [13:59:26] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3117788 (10Ottomata) [14:11:45] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3117855 (10Ottomata) [14:15:13] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3117868 (10Ottomata) [14:15:29] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#2989939 (10Ottomata) @halfak, does Huggle use irc.wikimedia.org, not RCStream? [14:21:30] 10Analytics-Tech-community-metrics: Updated data in mediawiki-identities DB not deployed onto wikimedia.biterg.io? - https://phabricator.wikimedia.org/T157898#3117882 (10Lcanasdiaz) The process should be restored by tomorrow morning. I'll keep you updated. [14:28:58] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3117906 (10Ottomata) [14:30:56] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#2989939 (10Ottomata) [14:47:09] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3117951 (10Halfak) @Petrb ^ [14:51:50] a-team: how come our workflows don't display properly in Hue anymore? https://hue.wikimedia.org/oozie/list_oozie_workflow/0028827-170228165458841-oozie-oozi-W/ [14:52:33] or I guess just some of them: https://hue.wikimedia.org/oozie/list_oozie_workflow/0028836-170228165458841-oozie-oozi-W/?coordinator_job_id=0001575-170228165458841-oozie-oozi-C [14:55:39] milimetric, +! [14:55:43] +1! [15:00:07] (03PS1) 10Milimetric: Add test2.wikipedia to the whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/343881 [15:00:53] a-team: standdupppp [15:01:32] elukey: standdup [15:20:27] (03CR) 10Milimetric: [V: 032 C: 032] "self-merging 'cause it's just a whitelist" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/343881 (owner: 10Milimetric) [15:25:08] 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3118049 (10Nuria) Ok, will be sending e-mail today with the change we are going and we can plan on doing it Thursday (48 hrs notice) [15:25:33] whitelist how-to in the oncall: https://wikitech.wikimedia.org/wiki/Analytics/Oncall#Find_and_fix_pageview_whitelist_exceptions [15:28:01] milimetric: thanks! [15:28:12] ottomata: updated puppet change for blacklist [15:28:39] great [15:33:14] !log restarting eventlogging client side processors with ImageMetrics blacklist change [15:33:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:37:17] ottomata: we also need to merge this change before taking the outage: https://gerrit.wikimedia.org/r/#/c/343122/ [15:38:22] oh ya nuria [15:38:24] merging now [15:38:42] ottomata: merge and restart eventlogging [15:38:52] ya [15:39:05] let's deploy on thurs as part of the downtime, since we have to stop El then anyway [15:39:23] nuria: think we can do that + the UA parser change in one? [15:39:31] that way the new tables all have JSON stirngs for ua? [15:39:59] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3118069 (10Nuria) We have merged change and restarted processors, tables can be deleted (cc @Maros... [15:40:48] ottomata: ah, i see, adding that change to the set of changes for UA? [15:41:47] ottomata: certainly, we can have all those together. Let me get a changeset that includes that one. [15:42:16] k [15:42:22] we can deploy that to beta now [15:42:28] and even test a table rename or two [15:42:39] actually we should probably rename all tables in beta too? [15:44:24] ottomata: for testing sure, but UA change was tested on beta throughly, let me get teh new changeset together and we can deploy all to beta [15:44:57] ya, i'm sure it'll work, but it'd be nice to have the tables with the longer varchars on beta [15:46:58] ottomata: much agree [15:47:44] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics, 13Patch-For-Review: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3118102 (10Marostegui) >>! In T141407#3118069, @Nuria wrote: > We have merged change and restarted... [15:55:05] ottomata: ok, patch is now on top of column change: https://gerrit.wikimedia.org/r/#/c/343895/, let me make sure things are working ok test wise [15:55:16] ottomata: have a meeting now but can pick this up later [15:58:50] ottomata: tests work fine [15:59:48] fdans, can I help with the pageviews.js / dashiki stuff? [16:00:40] nuria: ya k, i'll merge this, and deploy El in beta, and then also play with table renames there [16:00:44] mforns: I'm good for now, but we can have a sync up tomorrow morning if you want [16:01:08] fdans, ok, lets do that :] [16:01:34] ottomata: remember db on beta is a bit odd: https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/TestingOnBetaCluster#Database [16:15:48] milimetric: a few months ago (december?) you had pointed me at wmf.mediawiki_namespace_map table for translating between the project names in webrequest table ('en.wikipedia') to the wikiid's reported in eventlogging, and other places ('enwiki'). Seems this morning it was dropped, any replacement? [16:15:56] in hive [16:16:56] dropped?! ah... oh! I think it might've been renamed with the patches and deploys [16:16:57] one sec [16:17:18] well, perhaps not dropped, but no longer existing with previous name :) [16:18:53] mforns: I see domain_abbrev_map is there, but that's not quite what ebernhardson is looking for, what ended up happening to mediawiki_namespace_map again? [16:19:32] milimetric, has it disappeared??? [16:20:10] yep [16:20:31] milimetric, I didn't touch it! but... remember I told you guys that domain_abbrev_map disappeared misteriusly in standup a few days ago? [16:20:41] maybe the same happened to mediawiki_namespace_map! [16:20:42] mforns: yeah... hm [16:20:50] I didn't touch it [16:20:54] k, I'll just create it, no biggie [16:21:00] milimetric, do you want me to recreate it? [16:21:07] no blame, just wondering if you knew, haha [16:21:13] no no... [16:21:19] Nono, I'm oncall, I got it :) [16:21:26] k :] [16:25:32] (03PS1) 10Milimetric: Fix type-o in create table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/343910 [16:27:07] (03CR) 10Nuria: [V: 032 C: 032] Fix type-o in create table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/343910 (owner: 10Milimetric) [16:28:18] ebernhardson: sorry, bit of a non-communication. Table's in wmf_raw, and it has a new partition which we needed. We forgot to tell you, sorry. Here's how you get stuff now: [16:28:26] ebernhardson: select * from wmf_raw.mediawiki_project_namespace_map where snapshot='2017-01_private' limit 10; [16:29:06] (note you have to specify snapshot=... something, and right now the only snapshot is the 2017-01 one) [16:29:14] milimetric: ok i'll update my oozie job, thanks! [16:29:26] ebernhardson: is your oozie stuff in refinery? [16:29:38] milimetric: no, its in wikimedia/discovery/analytics [16:30:06] ok. Maybe if we mess up again we should think about merging or submodules or something, so we can detect when there's a breaking change/rename like this. [16:30:09] its not even really prod yet, is pre-work for our machine learning ranking that collects search queries + clicks to feed into label generation then training [16:39:39] nuria: when you compiled list of tables to rename [16:39:48] did you just choose all existent EL tables? [16:40:01] ottomata: no, the ones written to in teh last day [16:40:06] hm k [16:45:17] ottomata: compiled list from mysql consumer log [16:45:47] k [17:02:19] ottomata: staff? [17:02:51] OH [17:22:47] wikimedia/mediawiki-extensions-EventLogging#640 (wmf/1.29.0-wmf.17 - 838abb7 : Translation updater bot): The build has errored. [17:22:47] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.29.0-wmf.17 [17:22:47] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/213498354 [17:23:22] hello analytics! so I was hoping to create a table on the Analytics mariadb slaves that I can JOIN against production tables [17:23:28] is that doable/acceptable/etc.? [17:23:41] this is for https://phabricator.wikimedia.org/T156318 [17:42:48] musikanimal: production tables like which ones? [17:43:24] musikanimal: from that ticket seems that you could use the new data lake data no? [17:43:46] enwiki.revision, page, etc. I was under the belief I needed a large amount of production data to accurately test performance, but maybe I don't, in which case I can do this on my own machine [17:43:47] musikanimal: rather than joining against the application database, let me see [17:44:13] this in particular is what I need to do https://phabricator.wikimedia.org/T156318#3085518 [17:44:33] musikanimal: wait, take a look at this: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Mediawiki_page_history [17:44:39] perhaps the FLUSH STATUS stats will return similar results even if working off a smaller dataset [17:45:04] musikanimal: slow down for a sec [17:45:09] haha ok [17:45:23] this is to test performance for something that would go in core [17:45:59] musikanimal: if you need edit data you (most likely) do not need to connect to database tables but rather you can do with data lake data , data is teh same, it is just denormalized for analytics and thus queries are a lot faster [17:46:23] musikanimal: testing performance of php code that will get comitted to mediawiki? [17:46:33] musikanimal: did i understood correctly [17:46:54] yeah, I should make it clear this is not for research. I ideally would have real production data (a lot of it) and this new table I'm creating [17:47:10] there'd be no changes to any table except the new one [17:47:28] musikanimal: you cannot test that on anlytics replicas thoughm, they run tons of queries for other folks and thus their performance will be likely erratic [17:47:38] musikanimal: depending on what is running at any one time [17:47:55] musikanimal: makes sense? [17:48:37] musikanimal: you can get data from analytics replicas but not accurate numbers for code execution in production [17:49:18] yeah, there is no good way to do this I suppose [17:49:46] musikanimal: i bet there is but likely not on machines that are used to run long queries [17:49:47] but it may turn out the "handler stats" as they are called, do not require large datasets [17:50:03] these wouldn't be long queries [17:50:41] depending on your definition of long :) [17:50:49] hopefully if all goes as planned, it'd take 15 seconds at the most (per query) [17:50:55] but most will run very fast [17:51:39] musikanimal: everybody else is running long queries, they are analytics replicas [17:51:39] I can also try the beta cluster, since I assume it has some reasonable amount of data [17:51:50] musikanimal: that is why you cannot exytrapolate performance [17:52:10] yeah no worries, I'm going to try doing it locally first [17:52:20] thanks for the info :) [17:52:37] musikanimal: the amount of data in labs, prod and analytics replicas is teh same, teh setup however vastly differs [17:52:41] *the [17:53:03] actually for this I guess I could use the replicas... [17:53:09] normal replicas, I mean [17:53:15] not analytics [17:53:23] musikanimal: if that makes sense. thus it will be hard to extrapolate results from labs for the same reason: many other queries are being run at the same time [17:53:35] musikanimal: performance tests have to be run in isolation [17:53:51] do you know how the handler stats work? (I don't) [17:54:01] musikanimal: of other things going on on the environment [17:54:39] the idea is testing the "execution plan", so maybe it doesn't matter about the amount of data you have or if other queries being ran [17:54:57] very unclear to me, but like I said I could try locally first and see what I get [17:58:29] musikanimal: it matters, you cannot "test" in an environment you do not control what is going on [17:59:02] musikanimal: i would try first locally and second would give a [ping to perf team for advice on how best test your changes. [17:59:45] ok will do :) thanks again! [18:01:52] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3118566 (10leila) @Ottomata do you know if databases in staging in analytics-slave will be copied to some other place if we're decommissioning the suggested machines? I'm aski... [18:01:53] Hey milimetric and ebernhardson - I' sorry for the miscomunication on the ap table - I reset the mediawiki structure (hive + folders) this morning for productionisation and didn't know this was used somewhere [18:02:43] joal: no worries, as i mentioned its not a prod use case yet, it's actually still in a gerrit patch but i setup the oozie runner to make sure it would work over time [18:03:19] musikanimal: ok, please let me know what you end up doing that way we can best help the next person that comes around with the same question [18:03:20] k ebernhardson - Sorry nonetheless :) [18:03:36] ebernhardson: We'll need to find a way to keep you posted on new snapshots for this ... [18:03:44] sure! can do :) [18:04:24] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3118585 (10Ottomata) @leila, we can dump and copy to analytics-store, as long as there aren't any database.table name collisions. [18:21:25] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3118686 (10Ottomata) [18:23:39] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3118687 (10Ottomata) [18:27:42] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3118691 (10Ottomata) [18:32:37] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3118697 (10leila) sounds good, @Ottomata. [18:33:52] Hi! Any help on an apparently missing large file that I'd left in my home directory on stat1002? [18:37:11] AndyRussG: what's upi? [18:45:23] ottomata: there's a file w/ data that I think was in my home directory on stat1002. It was pretty big... Now it's apparently at 0 bytes :( [18:45:41] are there backups or something of home directories there? [18:46:16] It was pretty large so I'm wondering if someone or some automatic process purged or moved it? [18:46:18] no, but i don't know of anyone just deleting data [18:47:25] AndyRussG: I sent an email around about a month ago about home dirs getting big on stat1002, but i didn't delete anything [18:47:28] just asked folks to do so [18:47:32] can't remember if you were on that thread [18:47:41] i only emailed people who had a lot of data in their homes [18:49:18] ottomata: hmmm I don't see any such e-mail, though I could have missed it [18:49:27] well, then i probably didni't email you :) [18:49:29] as you weren't an offender [18:49:58] AndyRussG: which file [18:50:00] it is 0 bytes? [18:50:09] maybe you have something accidentally writing to it, and it didn't append? [18:50:54] Yeah it's 0 bytes... [18:51:43] * AndyRussG check bash history and cries [18:51:54] AndyRussG: which file? [18:51:55] just curious [18:51:57] ottomata: yep [18:52:44] /home/andyrussg/banner_history_nov_2016.out [18:53:20] It looks like that was it. I stupidly overwrote it. [18:53:32] oh ya? oh no? :/ [18:53:52] aaaarg [18:53:54] * AndyRussG hides [18:54:10] yaarr nov 2016 :( [18:54:28] ottomata: oh well... sh** happens, I guess [18:54:44] aaaarg thanks in any case!!!! [18:55:14] sho thing [19:39:29] 10Analytics, 10EventBus, 10Wikimedia-Stream: Puppetize event schema topic configuration - https://phabricator.wikimedia.org/T161027#3119072 (10Ottomata) [19:51:41] 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3119113 (10Tbayer) >>! In T160454#3118049, @Nuria wrote: > Ok, will be sending e-mail today with the change we are going and we can plan on doing it Thursday (48 hrs notic... [19:52:59] 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3119115 (10Nuria) >BTW, I assume we are going to update https://meta.wikimedia.org/wiki/Schema:EventCapsule beforehand too? We will be doing that once this work is finishe... [19:59:58] 10Analytics, 06Research-and-Data: geowiki data for Global Innovation Index - https://phabricator.wikimedia.org/T131889#3119148 (10leila) >>! In T131889#3085445, @Rafaesrey wrote: > 1. When will the data be available? Given the above launch date, time > is now of essence. I want to finish this work in th... [20:02:05] 10Analytics, 06Research-and-Data: geowiki data for Global Innovation Index - https://phabricator.wikimedia.org/T131889#3119153 (10leila) [20:31:08] 10Analytics, 06Research-and-Data: geowiki data for Global Innovation Index - https://phabricator.wikimedia.org/T131889#3119260 (10Rafaesrey) Dear Leila, Thank you very much! Yes, please use the data as suggested in c) since it is the same that I already have. This way we can produce a more complete set. I... [20:32:38] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3119307 (10Krinkle) [20:47:58] 10Analytics, 10Analytics-Dashiki: Move Dashiki config from CommonSettings to extension - https://phabricator.wikimedia.org/T161038#3119392 (10Reedy) [20:49:49] 10Analytics, 10Analytics-Dashiki: Move Dashiki config from CommonSettings to extension - https://phabricator.wikimedia.org/T161038#3119413 (10Reedy) [20:58:34] 10Analytics, 10Analytics-Dashiki: Move Dashiki config from CommonSettings to extension - https://phabricator.wikimedia.org/T161038#3119381 (10Milimetric) Thanks @Reedy, I'll make a patch right now. [21:00:48] 10Analytics, 10Analytics-Dashiki: Move Dashiki config from CommonSettings to extension - https://phabricator.wikimedia.org/T161038#3119437 (10Milimetric) Reedy, I was attempting to fix some of that with https://gerrit.wikimedia.org/r/#/c/336446/ but I didn't know I could move this stuff to extension.json. Is... [21:05:14] 10Analytics, 10Analytics-Dashiki: Move Dashiki config from CommonSettings to extension - https://phabricator.wikimedia.org/T161038#3119457 (10Reedy) >>! In T161038#3119437, @Milimetric wrote: > Reedy, I was attempting to fix some of that with https://gerrit.wikimedia.org/r/#/c/336446/ but I didn't know I could... [21:07:41] 10Analytics, 10Analytics-Dashiki: Move Dashiki config from CommonSettings to extension - https://phabricator.wikimedia.org/T161038#3119469 (10Milimetric) ok, cool, I'll tag this task with those patches, I'd appreciate merges 'cause none of us on our team have +2. [21:11:28] 06Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3119516 (10Krinkle) [21:20:14] 10Analytics, 06Developer-Relations, 10MediaWiki-API, 06Reading-Admin, and 3 others: Is User-Agent data PII when associated with Action API requests? - https://phabricator.wikimedia.org/T154912#3119641 (10Tgr) That would work in an ideal world where everyone uses a unique UA. In the real world, we have thre... [21:47:28] 10Analytics-Dashiki, 06Analytics-Kanban, 13Patch-For-Review: Move Dashiki config from CommonSettings to extension - https://phabricator.wikimedia.org/T161038#3119782 (10Milimetric) a:03Milimetric [22:51:23] (03PS1) 10Nuria: Adding renamed tables to sql union statements [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/344049 (https://phabricator.wikimedia.org/T160454) [22:53:30] (03CR) 10Bearloga: [C: 031] Adding renamed tables to sql union statements [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/344049 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:17:03] (03PS1) 10Nuria: Adding renamed tables to sql union statements [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) [23:19:29] (03CR) 10Nuria: "Is this repo and code still used? Ping @jdforrester" [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:19:49] (03CR) 10Nuria: "sorry ping @Jforrester" [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:26:16] (03PS1) 10Nuria: Adding renamed tables to sql union statements [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/344055 (https://phabricator.wikimedia.org/T160454) [23:26:44] (03CR) 10Nuria: "Is this data used at all?" [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:27:15] (03CR) 10Nuria: "Is this data used at all?" [analytics/limn-flow-data] - 10https://gerrit.wikimedia.org/r/344055 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:43:18] (03CR) 10Jforrester: "Which "this"?" [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:45:38] (03CR) 10Nuria: "Sorry @Jforrester, the code that updates this dashboard: https://edit-analysis.wmflabs.org/compare/" [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:47:54] (03CR) 10Jforrester: "Yes, I personally use it a few times a week and many other community people have looked at it from time to time. It's a dashboard, after a" [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria) [23:51:45] (03CR) 10Nuria: "We track usage and this receives 1 visit a day, pretty much. No harm in letting be if it is of use to you though." [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/344054 (https://phabricator.wikimedia.org/T160454) (owner: 10Nuria)