[00:20:05] 06Analytics-Kanban: Pagecounts all sites data issues - https://phabricator.wikimedia.org/T162157#3155870 (10Nuria) data from projectcount_raw, note the meta.m and meta.mw sites have spikes but the one present on reportcard lines up with data on meta.m, meta desktop counts, labeled as "meta" are not affected {F7... [02:49:46] RECOVERY - Hadoop DataNode on analytics1054 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [03:05:29] 06Analytics-Kanban, 10Wikimedia-Stream: EventStreams Blog Post - https://phabricator.wikimedia.org/T160080#3156031 (10Krinkle) [05:50:36] good afternoon! [06:23:28] hello fdans of the future [06:26:19] elukey: https://www.youtube.com/watch?v=BPLrNWAsBWU [06:30:08] aahhaha [08:29:19] fdans: oooh, they're huge ahead of us ! [08:29:48] joal: man the toilets are the best in this country [08:30:18] they redefine the whole experience of taking a dump [08:30:37] fdans: it [08:30:44] again .. [08:31:11] it actually means something to care or not about the comfort of those special moments [08:31:40] I'm not convinced about the seat-heating feature though [08:31:57] huhuhu :) [08:32:00] it always feels like someone has just used it [08:32:30] as long as it's not feeling like 'fire in da hole', I'm guessing it's still ok [08:33:43] * fdans writes down the expression "feeling like fire in da hole" [08:34:14] * joal hopes fdans doesn't associate his notes with names [08:34:23] :D [08:35:38] mediawiki history oozie jobs have successfully run [08:35:51] * joal dances around [08:37:44] nice! [08:42:40] reimaging an1057 to debian [09:12:44] 06Analytics-Kanban, 06Operations, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3156645 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1057.eqiad.wmnet'] ``` The log can... [09:36:30] 06Analytics-Kanban, 06Operations, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3156720 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1057.eqiad.wmnet'] ``` and were **ALL** successful. [09:40:08] 1057 almost completed, I'll do 1035 (journalnode) after lunch [09:40:13] and possibly 1028 [09:40:20] to complete the worker nodes [09:40:46] elukey: this makes me joyful :) [09:41:40] * elukey is happy if Jo is happy :) [09:42:32] joal: we'll need to reboot a lot of them to upgrade the kernel, probably later on this week [09:42:46] a bit invasive but we'll get 4.9 everywhere [09:48:36] elukey: no prob [09:56:12] it's just six hadoop nodes IIRC, the rest were re-imaged after the jessie installer was switched to 4.9 or are still on trusty [10:03:20] seems 11 from cumin [10:03:28] but it is not a big deal :) [10:03:44] I usually alert people since there is the risk of jobs failing [10:04:02] and sometimes joal runs things that if interrupted can cause severe pain and swearing [10:04:05] :D [10:09:23] elukey: I didn't know you knew about me swearing ;) [10:11:05] :D [10:19:06] yeah, 11 it is, my notes file was outdated [10:24:32] Hey all, has rstudio ever been requested on the stat* servers before? [10:38:07] (03PS1) 10Fdans: Remove subjective revision fields [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346519 (https://phabricator.wikimedia.org/T157362) [10:38:50] addshore: hey! Not sure, but if it is a UI it will probably not fit in the stats* use case [10:39:21] I'm asking for someone else, I actually have no idea what it is :D [10:44:27] (03PS1) 10Fdans: Removes subjective fields from mediawikihistory query - revision_is_productive - revision_first_identity_revert_timestamp + revision_time_to_identity_revert [analytics/refinery] - 10https://gerrit.wikimedia.org/r/346522 (https://phabricator.wikimedia.org/T157362) [10:45:08] (03PS2) 10Fdans: Removes subjective fields from mediawikihistory query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/346522 (https://phabricator.wikimedia.org/T157362) [10:46:33] joal: I'm out for a bit to get dinner, shall we batcave later about steps to deployment? [10:52:22] fdans: sure [10:52:50] fdans: 1h before standup would good for me (I'll be away for a reak sioon as well) [10:54:50] joal: perfect! [11:09:59] 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3156999 (10Milimetric) I've replicated the bug in vagrant. Annoyingly, there's a parameter called "pr... [11:21:06]  [12:13:33] * elukey brb [12:52:21] remaging analytics1035 (journal node) [12:58:24] :) nice! [12:58:31] did you already to 1057 elukey? [12:58:38] 06Analytics-Kanban, 06Operations, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3157141 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1035.eqiad.wmnet'] ``` The log can... [12:58:39] joal: i got some scala/spark qs for you in a bit... [12:58:42] if you have a min [12:59:10] ottomata: yep! [13:01:46] 10Analytics-Cluster, 06Analytics-Kanban: Hadoop cluster expansion. Add Nodes - https://phabricator.wikimedia.org/T152713#3157144 (10Ottomata) [13:12:13] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3157170 (10Ottomata) > How's the process to decommission db1047 going? I guess ok! I think we should just dump all the user created databases to a file and archive it before... [13:14:32] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3157185 (10Marostegui) >>! In T156844#3157170, @Ottomata wrote: >> How's the process to decommission db1047 going? > > I guess ok! I think we should just dump all the user c... [13:24:28] !log deployed slightly improved eventlogging_sync.sh script for on db1047 and dbstore1002 [13:24:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:25:18] 06Analytics-Kanban, 06Operations, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3157205 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1035.eqiad.wmnet'] ``` and were **ALL** successful. [13:26:08] 10Analytics-EventLogging, 06Analytics-Kanban: Research Spike: Better support for Eventlogging data on hive - https://phabricator.wikimedia.org/T153328#2877030 (10Ottomata) a:03Ottomata [13:44:05] chowning 1035 [13:44:18] lvm volume created etc.. [13:44:19] all good [13:50:04] yehaw [13:50:20] it should be back now :) [13:52:31] oh yeah dfs health is now reporting 3 journal nodes up to date [13:52:53] one to go then! [13:52:55] 1028 [13:59:36] joal talk now? [13:59:40] fdans: Heya [13:59:57] fdans: A la cave ! [14:04:41] joal: when you got a min i want you too! :) [14:05:40] (03CR) 10Ottomata: [WIP] Add Spark schema handler to refinery-core (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [14:08:28] elukey: i think it was down to you and joal, if you both have something earlier, that would work too [14:08:44] elukey: you, joal, and i, that is [14:08:59] Hey ottomata ! [14:09:06] I'm for you now :) [14:09:29] elukey, joal: and i still think we should look for a better time slot (this one is clearly problematic ;)) [14:10:23] joal: i think i'm on to something, but ya maybe good to talk anyway :) [14:10:47] in bc if you wanna [14:10:52] urandom: half an hour earlier is good for me! [14:12:15] joal: is 1/2 hour earlier for the cassandra stand-up OK with you? [14:12:34] urandom: Works for me :) [14:12:50] sweet! [14:21:09] Thanks urandom for moving the conflict! [14:23:27] no worries [14:43:33] For those liking images transformed by computers: https://github.com/junyanz/CycleGAN [14:46:03] zebras everywhereeeeeeee!! http://www.loupiote.com/photos_l/5056380696-traffic-zebras-la-paz-bolivia.jpg [14:47:00] so cool [14:47:27] in a few years we'll be sure not to trust reallity anymore :) [14:52:35] 06Analytics-Kanban, 10Analytics-Wikistats: Visual prototype for community feedback for Wikistats 2.0 iteration 1. - https://phabricator.wikimedia.org/T157827#3157514 (10Milimetric) // done - round/shorten numbers on dashboard from full to 7.2 billion, etc.// // done - show 2 years of data on widgets// //... [15:00:41] a-team, fdans : standddupp [15:13:12] !log removed /etc/cron.daily/blogreport from eventlog1001 (manual backup in /home/elukey) [15:13:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:13:16] ottomata: --^ [15:14:20] (03CR) 10Joal: "Samoe comments - Should work, but some improvement can be made" (0315 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346519 (https://phabricator.wikimedia.org/T157362) (owner: 10Fdans) [15:16:26] joal: FYI!!!! [15:16:33] // !!! THIS WORKS AND ADDS PARTITIONS TO HIVE!! :O :O :O [15:16:33] dfWithPartitions.write.mode("append").partitionBy("year", "month", "day").insertInto("otto.PageMove1") [15:16:41] 10Analytics-Dashiki, 06Analytics-Kanban: Refactor aqs api and usage for simplicity - https://phabricator.wikimedia.org/T161933#3157553 (10Nuria) a:03fdans [15:16:41] show partitions... [15:16:41] year=2017/month=3/day=31 [15:16:45] awesome [15:16:54] ottomata: and the partition is needed [15:17:32] yeah, and it seems (so far) that the order of the hive schema matters... not totally sure about that yet [15:17:37] when inserting [15:17:49] ottomata: right :( [15:17:59] ottomata: It means full sorting, or full unsorting [15:31:27] Hey folks. I'm looking for a good reference for our current number of monthly unique devices. Where would I find that? [15:33:42] elukey: i'm going to start 1028, s'ok? [15:33:43] halfak: what do you mean by reference? [15:33:50] ottomata: sureee [15:33:55] I want to link to it in a research paper [15:33:57] the metric is on the API: [15:34:05] A report would be fine [15:34:09] so you want the explanation for how it's computed? [15:34:15] Nope. Just the stat [15:34:21] in human-readable form [15:34:58] halfak: does this work? https://analytics.wikimedia.org/dashboards/vital-signs/#projects=enwiki/metrics=UniqueDevices [15:34:58] halfak: https://tools.wmflabs.org/siteviews/?platform=all-sites&source=unique-devices&range=latest-20&sites=en.wikipedia.org ? [15:35:12] halfak: FYI that that number is much less stable than the daily one [15:35:22] halfak: *seems* to be i'd say [15:35:27] joal: that's pageviews, halfak you want unique devices right? [15:35:37] milimetric: I don't think :) [15:35:37] nuria: I linked to the daily for that reason [15:35:45] I want monthly uniques [15:35:51] k, monthly is this one: [15:35:56] nuria, what do you mean monthly is "much less stable"? [15:36:06] https://analytics.wikimedia.org/dashboards/vital-signs/#projects=enwiki/metrics=MonthlyUniqueDevices [15:36:12] milimetric: siteviews include uniques : &source=unique-devices& [15:36:16] halfak: we have few measures of it and it oscillates a lot [15:36:19] milimetric, awesome. Thanks. [15:36:23] much more than daily [15:36:33] yes and for monthly: https://tools.wmflabs.org/siteviews/?platform=all-sites&source=unique-devices&start=2016-04&end=2017-03&sites=en.wikipedia.org [15:36:40] nuria, seems that daily is the oscilator because it's got a clear period of 7 days. [15:37:00] halfak: ya but the "overall" variability is less [15:37:13] joal: hm, oh I see now but musikanimal you should change the labels to say "unique devices" instead of "pageviews" when a different metric is used, that was confusing [15:37:33] yeah I was wondering about that [15:37:39] milimetric, do you expect that this URL will be stable? https://analytics.wikimedia.org/dashboards/vital-signs/#projects=enwiki/metrics=MonthlyUniqueDevices [15:37:57] halfak: yep, that's stable, but so is musikanimal's [15:38:21] it should just say "Unique devices: 7,116,448,104" and not "Pageviews: ..." [15:38:21] halfak: you can add other projects to the #projects=... if you want, too [15:38:50] musikanimal: yep, that would be useful, or on the label when you hover [15:38:53] or both [15:39:13] musikanimal: sub-title should also change (comparision of unique devices across multiple projects) [15:39:30] indeed [15:39:57] will fix, thank you! [15:40:15] thank you musikanimal ! [15:40:45] Thanks folks. This'll work for me. :) [15:45:36] (03PS3) 10Joal: [WIP] Add Spark schema handler to refinery-core [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) [15:51:01] joal: bad news, with lots of data with multiple schemas, spark isn't that great at finding all the possible schemas [15:51:12] NavigationTiming is a good example, it has two pretty active revisions [15:51:22] looking at spark code, it looks like it just does a dataset.sample [15:51:27] ottomata: mwarf [15:51:27] and then infers schema from the results [15:51:40] even if i do the map by revision thing [15:51:43] it doesn't help [15:51:47] because by that point the schema has alredy been loaded [15:52:03] so [15:52:16] we may need to look at the jsonschemas after all :/ [15:52:25] ottomata: another way - Load data as text, filter by revision using RDD, then laod into json [15:52:37] hm, just text matching? [15:52:39] hm [15:52:53] regexp matching in text for revision id [15:52:54] HMmmmmm [15:52:57] yeahhh [15:52:59] ok will try that, good idea [15:53:05] not optimal, but could work [15:53:08] ya [15:53:16] split your datasets, then read json dataframe out of rdd [15:53:21] ya [15:53:25] sqlContext.read.json(RDD) works [15:53:43] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add Spark schema handler to refinery-core [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [15:54:15] ja that's how i'm loading now, since i use rdd's sequenceFile to load [15:54:26] so i actually arledy have a string rdd [15:54:27] trying... [15:57:31] hmmm, joal I could modify Camus' SequenceFileRecordWriterProvider to have a configureable SequenceFile key [15:57:47] and we could make the sequence file loaded rdd alreday keyed by revision [15:58:24] ottomata: hm [15:58:36] right now it has https://github.com/linkedin/camus/blob/master/camus-etl-kafka/src/main/java/com/linkedin/camus/etl/kafka/common/SequenceFileRecordWriterProvider.java#L94-L97 [16:01:51] ottomata, joal, elukey : standup? [16:01:57] sorry staff? [16:02:29] we are just missing ottomata [16:03:05] !log restarted webrequest-load-wf-text-2017-4-5-14 [16:03:05] FYI for anyone curious about what I'm getting citations for, see https://meta.wikimedia.org/wiki/Research:Interpolating_quality_dynamics_in_Wikipedia_and_demonstrating_the_Keilana_Effect [16:03:06] sorry [16:03:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:03:07] :) [16:03:11] didn't click the join button [16:12:46] (03PS4) 10Joal: [WIP] Add Spark schema handler to refinery-core [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) [16:23:19] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3157874 (10Tbayer) >>! In T156844#3157170, @Ottomata wrote: >> How's the process to decommission db1047 going? > > I guess ok! I think we should just dump all the user creat... [16:26:19] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3157875 (10Ottomata) > I thought the plan was to import them (in particular the "staging" database) to dbstore1002, so that they can be queried there as before? Ah sure we can... [16:27:58] 06Analytics-Kanban, 06Operations, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3157878 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by otto on neodymium.eqiad.wmnet for hosts: ``` ['analytics1028.eqiad.wmnet'] ``` The log can b... [16:32:15] fdans: https://www.mediawiki.org/wiki/Scrum_of_scrums#Meeting_notes [16:32:25] milimetric: thank you! [16:32:28] should read all of them, you know, for completeness ;) [16:32:43] (kidding, it's like War and Peace) [16:32:58] too late [16:33:02] already printing em all [16:53:52] 06Analytics-Kanban, 06Operations, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3157937 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1028.eqiad.wmnet'] ``` and were **ALL** successful. [17:11:53] fdans: still here ???? [17:31:01] logging off people! [17:31:09] * elukey afk [18:16:40] joal: if you are still working, could use brain bounce [18:19:18] hey ottomata [18:19:23] batcave? [18:20:05] ya [18:32:22] halfak: one minute for me before I leave? [18:35:28] Later then halfak :) Bye a-team [18:35:39] sorry in meeting [18:35:45] Will be around tomorrow morning UTC-5 [18:35:46] :D [18:35:47] o/ [18:35:49] have a good night [19:54:12] nuria: since the version of mariadb we use doesn't support https://mariadb.com/kb/en/mariadb/json-functions/, could you please recommend a way to work with the new JSON-based user agents in SQL? thank you :) [20:03:25] bearloga: Given that our version does not support neither json-functions or json types i have no great recommendations to give cause neither parsing or casting at select time would work, sorry [20:03:54] okie dokie [20:17:07] 10Analytics, 07Performance: Eventlogging client needs to support offline events - https://phabricator.wikimedia.org/T162308#3158732 (10Nuria) [20:24:39] 10Analytics: Measuring non content pageviews - https://phabricator.wikimedia.org/T162310#3158770 (10Nuria) [20:24:55] 10Analytics: Measuring non content pageviews - https://phabricator.wikimedia.org/T162310#3158782 (10Nuria) [22:57:18] 06Analytics-Kanban: All Dashiki Dashboards down - https://phabricator.wikimedia.org/T162320#3159115 (10Milimetric)