[04:04:22] 06Analytics-Kanban, 10Analytics-Wikistats: Initial FE code for Wikistats 2.0. Dashboard skeleton - https://phabricator.wikimedia.org/T163814#3216608 (10Nuria) [04:04:43] 10Analytics, 10Analytics-Wikistats: Initial FE code for Wikistats 2.0. Dashboard skeleton - https://phabricator.wikimedia.org/T163814#3211082 (10Nuria) [08:31:03] hello people! [08:31:13] moar network changes today :D [08:31:25] starting at 4 PM UTC [08:31:52] this time two aqs nodes, thorium and 4 analytics workers [08:34:02] the thing that I am worried about are the data loss emails [08:34:04] from oozie [08:34:13] 1% of data is a lot [08:35:39] ah no wait this might be due to an issue with upload [08:37:50] asking to traffic, this might not be related to kafka [08:38:47] confirmed, all traffic related [09:31:26] elukey: ok to upgrade mysql on bohrium now? [09:34:37] moritzm: sure! Now that I played a bit with Piwik I am wondering if these upgrades needs that Piwik stops sending sql commands [09:35:02] there is an option to keep getting requests but not sending the relative INSERT to mysql [09:35:20] even if the upgrade is so quick that it shouldn't matter [09:40:36] sure, shall we test that option for this update? [09:43:21] yep, let me enable it [09:44:19] moritzm: done, you can upgrade [09:44:30] !log disabled tracking in piwik to allow mysql upgrade [09:44:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:44:37] I'll add a note in service restarts [09:44:52] it is basically a simple [09:44:54] [Tracker] [09:44:54] record_statistics = 0 [09:45:02] in config.ini + apache restart [09:45:34] well I guess I can simply apt-get update && apt-get install mysql-sever? [09:46:11] done [09:47:19] !log re-enabled tracking in piwik after maintenance [09:47:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:25:57] going to lunch + some errands to do, bbl! [12:06:23] !log Manually push pageview whitelist to silence oozie alerts [12:06:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:12:52] joal: o/ - how did you do it? [12:13:20] elukey: in up-to-date refinery folder [12:13:21] sudo -u hdfs hdfs dfs -put -f static_data/pageview/whitelist/whitelist.tsv /wmf/refinery/current/static_data/pageview/whitelist [12:13:57] ah yes just swapping the file [12:14:20] elukey: indeed [12:14:43] I don't remember if you once told me that this action would have used a single hdfs block rather than something more efficient? [12:17:50] elukey: I think you mix up with using "INSERT ... IN wmf.pageview_whitelist" [12:18:03] ahhh yesss [12:18:06] there you go [12:18:13] thanks :) [12:18:21] elukey: Hive insert creates a new file with the new line (1 full hdfs block for one new line ... mwarf) [12:18:48] Replacing the full file with the correct one if the correct way to go ;) [12:29:23] now it is super clear, thanks :) [12:31:51] joal: today's maintenance impact https://etherpad.wikimedia.org/p/analytics-row-d-maintenance [12:32:13] not as bad as yesterday but might need a bit of care [12:32:18] especially the cassandra instances [12:32:27] I asked if possible to do one host at the time [12:32:28] By the way elukey, forgot to thank you again for the extra work yesterday evening [12:32:39] :) [12:33:19] elukey: We could pause cassandra loading job for the time the hosts are down [12:33:24] elukey: to be extra sure [12:33:46] yep, writing it down [12:33:47] elukey: doesn't cost a lot, and removes moving part [12:34:33] yep yep +2 [12:38:09] elukey: except from that, I think it's good :) [12:39:19] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3217291 (10Marostegui) That is fine by us, but then we probably want to go ahead and fix this: T159266 [13:17:56] hi team :] [13:20:31] joal, whenever you have time today, can we talk about how we want oozie to generate the virtual _SUCCESS files for banner jobs? when we have a plan, if you're OK, I can implement it in both jobs [13:21:14] Hi mforns ! Currently reviewing ottomata scala stuff, we can talk about it after grooming? [13:21:23] joal, sure! [13:33:44] 06Analytics-Kanban: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3217461 (10mforns) As far as I know, master only holds data for the last 45 days. So, slaves and analytics-storage are the only one who keep the historical data. I may be wrong, but that is m... [13:34:05] be back shortly, bikign to manhattan w dan [13:35:05] I propose to have a onboard camera on Andrew's bike [13:35:22] it would be awesome to bike in manhattan once in a while :D [13:45:41] +1 elukey ! [13:55:33] (03CR) 10Joal: "Review only StructExtensions - Comments inline." (0312 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [14:01:27] milimetric: o/ - I came up with this config.ini for piwik - https://gerrit.wikimedia.org/r/#/c/348938/9/modules/piwik/templates/config.ini.php.erb [14:01:41] I added the options to disable tracking and maintenance mode [14:02:03] it may change from one version to the other but we'll have a consistent configuration [14:02:24] passwords are going to be stored in the puppet private repo [14:03:33] elukey: that's great, piwik is all proper now [14:03:50] milimetric: still need to work on the db :/ [14:03:58] My fellow mforns and fdans ! Where are you ? [14:04:07] joal, in the room! :] [14:04:10] omw! [14:04:18] Arf - Went to batcave this tiem :) Joining ! [14:18:11] elukey: hiyaa [14:18:17] just checkin in, how are things? [14:18:46] kafka looks balanced :) [14:20:44] elukey: did you see: https://gerrit.wikimedia.org/r/#/c/350485/ ? [14:20:54] elukey: are you comfortable with that going out in the Puppet SWAT? [14:21:26] elukey: it should be very safe, just a config change to cassandra-metrics-collector, to blacklist something that the collector doesn't yet collect [14:21:33] (in anticipation of when it does) [14:23:21] elukey: tl;dr super low risk, and the worst that could happen is broken cassandra metrics collection, but i want to avoid the situation where i throw changes at your cluster that you were unaware of :) [14:23:21] ottomata: o/ yesss everything looks good, but I've updated https://etherpad.wikimedia.org/p/analytics-row-d-maintenance with today's work [14:24:01] urandom: everything looks fine, thanks for the heads up :) [14:24:35] elukey: coo [14:25:37] elukey: i'm also going to deploy a new jar for that, but i'll give you another heads up before the change to link that in place [14:26:13] super [14:29:20] elukey: that's what needs to be done today? [14:29:28] elukey: dan and I are going to a city council meeting at 1pm our time :o [14:30:21] ottomata: yeah, from 16:00 UTC onwards (max 1 hour so it shouldn't be a big issue) [14:30:45] ok [14:41:57] 10Analytics: Better publishing of Annotations about Data Issues - https://phabricator.wikimedia.org/T142408#3217694 (10Milimetric) Just a warning: Config:Dashiki: is what the dashiki extension configures, so I think it's best to prefix with that because I really don't want to make another extension. We could th... [14:46:35] 06Analytics-Kanban, 10Analytics-Wikistats: Visual prototype for community feedback for Wikistats 2.0 iteration 1. - https://phabricator.wikimedia.org/T157827#3217733 (10Milimetric) It's possible we misunderstood, Erik, I think this is what you said about it: "Infobox Unique Devices It's no secret I feel t... [14:56:07] 06Analytics-Kanban: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3217801 (10Nuria) >As far as I know, master only holds data for the last 45 days. I think this is incorrect, we recently deleted data from master that was older than 45 days [14:56:32] joal: i'm thinking about your coment about the Seq[StructField] implicit class [14:57:16] 10Analytics: Better publishing of Annotations about Data Issues - https://phabricator.wikimedia.org/T142408#3217802 (10Nuria) +1 to Config:Dashiki, we do not need another namespace. [14:57:18] it def makes the code less DRY [14:57:29] since I have to repeat extra .map and sometimes .mkStrings around [14:57:39] but, maybe, I should get rid of the StructType implicit class [14:57:40] since [14:57:43] 06Analytics-Kanban: Create purging script for mediawiki-history data - https://phabricator.wikimedia.org/T162034#3217824 (10JAllemandou) Question to solve: In some tables, the snapshot partition has sub-partitions (for instance, in the wmf_raw.mediawiki_user table, table is partitioned by (snapshot, wikidb). Can... [14:57:48] StructType implements Seq[StructField] [14:59:36] and i guess i could always do struct.toSeq.normalize [14:59:45] struct.toSeq.hiveCreateDDL [15:00:54] ottomata: I'd do it the other way artound [15:01:16] joal,ottomata standup for your rights [15:01:23] juas [15:01:24] coominnggg [15:08:44] joal, but, if i do the implicit on Seq[StructField], i just realized I get the methods automatically on StructType [15:08:52] since it implements Seq[StructField] [15:08:57] so I can just as easily do [15:09:01] struct.hiveColumnsDDL [15:09:02] as I can [15:09:06] fields.hiveColumnsDDL [15:10:34] ottomata: I'd rather augment StructType - The function you add are for StructType, even if from a code perspective it's easier to add them to Iterable[StructField] [15:10:51] ottomata: Can be discussed though :) [15:14:10] hmm, i guess in that the user of these is expected to work with StructTypes [15:14:17] since that is what is returned by DataFrame .schema [15:14:41] but at the same time. StructType is a Seq[StructField] [15:14:42] so [15:14:48] df.schema.hiveCreateDDL [15:14:48] will work [15:14:57] if the implicit class Seq[StructField] exists [15:15:11] so will [15:15:24] schema.merge(otherSchema), where schema and otherSchema are both StructTypes [15:16:09] ottomata: My idea is more: merging StructTypes (because they are schemas etc) makes sense, merging Seq[StructFields] in general doesn't [15:16:35] but a StructType is a Seq[StructField] [15:17:11] with some extra methods [15:17:30] yes [15:17:36] and the merging is always done by iterating on StructType, which iterates on the StructFields in the struct [15:17:47] But a Seq[StructField] is not necessarily a StrucType [15:18:01] but it works exactly the same in both cases [15:18:13] It's not about how code works, it's more about what types represent [15:18:30] ottomata: But it's really details [15:18:38] it's like names :) [15:18:41] yeah, i see that, in that how the user will use the stuff they'll always be working with a StructType [15:18:41] :P [15:18:56] i guess i like it, because the other functions usually end up working with Seq[StructField] [15:18:56] urandom: helloooo [15:19:00] .map, filter, etc [15:19:01] time for a consult ? [15:19:13] so its nice to not have to convert the results of those into a StructType in order to keep working [15:19:18] with the implicit functions [15:19:38] elukey: ya! [15:19:55] elukey: do you want random advice, or about something specific? [15:20:10] elukey: i recommend you watch at least one sunset every day. [15:20:33] haha sorry joal, let's postpone discussion, i should pay more attention :p [15:20:36] :) [15:21:10] elukey: give a man a fish, and you'll have one less fish. [15:21:45] urandom: lol - aqs100[69] are going under row-d maintenance soon, just brief connectivity gap.. should I stop cassandra in there? [15:21:48] :) [15:21:58] it will be one instance at the time [15:21:59] elukey: naw [15:22:08] elukey: should be (better be) find [15:22:30] elukey: and the restbase nodes were [15:22:32] fwiw [15:22:47] s/find/fine/ [15:25:20] super thanks :) [15:25:30] elukey: this is the P in CAP fwiw, partition-tolerance :) [15:27:15] urandom: I don't take anything for granted when I hear "cassandra" [15:27:25] oh snap! [15:27:26] :P [15:27:29] ahhahaa [15:27:43] joking, you are completely right [15:27:57] you are not 100% joking [15:28:07] if you were, it wouldn't be so funny! :) [15:28:40] you got me [15:28:59] I'll have my revenge [15:29:18] heh [15:30:58] urandom: are you going to miami for apachecon? [15:31:38] elukey: nope [15:32:16] elukey: would have been nice, but no [15:51:15] 10Analytics: upgrade druid and pivot - https://phabricator.wikimedia.org/T157977#3218094 (10Nuria) * Let's update pivot from source first, let's test the sourcing of schemas (automatic updates). Pivot is on scap * Let's update storage 2nd. [15:51:28] 10Analytics: Update pivot to latest source - https://phabricator.wikimedia.org/T164007#3218097 (10Nuria) [15:51:40] 10Analytics: Update druid to latest source - https://phabricator.wikimedia.org/T164008#3218113 (10Nuria) [15:51:51] 10Analytics: Update druid to latest release - https://phabricator.wikimedia.org/T164008#3218113 (10Nuria) [15:56:04] (03PS1) 10Joal: Add dv.wikipedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/350589 [15:56:13] 10Analytics: Update pivot to latest source - https://phabricator.wikimedia.org/T164007#3218171 (10Nuria) Pivot source will need to be in a repository that is not public. Let's clear how to do this with release engineering, Pointed assuming this is possible. [15:56:35] 06Analytics-Kanban: Update pivot to latest source - https://phabricator.wikimedia.org/T164007#3218097 (10Nuria) [15:57:12] ottomata: for some reason ferm was not automatically brought up on hadoop worker nodes [15:57:25] 10Analytics: Update druid to latest release - https://phabricator.wikimedia.org/T164008#3218175 (10Nuria) Release notes for 0.10 version: https://github.com/druid-io/druid/issues/3944 [15:58:54] elukey: that is strange [15:58:56] shouldn't puppet just do it? [15:59:21] yeah, tried to run it but nothing.. [15:59:25] will open a phab task [15:59:33] I removed the downtime that I've set yesterday [15:59:35] and booom [15:59:36] :D [16:00:11] 10Analytics: Update druid to latest release - https://phabricator.wikimedia.org/T164008#3218194 (10Nuria) Some extra work for debianization for the new SQL apache calcilte server. Maybe puppet changes for overlord/coordinator setup. [16:00:43] 06Analytics-Kanban: Update druid to latest release - https://phabricator.wikimedia.org/T164008#3218113 (10Nuria) [16:02:44] 06Analytics-Kanban: upgrade druid and pivot - https://phabricator.wikimedia.org/T157977#3218202 (10Nuria) [16:08:01] stopping yarn and then hdfs on an1038->1041 [16:10:34] 10Analytics: Update pivot to latest version - https://phabricator.wikimedia.org/T161630#3218236 (10JAllemandou) [16:10:36] 06Analytics-Kanban: Update pivot to latest source - https://phabricator.wikimedia.org/T164007#3218234 (10JAllemandou) [16:10:41] 06Analytics-Kanban, 10Analytics-Wikistats: Initial FE code for Wikistats 2.0. Dashboard skeleton - https://phabricator.wikimedia.org/T163814#3218237 (10Nuria) [16:11:29] 06Analytics-Kanban: Finalize list of metrics, breakdowns, and filters for Wikistats 2.0 backend - https://phabricator.wikimedia.org/T163356#3218240 (10JAllemandou) [16:15:24] elukey: I restart failed jobs [16:15:53] thanks :) [16:16:21] I hoped that it wasn't necessary sigh [16:16:50] milimetric, mforns when do you want to take a look at the prototype? [16:17:15] fdans, milimetric, I'd like to close some of my ongoing stuff before starting another thing [16:17:23] oh actually mforns do you have a couple mins to talk about eventlogging beta? [16:17:35] fdans, yes, that's one of the things :] [16:17:52] voltams a la batcova? [16:18:00] (sorry for my horrible catalan mforns ) [16:18:08] tornem, hehehe [16:18:08] (03CR) 10Nuria: [V: 032 C: 032] Add dv.wikipedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/350589 (owner: 10Joal) [16:18:27] terrible galician mixture [16:18:35] fdans: i meant look at it yourself, and we start working on the real skeleton tomorrow [16:18:44] ahh cool cool [16:20:40] going to lunch then to meeting thing, back later afternoon [16:23:32] restarted maps job [16:32:47] network maintenance done on thorium and aqs1009 [16:32:48] all good [16:37:49] !log restart Workflow aqs-hourly-wf-2017-4-27-14 and Workflow pageview-hourly-wf-2017-4-27-14 [16:37:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:40:30] !log Manually push (again) pageview whitelist [16:40:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:44:36] (03CR) 10Nuria: "Thanks for doing changes." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/350247 (owner: 10Tjones) [16:49:42] ok maintenance done, we are good :) [16:50:32] un-suspended the aqs oozie jobs [17:16:34] * elukey going afk team! [17:16:41] bye elukey ! [17:16:48] there is a small network glitch but it seems not affecting us [17:16:53] will check later on [17:16:55] byeee [17:41:42] Hey mforns ! [17:41:48] joal, hi! [17:41:57] I'm with Fran in da cave [17:42:01] doing EL [17:42:07] Do we take 10 mminutes before metrics meeting on druid-reindexation-job-scheduling? [17:42:13] ah mforns :) [17:42:20] cool [17:42:32] do you want to cave with us and we can speak? [17:43:26] sure :) [17:44:22] 10Analytics: Webrequest tagging and distribution. Measuring non-pageview requests - https://phabricator.wikimedia.org/T164019#3218528 (10Nuria) [17:45:29] 10Analytics: Spike, test idea on spark job that reads tags and produces different outputs - https://phabricator.wikimedia.org/T164020#3218541 (10Nuria) [17:46:51] 10Analytics: Create tagging udf - https://phabricator.wikimedia.org/T164021#3218558 (10Nuria) [18:34:59] wikimedia/mediawiki-extensions-EventLogging#649 (REL1_29 - 76be198 : Umherirrender): The build has errored. [18:35:01] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/REL1_29 [18:35:01] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/226514519 [18:58:12] nuria: automated events looking good on beta, spent some time with mforns checking [18:58:27] just need to make tox stop complaining [18:58:42] fdans: then i think once jenkins runs clean we can merge! [18:58:44] thank you [18:59:47] megathanks to mforns , we took aaaaaages to find a minor blocker and he was a great sport [19:10:19] mforns zen master jiaa [19:40:40] (03CR) 10Milimetric: "bump" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/349723 (https://phabricator.wikimedia.org/T143119) (owner: 10Milimetric) [19:47:34] 10Analytics, 06DC-Ops, 06Operations, 10ops-eqiad, 13Patch-For-Review: Decom/Reclaim analytics1027 - https://phabricator.wikimedia.org/T161597#3218998 (10Cmjohnson) p:05Normal>03Low [19:55:15] :] [19:55:25] good night team, see you tomorrow [20:02:49] 10Analytics, 06DC-Ops, 06Operations, 10ops-eqdfw: SATA errors for stat1004 in the dmesg - https://phabricator.wikimedia.org/T162770#3219095 (10Cmjohnson) @elukey That is not a disk issue....What mode do you want the server to be in? Raid mode or AHCI? [20:14:31] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017), 07Regression: Only display organizations defined in Wikimedia's DB (disable assuming orgs via hostnames in email addresses) - https://phabricator.wikimedia.org/T161308#3219151 (10Qgil) a:03Albertinisg [20:15:43] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): https://wikimedia.biterg.io shows 2017 contributors who are not listed in mediawiki-identities/wikimedia-affiliations.json - https://phabricator.wikimedia.org/T161235#3219159 (10Qgil) a:03Albertinisg [20:16:33] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Updated data in mediawiki-identities DB not deployed onto wikimedia.biterg.io? - https://phabricator.wikimedia.org/T157898#3219161 (10Qgil) a:03Albertinisg [20:16:51] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): On the "Git" dashboard, filtering on one organization still lists authors who are with another organization - https://phabricator.wikimedia.org/T157709#3219163 (10Qgil) a:03Albertinisg [20:17:12] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): When indexing new users, identify identical email addresses and merge identities accordingly in the DB - https://phabricator.wikimedia.org/T151634#3219166 (10Qgil) a:03Albertinisg [20:30:01] (03PS9) 10Nuria: Changes api glue code to accept a project or array of same [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/347305 (owner: 10Fdans) [20:39:04] (03PS1) 10Milimetric: Implement showLastDays option on tab layout [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/350692 (https://phabricator.wikimedia.org/T160796) [20:40:28] (03CR) 10Milimetric: "to test, check out http://localhost:5000/src/layouts/tabs/#mobile-site-by-browser/browser-family-and-major-tabular-view and the other grap" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/350692 (https://phabricator.wikimedia.org/T160796) (owner: 10Milimetric) [20:46:45] (03CR) 10Milimetric: "minor cosmetics, will probably do these myself tomorrow and rebase" (032 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/347305 (owner: 10Fdans) [21:20:52] 10Analytics, 10MediaWiki-Releasing: Create dashboard showing MediaWiki tarball download statistics - https://phabricator.wikimedia.org/T119772#3219426 (10Nemo_bis) >>! In T119772#3168729, @Nuria wrote: > It is not clear what is the value of this data, can someone explain? It would aid MediaWiki development p... [21:31:16] (03CR) 10Ottomata: [WIP] Spark + JSON -> Hive (0313 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [21:31:26] (03PS16) 10Ottomata: [WIP] Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [21:37:13] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [21:49:09] (03PS17) 10Ottomata: [WIP] Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal)