[03:12:37] Analytics, Graph, Pageviews-API: Unable to get pageviews for the title with ' in the name - https://phabricator.wikimedia.org/T129346#2121260 (Milimetric) Sorry, no suggestions from me, I think the pageview API is doing the right thing, just blindly decoding. [03:17:52] Analytics: Find performance thresholds of piwik production instance - https://phabricator.wikimedia.org/T123640#2121263 (Milimetric) Open>Resolved a:Milimetric I think we can roughly handle two million requests per day. Something close to that seems to peg the CPU just while it's receiving data, to... [03:42:30] Analytics-Kanban, user-notice: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2121291 (Shizhao) Open>Resolved [03:51:43] Analytics-Cluster: Story: Community has periodic browser stats report generated from Hadoop data - https://phabricator.wikimedia.org/T69053#2121302 (Krinkle) duplicate>Open Re-opening since it is apparently not a duplicate as both have been closed without clear reason. [04:10:17] Analytics: Run browser reports on hive monthly - https://phabricator.wikimedia.org/T118330#2121310 (Krinkle) [04:10:19] Analytics-Kanban, Reading-Admin, Patch-For-Review: Tabular layout on dashiki [8 pts] {lama} - https://phabricator.wikimedia.org/T118329#2121311 (Krinkle) [04:10:21] Analytics-Cluster: Story: Community has periodic browser stats report generated from Hadoop data - https://phabricator.wikimedia.org/T69053#2121309 (Krinkle) [04:10:23] Analytics-Cluster, Analytics-Kanban, Reading-Admin, Easy, Patch-For-Review: PM sees reports on browsers (Weekly or Daily) {lama} [8 pts] - https://phabricator.wikimedia.org/T88504#2121312 (Krinkle) [04:39:52] Analytics-Kanban, Patch-For-Review: Create reportupdater browser reports that query hive's browser_general table {lama} - https://phabricator.wikimedia.org/T127326#2121318 (Krinkle) [04:39:54] Analytics-Cluster: Story: Community has periodic browser stats report generated from Hadoop data - https://phabricator.wikimedia.org/T69053#2121317 (Krinkle) [04:43:55] Analytics-Visualization: Limn reportcard "sign in" OAuth redirect to localhost - https://phabricator.wikimedia.org/T55096#2121321 (Krinkle) [05:25:28] (PS2) Milimetric: [WIP] Add a legend to each graph in the tabs layout [analytics/dashiki] - https://gerrit.wikimedia.org/r/276649 (https://phabricator.wikimedia.org/T129497) [06:45:41] (PS3) Mforns: Make datasets api more flexible [analytics/dashiki] - https://gerrit.wikimedia.org/r/276492 (https://phabricator.wikimedia.org/T129497) (owner: Milimetric) [06:51:30] (CR) Mforns: [C: 2 V: 2] "LGTM!" [analytics/dashiki] - https://gerrit.wikimedia.org/r/276492 (https://phabricator.wikimedia.org/T129497) (owner: Milimetric) [06:59:50] (CR) Mforns: "Ready to review :]" [analytics/reportupdater] - https://gerrit.wikimedia.org/r/276758 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [10:26:52] (CR) Mforns: "Ready for review :]" [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/276763 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [10:27:33] (CR) Mforns: "Ready to review :]" [analytics/reportupdater] - https://gerrit.wikimedia.org/r/277215 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [11:02:22] Analytics-Kanban: Build a Dashiki dashboard that shows basic OS and browser request breakdowns - https://phabricator.wikimedia.org/T129102#2122073 (mforns) I finished testing all the components together. Everything seems to work. The configuration of the Dashboard is here: https://meta.wikimedia.org/wiki/Con... [11:12:24] Analytics-Kanban, Patch-For-Review: Create reportupdater browser reports that query hive's browser_general table {lama} - https://phabricator.wikimedia.org/T127326#2122149 (mforns) I finished testing this and it seems to work, so it's ready for review. I'll list the patches included in this task: https:/... [11:56:51] Analytics-Kanban, Patch-For-Review: Dashiki visualization that shows a hierarchy {lama} - https://phabricator.wikimedia.org/T124296#1952269 (mforns) In progress again, because I'm adding the labels to the inner-most circle. [12:20:32] Analytics, Pageviews-API, I18n: Message [[Wikimedia:Pageviews-select2-max-items/en]] needs PLURAL support. - https://phabricator.wikimedia.org/T130005#2122274 (Purodha) [12:22:13] Analytics, Pageviews-API, I18n: [[Wikimedia:Pageviews-hosted/ksh]] i18n issue. - https://phabricator.wikimedia.org/T130006#2122287 (Purodha) [12:23:10] Analytics, Pageviews-API, I18n: [[Wikimedia:Pageviews-hosted/ksh]] i18n issue. - https://phabricator.wikimedia.org/T130006#2122300 (Purodha) [13:04:54] GOodo mooorning! [13:06:47] Hi ottomata :) [13:10:33] hellooo [13:11:23] morning :) [13:31:35] (PS1) Joal: Add cassandra unique devices load job [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) [13:45:41] (CR) Joal: [C: -1] "Problem with monthly dataset dependency time." [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) (owner: Joal) [14:26:13] Analytics-Tech-community-metrics, DevRel-March-2016, Regression: top-contributors.html only displays seven entries - https://phabricator.wikimedia.org/T129837#2117561 (Qgil) I see 7. @aklapper appears in the short list, and I don't. Now I am jealous. :) [15:30:15] a-team, having problems joining hangouts... [15:30:36] a-team: standdupppp [15:30:45] madhuvishy, ottomata holaaaa? [15:31:19] OO [15:34:23] Analytics-EventLogging, Analytics-Kanban, DBA, Patch-For-Review: Add autoincrement id to EventLogging MySQL tables. {oryx} - https://phabricator.wikimedia.org/T125135#2122744 (Nuria) a:jcrespo [15:50:38] Analytics-Cluster, Operations, hardware-requests: eqiad: New Hive / Oozie server node in eqiad Analytics VLAN - https://phabricator.wikimedia.org/T124945#2122817 (Ottomata) Bump, @Robh, could we use WMF4541 instead of waiting? [15:51:07] joal: moritzm wants to reboot an27 [15:51:18] i'm going to stop camus there by stopping puppet, etc. [15:51:27] and wait til there is no camus run running [15:51:44] sounds good [15:52:58] ok moritzm, ja, there is a camus job running, it should be done in a few mins [15:53:20] moritzm: if puppet is disabled, will it still be disabled after reboot? [15:54:24] moritzm: AFAIK disabling puppet is persistant across reboots [15:57:27] ok cool, thought so too [15:57:34] ok moritzm go ahead and reboot an27 [15:59:13] ok, going ahead [16:06:30] uhhh ottomata I messed up and deleted the 1.3.0-wmf2 tag too [16:06:58] this is why I shouldn't have push rights :/ [16:08:59] analytics1027 is back up [16:17:47] ottomata: ping me when you're back [16:26:39] ok great, tahnks moritzm [16:26:42] madhuvishy: what'sup [16:26:44] ohhh ok [16:26:45] lemme see.. [16:27:45] madhuvishy: no worries [16:27:47] i had the tag locallly [16:27:49] so i just repushed it [16:30:27] ottomata: ahh thanks [16:30:28] so [16:30:31] my problem is [16:30:41] there are now these commits on the wmf branch [16:30:50] which go like "prepare version for release" [16:31:22] ottomata: but those commits change the version number etc and the next time i try to release:prepare it complains [16:31:56] can i reset to the last sane commit and force push at that point? or is there a better way [16:34:38] did you actually push those to the remote? [16:34:47] (or did I?) [16:35:48] ottomata: maven does [16:35:59] ok, madhuvishy ahh in meeting now [16:36:03] will help as soon as i can [16:36:05] i would probably [16:36:09] just go into gerrrit [16:36:13] and delete the wmf branch [16:36:17] cause we are startign new [16:36:22] and it will be easiest [16:36:26] then you can recreate it in gerrit [16:36:39] and delete your local, fetch and recheckout wmf [16:36:40] and start over [16:37:00] ottomata: ok will do. enjoy your meeting :) [16:44:51] (PS1) Madhuvishy: Add release plugin to maven build [analytics/ua-parser/uap-java] (wmf) - https://gerrit.wikimedia.org/r/277539 [16:45:25] (CR) Madhuvishy: [C: 2 V: 2] "Self merging" [analytics/ua-parser/uap-java] (wmf) - https://gerrit.wikimedia.org/r/277539 (owner: Madhuvishy) [17:34:51] madhuvishy: i need lunch, but real quick, how goes [17:34:52] ? [18:01:15] Analytics-Cluster: Story: Community has periodic browser stats report generated from Hadoop data - https://phabricator.wikimedia.org/T69053#2123447 (Nuria) [18:01:17] Analytics: Run browser reports on hive monthly - https://phabricator.wikimedia.org/T118330#2123445 (Nuria) declined>Open [18:01:25] Analytics: Run browser reports on hive monthly - https://phabricator.wikimedia.org/T118330#1797540 (Nuria) Re-opening. [18:11:58] ottomata: o/ [18:12:20] quick question - have you rebooted the hadoop cluster already? Otherwise I am planning to do it tomorrow morning EU time [18:14:01] hadoop cluster reboot? [18:14:02] no [18:14:07] moritz rebooted an26 and an27 today [18:16:39] ottomata: i got to the point where all the tests fail on release perform [18:17:14] ok! now you where i was! Geeez sorry this is so weird [18:19:55] madhuvishy: we should make this work...buuut, if you don't want to bother, you can do a manual upload of the .jar [18:19:57] to archiva [18:20:22] ottomata: i see bugs filed about this test failing thing on ua-parser [18:20:23] https://wikitech.wikimedia.org/wiki/Archiva#Uploading_Dependency_Artifacts [18:20:31] https://github.com/ua-parser/uap-java/issues/6 [18:20:42] https://github.com/ua-parser/uap-java/issues/1 [18:20:48] huh, yeah [18:20:51] they are both closed [18:21:05] and the comments have nothing to do with the original issue [18:26:35] madhuvishy: am running out to a cafe, HMMMm, i dunno. can you skip the tests? do they matter? [18:26:40] should we manually upload artifact? [18:26:57] i'd say, if thing sare fine, and we can't skip the test to mvn release, then just upload the artifact :/ [18:26:58] ottomata: ya let me see if we can skipTests during release:perform [18:27:01] i'll be back shortly [18:27:01] ok [18:36:28] https://archiva.wikimedia.org/repository/releases/ua_parser/ua-parser/1.3.0-wmf3/ [18:36:50] I had to skipTests and javadoc generation while doing release:perform [18:53:02] Analytics-Cluster: Story: Community has periodic browser stats report generated from Hadoop data - https://phabricator.wikimedia.org/T69053#2123649 (Krinkle) [18:53:04] Analytics-Kanban: Build a Dashiki dashboard that shows basic OS and browser request breakdowns - https://phabricator.wikimedia.org/T129102#2123650 (Krinkle) [18:54:58] madhuvishy: vack [18:54:59] back [18:55:35] mforns: are we running browser reports weekly or monthly now? [18:59:29] nuria: I guess mforns is not online anymore (late in india) [19:00:48] Analytics-Kanban, Patch-For-Review: Create reportupdater browser reports that query hive's browser_general table {lama} - https://phabricator.wikimedia.org/T127326#2123693 (Milimetric) I'm going to review and merge this, and we'll consider my plan the back-up now. I'll pause my work beyond the datepicker. [19:01:18] nuria: I'm skipping the comms meeting today - will catch up with you on how it wen [19:01:25] ottomata: so I pushed the jar [19:01:29] madhuvishy: k [19:01:39] https://www.irccloud.com/pastebin/y5PzAmsz/ [19:02:32] ah nice [19:02:36] via release [19:02:40] great madhuvishy [19:02:50] sorry that was so hard [19:03:52] ottomata: it's okay! maven release is a terrrible tool though. It's in two steps and the first step creates changes at the git level with no good rollback strategy - and assumes the next steps will succeed [19:04:16] release:clean cleans up any local files but not the remote commits [19:04:40] yeah indeed [19:05:15] no idempotence [19:05:27] anyway [19:05:46] now I can update refinery-source and see if anything breaks [19:05:57] like tests [19:07:30] Analytics, Patch-For-Review: Make TimeseriesData reactive so it can support a legend talking to a graph - https://phabricator.wikimedia.org/T129497#2123768 (Milimetric) p:Triage>Normal [19:07:50] (PS2) Joal: Add cassandra unique devices load job [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) [19:09:29] gah ottomata all our tests fail with that streamclosed exception [19:09:37] looks like have to figure it out [19:09:48] oh no [19:09:50] :/ [19:10:26] elukey: WOO [19:10:28] varnishkafka works [19:10:29] to kafka! [19:10:30] yeehaw [19:10:40] ottomata: That's greaT :) [19:10:47] that's fantastic, wow [19:14:10] a-team I'm gone for diner, later [19:33:32] (CR) Nuria: Make reportupdater support removing columns (1 comment) [analytics/reportupdater] - https://gerrit.wikimedia.org/r/277215 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [19:35:44] (CR) Nuria: Add the query folder as the last parameter of scripts (1 comment) [analytics/reportupdater] - https://gerrit.wikimedia.org/r/276758 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [19:44:44] Analytics-Kanban: browser_general table should have documenting page in wikitech - https://phabricator.wikimedia.org/T130060#2124079 (Nuria) [19:53:34] (CR) Milimetric: [C: 2 V: 2] Make reportupdater support removing columns (1 comment) [analytics/reportupdater] - https://gerrit.wikimedia.org/r/277215 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [19:56:30] (CR) Milimetric: Add the query folder as the last parameter of scripts (1 comment) [analytics/reportupdater] - https://gerrit.wikimedia.org/r/276758 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [19:58:37] Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2124118 (Nuria) [20:07:16] (CR) Nuria: "I see all these report raw view counts. I do not remember exactly but...weren't we reporting percentages?" [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/276763 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [20:17:42] Ironholds: u got probs? [20:18:08] ottomata, I'm allegedly now in the analytics-search-user account. How do I actually su into it? [20:21:51] sudo -u analytics-search [20:21:57] sudo -u analytics-search [20:22:16] you can't become the user [20:22:26] (i think), but you can execute commands as it [20:24:51] ottomata: where can i find hardware spect s for bohrium.eqiad.wmnet ? [20:24:57] ottomata: are those available? [20:25:36] ironholds@stat1002:~$ sudo -u analytics-search-user [20:25:36] sudo: unknown user: analytics-search-user [20:25:36] sudo: unable to initialize policy plugin [20:27:07] iron [20:27:07] Ironholds: can you do sudo -u analytics-search-user ls? [20:27:08] analytics-search [20:27:11] is the username [20:27:27] Ironholds: ^^ [20:27:29] nuria: uhhhh [20:27:35] i can find them in racktables, but that is not public [20:27:42] s/analytics-search-user/analytics-search [20:27:43] k [20:28:07] milimetric: what is the comand to build the layouts marcel was working on? [20:28:10] gulp --layout tabs --config SimpleRequestBreakdowns [20:28:12] no work [20:28:39] you need the patch that we were both working on, do you have that checked out? [20:28:52] oh he merged it: https://gerrit.wikimedia.org/r/#/c/276492/ [20:29:10] ottomata, ta [20:29:19] um, but yeah, this should work: https://meta.wikimedia.org/wiki/Config:SimpleRequestBreakdowns [20:29:32] so I can't su in but I can run commands? [20:29:45] milimetric: like "gulp --layout tabs --config SimpleRequestBreakdowns" [20:29:49] ? [20:29:58] yes, what's breaking? [20:30:04] and..analytics-search doesn't have a /home/ directory? [20:30:57] milimetric: data is look for at this url: http://datasets.wikimedia.org/limn-public-data/metrics/browser/mobile_site_by_browser_family.tsv [20:31:02] which is a 404 [20:31:58] milimetric: but wait maybe it needs ... [20:32:10] nuria: I see, I think he must have looked at that locally [20:32:27] run the scripts and copied the result somewhere [20:32:28] milimetric: i think it needs this one: https://gerrit.wikimedia.org/r/#/c/276763/ [20:32:36] yeah, to run in prod yea [20:32:56] nuria: what are you trying to look for, how it looks in general or? [20:33:08] milimetric: those scripts report raw viewcounts though [20:33:16] yes [20:33:42] milimetric:so percentages are calculated client side? [20:33:48] yes [20:34:04] otherwise any kind of pivoting or navigation client side would be impossible [20:34:43] k [20:35:01] then i guess the hive scripts are reday to be merged? [20:35:03] *reday [20:35:07] *ready [20:35:15] ^ milimetric [20:36:14] nuria: no [20:36:32] they'll be deployed right away and there's still a problem with the reportupdater that I commented [20:36:40] wait... [20:36:41] milimetric: aham.. i see [20:37:16] this one I want a comment I think: https://gerrit.wikimedia.org/r/#/c/276758/1/reportupdater/executor.py [20:37:30] but... I'll just add it and get it merged [20:37:45] milimetric: ok [20:40:09] (PS2) Milimetric: Add the query folder as the last parameter of scripts [analytics/reportupdater] - https://gerrit.wikimedia.org/r/276758 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [20:40:24] (CR) Milimetric: [C: 2 V: 2] Add the query folder as the last parameter of scripts [analytics/reportupdater] - https://gerrit.wikimedia.org/r/276758 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [20:40:39] (PS2) Milimetric: Make reportupdater support removing columns [analytics/reportupdater] - https://gerrit.wikimedia.org/r/277215 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [20:40:46] (CR) Milimetric: [V: 2] Make reportupdater support removing columns [analytics/reportupdater] - https://gerrit.wikimedia.org/r/277215 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [20:43:44] (CR) Milimetric: [C: 2 V: 2] Improve the browser queries [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/276763 (https://phabricator.wikimedia.org/T127326) (owner: Mforns) [20:44:24] k, nuria, the new job will be deployed automatically and that dashboard should be functional within a few hours (after it starts getting some data) [20:44:34] milimetric: ok, will check [20:45:19] Analytics-Kanban: Deploy new browser dashboard to 'browser-reports.wmflabs.org' - https://phabricator.wikimedia.org/T130069#2124345 (Nuria) [20:45:55] milimetric: ok, we need a new top domain and such, how did you provisioned a dns again? [20:47:56] nuria: I think we can just serve this on stats.wikimedia.org, right? [20:48:01] and do a manual deploy [20:48:20] like stats.wikimedia.org/dashboards/request-breakdowns or something [20:48:41] it'd be a bit weird to redirect to a totally different domain name [20:50:25] milimetric: mmm... that is going to be a bit of a pain to update as we would need to move code by hand there everytime [20:50:30] milimetric: why not a redirect? [20:50:56] I don't like cross-domain anything :) [20:51:15] wait.. it is a redirect, a 301 [20:51:21] not cross-domain [20:51:33] well, I mean it's redirecting from one domain to another [20:51:45] and I especially don't like it when things redirect from the prod cluster to labs [20:52:16] no technical reason or anything [20:52:21] ya, i would like to get away from having to modify wikistats code and rather delete what we no longer need [20:53:07] and deploying dashiki on labs or prod should be the same as it is only a client side app [20:53:10] this wouldn't modify wikistats code, it's just a matter of copying files to a specific directory. I think it can be done automatically with the fab deployer that madhu made, just a config to go to stat1001 instead of labs [20:53:47] milimetric: we do not have ssh-ing to 1001 [20:53:52] at least i do not [20:53:54] I do [20:54:09] and you should too, if we're going to work on wikistats it's pretty essential [20:55:21] we can talk about this tomorrow but I seems a lot easier to deploy dashboard in an easy to reach place [20:59:48] nuria: I'm not sure why stat1001 is not easy to reach... [20:59:59] it's not a windows machine :) [21:00:34] dashiki deployment is literally just 1. build, 2. copy over ssh [21:01:02] all I'm suggesting is that the target of 2. is stat1001 instead of dashiki.eqiad.wmflabs [21:08:58] Ironholds: sorrry [21:09:00] yeah no home dir [21:09:02] but put things in /a [21:09:18] ookay. [21:21:49] (PS3) Milimetric: [WIP] Add a legend to each graph in the tabs layout [analytics/dashiki] - https://gerrit.wikimedia.org/r/276649 (https://phabricator.wikimedia.org/T129497) [21:23:36] Ironholds: you guys ok? [21:23:44] just saw mikhail's email (what' his IRC?) [21:24:15] ottomata, we're probably fine, I'll handle it tomorrow [21:24:18] ok [21:27:03] ok, shodul I respond? [21:32:05] ottomata, well, where should I put things in /a/? [21:32:48] Ironholds: your choice i suppose [21:33:05] /a/search/research? /a/discovery/... [21:33:12] hmm [21:33:13] yeah [21:33:16] maybe /a/discovery [21:33:27] they're putting stuff in /wmf/discovery in hdfs [21:34:18] that works; thanks. [22:03:56] laters all! [22:12:26] Analytics, Operations, hardware-requests, Patch-For-Review: eqiad: (3) AQS replacement nodes - https://phabricator.wikimedia.org/T124947#1970805 (RobH) So we have a specification for this, it is actually our new spare pool specification on T128910. I've added this as dependent on that specificati... [22:15:42] milimetric: How does this sound? 1) let's deploy to browser-report.blah and 2) send erik z a link so he can see reports . Lets aks him what he wants to do with squid reports and if his preference is to report on the same domain we can do that [22:41:29] Analytics-Cluster, EventBus, Operations, Services: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#2124807 (RobH) [22:45:26] (PS1) Bearloga: Search engine detection bug fix [analytics/refinery/source] - https://gerrit.wikimedia.org/r/277679 [22:46:41] (CR) Bearloga: "recheck" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/277679 (owner: Bearloga) [22:53:10] (CR) OliverKeyes: Search engine detection bug fix (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/277679 (owner: Bearloga) [22:56:29] (PS2) Bearloga: Google detection bug fix [analytics/refinery/source] - https://gerrit.wikimedia.org/r/277679 [22:57:18] (CR) OliverKeyes: [C: 2 V: 2] Google detection bug fix [analytics/refinery/source] - https://gerrit.wikimedia.org/r/277679 (owner: Bearloga) [23:06:58] nuria: Erik's not going to care most likely, but if you insist that it should be on labs, ok, no problem [23:37:49] "ERROR 1030 (HY000): Got error 1152 "Unknown error 1152" from storage engine TokuDB" [23:38:01] ...while querying eventlogging data on analytics-store.eqiad.wmnet [23:38:05] not awesome :( [23:40:45] HaeB: what's your query? [23:41:23] (sent) [23:42:10] 101156 rows per "explain"