[00:07:07] Analytics, Analytics-EventLogging, Analytics-Kanban, operations, Patch-For-Review: Disk space full on vanadium from logs in /var/log/upstart - https://phabricator.wikimedia.org/T93185#1172575 (Dzahn) Open>Resolved a:Dzahn [00:33:55] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Utf-8 names on json reports appear as unicode code points: "\u0623\u0645\u064a\u0646" - https://phabricator.wikimedia.org/T93023#1172646 (Nuria) In this case is the right to left versus left to right what is tricking... [01:51:08] (PS1) Nuria: Reducing polling of report screen when not visible [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/201407 (https://phabricator.wikimedia.org/T94193) [01:52:51] (PS2) Nuria: Reducing polling of report screen when not visible [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/201407 (https://phabricator.wikimedia.org/T94193) [09:28:44] Hiya [13:10:15] Analytics-Wikimetrics, Community-Wikimetrics: Uploading cohort or running a large report fails - https://phabricator.wikimedia.org/T87596#1173584 (Aklapper) This task has "Unbreak now" priority since it was filed in January 2015 which [[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Prior... [13:28:58] Analytics-Cluster, Analytics-Kanban: Update Pageview UDF with dialect-specific directories {hawk} - https://phabricator.wikimedia.org/T92020#1173626 (JAllemandou) Discussed with Oliver: on each wiki listed in the [[ https://meta.wikimedia.org/wiki/Wikipedias_in_multiple_writing_systems#With_Automatic_Conv... [13:59:28] Analytics-Cluster, Analytics-Kanban: Update Pageview UDF with dialect-specific directories {hawk} - https://phabricator.wikimedia.org/T92020#1173672 (Ironholds) Why are we overcounting Chinese? [14:04:58] Analytics-Cluster, Analytics-Kanban: Update Pageview UDF with dialect-specific directories {hawk} - https://phabricator.wikimedia.org/T92020#1173675 (JAllemandou) I'd say so: we match zh(-(cn|hans|hant|hk|mo|my|sg|tw))? while, with the language drop-down we saw yesterday, I can only access zh(-(cn|hk|mo|s... [14:07:29] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Utf-8 names on json reports appear as unicode code points: "\u0623\u0645\u064a\u0646" - https://phabricator.wikimedia.org/T93023#1173679 (Nuria) Open>Resolved [14:07:42] (CR) Milimetric: Reducing polling of report screen when not visible (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/201407 (https://phabricator.wikimedia.org/T94193) (owner: Nuria) [14:12:14] Ironholds: what do you think about the last comment I made (overcounting chinese pv) ? [14:12:57] I replied to it! [14:13:07] already ? [14:13:08] oh, you commented again! [14:13:19] :) [14:13:22] Analytics-EventLogging, Analytics-Kanban, Wikimedia-Search: Estimate maximum throughput of Schema:Search (capacity) {oryx} - https://phabricator.wikimedia.org/T89019#1173697 (Nuria) Ticket for code changes as they pertain to the schema: https://phabricator.wikimedia.org/T90518 (*) https://gerrit.wikim... [14:14:21] Analytics-Cluster, Analytics-Kanban: Update Pageview UDF with dialect-specific directories {hawk} - https://phabricator.wikimedia.org/T92020#1173698 (Ironholds) Well, https://zh.wikipedia.org/zh-hans/ and https://zh.wikipedia.org/zh-hant/ - they exist, they're just presumably not common enough to be expli... [14:15:33] Analytics-Wikimetrics, Community-Wikimetrics: some non-ascii characters do not show up in uploaded usernames and result in invalid usernames - https://phabricator.wikimedia.org/T93646#1173702 (TFlanagan-WMF) I just got feedback from a user today that they couldn't upload their usernames in Arabic (assumi... [14:30:05] (CR) Milimetric: [C: 2 V: 2] Adding wikitext analysis to queries [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/200159 (owner: Milimetric) [14:53:39] (PS1) Ottomata: [WIP] spark streaming playtime [analytics/refinery/source] - https://gerrit.wikimedia.org/r/201474 [14:54:02] (PS2) Ottomata: [WIP] spark streaming playtime [analytics/refinery/source] - https://gerrit.wikimedia.org/r/201474 [14:54:23] (CR) Ottomata: [C: -2] "Just parking this here. This is not meant to be merged as is." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/201474 (owner: Ottomata) [15:07:33] ottomata, I have left unsolicited comments in your Scala [15:07:35] I am so helpful ;p [15:07:42] (CR) OliverKeyes: [WIP] spark streaming playtime (4 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/201474 (owner: Ottomata) [15:10:15] haha [15:10:49] haha, thank you Ironholds :p [15:11:16] :D [15:11:20] also, scala is BEAUTIFUL [15:11:32] it's like someone looked at Java and went "let's make this look less stupid", and then actually did [15:11:35] its really nice, ja? especially if you get good at it, then you can make it so much cleaner [15:11:43] it is really fun to play with too [15:11:48] if you tell me I can (a) avoid semicolons [15:12:28] and (b) the namespace system has been cleared up so I no longer have to import java.lang.hive.foo.bar.baz.another_namespace_because_static_all_the_things_sod_it.AbstractBeanConstructorValidatorFactory [15:12:33] I will start learning Scala [15:12:51] ha, uh, yes i believe that is true [15:12:59] graaaand [15:13:07] did you see marcel's patch? [15:13:12] it still needs cleanup, but it is nicer than mine [15:13:14] https://gerrit.wikimedia.org/r/#/c/199935/5/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/AppSessionMetrics.scala [15:14:27] I did! [15:14:30] it amused the shit out of me [15:15:30] the logic he's used is pretty much identical to my C++, absent language differences [15:15:33] great minds or something? [15:18:41] haha, awesome :) [15:20:29] (PS1) Yurik: Implemented clipping, 0-insertion, more reports [analytics/zero-sms] - https://gerrit.wikimedia.org/r/201478 [15:21:06] (CR) Yurik: [C: 2 V: 2] Implemented clipping, 0-insertion, more reports [analytics/zero-sms] - https://gerrit.wikimedia.org/r/201478 (owner: Yurik) [16:10:28] tnegrin: Do you know if there is really going to be a hangout for the "Quarterly Reports Workshop - Optional" meeting? [16:17:58] I was going to have us go to the Tuesday meeting [16:20:22] tnegrin: *nod* My prep meeting is tuesday so I guess I should have most of a deck before then [16:25:59] seems reasonable [16:26:31] I did talk to Terry and the meetings are open to everyone -- he had a very different expectation about how communications flow here [16:26:48] there's one small hitch about developers attending which I will clarify today [16:28:45] Let me guess, he thought that this was a top down hierarchical organization rather than a loose collective of anarchists and social freedom advocates? [16:40:27] Is that the new COO? [16:42:25] yes [16:44:07] Hm. That's concerning. [16:45:53] I honestly feel like it's a scale thing -- he's used to larger organizations where sending out an email to everyone isn't practical. When I talked to him about it, he's been very open about collaboration and open meetings and so forth. [16:46:38] as far as I can tell, his expectation was that the managers would organize their people but we just haven't rolled that way in the past. [17:26:45] Analytics-Tech-community-metrics: Connecting wikitech.wikimedia.org user profiles with community metrics - https://phabricator.wikimedia.org/T53050#1174683 (Krenair) Needs to support multiple affiliations [18:07:15] you guys notice recently that labs shells sometimes hang for a few seconds? [18:07:25] all my open shells to labs usually do it at the same time [18:19:30] (CR) Bmansurov: Reducing polling of report screen when not visible (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/201407 (https://phabricator.wikimedia.org/T94193) (owner: Nuria) [19:03:10] phew, finally working! milimetric, i got a kafka cluster in labs running [19:03:21] i'm producing raw json rsctream to it, as well as an avro version [19:03:41] using the kafka rest proxy and the schema registry [19:08:56] ottomata: that's really cool [19:09:18] how are you making the avro version? [19:09:25] consuming the raw one and producing it back? [19:11:47] mforns: yay I figured out how to do "sudo -u stats" all by myself [19:11:56] * milimetric high fives himself [19:12:01] milimetric, xD [19:12:21] milimetric, ok, I was writing an email to internal, because I'll be leaving now to go to the doctor [19:12:29] my left foot is bigger than usual [19:12:44] so I think I maybe have an infection, I dunno [19:13:44] mforns: that's bad, go get that fixed [19:14:00] milimetric, yea, see you tomorrow! [19:14:04] night! [19:14:06] night [19:15:51] (makin lunch) [19:22:44] milimetric: ? [19:22:49] no, that would be interesting [19:22:52] but no, i'm producing it via node [19:23:04] the rest proxy lets me produce the json data [19:23:08] and the rest proxy converts it to avro [19:23:20] i give it the schema_id to use, and the json record in the post body [19:27:34] milimetric: i'm eating lunch, but in a bit, wanna hangout and talk about this a bit? [19:44:52] milimetric: ehhHHh? [19:48:27] ah! sorry [19:48:30] pinging not working :( [19:48:46] ottomata: yes, I can hangout, anytime [19:48:56] in batcave now, cmon in! [20:21:16] 'night all ! [20:37:01] nite [20:46:18] Analytics-Tech-community-metrics, MediaWiki-Developer-Summit-2015, ECT-March-2015, developer-notice: Achievements, lessons learned, and data related with the MediaWiki Developer Summit 2015 - https://phabricator.wikimedia.org/T87514#1175522 (Rfarrand) Open>Resolved [20:57:01] Analytics-Cluster, Analytics-Kanban: Expand people's ability to use Hive/Cluster {hawk} - https://phabricator.wikimedia.org/T94903#1175573 (kevinator) NEW [21:32:01] Analytics-EventLogging, Analytics-Kanban, operations: Upgrade box for EventLogging (vanadium) - https://phabricator.wikimedia.org/T90363#1175784 (yuvipanda) ``` PROBLEM - RAID on vanadium is CRITICAL: CRITICAL: Active: 4, Working: 4, Failed: 2, Spare: 0 ``` Just happened, so this box needs to be repla... [21:35:57] Analytics-EventLogging, Analytics-Kanban, operations: Upgrade box for EventLogging (vanadium) - https://phabricator.wikimedia.org/T90363#1175804 (yuvipanda) p:High>Unbreak! since this is a SPOF for EventLogging, and plenty of people will be sadface if eventlogging dies. [22:18:10] Analytics, Multimedia, Multimedia-Sprint-2015-03-25, Patch-For-Review: Measure how many users have CORS-hostile proxies - https://phabricator.wikimedia.org/T507#1175978 (Tgr) ``` mysql:research@analytics-store.eqiad.wmnet [log]> select sum(event_xhrSupported and event_scriptAttributeSupported and !e... [22:38:51] nuria: https://phabricator.wikimedia.org/T90363#1175784 [22:39:08] YuviPandaa: looking [22:39:35] YuviPandaa: ah yes, ori said he had two boxes provisiones (1 to use 1 to spare) [22:39:40] *provisioned [22:39:47] ori, yt? [22:41:48] Ironholds: Minor disaster with the apps right now. Account creation is totally broken in production. Might not be able to make our meeting. [22:45:56] Deskana, kk! [22:59:28] Ironholds: Issue still ongoing, I have to stick with it. [23:00:04] Deskana, okie-dokes! LMK if it clears up in the next hour, otherwise we can reschedule :) [23:00:19] dbrant might be there to chat to you, too [23:02:07] * Ironholds nods [23:02:17] should we just reschedule? Fewer things for you to juggle while resolving :) [23:02:32] Ironholds: Sure, if you can find a spot on my calendar! [23:02:33] oop, we have a dmitry [23:03:17] Ironholds: Actually, it seems like they don't need me right now, so I'll come on. [23:03:30] nuria: I looked at EL rates when vanadium lost those drives and I looked again just now [23:03:40] the rates seem normal so it doesn't look like it's dying [23:03:46] but we should get on it as soon as we can [23:04:22] milimetric: ya, YuviPandaa caught that too, but we cannot change the box ourselves as we do not have permits, someone from ops needs to do it [23:05:32] yeah, of course [23:05:49] I just meant - in case that makes you feel better about waiting until tomorrow :P [23:05:53] milimetric: haha [23:06:31] milimetric: i was gone from 1 to 3 so didn't see the updates until a bit ago, i will check EL db counts vs log counts [23:07:21] nuria: the valid vs invalid rate is almost identical in graphite, so if the db vs. log are different then we have a problem [23:08:42] milimetric: that is a good sign but db inserts do not get to graphite and they are "sent" from vanadium so will check those too in a bit [23:08:45] just remember that if another disk dies then basically EL will be dead. [23:09:17] instead of the machine being a SPOF it's now an individual disk in an array that's just had failures [23:09:20] yep, we knew this was a risk, so this is on us if it dies [23:09:35] cool :) [23:09:59] I mean, we have asked for ops resources to swap it for about two months [23:10:05] but still - we should've asked harder [23:11:33] YuviPandaa: as we can do nothing to change that box, i respectfully disagree that is on us [23:12:17] telling me isn't going to change much, nuria :) That ticket didn't even have operations project added so nobody in ops even saw it. [23:12:36] YuviPandaa: ya, you are right. [23:12:43] well, for some definition of 'ops', otto was on it... [23:13:02] well, to be fair, otto did bring it up at a few ops meetings if I remember correctly [23:13:11] nuria: anyway, I just saw an alert for vanadium and poked you guys, so don't shoot the messenger :) [23:13:12] and Toby brought it up with Damon several times [23:13:18] nono, no shooting [23:13:26] that's why I was assuming responsibility [23:13:34] nuria - Yuvi didn't say it was on us, I said that [23:14:16] we thought we did everything possible to get heard, but I think considering the severity we could've done more [23:14:40] YuviPandaa: you are right to poke us, i do not even see those vanadium alarms (which kind of makes sense as i cannot do much about them) [23:15:16] :D Let me know if there's anything I can do to help, but I bet otto's going to be able to get it all covered. [23:15:32] I suspect in the 'short term' we can basically replace it with a server spare, and then order a new machine maybe... [23:15:40] or ori already has it covered. not sure [23:16:25] last I heard we already had a box we could use [23:17:30] right [23:22:11] Analytics-Cluster, Analytics-Kanban: HDFS Blancing, productionize - https://phabricator.wikimedia.org/T94933#1176267 (Aklapper) [23:26:48] (PS3) Nuria: Reducing polling of report screen when not visible [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/201407 (https://phabricator.wikimedia.org/T94193) [23:27:49] YuviPanda: let's see if we can have the vanadium stuff resolved by tomorrow [23:28:43] milimetric: please take a look: https://gerrit.wikimedia.org/r/#/c/201407/ I removed the changes for visibility API out of default namespace. [23:30:43] (CR) Nuria: Reducing polling of report screen when not visible (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/201407 (https://phabricator.wikimedia.org/T94193) (owner: Nuria) [23:35:18] (CR) Milimetric: [C: 2] Reducing polling of report screen when not visible [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/201407 (https://phabricator.wikimedia.org/T94193) (owner: Nuria) [23:39:24] YuviPanda: I cannot even ssh into vanadium [23:39:31] oh? [23:39:33] just now or in general? [23:39:35] YuviPanda: can you? [23:39:40] YuviPanda: no, just now [23:39:41] trying [23:39:46] yup I can [23:40:08] YuviPanda: so my permits got revoked? [23:40:40] nuria: try again? I’m watching the ssh logs [23:40:54] ah no, now it worked.... [23:40:58] #scary... [23:41:05] or #scared [23:41:09] :) [23:41:19] YuviPanda: ok, let me poke arround to see state of affairs [23:41:31] nuria: cool [23:42:30] Analytics-Cluster, Ops-Access-Requests, operations: Requesting access to analytics-users (stat1002) for Jkatz - https://phabricator.wikimedia.org/T94939#1176345 (JKatzWMF) NEW a:Ottomata