[00:19:18] (CR) Bmansurov: [C: 1] Integrated logging [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180828 (https://phabricator.wikimedia.org/T84892) (owner: Rtnpro) [08:08:23] Multimedia, Analytics: Set up varnish 204 beacon endpoint for virtual image views - https://phabricator.wikimedia.org/T89088#1027235 (Gilles) NEW [08:09:00] Multimedia, Analytics: Set up varnish 204 beacon endpoint for virtual image views - https://phabricator.wikimedia.org/T89088#1027235 (Gilles) [08:11:06] Multimedia, Analytics: Per-file view stats - https://phabricator.wikimedia.org/T77541#1027250 (Gilles) [08:11:49] Multimedia, Analytics: Set up varnish 204 beacon endpoint for virtual image views - https://phabricator.wikimedia.org/T89088#1027235 (Gilles) [10:12:42] Project-Creators, Engineering-Community, Phabricator: Analytics-Volunteering and Wikidata's Need-Volunteer tags - https://phabricator.wikimedia.org/T88266#1027437 (Qgil) In T78617 we have been discussing that the opposition between "volunteers" and the maintainers of a project (assumed to be non-volunteers) i... [10:14:30] (CR) QChris: [C: -1] [WiP] Add new UDF to determine client IP given source IP and XFF header (10 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [11:06:57] Analytics-Tech-community-metrics, Possible-Tech-Projects: Wikimedia Identities Editor - https://phabricator.wikimedia.org/T89111#1027636 (NiharikaKohli) NEW [11:23:02] Analytics-Tech-community-metrics, Possible-Tech-Projects: Wikimedia Identities Editor - https://phabricator.wikimedia.org/T89111#1027689 (Qgil) [11:23:04] Analytics-Tech-community-metrics: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1027690 (Qgil) [12:54:02] Analytics-Tech-community-metrics: "Contributors new and gone" in korma is stalled - https://phabricator.wikimedia.org/T88278#1027812 (Acs) Working on it now! Once fixed I will report about it here! [13:22:14] (CR) Ananthrk: [WiP] Add new UDF to determine client IP given source IP and XFF header (10 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [13:22:35] (PS6) Ananthrk: Add new UDF to determine client IP given source IP and XFF header [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 [13:36:57] (CR) QChris: Add new UDF to determine client IP given source IP and XFF header (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [13:38:07] (CR) QChris: [C: -1] "CR-1 due to the "UDF vs GenericUDF" discussion from Patch Set 5" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [13:47:29] (PS7) Ananthrk: Add new UDF to determine client IP given source IP and XFF header [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 [13:47:41] (CR) Ananthrk: "Made ClientIpUDF extend UDF rather than GenericUDF" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [14:06:56] Analytics-Tech-community-metrics, Possible-Tech-Projects: Support for vizGrimoireJS-lib widgets - https://phabricator.wikimedia.org/T89132#1027939 (Qgil) [14:09:54] Analytics-Tech-community-metrics, Possible-Tech-Projects: Support for vizGrimoireJS-lib widgets - https://phabricator.wikimedia.org/T89132#1027944 (Qgil) p:Triage>Low After our conversations in FOSDEM, I'm not sure whether it is a good idea to dedicate GSoC/OPW resources on vizGrimoire. Also, if this is... [14:11:15] (CR) QChris: "Yay, we're close!" (4 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [14:16:08] Analytics-Tech-community-metrics, Possible-Tech-Projects: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1027981 (Qgil) [14:18:50] (PS8) Ananthrk: Add new UDF to determine client IP given source IP and XFF header [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 [14:20:20] (CR) Ananthrk: Add new UDF to determine client IP given source IP and XFF header (4 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [14:23:13] Analytics-Tech-community-metrics, Possible-Tech-Projects: Improving MediaWikiAnalysis - https://phabricator.wikimedia.org/T89135#1027996 (Qgil) p:Triage>Low [14:23:52] Analytics-Tech-community-metrics, Possible-Tech-Projects: Improving MediaWikiAnalysis - https://phabricator.wikimedia.org/T89135#1027974 (Qgil) @Dicortazar, @Acs, are you still interested in proposing this project for GSoC / OPW? A new round is starting. [14:39:50] Analytics-Tech-community-metrics, Possible-Tech-Projects: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#622887 (Qgil) @Dicortazar, @Acs, are you still interested in proposing this project for GSoC / OPW? A new round is starting. [14:54:53] Analytics-General-or-Unknown, Possible-Tech-Projects: Pageviews for Wikiprojects and Task Forces in Languages other than English - https://phabricator.wikimedia.org/T56184#1028071 (Qgil) [15:28:04] Analytics-Tech-community-metrics: "Contributors new and gone" in korma is stalled - https://phabricator.wikimedia.org/T88278#1028135 (Acs) Open>Resolved Ok, now there is updated information for new and gone contributors: http://korma.wmflabs.org/browser/code_contrib_new_gone.html [16:03:16] Hi lads, newby just arrived :) [16:15:00] hi joal, (Dan here) [16:24:27] newbie++ :) [16:25:14] madhuvishy: hahaha [16:25:15] hi joal [16:25:18] hi joal [16:25:22] welcome Madhu :) [16:25:41] Hey Dan :) [16:25:52] hi madhuvishy [16:26:44] Hi mforns [16:30:19] Hi milimetric, hi forns, hi kevinator, hi Madhu :) [16:39:49] hey joal :) [16:51:56] joal, let us know if you need anything once you start requesting permits and such [16:53:41] Will do, thx Nuria [17:14:29] Project-Creators, Engineering-Community, Phabricator: Analytics-Volunteering and Wikidata's Need-Volunteer tags - https://phabricator.wikimedia.org/T88266#1028382 (Aklapper) >>! In T88266#1027437, @Qgil wrote: > So what about #Contributors-Welcome? Contributors are basically always welcome, except for when... [17:15:16] Project-Creators, Engineering-Community, Phabricator: Analytics-Volunteering and Wikidata's Need-Volunteer tags; "New contributors" vs "volunteers" terms - https://phabricator.wikimedia.org/T88266#1028383 (Aklapper) [17:20:25] Project-Creators, Engineering-Community, Phabricator: Analytics-Volunteering and Wikidata's Need-Volunteer tags; "New contributors" vs "volunteers" terms - https://phabricator.wikimedia.org/T88266#1028394 (Lydia_Pintscher) Those tasks are not just for new volunteers ;-) And I'd contest "contributors are alway... [17:26:07] * milimetric going for lunch [17:26:39] qchris: Hola. [17:26:43] qchris: looking at changes that went in I think this is the most likely one to have caused the issues we are seeing: [17:26:45] looking at changes that went in I think this is the most likely one to have caused the issues we are seeing: [17:26:46] Hi nuria. [17:26:57] http://git.wikimedia.org/blobdiff/mediawiki%2Fextensions%2FEventLogging.git/18b7693275ec6d693c6536f09293455191aa8e5a/server%2Feventlogging%2Futils.py [17:27:35] Is the above link correct? It shows a commit from 2014-11-23 for me. [17:27:39] qchris: but it is from the end of November [17:27:50] We had good deploys after that. [17:27:53] right, so .... was that deployed to the box ? [17:27:54] ok [17:28:23] As usual, you can walk SAL. [17:28:30] qchris: when was teh last time you deployed? [17:28:35] People are expected to log deployments there. [17:28:36] SAL? [17:28:46] https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:28:49] ^ = SAL [17:28:50] system-admin-log [17:28:53] ? [17:29:40] I mentioned that several times before. [17:30:18] qchris: ya, i looked at that yesterday [17:30:24] But let me grab a deploy from there. [17:30:35] qchris: and show a deployment from dec 18th [17:30:49] Right. [17:31:01] So 937d804 seems like a good commit. [17:33:17] 1sec [17:34:08] * joal will be back in a half hour or so [17:34:17] \q [17:34:20] oops :) [17:36:33] nuria, I suppose you postponed EL troubleshooting in hangout? [17:36:34] Between that and master, there are only 8 commits. /If/ it is the backend (not saying that it is) that should be ok to test. [17:37:21] qchris: I am going to revert in beta labs to that one 1st [17:38:03] nuria: Sure. [17:38:53] But are you convinced that it actually is a backend issue? [17:39:25] Meh. Sorry. [17:39:36] * qchris shuts up and tries to not get in the way. [17:39:39] qchris: seeing that validation is ok and that capsule changes are backwards compatible it seems that be is more likely than client [17:39:49] qchris: DO get in the way [17:41:23] mforns: well, i am trying to rollback to last known good version, will do that 1st on beta labs [17:43:20] nuria, can I watch? [17:43:48] mforns: sure, do you have permits to access deployment-eventlogging02 [17:44:03] mforns: or do you want to do it in teh batcave? [17:44:21] let me try to ssh [17:45:09] nuria, is that deployment-eventlogging02.eqiad.wmnet? [17:45:16] mforns: yes [17:45:28] no, no permits [17:45:35] batcave? [17:45:42] sure [18:00:27] qchris: is there a way (we) can look at tables created since february 5th? [18:00:41] qchris: i think other than those everything else will work if we revert [18:01:08] Not sure ... does mariadb hold meta information when a table got created? [18:02:44] qchris: i bet it does but not that i can see, ok, this one will be for sean [18:03:53] nuria: The TABLES table of the INFORMATION_SCHEMA database has a CREATE_TIME column. [18:06:50] show table status in log; [18:07:01] qchris: k [18:07:04] nuria: ^ does the trick, as it exposes the Create_time column. [18:07:42] qchris: ahhh [18:31:53] Hi again [18:32:09] Hi joal [18:59:24] milimetric: have you ever thought of using Jenkins as a scheduler for the EL -> Dashboard pipeline? [19:11:09] milimetric: ping me if/when you want to discuss the PV drop postmortem for SoS [19:25:26] kevinator: yes, but I should look at it again. I'm really bad with these systems that require mountains of text to configure [19:25:35] but it seems that people really trust and depend on jenkins [19:26:01] I was reading up on it, and one thing that concerns me is that it becomes a single point of failure [19:26:06] so it needs monitoring [19:26:14] our jenkins is already monitored [19:26:24] there are periodic problems but if we don't need to deal with them, that's cool [19:26:41] DarTar: no prob, I'm trying to help with EL right now, I'll ping after [19:26:59] milimetric: kk cool [19:27:00] good point… I was thinking of using our own Jenkins, but it’s better to use one already monitored [19:27:49] Quick question [19:28:06] Why to use jenkins if the only need is scheduling / [19:28:06] we're listening :) [19:28:22] oh, "scheduling" is a big term in this context [19:28:31] it means more like "job management" [19:28:40] like - rerunning older instances of a schedule [19:28:51] running missing or errored instances [19:29:08] having state to query about "did this job run reliably from time A to time B?" [19:29:49] and when you say job, are you talking hadoop? [19:30:09] well, right now we have a few separate schedulers that do different things [19:30:20] Right :) [19:30:27] one sec [19:31:06] My point ws, jenkins being a continuous integration server, why use it as a scheduler ... [19:31:32] oozie does hadoop jobs, cron does some "almost" ad-hoc sql and python, and wikimetrics uses celery and celery beat to do some standard sql metrics on a schedule [19:31:46] But I should read and understand more before making opinions ;) [19:31:48] in my opinion, ideally, we'd have one "scheduler" that managed all those things [19:32:10] yeah, but that's true of everyone, and old crusty opinions are not any better either :) [19:32:44] I like that when you make me feel young ! [19:32:49] :D [19:43:48] Analytics-Kanban, Analytics-Cluster: Mobile Apps PM has monthly report from oozie about apps uniques - https://phabricator.wikimedia.org/T88308#1028798 (kevinator) Spoke to Dan and they need this within a month. [19:49:30] joel, milimetric righttt, i do not see jenkins+ hadoop elsewhere maybe, sure [19:52:58] Analytics-Kanban, Analytics-EventLogging: Eventlogging should log timestamps for all error messages. - https://phabricator.wikimedia.org/T89162#1028819 (Nuria) NEW [20:06:38] milimetric: On Scheduling, have you heard of Azkaban ? http://azkaban.github.io/ [20:06:56] Should take care of at least the first two points above [20:13:28] qchris: reverted to last known deployment, but i really do not get how the code could be the problem, seems like it needs to be related to throughput rather than code changes. [20:15:00] nuria: If the code is not the problem and the issues persist, you ruled a one big chunk of unknowns and can focus on the next piece. [20:15:10] qchris: much agreed [20:27:26] joal: yes, Azkaban and Hudson have been brought up (https://lists.wikimedia.org/pipermail/analytics/2013-August/000847.html) but some people had problems with the lack of a CLI interface and the requirement to zip up all "job" files together to upload or something along those lines [20:27:34] so we never looked closely at those systems [20:27:49] but if you have experience or are curious, that'd be great to know more about them [20:32:10] Well, hmmm, my last knowledge of oozie was that it was full XML ... [20:34:51] Analytics-Visualization, Analytics-Kanban: Build high level timeseries view of key metrics - https://phabricator.wikimedia.org/T88367#1028919 (Milimetric) a:Milimetric [20:58:07] Analytics-EventLogging, Continuous-Integration: Set up jsduck test job for EventLogging - https://phabricator.wikimedia.org/T88343#1029008 (hashar) p:Triage>Volunteer? [21:45:02] Project-Creators, Engineering-Community, Phabricator: Analytics-Volunteering and Wikidata's Need-Volunteer tags; "New contributors" vs "volunteers" terms - https://phabricator.wikimedia.org/T88266#1029208 (Qgil) "Good-For-Volunteers" or "Ready-For-Volunteers"? I believe this is what Lydia and Kevin have in m... [21:49:20] Project-Creators, Engineering-Community, Phabricator: Analytics-Volunteering and Wikidata's Need-Volunteer tags; "New contributors" vs "volunteers" terms - https://phabricator.wikimedia.org/T88266#1029215 (Lydia_Pintscher) Yes! Those are both covering what I have in mind much better. [22:33:20] (CR) QChris: [C: 1 V: 2] "Only CR+1 instead of merging right away, as the new" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [23:10:22] how can I fix this query http://quarry.wmflabs.org/query/1939 ? [23:20:20] I'm not much with SQL or the mediawiki schema, but I'm trying to figure out how many eligible voters there are in the current stewards election. In #wikimedia-stewards-elections, huh has done some relevant queries, e.g. http://quarry.wmflabs.org/query/1939 (timed out) and http://quarry.wmflabs.org/query/1942 [23:20:45] see also https://en.wikipedia.org/wiki/Wikipedia_talk:Wikipedians#How_many_eligible_to_vote_here.3F [23:21:27] it would be nice to get some sort of handle on what our turnout looks like, since that is a common measure of election quality..... [23:25:40] nealmcb: I'll be busy for a while. When I come back I'll try to help some more (unless someone else answers your question first!) [23:25:57] BTW the query I gave only counts enwiki, which seemed much, much easier at the time [23:30:33] hi nealmcb and huh [23:31:37] mforns: howdy! [23:31:50] I've had a look at huh's queries, I thinks nothing is wrong with them [23:32:07] but the second one takes more than 10 mins so quarry kills it [23:32:33] I think it's normal though to take so long, it's a pretty intense query [23:32:47] So, how can I help you? [23:34:06] is there a way to run the query without the 10-minute limit? would that be too much to impose on the server? [23:35:34] I will try that on my side and let you know, ok? I don't know exactly how much time it is going to take though... [23:38:30] nealmcb, how can I contact you? [23:40:49] mforns: neal@bcn.boulder.co.us [23:41:34] nealmcb, ok, thanks, I'll let you know as soon as I can :] [23:41:51] mforns: many thanks! [23:42:13] thank you! [23:54:44] nealmcb, I tried a similar query, relaxing a bit the conditions and it finished in 8 minutes [23:55:01] I think it quite matches the definition you wrote in the wiki [23:55:32] I'll pass it to you so that you can run it in quarry [23:57:50] nuria, milimetric: my microphone has stopped working for some reason. can we do irc instead of hangout? [23:58:04] ok with me [23:58:30] springle: sure, sounds good. mforns is been doing the last vetting on the db so i will let him take it from there. [23:58:55] nuria, springle OK [23:59:15] just a sec [23:59:19] thanks [23:59:21] springle: i have a -completely- unrelated EL question but we can leave that for last [23:59:34] nealmcb, the query: http://pastebin.com/LXJ2Br5u