[00:00:00] qchris: it seems i see this (bogus) error in the logs a bunch: java.io.FileNotFoundException: File file:/srv/deployment/analytics/refinery/artifacts/refinery-hive.jar , do we know what causes that? [00:00:36] Are you using the hive-site.xml from stat1002 to get oozie jobs running? [00:01:06] Or where/how are you getting that message? [00:01:57] nuria / mforns: in case you were on the edge of your seats: [00:02:23] the reason EL qunit tests are failing is because they use mw.format and that apparently just got merged 6 days ago by Ori [00:02:24] i imagine i am using the one on etc as i did not setup any other ^qchris [00:02:36] it's not available yet in the version of mediawiki we use in vagrant [00:02:47] milimetric: :-D [00:03:25] nuria: Like /etc/hive/conf/hive-site.xml from stat1002? [00:03:34] qchris: yes [00:03:52] k. [00:04:13] Then create a new one, where you strip the [00:04:22] ,file:///srv/deployment/analytics/refinery/artifacts/refinery-hive.jar [00:04:29] from the value of hive.aux.jars.path [00:04:31] milimetric: so how come.. ah, jenkins is running a dif version of mw then [00:04:47] Alternatively, you can use some of the deployed hive-site.xmls from hdfs. [00:05:19] Like: [00:05:23] hdfs:///wmf/refinery/current/oozie/util/hive/hive-site.xml [00:06:30] qchris: so i should remove references to jar entirely, ok, i see, i will grab that hive site [00:06:54] Using that one from hdfs is the simpler way. [00:07:13] milimetric: you did update your mw checkout right? [00:07:43] nuria: If you are only using that for Oozie jobs you want to productionize, you can add [00:07:45] hive_site_xml = ${oozie_directory}/util/hive/hive-site.xml [00:07:50] to your properties file. [00:08:00] nuria: not sure about jenkins but if vagrant is using mw 1.25alpha and mw.format is not available until 1.25 proper. So I just did git pull in the mediawiki directory itself and everything works ok [00:08:06] like the other jobs do. [00:08:23] i hadn't updated the mw checkout, no, is that common knowledge or documented? [00:08:32] If so, I missed it and cost myself a lot of time :( [00:09:13] milimetric: thanks, i'll cherry-pick it [00:09:31] i totally forgot about that [00:09:42] no prob, not your fault, it was just the first time I debugged anything there so I was uncertain [00:10:10] totally *is* my fault:P [00:32:27] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Collect more data in MediaViewer network performance logging - https://phabricator.wikimedia.org/T86609#972548 (Tgr) NEW [00:36:20] (PS1) Nuria: Escaping leading 0. Adding empty done flag. [analytics/refinery] - https://gerrit.wikimedia.org/r/184531 [00:51:33] nuria: when I try to run tox in EventLogging/server inside vagrant, it blows up while installing dependencies [00:51:43] do you give it more memory or run it outside of vagrant? [00:52:19] no, i run it inside, but my vagrant has a ton of memory that i needed to install the hadoop role [00:52:35] milimetric: so it might be benefiting from that [00:52:45] gotcha, thanks [01:41:20] (PS15) Nuria: Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 [01:59:47] (PS16) Nuria: Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 [02:05:28] (CR) Nuria: ">generate_daily_uniques.hql was a good filename" [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 (owner: Nuria) [02:06:52] (CR) Nuria: Mobile apps oozie jobs (5 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 (owner: Nuria) [04:29:03] Analytics-Cluster: Write Success Flags for refine-webrequest.* Oozie Coordinator when partition is created - https://phabricator.wikimedia.org/T86615#972746 (ellery) NEW [06:56:24] (CR) Bmansurov: [C: 1] Update scripts in light of recent changes [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/181428 (owner: Jdlrobson) [07:20:34] Wikimedia-Git-or-Gerrit, Analytics-Tech-community-metrics: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#972811 (Qgil) It's not that we have dozens of types of affiliation. In order to understand "Independent" we need to see the magnitude of "Unknown", bec... [08:11:46] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#972901 (Nemo_bis) That's not quite a detail. The WMF people doing code review for non-WMF people are typically those who were active in MediaWiki befor... [08:13:28] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#972907 (ori) >>! In T59038#972901, @Nemo_bis wrote: > That's not quite a detail. The WMF people doing code review for non-WMF people are typically thos... [09:01:43] Wikimedia-Git-or-Gerrit, Analytics-Tech-community-metrics: Metrics about contributors with +2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#972978 (Qgil) Nemo, I don't understand your arguments in this task. First you say that it is not worth analyzing data by affiliation. Then you make a s... [09:26:13] Wikimedia-Git-or-Gerrit, Analytics-Tech-community-metrics: Basic metrics about contributors exercising +2/-2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#973027 (Qgil) [09:27:13] Wikimedia-Git-or-Gerrit, Analytics-Tech-community-metrics: Basic metrics about contributors exercising +2/-2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#641085 (Qgil) I have edited the description proposing the data we want to gather in this task. Suggestions for improvement are welcome. [09:36:45] Analytics-Tech-community-metrics: Consolidating time ranges across tech community metrics - https://phabricator.wikimedia.org/T86630#973043 (Qgil) NEW [09:37:21] Wikimedia-Git-or-Gerrit, Analytics-Tech-community-metrics: Basic metrics about contributors exercising +2/-2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#973049 (Qgil) [09:38:19] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#973051 (Qgil) I would like to propose some changes in the data ranges used in this report, but before let's agree on {T86630}. You opinions are wel... [12:04:44] Analytics-Cluster: Write Success Flags for hive_webrequest.* Oozie Coordinator when partition is created - https://phabricator.wikimedia.org/T86616#973230 (Aklapper) [14:18:52] operations, ops-core, Analytics: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#973506 (faidon) NEW a:faidon [14:21:18] Analytics-Cluster: Raw text webrequest partition for 2015-01-12T20/1H not marked successful - https://phabricator.wikimedia.org/T86657#973522 (QChris) NEW [14:21:32] Analytics-Cluster: Raw webrequest partitions that were not marked successful - https://phabricator.wikimedia.org/T72085#973531 (QChris) [14:21:34] Analytics-Cluster: Raw text webrequest partition for 2015-01-12T20/1H not marked successful - https://phabricator.wikimedia.org/T86657#973522 (QChris) Open>Resolved a:QChris cp1068 had 92 duplicates in 2015-01-12T20:07:07 -- 2015-01-12T20:07:08. I deduped the partition, and the partition is good now. [14:56:13] milimetric: Guess who is stuck again in "Trying to join the call. Please wait ...". [14:56:22] k, trying [14:56:24] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Collect more data in MediaViewer network performance logging - https://phabricator.wikimedia.org/T86609#973685 (Gilles) [14:56:53] i removed and added you [14:57:22] Ok. [14:57:24] * qchris retries. [14:58:20] Did not help :-/ [14:58:49] Now I am in. [14:58:55] Thanks milimetric! [15:04:21] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Find a robust way of filtering local cache hits out of performance figures - https://phabricator.wikimedia.org/T86672#973711 (Gilles) NEW [15:11:31] Analytics-Wikistats: tyv.wikipedia.org is missing - https://phabricator.wikimedia.org/T57605#973733 (ezachte) Open>Resolved a:ezachte Wikistats reports for tyv are currently available, see e.g. http://stats.wikimedia.org/EN/SummaryTYV.htm http://stats.wikimedia.org/EN/TablesWikipediaTYV.htm [15:36:49] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Fix filter differentiating between varnish hits and misses in performance queries - https://phabricator.wikimedia.org/T86675#973778 (Gilles) NEW a:Gilles [16:20:03] (PS1) Gilles: Fix filter differentiating between varnish hits and misses [analytics/multimedia] - https://gerrit.wikimedia.org/r/184646 (https://phabricator.wikimedia.org/T86675) [16:22:35] ottomata1, this is your daily oliver-being-pestery poke to look at the legacy UDF :D [16:22:57] wha, i thought i was caught up on that! [16:23:16] oh suppose not! [16:23:21] you patched day of! [16:23:33] ok, um, lots of meetings today, and christian wants me to deploy something, but i will keep you on my mind [16:24:24] ottomata, thankee :) [16:24:27] and yeah, I patch endlessly [16:24:32] I want that on a jacket patch, actually [16:24:38] This Machine Kills Code Reviews [16:33:45] milimetric, just updated http://etherpad.wikimedia.org/p/diff_test [16:33:52] * milimetric looks [16:33:58] Last ran worked without issue. Still processing at 30 revs/second [16:35:16] huh [16:35:18] interesting [16:36:35] milimetric, I was considering restarting without the speculative param change to see if we can get the error to appear again. [16:36:49] I'm worried that it is intermittent and we fixed nothing. [16:36:52] Thoughts? [16:38:00] halfak: I agree, Christian suggested the same thing [16:38:02] though [16:38:09] before you do, we should study the job the same way Christian did [16:38:24] and see how the jobs interacted with the temporary directories [16:38:32] and look at all the same details he looked at [16:38:38] Seems like we should be able to do both. [16:38:39] sadly - I don't have any time :( [16:38:51] I don't believe that the logs get deleted. [16:38:52] really sorry, this sprint is looking bad [16:39:01] no worries milimetric [16:51:31] nuria mforns milimetric : I created a deck for this showcase [16:51:32] https://docs.google.com/presentation/d/14eAWBLZFqMAC9qdqg7WS3-OseC0E-LeXENNc6yOEMUA/edit?usp=sharing [16:51:45] ok kevinator [16:51:59] I think we can have one person showcase wikimetrics, and one for Vital Signs [16:52:09] and I’ll leave up to you guys to decide who [16:52:11] :-) [16:52:22] yes, ok [16:55:23] yall want a webrequest_refined demo? [16:56:15] (Abandoned) QChris: Add Oozie legacy_tsv job for landingpages TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184490 (owner: QChris) [16:56:30] ottomata: Talking about perf results is a MUST [16:56:33] (Abandoned) QChris: Add Oozie legacy_tsv job for bannerImpressions TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184491 (owner: QChris) [16:56:42] (Abandoned) QChris: Add Oozie legacy_tsv job for bannerRequests TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184492 (owner: QChris) [16:56:58] ottomata: i run the daily queries yesterday for apps in minutes rather than hours [16:58:39] kevinator: we have to show two other things 1) perf improvementes and refined datasets by ottomata 2)sendbeacon changes to EL [16:58:43] do they have phabricator tasks? [16:58:54] I’m happy showing those 2 things as well [16:59:21] kevinator: no, tehy do not [16:59:44] *the send beacon has a phab task yes, the perf improvements not [16:59:51] kevinator: [17:00:18] ok [17:01:44] I see the sendbeacon task: https://phabricator.wikimedia.org/T78110 [17:01:49] (CR) Ottomata: "Can you copy paste Christian's comment about how this works from the pagecounts-all-sites datasets.xml file? It is pretty confusing when " [analytics/refinery] - https://gerrit.wikimedia.org/r/184531 (owner: Nuria) [17:02:32] nuria: we could retroactively add this task to that sprint so our velocity get a 5 point increase :-) [17:02:50] kevinator: it was done a month ago though [17:02:58] kevinator: that would be cheating [17:03:35] yeah, I mean adding it to the last sprint of 2014. and yes, won’t do it. Next time… [17:03:58] next time we’ll add it to the sprint on time. [17:12:27] nuria, so, you are setting the mobile apps archive up so that there is one file per directory? [17:12:54] ottomata: yes, unless that is terrible, there will be 1 file per dir per day [17:13:38] ottomata: like /mobile_apps/uniques/2015/1/1 [17:13:40] i guess: why? all the other 'archive' things we do have many files in one directory, that way its easy to see what is available via asimple ls [17:14:10] ottomata: but do let me know if this is not acceptable, before i had them by hour to run tests faster [17:14:11] (and executed shorter jobs) [17:14:24] aye, i mean, haha, 'acceptable' :p [17:14:47] it seems easier for the end user if there are multiple files ina directory, and that is how we do it in other jobs [17:14:57] if there is a particular reason to do it here, sure, but you gotta have the reason :) [17:16:10] also, i really think that $archive_directory should point directly to /wmf/data/archive, not /wmf/data/archive/mobile_apps [17:16:21] ottomata: then, how do you do for naming schemes? naming them with the date? [17:16:23] other jobs have this variable pointing there [17:17:17] nuria: https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/legacy_tsvs/workflow.xml#L125 [17:17:38] you do'nt need to stick with that naming convention, but you can see how the filename is created [17:18:11] ottomata: ok, will change those two [17:19:49] (CR) Ottomata: Mobile apps oozie jobs (4 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 (owner: Nuria) [17:20:58] (CR) Ottomata: "We just talked in IRC about changing:" [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 (owner: Nuria) [17:21:09] nuria, i think whether or not you do per month directories or not is up to you [17:21:14] i'm fine either way [17:21:29] ottomata: i will change it to follow the convention [17:21:34] hm, also, nuria, maybe 'daily' should be in the path too? [17:21:44] i've seen that before, camus does this by default too [17:22:05] in case someone wants the daily files aggregated per week or month or something too later (maybe using q c hris' aggregator jobs, dunno) [17:22:49] $archive_directory/mobile_apps/uniques/daily/$year/$month/mobile_app_uniques-$year-$month-$day.gz or whatever [17:23:01] if you like [17:25:34] (CR) Nuria: [C: 2] Check ownership before adding tag to cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/182391 (https://phabricator.wikimedia.org/T68483) (owner: Bmansurov) [17:27:24] (CR) Ottomata: [C: 2 V: 2] Add Oozie legacy_tsv job for api-usage TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184493 (owner: QChris) [17:27:46] qchris_away: your tsv patch current depends on an abandoned patch. I will let you resolve :) [17:28:24] (CR) Ottomata: "Should we wait until geocode udf is available before we do this then?" [analytics/refinery] - https://gerrit.wikimedia.org/r/184494 (owner: QChris) [17:29:19] (CR) Ottomata: [C: 2 V: 2] Add Oozie legacy_tsv job for edits TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184495 (owner: QChris) [17:30:42] (CR) Ottomata: "Since we won't be able to use this data until bits and upload are cool, let's wait on this, eh?" [analytics/refinery] - https://gerrit.wikimedia.org/r/184496 (owner: QChris) [17:35:25] (PS2) QChris: Add Oozie legacy_tsv job for 5xx TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184496 [17:35:28] (PS2) QChris: Add Oozie legacy_tsv job for api-usage TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184493 [17:35:29] (PS2) QChris: Add Oozie legacy_tsv job for glam_nara TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184494 [17:35:31] (PS2) QChris: Add Oozie legacy_tsv job for edits TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184495 [17:36:40] (CR) Ottomata: "FIX YOUR TABS! >:O :)" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/182971 (owner: OliverKeyes) [17:37:04] (CR) Ottomata: Legacy pageviews definition UDF (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/182971 (owner: OliverKeyes) [17:37:32] ottomata, thanks for the review [17:37:38] will address shortly (writing unit tests for something) [17:39:10] ottomata: the abandoned ones were the fundraising tsvs that seems are no longer needed. I rebased the changes. [17:39:20] But I am not sure about the changes with TODOs. [17:39:33] I'd argue that it's better to merge them with the TODOs, [17:39:44] as then we'd already now see how things behave. [17:40:15] And things like "perfect upload partition" will take some more time. [17:40:24] So we'd otherwise be blocked on that. [17:42:12] (CR) QChris: "> Should we wait until geocode udf is available before we do this then?" [analytics/refinery] - https://gerrit.wikimedia.org/r/184494 (owner: QChris) [17:44:32] (CR) QChris: "> Since we won't be able to use this data until bits and upload are" [analytics/refinery] - https://gerrit.wikimedia.org/r/184496 (owner: QChris) [17:47:27] (PS2) Nuria: Escaping leading 0. Adding empty done flag. [analytics/refinery] - https://gerrit.wikimedia.org/r/184531 [17:48:20] (CR) Nuria: "> Can you copy paste Christian's comment" [analytics/refinery] - https://gerrit.wikimedia.org/r/184531 (owner: Nuria) [17:55:45] !seen ottomata [17:56:03] anyone else who is familiar with the different analytics access groups for shell access request [17:58:21] Analytics-Cluster: Write Success Flags for refine-webrequest.* Oozie Coordinator when partition is created - https://phabricator.wikimedia.org/T86615#974037 (ellery) Open>Invalid [18:11:08] milimetric, nuria: do you want to showcase anything special? have you already spoken about this? [18:11:34] mutante: what was the question? [18:12:12] mforns: no, we can talk briefly on batcave if you want. I was planning on talking a bit about sendbeacon and that's it. cc milimetric [18:12:31] no preferences guys [18:12:43] i'm just struggling with this validation stuff and have no brainpower for demos [18:12:52] if you want me to demo something, I'm happy to, just let me know [18:13:07] milimetric, do you want to show annotations? [18:13:27] milimetric, nuria: I can do wikimetrics [18:13:58] milimetric, I mean Vital Signs [18:14:35] mforns: sure I can show annotations [18:14:41] (CR) Ananthrk: [WIP] UDF to get country code from IP address UDF to determine client IP address given values from remote_addr and XFF headers Change-Id: I6 (17 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 (owner: Ananthrk) [18:16:02] milimetric: that's fine. I can do dashiki and sendbeacon , if mforns does wikimetrics we are good [18:16:50] milimetric, nuria: sure I'll do Wikimetrics, and if you want I can do Dashiki too [18:17:57] mforns: it's fine, the dashiki and send beacon part should be pretty small [18:18:13] ok nuria, I take Wikimetrics then [18:29:32] ori: well, the request says " simple queries against the EventLogging database containing the pageview data of the Wikimedia blog" and ottomata said for that we don't even need those privatedata groups [18:29:58] i know, but like, the guy edited (founded?) the wikimedia research newsletter [18:30:02] i was going to change that to just put him in a bastion group and on stat1003 as otto said [18:30:15] and he's now one of the senior people in product/strategy [18:30:26] IMO let's just clear his path [18:31:05] i like to err on the side of having people with advanced math degrees from cambridge and a history of producing useful research having access to *more* data than they strictly need right this moment [18:31:14] but up to you [18:31:58] it's not about who he is, it's that he never said he needs anything else. and usually you go by the minimum permissions needed [18:32:42] ok, it's certainly not the end of the world to add him to more groups in the future [18:32:46] so fine by me [18:33:52] (PS2) Ananthrk: [WIP] UDF to get country code from IP address UDF to determine client IP address given values from source IP address and XFF headers Added IntelliJ related files to .gitignore Split existing Geo UDF into two - GeoCodedCountryUDF and GeoCodedDataUDF Both U [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 [18:34:01] it's the admin in me who wants http://en.wikipedia.org/wiki/Principle_of_least_privilege .. [18:34:11] always fine to add more groups when needed.of course [18:36:04] there's a nice post from domas on wikitech-l from a few years ago [18:36:37] about how if tim had not been bold and entrusted him and some other early volunteers with the right and responsibility to care for some essential services we would have never gotten anywhere [18:36:41] i can't dig it up right now [18:37:21] but generally i think what made wikimedia successful is a history of assuming good faith and giving people more trust than the minimum required. of course we shouldn't compromise the security of the cluster, etc. [18:37:37] anwyays, i don't want you to feel that this is a bigger deal than it is, tilman himself would probably tell me to shut up by now :P [18:37:46] i think it's fine to just restrict it to stat1003 for now [18:45:32] ori: i took the task of "allow null to be passed in a boolean field" on and I am realizing it's kind of problematic [18:45:47] where's the task? [18:46:05] firstly, it seems people would like "null" to be passable in string fields too. task is at: https://phabricator.wikimedia.org/T78325 [18:46:21] and the linked problem talks about strings: https://phabricator.wikimedia.org/T85963 [18:46:42] so I'm wondering if you gave this any thought [18:46:52] i think it's a bad idea [18:46:55] yea [18:46:58] that's what i figured [18:46:59] :) [18:48:06] so when people need such a thing as a nullable boolean, they could just use an enum? [18:48:21] a nullable boolean should just not be set [18:48:31] if the property is not required, it's null in the database [18:49:07] it may seem fussy to be strict when there is a https://en.wikipedia.org/wiki/DWIM - style solution that seems obvious [18:49:08] supposedly validation fails somewhere along the line, so I'm still thinking about that [18:49:23] but really the cost of making analysts work with shitty data that was improperly logged is so much greater [18:49:38] it's worthwhile to make devs sweat a tiny bit more to ensure correctness [18:50:00] in fact you may want to ask halfak or DarTar to weigh in [18:50:06] since i suspect they're of the same opinion [18:50:14] or lzia [18:50:38] * ori wasn't sure what leila's irc nick was [18:50:55] ok, hm [18:51:05] hey both, I haven’t read the thread on event validation yet [18:51:05] yo [18:51:06] yes, ori. catching up on the conversation [18:51:15] sorry, mailing list meltdown [18:51:21] kaldari is wants license to be sloppy and i don't agree [18:51:38] i'm not sure if it's license to be sloppy [18:51:43] there seems to be a problem with validation: https://phabricator.wikimedia.org/T85963 [18:51:49] which I couldn't reproduce yet [18:52:11] I think validation + access to the raw logs (including those failing validation) as discussed would be the best solution [18:52:21] but again, I should read the thread first [18:52:27] a value set to null is not the same as a value being absent [18:52:42] so, "required" would mean that "you can have the field", "you and not include the field" or "you can set the field to null". [18:52:44] if the key is present, it has to be a string; if it's not required, then it can be absent, but it can't be present and null [18:52:55] Sorry "required: false" [18:53:17] That doesn't sound like "required". [18:53:29] It sounds like "nullable" or something like that [18:53:33] right [18:53:38] present and null is being requested [18:53:46] i'm almost in agreement with ori but wasn't sure [18:54:02] i'm looking for the relevant part of the json schema spec [18:54:05] I don't really have a strong opinion on this one. [18:54:10] DarTar, I wonder if there is an easier way for doing this. having to dig raw logs make the QA very hard. [18:54:24] as far as jsonschema is concerned it looks like they expect you to say type: ["string", "null"] [18:54:29] to do what's being requested here [18:54:34] But I side slightly with the having "required" mean what it says. [18:54:37] leila: it’s a stopgap measure: way better than the current workflow [18:54:59] leila, are you using jq to look in the logs? [18:55:06] leila: are you talking about the eventlogging logs? [18:55:07] ultimately, we will want per schema monitoring, automated anomaly reporting and fuzzy event validation [18:55:20] ori, yes, eventloggings logs [18:55:27] "fuzzy event validation"? [18:55:29] yay, jq – best thing since sliced bread [18:55:35] milimetric: there is a facility for having nullable JSON and that's union types, see https://tools.ietf.org/html/draft-zyp-json-schema-03#section-5.1 , subsection 'null' [18:55:37] halfak: I’m just talking nonsense :) [18:55:44] :p [18:55:54] but absent and null are two different things [18:56:08] ori +1 [18:56:20] (CR) OliverKeyes: [WIP] UDF to get country code from IP address UDF to determine client IP address given values from source IP address and XFF headers Added I (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 (owner: Ananthrk) [18:56:27] It's only out of necessity that we equate them when inserting in the DB. [18:56:28] ori: right, union types is what I was saying above, type: ["string", "null"] [18:56:30] halfak, in my case the choice of jq didn't help much [18:56:37] if we were working in raw JSON, we wouldn't abide nulls. [18:56:38] do you think we should enable union types? [18:56:47] leila: i have an idea on how to make it more accessible, i'll have to corner milimetric / nuria for some dev work :P [18:57:00] I need to make a complete funnel for WikiGrok QA and having to look at raw files and the database at the same time make things harder [18:57:11] halfak, ^ [18:57:25] Why do you need to look at the raw files? [18:57:27] milimetric: we could, but it would take additional work to implement, and i bet you anything that kaldari wouldn't use that feature, because what he really wants is permission to be sloppy, and union types are even more elaborate [18:57:34] ahem , /nuria doesn't know what we are talking about but she is going to find out [18:57:54] ori wonders if nuria is talking about herself in the third person now [18:58:06] * halfak talks in 3rd person all the time [18:58:09] just like the pope ori [18:58:18] you should speak latin for the full effect [18:58:27] * ori has to interview someone, bbiab [18:59:14] guys I need to go back to WikiGrok QA. [18:59:22] later [19:31:35] kevinator: https://phabricator.wikimedia.org/T85963#974294 [19:31:42] :( [19:31:48] sorry i was so slow to figure this out [19:32:08] i think i need to get emails under control so it's not so distracting [19:32:18] milimetric: it sounds like we are not going to implement the 'allow null' ticket right? [19:34:13] nuria: I don't know, I can still try to do it, but the code I can think of would be kind of ugly and the reasoning to not do it seems stronger than the reasoning to do it [19:34:40] milimetric: agreed, let's just document why we didi not do it and tell kaldari as of his options [19:35:03] well, for now my comment just says I'm pausing and I explained some of the reasoning [19:35:17] but I'm happy to go further into detail, I just wondered what kevinator thought [19:35:33] anyway, that being on hold, nuria or mforns: can I help with anything? [19:35:37] are there reviews or work? [19:36:48] milimetric: just a sec… let me finish something then I’ll look at the comment in phab [19:36:58] milimetric: i just finished all the reviews ones i had ongoing, Iam not sure if mforns is working further on the eventlogging devserver [19:37:25] milimetric: next task will be validating teh warehouse [19:37:34] milimetric: for the 2nd time [19:37:37] hey do you guys use teh spring extension for phab? [19:38:22] milimetric: https://wikitech.wikimedia.org/wiki/Analytics/DataWarehouse/Requirements [19:38:46] chasemp: yes, "sprint" right? [19:38:53] yep [19:38:54] nuria: ok, great, I'll validate the data [19:39:12] milimetric: excellent [19:39:17] chasemp: yes, thank you, we were one of the main teams that needed it [19:39:28] milimetric: so I understand christopher has fixed some things [19:39:36] I don't really use it, am not a good person to verify [19:39:40] could you poke at https://phab-01.wmflabs.org/sprint/ [19:39:42] if you have a moment [19:39:46] to see how it seems [19:39:52] sure, kevinator might want to as well [19:39:54] that is new phab version on latest sprint app [19:40:03] (that labs install has local creds fyi) [19:40:50] ( did get an error upgrading but I believe it to be benign but if it blows up big time don't waste too much time just let me know ) [19:40:56] ( but should be fine) [19:42:52] qchris: ! let's do it! [19:42:52] milimetric, nuria: yes, I made some progress on puppet+apache inside vagrant, but if you milimetric have 10 minutes, I would appreciate your help [19:42:52] in particular https://phabricator.wikimedia.org/T78679#973628 is supposed to be addressed here [19:42:52] thanks [19:42:52] mforns: sure, batcave? [19:42:52] ok! [19:42:53] chasemp: what do you mean local creds? [19:42:53] have to sign up for a local account to that instance as the labs "preview" box isn't tied into ldap etc [19:43:03] but it's not ephemeral just not tied in to auth [19:46:30] ottomata: (Sorry, missed ping) Yes, let's do it. [19:46:45] I just rebased the change just some minutes ago. [19:46:50] https://gerrit.wikimedia.org/r/#/c/172201/8 [19:47:40] (PS1) QChris: Add projectcounts for 2015-01-03 -- 2015-01-12 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/184719 [19:48:33] (CR) QChris: [C: 2 V: 2] "See Nuria's comment on" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/184719 (owner: QChris) [19:49:52] milimetric: I looked at the null issue for EL [19:50:07] are you still in the batcave? [19:50:35] okey [19:53:29] yes kevinator we're here [19:55:37] qchris: [19:55:47] this should be command, yes? [19:55:55] https://gerrit.wikimedia.org/r/#/c/172201/10/manifests/misc/statistics.pp [19:56:00] line 1147 [19:56:28] ottomata: Yes. That's the one for the monitoring. [19:59:44] ottomata, hi! do you have 10 minutes to help me? I'm stuck with eventlogging, trying to configure apache inside vagrant via puppet eventlogging role... [20:00:42] qchris: is there a reason to do 750 and not 755 on these aggregator directories? [20:00:57] mforns, yes, lemme finish up with qchris' thing then i gotcha [20:01:12] ottomata, fine! [20:02:00] ottomata: Nothing secret in there, so if you prefer 755, I'll patch it. [20:02:06] i'm patchin gnow [20:02:11] k. Thanks. [20:08:50] halfak: didn't the original job take ~18 hours? The diff job that failed, that is? [20:09:08] I believe so. [20:09:49] ok, updating our etherpad [20:12:32] there it goes qchris [20:12:33] :) [20:12:47] ottomata: Thanks for all those fixes. [20:12:54] Thanks for making it work. [20:12:58] qchris, i'm going to run the cron manually to double check, ja? [20:12:59] s'ok ? [20:13:03] Sure. [20:13:13] The monitoring one should pass just fine. [20:13:39] hm [20:13:40] 2015-01-13 20:13:30,338 ERROR Source directory '/wmf/data/archive/pagecounts-all-sites' does not point to an existing directory [20:13:40] The aggregating one too. [20:13:44] maybe mount path didn't resolve [20:13:45] checking [20:16:41] ottomata: Might be missing leading "::"? [20:17:11] maybe [20:17:14] i woudln't think it would need it here [20:17:16] going to try it [20:20:15] kevinator: how do you get a project to be a "sprint" in phab? like this one: https://phabricator.wikimedia.org/project/view/997/ [20:20:37] hm, no change qchris, i didn't really expect any. [20:20:38] hm [20:20:43] Mhmm :-/ [20:20:57] I'm trying to replicate in the labs phab (https://phab-01.wmflabs.org/) so I can test fixes to performance, etc. [20:21:12] ottomata: manifests/role/analytics.pp used it in the same way :-/ [20:21:24] ya [20:21:41] milimetric: I have to dig it up… lately I found a “create sprint” button when looking a a sprint [20:22:05] milimetric: also sprint names need to start witih § [20:22:31] I recall starting by creating a regular project and then adding to it the sprint fields. [20:23:34] HM [20:23:36] this should work. [20:29:34] milimetric: any success? [20:29:57] milimetric: all i think you need is to create a project and have the name start with § [20:30:34] kevinator: I can't find the "create sprint" button anymore. It was there before I signed up and now it's gone [20:30:35] milimetric: then you can go back to edit the properties of the project with a start/end date etc [20:30:49] I did make it start with that section thig [20:31:03] I think the button only shows up when you are looking at a sprint [20:32:15] oh snap, kevinator: https://phab-01.wmflabs.org/sprint/board/54/ [20:32:46] no change either :( [20:33:01] * qchris looks puzzled. [20:35:26] milimetric: cool…. but it doesn’t feel faster. [20:36:14] I don’t see the create sprint button either… like i see here: https://phabricator.wikimedia.org/project/view/997/ [20:36:40] kevinator: i think the settings / interface is different [20:36:49] regardless, it does seem faster to me [20:36:51] yeah [20:37:00] i loaded up 7 tasks in that new sprint [20:37:06] and the sprint board takes about 3 seconds to load [20:37:09] compared to 9 seconds in prod [20:37:15] and i think this server is a lot tinier [20:37:22] yeah, it does seem a little faster [20:37:30] chasemp: I only tested a little bit, and it does seem a little bit faster [20:37:36] I say go for it, nothing seems broken [20:37:42] and we'll let you know if anything blows up in prod [20:37:46] I was wondering too if it was the server… [20:37:59] o/ [20:38:06] ottomata: At least the hardcoding fixed it :-) [20:38:12] Thanks [20:38:26] chasemp: yep, we’ll let you know if we run into issues [20:39:35] phab-01 is a lot faster right now. I wonder if it was bogged down while I originally tested. [20:39:54] uhm, ok, i htink it is doing? [20:39:58] qchris, i thikn you can test from here? [20:40:01] ottomata: The script is complaining that there is no data to write. [20:40:09] But that's just fine. [20:40:16] in git, right? [20:40:24] Right. [20:40:32] it shells out to git to commit and push. [20:41:25] I'll check tomorrow whether or not it nicely did its thing. [20:42:23] (PS17) Nuria: Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 [20:43:25] (CR) Nuria: ">$archive_directory should mean the top level archive directory," [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 (owner: Nuria) [20:43:42] k danke [20:44:06] ottomata: Thanks for banging that change into shape! [20:44:09] mforns: did you get the help you needed with the vagrant devserver? [20:45:01] nuria, I spoke with dan and we could not find a solution, I'm waiting for andrew to help me, he will ping me when done with chris [20:45:29] btw nuria, I'll look at your review today [20:45:55] mforns: ok, otherwise I can try to look at the changes later on today if you push them (i will be off for couple hours) [20:46:23] ok, I'll push them [20:50:27] ah, mforns [20:50:30] i am done with qchris, worry! [20:50:32] sorry* [20:50:33] what's up? [20:50:36] hey [20:51:10] so.. I'm trying to do apache config inside vagrant [20:51:32] to be able to execute the eventlogging-devserver inside vagrant [20:52:06] I think I'll be able to express myself better in hangout, is it ok for you? [20:52:08] sure [20:52:09] ja [20:52:11] joining batcave [20:52:12] batcave? [20:52:14] ok [21:36:24] kevinator: thanks guys, read the backlog seems good so far :) [21:37:13] (CR) Ottomata: [WIP] UDF to get country code from IP address UDF to determine client IP address given values from source IP address and XFF headers Added I (7 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 (owner: Ananthrk) [21:41:52] nuria i thikn you accidentally added a change to datasets_reifned that we had already merged, ja? [21:41:58] maybe it just needs a rebase? [21:42:10] oh, hadn't merged yet [21:42:21] but it is in a different patchset [21:42:32] (CR) Ottomata: [C: 2 V: 2] Escaping leading 0. Adding empty done flag. [analytics/refinery] - https://gerrit.wikimedia.org/r/184531 (owner: Nuria) [21:43:06] nuria: also, i think you didn't see the inline comments on the previous patchset [21:48:22] (CR) Ottomata: Generalised class of UDFs for handling the webrequests table (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/181939 (owner: OliverKeyes) [21:48:57] (CR) Ottomata: "If you rebase your change, you shouldn't need the refinery-tools/pom.xml file in this commit anymoire." (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/181939 (owner: OliverKeyes) [21:50:04] (CR) Ottomata: Generalised class of UDFs for handling the webrequests table (5 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/181939 (owner: OliverKeyes) [21:52:02] (PS3) Ottomata: Add Oozie legacy_tsv job for api-usage TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184493 (owner: QChris) [21:52:07] (CR) Ottomata: [C: 2 V: 2] Add Oozie legacy_tsv job for api-usage TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184493 (owner: QChris) [21:52:19] (PS3) Ottomata: Add Oozie legacy_tsv job for glam_nara TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184494 (owner: QChris) [21:54:28] (CR) Ottomata: "But we won't be able to replace the udp2log data with this until this data is good, right? I'm ok with merging, but maybe we shouldn't bo" [analytics/refinery] - https://gerrit.wikimedia.org/r/184496 (owner: QChris) [21:54:48] (PS3) Ottomata: Add Oozie legacy_tsv job for 5xx TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184496 (owner: QChris) [21:58:41] da ircs are getting weeeeird... :) [21:58:57] (CR) QChris: "> But we won't be able to replace the udp2log data with this until this data is good, right?" [analytics/refinery] - https://gerrit.wikimedia.org/r/184496 (owner: QChris) [22:07:44] qchris: ok let's merge them, but only start the ones that will be useable for now [22:07:52] we should also talk about using refined table where we can [22:08:18] ottomata: hokey, about only starting the ones that are usable. [22:08:29] (CR) Ottomata: [C: 2 V: 2] Add Oozie legacy_tsv job for glam_nara TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184494 (owner: QChris) [22:08:36] Are the refined tables already production-grade? [22:08:42] (CR) Ottomata: [V: 2] Add Oozie legacy_tsv job for edits TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184495 (owner: QChris) [22:09:02] (CR) Ottomata: [C: 2 V: 2] Add Oozie legacy_tsv job for 5xx TSVs [analytics/refinery] - https://gerrit.wikimedia.org/r/184496 (owner: QChris) [22:09:44] qchris: hm, good q. [22:09:55] i mean, yes/no? nuria is going to use them for the mobile_apps job [22:10:04] it seems to be working fine. [22:10:15] i'd like to refactor the oozie dataset names [22:10:41] so that the refined ones are called 'webrequest_*', and the other are 'webrequest_*_raw' [22:10:45] If I understood it correctly, after the refactor, the jobs would switch automatically? [22:10:51] hm. [22:10:54] suppose that is true! [22:10:58] ;-) [22:11:00] :-D [22:11:08] we'd have to relaunch them, but ja [22:11:30] We'd have to relaunch them anyways. [22:11:32] aye [22:11:56] Awesome. [22:12:18] about how long do these take right now? [22:12:21] to generate a day? [22:12:34] Depends on the job and how busy the cluster is. [22:12:49] A few where running three hours (but they [22:12:59] used the upload partition, which is typically faulty these days, so [22:13:08] they would not start for current days) [22:15:42] Basically, those jobs are more preparation for when all partitions are cool and flawless. [22:16:16] aye [22:16:40] hm, whatcha think, should I refactor the dataset names now? [22:16:43] sooner rather than later? [22:16:46] why not, eh? [22:16:56] Refactoring is fine by me. [22:17:25] ok, then we will launch the mobile and text tsv ones with the refined data then. [22:17:25] ok [22:17:27] I'm about to bail out for today, but I guess I would not be much help anywmays. [22:17:56] Cool. [22:19:11] ok cool [22:19:17] ttyt [22:19:43] ok. good night :-) [22:29:35] Analytics-Engineering, Analytics-EventLogging: Researchers access EventLogging logs to troubleshoot new experiments - https://phabricator.wikimedia.org/T85027#974715 (kevinator) [22:29:57] Analytics-Engineering, Analytics-EventLogging: Researchers access EventLogging logs to troubleshoot new experiments - https://phabricator.wikimedia.org/T85027#974722 (kevinator) a:Ottomata [22:30:06] ottomata: can you have a look at https://phabricator.wikimedia.org/T85027 [22:30:12] let me know if you have questions. [22:31:25] ottomata: One thing about the refined datasets is worrying me. I know I should not care ... but still ... is there a reason to not use a done-flag? [22:32:16] Hive screwed us so often already. And honestly, I do not fully buy "$X is atomic" on the cluster. [22:32:25] Analytics-Engineering, Analytics-EventLogging: Researchers access EventLogging logs to troubleshoot new experiments - https://phabricator.wikimedia.org/T85027#974723 (Ottomata) We don't really have a mechanism for keeping folks from running 'computationally intensive operations'. If you want folks to have un... [22:32:48] qchris: there is no reason other than it seemed an unnecessary step [22:32:54] I found files ending in ".copying" lying around and some such, although they should not be there. [22:33:08] hm, in the table data? [22:33:10] after inserting? [22:33:37] I do not know the exact details any longer. [22:33:44] But in general, I do not trust Hive. [22:34:15] :) [22:34:20] I'm fine to add a done-flag [22:34:32] Meh. I guess I am over-cautuous. [22:34:49] But if you refactor anyways ... a done-flag would make [22:34:55] things a bit more robust. [22:35:03] And more cluster-friendly. [22:35:17] yeah [22:35:18] i'll do it. [22:35:33] There is a workflow for the done-flag. ... Let me find it... [22:35:57] https://git.wikimedia.org/blob/analytics%2Frefinery/e1bccdcf65425f35084b7753f93be223cbe8ec60/oozie%2Futil%2Fmark_directory_done%2Fworkflow.xml [22:36:20] ^ that workflow abstracts "writing the done-flag". [22:36:51] great, danke [22:37:00] Thanks for considering to add it. [22:37:04] * qchris hugs ottomata [22:37:06] :-) [22:39:06] Good night (again). [22:42:21] hey ottomata [22:42:57] hey kevinator [22:43:03] i replied, didya see? [22:43:23] ottomata: Ryan commented a few days ago on analytics-l that he lost access to the logs on EL beta [22:43:34] yeah, I saw the reply [22:43:41] I’m dealing with something else at the moment [22:44:27] do you know if anything on those machines could have caused access permisions to change? [22:44:53] naw, kevinator, i don't know much about EL beta, that is in labs, rigth? [22:45:02] yes, in labs [22:45:20] I don’t know much about it either… (machine name, location of logs etc) [22:45:45] I can reply with a request for more data [22:45:52] but thought I’d ping you first [22:46:29] aye, ¯\_(ツ)_/¯ [22:47:32] yeah, I’ll get more info [23:03:18] (PS1) Ottomata: Refactor webrequest dataset names [analytics/refinery] - https://gerrit.wikimedia.org/r/184796 [23:15:36] nuria: yay irc madness is over [23:15:45] ok, so I vetted the data the same way you did [23:15:49] and found all the same problems you did [23:16:04] so now I'm working on writing a script that just does the vetting for us, looking across databases. [23:16:34] i've gotta run for tonight, but will be back on it tomorrow [23:16:46] however, if we want to tell Sean something, it's that the problems are all still there [23:17:05] well, sorry, i have to double check the user bot flag thing [23:17:08] because that seemed weird [23:19:42] (PS2) Ottomata: Refactor webrequest dataset names [analytics/refinery] - https://gerrit.wikimedia.org/r/184796 [23:19:44] (PS1) Ottomata: Add _SUCCESS done-flag in refined webrequest data directories after successful refinement [analytics/refinery] - https://gerrit.wikimedia.org/r/184804 [23:36:12] MediaWiki-extensions-MultimediaViewer, Analytics, Multimedia: Find a robust way of filtering local cache hits out of performance figures - https://phabricator.wikimedia.org/T86672#974833 (Tgr) Small event_total (RT.duration) with old event_timestamp (HTTP.Date)? [[ http://www.slideshare.net/turbobytes/state-... [23:55:48] MediaWiki-extensions-MultimediaViewer, Analytics, Multimedia: Find a robust way of filtering local cache hits out of performance figures - https://phabricator.wikimedia.org/T86672#974875 (Tgr) >>! In T86672#974833, @Tgr wrote: > Small event_total (RT.duration) with old event_timestamp (HTTP.Date)? Which of c... [23:57:49] (CR) Gergő Tisza: [C: 2] Fix filter differentiating between varnish hits and misses [analytics/multimedia] - https://gerrit.wikimedia.org/r/184646 (https://phabricator.wikimedia.org/T86675) (owner: Gilles) [23:57:58] (Merged) jenkins-bot: Fix filter differentiating between varnish hits and misses [analytics/multimedia] - https://gerrit.wikimedia.org/r/184646 (https://phabricator.wikimedia.org/T86675) (owner: Gilles)