[01:30:06] <grrrit-wm>	 (PS1) Gergő Tisza: Fix date format in pageview SQL query [analytics/multimedia] - https://gerrit.wikimedia.org/r/185362 (https://phabricator.wikimedia.org/T78189)
[01:30:56] <grrrit-wm>	 (CR) Gergő Tisza: [C: 2] "Self-merge, trivial bug that prevents me from testing the limn half of the patch." [analytics/multimedia] - https://gerrit.wikimedia.org/r/185362 (https://phabricator.wikimedia.org/T78189) (owner: Gergő Tisza)
[01:31:02] <grrrit-wm>	 (Merged) jenkins-bot: Fix date format in pageview SQL query [analytics/multimedia] - https://gerrit.wikimedia.org/r/185362 (https://phabricator.wikimedia.org/T78189) (owner: Gergő Tisza)
[02:16:56] <grrrit-wm>	 (PS1) Ewulczyn: first draft of host parsing udf. extracts the project and project qualifier from the uri_host. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/185377
[02:20:49] <grrrit-wm>	 (PS2) Ewulczyn: first draft of host parsing udf. extracts the project and project qualifier from the uri_host. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/185377
[02:24:54] <grrrit-wm>	 (PS1) Yurik: weblog2: monthly sums, enabled [analytics/zero-sms] - https://gerrit.wikimedia.org/r/185380
[02:25:10] <grrrit-wm>	 (CR) Yurik: [C: 2 V: 2] weblog2: monthly sums, enabled [analytics/zero-sms] - https://gerrit.wikimedia.org/r/185380 (owner: Yurik)
[03:17:40] <wikibugs>	 Analytics-EventLogging: Can't save new schemas on MetaWiki - https://phabricator.wikimedia.org/T86989#981125 (ori) Open>Resolved a:ori Fixed by cherry-picking https://gerrit.wikimedia.org/r/#/c/184111/ to wmf14
[03:32:51] <grrrit-wm>	 (CR) OliverKeyes: first draft of host parsing udf. extracts the project and project qualifier from the uri_host. (11 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/185377 (owner: Ewulczyn)
[08:06:46] <grrrit-wm>	 (CR) Gilles: [C: 2] Show page view / image view comparison [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/185336 (https://phabricator.wikimedia.org/T78189) (owner: Gergő Tisza)
[08:07:35] <grrrit-wm>	 (CR) Gilles: [V: 2] Show page view / image view comparison [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/185336 (https://phabricator.wikimedia.org/T78189) (owner: Gergő Tisza)
[08:08:08] <grrrit-wm>	 (CR) Gilles: "Deployed" [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/185336 (https://phabricator.wikimedia.org/T78189) (owner: Gergő Tisza)
[09:03:54] <wikibugs>	 Analytics-EventLogging: Can't update schemas on meta - https://phabricator.wikimedia.org/T86926#981315 (Gilles) Maybe related to your recent changesets, Timo?
[09:06:19] <wikibugs>	 Analytics-EventLogging: sendBeacon throws an exception in Firefox when event.gif is adblocked - https://phabricator.wikimedia.org/T86680#981317 (Gilles) a:MarkTraceur>ori
[09:11:16] <wikibugs>	 Analytics-EventLogging: sendBeacon throws an exception in Firefox when event.gif is adblocked - https://phabricator.wikimedia.org/T86680#981323 (Gilles) I can reproduce with the subscriptions listed by @oseveno and as @Rillke already confirmed, the exception is correctly trapped by wrapping the navigator.send...
[11:30:19] <wikibugs>	 Analytics-EventLogging: PHP Notice:  Undefined index: REQUEST_TIME in extensions/EventLogging/includes/EventLogging.php on line 56 - https://phabricator.wikimedia.org/T70629#981545 (hashar) Seems EventLogging works fine under HHVM with a bunch of other extensions. Proposed https://gerrit.wikimedia.org/r/#/c/1...
[13:10:35] <wikibugs>	 Analytics-EventLogging: PHP Notice:  Undefined index: REQUEST_TIME in extensions/EventLogging/includes/EventLogging.php on line 56 - https://phabricator.wikimedia.org/T70629#981702 (Nemo_bis) Your definition of "a bunch" differs of 1-2 orders of magnitude from mine. :) I still have open patches to fix the bug...
[13:11:34] <wikibugs>	 Analytics: Firewall changes on 2015-01-13 affect udp2log - https://phabricator.wikimedia.org/T86973#981704 (QChris)
[13:13:21] <wikibugs>	 Analytics: Firewall changes on 2015-01-13 affect udp2log - https://phabricator.wikimedia.org/T86973#980599 (QChris) (Removed Incident-20141129-Network project as that is not a general catch-all for network issies, but was only for the Incident report of network issues that occurred on 2014-11-19)
[14:21:39] <grrrit-wm>	 (PS1) QChris: Add script to dump status of webrequest partitions [analytics/refinery] - https://gerrit.wikimedia.org/r/185436
[14:21:41] <grrrit-wm>	 (PS1) QChris: Add quiet mode to webrequest partition status dumping [analytics/refinery] - https://gerrit.wikimedia.org/r/185437
[14:36:38] <wikibugs>	 Analytics-EventLogging: Recent EventLogging change breaking Echo, Popups on blocked event.gif calls / on Firefox with Adblock Plus - https://phabricator.wikimedia.org/T86918#981898 (Aklapper) This seems to be the same as T86680, hence merging.
[14:36:46] <wikibugs>	 Analytics-EventLogging: Recent EventLogging change breaking Echo, Popups on blocked event.gif calls / on Firefox with Adblock Plus - https://phabricator.wikimedia.org/T86918#981900 (Aklapper)
[14:36:48] <wikibugs>	 Analytics-EventLogging: sendBeacon throws an exception in Firefox when event.gif is adblocked - https://phabricator.wikimedia.org/T86680#973983 (Aklapper)
[14:39:05] <wikibugs>	 Analytics-EventLogging: sendBeacon throws an exception in Firefox when event.gif is adblocked - https://phabricator.wikimedia.org/T86680#981904 (Aklapper) p:High>Unbreak! This creates problems and confusion all over the VillagePumps etc.; raising priority.
[14:42:47] <grrrit-wm>	 (PS1) QChris: Add --hdfs-mount option to script to dump webrequest status [analytics/refinery] - https://gerrit.wikimedia.org/r/185440
[14:55:10] <halfak>	 ottomata, do you have time to look at the altiscale data transfer options with me today?
[14:55:41] <ottomata>	 sure, let's do it after I do the haddoop namenode migration this morning, i'm starting that in 45 minutes
[14:55:51] <ottomata>	 i think it will take me 30 - 60 mintues to do, hopefully less
[14:55:55] <halfak>	 Sure.  Sounds good.
[14:55:57] <ottomata>	 i've scheduled 2 hours just in case
[14:56:43] <halfak>	 kk.
[14:57:10] <halfak>	 hmm... I have something writing to HDFS.  It seems like it might have stalled.
[15:22:37] <wikibugs>	 Analytics-Engineering, Analytics-EventLogging: Community Engineers have a working EventLogging dev server in vagrant [8 pts] - https://phabricator.wikimedia.org/T86067#981952 (mforns) Open>Resolved
[15:26:14] <ottomata>	 hmm, nuria, the first mobile daily job ran
[15:26:24] <ottomata>	 but, i would have expected it to wait until then end of today
[15:26:29] <ottomata>	 since I the start time was jan 16
[15:26:35] <ottomata>	 maybe we've got our offsets wring?
[15:26:36] <ottomata>	 wrong?
[15:26:40] <ottomata>	 jan 16 contains data for jan 15?
[15:27:30] <ottomata>	 hmm, also, the filename is
[15:27:33] <ottomata>	 2015-1-16.gz
[15:27:39] <ottomata>	 probably would be better to have those 0 padded
[15:27:45] <ottomata>	 so they sort properly in displays
[15:27:58] <ottomata>	 but, it seems to have worked!
[15:27:59] <ottomata>	 Android        56560
[15:27:59] <ottomata>	 iOS        825
[15:28:05] <ottomata>	 nuria, also, these files are super small
[15:28:15] <ottomata>	 do we need to gzip compress them?
[15:28:18] <ottomata>	 could we just leave them uncompressed?
[15:28:25] <wikibugs>	 Analytics-EventLogging: Client Engineer passes null to EventLogging::logEvent for optional boolean fields [8 pts] - https://phabricator.wikimedia.org/T78325#981960 (kevinator)
[15:28:46] <wikibugs>	 Analytics-EventLogging: Client Engineer passes null to EventLogging::logEvent for optional boolean fields [8 pts] - https://phabricator.wikimedia.org/T78325#842558 (kevinator) We are canceling our commitment to complete this task in our sprint.  I removed the sprint project.
[15:30:17] <nuria>	 ottomata: i think compressing as a default is an ok default
[15:30:40] <nuria>	 ottomata: as for the zero padding i will get a patch together.
[15:31:58] <ottomata>	 k
[15:31:58] <ottomata>	 cool
[15:32:08] <ottomata>	 also, check that the date offset is right
[15:32:23] <ottomata>	 the nominal time for this job was  2015-01-16 00:00 GMT
[15:32:41] <ottomata>	 and it was started at 2015-01-16 02:16 GMT
[15:32:45] <ottomata>	 the 16th isn't over yet
[15:32:50] <ottomata>	 so that means that the data in that file is for the 15th
[15:33:11] <ottomata>	 maybe you want dataset files 0, 23, instead of -23?
[16:09:19] <nuria>	 milimerric: will be back in 40 mins and look at pentaho then
[16:09:25] <nuria>	 cc kevinator
[16:11:13] <wikibugs>	 Analytics-EventLogging: Client Engineer passes null to EventLogging::logEvent for optional boolean fields [8 pts] - https://phabricator.wikimedia.org/T78325#981990 (kevinator) Open>declined We are not going implement this functionality... see my longer comment above.
[16:12:57] <wikibugs>	 Analytics-Engineering, Analytics-EventLogging: Community Engineers have a working EventLogging dev server in vagrant [8 pts] - https://phabricator.wikimedia.org/T86067#981993 (kevinator) Resolved>Open temporarily re-opening this task so its points show up on our burndown chart.
[16:46:29] <ottomata>	 migration done!  hadoop is back
[16:47:24] <nuria>	 ottomata: super fast!, ok, will start trying changes in abit
[16:51:39] <wikibugs>	 Analytics, operations: Upgrade Analytics Cluster to Trusty, and then to CDH 5.3 - https://phabricator.wikimedia.org/T1200#982081 (Ottomata)
[16:52:57] <wikibugs>	 Analytics, operations: Upgrade Analytics Cluster to Trusty, and then to CDH 5.3 - https://phabricator.wikimedia.org/T1200#20804 (Ottomata) analytics1001 and analytics1002 have been provisioned, and the Hadoop NameNode and YARN master services have been migrated off of analytics1010 and analytics1004 (ciscos)....
[16:55:50] <ottomata>	 nuria: ananthrk, lemme know if you want me to chime in, maybe chat here?
[16:56:32] <nuria>	 yes.
[16:56:51] <ananthrk>	 ottomata: nuria, just to summarize our discussions so far -
[16:57:20] <ananthrk>	 the reason why we have the constructor is to pass the db path from tests even though the main code tries to read it from a property
[16:57:41] <ananthrk>	 this property would typically be set via hive-site or via the CLI
[16:57:46] <nuria>	 ananthrk, ottomata : what we wnat to avoid is having test specific code mixed with runtime code
[16:57:55] <nuria>	 ananthrk: as that is a source of potential bugs
[16:58:14] <ananthrk>	 would making the constructor protected minimize the confusion?
[16:58:15] <nuria>	 ananthrk, ottomata : also we want to avoid testing code "undoing" teh static initializer
[16:58:51] <nuria>	 ananthrk: actually you should be able to pass properties to the test with mvn, did you tried that?
[16:59:18] <ananthrk>	 that would be system or env level properties
[16:59:31] <ananthrk>	 but the ones that we read in our code are in the "hiveconf" namespace
[16:59:40] <nuria>	 ananthrk: regardless of whether you make the constructor private you woudl still be undoing the static initialization, which is not good practice.
[16:59:52] <ottomata>	 ananthrk, i'm pretty sure hiveconf will use sys level properties too, at least in some way.
[16:59:59] <ottomata>	 i'm fine if they are sys level propeties and not avaiable in hive
[17:00:00] <ottomata>	 e.g.
[17:00:08] <ottomata>	 -Dmaxmind.database.country=/path/to/db
[17:00:35] <ottomata>	 hm, maybe it won't, not sure,  even if it doesn't though, i'm ok if they are sys level
[17:00:52] <ottomata>	 if they are sys level, then you could move the logic to choose the database into the Geocode class, rather than the hive class
[17:01:11] <nuria>	 agreed, the important thing is removing testing specific code and let configuration handle that
[17:01:54] <ananthrk>	 okay..in that case, we can even set it in tests using System.setProperty
[17:02:17] <ananthrk>	 the part I was not sure of was the way it would be set in the cluster
[17:04:14] <ananthrk>	 also - this cannot be in Geocode because the class it common to both Country and City UDF - and we want to use two different DBs for these two usecases
[17:05:11] <ananthrk>	 otherwise we have to split the underlying Geocode class into two as well - one for country and the other for city
[17:06:48] <ottomata>	 hm, ananthrk, as is now, someone could call country() on the city db, right?
[17:07:02] <ottomata>	 if they were just using the Geocdoe class
[17:07:03] <ananthrk>	 yes..they can
[17:07:13] <ottomata>	 or even city() on the country db
[17:07:19] <ottomata>	 if they instantiated Geocode witih the country db
[17:07:44] <ananthrk>	 city() on the country DB will not work
[17:07:57] <ananthrk>	 but country() on city DB will
[17:08:00] <ottomata>	 right, but someone coudl maek that happen right now, right?
[17:08:20] <ottomata>	 g = new Geocode(/path/to/country); g.getGeocodedData()
[17:08:29] <ananthrk>	 yes..by changing UDFs code
[17:08:37] <ottomata>	 no, this class will be used by more than just hive
[17:08:44] <ottomata>	 the UDF is just for the Hive interface
[17:08:55] <ottomata>	 say someone just wrote a mapreduce job that needed Geocoding
[17:08:57] <ottomata>	 they would use this class
[17:09:54] <ottomata>	 so, probably
[17:10:03] <ottomata>	 the class should know how to work with both databases
[17:10:33] <ottomata>	 maybe have two DatabaseReaders?
[17:10:44] <ananthrk>	 thats what I am thinking now
[17:11:21] <ananthrk>	 in that case we will have two readers and initialize both of them with their respective system-level properties?
[17:11:23] <ottomata>	 we've already got two settings
[17:11:25] <ottomata>	 yes
[17:11:28] <ottomata>	 that sounds good to me
[17:12:42] <ananthrk>	 so any user of this class is expected to set these two properties - or atleast have the respective DBs in the classpath..is that a reasonable assumption to make?
[17:16:04] <wikibugs>	 Analytics-EventLogging: sendBeacon throws an exception in Firefox when event.gif is adblocked - https://phabricator.wikimedia.org/T86680#982160 (greg) @gilles / @ori: Feel free to fix/deploy this ASAP given Andre's previous comment without checking in with me further.
[17:17:28] <ottomata>	 ananthrk: they are expected to set them if they want to change the defaults
[17:17:43] <ottomata>	 i think having the defaults at /usr/share/GeoIP/... is cool
[17:17:51] <ottomata>	 since that should be present everywhere on our cluster
[17:17:54] <ottomata>	 does that make sense?
[17:17:56] <ananthrk>	 okay...fine
[17:18:17] <ottomata>	 cool
[17:19:27] <ananthrk>	 a quick question on setting this property from Hive CLI
[17:19:47] <ottomata>	 ja?
[17:19:49] <ananthrk>	 will the users need to explicitly use system:maxmind.database.country=<path>
[17:22:10] <ottomata>	 I'm not sure, if that works, then that is fine.  or, they may be able to just use -D on the CLI when they start hive
[17:22:18] <ottomata>	 hive -Dmaxmind.database.country=<path>
[17:22:52] <ananthrk>	 okay...thanks
[17:23:12] <ottomata>	 yup :)
[17:23:32] <ananthrk>	 nuria: do you see any issues in setting the property in tests explicitly via System.setProperty?
[17:25:08] <nuria>	 ananthrk: in teh setup method , I think that will be preferable but i suspect that  you might run into issues, it might be be sytem properties have been read alredy and test cannot change them, that is why the mvn route seemed more likely to work. But you can try it and see.
[17:27:53] <nuria>	 ananthrk: aren't JVM properties read by JVM upon startup? I think that is the case for tests too
[17:28:35] <nuria>	 ananthrk: but as i said you can try it and see, it is possible to set up properties just for tests, we have to find teh best way to do it in our env.
[17:29:41] <ananthrk>	 you are right...the ones initialized in setup are not available in the class
[17:30:37] <nuria>	 https://www.irccloud.com/pastebin/NnNqEka1
[17:31:25] <nuria>	 So you can do the same on mvn i bet.
[17:31:28] <nuria>	 cc ananthrk
[17:31:51] <ananthrk>	 nuria: thanks...will check
[17:33:37] <nuria>	 ananthrk: k, please see: http://maven.apache.org/surefire/maven-surefire-plugin/examples/system-properties.html
[17:35:01] <ananthrk>	 yup...saw them
[17:36:27] <ananthrk>	 we currently do not explicitly add the surefire-plugin in our pom.xml
[17:36:44] <ananthrk>	 i trust it is okay to add them?
[17:41:27] <nuria>	 ananthrk: that is a question for ottomata the man, i would need to look a bit more about what that pluging does exactly but qchris and ottomata might have looked at options whn they setup mvn infrastructure.
[17:42:02] <ananthrk>	 okay..cc ottomata ?
[17:43:08] <ottomata>	 ha you can add whatever you want
[17:43:18] <ottomata>	 once we get this patchset ready, we'll add the jars to our archiva mirror
[17:43:19] <qchris>	 We're using it already. It's just not in the poms.
[17:43:34] <ananthrk>	 ok :)
[17:44:50] <qchris>	 Actually ... it should already be in archiva ...
[17:46:16] <qchris>	 There you go: http://archiva.wikimedia.org/#artifact/org.apache.maven.plugins/maven-surefire-plugin
[17:48:37] <ananthrk>	 thanks qchris
[17:48:46] <qchris>	 yw
[17:53:09] <ananthrk>	 qchris - is 2.18.1 a valid version for the surefire plugin?
[17:54:11] <qchris>	 Not sure what "valid" means in this context ... but 2.18.1 is in wmf's archiva.
[17:54:26] <qchris>	 ananthrk: ^ http://archiva.wikimedia.org/#artifact/org.apache.maven.plugins/maven-surefire-plugin/2.18.1
[17:55:13] <ananthrk>	 i saw a checksum failure when i first added the plugin to pom
[17:55:37] <ananthrk>	 but i now think it was probably due to a broken download in my end
[17:55:56] <qchris>	 So it's working now?
[17:56:16] <ananthrk>	 hm...just moved to a different one now
[17:56:28] <ananthrk>	 https://www.irccloud.com/pastebin/Y2T9Tgp6
[17:57:08] <qchris>	 ananthrk: that ^ error is for maven-plugin-tools-3.3
[17:57:14] <qchris>	 not for the surefire plugin.
[17:58:03] <ananthrk>	 got the same msg for surefire earlier, but it got resolved
[17:58:08] <ananthrk>	 https://www.irccloud.com/pastebin/PphTlUXj
[17:58:21] <ananthrk>	 so may be it is the connection in my end??
[17:59:01] <qchris>	 Are you downloading manually or through automatic means?
[18:02:14] <ananthrk>	 mvn is automatically downloading the dependencies
[18:02:28] <qchris>	 Mhmmm ... there is something weird going on then.
[18:02:33] <qchris>	 The error message you gave:
[18:02:42] <qchris>	 [WARNING] Checksum validation failed, expected <!-- but is 86c69e79aa595f22b02b2bdd4e46ab34e3431375 for http://archiva.wikimedia.org/repository/mirrored/org/apache/maven/surefire/surefire/2.18.1/surefire-2.18.1.pom
[18:03:01] <qchris>	 Says that it is expecting "<!--" (which the url has for me)
[18:03:20] <qchris>	 but it gets "86c69e79aa595f22b02b2bdd4e46ab34e3431375" which is the sha1sum for the url's content.
[18:03:44] <qchris>	 Somethings wrong here.
[18:03:57] <qchris>	 ananthrk: Can you share the code that produces this error for you?
[18:04:04] <qchris>	 (So I can try reproducing)
[18:04:44] <ananthrk>	 just added the plugin to refinery-core/pom.xml
[18:04:54] <ananthrk>	 https://www.irccloud.com/pastebin/HM1gKgbZ
[18:06:20] <qchris>	 Can you paste your whole pom.xml file somewhere?
[18:06:32] <qchris>	 (Otherwise I would not know where exactly you added it)
[18:06:43] <qchris>	 Also ... on top of which git commit are you using it?
[18:06:57] <qchris>	 (And which of the many pom.xml files in that repo)
[18:07:27] <ananthrk>	 i think it is my connection that is being patchy
[18:07:48] <ananthrk>	 let me try that again and let you know
[18:07:53] <qchris>	 Mhmm... ok.
[18:08:34] <ananthrk>	 (fwiw, i am updating the pom.xml in analytics/refinery/source/refinery-core)
[18:14:25] <wikibugs>	 Mobile-Web, Analytics-EventLogging: Many of the mobile report cards are broken - https://phabricator.wikimedia.org/T86972#982320 (bmansurov) https://phabricator.wikimedia.org/T86059
[18:17:23] <bmansurov>	 Hello all, I was wondering if there were any updates related to this bug: https://phabricator.wikimedia.org/T86059 Mobile team would like to get their nice looking graphs back ;)
[18:20:38] <qchris>	 bmansurov: This task is claimed by mili|lunch ... but judging from his nick, mili|lunch is at lunch.
[18:20:48] <qchris>	 kevinator is not around...
[18:20:54] <bmansurov>	 qchris: ok thanks
[18:20:58] <qchris>	 nuria, mforns: Do you know about the bug?
[18:21:48] <mforns>	 qchris, looking
[18:24:40] <nuria>	 qchris: reading
[18:28:29] <nuria>	 bmansurov, mforns , i think mili|lunch had some updates regaring that bug that for my life i cannot recall
[18:29:07] <bmansurov>	 nuria: thanks ;)
[18:29:32] <nuria>	 mforns: let's wait a bit until he comes back from lunch
[18:33:15] <grrrit-wm>	 (CR) Ottomata: "You can remove refinery-tools/pom.xml from this patchset. Remove it, and then rebase this patchset." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/181939 (owner: OliverKeyes)
[18:34:07] <leila>	 hola nuria.
[18:34:16] <nuria>	 hola amiga!
[18:34:23] <nuria>	 hola amiga leila that is
[18:34:31] <leila>	 we have an option for turning off WikiGrok eod today or letting it run until Tuesday morning. How is EL doing nuria?
[18:34:54] <nuria>	 leila: it's fine, we should let it run, talked to kaldari about this yesterday
[18:35:28] <ananthrk>	 qchris - it was my connection
[18:35:39] <ananthrk>	 just got it resolved and after that my download went through fine
[18:35:54] <leila>	 thanks, nuria. just wanted to make sure since Monday is a holiday and if we expect something to go wrong, it's better to turn it off today.
[18:36:07] <qchris>	 ananthrk: k.
[18:36:13] <qchris>	 Cool.
[18:41:28] <Ironholds>	 ottomata, how would you go about removing the pom changes? bah :/
[18:42:40] <ottomata>	 hm
[18:42:41] <ottomata>	 git reset?
[18:42:52] <wikibugs>	 Mobile-Web, Analytics-EventLogging: Many of the mobile report cards are broken - https://phabricator.wikimedia.org/T86972#982452 (Jdlrobson)
[18:42:53] <wikibugs>	 Analytics-Engineering, Analytics-Visualization: generate.py produced broken CSV files - https://phabricator.wikimedia.org/T86059#982454 (Jdlrobson)
[18:43:04] <wikibugs>	 Mobile-Web, Analytics-Engineering, Analytics-Visualization: generate.py produced broken CSV files - https://phabricator.wikimedia.org/T86059#982455 (Jdlrobson) #Mobile-Web
[18:47:29] <mforns>	 nuria, I'm reading the eventlogging docs now, but if you feel I could be more useful doing other stuff like helping in the presentation, let me know
[18:47:57] <nuria>	 mforns: did you updated the part about EL devserver?
[18:48:04] <mforns>	 nuria, yes
[18:48:09] <nuria>	 mforns: so it is clear that now can be run insie vagrant
[18:48:13] <nuria>	 *inside
[18:48:19] <nuria>	 mforns: i see, great then,
[18:48:21] <mforns>	 yes,
[18:48:42] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add script to dump status of webrequest partitions [analytics/refinery] - https://gerrit.wikimedia.org/r/185436 (owner: QChris)
[18:48:46] <nuria>	 mforns: we have no further guidelines for EL presentation so i think we are ok on that front
[18:48:49] <mforns>	 nuria, I removed the doc about running it outside vagrant
[18:48:58] <nuria>	 mforns: excellent
[18:49:02] <mforns>	 because it did not say anyting about port forwarding
[18:49:12] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add quiet mode to webrequest partition status dumping [analytics/refinery] - https://gerrit.wikimedia.org/r/185437 (owner: QChris)
[18:49:29] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add --hdfs-mount option to script to dump webrequest status [analytics/refinery] - https://gerrit.wikimedia.org/r/185440 (owner: QChris)
[18:58:46] <wikibugs>	 Analytics-EventLogging: Create staging environment - https://phabricator.wikimedia.org/T74767#982584 (Nuria) Resolving. We have beta-labs now.
[18:58:56] <wikibugs>	 Analytics-EventLogging: Create staging environment - https://phabricator.wikimedia.org/T74767#982585 (Nuria) Open>Resolved
[19:00:02] <nuria>	 mforns: so I think there is nothing pressing
[19:02:40] <nuria>	 mforns: weekend for you!
[19:04:00] <mforns>	 nuria, ok, but I'll read some EventLogging docs, I feel a bit behind on that
[19:04:15] <grrrit-wm>	 (CR) Ottomata: first draft of host parsing udf. extracts the project and project qualifier from the uri_host. (4 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/185377 (owner: Ewulczyn)
[19:04:36] <nuria>	 mforns: ok, i think testing in vagrant with navtiming is teh good way to understand how things work
[19:04:54] <mforns>	 ok
[19:09:46] <mili|lunch>	 grrr, my nick ... grrr
[19:11:00] <milimetric>	 bmansurov: sorry man, I forgot about that among the other stuff this week, I'm going to treat it like the prod issue that it is, right now
[19:11:20] <bmansurov>	 milimetric: np, thanks
[19:11:54] <nuria>	 Ironholds: pentaho is alive again cc milimetric , mforns
[19:12:08] <Ironholds>	 nuria, thanks! will poke Ellery
[19:12:12] <mforns>	 oh! what was that?
[19:12:26] <nuria>	 mforns: pentaho?
[19:12:39] <mforns>	 nuria, yes, do you know the problem?
[19:13:31] <nuria>	 mforns: yes, "crappy setup" as called out by milimetric before in many instances
[19:13:59] <nuria>	 mforns: what happens a bunch with stuff like this is that when turning it on/off gets horked on some permit issue
[19:14:16] <mforns>	 nuria, ok
[19:14:18] <nuria>	 mforns: like "i cannot write to this log" but "hey , iam going to tell you i exit anyways"
[19:14:29] <mforns>	 xD
[19:14:40] <nuria>	 mforns: "but i leave the port occupied" just in case
[19:14:58] <mforns>	 hehe
[19:15:02] <mforns>	 ok I see
[19:15:36] <nuria>	 mforns: in sum killing all javaprocesses (visible with ps auxfw) and after checking ports (with netstat -putona, love this one)
[19:16:49] <nuria>	 mforns: and restarting again should fix this one issue, I am sure there are many other problems we can run into, i logged in but did not tried any data .. hopefully that's ok, lemme see
[19:17:32] <nuria>	 mforns: well, i do not think i have data to load, i guess i will wait for someone to use it
[19:18:37] <nuria>	 mforns: do you have a cube that you can try?
[19:18:41] <mforns>	 nuria, ok, let me import the cube
[19:19:01] <nuria>	 mforns: if you are doing nothing else that is, this is super low priority
[19:19:15] <mforns>	 nuria, yes, that should be a sec
[19:20:28] <mforns>	 nuria, the data is still there, an the cube also
[19:20:42] <nuria>	 mforns: ok, will send notice to list, thank you
[19:20:58] <mforns>	 nuria, ok!
[19:26:52] <milimetric>	 (thanks for that nuria - I tried to do the same but of course screwed up my "ps" call)
[19:27:17] <nuria>	 milimetric: np
[19:30:57] <grrrit-wm>	 (PS11) OliverKeyes: Generalised class of UDFs for handling the webrequests table [analytics/refinery/source] - https://gerrit.wikimedia.org/r/181939
[19:31:17] <grrrit-wm>	 (PS12) OliverKeyes: Generalised class of UDFs for handling the webrequests table [analytics/refinery/source] - https://gerrit.wikimedia.org/r/181939
[19:46:25] <milimetric>	 nuria: you got a second to talk about this generate.py mess? :)
[19:46:37] <nuria>	 milimetric: of course
[19:46:44] <nuria>	 give 2 mins to get to the batcave
[19:47:16] <milimetric>	 k, omw
[19:55:17] <milimetric>	 mforns: are you still working for another hour or so?
[20:15:13] <nuria>	 ottomata: yt?
[20:15:34] <ottomata>	 yes
[20:16:48] <nuria>	 ottomata: ok , regarding the ${coord:current(23)
[20:17:03] <ottomata>	 ja
[20:17:31] <nuria>	 ottomata: if I specify frequency="${coord:days(1)}" and frequency="${coord:current(-1)}"
[20:17:35] <nuria>	 sorry
[20:17:47] <nuria>	 if I specify frequency="${coord:days(1)}"
[20:17:48] <nuria>	 and
[20:17:56] <nuria>	 "${coord:current(-1)}"
[20:18:08] <nuria>	 this will count backwards from execution time
[20:18:17] <nuria>	 "${coord:current(1)} will count forward
[20:18:24] <ottomata>	 yes oh yes, right, by days.
[20:18:25] <ottomata>	 OH
[20:18:27] <nuria>	 but both are referenced from execution time
[20:18:27] <ottomata>	 right,
[20:18:27] <ottomata>	 hm
[20:18:30] <ottomata>	 wait, gm
[20:18:31] <ottomata>	 hm
[20:18:42] <ottomata>	 hm
[20:18:45] <nuria>	 not in days, in hours
[20:18:50] <ottomata>	 aye right, because that is dataset
[20:18:51] <ottomata>	 sorry
[20:19:17] <ottomata>	 coord:current(0) refers to the nominal time for the dataset for which the coordinator was launched
[20:19:26] <ottomata>	 it shouldn't ahve anything to do with execution time
[20:19:29] <nuria>	 sorry myself i am getting tangled up
[20:20:18] <nuria>	 starting all over...
[20:20:47] <nuria>	 coord:current(0) starts the count and the count depends on the frequency of coord
[20:21:22] <nuria>	 so ${coord:current(-1)} will be minus 1 day for a job that runs daily
[20:21:28] <nuria>	 according to docs
[20:21:36] <ottomata>	 ok, that makes sense.  so hm
[20:22:07] <ottomata>	 i am looking at qchris' legacy_tsv jobs
[20:22:07] <ottomata>	 it has
[20:22:12] <ottomata>	     frequency="${coord:days(1)}"
[20:22:16] <nuria>	 ya
[20:22:21] <ottomata>	             <start-instance>${coord:current(0)}</start-instance>
[20:22:22] <ottomata>	             <end-instance>${coord:current(23)}</end-instance>
[20:22:37] <nuria>	 right, sooooo
[20:22:53] <nuria>	 because of the dataset those are translated to hours?
[20:23:07] <ottomata>	 i think so.  i think that in dataset context, we are talking about the dataset frequency
[20:23:21] <ottomata>	 which is defined as frequency="${coord:hours(1)}"
[20:23:23] <nuria>	 ok, taht makes a lot of sense
[20:23:49] <nuria>	 so i will change mobile jobs to the tsv convention
[20:23:50] <ottomata>	 but ja, if you use the coord: time functions in the coordinaor context, they will be relative to the coord frequency, i think
[20:24:00] <nuria>	 ottomata: i think so
[20:24:38] <nuria>	 ottomata: ok, changes on the way
[20:28:42] <mforns>	 milimetric, sorry I did not see your message, yep, for 2 more hours
[20:30:28] <grrrit-wm>	 (PS1) Ottomata: Add extra step in webrequest load that adds a _PARTITIONED done-flag [analytics/refinery] - https://gerrit.wikimedia.org/r/185538
[20:31:24] <mforns>	 milimetric, you wanted me to take something?
[20:33:21] <ottomata>	 nuria, would love a quick once over of this if you got a sec: https://gerrit.wikimedia.org/r/#/c/185538/1
[20:33:29] <nuria>	 ottomata: looking
[20:33:59] <nuria>	 ottomata: this is for what ellery was asking on e-mail right?
[20:34:34] <ottomata>	 yes
[20:34:58] <ottomata>	 there are a few recent times in the refined data where we didn't have anything imported at all
[20:35:13] <ottomata>	 especially when dealing with mobile and text, missing data is rare and if it happens is small
[20:35:22] <ottomata>	 ideally: there would be 0 data loss...we may yet get there.
[20:35:42] <ottomata>	 i also would like to make another dataset, that would be "minimal or 0 loss"
[20:35:53] <ottomata>	 that way we could tolerate SOME loss without blocking jobs
[20:35:57] <ottomata>	 but big loss would
[20:36:01] <ottomata>	 not sure how to do that yet
[20:38:53] <grrrit-wm>	 (CR) Nuria: "Some comments about usage in refined dataset." (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/185538 (owner: Ottomata)
[20:39:40] <nuria>	 ottomata: ah i see, if there 1 partition a miss, jobs halt not running subsequent partiotions
[20:40:48] <nuria>	 ottomata: like if we loose data on the 5th (no partition) and we do not fix that by hand no jobs will run from that moment on even if data is available for the 6th (that would be bad)
[20:46:04] <ottomata>	 yes
[20:46:06] <ottomata>	 exactly
[20:46:27] <nuria>	 ottomata: you knowwww
[20:46:30] <ottomata>	 ellery says he is currently kinda side stepping this issue, by only depending on a single hour, and then running against a whole day
[20:46:42] <ottomata>	 which will mostly work ( as long as that one hour doesn't have loss), but is not ideal
[20:46:43] <nuria>	 ottomata: that seems that oozie must have a way to side step this issue
[20:46:51] <ottomata>	 ?
[20:46:59] <nuria>	 as in " i will not process 5th" but keep on going with "6th"
[20:47:02] <ottomata>	 oh
[20:47:04] <nuria>	 makes sense?
[20:47:08] <ottomata>	 it will not stop all future jobs
[20:47:20] <ottomata>	 it just won't launch a job for which it doesn't have all required datasets
[20:47:27] <ottomata>	 sorry
[20:47:30] <ottomata>	 misread your statement
[20:47:32] <ottomata>	 so
[20:47:33] <nuria>	 ahhhhhh
[20:47:36] <ottomata>	 if there is one hour missing from the 5th
[20:47:40] <ottomata>	 the daily 5th job will never run
[20:47:52] <ottomata>	 but as long as all data is present for the 6th
[20:47:55] <ottomata>	 the 6th  job will still run
[20:48:15] <nuria>	 mmmm... i think that is the right thing to do (what oozie is doing now)
[20:48:21] <ottomata>	 yes agree.  it is
[20:48:36] <ottomata>	 but until we have 100% data, or some automated way to check and mark data ok anyway, or to fix data
[20:48:39] <ottomata>	 this is going to be annoying
[20:48:43] <nuria>	 but if you want to be more tolerable with failure (ellery's case) then you can use the new dataset
[20:48:46] <ottomata>	 especially with daily  jobs
[20:48:50] <ottomata>	 yes.
[20:48:56] <ottomata>	 so, right now
[20:49:08] <ottomata>	 this change will make the refined data dependened only on the partiioned dataset
[20:49:12] <ottomata>	 not the checked dataset
[20:49:27] <ottomata>	 that way refinfed data jobs will always launch, even if they have faulty data
[20:49:31] <nuria>	 ya i kind of lean towards having the  the refined dataset  only have "good" data
[20:49:46] <ottomata>	 well, then folks will just do what ellery is doing now, or depend on the json one
[20:49:51] <ottomata>	 i think better
[20:49:57] <ottomata>	 would be to have the refeind data only have the 'pretty good data'
[20:50:03] <ottomata>	 that is, as long as percent_different is low
[20:50:07] <ottomata>	 in the seqeuence stats
[20:50:13] <ottomata>	 like, maybe, less than 1% off
[20:50:17] <ottomata>	 then go ahead and use the data
[20:50:17] <nuria>	 run, ya i understand
[20:50:27] <ottomata>	 but, i don' thave that yet
[20:50:32] <ottomata>	 and in the absence of that, i think this is better
[20:50:50] <ottomata>	 also, i'd like to have the faulty data visible, qchris is working on making an email report now
[20:50:52] <nuria>	 unless we have a major operational issue which this way might go unnoticed
[20:50:53] <ottomata>	 so we can always see what's missing
[20:51:00] <ottomata>	 yes.  but at least there are stats
[20:51:10] <ottomata>	 we have the info abou twhat is missing still
[20:51:16] <ottomata>	 so, researchers can always check
[20:51:20] <ottomata>	 if they think something is weird
[20:51:38] <nuria>	 but what about automated jobs ?
[20:51:50] <ottomata>	 sure, they will not notice unless there is something seriously wrong
[20:51:53] <nuria>	 we will not be cheking those , they are kind of fire and forget
[20:52:14] <ottomata>	 i mean, so is udp2log and webstats collector now.
[20:52:18] <ottomata>	 ideally there would be no loss
[20:52:28] <ottomata>	 but there is no way I am going to manually check on every single missing line right now
[20:52:34] <nuria>	 ottomata: of course
[20:52:36] <ottomata>	  i think qchris does that :p
[20:52:38] <ottomata>	 i will not :)
[20:53:03] <ottomata>	 i will spend time on making the missing data frequency rare
[20:53:19] <nuria>	 ottomata: but i think is peferable to have "less" data, how frequent are  the bad data/missing hours?
[20:53:26] <ottomata>	 for mobile and text, not frequent
[20:53:38] <nuria>	 i see
[20:53:38] <ottomata>	 but, when it does happen, the amount of missing data is very small
[20:53:45] <ottomata>	 like 0.007 percent, in one i just checked
[20:53:55] <wikibugs>	 Mobile-Web, Analytics-Engineering, Analytics-Visualization: generate.py produced broken CSV files - https://phabricator.wikimedia.org/T86059#982917 (Milimetric) so weird!  always 711, 84... hm... (debugging this guys, don't worry I'll fix it soon)  also, fyi, we're getting the green light to make this better...
[20:54:10] <ottomata>	 so, it sucks to hold jobs for which there are only a few missing lines
[20:54:14] <ottomata>	 e.g.
[20:54:19] <ottomata>	 if there are 1000 missing lines in a single hour
[20:54:25] <ottomata>	 then the whole daily job won't run
[20:54:39] <nuria>	 ottomata: i see, and who populates teh stats table?
[20:54:44] <ottomata>	 that is automated
[20:54:50] <nuria>	 ottomata: with the percentages of data missing?
[20:55:15] <ottomata>	 yup
[20:55:19] <ottomata>	 part of the load job
[20:55:20] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/generate_sequence_statistics.hql
[20:55:24] <nuria>	 ottomata: and we cannot tap on that automation and set the done flag accordingly?  (maybe this is scheduled for future work)
[20:55:54] <nuria>	 cause refined dataset could check that table right?
[20:56:09] <ottomata>	 nuria: that is what happens now
[20:56:14] <ottomata>	 if there is faulty data
[20:56:18] <ottomata>	 _SUCCESS never gets added
[20:56:32] <ottomata>	 future work is to make another done-flag stage
[20:56:37] <ottomata>	 _MOSTLY_GOOD
[20:56:38] <ottomata>	 heheh
[20:56:39] <ottomata>	 or whatever
[20:56:43] <nuria>	 _NOT_CRAP
[20:56:45] <ottomata>	 yeah
[20:56:52] <ottomata>	 where perecent_different is very low
[20:56:59] <ottomata>	 AND where min(sequence) != 0
[20:57:09] <ottomata>	 (that indicates a varnishkafka restart, not missing data)
[20:57:18] <nuria>	 ok, and we are planning on getting to this work in the near future ?
[20:57:22] <ottomata>	 well i mean
[20:57:25] <ottomata>	 i would like to
[20:57:32] <ottomata>	 but i doubt it will happen over the next few weeks
[20:57:44] <nuria>	 because your plate is too full?
[20:57:47] <ottomata>	 and i would not like ellery's jobs, ro the mobile apps jobs not to run
[20:57:50] <ottomata>	 because of travel etc.
[20:58:26] <nuria>	 ottomata: ok, let's do this, let's file a phabricator task that describes this problem, and the solution (which we have)
[20:58:37] <ottomata>	 aye k
[20:58:39] <nuria>	 ottomata: let's give it a very high priority
[20:58:39] <ottomata>	 pon it
[20:58:40] <ottomata>	 on it
[20:58:46] <ottomata>	 naw normal, let's not abuse high
[20:59:08] <nuria>	 and whoever is free can get to it next , maybe ananath, maybe you, manbe me, maybe madhu
[20:59:22] <nuria>	 (after bootsrapping)
[20:59:43] <nuria>	 and we list the phab task on the commit message of the gerrit change and the other way arround
[21:00:14] <nuria>	 cause this is not a super hard task, it's just a little time consuming but we know 100% of what needs to happen here
[21:01:27] <ottomata>	 well, i am not certain how to do it, it will require some oozie fanciness that i don't know exists yet
[21:01:48] <ottomata>	 checking the value of something and firing an action basd on it? i think qchris does it based on a filesize now, but comparing a value somewhere may be more difficult
[21:01:51] <ottomata>	 we will see
[21:01:59] <nuria>	 wait, where is the code that checks and sets "_SUCCESS" now?
[21:02:01] <wikibugs>	 Analytics-Cluster: Create another done-flag level for webrequest_*_raw oozie datasets that indicates the partition is mostly good - https://phabricator.wikimedia.org/T87060#982935 (Ottomata) NEW a:Ottomata
[21:02:02] <ottomata>	 done https://phabricator.wikimedia.org/T87060
[21:02:32] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/check_sequence_statistics_workflow.xml#L82
[21:02:44] <ottomata>	 which is included by
[21:02:55] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/workflow.xml#L119
[21:03:08] <nuria>	 but then wouldn't the _NOT_BAD be a subsequent workflow to this one?
[21:03:23] <ottomata>	 nuria, if you +1 that change, i will restart the load and refine  jobs now
[21:03:41] <ottomata>	 it could be, but i think we could just insert it as a nother action
[21:03:43] <ottomata>	 in the chain
[21:04:19] <nuria>	 instead of
[21:04:19] <ottomata>	 or as another decision case in check_sequence_statistics_workflow.xml
[21:04:31] <nuria>	 <ok to="mark_dataset_done"/>
[21:04:31] <nuria>	         <error to="kill"/>
[21:04:38] <ottomata>	 in addition to.
[21:04:40] <ottomata>	 before that
[21:04:40] <ottomata>	 first
[21:04:49] <nuria>	 it will be <error to="check_dataloss"/>
[21:04:50] <ottomata>	 check if NOT BAD, and if so add _NOT_BAD
[21:04:51] <ottomata>	 yes
[21:04:53] <ottomata>	 uh
[21:05:03] <ottomata>	 no, because we want _NOT_BAD AND _SUCCESS to exist
[21:05:05] <ottomata>	 so frist
[21:05:16] <ottomata>	 check if not bad, and if not bad write a _NOT_BAD file
[21:05:33] <ottomata>	 then move on and check if 100% good, and if 100% good, then write _SUCCESS
[21:05:39] <ottomata>	 something like that
[21:05:45] <ottomata>	 whatever, we can figure it out later,
[21:05:49] <nuria>	 so success overrides not_bad?
[21:05:55] <ottomata>	 gimme +1 i want to do this before I peace out for the weekend! :p
[21:06:14] <ottomata>	 like this:
[21:06:27] <ottomata>	 directory_exists < _PARTITIONED < _NOT_BAD < _SUCCESS
[21:06:56] <ottomata>	 oh i see you commented
[21:06:56] <ottomata>	 looking
[21:07:27] <grrrit-wm>	 (CR) Nuria: [C: 1] "Filed phab task to address halted jobs due to small dataloss in a more stylish fashion: https://phabricator.wikimedia.org/T87060#982935" (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/185538 (owner: Ottomata)
[21:07:50] <nuria>	 ottomata: done.
[21:08:05] <nuria>	 ottomata: it's fine my comment was to do with the badness
[21:08:15] <ottomata>	 thanks :)
[21:08:21] <ottomata>	 i will augment the comment too
[21:08:35] <grrrit-wm>	 (PS2) Ottomata: Add extra step in webrequest load that adds a _PARTITIONED done-flag [analytics/refinery] - https://gerrit.wikimedia.org/r/185538
[21:08:44] <ottomata>	 OO, btw, you just reminded me that I want to add de-uplication do the refine phase
[21:08:54] <ottomata>	 that way we can at least be sure there are no duplicates in the refined table :)
[21:08:56] <ottomata>	 another TODO! :)
[21:09:27] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add extra step in webrequest load that adds a _PARTITIONED done-flag [analytics/refinery] - https://gerrit.wikimedia.org/r/185538 (owner: Ottomata)
[21:11:05] <ottomata>	 uh oh, I am a little worried about the load step since I restarted the cluster... somethign is up.
[21:11:12] <ottomata>	 missing a lot of _SUCCESS files since then.  on it.
[21:13:04] <nuria>	 ottomata: ok, here, let me know if i can help
[21:32:09] <ottomata>	 OHHHHHHHHH
[21:32:11] <ottomata>	 i know why!
[21:36:44] <grrrit-wm>	 (PS3) Ewulczyn: first draft of host parsing udf. extracts the project and project qualifier from the uri_host. Added mobile_qualifier option. Changed tabs to spaces. Use '__null__' string to denote expected null values in test csv Change-Id: I3b48f0cf2b4836d452e00a460a7b [analytics/refinery/source] - https://gerrit.wikimedia.org/r/185377
[21:40:15] <ottomata>	 my fingers are crosseeeeeed, oozie jobs have not been submitting since hadoop cluster came back up
[21:40:20] <ottomata>	 i thikn i jsut fixed it, no luck yet htough
[21:40:22] <ottomata>	 cmonoooozie
[21:40:26] <ottomata>	 AH
[21:40:26] <ottomata>	 yes
[21:40:27] <ottomata>	 there is god
[21:40:28] <ottomata>	 goes
[21:40:31] <ottomata>	 patience!
[21:41:51] <milimetric>	 tnegrin: did you want to catch up before the meeting
[21:41:55] <ottomata>	 uh oh, no still problem, HMMM
[21:41:58] <milimetric>	 when we were talking earlier you weren't on
[21:42:15] <ottomata>	 gah, cached DNS
[21:42:16] <tnegrin>	 yes -- that's a good plan -- 2 o'clock?
[21:42:57] <ottomata>	 crap crackers, it is cached for an hour
[21:43:35] <milimetric>	 tnegrin: sure, meet up in 15 min. in the batcave then?
[21:43:42] <tnegrin>	 cool
[21:45:41] <grrrit-wm>	 (PS4) Ewulczyn: first draft of host parsing udf. extracts the project and project qualifier from the uri_host. Added mobile_qualifier option. Changed tabs to spaces. Use '__null__' string to denote expected null values in test csv Change-Id: I3b48f0cf2b4836d452e00a460a7b [analytics/refinery/source] - https://gerrit.wikimedia.org/r/185377
[21:49:24] <ottomata>	 phew, there it goes
[21:55:23] <ottomata>	 crappppi crackers, nope.  dns cached all around the cluster
[21:55:24] <ottomata>	 :( :(
[21:55:39] <ottomata>	 oozie just takes so long to tell me something failed!
[21:55:40] <ottomata>	 GAH
[22:02:03] <wikibugs>	 Mobile-Web, Analytics-Engineering, Analytics-Visualization: generate.py produced broken CSV files - https://phabricator.wikimedia.org/T86059#983110 (Milimetric) Open>Resolved The files had things like  711,84 in them because something (either generate.py or a person who had a editing hiccup while they w...
[22:02:38] <milimetric>	 can I help with anything ottomata?
[22:02:44] <milimetric>	 order you some pizza?  sounds bad
[22:02:52] <ottomata>	 naw, it isn't a huge emergency, i just wanna go do friday eve plans!
[22:02:59] <ottomata>	 that's what I get for doing a change on  friday!
[22:03:10] <ottomata>	 it is just keeping oozie jobs from launching
[22:03:19] <ottomata>	 but, what's nice about oozie, is that it should be easy to relaunch them
[22:03:22] <ottomata>	 if it doesn't do it on it sown
[22:03:27] <milimetric>	 ah, is there something that "should work" if ran at a certain time or something?
[22:03:31] <milimetric>	 i can do that, i'm around for a while
[22:03:42] <ottomata>	 well, i'm trying to force make it work now
[22:03:43] <ottomata>	 lemme see
[22:04:03] <milimetric>	 k, good luck and shoot me an email if i can help
[22:04:07] <milimetric>	 i'll be in a meeting for a bit
[22:04:08] <ottomata>	 ha ok, thanks
[22:05:14] <halfak>	 ottomata, if you've got some time before the EOD, I'm failing to xfer data to altiscale and could use another set of eyes.
[22:05:26] <halfak>	 See my attempts here: https://gist.github.com/halfak/7a965541648d6e979f46
[22:05:55] <halfak>	 Looks like I can scp just fine, but hdfs copy isn't working
[22:06:02] <halfak>	 *hdfs commands
[22:09:32] <ottomata>	 nuria: did your workflow apps test just die too?
[22:09:42] <ottomata>	 halfak: sorry, iam in emergency fix mode on friday eve already :/'
[22:09:47] <halfak>	 no worries
[22:09:55] <halfak>	 Will keep digging and expanding my notes.
[22:10:08] <halfak>	 I'm starting to think that it is port forwarding that is failing.
[22:12:29] <nuria>	 ottomata: ya, but they were tests so no big deal
[22:15:31] <grrrit-wm>	 (PS1) Yurik: Adjust launched date for long-running carriers [analytics/zero-sms] - https://gerrit.wikimedia.org/r/185559
[22:16:12] <ottomata>	 did it die?  i'm very confused as to what is happening right now
[22:16:33] <ottomata>	 it sseems the job succeeds according to yarn, but when wrapping up, oozie thinks it died, because it can't find the job id in yarn
[22:17:35] <grrrit-wm>	 (CR) Yurik: [C: 2 V: 2] Adjust launched date for long-running carriers [analytics/zero-sms] - https://gerrit.wikimedia.org/r/185559 (owner: Yurik)
[22:19:58] <nuria>	 ottomata: no, it did not started
[22:20:13] <nuria>	 ottomata: it said :
[22:20:17] <nuria>	 https://www.irccloud.com/pastebin/i09K9Zyx
[22:20:50] <nuria>	 https://www.irccloud.com/pastebin/E6KC7Vpy
[22:20:55] <nuria>	 i can look at logs
[22:21:29] <ottomata>	 aye
[22:21:32] <ottomata>	 i think i'm onto something...
[22:27:17] <nuria>	 ottomata: ok, i looked at logs but it's not the job as you already know
[22:27:37] <ottomata>	 ja, i think it is history server and oozie problem, possibly file permission in hdfs history
[22:27:55] <ottomata>	 oozie thinks the job doesn't exist after it finishes
[22:28:00] <ottomata>	 so it kills everything else that it would have to do
[22:28:14] <ottomata>	 not certain this is the problem
[22:28:21] <nuria>	 ottomata: i see, taht adds up with logs appearing as partial
[22:28:23] <ottomata>	 i'm about 65% sure
[22:28:24] <ottomata>	 yes
[22:28:36] <nuria>	 ottomata: it's not proceeding to hive step
[22:44:45] <ottomata>	 phew, nuria, think is is better
[22:45:35] <ottomata>	 -rw-r--r--   3 hdfs analytics-privatedata-users          0 2015-01-16 22:44 /wmf/data/raw/webrequest/webrequest_mobile/hourly/2015/01/16/14/_PARTITIONED
[22:45:37] <ottomata>	 that's good
[22:56:32] <ottomata>	 ok, nuria, i think things are fixed
[22:56:38] <ottomata>	 refine is lookiing good too
[22:56:47] <ottomata>	 starti g at 2015-01-16T14
[22:56:57] <ottomata>	 i'm not going to relauch the mobile apps one
[22:57:04] <ottomata>	 cause we want to make some changes thre anyway, right?
[23:00:31] <ottomata>	 phew ok, laters all!
[23:38:00] <milimetric>	 later everyone