[00:05:34] (CR) Mforns: [C: 1] "LGTM!" (3 comments) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [00:38:12] (CR) Mforns: "LGTM" (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [00:38:19] (CR) Mforns: [C: 1] Add basic monitoring script for projectcount aggregates [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [00:43:21] (CR) Mforns: [C: 1] "LGTM" (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [01:05:41] (PS6) Nuria: Add Separated Values (CSV, TVS) data converter [analytics/dashiki] - https://gerrit.wikimedia.org/r/168488 (owner: Milimetric) [01:07:46] (PS7) Nuria: Add Separated Values (CSV, TVS) data converter [analytics/dashiki] - https://gerrit.wikimedia.org/r/168488 (owner: Milimetric) [01:08:25] (CR) Nuria: Add Separated Values (CSV, TVS) data converter (4 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/168488 (owner: Milimetric) [10:17:02] Analytics / EventLogging: Get back to 90 days of logs on vanadium - https://bugzilla.wikimedia.org/69029 (christian) PATC>RESO/FIX [11:17:58] (PS3) QChris: Add basic implementation of projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) [11:18:00] (PS3) QChris: Add switch for automatic pushing of data repo for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172198 (https://bugzilla.wikimedia.org/72740) [11:18:02] (PS3) QChris: Allow additional logging to disk for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740) [11:18:04] (PS3) QChris: Add basic monitoring script for projectcount aggregates [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) [11:18:06] (PS3) QChris: Add total column for project aggregation CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237 [11:18:08] (PS3) QChris: Allow to force recomputation of existing data [analytics/aggregator] - https://gerrit.wikimedia.org/r/172236 [11:18:10] (PS3) QChris: Switch to CRLF as line terminator for CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172235 [11:18:12] (PS3) QChris: Add header line to CSVs for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172234 [11:18:14] (PS1) QChris: Sort list of CSVs before iterating over them for projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/173257 [11:27:17] (CR) QChris: Add basic implementation of projectcount aggregation (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [11:33:26] (CR) QChris: Add basic monitoring script for projectcount aggregates (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [11:36:53] (CR) QChris: Allow additional logging to disk for projectcounts aggregation (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [12:15:46] qchris: I am adding in analytics-aggregator-tox-py27 ( https://gerrit.wikimedia.org/r/#/c/172954/ ) [12:16:19] Great! [12:16:35] However, I am oin a meeting. back in ~45 minutes [12:23:09] (CR) Hashar: "recheck" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [12:23:26] pass https://integration.wikimedia.org/ci/job/analytics-aggregator-tox-py27/1/ [12:23:31] qchris_meeting: all done. Happy meeting [14:00:22] hashar: Yaaaay! Thanks a lot! [14:55:47] Analytics / Refinery: Most raw webrequest partitions for 2014-10-13T20/1H not marked successful - https://bugzilla.wikimedia.org/73418 (christian) NEW p:Unprio s:normal a:None Three of the webrequest partitions [1] for 2014-10-13T20/1H have been been marked successful. What happened? [1]... [14:58:47] Analytics / Refinery: Most raw webrequest partitions for 2014-10-13T20/1H not marked successful - https://bugzilla.wikimedia.org/73418#c1 (christian) The three jobs for 2014-11-13T20/1H were in SUSPENDED state. Some internal workflows got stuck with exception about RM issues [1]. This nicely matches y... [14:58:47] Analytics / Refinery: Raw webrequest partitions that were not marked successful due to configuration updates - https://bugzilla.wikimedia.org/72300 (christian) [15:35:58] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22262 (Ottomata) Things are looking fine. I'm going to continue with analytics1024 and then analytics1025. Once they are upgraded I will modify puppet so it is happy again. [15:37:43] Wait ... wmbugs? is now called phawkes ? :-D [15:37:52] I AM NOT WEARING MASKS! [15:44:37] phabricator! [15:45:05] phuntastic :-) [15:48:02] Analytics / Refinery: Most raw webrequest partitions for 2014-10-13T20/1H not marked successful - https://bugzilla.wikimedia.org/73418#c2 (christian) NEW>RESO/FIX Now the jobs succeeded, and the partitions got marked ok. [16:25:34] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22266 (Ottomata) [16:49:56] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22283 (Ottomata) I just noticed the same diskfilter warning on analytics1024. I think we should apply this to each node after we upgrade. I've attached the patched 00_header file. {F826} Steps are... [16:58:33] Hi, leila. Is Ori's fix still pending on Ops approval? [16:58:47] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22285 (Ottomata) analytics1024 is done, moving on to analytics1025. [16:58:56] Hi Joel. I'm not sure, but I can see if we have events [16:59:02] jsahleen, ^ [17:00:02] leila: Thanks. There may not have been any events sent since the fix. I will trigger one now. [17:00:05] jsahleen, do you expect to see any events from last night to this morning? [17:00:10] okay. thanks! [17:00:14] let me know once you have that [17:02:18] Ok. Done. [17:13:48] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [17:20:48] (CR) Mforns: [C: 1] "LGTM" (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [17:22:27] (CR) Mforns: [C: 1] Allow additional logging to disk for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [17:28:06] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172198 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [17:30:23] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172234 (owner: QChris) [17:32:23] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172235 (owner: QChris) [17:34:17] hashar, around? [17:34:51] trying to figure out how wp-zero git repo is functioning, we are getting very different results between our own analytics and LIMN graphs [17:38:36] yurikR1: hop sorry. bugzilla is the preferred input [17:38:42] or the lists :] [17:39:14] hehe, i thought we moved to the ph* [17:40:01] hashar, basically i am just looking for some starting points dicyphering those scripts, namely - where are the conditions that say "include/explude" a certain log line from counting [17:41:32] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator] - https://gerrit.wikimedia.org/r/173257 (owner: QChris) [17:50:24] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172181 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:06:08] (CR) Mforns: [C: 1] "The generation looks good." [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172182 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:11:52] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22295 (Ottomata) analytics1025 is done. [18:12:01] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172183 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:13:53] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22296 (Ottomata) Kafkas are next: analytics1012, 1018, 1021, 1022. Will do them one at a time in that order. [18:18:04] (CR) Mforns: "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172185 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:22:19] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22297 (Ottomata) Actually, on second thought, there is no immediate need to upgrade the Kafka servers to Trusty, and to do so I would have to build a Trusty version of the .deb. Kafka 0.8.2 will be r... [18:22:46] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#22298 (Ottomata) Next up is the Hadoop cluster then. Since it is Friday, I will wait until next week before continuing. [18:26:47] (CR) Mforns: [C: 1] Backfill daily projectcounts for 2010 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172185 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:30:44] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172186 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:33:34] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172187 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:37:21] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172188 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:41:43] (CR) Mforns: [C: -2] "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172189 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:42:14] (CR) Mforns: [C: 1] "Sorry, I messed up." [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172189 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:44:17] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172190 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [19:35:01] Analytics / EventLogging: Beta setup of event logging not working - https://bugzilla.wikimedia.org/73388#c3 (nuria) Rebooted machine & manually updated code, looks like there are events appearing now. Not sure if any other changes took place. [19:38:15] hey guys how do I stop a running wikimetrics server? [19:38:30] hi baha [19:38:38] inside vagrant? [19:38:52] you can go sudo service wikimetrics-web stop [19:39:08] there's also wikimetrics-queue service running [19:39:35] maybe you want to stop both [19:39:48] mforns: hi, thanks. yes inside vagrant. it's interfering with limn server, which also listens to 5000 [19:40:04] ok, try: [19:40:10] sudo service wikimetrics-web stop [19:40:14] sudo service wikimetrics-queue stop [19:40:31] mforns: great, thanks [19:40:33] worked [19:40:40] good! [19:42:29] is that a competition between religious beliefs? [19:42:51] oops, wrong channel [19:42:56] xD [19:46:30] (Abandoned) Nuria: Adding stubs for pageview MVP [analytics/dashiki] - https://gerrit.wikimedia.org/r/172480 (owner: Nuria) [20:00:47] Analytics / Visualization: limn-mobile-data requirements.txt file needs distribute version - https://bugzilla.wikimedia.org/73431 (Bahodir Mansurov) NEW p:Unprio s:normal a:None I have distribute-0.7.3 installed, but I cannot compile MySQL-python==1.2.4. However, I am able to install MySQL-... [20:05:32] Analytics / Visualization: limn-mobile-data requirements.txt file is missing python-dateutil - https://bugzilla.wikimedia.org/73432 (Bahodir Mansurov) NEW p:Unprio s:normal a:None The python-dateutil module is used in the code but is not present in the requirements.txt [23:20:08] (CR) Mforns: [C: 2] Backfill daily projectcounts up to 2014-11-08 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172190 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:20:29] (CR) Mforns: [C: 2] Backfill daily projectcounts for 2014 up to 2014-09-22 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172189 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:20:39] (CR) Mforns: [C: 2] Backfill daily projectcounts for 2013 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172188 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:20:56] (CR) Mforns: [C: 2] Backfill daily projectcounts for 2012 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172187 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:21:06] (CR) Mforns: [C: 2] Backfill daily projectcounts for 2011 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172186 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:21:16] (CR) Mforns: [C: 2] Backfill daily projectcounts for 2010 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172185 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:21:27] (CR) Mforns: [C: 2] Backfill daily projectcounts for 2009 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172183 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:21:48] (CR) Mforns: [C: 2] Backfill daily projectcounts for 2008 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172182 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:21:58] (CR) Mforns: [C: 2] Create empty CSVs for relevant wikis [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172181 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:22:44] (CR) Mforns: [C: 2] Allow to force recomputation of existing data [analytics/aggregator] - https://gerrit.wikimedia.org/r/172236 (owner: QChris) [23:22:52] (CR) Mforns: [C: 2] Add total column for project aggregation CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237 (owner: QChris) [23:23:03] (CR) Mforns: [C: 2] Add basic implementation of projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:14] (CR) Mforns: [C: 2] Add basic monitoring script for projectcount aggregates [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:18] (Merged) jenkins-bot: Add basic implementation of projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:20] (Merged) jenkins-bot: Add basic monitoring script for projectcount aggregates [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:23] (CR) Mforns: [C: 2] Allow additional logging to disk for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:31] (CR) Mforns: [C: 2] Add switch for automatic pushing of data repo for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172198 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:41] (CR) Mforns: [C: 2] Add header line to CSVs for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172234 (owner: QChris) [23:23:44] (Merged) jenkins-bot: Allow additional logging to disk for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:48] (Merged) jenkins-bot: Add switch for automatic pushing of data repo for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172198 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:23:51] (Merged) jenkins-bot: Add header line to CSVs for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172234 (owner: QChris) [23:24:05] (CR) Mforns: [C: 2] Switch to CRLF as line terminator for CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172235 (owner: QChris) [23:24:12] (Merged) jenkins-bot: Switch to CRLF as line terminator for CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172235 (owner: QChris) [23:24:14] (Merged) jenkins-bot: Allow to force recomputation of existing data [analytics/aggregator] - https://gerrit.wikimedia.org/r/172236 (owner: QChris) [23:24:16] (CR) Mforns: [C: 2] Sort list of CSVs before iterating over them for projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/173257 (owner: QChris) [23:24:18] (Merged) jenkins-bot: Add total column for project aggregation CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237 (owner: QChris) [23:24:23] (Merged) jenkins-bot: Sort list of CSVs before iterating over them for projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/173257 (owner: QChris) [23:24:31] such merging [23:29:46] (CR) Mforns: [V: 2] Backfill daily projectcounts for 2011 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172186 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:35:37] nuria__: Hi, I'd like to work on https://bugzilla.wikimedia.org/show_bug.cgi?id=72116. Any objections? [23:36:57] bmansurov: no, sounds good, now other timezones should be available for "input" only (to keep things scoped) [23:37:05] output of dates will still be in utc [23:37:14] bmansurov: makes sense? [23:37:27] bmansurov: output as in what the UI displays [23:37:55] nuria__: kind of, so the user is allowed to enter in different timezones, which I will convert to UTC for consistency? [23:40:47] bmansurov: right [23:40:54] there is moment.js [23:40:59] which handles all that [23:41:07] (CR) Mforns: [V: 2] Backfill daily projectcounts for 2010 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172185 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:41:09] bmansurov: alredy at your disposal [23:41:18] (CR) Mforns: [V: 2] Backfill daily projectcounts for 2009 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172183 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:41:21] nuria__: ok thanks [23:41:29] (CR) Mforns: [V: 2] Backfill daily projectcounts for 2008 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172182 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:41:32] bmansurov: thanks to you man! [23:41:54] np, this stuff is interesting [23:43:08] Analytics / Wikimetrics: Accept more timezones as input - https://bugzilla.wikimedia.org/72116 (Bahodir Mansurov) a:Bahodir Mansurov [23:43:38] (CR) Mforns: [V: 2] Create empty CSVs for relevant wikis [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172181 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:43:49] (CR) Mforns: [V: 2] Backfill daily projectcounts up to 2014-11-08 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172190 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:43:59] (CR) Mforns: [V: 2] Backfill daily projectcounts for 2014 up to 2014-09-22 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172189 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:44:28] (CR) Mforns: [V: 2] Backfill daily projectcounts for 2013 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172188 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [23:44:36] (CR) Mforns: [V: 2] Backfill daily projectcounts for 2012 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172187 (https://bugzilla.wikimedia.org/72740) (owner: QChris)