[03:48:50] <wikibugs>	 Analytics-EventLogging: Add sampling support in EventLogging - https://phabricator.wikimedia.org/T67500#994061 (Nuria) >mw.eventLog.logWithSampling( 'Foo', 100, data ) I certainly agree that having a method that eases logging with sampling on the js end  is a must.   >If I need to look up EL data from three m...
[04:54:36] <wikibugs>	 Analytics-EventLogging: Add sampling support in EventLogging - https://phabricator.wikimedia.org/T67500#994095 (Tgr) In my experience with MediaViewer, we rarely cared about absolute numbers (there were cases when we did, but only a few), but we cared a lot about relative differences between various events (d...
[14:06:25] <grrrit-wm1>	 (PS2) QChris: Load and refine logs from 'misc' caches [analytics/refinery] - https://gerrit.wikimedia.org/r/184191
[14:06:27] <grrrit-wm1>	 (PS1) QChris: Refine bits and upload webrequests [analytics/refinery] - https://gerrit.wikimedia.org/r/186774
[14:06:29] <grrrit-wm1>	 (PS1) QChris: Fix legacy_tsvs' use of datasets after switch to refined tables [analytics/refinery] - https://gerrit.wikimedia.org/r/186775
[14:06:31] <grrrit-wm1>	 (PS1) QChris: Use bits and misc when producing legacy tsvs [analytics/refinery] - https://gerrit.wikimedia.org/r/186776
[14:06:33] <grrrit-wm1>	 (PS1) QChris: Sort webrequest_sources in legacy_tsvs' HiveQL files [analytics/refinery] - https://gerrit.wikimedia.org/r/186777
[14:06:35] <grrrit-wm1>	 (PS1) QChris: Clarify why there are no bits or misc requests in sampled-1000 [analytics/refinery] - https://gerrit.wikimedia.org/r/186778
[14:30:40] <grrrit-wm1>	 (CR) QChris: "The corresponding change to get webrequest_misc into kafka is at Ia3712a0d85ffa893912af2dff312017f03a7a935." [analytics/refinery] - https://gerrit.wikimedia.org/r/184191 (owner: QChris)
[14:44:06] <wikibugs>	 Wikidata, Analytics, wikidata-query-service, operations, Services, MediaWiki-General-or-Unknown: Reliable publish / subscribe event bus - https://phabricator.wikimedia.org/T84923#994455 (JanZerebecki) >>! In T84923#993443, @GWicke wrote: > Since 0mq is not actually durable or replicated this does not cover th...
[14:55:42] <wikibugs>	 Analytics: analytics-logbot is no longer in the #wikimedia-analytics channel - https://phabricator.wikimedia.org/T85698#994484 (QChris) Open>Resolved a:QChris The bot returned somewhen in the last few days.  Not sure who restarted the service, but whoever did it: Thanks! :-)
[17:18:51] <wikibugs>	 MediaWiki-Developer-Summit-2015, Analytics-EventLogging: Using EventLogging and Dashboards - https://phabricator.wikimedia.org/T85280#994594 (kevinator)
[17:50:35] <wikibugs>	 MediaWiki-extensions-UniversalLanguageSelector, Analytics, Wikipedia-App-iOS-App, Wikipedia-App-Android-App, Language-Engineering, Mobile-Web, Mobile-Apps: there should be a comparison of clicks count on interlanguage on different platforms - https://phabricator.wikimedia.org/T78351#994626 (Amire80)
[18:00:00] <DarTar>	 is it just me or hadoop is ridiculously slow today? just launching a hive query takes forever
[18:22:34] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] "HA! that's it! Amazing." [analytics/refinery] - https://gerrit.wikimedia.org/r/177522 (owner: QChris)
[18:29:22] <grrrit-wm1>	 (CR) Ottomata: [C: 1] "Just a few comments. Aside from those, LGTM. Just respond to them however you see fit, and then feel free to self merge this, just in ca" (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/185708 (owner: QChris)
[18:52:52] <wikibugs>	 Analytics-Dashiki: Commons page views in webstatscollector drop precipitously in 2015 - https://phabricator.wikimedia.org/T87589#994717 (Tnegrin) NEW a:kevinator
[19:01:59] <grrrit-wm1>	 (CR) Ottomata: "wmf.webrequest is external!" [analytics/refinery] - https://gerrit.wikimedia.org/r/185179 (owner: QChris)
[19:05:34] <grrrit-wm1>	 (CR) Ottomata: "Hm, I dunno. I actually really like this feature. Why should a job that doesn't need particular data depend on that data. Even if in th" [analytics/refinery] - https://gerrit.wikimedia.org/r/186775 (owner: QChris)
[19:06:32] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Sort webrequest_sources in legacy_tsvs' HiveQL files [analytics/refinery] - https://gerrit.wikimedia.org/r/186777 (owner: QChris)
[19:06:54] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Clarify why there are no bits or misc requests in sampled-1000 [analytics/refinery] - https://gerrit.wikimedia.org/r/186778 (owner: QChris)
[19:10:23] <grrrit-wm1>	 (CR) Ottomata: "Perhaps the unneeded dependencies are just an artifact of the abstractions of all legacy_tsvs into a single coordinator. Should we consid" [analytics/refinery] - https://gerrit.wikimedia.org/r/186775 (owner: QChris)
[19:14:44] <wikibugs>	 Analytics-Wikimetrics: Uploading cohort or running a large report fails - https://phabricator.wikimedia.org/T87596#994809 (kevinator) NEW
[19:42:52] <wikibugs>	 Analytics, Wikidata, wikidata-query-service, operations, Services, MediaWiki-General-or-Unknown: Reliable publish / subscribe event bus - https://phabricator.wikimedia.org/T84923#994876 (mobrovac) Re: reliability, [RELP](http://www.rsyslog.com/doc/relp.html) might be of help on the application level.
[20:31:56] <wikibugs>	 Analytics-Engineering: Add time range selection to Limn dashboards (or new Dashiki dashboards) - https://phabricator.wikimedia.org/T87603#994953 (Milimetric) NEW a:Milimetric
[20:35:59] <wikibugs>	 Analytics-Engineering: Add ops-reportcard dashboard with analysis that shows the http to https slowdown on russian wikipedia - https://phabricator.wikimedia.org/T87604#994962 (Milimetric) NEW a:Milimetric
[21:36:29] <qchris>	 ottomata: Thanks for the merge and the comments!
[21:38:46] <ottomata>	 qchris:  hello!
[21:38:48] <ottomata>	 yup :)
[21:38:53] <ottomata>	 thanks for the poke :)
[21:49:43] <wikibugs>	 Analytics, operations: Hadoop logs on logstash are being really spammy - https://phabricator.wikimedia.org/T87206#995149 (Ottomata) Merged, but Hadoop daemons will need to be restarted to pick up this change.  If you don't mind waiting, I will likely be restarting all of them soon (hopefully within the next f...
[21:57:21] <grrrit-wm1>	 (PS3) QChris: Explain why tables that should be external are internal [analytics/refinery] - https://gerrit.wikimedia.org/r/185179
[21:58:22] <grrrit-wm1>	 (PS4) Ottomata: Explain why tables that should be external are internal [analytics/refinery] - https://gerrit.wikimedia.org/r/185179 (owner: QChris)
[21:58:31] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Explain why tables that should be external are internal [analytics/refinery] - https://gerrit.wikimedia.org/r/185179 (owner: QChris)
[21:58:44] <qchris>	 You so fast to merge!
[21:59:17] <grrrit-wm1>	 (CR) QChris: "> wmf.webrequest is external!" [analytics/refinery] - https://gerrit.wikimedia.org/r/185179 (owner: QChris)
[21:59:58] <ottomata>	 vrrroooooom
[22:04:49] <qchris>	 :-D
[22:05:03] <ottomata>	 keep poking me, these are fun! :)
[22:05:07] <ottomata>	 and easy to do for the most part
[22:05:12] <ottomata>	 while i'm watching james F talk about the future of editing
[22:05:22] <ottomata>	 with 20 people all editing an etherpad behind him on the projector
[22:05:35] <qchris>	 k. In a few minutes I'll convince you to merge the legacy_tsv dependency change :-)
[22:05:49] <qchris>	 Whoa. Sounds cool.
[22:06:00] <ottomata>	 it is distracting and hilarious
[22:06:10] <ottomata>	 but interesting too!
[22:06:11] <qchris>	 Link to the etherpad?
[22:06:20] <ottomata>	 https://etherpad.wikimedia.org/p/MWDS2015-FutureOfEditing
[22:06:39] <qchris>	 Thanks
[22:06:56] <qchris>	 32 participants in the etherpad :-D
[22:13:37] <grrrit-wm1>	 (CR) QChris: "> Perhaps the unneeded dependencies are just an artifact of the" [analytics/refinery] - https://gerrit.wikimedia.org/r/186775 (owner: QChris)
[22:23:38] <grrrit-wm1>	 (CR) Ottomata: "> As there are no missing datasets, missing datasets can no longer block. So it's ok to depend on all of them and be on the safe side." [analytics/refinery] - https://gerrit.wikimedia.org/r/186775 (owner: QChris)
[22:27:52] <ottomata>	 qchris: maybe live chat about this one real quick?
[22:28:04] <ottomata>	 you can probably convince me, especially because I want this stuff running sooner rather than later :)
[22:28:15] <qchris>	 I was just about to respond to it.
[22:28:23] <ottomata>	 ok
[22:28:26] <ottomata>	 das fine
[22:28:33] <qchris>	 So ... you know ... I am not too fond of the way refined datasets are currently set up.
[22:28:48] <qchris>	 But I figured that the current setup is the way you want it,
[22:28:52] <qchris>	 so I wanted to roll with it.
[22:29:01] <ottomata>	 yes, because they are not verified and because the data quality is not currently visible
[22:29:10] <ottomata>	 right?
[22:29:13] <qchris>	 Right.
[22:29:16] <qchris>	 But that's ok.
[22:29:25] <qchris>	 And if one turns off bits again,
[22:29:35] <qchris>	 The refined partitions will get created,
[22:29:38] <qchris>	 but they are empty.
[22:29:46] <qchris>	 So they are ok is some sense,
[22:29:49] <ottomata>	 oh?
[22:29:50] <ottomata>	 really?
[22:29:51] <qchris>	 and Oozie jobs should run on them.
[22:30:03] <ottomata>	 camus creates the directories even if it doesn't have any timestamps to put there?
[22:30:13] <qchris>	 No.
[22:30:25] <qchris>	 Oh. Sorry.
[22:30:42] <qchris>	 I screwed up, by mixing the turning off of esams bits with turning off bits fully.
[22:30:59] <qchris>	 For only esams bits turned off, it created the directories.
[22:31:10] <qchris>	 Obviously, because the other bits were present.
[22:31:17] <ottomata>	 aye, yeah, no bits directories since the 14th :)
[22:32:13] <qchris>	 If you turn off bits and want only those jobs to run that do not depend on bits,
[22:32:23] <qchris>	 the trick I used for raw would not help.
[22:32:32] <qchris>	 Hence we'd really need separate coordinators.
[22:32:45] <ottomata>	 why wouldn't the trick work?
[22:33:03] <qchris>	 Because I'd need an "raw_unchecked" dataset to exist.
[22:33:10] <qchris>	 And if you turn it off completely,
[22:33:25] <qchris>	 the bits "raw_unchecked" would not be marked done either.
[22:33:49] <qchris>	 So, we're basically back to 4 coordinators (which can share the same workflow).
[22:33:55] <qchris>	 It's easy to setup.
[22:33:59] <ottomata>	 oh, yes.
[22:34:00] <ottomata>	 true
[22:34:01] <qchris>	 But probably tedious to maintain.
[22:34:19] <ottomata>	 hmMMMmmmMMmmMmm
[22:34:32] <qchris>	 Like having to propagate any change to all four coordinators.
[22:34:41] <ottomata>	 tedious in that we have to manage them separately in oozie
[22:34:44] <ottomata>	 restart all 4 of them
[22:34:46] <ottomata>	 etc.
[22:34:48] <ottomata>	 oh
[22:34:59] <ottomata>	 and edit all of them for changes in  <property>
[22:35:01] <ottomata>	 yeah
[22:35:02] <ottomata>	 :
[22:35:02] <ottomata>	 :?
[22:35:04] <ottomata>	 :/
[22:35:10] <ottomata>	 but, then again, oozie is already tedious
[22:35:11] <ottomata>	 heh
[22:35:31] <qchris>	 Mhmmm ... I guess I could make the trick work if we add a dummy partition, that automatically gets added and marked done.
[22:35:42] <qchris>	 But that's ugly.
[22:35:54] <qchris>	 Well ... :-/
[22:36:08] <qchris>	 We're building against a case that I doubt it will become relevant.
[22:36:17] <qchris>	 So it's hard to say how likely one will need it.
[22:36:20] <ottomata>	 aye
[22:36:30] <ottomata>	 i feel like the right thing to do is to have multiple coordinators :/
[22:36:44] <qchris>	 k.
[22:36:48] <ottomata>	 i mean
[22:36:50] <qchris>	 Multiple coordinators it is.
[22:36:52] <ottomata>	 maybe not?  right?
[22:36:58] <ottomata>	 i feel ya on the tediousness
[22:37:07] <ottomata>	 but the artificial dependencies feel way worse to me
[22:38:04] <qchris>	 By artificial you mean the fake ones on "raw_unchecked", or the unneeded ones (like bits for the mobile-sampled-100)?
[22:38:28] <ottomata>	 uh, undeeded ones
[22:38:34] <qchris>	 k.
[22:38:58] <qchris>	 Then it sounds like we want the multiple coordinators.
[22:39:01] <ottomata>	 haha
[22:39:05] <ottomata>	 i think so!?
[22:39:13] <qchris>	 Yes, you want them!
[22:39:14] <ottomata>	 that sounds right to me, but I am not decreeing it!
[22:39:22] <qchris>	 YOU WANT THEM !!!!!
[22:39:25] <ottomata>	 hahaha
[22:39:32] <qchris>	 :-P
[22:40:19] <qchris>	 If I've got you already ... about the pagecounts-raw workflow directories ...
[22:40:29] <qchris>	 I do not find "output_archive_directory" or "dataset_archive_directory" too convincing, as they do not really conway in which way they differ from the "archive_directory" from the properties file.
[22:40:29] <qchris>	 In general "archive_directory" is "/wmf/data/archive", and we want a subdirectory underneath that.
[22:40:29] <qchris>	 What we want is the directory that is specific to the workflow's configuration (either pagecounts-all-sites or pagecounts-raw).
[22:40:29] <qchris>	 Would "workflow_specific_archive_directory" work for you?
[22:40:31] <qchris>	 "pagecounts_kind_archive_directory",
[22:40:33] <qchris>	 "pagecounts_variant_archive_directory"?
[22:42:30] <qchris>	 ottomata: ^
[22:43:58] <ottomata>	 how about jsut pagecounts_archive_directory
[22:44:04] <ottomata>	 ?
[22:44:27] <qchris>	 Is thaht also ok for the jobs that generate projectcoutns files?
[22:44:41] <ottomata>	 hm. they use the same property?
[22:44:48] <qchris>	 Yes.
[22:44:51] <ottomata>	 hm, then no. :p
[22:45:11] <qchris>	 It's where the workflow will put the generated file.
[22:45:40] <qchris>	 And projectcounts is kind of a "pagecounts_variant" ...
[22:46:15] <ottomata>	 qchris, how about just
[22:46:18] <ottomata>	 output_directory?
[22:46:26] <ottomata>	 this is where the workflows output will go
[22:46:37] <qchris>	 Mhmm.
[22:46:55] <qchris>	 Isn't that too generic.
[22:47:04] <qchris>	 An "output_directory" could be anything.
[22:47:07] <qchris>	 Meh.
[22:47:08] <ottomata>	 naw, why?  it is a variable name
[22:47:13] <qchris>	 I guess it's ok.
[22:47:18] <ottomata>	 yes!
[22:47:21] <ottomata>	 it is ok!
[22:47:30] <ottomata>	 it is not a directory of archived workflows
[22:47:33] <qchris>	 After all ... this is not in the properties file, but only lives from the coordinator onwards.
[22:47:35] <ottomata>	 it is the output of some job!
[22:47:37] <ottomata>	 yes
[22:47:43] <ottomata>	 but can be overridden!
[22:47:49] <qchris>	 Ok. output_directory it is.
[22:47:51] <ottomata>	 yes!
[22:47:52] <qchris>	 Thanks!
[22:47:59] <ottomata>	 i am getting my way all over the place today!
[22:48:00] <ottomata>	 :)
[22:48:17] <qchris>	 You're a winner! :-D
[22:48:21] <ottomata>	 hahaha
[22:55:05] <grrrit-wm1>	 (PS2) QChris: Add pagecounts-raw computation to pagecounts-all-sites [analytics/refinery] - https://gerrit.wikimedia.org/r/185708
[22:55:41] <grrrit-wm1>	 (CR) QChris: Add pagecounts-raw computation to pagecounts-all-sites (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/185708 (owner: QChris)
[22:58:28] <grrrit-wm1>	 (PS2) QChris: Clarify why there are no bits or misc requests in sampled-1000 [analytics/refinery] - https://gerrit.wikimedia.org/r/186778
[23:03:02] <grrrit-wm1>	 (CR) QChris: [V: 2] Clarify why there are no bits or misc requests in sampled-1000 [analytics/refinery] - https://gerrit.wikimedia.org/r/186778 (owner: QChris)
[23:08:08] <grrrit-wm1>	 (CR) QChris: [C: -1] "Per IRC discussion on 2015-01-26 in wikimedia-analytics," [analytics/refinery] - https://gerrit.wikimedia.org/r/186775 (owner: QChris)
[23:09:05] <grrrit-wm1>	 (PS3) Ottomata: Add pagecounts-raw computation to pagecounts-all-sites [analytics/refinery] - https://gerrit.wikimedia.org/r/185708 (owner: QChris)
[23:09:15] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Add pagecounts-raw computation to pagecounts-all-sites [analytics/refinery] - https://gerrit.wikimedia.org/r/185708 (owner: QChris)
[23:10:06] <qchris>	 Thanks!
[23:12:23] <qchris>	 ottomata: How do feel about the changes to add misc to kafka?
[23:12:28] <qchris>	 (I'll call it a day soon and need something to work on tomorrow besides deploying and add the coordinators for the legacy_tsvs :-) )
[23:14:53] <ottomata>	 hmm
[23:14:58] <ottomata>	 ja can work on that i think
[23:15:01] <ottomata>	 looking
[23:15:25] <qchris>	 (Changes for it are already in gerrit)
[23:15:38] <ottomata>	 bits too?
[23:15:55] <qchris>	 https://gerrit.wikimedia.org/r/#/c/186641/
[23:16:02] <qchris>	 ottomata: ^ is bits re-enabling.
[23:16:20] <qchris>	 (Hopefully :-) )
[23:17:48] <ottomata>	 yeah saw it, just was wondering if we really wanted to turn it back on
[23:17:59] <ottomata>	 i quieted most of the upload alerts so they wouldn't spam the ops room.
[23:18:19] <qchris>	 The 5xx legacy tsvs would include bits.
[23:18:22] <ottomata>	 i guess we do
[23:18:23] <ottomata>	 yeah
[23:18:29] <qchris>	 But we can keep them out for now if you prefer.
[23:18:40] <ottomata>	 dunno.
[23:18:56] <ottomata>	 i mean,i prefer to turn it back on, but i might prefer to do it when i am back on regular work hours
[23:19:01] <ottomata>	 so i can respond to things more easily
[23:19:11] <ottomata>	  misc  no problem
[23:19:12] <qchris>	 k.
[23:19:13] <ottomata>	 we can do that now
[23:19:20] <qchris>	 Then let's leave bits out for now.
[23:19:22] <ottomata>	 but it depends on bits
[23:19:24] <ottomata>	 that change
[23:19:34] <qchris>	 I'll rebase.
[23:20:06] <ottomata>	 k
[23:20:23] <qchris>	 https://gerrit.wikimedia.org/r/184183
[23:20:25] <qchris>	 Done ^
[23:21:58] <qchris>	 I guess I'll leave splitting the corresponding refinery change for tomorrow (so I can test them a bit)
[23:22:23] <ottomata>	 ok
[23:24:30] <ottomata>	 there they go!
[23:24:44] <ottomata>	 qchris:
[23:24:51] <ottomata>	 we should merge the camus change
[23:24:53] <ottomata>	 whatcha think?
[23:25:15] <qchris>	 Sure.
[23:25:20] <qchris>	 But I guess thaht needs rebasing too.
[23:25:20] <ottomata>	 oh i see it
[23:25:21] <ottomata>	 coool
[23:25:23] <ottomata>	 ah and refine
[23:25:24] <qchris>	 Let me check.
[23:25:25] <ottomata>	 yes i'm for it.
[23:25:31] <ottomata>	 it looks good i think
[23:25:34] <ottomata>	 just adds everything for misc
[23:25:39] <qchris>	 ok.
[23:25:41] <ottomata>	 on its own, upload+bits refine are separate
[23:25:56] <ottomata>	 gonna do it, and git-deploy
[23:26:05] <qchris>	 Cool!
[23:26:06] <qchris>	 Thanks.
[23:26:14] <ottomata>	 oh, it has dependency
[23:26:17] <ottomata>	 on bits/upload
[23:26:18] <ottomata>	 rebase?
[23:26:25] <qchris>	 will do.
[23:27:34] <qchris>	 Meh. The dia graph dependency gets in the way.
[23:27:56] <ottomata>	 ?
[23:28:00] <qchris>	 Let's keep that for tomorrow?
[23:28:13] <qchris>	 I always update the oozie overview diagram with my changes.
[23:28:23] <ottomata>	 i think i would rather merge the camus changes asap, so we don't have to consume a bunch later
[23:28:35] <qchris>	 And since that diagram is a binary file, git cannot rebase nicely for diagram changes.
[23:28:35] <ottomata>	 hm, unless it will be smart enough to start at end
[23:28:38] <ottomata>	 oh manbye it will
[23:29:06] <qchris>	 Ok. Then I'll update the graph.
[23:29:57] <ottomata>	 meh i dunno what it does
[23:30:01] <ottomata>	 not clear from the properties
[23:33:08] <grrrit-wm1>	 (PS3) QChris: Load and refine logs from 'misc' caches [analytics/refinery] - https://gerrit.wikimedia.org/r/184191
[23:33:22] <qchris>	 ottomata: rebased change ^
[23:33:36] <ottomata>	 thank you
[23:34:16] <grrrit-wm1>	 (PS4) Ottomata: Load and refine logs from 'misc' caches [analytics/refinery] - https://gerrit.wikimedia.org/r/184191 (owner: QChris)
[23:34:25] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Load and refine logs from 'misc' caches [analytics/refinery] - https://gerrit.wikimedia.org/r/184191 (owner: QChris)
[23:40:25] <ottomata>	 hm, wouldn't it be cool if i could start PART of a bundle without killing the running parts?
[23:40:36] <ottomata>	 e.g. start misc refinement without killing the whole bundle
[23:40:37] <ottomata>	 hm.
[23:40:41] <ottomata>	 i think not possible :p
[23:40:58] <qchris>	 I also think it is not possible.
[23:41:12] <qchris>	 There is a misc directory!
[23:41:24] <qchris>	  /mnt/hdfs/wmf/data/raw/webrequest/webrequest_misc
[23:41:32] <ottomata>	 :)
[23:41:35] <qchris>	 So camus seems to be doing its job.
[23:41:50] <qchris>	 But do not worry about the refining.
[23:41:57] <qchris>	 I can take care of that tomorrow.
[23:43:25] <qchris>	 Content looks good too.
[23:43:27] <qchris>	 Awesome.
[23:43:27] <ottomata>	 i am starting the refine job now :)
[23:43:33] <qchris>	 Ok. :-D
[23:43:51] <ottomata>	 3 coordinators now!
[23:43:52] <ottomata>	 :)
[23:44:06] <ottomata>	 woot Job Name    : refine-webrequest_misc
[23:44:07] <qchris>	 Awesome! What a productive evening.
[23:57:05] <ottomata>	 oh, qchris, i need to restart the load job too, eh?
[23:57:14] <ottomata>	 yup, doh.
[23:57:18] <qchris>	 Yup.
[23:59:28] <ottomata>	 ok done
[23:59:32] <ottomata>	 now 5 load coordinators running
[23:59:41] <qchris>	 Cool.