[00:13:05] <wikibugs>	 3MediaWiki-extensions-OpenStackManager, Librarization, MediaWiki-extensions-Translate: Bring in spyc for OpenStackManager and Translate via composer - https://phabricator.wikimedia.org/T75945 (10Legoktm) 3NEW p:3Triage
[00:44:32] <wikibugs>	 3Wikimedia-Labs-tools: Set up a tileserver for OSM in Labs - https://phabricator.wikimedia.org/T62819#786794 (10Mattflaschen)
[00:45:33] <wikibugs>	 3Wikimedia-Labs-wikistats: Add sourceforge farm - https://phabricator.wikimedia.org/T60396#786802 (10Dzahn)   > (In reply to Robert Hanke from comment #14) > WikiTeam's up to date lists are at https://github.com/WikiTeam/wikiteam/blob/master/listsofwikis/ but SF wasn't updated recently, IIRC.  the first 14 wikis...
[00:47:17] <wikibugs>	 3Wikimedia-Labs-wikistats: Add sourceforge farm - https://phabricator.wikimedia.org/T60396#786809 (10Dzahn) 5Open>3Resolved setting to resolved either way, because we do have that sourceforge table  but ..most or all links will be broken. some redirect to non-sourceforge URLs it seems  http://wikistats.wmfla...
[00:47:59] <wikibugs>	 3Wikimedia-Labs-wikistats: deploy a replacement for the old "wikistats admin" (WSA) script - https://phabricator.wikimedia.org/T38287#786811 (10Dzahn)
[00:57:12] <wikibugs>	 3Wikimedia-Labs-wikistats: Fix all the Wikia stats - https://phabricator.wikimedia.org/T61943#786843 (10Dzahn) >>! In T61943#641686, @Nemo_bis wrote: > Yes, as noted it the commit message the patch doesn't actually enable updates. just pave the way for them. I suppose it's a hardcoded crontab, unless there's som...
[04:19:53] <ragesoss>	 OAuth logins seem to be broken. Quarry isn't working, and neither is my app.
[06:41:46] <wikibugs>	 3Wikimedia-Labs-tools: Tool Labs: Node.js and npm broken due to outdated certificate (install minor update to fix certificate) - https://phabricator.wikimedia.org/T72120#787341 (10Danmichaelo) Why was this closed? Node / npm is still broken on Tool Labs
[06:41:50] <Krinkle>	 Coren: andrewbogott_afk: Seems wikitech user sessions lost backend connection to openstack again. Getting weird errors like "The requested host does not exist." when clicking "reboot"
[06:42:26] <wikibugs>	 3Wikimedia-Labs-tools: Tool Labs: Node.js and npm broken due to outdated certificate (install minor update to fix certificate) - https://phabricator.wikimedia.org/T72120#787342 (10yuvipanda) @danmichaelo use trusty.tools.wmflabs.org and use jsub with -l release=trusty. Both have newer versions.
[06:44:38] <Krinkle>	 logout/login makes it work.
[07:48:46] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#787392 (10yuvipanda) Consider three things we want beta to have:  # Same as prod (including SSL Certs) # sudo for as many people as possible  They're incompati...
[08:28:13] <wikibugs>	 3Wikimedia-Labs-General: tables views missing for idwiki_p (source s3) on s3.labsdb - https://phabricator.wikimedia.org/T75975#787463 (10Merl)
[08:29:12] <wikibugs>	 3Wikimedia-Labs-General: tables views missing for idwiki_p (source s3) on s3.labsdb - https://phabricator.wikimedia.org/T75975#787463 (10Merl)
[08:32:51] <wikibugs>	 3Wikimedia-Labs-tools: Tool Labs: Node.js and npm broken due to outdated certificate (install minor update to fix certificate) - https://phabricator.wikimedia.org/T72120#787487 (10Petrb)
[08:34:19] <wikibugs>	 3Wikimedia-Labs-General: tables views missing for idwiki_p (source s3) on s3.labsdb - https://phabricator.wikimedia.org/T75975#787492 (10Merl) see also T75493 for the same problem with s5 as source database cluster
[08:40:29] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#787515 (10Petrb)
[08:40:34] <wikibugs>	 3Wikimedia-Labs-General: add user filter to status page - https://phabricator.wikimedia.org/T75384#787517 (10Petrb)
[08:40:37] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Get Parsoid (and thus VisualEditor) working again on Wikitech - https://phabricator.wikimedia.org/T75104#787519 (10Petrb)
[08:40:41] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Thumbnails for Commons images are not displayed - https://phabricator.wikimedia.org/T74245#787521 (10Petrb)
[08:40:45] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Not possible to delete files - https://phabricator.wikimedia.org/T73208#787523 (10Petrb)
[08:52:22] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Commons replica DB excessively out-of-date - https://phabricator.wikimedia.org/T75980#787543 (10Magnus)
[09:58:48] <grrrit-wm>	 (03PS1) 10Filippo Giunchedi: add privacy disclaimer [labs/private] - 10https://gerrit.wikimedia.org/r/175969 
[10:03:12] <wikibugs>	 3Wikimedia-Labs-tools: Tool Labs: Node.js and npm broken due to outdated certificate (install minor update to fix certificate) - https://phabricator.wikimedia.org/T72120#787650 (10Danmichaelo) Sorry, missed that email. Thanks for clarifying!
[10:33:45] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "A nice follow up to the warning I added in modules/passwords/manifests/init.pp with:" [labs/private] - 10https://gerrit.wikimedia.org/r/175969 (owner: 10Filippo Giunchedi)
[10:38:41] <YuviPanda>	 !log shinken killed old instances shinken-server and tustyshinkencheck
[10:38:44] <labs-morebots>	 Logged the message, Master
[10:40:25] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] add privacy disclaimer [labs/private] - 10https://gerrit.wikimedia.org/r/175969 (owner: 10Filippo Giunchedi)
[11:38:51] <wikibugs>	 3Tool-Labs-tools-Database-Queries: DBQ-197 User preferences usage on Portuguese Wikipedia - https://phabricator.wikimedia.org/T61480#787774 (10Steinsplitter)
[11:42:42] <wikibugs>	 3Tool-Labs-tools-Database-Queries: DBQ-197 User preferences usage on Portuguese Wikipedia - https://phabricator.wikimedia.org/T61480#787792 (10TTO) p:5Triage>3Volunteer?
[11:55:09] <wikibugs>	 3Wikimedia-Labs-tools: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#787814 (10He7d3r)
[12:23:14] <wikibugs>	 3Wikimedia-Labs-tools: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#787859 (10Qgil)
[13:35:56] <Betacommand>	 was idwiki deleted?
[13:38:43] <Betacommand>	 idwiki has no data in the database
[13:38:54] <Betacommand>	 Coren: ^ 
[13:39:09] <YuviPanda>	 Betacommand: try in the other databases?
[13:39:22] <YuviPanda>	 I mean, connect to enwiki.labsdb and try 'use idwiki_p'?
[13:41:13] <Betacommand>	 YuviPanda: its there, but idwiki.labsdb is showing zero tables
[13:41:27] <YuviPanda>	 yeah, Coren just fixed a similar issue yesterday
[13:41:30] <YuviPanda>	 Coren: what's causing the issues, btw?
[13:45:15] <YuviPanda>	 andrewbogott_afk: looks like wikitech-test is essentially hosed.
[13:48:05] <YuviPanda>	 ok, slightly unfucked now.
[13:50:40] <wikibugs>	 3Wikimedia-Labs-tools: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#788034 (10Petrb)
[13:51:01] <YuviPanda>	 nope, not really. lol
[14:26:12] <ptqa>	 hi everyone
[14:29:15] <ptqa>	 I have a question about using ruby in tools, how to use gems there? I can install them only in my home dir, should I use wrapper for ruby or smthng?
[14:36:51] <ptqa>	 oh yeh
[14:37:03] <ptqa>	 wrapper thing really works, thanks everyone)
[15:23:33] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788195 (10hashar) >>! In T75919#785867, @yuvipanda wrote: > Can't actually use dynamicproxy, since the cert there is just for *.wmflabs.org. Also, this termina...
[15:26:10] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788206 (10yuvipanda) I'll also note that from a security perspective, since people without NDA *had* root in the past, it would've been trivial for them to acq...
[15:28:14] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 032 V: 032] add privacy disclaimer [labs/private] - 10https://gerrit.wikimedia.org/r/175969 (owner: 10Filippo Giunchedi)
[15:31:34] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788221 (10hashar) >>! In T75919#786302, @greg wrote: >>>! In T75919#786292, @hashar wrote: >> I loose track on the rather long {T50501} , but I commented about...
[15:53:52] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788261 (10greg) >>! In T75919#788221, @hashar wrote: >>>! In T75919#786302, @greg wrote: >>>>! In T75919#786292, @hashar wrote: >>> I loose track on the rather...
[16:07:57] <YuviPanda>	 andrewbogott_afk: hey! I'm going to step away for dinner shortly, but is there anywhere I can test wikitech patches? LDAP/Nova on wikitech-test-frontend seems dead. Mediawiki was dead too, but I switched vagrant to the precise-compat branch and it works reasonably. Not sure how that instance was set up.
[16:08:00] <YuviPanda>	 https://gerrit.wikimedia.org/r/#/c/175984/ is the patch
[16:10:25] <YuviPanda>	 andrewbogott_afk: also I've added you as a contact in shinken for https://gerrit.wikimedia.org/r/#/c/175964/ which sets up a test for the labs puppetmaster. email only. do +1/CR
[16:45:27] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788525 (10chasemp) >>! In T75919#788221, @hashar wrote: >>>! In T75919#786302, @greg wrote: >>>>! In T75919#786292, @hashar wrote: >>> I loose track on the rat...
[16:57:25] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788615 (10bd808) Self-signed SSL certificate chains have historically been a huge pain for automated browser testing. @Cmcmahon has pointed this out elsewhere...
[16:59:20] <wikibugs>	 3Beta-Cluster, Wikimedia-Labs-Infrastructure: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#788637 (10greg)
[17:41:43] <wikibugs>	 3MediaWiki-extensions-OpenStackManager, Librarization, MediaWiki-extensions-Translate: Bring in spyc for OpenStackManager and Translate via composer - https://phabricator.wikimedia.org/T75945#788713 (10Legoktm) I filed https://github.com/mustangostang/spyc/issues/38 asking for a new release to be tagged since bo...
[17:44:06] <wikibugs>	 3Wikimedia-Labs-Infrastructure: wikidata (federated?) database not available for many wikis - https://phabricator.wikimedia.org/T75511#788717 (10Merl)
[17:44:07] <wikibugs>	 3Wikimedia-Labs-General: tables views missing for s5 databases (dewiki_p/wikidatawiki_p) on s3.labsdb - https://phabricator.wikimedia.org/T75493#788718 (10Merl)
[18:46:07] <wikibugs>	 3Wikimedia-Labs-tools: Set up a trusty web host on tools - https://phabricator.wikimedia.org/T76055#788851 (10Legoktm)
[18:49:23] <wikibugs>	 3MediaWiki-extensions-OpenStackManager, Librarization, MediaWiki-extensions-Translate: Bring in spyc for OpenStackManager and Translate via composer - https://phabricator.wikimedia.org/T75945#788863 (10Nikerabbit)
[18:58:31] <YuviPanda>	 andrewbogott: hey!
[18:59:22] <andrewbogott>	 hello!
[18:59:51] <YuviPanda>	 andrewbogott: so... where do we test wikitech patches? :) saw my messages about wikitech-test-frontend?
[19:00:23] <andrewbogott>	 wikitech-test is broken because heira doesn't work on trusty in labs
[19:00:29] <YuviPanda>	 oh
[19:00:31] <andrewbogott>	 Giuseppe is fixing it, in theory
[19:00:31] <YuviPanda>	 I thought that got fixed?
[19:00:37] <andrewbogott>	 Maybe?  If so I haven't head
[19:00:38] <YuviPanda>	 ori sent a patch in
[19:00:39] <andrewbogott>	 heard
[19:00:48] <YuviPanda>	 on the email thread you started
[19:01:32] <andrewbogott>	 as far as I know that was just a tool for testing, not a fix
[19:01:46] <YuviPanda>	 andrewbogott: oh, I didn't actually look at the patch :) ok
[19:02:02] <YuviPanda>	 andrewbogott: hmm, so I guess I'll just poke him
[19:02:17] <YuviPanda>	 andrewbogott: outside of that, how hard do you think it'll be to make the wikitech vagrant role fully functional?
[19:02:21] <andrewbogott>	 yeah.  wikitech-test may need work beyond that, but that's what's blocking me for now
[19:02:41] <andrewbogott>	 I don't think that wikitech-vagrant will ever install openstack.  But having it work alongside production openstack puppet works OK.
[19:02:52] <bd808>	 !log integration Changed worker count on wikidata-jenkins[1-3] from 5 to 3 as requested by hoo
[19:02:56] <labs-morebots>	 Logged the message, Master
[19:03:05] <andrewbogott>	 But  setting up a full openstack cluster is a bit outside the scope of puppet.  Requires some hand tuning
[19:03:22] <YuviPanda>	 andrewbogott: hmm, we can't set it up to the same state as wikitech-test?
[19:03:58] <andrewbogott>	 'it'?
[19:04:46] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Commons replica DB excessively out-of-date - https://phabricator.wikimedia.org/T75980#788906 (10MZMcBride)
[19:05:57] <YuviPanda>	 andrewbogott: it -> wikitech-vagrant
[19:06:22] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Commons replica DB excessively out-of-date - https://phabricator.wikimedia.org/T75980#788910 (10MZMcBride) p:5Triage>3High
[19:06:24] <andrewbogott>	 I'm not sure I understand what you're asking
[19:06:30] <andrewbogott>	 We could duplicate a bunch of prod puppet code in vagrant
[19:06:40] <andrewbogott>	 but that would diverge, and not be so useful for testing
[19:06:51] <YuviPanda>	 andrewbogott: hmm, true.
[19:06:57] <wikibugs>	 3Wikimedia-Labs-General: tables views missing for s5 databases (dewiki_p/wikidatawiki_p) on s3.labsdb - https://phabricator.wikimedia.org/T75493#788923 (10MZMcBride)
[19:07:08] <YuviPanda>	 andrewbogott: I'm talking about setting up a local wikitech on my vagrant that has at least LDAP and instance/project creation working
[19:07:18] <andrewbogott>	 'at least'?
[19:07:20] <andrewbogott>	 :)
[19:07:29] <YuviPanda>	 hehe :D
[19:07:32] <andrewbogott>	 I don't know that I've ever been able to get instance creation to work in labs
[19:07:34] <andrewbogott>	 due to network issues
[19:07:44] <andrewbogott>	 project creation is easy
[19:07:45] <YuviPanda>	 yeah, but we could get it get stuck on creation, so the API can be tested
[19:07:51] <andrewbogott>	 Sure
[19:08:19] <andrewbogott>	 I think that… what you're asking for is basically done.  Except that there's a step where you have to tell keystone where to find services and such.
[19:08:31] <andrewbogott>	 We could make a minimal default config that runs automatically, maybe.  But it will be ugly
[19:09:01] <YuviPanda>	 that's ok :)
[19:09:26] <YuviPanda>	 andrewbogott: really? local vagrant, with wikitech applied, if I try to log in I get:
[19:09:29] <YuviPanda>	 Unexpected object type Unknown.
[19:10:33] <andrewbogott>	 That doesn't give me much to go on :)
[19:10:54] <YuviPanda>	 me neither
[19:11:01] <YuviPanda>	 I try to log in, get blank page with just that
[19:11:51] <andrewbogott>	 If you don't have ldap then everything will probably fail immediately
[19:12:02] <YuviPanda>	 yeah, and does the wikitech role set up ldap?
[19:12:46] <andrewbogott>	 no
[19:13:02] <andrewbogott>	 it's not… trivial to create a new working ldap server
[19:13:07] <YuviPanda>	 right.
[19:13:12] <andrewbogott>	 even the prod classes don't work that great
[19:13:26] <YuviPanda>	 to rephrase, I'm wondering if it's worth our time to clean those up and set it up for wikitech
[19:13:39] <YuviPanda>	 we would also probably want a horizon vagrant, and this might help
[19:14:00] <andrewbogott>	 I don't think that getting ldap working in vagrant is realistic
[19:14:15] <YuviPanda>	 oh...
[19:14:16] <YuviPanda>	 ok.
[19:14:25] <YuviPanda>	 is LDAP *that* bad?
[19:14:48] <YuviPanda>	 andrewbogott: sorry if this seems pestering, you're pretty much the most knowledgeable person about wikitech around now :)
[19:15:33] <andrewbogott>	 Well… opendj setup isn't very automatable.  It requires some interactive tools.
[19:15:42] <andrewbogott>	 It might be possible, I just haven't succeeded in past attempts.
[19:16:34] <andrewbogott>	 I think that the only way to get a test setup will be using a combination of vagrant and prod puppet.  Like wikitech-test already is.
[19:16:54] <andrewbogott>	 I'm not sure how duplicating any of the steps from prod puppet into vagrant helps us -- I can definitely think of ways it harms though
[19:17:23] <YuviPanda>	 helps us in the sense the current wikitech role doesn't actually work in vagrant.
[19:17:58] <andrewbogott>	 But if it did work (by cutting out ldap and openstack) how would it be useful?
[19:21:00] <YuviPanda>	 andrewbogott: don't think so, I think
[19:21:10] <YuviPanda>	 andrewbogott: without those it's just another wiki (with some SMW stuff)
[19:21:18] <andrewbogott>	 YuviPanda: if vagrant could set up a full ldap/keystone/nova cluster locally, that would be pretty handy for testing OSM.
[19:21:35] <YuviPanda>	 definitely! exactly what I was hoping we could do :)
[19:21:41] <andrewbogott>	 That that would be a ton of work, and would diverge from production puppet.  And inasmuch as we want to kill OSM.
[19:21:43] <YuviPanda>	 we could definitely do execs (eugh, but would work)
[19:21:43] <YuviPanda>	 yeah
[19:21:56] <andrewbogott>	 So, it's not that it's impossible...
[19:21:58] <YuviPanda>	 well, I guess Horizon will still have LDAP and keystone/nova
[19:22:03] <andrewbogott>	 just, we already have a pretty good solution
[19:22:06] <andrewbogott>	 (or had, until hiera broke it)
[19:22:26] <andrewbogott>	 I'm not sure that making a local-to-a-laptop solution is worth the trouble.  It will be a LOT of trouble
[19:22:27] <YuviPanda>	 hmm, right.
[19:22:40] <YuviPanda>	 I guess that's what I wanted to know :) if it's a lot of trouble then meh
[19:23:18] <andrewbogott>	 It's definitely a lot of trouble
[19:23:32] <andrewbogott>	 If you want, you can poke around on wikitech-test and see what's going on with keystone & co.
[19:23:51] <andrewbogott>	 But I think the issue right now is just that we have icehouse configs with havana packages.
[19:24:03] <andrewbogott>	 Proably you could override hiera locally if you don't want to wait for a proper fix
[19:25:44] <YuviPanda>	 yeah, I can wait, I think
[19:25:47] <YuviPanda>	 plenty of other things to work on
[19:26:02] <YuviPanda>	 !log tools created tools-webgrid-05 on trusty to set up a working webnode for trusty
[19:26:05] <labs-morebots>	 Logged the message, Master
[19:26:05] <bd808>	 YuviPanda: http://thornelabs.net/2013/12/19/deploy-rackspace-private-cloud-v42x-powered-by-openstack-havana-with-neutron-networking-using-virtualbox-or-vmware-fusion-and-vagrant.html
[19:26:46] <YuviPanda>	 bd808: :D I was just looking at devstack
[19:26:47] <YuviPanda>	 nice
[19:26:53] <YuviPanda>	 so I guess we don't need to hack havana into ours
[19:35:53] <YuviPanda>	 Coren: am adding a trusty webgrid node. for completeness, plus legoktm requested :)
[19:35:58] <YuviPanda>	 there's a phab task somewhere
[19:36:07] <Coren>	 The sense, you are making it.
[19:36:21] <legoktm>	 https://phabricator.wikimedia.org/T76055
[19:36:46] <wikibugs>	 3Wikimedia-Labs-General: replication for s5 source wikis (dewiki/wikidata) stopped on labs1002 (dewiki.labsdb)  - https://phabricator.wikimedia.org/T76070 (10Merl) 3NEW p:3Triage
[19:36:53] <YuviPanda>	 Coren: do we have docs on this? I'm writing down at https://etherpad.wikimedia.org/p/new-webnode-toollabs as I go along
[19:37:15] <Coren>	 I forsee two plausible issues you may want to watch for: lighttpd version bump which might cause the default config to have issues, and 'webservice' has no provision atm for specifying -l requirements (but that should be relatively easy to fix).
[19:37:15] <wikibugs>	 3Wikimedia-Labs-General: replication for s5 source wikis (dewiki/wikidata) stopped on labs1002 (dewiki.labsdb) - https://phabricator.wikimedia.org/T76070#789042 (10Merl)
[19:37:28] <YuviPanda>	 yeah
[19:38:10] <YuviPanda>	 puppet run's gonna take a whillleee
[19:38:43] <Coren>	 YuviPanda: No, but it's a trivial thing to do after you added the right classes to the instance config (the gridengine puppet thang can't do that itself yet): add the node to gridengine proper, add it to the apropriate queues, and restart gridengine-exec
[19:38:56] <YuviPanda>	 right.
[19:39:04] <YuviPanda>	 Coren: you were going to do a short hangout on SGE :)
[19:39:06] <YuviPanda>	 we should set that up
[19:39:40] <Coren>	 We should.  Sometime next week unless you are ridiculously busy then?
[19:40:12] <YuviPanda>	 nope, sometime next week sounds fine! except for perhaps wednesday or something.
[19:40:16] <YuviPanda>	 andrewbogott: ^ would you also be interested?
[19:41:00] <andrewbogott>	 SGE?  Yes please
[19:45:58] <YuviPanda>	 Coren: andrewbogott tuesday?
[19:46:26] <andrewbogott>	 I have a meeting from 11-noon CST.  Any other time is good
[19:46:35] <andrewbogott>	 um… that's 9AM in SF
[19:46:35] <Coren>	 YuviPanda: Find some time that works for you between 14h and 23h UTC and I'm happy.
[19:46:48] <YuviPanda>	 so many timezones.
[19:47:01] <Coren>	 Heh.  That's why I only speak UTC now.  :-)
[19:47:04] <YuviPanda>	 :D
[19:47:09] <YuviPanda>	 I still don't have a proper sense for UTC
[19:47:21] <YuviPanda>	 I have been meaning to write an app that tracks what timezone I'm in based on my sleeping habits
[19:49:13] <YuviPanda>	 andrewbogott: Coren sent an invite, assuming your gcals are accurate...
[19:49:28] <YuviPanda>	 I think it's between Coren's time specifications too
[19:50:20] <Coren>	 YuviPanda: It might actually be worthwhile to include Tim if he's willing and available.
[19:51:19] <YuviPanda>	 Coren: totally agree.
[19:51:24] <YuviPanda>	 Coren: let me email him
[19:52:21] <YuviPanda>	 emailed
[19:52:33] <YuviPanda>	 so many puppet failures
[19:52:35] <YuviPanda>	 that's weird
[19:52:39] <YuviPanda>	 they're not failing on the exec node
[19:53:31] <Coren>	 YuviPanda: Remember that on a new instance it sometimes takes more than one puppet run because the first adds the local repos (and ordering is random).
[19:54:19] <YuviPanda>	 aaah
[19:54:19] <YuviPanda>	 right
[19:54:25] <YuviPanda>	 we need to use labsdebrepo
[19:54:36] <YuviPanda>	 and also have labsdebrepo support mixed precise/trusty things
[19:54:45] <YuviPanda>	 I wonder if toollabs' supports that
[19:55:24] <Coren>	 No, it's just a dumb file repo.
[19:55:30] <YuviPanda>	 yeah.
[19:57:07] <YuviPanda>	 hmm, how does nginx logrotate on tools-webproxy?
[19:57:48] <YuviPanda>	 should probably make it keep logs for only one day
[20:12:12] <wikibugs>	 3Tool-Labs-tools-Commons-Delinker: Commons Delinker is down, and web-interface is broken - https://phabricator.wikimedia.org/T76077#789118 (10NahidSultan)
[20:15:08] <wikibugs>	 3Tool-Labs-tools-Commons-Delinker: Commons Delinker is down, and web-interface is broken - https://phabricator.wikimedia.org/T76077#789151 (10NahidSultan)
[20:25:52] <Steinsplitter>	 hello, somone has renamed today databases?
[20:25:57] <Steinsplitter>	 petan?
[20:27:54] <Steinsplitter>	 hello?
[20:32:40] <YuviPanda>	 Steinsplitter: heya! what do you mean by 'today databases'?
[20:32:54] <YuviPanda>	 if it's databses missing data, I think Coren and springle are looking into it already
[20:33:03] <Steinsplitter>	 YuviPanda: ProgrammingError: (1146, "Table 'p50380g51602_p_delinker_p.replacer' doesn't exist")
[20:33:03] <Steinsplitter>	 STDOUT: [2014-11-26 20:15:50] Stopping CommonsDelinker
[20:33:11] <YuviPanda>	 Coren: ^
[20:33:16] <YuviPanda>	 uh oh, that's a new one
[20:34:21] <Coren>	 Steinsplitter: You may have to specify a specific server to connect to; there was some movement of aliases to databases to circumvent a problem that may have bit you.  Which server were you connecting to?
[20:36:46] <Steinsplitter>	 commonswiki.labsdb
[20:36:51] <Steinsplitter>	 show tables;
[20:37:00] <Steinsplitter>	 the database isn't  there :(
[20:37:10] <Steinsplitter>	 not sure why you move db's around
[20:37:12] * Coren ponders.
[20:37:24] <Steinsplitter>	 *databases; i mean
[20:38:31] <Coren>	 We didn't, but some of the aliases temporarily point to other servers to work around an issue.  Connect to 'c2.labsdb' and you'll be golden.
[20:39:22] <Coren>	 (In general, if you use tools-specific databases it's better to rely on a stable server than on an alias, but it's true that this has never been spelled out - documentation error)
[20:40:08] <Coren>	 c2 will, by definition, always point to the same place.
[20:41:01] <Steinsplitter>	 ok, thx :)
[20:41:01] <Steinsplitter>	 works
[20:47:54] <cscott>	 !log deployment-prep updated OCG to version 7d8f2b8bd496464041e3ef9c092732457cc8f7ef
[20:47:56] <labs-morebots>	 Logged the message, Master
[20:54:08] <wikibugs>	 3Tool-Labs-tools-Commons-Delinker: Commons Delinker is down, and web-interface is broken - https://phabricator.wikimedia.org/T76077#789282 (10Steinsplitter) fix for the webinterface: https://github.com/commonsdelinker/delinker-site/commit/6a186e6702707f2ebf071c1fd4ef973f7e0d2114
[20:54:43] <wikibugs>	 3Tool-Labs-tools-Commons-Delinker: Commons Delinker is down, and web-interface is broken - https://phabricator.wikimedia.org/T76077#789283 (10Steinsplitter)
[20:56:40] <wikibugs>	 3Wikimedia-Labs-tools: Set up a trusty web host on tools - https://phabricator.wikimedia.org/T76055#789291 (10yuvipanda) Created tools-webgrid-05, puppet has been running forever so far :)
[20:57:45] <Merlissimo>	 Steinsplitter: problems on s2.labsdb are also geting more. Maybe s1.labsdb would be the better choise
[20:58:22] <Steinsplitter>	 Merlissimo: ok, thanks.
[21:03:26] <YuviPanda>	 Coren: uh, do you have the magic incantation handy to setup new queue?
[21:03:32] <YuviPanda>	 I have done it in the past, but have no memory.
[21:03:43] <YuviPanda>	 qconf, I think?
[21:03:45] * YuviPanda mans
[21:03:45] <Coren>	 ... wait why would you want a new queue?
[21:03:47] <YuviPanda>	 err
[21:03:48] <YuviPanda>	 I meant
[21:03:54] <YuviPanda>	 new instance
[21:03:58] <YuviPanda>	 new node, even
[21:04:52] <Merlissimo>	 the word you are looking for: new execution host
[21:05:23] <YuviPanda>	 thank you, Merlissimo
[21:05:24] <Coren>	 Easiest thing to do is to copy my template.  Look in ~marc/gridengine/webnode.e
[21:05:27] * YuviPanda is a SGE newbie
[21:05:29] <YuviPanda>	 ah, ok
[21:05:39] <YuviPanda>	 Coren: also need to add a resource release=trusty to it
[21:05:46] <YuviPanda>	 (hopefully correct terminology)
[21:06:17] <Coren>	 Copy it, edit it, and do a 'qconf -Ae <filename>'
[21:06:46] <YuviPanda>	 now I remember :)
[21:06:56] <Coren>	 YuviPanda: Correct, you want to add 'release=trusty' to the complex_values
[21:07:24] <Coren>	 You can look at 'qconf -me tools-exec-12' for an example
[21:08:18] <YuviPanda>	 cool, added
[21:08:21] <Coren>	 Remember that web nodes have to be submit nodes too: 'qconf -as <the-hostname>'
[21:08:38] <Coren>	 Once that's done, you need to add it to the queue
[21:08:39] <YuviPanda>	 yeah, added
[21:09:05] <Merlissimo>	 and it to the @webgrid host list
[21:09:17] <Coren>	 qconf -mhgrp @webgrid
[21:09:20] <Coren>	 Right.
[21:09:32] <Coren>	 final step is restart gridengine-exec on the node.
[21:09:58] <Coren>	 Once that's done, you'll see the host load etc with qhost
[21:10:54] <YuviPanda>	 restarted+!
[21:11:09] <YuviPanda>	 I do see it!
[21:11:10] <YuviPanda>	 yay
[21:11:22] <YuviPanda>	 let me write all these down
[21:11:30] <Merlissimo>	 Coren: can you switch dewiki.labsdb to s1?
[21:12:09] <Coren>	 If at all possible, I'd rather leave enwiki alone.  Is there a user db there you need to access?
[21:12:23] <Merlissimo>	 it is the only db having last data available
[21:12:37] <Coren>	 You can always connect to c1.labsdb directly.
[21:12:41] <Merlissimo>	 on s3 there are no dewiki views and on s2 there is not replication
[21:13:05] <Coren>	 I know Sean is currently restoring data to sync them up, but he told me it will take several days because of tables with no primary keys.
[21:13:23] <wikibugs>	 3Tool-Labs-tools-Commons-Delinker: Commons Delinker is down, and web-interface is broken - https://phabricator.wikimedia.org/T76077#789332 (10Steinsplitter) 5Open>3Resolved the same for cdh and cd.
[21:13:41] <YuviPanda>	 Coren: hmm, an email update would be nice.
[21:14:02] <Coren>	 YuviPanda: My own information is old-ish.  I shall poke Sean to give an update.
[21:14:11] <YuviPanda>	 cool
[21:14:15] <YuviPanda>	 but still, better than none.
[21:15:20] <Merlissimo>	 moving the wikidata request to s1 will incread load
[21:33:47] <Coren>	 Merlissimo: Like I said, you can connect to c1 directly at need; though I'll ask you to make this a temporary measure while the other servers stabilize.
[21:44:29] <annika_>	 i changed practically nothing to my cgi script and now it complains about oom
[21:45:53] <annika_>	 and now it works … weird
[21:57:13] <a930913>	 Coren: Looks like my email to the list a while back didn't come through. Any idea why? (First one, needs moderation?)
[21:57:34] <a930913>	 annika_: Seems to be happening randomly of late :/
[22:00:12] <Coren>	 annika_: We're running out of resources, and Labs doesn't have that much elbow room to add new nodes.  We've got moar hardware on the way though.
[22:00:45] <Coren>	 a930913: Normally, email only needs moderation if it came from an unsubscribed address.
[22:01:26] <Betacommand>	 Coren: whats the ETA on the new hardware?
[22:01:56] <Coren>	 Rob would know for certain, but I think it's about a week away.
[22:02:17] <Betacommand>	 Coren: close enough, just looking for a ballback :P
[22:02:24] <Betacommand>	 *ball park
[22:02:30] <Betacommand>	 long day
[22:17:51] <a930913>	 Coren: 2014-11-19 15:13, sent to labs-l@lists.wikimedia.org
[23:01:34] <darkblue_b>	 this is a good page for overview ?  https://wikitech.wikimedia.org/wiki/Help:Tool_Labs
[23:02:45] <YuviPanda>	 darkblue_b: yup!
[23:02:51] <darkblue_b>	 thx
[23:03:20] <darkblue_b>	 SGE means Sun Grid Engine, an ancestor of Open Grid Engine, I suppose ?
[23:04:03] <darkblue_b>	 http://gridscheduler.sourceforge.net/
[23:04:05] <darkblue_b>	 ?
[23:04:19] <YuviPanda>	 yeah
[23:05:06] <darkblue_b>	 ok - I am in the Berkeley community, so there are people working on this .. http://mesos.apache.org/
[23:05:24] <YuviPanda>	 ah, yeah, I remember...
[23:05:31] <YuviPanda>	 mesos + docker wouldn't be too bad
[23:06:24] * bd808 swats YuviPanda with a trout for wanting docker
[23:06:31] <YuviPanda>	 heh
[23:06:35] <YuviPanda>	 I wanna play with it, at least
[23:07:08] <bd808>	 I made a docker provider config for mw-vagrant. Docker wants to do things very differently than I want to
[23:07:10] <darkblue_b>	 I wrote an overview of the mapping tiles rendering pipeline used on another lab channel.. its stable and works well
[23:07:28] <darkblue_b>	 .. its unpublished, but if someone specifically wanted to see I could chat about that
[23:07:52] <darkblue_b>	 seems like there are bigger things to think about here right now
[23:08:50] <darkblue_b>	 bd808: seems like docker is something for very light weight, short lived instance things.. I am not working with it, however
[23:08:56] <bd808>	 YuviPanda: in my estimation, a docker container is basically the same a a heroku deploy
[23:09:13] <darkblue_b>	 .. people who are doing mobile app backends are all over docker
[23:09:28] <YuviPanda>	 bd808: well, if VMs are pets I guess docker containers are... single use gloves?
[23:09:40] <darkblue_b>	 something like that
[23:09:41] <bd808>	 I just don't like that they invented yet another way to manage provisioning and subvert running real init process
[23:09:48] <YuviPanda>	 heh
[23:10:41] <bd808>	 I get what they are doing and why, but docker is not a vm and the random developer community seems unaware of that
[23:10:53] <YuviPanda>	 ah, *that* is true
[23:10:58] <bd808>	 1 container = 1 process
[23:11:12] <bd808>	 and thea's cool for a lot of reasons but requires you to think about things very differently
[23:11:13] <YuviPanda>	 but random developer community also seems to be ok with using redis as a backing store for a forum...
[23:11:30] <bd808>	 "it's persistent"
[23:11:40] <bd808>	 and at least as web-scale as mongo
[23:11:41] <darkblue_b>	 the world is a big place, lots of use cases and kinds of development
[23:12:32] <YuviPanda>	 darkblue_b: sure, but none for redis as a sole important-persistant data provider
[23:12:35] <darkblue_b>	 I looked at mongo for the first time in the last six weeks, but Postgres 9.4 with jsonb will fit some aspects of interop for me
[23:13:35] <darkblue_b>	 re redis, no opinion there.. I work with more persistant data than forums :-)