[01:45:34] <yurikR>	 anyone knows why "mwscript eval.php --wiki enwiki
[01:45:34] <yurikR>	 " does not seem to function on betalabs?
[01:46:00] <yurikR>	 $wgSite and $wgTitle are null
[02:06:45] <wikibugs>	 3Wikimedia Labs / 3tools: Some users can't connect to replica DB servers - 10https://bugzilla.wikimedia.org/69679 (10Tim Landscheidt) 3NEW p:3Unprio s:3major a:3Marc A. Pelletier (Split out from comment #1 of bug #54330.)  Some recently created user and tool accounts seem to not be able to connect to...
[02:07:43] <wikibugs>	 3Wikimedia Labs / 3tools: Some users can't connect to replica DB servers - 10https://bugzilla.wikimedia.org/69679 (10Tim Landscheidt)
[02:07:44] <wikibugs>	 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10Tim Landscheidt)
[02:09:29] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Some databases missing - 10https://bugzilla.wikimedia.org/54330#c2 (10Tim Landscheidt) 5NEW>3RESO/FIX a:5Ryan Lane>3Marc A. Pelletier The issue in comment #0 seems to be fixed now; I've filed bug #69679 for the different issue in comment #1.
[02:12:08] <shilad>	 Thanks, Marc. This only happens for a small fraction of language editions for me.
[02:19:17] <legoktm>	 > Failed to create instance. 
[02:19:22] <legoktm>	 thanks for the help :<
[02:20:41] <legoktm>	 wtfff
[02:20:48] <legoktm>	 tell me why???
[02:21:58] <legoktm>	 YuviPanda: can you try creating a new instance for me?
[02:24:17] <YuviPanda>	 legoktm: you are probably out of quota?
[02:24:27] <legoktm>	 there's a limit?
[02:24:46] <legoktm>	 hm
[02:24:50] <legoktm>	 there are 15 instances
[21:06:22] <mutante>	 and one of them happens to be this, just because i search for "sites-enabled" and such
[21:06:34] <milimetric>	 mutante: I gotcha
[21:06:41] <milimetric>	 it seems safe but I'm always scared of puppet :)
[21:07:03] <milimetric>	 especially because that instance has a bunch of instances of limn, but I can check all of them if you want to run it
[21:07:12] <milimetric>	 so it's limn1.eqiad.wmflabs
[21:07:16] <mutante>	 if you currently look into sites-enabled and sites-available,
[21:07:20] <milimetric>	 and then reportcard1.eqiad.wmflabs in project reportcard
[21:07:21] <mutante>	 you _may_ have a messed up situation a bit
[21:07:27] <mutante>	 and have 2 files with different content
[21:07:31] <mutante>	 where actually one should be a symlink
[21:07:39] <Cyberpower678>	 Coren, I'm running a bot process on -login, in case you're wondering.
[21:07:49] <milimetric>	 but if you could, don't do anything on reportcard1.eqiad.wmflabs, because that's running the "real" reportcard
[21:08:04] <Thehelpfulone>	 !project mailman
[21:08:04] <wm-bot>	 There are multiple keys, refine your input: project-access, project-discuss, projects, 
[21:08:07] <milimetric>	 ok mutante, I'm checking that out now (-available vs. -enabled)
[21:08:14] <Thehelpfulone>	 !project-access mailman
[21:08:14] <wm-bot>	 To request access to a project, use a project's discussion page; see !project-discuss <project>
[21:08:16] <mutante>	 milimetric: i recently made a bug about reportcard.wikimedia.org
[21:08:26] <milimetric>	 that's different mutante 
[21:08:35] <mutante>	 i know, but it's broken
[21:08:41] <milimetric>	 reportcard1.eqiad.wmflabs serves reportcard.wmflabs.org
[21:08:47] <mutante>	 then i found out it was decided it should not become production
[21:09:12] <mutante>	 imho redirect would be nicer than the error message
[21:10:04] <milimetric>	 mutante: there's nothing in sites-available on limn1
[21:10:04] <mutante>	 let me see which roles those instances have really configured
[21:10:15] <milimetric>	 so I'm not sure what you mean now
[21:10:24] <mutante>	 milimetric: ..which is wrong 
[21:10:31] <mutante>	 the sites should be in sites-available
[21:10:38] <mutante>	 and enabled in sites-enabled
[21:10:48] <milimetric>	 by symlinks
[21:10:49] <milimetric>	 gotcha
[21:10:51] <mutante>	 right
[21:10:53] <milimetric>	 and is that what your change does?
[21:10:58] <mutante>	 yes
[21:11:23] <milimetric>	 ok, cool, not super important for us (would rather the stability that comes with laziness) but I'm all for cleaner better code
[21:11:33] <mutante>	 well, it's more than that
[21:11:35] <milimetric>	 so I'm happy to help test, and I think it should be pretty low risk
[21:11:44] <mutante>	 it's also that almost every Apache setup has slightly different dependencies
[21:11:46] <milimetric>	 one sec lemme back up those sites-available
[21:11:50] <mutante>	 and this makes them do the same stuff
[21:12:12] <mutante>	 it's nice that the config is already in a template , so it's not touching that line
[21:12:52] <milimetric>	 cool, ok mutante, do your worst and let me know when I should test
[21:13:00] <ebernhardson>	 hashar: i notice zuul/jenkins fallen over once in awhile ... is there perhaps an easy way i can reset it without having to poke people?
[21:13:00] <milimetric>	 I'll only be around for another little bit though
[21:13:20] <hashar>	 ebernhardson: yeah I need to write the step by step documentation to restart it
[21:13:26] <hashar>	 though restarting Jenkins would fix it :-]
[21:13:55] <hashar>	 ebernhardson: I have filled a bug for upstream (openstack) , but not much traces are available ::(
[21:14:18] <mutante>	 milimetric: cool, thanks, let me connect to instanes
[21:14:40] <ebernhardson>	 hashar: where would i bounce jenkins from?  i can't seem to ssh to integration.wikimedia.org directly(although maybe just permissions)
[21:14:48] <milimetric>	 mutante: you know those are self-hosted and have secret data locally right?
[21:14:59] <hashar>	 ebernhardson: few folks have permissions. But ops can give it a poke
[21:15:02] <mutante>	 milimetric: denied public key, where i would expect to have root 
[21:15:03] <milimetric>	 using "secret" very loosely
[21:15:09] <ebernhardson>	 hashar: ok
[21:15:10] <mutante>	 milimetric: no
[21:15:21] <milimetric>	 one sec mutante, lemme get you access
[21:16:39] <hashar>	 ebernhardson: I will probably write a script to unstuck it :-]
[21:17:45] <wikibugs>	 3Tool Labs tools / 3X!'s tools: Checkwiki link on gadget - 10https://bugzilla.wikimedia.org/69718 (10bgwhite) 3UNCO p:3Unprio s:3normal a:3None Wow, the Xlink tools and gadget are impressive. I did not know about the Checkwiki feature on the gadget.  I'm the maintainer of Checkwiki.    There is no "b...
[21:25:46] <wikibugs>	 3Wikimedia Labs / 3tools: I can't access ptwiki_p database - 10https://bugzilla.wikimedia.org/69701 (10Tim Landscheidt)
[21:25:46] <wikibugs>	 3Wikimedia Labs / 3tools: Some users can't connect to replica DB servers - 10https://bugzilla.wikimedia.org/69679 (10Tim Landscheidt)
[21:27:19] <mutante>	 how fixable is the mariadb puppet error?
[21:28:29] <mutante>	 we would like to apply other changes, but are blocked
[21:28:45] <spagewmf>	 Every few days I get a failure when I try to login to beta labs.  It comes from http://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralLogin/complete?token=<blahblah>, this time "[91e46ac0] /wiki/Special:CentralLogin/complete?token=393bfc3673da806cbc1c33ebef9b2516 Exception from line 167 of /srv/common-local/php-master/extensions/CentralAuth/specials/SpecialCentralLogin.php: The user account logged into does not exist."
[21:29:37] <spagewmf>	 the workaround is to visit beta-labs Special:UserLogout , then login works.
[21:37:46] <mutante>	 somebody testing reboots?
[21:37:54] <mutante>	 info: mpt raid status change on testlabs-reboottest6
[21:38:30] <mutante>	 we just got like 10 mails similar to that 
[21:38:42] <mutante>	 should the entire raid status check be removed?
[21:38:49] <mutante>	 because virtual 
[21:50:56] <scfc_de>	 mutante: What mariadb puppet error?
[21:51:52] <mutante>	 Error 400 on SERVER: Could not parse for environment production: Syntax error at '{'; expected  '}' at /etc/puppet/manifests/role/mariadb.pp:86
[21:51:56] <mutante>	 scfc_de: 
[21:54:01] <andrewbogott>	 mutante: that's me, sorry.
[21:54:07] <andrewbogott>	 Rebooting frantically, trying to trigger a race.
[21:56:21] <mutante>	 gotcha, alright
[21:56:42] <mutante>	 andrewbogott: you mean the raid status right
[21:57:08] <andrewbogott>	 I don't know what 'raid status' means in this context, since testlabs-reboottest6 is a VM...
[21:57:11] <andrewbogott>	 But it's probably related.
[21:57:29] <mutante>	 14:40 < mutante> should the entire raid status check be removed?
[21:57:29] <mutante>	 14:40 < mutante> because virtual 
[21:57:47] <andrewbogott>	 Ah, yes, probably should be removed.  I'm surprised that it's there at all.
[21:58:15] <mutante>	 subject:info: mpt raid status change
[22:11:43] <^d>	 bd808: Can you think of any deployment-prep instances we could drop? We're maxed out at 40/40 already :\
[22:12:07] <^d>	 Maybe could lower elastic from 4 to 3, hmm.
[22:12:17] <bd808>	 ^d: Let me take a quick look. We can always ask for more quote
[22:12:21] <bd808>	 quota
[22:12:26] <bd808>	 labs has room to spare
[22:18:25] <bd808>	 ^d: We could kill deployment-apache01 and deployment-apache02 if we had to. They are unused now that we are running hhvm.
[22:19:07] <bd808>	 I'm not sure what deployment-sandbox and deployment-fluoride are doing for us either but I didn't build them
[22:19:20] <^d>	 I think -fluoride is mostly broken.
[22:20:44] <ebernhardson>	 probably not related, but deployment-mediawiki01 isn't serving any requests either
[22:20:47] <ebernhardson>	 but thats probably a bug :P
[22:21:05] <ebernhardson>	 (or maybe leftover from swtaars debugging?)
[22:21:07] <^d>	 2014-07-31T22:18:09 was the launch time for -sandbox.
[22:21:15] <^d>	 No used storage yet.
[22:22:28] <^d>	 !log deployment-prep dropped apache01/02 instances, unused and need the resources
[22:22:31] <labs-morebots>	 Logged the message, Master
[22:23:09] <bd808>	 ebernhardson: It's a bug/leftover from debugging.
[22:23:34] <bd808>	 I tried to repool it this morning and found out that puppet can't configure the apache vhosts right now
[22:23:54] <bd808>	 deployment-mediawiki02 has puppet disabled :/
[22:24:28] <mutante>	 wait a minute, please tell me more about "can't configure apache vhosts"
[22:24:33] <bd808>	 And the guy I'd bug to fix that is on vacation this week
[22:24:53] <bd808>	 mutante: Log into deployment-mediawiki01 and try to start apache
[22:25:08] <bd808>	 lots of "oops this is wrong" and then it dies
[22:25:20] <mutante>	 sigh, let's see if i can log in
[22:25:23] <ebernhardson>	 heh
[22:25:33] <bd808>	 deployment-mediawiki02 is running hand coded something from _joe_
[22:26:44] <bd808>	 All related to the move of apache config to the puppet repo I imagine. We had a very non-prod config in the apache repo under a separate branch
[22:26:54] <mutante>	 apache   25382  0.0  0.1 475312 11960 ?        S    21:31   0:00 /usr/sbin/apache2 -D HHVM -k start
[22:26:58] <mutante>	 _D HHVM ?
[22:27:05] <mutante>	 it's running?
[22:27:16] <bd808>	 It is? /me looks again
[22:27:27] <mutante>	 starting it = no error
[22:27:40] <mutante>	  * Starting web server apache2                                                                                                      * 
[22:27:46] <mutante>	 but it was already before ?!
[22:27:57] <bd808>	 on deployment-mediawiki01?
[22:28:08] <mutante>	 02 ..argg
[22:28:46] <mutante>	 yea, 01 is different
[22:29:01] <mutante>	 please say it's different roles and not all manual 
[22:29:09] <bd808>	 all manual
[22:29:14] <mutante>	 (98)Address already in use: AH00072: make_sock: could not bind to address [::]:9002
[22:29:14] <bd808>	 puppet is disabled on 02
[22:29:18] <mutante>	 ok, i'm logging out
[22:29:38] <bd808>	 that was my response too. _joe_ will fix it next week
[22:29:55] <bd808>	 Until then lets hope that vm doesn't vanish in a puff of smoke
[22:29:56] <mutante>	 AH00112: Warning: DocumentRoot [/usr/local/apache/common/docroot/wikispecies.org] does not exist
[22:29:59] <mutante>	 AH00112: Warning: DocumentRoot [/mnt/upload7] does not exist
[22:30:02] <mutante>	 AH00112: Warning: DocumentRoot [/usr/local/apache/common/docroot/config] does not exist
[22:30:05] <mutante>	 we could probably start with that part
[22:30:07] <mutante>	 AH00112: Warning: DocumentRoot [/usr/local/apache/common/docroot/ee-prototype] does not exist
[22:30:11] <mutante>	 and have the document roots
[22:48:26] <Thehelpfulone>	 !log mailman Deleted corrupt mailman-01 instance
[22:48:28] <labs-morebots>	 Logged the message, Master
[22:48:57] <Thehelpfulone>	 !log wm-bot Restarted instance and bot as frozen for almost 24 hrs
[22:48:57] <labs-morebots>	 wm-bot is not a valid project.
[22:49:11] <Thehelpfulone>	 !log bots Restarted instance as wm-bot as frozen for almost 24 hrs
[22:49:12] <labs-morebots>	 Logged the message, Master
[22:51:55] <^d>	 Gah, new instance built. SSH'd fine. Exited. Can't ssh now, pubkey denied.
[22:51:57] <^d>	 What gives.
[22:53:27] <Thehelpfulone>	 ^d: that happened to me for wm-bot
[22:53:32] <Thehelpfulone>	 restart the instance?
[22:54:12] <^d>	 {{doing}}
[22:57:37] <^d>	 {{no dice}}
[22:58:47] <andrewbogott>	 ˙^d, what project and instance?
[22:59:09] <^d>	 deployment-swift01 (i-0000054d.eqiad.wmflabs), deployment-prep.
[23:00:08] <andrewbogott>	 Hm, just as soon as my internet starts working again, I'll have a look.
[23:01:13] <andrewbogott>	 bah, virt1006
[23:01:20] <andrewbogott>	 ^d:  that host seems to be rapidly decaying :(
[23:01:30] <^d>	 :(
[23:01:36] <andrewbogott>	 I'm in the process of depooling so it won't get new instances.  But instances that are on it… I don't know yet if they can be saved.
[23:01:44] <andrewbogott>	 I'll use yours as an experiment in just a moment.
[23:02:00] <^d>	 Do whatever you'd like to it. It's brand-new from like 15 minutes ago.
[23:04:41] <andrewbogott>	 actually… ^d, I have root on that instance, no problems.
[23:04:45] <andrewbogott>	 You can ssh to other beta vms?
[23:05:23] <^d>	 Hmm, swift01 works now
[23:05:30] <^d>	 Other vms have been fine
[23:05:53] <andrewbogott>	 ok, well, lemme try to migrate it anyway, as an exercise.  I'll need to shut it down first
[23:15:29] <andrewbogott>	 well… ^d, I'm stumped, probably best to just delete that instance and build a fresh one.
[23:15:53] <andrewbogott>	 The issue shouldn't recurr, I'm just not sure what to do about instances that already live on virt1006 :(
[23:16:13] * bd808 hopes that deployment-mediawiki02 isn't there
[23:17:03] <bd808>	 That's the box that I figured out this morning is hand built and the only apache powering all of beta
[23:17:06] <andrewbogott>	 nope, it's on 1008
[23:18:42] <andrewbogott>	 YuviPanda|brb: is it ok if I 'accidentally' break quarry-web-test?