[01:12:16] <mutante>	 i get this again on my project
[01:12:19] <mutante>	 channel 0: open failed: administratively prohibited: open failed
[01:12:24] <mutante>	 ssh_exchange_identification: Connection closed by remote host
[01:13:05] <mutante>	 no, taking it back, wrong instance name
[01:13:35] <jeremyb>	 crossposter!
[01:14:48] <mutante>	 you would have told me "-labs" in 10 seconds otherwise, no ?:)
[01:17:32] <mutante>	 nslcd[17332]: [c0ce9b] <group=550> error writing to client: Broken pipe
[01:17:33] <mutante>	 hmmm
[01:17:38] <mutante>	 jeremyb: do you know those ?
[01:19:57] <jeremyb>	 i've not seen that particular broken pipe before
[01:21:08] <scfc_de>	 IIRC that has been around since Labs inception.  Don't know what causes it, though (i. e., whether the error is client or server wise)..
[01:21:11] <mutante>	 yea, uhm, cant run puppet 
[01:28:35] <jeremyb>	 scfc_de, intermittent?
[01:31:23] <scfc_de>	 jeremyb: On tools-login it's about every ~ 2 minutes (/var/log/syslog).  That would look like toolwatcher which basically does "getent passwd" or something like that.
[01:32:22] <scfc_de>	 (Because it has a "sleep 120" in a forever loop.)
[01:34:32] <mutante>	 so my labs instance says
[01:34:33] <mutante>	 The last Puppet run was at Sat Jun 14 19:38:55 UTC 2014 (88195 minutes ago).
[01:34:36] <mutante>	 sigh
[01:34:56] <mutante>	 or it's "in progress" since then :p
[01:36:32] <mutante>	 kills the lockfile and gets changes in  module Puppet::Parser::Functions
[01:37:18] <mutante>	 applies tons of changes and red lines flying by :p
[01:37:43] <mutante>	 "sweet" - Warning: /Stage[main]/Apache/Service[apache2]: Skipping because of failed dependencies
[01:38:33] <mutante>	 but the site setup uses new method.. something :)
[01:38:34] <ebernhardson>	 also looks like scap on beta is failing :( 
[01:39:11] <mutante>	  mariadb-server : Depends: mariadb-server-5.5 (= 5.5.38+maria-1~precise) but 5.5.36+maria-1~precise is to be installed
[01:39:14] <mutante>	 groar
[01:40:02] <mutante>	 i had puppetized that back when we had the "test" branch , hah
[01:40:27] <mutante>	 Warning: /Stage[main]/Certificates::Wmf_labs_ca/File[/etc/ssl/certs/wmf-labs.pem]: Skipping because of failed dependencies
[01:41:18] <mutante>	  apache2 : Depends: apache2-mpm-worker (= 2.2.22-1ubuntu1.7) but it is not going to be installed or
[01:41:34] <mutante>	 ok, it's broken in a bunch of ways 
[01:42:01] <mutante>	 db, webserver, nscld, puppet itself.. 
[01:42:51] <mutante>	 what i wanted to do was just apply a little change, now time to fix all that now
[01:43:06] <mutante>	 s/now/no  .. laters cya
[01:51:05] <spagewmf>	 greg-g: ^ 18:38 < ebernhardson> also looks like scap on beta is failing :( !  Flow doesn't have its latest JS files deployed on  the labs cluster
[01:51:43] <spagewmf>	 does scap use hhvm? :)
[01:53:40] <ebernhardson>	 its actually the rsync thats failing, afaict.  failures start showing up in scap.log and syslogaug 14 00:31:27
[02:12:57] <spagewmf>	 ebernhardson: I'm filing a bug...
[02:14:48] <spagewmf>	 ebernhardson , bd808|BUFFER , greg-g : Bug 69590 - rsync errors to beta cluster, inconsistent state after scap
[02:25:31] <ebernhardson>	 sigh, its the obvious things you dont think to look at
[02:25:31] <ebernhardson>	 /dev/vda1                                                  7.6G  7.2G  1.5M 100% /
[02:25:35] <ebernhardson>	 spagewmf: ^^
[02:27:17] <spagewmf>	 let's take this to ops
[02:28:21] <jeremyb>	 ebernhardson, what host is that?
[02:28:56] <ebernhardson>	 jeremyb: deployment-bastion.eqiad.wmflabs
[02:31:12] <spagewmf>	 bus homeward. Thanks ebernhardson !
[02:33:21] <jeremyb>	 ok, i'm taking a look
[02:33:39] <jeremyb>	 hrmmm
[02:33:53] <jeremyb>	 or do i not have root? so hard to remember :)
[02:34:23] <jeremyb>	 gah, why no root?
[02:34:43] <jeremyb>	 i thought only the varnish/nginx hosts should have root removed?
[02:34:58] <ebernhardson>	 i have root, but not sure what exactly would be the "extra" data
[02:37:22] <ebernhardson>	 generally, it looks to me like this server just needs a bigger allocation,  /srv/scap-stage-dir/php-master is 5.3G on a 7.6G partition
[02:38:25] <jeremyb>	 ebernhardson, no, /srv is not on /
[02:38:49] <ebernhardson>	 no? hmm didnt see it on df but i'm known to miss things occasionally :)
[02:39:09] <ebernhardson>	 of course, /srv is a symlink :P 
[02:39:20] <jeremyb>	 for i in / /srv /srv/; do stat "$i"; done
[02:39:38] <jeremyb>	 compare device ids. and yes, look for symlinks too
[02:40:18] <jeremyb>	 i'm going to try moving something
[02:40:25] <jeremyb>	 yeah, no that don't work
[02:40:41] <jeremyb>	 $ mv -v /srv-old /mnt/srv-old
[02:40:52] <jeremyb>	 mv: cannot create directory `/mnt/srv-old': Permission denied
[02:41:58] <jeremyb>	 /var is nearly full too
[02:46:37] <ebernhardson>	 !log beta /dev/vda1 full. moved /srv-old to /mnt/srv-old and freed up 2.1G
[02:46:37] <labs-morebots>	 beta is not a valid project.
[02:47:12] <ebernhardson>	 :P
[02:47:32] <jeremyb>	 ebernhardson, so, now /var ?
[02:48:11] <ebernhardson>	 yup, looking
[02:53:38] <ebernhardson>	 mostly logs, nothing too inspring.  500M of zip'd access history in /var/log/account/
[02:53:44] <ebernhardson>	 s/zip/gz/
[03:01:10] <jeremyb>	 ebernhardson, band-aid could be clear the apt cache
[03:01:24] <jeremyb>	 281496  /var/cache
[03:01:24] <jeremyb>	 247344  /var/cache/apt
[03:01:33] <jeremyb>	 (KB)
[03:17:12] <jeremyb>	 greg-g, ping
[03:17:12] <greg-g>	 jeremyb: You sent me a contentless ping.  This is a contentless pong.  Please provide a bit of information about what you want and I will respond when I am around.
[03:17:18] <jeremyb>	 hah
[03:17:23] <jeremyb>	 greg-g, ping ping
[03:20:40] <jeremyb>	 !log deployment-prep 02:46:37 UTC <ebernhardson> !log beta /dev/vda1 full. moved /srv-old to /mnt/srv-old and freed up 2.1G
[03:20:42] <labs-morebots>	 Logged the message, Master
[03:44:47] <spagewmf>	 bits.beta.wmflabs.org is unresponsive, not accepting connections. So no load.php from beta cluster
[03:44:59] <spagewmf>	 jeremyb: thanks for freeing up space, dunno if this is related
[03:45:19] <spagewmf>	 I guess apaches need a restart
[03:46:15] <jeremyb>	 spagewmf, ebernhardson actually did it not me :)
[03:47:09] <spagewmf>	 but http://en.wikipedia.beta.wmflabs.org/wiki/Dido_Sotiriou is OK, so hmm
[11:37:49] <ori>	 !log deployment-prep deployment-cache-bits01 unresponsive; console shows OOMs: https://dpaste.de/LDRi/raw . rebooting
[11:37:51] <labs-morebots>	 Logged the message, Master
[12:28:31] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - 10https://bugzilla.wikimedia.org/69601 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None The instances provided on labs eqiad (such as deployment-bastion.eqiad.wmflabs) have a /var/ of...
[12:30:28] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - 10https://bugzilla.wikimedia.org/69601 (10Antoine "hashar" Musso)
[12:30:28] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: diamond does not compress its logs - 10https://bugzilla.wikimedia.org/69602 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None The python stats collector diamond does not compress its log under /var/log/diamond/ . That might lead to /var/ filling up entirel...
[12:36:56] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - 10https://bugzilla.wikimedia.org/69601 (10Antoine "hashar" Musso)
[12:37:00] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: acct (process and login accounting) fill up instances /var/ partition - 10https://bugzilla.wikimedia.org/69604 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None I noticed 'acct' generates log files under /var/log/account/ which is on a 2GB partition.  It i...
[12:42:13] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - 10https://bugzilla.wikimedia.org/69601 (10Antoine "hashar" Musso)
[12:42:15] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: atop (monitoring system) logs fill up instances /var/ partition - 10https://bugzilla.wikimedia.org/69605 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None I noticed 'atop' generates log files in /var/log/ which is on a 2GB partition. atop has a logrotate r...
[13:05:08] <grrrit-wm>	 (03PS1) 10Addshore: Send codesniffer changes to #wikimedia-qa [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154262 
[13:06:41] <grrrit-wm>	 (03PS1) 10Addshore: Remove deleted extension Diff from #wikidata [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154263 
[13:14:13] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Wikitech: SAL transclusion on project pages should be limited - 10https://bugzilla.wikimedia.org/48065 (10Andre Klapper) p:5High>3Normal a:5Ryan Lane>3None
[13:27:02] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] Send codesniffer changes to #wikimedia-qa [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154262 (owner: 10Addshore)
[15:12:57] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - 10https://bugzilla.wikimedia.org/69601 (10Greg Grossmeier) p:5Unprio>3High
[15:13:11] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: diamond does not compress its logs - 10https://bugzilla.wikimedia.org/69602 (10Greg Grossmeier) p:5Unprio>3Normal
[15:13:26] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: acct (process and login accounting) fill up instances /var/ partition - 10https://bugzilla.wikimedia.org/69604 (10Greg Grossmeier) p:5Unprio>3Normal
[15:14:26] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: atop (monitoring system) logs fill up instances /var/ partition - 10https://bugzilla.wikimedia.org/69605 (10Greg Grossmeier) p:5Unprio>3Normal
[15:16:43] <andrewbogott>	 Coren: There sure have been a lot of bugs about /var/log filling up today.  Should we rethink the default partitioning of new instances?  I could adjust the instance flavors to make /var bigger, or could turn on the 'biglogs' class by default for new instances...
[15:16:57] <andrewbogott>	 (although that latter might cause weird races)
[15:48:13] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: diamond does not compress its logs - 10https://bugzilla.wikimedia.org/69602#c1 (10Yuvi Panda) 5NEW>3RESO/WON This used to happen really consistently (see bug 66458). We 'fixed' it by essentially making diamond log only on errors, which shouldn't happen often. The role a...
[15:48:56] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: diamond does not compress its logs - 10https://bugzilla.wikimedia.org/69602#c2 (10Yuvi Panda) You might have leftover archive.log and diamond.log files from before the fixes. If so, feel free to rm -rf them, diamond won't re-create.
[16:33:57] <ebernhardson>	 something odd still going on in labs,  `dsh -M -g mediawiki-installation md5sum /a/common/php-master/extensions/Flow/includes/Formatter/TopicListFormatter.php` gives one set of md5 for -bastion and -rsync01, and a different set for the rest
[16:34:03] <ebernhardson>	 in beta labs i mean
[16:34:32] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Send codesniffer changes to #wikimedia-qa [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154262 (owner: 10Addshore)
[16:34:48] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Remove deleted extension Diff from #wikidata [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154263 (owner: 10Addshore)
[16:35:26] <grrrit-wm>	 (03CR) 10Legoktm: [V: 032] Remove deleted extension Diff from #wikidata [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154263 (owner: 10Addshore)
[16:35:32] <grrrit-wm>	 (03CR) 10Legoktm: [V: 032] Send codesniffer changes to #wikimedia-qa [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154262 (owner: 10Addshore)
[16:36:24] <legoktm>	 !log tools restarting grrrit-wm
[16:36:26] <labs-morebots>	 Logged the message, Master
[16:38:15] <legoktm>	 why isn't it back...?
[16:40:17] <legoktm>	 ugh
[16:45:16] <legoktm>	 :O
[16:45:23] <legoktm>	 interesting
[16:45:30] <legoktm>	 !log tools fixed grrrit-wm 
[16:45:33] <labs-morebots>	 Logged the message, Master
[16:45:54] <legoktm>	 I guess it's not going to spam the missed messages.
[16:46:09] <grrrit-wm>	 (03PS1) 10Legoktm: Fix syntax error [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154288 
[16:46:12] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Fix syntax error [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154288 (owner: 10Legoktm)
[16:46:15] <grrrit-wm>	 (03Merged) 10jenkins-bot: Fix syntax error [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/154288 (owner: 10Legoktm)
[16:46:17] <legoktm>	 :D
[16:54:06] <addshore>	 whoops legoktm :P
[16:54:41] <legoktm>	 it wasn't your fault
[16:54:48] <legoktm>	 I merged both together which caused the problem
[16:54:54] <legoktm>	 I think
[16:54:57] <legoktm>	 I blame jenkins
[16:55:00] <addshore>	 doesnt jenkins lint it? :P
[16:55:04] <legoktm>	 it should!
[16:55:18] <legoktm>	 I also did bypass it on V+2, but the initial patch should have
[16:55:42] <Cyberpwer678>	 LEGO!!!
[16:56:24] <legoktm>	 o/
[16:56:34] <Cyberpwer678>	 Hi
[16:58:00] <legoktm>	 what's up?
[16:58:05] <Cyberpwer678>	 legoktm, do you suppose tool labs could use an OAuth library.  I came up with an ultra easy to use library, that does all the work for you essentially.
[16:58:25] <legoktm>	 like......https://github.com/valhallasw/flask-mwoauth ?
[16:58:32] <legoktm>	 https://github.com/valhallasw/flask-mwoauth maybe.
[16:58:57] <legoktm>	 also even https://github.com/wikimedia/MediaWiki-OAuth
[16:59:06] <valhallasw`cloud>	 Cyberpwer678 is more of the PHP-aligned kind, if I remember correctly :-)
[16:59:29] <legoktm>	 psh
[16:59:46] * Cyberpwer678 is honored that users remember specifics about me.
[17:00:05] <Cyberpwer678>	 legoktm, valhallasw`cloud I meant a PHP version.
[17:00:21] <valhallasw`cloud>	 legoktm: yes, the dark side! But even the dark side deserves a sensible OAuth library that doesn't make you poke your eyes out.
[17:00:55] <legoktm>	 Cyberpwer678: yeah, seems useful then I guess. There's also anomie's oauth-hello-world which is in PHP
[17:01:18] <Cyberpwer678>	 valhallasw`cloud, it may be the dark side, but it only affects users who can't control the dark powers of PHP. :D
[17:01:29] <Cyberpwer678>	 I'm in control.
[17:01:46] <valhallasw`cloud>	 legoktm: doesn't that one do some creepy json-based oauth?
[17:02:04] <valhallasw`cloud>	 the php example on the oauth page does
[17:02:17] * valhallasw`cloud keeps reading Cyberpwer as Cyberpwner
[17:02:24] <legoktm>	 https://tools.wmflabs.org/oauth-hello-world/index.php?action=download
[17:03:03] <legoktm>	 you mean the JWT stuff?
[17:03:22] <legoktm>	 I don't really understand how most of this works anyways...
[17:03:24] <valhallasw`cloud>	 https://github.com/Stype/mwoauth-php
[17:03:26] <valhallasw`cloud>	 look what I found :-p
[17:04:31] <Cyberpwer678>	 legoktm, it's similar, but the implementation is simpler.  $OAuth = new OAuth( $key, $secret, $baseURL, $apiURL );
[17:04:38] <valhallasw`cloud>	 the hello world one is better than I remember (although all the CURLing still makes me want to poke my eyes out)
[17:05:04] <valhallasw`cloud>	 https://github.com/Stype/mwoauth-php/blob/master/demo.php is quire nice
[17:05:13] <Cyberpwer678>	 If the user is not authenticated, it essentially does nothing, and waits for $OAuth->login();
[17:05:43] <Cyberpwer678>	 It will automatically redirect to the correct location.
[17:07:06] <Cyberpwer678>	 When oauth_verifier is defined in the URL, $OAuth = new OAuth( ... ); will complete the authentication and redirect the user back to the page they left.
[17:08:34] <Cyberpwer678>	 And $OAuth->apiQuery( $post, $params ); will do an API query as that logged in user.
[17:09:06] <Cyberpwer678>	 $OAuth->logout() is self-explanatory.
[17:09:29] <Cyberpwer678>	 valhallasw`cloud, simple right? ^
[17:11:12] <Cyberpwer678>	 valhallasw`cloud, legoktm: simple right?
[17:25:06] <spagewmf>	 greg-g Flow team is hurting from incomplete/inconsistent rsync to beta labs, https://bugzilla.wikimedia.org/show_bug.cgi?id=69590 
[17:26:04] <greg-g>	 spagewmf: yeah, and the scap-aware people are mostly still traveling :/
[17:26:08] <spagewmf>	 greg-g: looks like bd808|BUFFER  is out and ori is well-deserved vacay, who can help ebernhardson ?
[17:26:10] * greg-g pings some others
[17:26:40] <greg-g>	 Reedy: can you help debug whats going on with https://bugzilla.wikimedia.org/show_bug.cgi?id=69590 ?
[17:45:13] <spagewmf>	 If scap on beta labs runs every 10 minutes, it's strange the cluster is still inconsistent
[19:06:43] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Deploy BounceHandler extension to beta cluster - 10https://bugzilla.wikimedia.org/69621 (10Kunal Mehta (Legoktm)) 3NEW p:3Unprio s:3normal a:3None Before a full deployment to production.
[19:08:56] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Deploy BounceHandler extension to beta cluster - 10https://bugzilla.wikimedia.org/69621#c1 (10Antoine "hashar" Musso) Have the extension registered in mediawiki/extensions.git (use sync-with-gerrit.py script at the root of that repository to do so).  Then it is all...
[19:30:14] <greg-g>	 YuviPanda: around?
[19:30:52] <YuviPanda>	 greg-g: 'sup
[19:31:25] <greg-g>	 YuviPanda: this bug is eluding us: https://bugzilla.wikimedia.org/show_bug.cgi?id=69590
[19:31:57] <YuviPanda>	 greg-g: ah, I saw around. sadly I've never seen scap in my life before nor have I done anything in deployment-prep... :(
[19:32:13] <YuviPanda>	 such low bus factor
[19:32:17] <greg-g>	 yeah :(
[19:34:11] <YuviPanda>	 greg-g: can't really do much there, sorry :(
[19:35:08] <greg-g>	 YuviPanda: s'ok
[19:35:13] <greg-g>	 thought I'd try :)
[19:41:43] <hashSpeleology>	 YuviPanda: I am going to dig in scap :-D
[19:41:55] <YuviPanda>	 hashSpeleology: \o/ good luck
[19:42:22] <hashSpeleology>	 I havent closely followed scap dev unfortunately :-D
[19:42:28] <hashSpeleology>	 but it must log something somewhere
[20:07:49] <ebernhardson>	 hashSpeleology: there are logs, but the funny thing is the logs claim its working (but the resulting files on the servers claim it doesnt) :(
[20:13:03] <hashSpeleology>	 !log deployment-prep bunch of instances have a full /var/log :-/
[20:13:06] <labs-morebots>	 Logged the message, Master
[20:13:50] <hashSpeleology>	 !log deployment-prep on deployment-mediawiki01 deleting /var/log/apache2/debug.log.1
[20:13:51] <labs-morebots>	 Logged the message, Master
[20:13:59] <hashSpeleology>	 !log deployment-prep on deployment-mediawiki01 deleting /var/log/apache2/access.log.1
[20:14:01] <labs-morebots>	 Logged the message, Master
[20:14:06] <hashSpeleology>	 no clue why they log locally that is lame
[20:16:39] <hashSpeleology>	 !log deployment-prep cleaning up /var/log on deployment-mediawiki02
[20:16:41] <labs-morebots>	 Logged the message, Master
[20:20:29] <hashSpeleology>	 !log deployment-prep rebooting mediawiki01  , /var refuses to clear out and stick at 100% usage
[20:20:32] <labs-morebots>	 Logged the message, Master
[20:39:57] <hashSpeleology>	 !log deployment-prep in case it is not in SAL.  MediaWiki is no more synced to app server {{bug|69590}}
[20:40:00] <labs-morebots>	 Logged the message, Master
[20:52:27] <hashSpeleology>	 !log deployment-prep attempting to unbreak mediawiki code update {{bug|69590}} by cherry picking {{gerrit|154329}}
[20:52:30] <labs-morebots>	 Logged the message, Master
[20:54:30] <hashSpeleology>	 !log deployment-prep puppet is proceeding on mediawiki01
[20:54:32] <labs-morebots>	 Logged the message, Master
[20:55:04] <hashSpeleology>	 !log deployment-prep puppet administratively disabled on mediawiki02 . Assuming some work in progress on that host. Leaving it untouched
[20:55:06] <labs-morebots>	 Logged the message, Master
[20:57:21] <hashSpeleology>	 !log deployment-prep deployment-rsync01 : deleting /usr/local/apache/common-local content.  Then ln -s /srv/common-local /usr/local/apache/common-local  as set by beta::common which is not applied on that host for some reason.  {{bug|69590}}
[20:57:22] <labs-morebots>	 Logged the message, Master
[21:06:30] <hashSpeleology>	 ebernhardson: thank you a ton for your investigation on the beta cluster not updating code ( https://bugzilla.wikimedia.org/show_bug.cgi?id=69590 )
[21:06:40] <hashSpeleology>	 ebernhardson: you saved me a looooot of time.
[21:06:54] <hashSpeleology>	 spagewmf: beta cluster should be updating again
[21:07:23] <spagewmf>	 je t'aime mon frere
[21:15:33] <hashSpeleology>	 -:]
[21:28:15] <spagewmf>	 next problem: it doesn't look like DB updates are running.  http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA gets Error: 1146 Table 'enwiki.echo_target_page' doesn't exist .
[21:30:20] <spagewmf>	 Echo extension adds this in getSchemaUpdates hook, $updater->addExtensionTable( 'echo_target_page', "$dir/db_patches/echo_target_page.sql" );
[21:31:25] <ebernhardson>	 that problem is shown here: https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=aawiki/3304/console
[21:31:31] <ebernhardson>	 and i think this would fix it: https://gerrit.wikimedia.org/r/#/c/154231/
[21:31:34] <spagewmf>	 on https://integration.wikimedia.org/ , there's a broken icon next to Beta cluster DB update and https://integration.wikimedia.org/ci/view/Beta/ shows failures for 3 days
[21:35:29] <spagewmf>	 ebernhardson: I think Adam's around, I'll ping to look at that patch
[21:35:46] <hashSpeleology>	 spagewmf: ebernhardson yeah the DB is broken 
[21:35:50] <hashSpeleology>	 https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=enwiki/3323/console
[21:35:53] <hashSpeleology>	 enwiki on beta cluste
[21:35:54] <hashSpeleology>	 r
[21:36:03] <hashSpeleology>	 21:34:45 Adding template_logging_comments field to table cn_template_log ...A database query error has occurred.
[21:36:03] <hashSpeleology>	 21:34:45 Query: ALTER TABLE `cn_template_log` ADD `tmplog_comment` varchar(255) DEFAULT NULL
[21:36:03] <hashSpeleology>	 21:34:45 
[21:36:03] <hashSpeleology>	 21:34:45 Function: DatabaseBase::sourceFile( /mnt/srv/scap-stage-dir/php-master/extensions/CentralNotice/patches/patch-template-logging-comments.sql )
[21:36:03] <hashSpeleology>	 21:34:45 Error: 1060 Duplicate column name 'tmplog_comment' (10.68.16.193)
[21:36:04] <hashSpeleology>	 :D
[21:36:31] <hashSpeleology>	 looks like some code does not notice the column is already there
[21:36:44] <hashSpeleology>	 maybe it can just be dropped, or the database updater is broken
[21:36:58] <ebernhardson>	 db updater is broken, central notice has wrong info in the updater: https://gerrit.wikimedia.org/r/#/c/154231/
[21:37:37] <hashSpeleology>	 how the hell do you manage to write and propose a patch so fast?
[21:38:15] <ebernhardson>	 i wrote it last night :P
[21:38:37] <ebernhardson>	 last night i noticed that while debugging the other beta issues,  but it was just another side issue
[21:40:56] <hashSpeleology>	 so the databases will remain broken until that patch lands in
[21:41:04] <hashSpeleology>	 or the culprit is reverted / merged :D
[21:41:16] <hashSpeleology>	 I dont want to mess with CentralNotice so just gave it a +1
[21:41:41] <hashSpeleology>	 I need the jobs to whine on IRC / mail
[21:42:25] <hashSpeleology>	 !log deployment-prep Beta cluster database updates are broken due to CentralNotice. Fix up is {{gerrit|154231}}
[21:42:27] <labs-morebots>	 Logged the message, Master
[21:55:52] <spagewmf>	 awight +2d the CentralNotice fix... onward :)
[21:57:48] <legoktm>	 !log deployment-prep set $wgVERPsecret in PrivateSettings.php
[21:57:50] <labs-morebots>	 Logged the message, Master
[22:04:12] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Deploy BounceHandler extension to beta cluster - 10https://bugzilla.wikimedia.org/69621#c3 (10Kunal Mehta (Legoktm)) I set $wgVERPsecret in PrivateSettings.php and have scheduled the deployment to beta labs for next Monday's SWAT: <https://wikitech.wikimedia.org/w/i...
[22:23:16] <spagewmf>	 labs is back in business.  Thanks so much ebernhardson and hashSpeleology 
[22:23:20] <spagewmf>	 *beta labs
[22:24:55] <hashSpeleology>	 spagewmf: oh you guys did all the job :-]
[22:25:05] <hashSpeleology>	 I just created a symlink :D
[22:25:29] <hashSpeleology>	 https://integration.wikimedia.org/ci/view/Beta/  might be worth bookmarking
[22:26:37] <jeremyb>	 what is the meaning of that nick?
[22:28:02] <mutante>	 he's in a cave
[22:28:10] <mutante>	 per http://en.wiktionary.org/wiki/speleology
[22:28:17] <mutante>	 i wonder how he is online in there :)
[22:33:29] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Jenkins jobs updating beta cluster should whine - 10https://bugzilla.wikimedia.org/69628 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None Beta cluster is updated by Jenkins. Whenever a build breaks, nobody get informed though we can always have a...
[22:38:42] <spagewmf>	 "You are in a maze of symlinks, all alike".  Zuul as Zork
[22:41:22] <spagewmf>	 hashSpeleology: thanks. I go to https://integration.wikimedia.org/ and click "Jenkins view for Beta"
[22:42:38] <MC8>	 hm, those icons on the right are broken for me
[22:46:17] <hashSpeleology>	 MC8: they point to labs Ganglia which is broken :/
[22:46:46] <hashSpeleology>	 spagewmf: hurrah.  The source code for that page is in integration/docroot.git if you ever want to add more links / views / details etc :/°
[22:46:48] <hashSpeleology>	 :-]
[22:46:53] <hashSpeleology>	 I am off to bed *wave*
[22:47:01] <MC8>	 jenkins' 500 page logo is downright terrifying
[23:11:13] <mutante>	 !log puppet-compiler - deleted job #200 because it was 3GB , larger than any other job, and the instance ran out of disk space, breaking compiler
[23:11:13] <labs-morebots>	 puppet-compiler is not a valid project.
[23:11:19] <mutante>	 !instance puppet-compiler
[23:11:19] <wm-bot>	 need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances  want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance  want resources? use !resource <instance id>
[23:11:28] <mutante>	 gimme the project for the instance :p
[23:11:30] <mutante>	 wm-bot: 
[23:11:30] <wm-bot>	 Hi mutante, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-)
[23:16:39] <mutante>	 what is the file again in the instance filesystem that holds the project name? looks
[23:17:26] <mutante>	 ah, $INSTANCEPROJECT of course
[23:18:06] <mutante>	 !log puppet3-diffs - puppet-compiler02 ran out of disk - deleted job #200 because it was 3GB 
[23:18:08] <labs-morebots>	 Logged the message, Master
[23:18:13] <jeremyb>	 /etc/wmflabs-project ?
[23:18:25] <mutante>	 echo $INSTANCEPROJECT  does it
[23:18:35] <jeremyb>	 or `cat /etc/wmflabs-project` :)
[23:18:35] <mutante>	 that works too , thx:)
[23:23:50] <jeremyb>	 fgrep INSTANCE /etc/bash.bashrc | egrep -e '^export ' | cut -d' ' -f 2-