[00:00:45] <wikibugs>	 3Wikimedia Labs / 3tools: tools-webgrid-01 / full - 10https://bugzilla.wikimedia.org/66864 (10metatron) 3UNCO p:3Unprio s:3critic a:3Marc A. Pelletier And unfortunately, the grid directs all web(re)starts in the lighty-queue to this host.  tools-webgrid-01:~$ df -h Filesystem...
[00:33:20] <Betacommand>	 Mono: you still around?
[00:40:31] <wikibugs>	 3Wikimedia Labs / 3tools: tools-webgrid-01 / full - 10https://bugzilla.wikimedia.org/66864#c1 (10Tim Landscheidt) 5UNCO>3RESO/FIX a:5Marc A. Pelletier>3Tim Landscheidt There were 3,7 GByte of VIPS temporary files in /tmp:  | scfc@tools-webgrid-01:~$ ll -rt /tmp/vips-* | -rw-r--r-- 1 tools.zoomviewer...
[04:18:24] <TParis>	 Hey guys, I'm having database problems, can anyone help me?
[04:18:41] <TParis>	 array ( 0 => 'HY000', 1 => 29, 2 => 'File \'./utrs/loggedInUsers.MYD\' not found (Errcode: 30)', )'
[04:18:57] <TParis>	 I can open my database over ssh, but I cant open any tables
[07:20:31] <wikibugs>	 3Tool Labs tools / 3[other]: Migrate http://toolserver.org/~dispenser/* to Tool Labs - 10https://bugzilla.wikimedia.org/66868 (10Thorsten) 3UNCO p:3Unprio s:3normal a:3None Heaviliy used tools by Dispenser (https://toolserver.org/~dispenser/view/Main_Page) have to be moved to Tool Labs.  See also htt...
[07:20:43] <wikibugs>	 3Tool Labs tools / 3[other]: Migrate http://toolserver.org/~dispenser/* to Tool Labs - 10https://bugzilla.wikimedia.org/66868 (10Thorsten)
[07:20:59] <wikibugs>	 3Tool Labs tools / 3[other]: [tracking] toolserver.org tools that have not been migrated - 10https://bugzilla.wikimedia.org/58865 (10Thorsten)
[07:49:28] <wikibugs>	 3Wikimedia Labs / 3tools: Install libGeoIP - 10https://bugzilla.wikimedia.org/62649#c3 (10fox) 5PATC>3RESO/FIX The library is now installed and also the python bindings are importable :) thanks!
[08:38:51] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs was modified, changed by Sandaru link https://www.mediawiki.org/w/index.php?diff=1041603 edit summary: [+13] 
[08:40:44] <YuviPanda|zzz>	 `fox`: :) yw! (just saw bz)
[12:38:46] <wikibugs>	 3Tool Labs tools / 3[other]: Erwin's related changes does work properly with wikidata. - 10https://bugzilla.wikimedia.org/66874 (10Andre Koopal) 3NEW p:3Unprio s:3normal a:3None From bug 60872 by Helder, creating new bug as it is a different issue:  As reported on https://pt.wikipedia.org/w/index.php...
[12:40:29] <wikibugs>	 3Tool Labs tools / 3[other]: Erwin's related changes does work properly with wikidata. - 10https://bugzilla.wikimedia.org/66874#c1 (10Andre Koopal) My comment in the other bug:  I know the issue, at the moment I don't have time to look into this in detail, if I make time I first want to look at why commons i...
[12:40:43] <wikibugs>	 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~erwin85/relatedchanges.php to Tool Labs - 10https://bugzilla.wikimedia.org/60872#c7 (10Andre Koopal) I have split of the report above to bug 66874, as it is a different issue.
[12:48:31] <akoopal>	 Hi, what is te process to get a new component in bugzilla?
[12:48:38] <akoopal>	 ^te^the
[12:49:48] <Nemo_bis>	 akoopal: filing a bug against Wikimedia>Bugzilla
[12:50:01] <Nemo_bis>	 there are docs under [[mw:Bugzilla]]
[12:50:24] <akoopal>	 thanks
[12:51:20] <valhallasw>	 Nemo_bis: filtering for *.sql should be possible, tes
[12:51:21] <valhallasw>	 yes
[12:52:03] <valhallasw>	 Nemo_bis: note that the *filename* should start with mediawiki in your example
[12:52:28] <Nemo_bis>	 valhallasw: yes, I intentionally added it wrong so that he doesn't get emails for now
[12:52:44] <Nemo_bis>	 how hard is it to set that for a list of extensions
[12:52:55] <valhallasw>	 impossible without enumerating at the moment
[12:53:18] <Nemo_bis>	 valhallasw: each in its own section?
[12:53:24] <valhallasw>	 but something can be added to https://github.com/valhallasw/gerrit-reviewer-bot/blob/master/add_reviewer.py to support it
[12:53:27] <valhallasw>	 yes
[12:53:42] <Nemo_bis>	 k
[12:54:21] <valhallasw>	 e.g. by adding a 'project_regex=' parameter
[12:56:28] <valhallasw>	 I'd be happy to merge a patch, but I have no time to develop it at the moment
[13:30:44] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate http://tools.wmflabs.org/erwin85/catanalyzer.php to Tool Labs - 10https://bugzilla.wikimedia.org/60868 (10Andre Koopal)
[13:31:14] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/categorycount.php to Tool Labs - 10https://bugzilla.wikimedia.org/60869 (10Andre Koopal)
[13:31:29] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/contribs.php to Tool Labs - 10https://bugzilla.wikimedia.org/60870 (10Andre Koopal)
[13:31:30] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/randomarticle.php to Tool Labs - 10https://bugzilla.wikimedia.org/60871 (10Andre Koopal)
[13:31:43] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/shortpages.php to Tool Labs - 10https://bugzilla.wikimedia.org/60873 (10Andre Koopal)
[13:31:59] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/talkcatintersect.php to Tool Labs - 10https://bugzilla.wikimedia.org/60874 (10Andre Koopal)
[13:31:59] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/newpages.php to Tool Labs - 10https://bugzilla.wikimedia.org/60875 (10Andre Koopal)
[13:32:13] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/subpages.php to Tool Labs - 10https://bugzilla.wikimedia.org/60876 (10Andre Koopal)
[13:32:29] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/xwiki.php to Tool Labs - 10https://bugzilla.wikimedia.org/60878 (10Andre Koopal)
[13:32:30] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/delete.php to Tool Labs - 10https://bugzilla.wikimedia.org/60877 (10Andre Koopal)
[13:32:59] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/blockfinder.php to Tool Labs - 10https://bugzilla.wikimedia.org/60880 (10Andre Koopal)
[13:32:59] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/projects.php to Tool Labs - 10https://bugzilla.wikimedia.org/60879 (10Andre Koopal)
[13:33:14] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Erwin's related changes does work properly with wikidata. - 10https://bugzilla.wikimedia.org/66874 (10Andre Koopal)
[13:33:28] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/relatedchanges.php to Tool Labs - 10https://bugzilla.wikimedia.org/60872 (10Andre Koopal)
[13:33:30] <akoopal>	 Sorry for the spam :-)
[13:43:43] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Migrate http://tools.wmflabs.org/erwin85/catanalyzer.php to Tool Labs - 10https://bugzilla.wikimedia.org/60868#c1 (10Andre Koopal) From Jira:  Retrieved categories.  Warning: chmod(): No such file or directory in /home/erwin85/public_html/catanalyzer.php on line 341 at Mon,...
[14:32:29] <wikibugs>	 3Wikimedia Labs / 3tools: Install libGeoIP - 10https://bugzilla.wikimedia.org/62649#c4 (10Tim Landscheidt) a:5Marc A. Pelletier>3Yuvi Panda (Quim uses the assignee field for statistics IIRC.)
[14:39:11] <scfc_de>	 @seen Mono
[14:39:11] <wm-bot>	 scfc_de: Last time I saw Mono they were quitting the network with reason: Quit: Connection closed for inactivity N/A at 6/20/2014 2:16:08 AM (12h23m3s ago)
[14:39:16] <scfc_de>	 @notify Mono
[14:39:16] <wm-bot>	 I'll let you know when I see Mono around here
[14:40:53] <scfc_de>	 Does someone know if Mono has a Bugzilla account?
[14:47:40] <scfc_de>	 !log tools Restarted webservice for mono; cf. bug #64219
[14:47:42] <labs-morebots>	 Logged the message, Master
[14:48:58] <wikibugs>	 3Wikimedia Labs / 3tools: webservice creates blocking files and jobs when called from a user account with an eponymous tool - 10https://bugzilla.wikimedia.org/64219#c1 (10Tim Landscheidt) This happened again for mono whose webservice I just restarted.
[15:06:43] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Erwin's related changes does work properly with wikidata. - 10https://bugzilla.wikimedia.org/66874#c2 (10Andre Klapper) Hmm, does the bug summary "Erwin's related changes does work properly with wikidata" really summarize the issue described in this ticket?
[15:49:30] <wikibugs>	 3Wikimedia Labs / 3tools: Cannot submit sge jobs from tools-webgrid-tomcat - 10https://bugzilla.wikimedia.org/66882 (10Peter Schlömer (dapete)) 3NEW p:3Unprio s:3major a:3Marc A. Pelletier I assume this is basically the same as Bug 54786 ('Make exec hosts as submit hosts too'), just concerning tools-...
[15:59:18] <DeltaQuad>	 Anyone here able to assist with an sql instance that went to lunch and never returned.
[15:59:21] <ali_king_intern>	 I requested a project through the online form a few days ago, and haven't heard anything back.  Does anyone know how long it usually takes to get approval?
[16:00:04] <gifti>	 ali_king_intern: forever
[16:00:48] <ali_king_intern>	 gifti: thanks.  I'm wanting to get developing ASAP so may have to go for another option
[16:01:34] <gifti>	 you could ping someone who is responsible and around (don't ask me who)
[16:02:45] <greg-g>	 andrewbogott: ^ re project approval
[16:03:02] <greg-g>	 ping - {{done}}
[16:03:13] <andrewbogott>	 ali_king_intern: looking...
[16:04:16] <andrewbogott>	 ali_king_intern: RDFIO?
[16:04:53] <ali_king_intern>	 andrewbogott: yes, that's the one!
[16:05:42] <ali_king_intern>	 thanks :)
[16:05:57] <andrewbogott>	 ali_king_intern: ok -- no caps allowed, so it's rdfiodev.
[16:06:09] <andrewbogott>	 btw, if you figure out a way to install smw w/out composer… I want to hear about it :/
[16:06:14] <ali_king_intern>	 fine by me
[16:08:02] <ali_king_intern>	 I think I did it without as I had to install a previous version using Extension Manager, but that's deprecated
[16:08:23] <ali_king_intern>	 I'll check with the project mentors
[16:15:00] <DeltaQuad>	 andrewbogott: you able to help me start up the SQL service again...I think my n00bness is showing...
[16:15:39] <andrewbogott>	 DeltaQuad: maybe… on tools, or do you have a private project?
[16:16:00] <DeltaQuad>	 private project.
[16:16:05] <DeltaQuad>	 it seems like it just died
[16:17:45] <andrewbogott>	 generally you can just 'sudo service mysql restart'
[16:17:55] <andrewbogott>	 or, hm, mysql-server?  lemme check
[16:18:19] <DeltaQuad>	 its mysql
[16:18:26] <DeltaQuad>	 and I can't stop or start the job
[16:18:38] <andrewbogott>	 'the job'?
[16:18:42] <andrewbogott>	 what does it say when you restart?
[16:19:02] <DeltaQuad>	 deltaquad@utrs-primary:~$ sudo service mysql status
[16:19:03] <DeltaQuad>	 mysql stop/waiting
[16:19:05] <DeltaQuad>	 deltaquad@utrs-primary:~$ sudo service mysql restart
[16:19:06] <DeltaQuad>	 stop: Unknown instance:
[16:19:08] <DeltaQuad>	 start: Job failed to start
[16:20:15] <andrewbogott>	 ah, well… then something is broken :)  You'll have to look in the logs to see what's happening.  Probably /var/log/mysql
[16:20:38] <andrewbogott>	 And, I'd say, start with checking that you have disk space in the places that mysql needs to work
[16:22:15] <wikibugs>	 3Wikimedia Labs / 3tools: Cannot submit sge jobs from tools-webgrid-tomcat - 10https://bugzilla.wikimedia.org/66882#c1 (10Tim Landscheidt) 5NEW>3ASSI a:5Marc A. Pelletier>3Tim Landscheidt lighttpd nodes already allow that, so IMHO it's not controversial.  I'll look into how to do this later.  Re Catg...
[16:24:42] <DeltaQuad>	 andrewbogott: the first thing close to the bottom of the sql error log that seems relevant is "140620  0:01:36 [ERROR] /usr/sbin/mysqld: Incorrect key file for table './utrs/appeal.MYI'; try to repair it"
[16:25:41] <andrewbogott>	 hm, I'd be surprised if that stopped the service from coming up
[16:25:45] <andrewbogott>	 But it seems serious in any case.
[16:26:22] <andrewbogott>	 DeltaQuad: what instance is this?
[16:26:22] <DeltaQuad>	 I have to run for now, but i'll be back
[16:26:33] <andrewbogott>	 oh, utrs-primary
[16:28:58] <andrewbogott>	 DeltaQuad: looks like app-armor is preventing you from running for some reason
[18:03:30] <rillke>	 andrewbogott_afk, hello, is it possible that I missed something while filing a new project request?
[18:04:17] <rillke>	 just because at, https://wikitech.wikimedia.org/wiki/New_Project_Request/MediaHandler_tests hasn't been processed for nearly one week now
[18:28:31] <YuviPanda>	 rillke: mostly because he's been busy, I guess.
[18:28:33] <YuviPanda>	 andrewbogott_afk: ^
[18:28:43] <YuviPanda>	 andrewbogott_afk: can I also get righs to approve new project creation on wikitech?
[18:49:20] <TParis>	 YuviPanda: Ping
[18:49:28] * YuviPanda wa ves at TParis
[18:49:31] <YuviPanda>	 hello
[18:55:59] <liangent>	 YuviPanda: are you root?
[18:56:45] <liangent>	 nvm I resolved it myself
[18:58:54] <YuviPanda>	 liangent: yeah
[18:58:55] <YuviPanda>	  ok
[19:03:47] <andrewbogott>	 rillke: all you missed was nagging me here :)
[19:05:17] <andrewbogott>	 rillke: no CamelCase allowed, though, so the project is called 'mediahandler-tests'
[19:06:09] <YuviPanda>	 andrewbogott: can you login to utrs-primary? I can't seem to...
[19:06:31] <andrewbogott>	 I did a couple hours ago...
[19:06:41] <YuviPanda>	 ah, hmm
[19:06:48] <andrewbogott>	 YuviPanda: yep, working fine
[19:07:04] <andrewbogott>	 I don't know what project it's in, though… maybe you aren't a member?
[19:08:04] <YuviPanda>	 andrewbogott: I am projectadmin
[19:08:25] <YuviPanda>	 andrewbogott: nevermnd, got it now
[19:09:51] <rillke>	 thank you, andrewbogott
[19:15:48] <Krenair>	 https://wikitech.wikimedia.org/wiki/Special:Contributions/127.0.0.1 - is this supposed to be happening?
[19:16:53] <andrewbogott>	 Krenair: it's a bug in the nova callback that updates instance pages.  It's been there for a while and I haven't looked at it
[19:17:44] <Krenair>	 Also why wasn't https://wikitech.wikimedia.org/wiki/Nova_Resource:Otrs updated to include me as admin and the new instance?
[19:17:51] <Krenair>	 The special pages show it...
[19:19:12] <YuviPanda>	 !log utrs ran sudo dpkg-reconfigure mysql-server-5.5, since starting mysql kept failing
[19:19:14] <labs-morebots>	 Logged the message, Master
[19:20:14] <andrewbogott>	 Krenair: I think it's because SMW is broken.
[19:20:25] <andrewbogott>	 Those pages often fall out of sync and I don't know why
[19:33:29] <Krenair>	 andrewbogott, so, I'm trying to set up an OTRS labs instance
[19:33:47] <Krenair>	 But I'm pretty sure http://git.wikimedia.org/blob/operations%2Fpuppet.git/7cd814686241a19ba5b4fa51da78d4fbca2c7f8a/manifests%2Frole%2Fotrs.pp has production-specific stuff in that will stop this from working..?
[19:34:11] <YuviPanda>	 Krenair: try it and see? :) but yeah, most probably yes
[19:34:27] <YuviPanda>	 Krenair: yeah, seems to
[19:34:30] <Krenair>	 Yeah it doesn't appear to be running anything on port 80
[19:35:05] <YuviPanda>	 Krenair: looking at it, it also is hard coding db names to cluster things
[19:35:23] <Krenair>	 But also no /opt/otrs directory...
[19:35:35] <YuviPanda>	 and the passwords
[19:35:37] <Krenair>	 YuviPanda, yeah, that's what I saw first
[19:35:40] <YuviPanda>	 Krenair: did you force a puppet run?
[19:35:44] <Krenair>	 no...
[19:35:51] * Krenair does not know puppet at all
[19:35:59] <YuviPanda>	 Krenair: 'sudo puppet agent -tv'
[19:36:28] <Krenair>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Class[Exim4] is already declared in file /etc/puppet/manifests/role/mail.pp:5; cannot redeclare at /etc/puppet/manifests/mail.pp:64 on node i-00000434.eqiad.wmflabs
[19:36:28] <Krenair>	 Warning: Not using cache on failed catalog
[19:36:28] <Krenair>	 Error: Could not retrieve catalog; skipping run
[19:36:29] <YuviPanda>	 Krenair: see if that errors?
[19:36:35] <YuviPanda>	 Krenair: heh
[19:36:52] <YuviPanda>	 Krenair: did you also pick the mail role or something?
[19:36:56] <Krenair>	 nope.
[19:37:40] <andrewbogott>	 Krenair: you are free to drop dummy passwords into the labs-private repo in order to make puppet happy.
[19:37:52] <YuviPanda>	 andrewbogott: I don't think puppet even got to that part
[19:37:55] <andrewbogott>	 Probably you can commit to the repo; if not I'll do it for you.
[19:38:01] <andrewbogott>	 Oh, ok then...
[19:38:07] <Krenair>	 andrewbogott, I think there are more problems than just passwords
[19:38:15] <YuviPanda>	 andrewbogott: see the puppet error he pasted, there's a class conflict...
[19:38:42] <YuviPanda>	 andrewbogott: although I don't see how that's happening, since role mail isn't really invoked directly from there
[19:40:12] <andrewbogott>	 The labs base class includes some basic mail roles.
[19:40:17] <andrewbogott>	 Or, a basic mail role
[19:40:41] <YuviPanda>	 aaah
[19:40:46] <YuviPanda>	 that... might be the problem
[19:41:11] <andrewbogott>	 role::labs::instance
[19:41:20] <andrewbogott>	 includes role::mail::sender
[19:43:25] <YuviPanda>	 Krenair: yeah, otrs role would need... some work before it can work on labs
[19:43:30] <YuviPanda>	 realm branching should be ok, I'd think
[19:43:58] <YuviPanda>	 Krenair: 1. setup a self-hosted puppetmaster thing on an instance, 2. hack at the puppet role until it works, 3. commit / get that patch to make it work merged, 4. create another instance
[19:44:22] <Krenair>	 oh, fun. okay
[19:44:54] <YuviPanda>	 Krenair: :D shouldn't be too hard, I think
[19:45:11] <Krenair>	 yeah, well, I'll just have to learn about puppet.
[19:45:31] <YuviPanda>	 Krenair: :D puppet is fun
[19:45:58] <Krenair>	 made a new instance to work on this
[19:46:15] <Krenair>	 Hi Jeff_Green 
[19:46:39] <YuviPanda>	 Krenair: ideal would be to convert the OTRS role into a module
[19:46:50] <andrewbogott>	 In most cases you won't have to branch on realm, just add some if ! defined checks for your duplicate packages
[19:46:52] <Jeff_Green>	 ohai
[19:47:06] <Krenair>	 Jeff_Green, the otrs role has a bunch of production-specific things in
[19:47:23] <Jeff_Green>	 yeah, that makes sense
[19:47:23] <YuviPanda>	 andrewbogott: hmm, true. but I think even better than ifs would be to fix the underlying code itself to not cause that to happen :D
[19:47:24] <Krenair>	 So we'll have to:
[19:47:27] <Krenair>	 <YuviPanda> Krenair: 1. setup a self-hosted puppetmaster thing on an instance, 2. hack at the puppet role until it works, 3. commit / get that patch to make it work merged, 4. create another instance
[19:47:43] <wikibugs>	 3Tool Labs tools / 3Erwin's tools: Erwin's related changes doesn't handle wikidata edits properly - 10https://bugzilla.wikimedia.org/66874 (10Andre Koopal)
[19:47:45] <Jeff_Green>	 also . . . it doesn't do a whole lot anyway iirc.
[19:48:07] <YuviPanda>	 yeah, just seems to setup packages and monitoring and a mailer and spamassasin
[19:48:13] <Jeff_Green>	 right
[19:48:22] <Jeff_Green>	 otrs is not particularly friendly for packaging or puppet
[19:48:29] <YuviPanda>	 yeah.
[19:48:33] <YuviPanda>	 Jeff_Green: how is it deployed right now?
[19:48:36] <YuviPanda>	 just a git checkout?
[19:48:41] <Jeff_Green>	 not even
[19:48:43] <Jeff_Green>	 sec
[19:48:58] <Jeff_Green>	 it's not good for git either :-(
[19:49:04] <YuviPanda>	 ...
[19:49:05] <YuviPanda>	 ouch
[19:49:13] <Jeff_Green>	 the issue--flat file config is modified on the fly by the webUI
[19:49:20] <Jeff_Green>	 and there is *also* config stored inthe database
[19:49:28] <Jeff_Green>	 and it's modperl :-)
[19:49:30] <YuviPanda>	 aaah, the ZNC way of doing things that makes everything a PITA
[19:49:36] <Jeff_Green>	 yeah
[19:49:54] <Jeff_Green>	 I did a doc overhaul when we did the big upgrade
[19:49:55] <Krenair>	 Hm... So do I tick the 'puppetmaster' box under the puppet group?
[19:50:04] <Jeff_Green>	 https://wikitech.wikimedia.org/wiki/OTRS
[19:50:10] <Krenair>	 Do I tick the 'role::puppet::self' under puppet-development?
[19:50:16] <andrewbogott>	 Krenair: no, there's a puppet development group...
[19:50:18] <YuviPanda>	 Krenair: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster
[19:50:19] <Jeff_Green>	 mind you the upgrade doc we worked from was over 20 pages long
[19:50:21] <andrewbogott>	 yes, role::puppet::self is all you need.
[19:50:48] <Krenair>	 I saw that but admittedly should have read all of it rather than jumping to 'Set up a single instance self hosted puppetmaster'
[19:50:55] <YuviPanda>	 Krenair: :D
[19:51:15] <Jeff_Green>	 the 100 lines or so are the important part
[19:51:16] <YuviPanda>	 Krenair: 1. tick the role, 2. run sudo puppet agent -tv, 3. go to /var/lib/git/operations/puppet 4. do git.
[19:51:19] <Jeff_Green>	 under Upgrading
[19:53:00] <YuviPanda>	 Krenair: and welcome to puppet :)
[19:53:52] <Jeff_Green>	 puppet doesn't seem like a good name for it
[19:54:26] <Jeff_Green>	 it should be maybe "donkey" to account for all the times it stubbornly refuses to do things
[19:55:17] <Krenair>	 okay, starting the instance with the otrs role was a mistake. seem like it made the run fail without becoming a puppetmaster
[20:01:49] <Krenair>	 YuviPanda, hmm... I don't think it worked properly...
[20:02:00] <Krenair>	 console log coming in a sec
[20:02:43] <Krenair>	 YuviPanda, https://gist.github.com/Krenair/ee7b84e3ad09f4d7e66e
[20:03:19] <Krenair>	 Ah... I notice the puppet run I did after turning on puppetmaster gave errors for it
[20:03:40] <Krenair>	 Error: /Stage[main]/Puppet::Self::Master/Service[puppetmaster]: Failed to call refresh: Could not start Service[puppetmaster]: Execution of '/etc/init.d/puppetmaster start' returned 1: 
[20:03:40] <Krenair>	 Error: /Stage[main]/Puppet::Self::Master/Service[puppetmaster]: Could not start Service[puppetmaster]: Execution of '/etc/init.d/puppetmaster start' returned 1: 
[20:04:42] <Krenair>	 ran "sudo /etc/init.d/puppetmaster start" myself, it's fine now
[20:04:53] <Krenair>	 gives the otrs exim error I was expecting
[20:12:19] <Krenair>	 Jeff_Green, YuviPanda: So with this error it's the Class['exim4::ganglia'] requirement for monitoring ganglia's 'exim_messages_in' that is breaking stuff?
[20:14:45] <Jeff_Green>	 that is dependency hell
[20:15:29] <Jeff_Green>	 exim4::ganglia I think installs a script that scrapes exim mail logs and feeds message counts to ganglia
[20:16:59] <Krenair>	 ew
[20:27:37] <Krenair>	 Jeff_Green, I put the monitor_ganglia block referencing exim4 inside a check for $::realm == 'production'
[20:27:40] <Krenair>	 but I get the same error
[20:30:07] <Jeff_Green>	 it's probably required by exim or something
[20:30:12] <Jeff_Green>	 lemme add it to the things you can install
[20:37:20] <Jeff_Green>	 alright, exim4::ganglia added
[20:38:41] <Krenair>	 Jeff_Green, I have no idea what you just did or what I need to do to make it take effect
[20:39:00] <Jeff_Green>	 i just added it to the list of things that puppet can install on an instance
[20:39:31] <Jeff_Green>	 just another class you can add like role::otrs
[20:40:08] <Krenair>	 I see
[20:41:56] <Krenair>	 I ticked the box on wikitech, saved and re-ran puppet. no difference
[20:45:30] <Jeff_Green>	 unfortunatetly I don't know--labs vs puppet is not something I'm very familiar with
[20:46:06] <Cyberpower678>	 anomie, I have a question.
[20:46:30] <anomie>	 Cyberpower678: umm, good?
[20:46:38] <Cyberpower678>	 anomie, you take part in the development of the API correct?
[20:46:43] <anomie>	 Cyberpower678: yes
[20:47:15] <Cyberpower678>	 Well I had a look at ApiFormatJson.php
[20:47:27] <Cyberpower678>	 Seems to be overly complex to format json
[20:47:34] <bd808>	 Krenair: need puppet help? I may be useful with that than I was with beta questions yesterday.
[20:47:46] <Krenair>	 Yes
[20:47:52] <Krenair>	 (I have no idea what I'm doing etc.)
[20:48:08] <bd808>	 :) I didn't really when I started either
[20:48:12] <anomie>	 Cyberpower678: How so?
[20:49:04] <bd808>	 Krenair: Can I get access to your instance or do you want to debug using pastebin to show me what's going wrong?
[20:50:02] <Cyberpower678>	 anomie, wouldn't using the function $this->printText( json_encode( $this->getResultData() ) ); be a whole lot simpler. 
[20:50:31] <Cyberpower678>	 json_encode is a PHP function so I see no need to write your own.
[20:50:57] <anomie>	 Cyberpower678: Except for the callback feature and the fact that FormatJson::encode wraps json_encode to account for missing features in PHP 5.3's json_encode, that's pretty much exactly what it *does* do.
[20:51:20] <Krenair>	 bd808, added you
[20:51:36] <Krenair>	 see instance otrs-puppetisation
[20:51:45] <Cyberpower678>	 anomie, ah.  Very well.
[20:51:46] <Krenair>	 I'm dealing with /var/lib/git/operations/puppet/manifests/role/otrs.pp
[20:52:49] * bd808 logs in and runs `sudo -s; puppet agent --test --verbose`
[21:00:39] <bd808>	 Krenair: Looking now. Looks like you may have applied two classes/roles that haven't been put on the same host at the same time before.
[21:01:37] <scfc_de>	 !log tools tools-webgrid-tomcat: Added to submit host list with "qconf -as" for bug #66882
[21:01:40] <labs-morebots>	 Logged the message, Master
[21:01:44] <wikibugs>	 3Wikimedia Labs / 3tools: Cannot submit sge jobs from tools-webgrid-tomcat - 10https://bugzilla.wikimedia.org/66882#c2 (10Tim Landscheidt) I added tools-webgrid-tomcat to the list of submit hosts; I'll leave this bug open until I've puppetized that.  Please test if it works for you now.
[21:02:31] <Krenair>	 bd808, yep. Just not sure about much more details than that...
[21:09:05] <bd808>	 Krenair: role::labs::instance (which you need in labs) applies role::mail::sender; role::otrs applies exim::roled
[21:11:00] <bd808>	 Both classes apply class exim4 with different parameters
[21:11:42] <bd808>	 Fixing will be interesting...
[21:12:28] <bd808>	 Krenair: do you just need this host to work or are you trying to make this more generally usable (installing otrs in labs via puppet)
[21:13:01] <Krenair>	 bd808, this is not work-related, I help with OTRS stuff as a volunteer
[21:14:08] <Krenair>	 Someone wanted to query the DB, I was going to help but needed to check some things. Turns out we don't actually have a test setup in labs
[21:14:44] <Krenair>	 Went to do that, and it turns out the puppet manifest for otrs doesn't actually work in labs
[21:19:31] <bd808>	 Something weird is happening on that host with respect to the puppetmaster. I just commented out one of the conflicting definitions and the puppet run still fails with the same error message.
[21:20:23] <bd808>	 Krenair: Did you have puppet running cleanly from the self-hosted master before you turned on the otrs class?
[21:20:38] <Krenair>	 The puppet master didn't turn on properly when I enabled it
[21:20:56] <Krenair>	 I ran "sudo /etc/init.d/puppetmaster start" myself, and it appeared to start
[21:21:20] <bd808>	 Ok. Can you uncheck the role::otrs class in wikitech and I'll fix that problem first
[21:21:39] * bd808 can't see the config page in wikitech for this instance; no admin right?
[21:22:06] <Krenair>	 done
[21:22:24] <Krenair>	 (and added you to projectadmin)
[21:32:34] <bd808>	 Krenair: I got the puppetmaster working. Now I'll try my hack for otrs again
[21:36:32] <Krenair>	 bd808, thanks for looking at this for me
[21:46:57] <bd808>	 Krenair: Partial success. The catalog compiles and applies now. There are a bunch of failures that look like bugs in the otrs module/role itself. Nothing makes the /opt/otrs directory :/
[21:48:04] <bd808>	 My "fix" was a complete hack: I commented out the guts of the role::mail::sender class and left an alert() saying I did so
[21:48:12] <Krenair>	 :/
[21:48:43] <bd808>	 More than one entry point requires role::mail::sender and I didn't try to track them all down
[22:16:58] <bd808>	 Krenair: I got a little farther, but not all the way to success. It looks to me like role::otrs doesn't actually install the perl otrs code expected in /opt/otrs ; maybe that has to be done manually? :/
[22:20:13] <Krenair>	 Jeff might know