[06:09:39] !log project-proxy a default robots.txt is served now. see [[Help:Proxy#robots.txt]] [06:09:41] Logged the message, Master [06:18:27] [bz] (NEW - created by: Legoktm, priority: Unprioritized - normal) [Bug 47278] Unable to log in with api.php on wikitech.wikimedia.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=47278 [08:56:44] !log wikidata-dev wikidata-testrepo commented out "$wgWBSettings['entityNamespaces'][CONTENT_MODEL_WIKIBASE_QUERY]" in /srv/mediawiki/wikidata_repo_requires.php [08:56:47] Logged the message, Master [09:11:23] [bz] (ASSIGNED - created by: Antoine "hashar" Musso, priority: Normal - enhancement) [Bug 44041] [OPS] adapt varnish mobile puppet class on beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=44041 [09:25:40] !log deployment-prep Updating mobile cache (deployment-varnish-t3) to patchset 47567/9 . Some puppet changes got merged in this morning :-] [09:25:42] Logged the message, Master [09:28:40] zeljkof: I have updated the mobile cache on beta :-] [09:28:51] zeljkof: that might have broken something though my first tests do not show any troubles. [09:29:21] hashar: thanks for letting me know, will check :) [09:34:57] !log applying role::cache::mobile to deployment-cache-mobile01 (that will replace deployment-varnish-t3 eventually) [09:34:58] applying is not a valid project. [09:35:17] !log deployment-prep applying role::cache::mobile to deployment-cache-mobile01 (that will replace deployment-varnish-t3 eventually) [09:35:19] Logged the message, Master [09:37:00] [bz] (NEW - created by: Željko Filipin, priority: High - normal) [Bug 47194] GettingStarted extension not at beta.wmflabs.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=47194 [12:34:08] [bz] (NEW - created by: Željko Filipin, priority: High - normal) [Bug 47203] Math extension broken at beta.wmflabs.org: "No backend defined with the name `global-multiwrite`." or "Missing texvc executable" - https://bugzilla.wikimedia.org/show_bug.cgi?id=47203 [13:59:04] !ping [13:59:04] pong [13:59:36] hi [13:59:55] how can I tell puppet to install mysql or some other module [14:00:13] do I have to add it to some other file ? [14:07:32] labs-morebots, you keeping it together? [14:07:32] I am a logbot running on tools-exec-02. [14:07:32] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [14:07:32] To log a message, type !log . [14:10:35] hi sumanah [14:10:45] Hi Oren_Bochman [14:10:54] sumanah: is there some updated puppet docs for labs ? [14:11:21] Oren_Bochman: I'm sorry, I don't know. petan or Damianz or andrewbogott or other people here would know, though [14:11:39] ok [14:12:14] sumanah: bye [14:12:17] Bye [14:12:22] you could also ask labs-l [14:12:24] the list [14:13:39] there was a tutorial session last wikimania but we could not upload the video [14:14:49] Oren_Bochman, can you tell me what you're looking for? [14:14:55] yep [14:15:13] I found a puppet def on github for installing moodle [14:15:48] but it requires mysql as well as definitions for the database's password [14:16:13] I dont know 1. where the mysql password is defined [14:16:36] The password for moodle's db, or the root mysql password? [14:16:58] root mysql password [14:17:33] Hm… I'm pretty sure it's empty to start off. [14:17:43] also for somereason mysql was not installed on that instance [14:17:45] could be [14:17:56] Have you spent any time on the 'configure instance' page? [14:18:04] yes [14:18:19] Which package did you select for mysql? [14:18:24] sec [14:18:52] Oren_Bochman: look at https://commons.wikimedia.org/wiki/File:Puppet_Tutorial.ogv [14:19:04] Video from the Wikimedia Foundation tech days, September 11, 2012, explaining Puppet configuration management in the context of Wikimedia's site/services infrastructure. Speaker/slides: Ryan Lane. [14:19:24] sumanah: thanks [14:19:38] Oren_Bochman, I think the one I use generally is role::labs-mysql-server (which, I now see, I failed to document :( ) [14:19:44] andrewbogott: webserver::php5-mysql [14:20:02] php works [14:20:20] Hm, that should do it. If mysql isn't getting installed then you should see obvious errors when you run puppet [14:20:25] but mysql seems not to be installed [14:20:36] Um… to run puppet by hand w/error messages, run 'sudo puppetd -tv' [14:20:48] I run puppet but I don't think there are any errors [14:20:58] what instance is this? I can poke around. [14:21:16] sure it is he-moodle [14:21:19] (And, to answer your original question: I don't think there's any good top-level documentation about this. I'll think about writing some.) [14:21:54] I started writing a puppet tutorial last year [14:22:09] it is at https://meta.wikimedia.org/wiki/User:OrenBochman/Labs_2 [14:22:29] but I don't know enough yet to finish it [14:23:12] where is the puppet error log supposed to be ? [14:25:40] It would appear on the console when you run puppet. [14:25:45] Oren_Bochman, run $ dpkg --list | grep mysql [14:25:59] There are some mysql packages there. I'm surprised there isn't a server though... [14:30:13] Oren_Bochman: I'm looking at a role I wrote to install LAMP in labs, and it installs several things: "apachesetup", "role::labs-mysql-server", "webserver::php5-mysql", "webserver::php5" [14:30:20] I'm quite surprised about that last one... [14:30:26] but the first three should get you going. [14:30:48] Meanwhile, I will probably make a labs:LAMP role to simplify this. [14:31:22] webserver::php5-mysql might be broken a bit [14:31:32] Or just misnamed. [14:31:45] which file has the local definitinos [14:32:07] Which definitions [14:32:08] ? [14:32:17] for the local machine [14:32:33] i.e. which modules to have on the local machnine [14:32:55] You mean, what determines which puppet classes are included? [14:33:14] yes [14:33:41] It's done via ldap, controlled by the web interface. [14:33:52] So even though you're using puppetmaster::self you can still do things via 'configure instance' [14:34:09] I was told it only works for the initial install [14:34:26] Hm… nope. [14:34:33] also if I want to add stuff using pupet master self [14:34:47] like I added a moodle module [14:34:51] You can add custom classes for your project here: https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup [14:35:02] I need to add that [14:35:18] Um… I am describing my workflow, here. I think that some people edit the site.pp file directly, but I'm not sure how they get the changes in the right place... [14:35:26] And of course you'd want to be careful not to commit any changes to site.pp [14:35:52] And I'm not sure how site.pp interacts with the ldap definitions… I just use the web interface for all this. [14:35:57] site.pp [14:36:07] that is what I need [14:36:10] I think [14:36:34] there was a tutorial by ryan about this - see the link by sumanah above [14:36:57] https://commons.wikimedia.org/wiki/File:Puppet_Tutorial.ogv [14:37:00] Video from the Wikimedia Foundation tech days, September 11, 2012, explaining Puppet configuration management in the context of Wikimedia's site/services infrastructure. Speaker/slides: Ryan Lane. [14:37:31] I'm viewing it now [14:37:40] Oren_Bochman: I need to step away but will be back in 20. Let me know if you run into trouble. [14:37:49] ok thanks [15:07:06] hi [15:09:07] Oren_Bochman, any luck? [15:16:54] still trying [15:47:42] paravoid: When talking about the debian-packaging of mod_tile / renderd, you mentioned that you could push packages into debian, but would obviously want to contact debian-gis first. [15:48:37] I was planning on sending a mail to the debian-gis list to try and get their response on packaging mod_tile [15:49:13] Do you think that is a good idea, or should I wait, till you think the packages are ready? [16:02:22] !log deployment-prep Updating mobile cache to use some mark change {{gerrit|59401}} [16:02:25] Logged the message, Master [16:28:32] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 47249] puppetmaster::self unusuable - https://bugzilla.wikimedia.org/show_bug.cgi?id=47249 [16:59:28] ottomata, puppetmaster::self and role::puppet::self are broken. Should I revert your changes, or are you already furiously working to fix that? [16:59:47] hmmmm, i did not know they were broken! [16:59:52] i will work furiously though! [17:00:03] whats up? [17:00:36] Neither can be installed on a new instance… "Exiting; no certificate found and waitforcert is disabled" [17:00:40] i made changes yesterday and tested them, they were working for me then... [17:00:40] oh [17:00:44] Or, rather, they can be installed, but then they never work after that. [17:00:49] hmmmm [17:00:56] https://bugzilla.wikimedia.org/show_bug.cgi?id=47249 [17:01:18] on friday, i had that problem on a new host creation, before even applying self hosted stuff [17:01:49] That's something different. Previously (up until a couple of weeks ago) you couldn't log into an instance before the first puppet run completed. [17:01:57] Now you can. But if you try to run puppet /during/ that puppet run you will get a similar error. [17:02:07] I'm pretty sure the two things are unrelated. [17:02:26] hm [17:03:10] can you get me the puppet.conf file? [17:03:15] /etc/puppet/puppet.conf [17:03:54] The failure is on testlabs-abogott-dev7 [17:03:59] I can add you to that project if you aren't in it already [17:04:13] i'min [17:06:17] ottomata, what's your @wikimedia email? [17:06:33] otto@ [17:06:38] i'm in the project [17:07:37] yep, just adding you to the bug [17:07:40] k [17:16:33] [bz] (RESOLVED - created by: Legoktm, priority: Unprioritized - normal) [Bug 47278] Unable to log in with api.php on wikitech.wikimedia.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=47278 [17:41:23] andrewbogott: is this a new instance, or something that was running puppetmaster::self before my changes went in? [17:42:00] ottomata: New. And, it's number seven because I reproduced the problem on several other times on fresh instances today. [17:42:02] also, is it using puppet::self::master or puppetmaster::self? [17:42:06] ok [17:42:07] hm [17:42:16] The same error happens with both the role and puppetmaster::self. [17:42:21] k [17:42:21] But the one you're looking at uses the role class. [17:42:44] yeah sorry, role::puppet:;self is what I meant to say. ok thanks [17:45:51] I added a node def [17:45:56] but I get an error [17:46:16] Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class php for he-moodle.pmtpa.wmflabs at /etc/puppet/manifests/site.pp:2929 on node he-moodle.pmtpa.wmflabs [18:04:33] hm. it seems no one is doing shell requests :( [18:04:59] does Special:UserRights have an api? [18:05:18] I kind of want to make that queue have a single-click interface [18:05:32] !log wikiversity-sandbox Updated wikiversity-sandbox-frontend and rebooted it [18:05:33] Logged the message, Master [18:05:59] Ryan_Lane: Yes, it should [18:06:03] it does :) [18:06:04] yeah [18:06:11] cool. that's easy enough to add [18:06:16] hell, I can do it with a gadget [18:06:27] Ryan_Lane: I'm doing them right now but, yeah, I don't check regularly. [18:06:40] I need to give more volunteers this ability [18:07:16] andrewbogott: i'm still testing some new instances to make sure things are ok, but your instance looks happier right now [18:07:27] I can do a action=userrights and an action=edit [18:07:41] ottomata: OK, I will retest shortly. [18:07:57] then remove the object from the page [18:11:21] that would be better! [18:11:44] yeah :) [18:11:52] I started doing some api work the other day [18:11:56] and some javascript modules [18:12:17] to make the interface suck less [18:13:01] hmm, andrewbogott, i'm not so sure if I fixed it. your instance's puppet ran…hm, [18:13:39] Ryan_Lane, are you adding javascript smarts to webforms in general? Or is this labs-specific? [18:14:01] openstackmanager specific [18:14:45] Seems like HTMLforms could generally use some tightening up, although I haven't thought very hard about that. [18:15:48] @notify Nullzero [18:15:48] I will notify you, when I see Nullzero around here [18:16:01] !ping [18:16:02] pong [18:16:06] !pong [18:16:07] don't play with me [18:16:11] :-D [18:31:32] petan: can you make the bot complain when shell reqs are waiting too long? (<--- andrewbogott) [18:52:50] jeremyb_: Maybe, but probably just sending an echo notification on account creation would be better. [18:53:19] huh [18:53:35] idk. first echo needs fixing with monobook [18:53:47] it's not so useful now. especially not if it's every 5 mins [18:54:14] Is monobook what wikitech is using now? [18:54:20] It seems sort of reasonable to me as is [18:54:31] Hey Coren. YOu asked me to ping me 24h later about a non-glusterfs filesystem :) [18:54:32] monobook is what i use... [18:54:48] ah [18:54:54] you click the echo button and it pops up under the page body [18:55:30] Wikitech defaults to vector, andrewbogott [18:56:20] This is monobook: https://wikitech.wikimedia.org/w/index.php?title=Main_Page&useskin=monobook [18:56:28] andrewbogott: https://bugzilla.wikimedia.org/46059 [18:56:51] I think we should strip all skins but vector from wikitech [18:57:01] I don't want to test against a bunch of skins [18:57:33] YuviPanda|Meds: Added you to shellmanagers [18:57:51] thanks Ryan_Lane. I'll watchlist the shellreqs [18:58:06] not sure that's possible [18:58:14] it's a SMW query [18:58:20] you can get an RSS feed of it [18:58:42] what's Matt Walker's wiktiech name? [18:59:30] ah, hmm [18:59:33] i'll add it to feedly then [18:59:53] Ryan_Lane: this isn't a labs bug anyway. it's upstream [19:00:15] ah, yeah [19:00:23] echo only works in vector [19:00:29] this is a problem for strapping, too [19:00:46] it kinda works. you just can't read what it says [19:00:56] yeah. in strapping you can read it [19:01:01] but it's not aligned properly [19:01:24] Ryan_Lane: 16 18:31:32 < jeremyb_> petan: can you make the bot complain when shell reqs are waiting too long? (<--- andrewbogott) [19:01:39] (you weren't here at the time) [19:01:42] which bot [19:01:48] wm-bot [19:01:50] ah [19:01:50] or whatever [19:01:55] yes [19:02:04] that is actually very easy [19:23:21] any idea why ... err: /Stage[main]/Ganglia/Service[gmond]/ensure: change from stopped to running failed: Could not start Service[gmond]: Execution of '/etc/init.d/ganglia-monitor start' returned 1: at /etc/puppet/manifests/ganglia.pp:241 [19:24:08] Oren_Bochman: That's pretty well understood; you can ignore it for now. [19:24:16] Or if it bothers you, do 'sudo apt-get install ganglia-monitor' [19:24:31] thanks [19:26:07] andrewbogott: this is related to the image issue? [19:26:17] of course our package is also broken [19:26:25] so it causes issues in vmbuilder [19:26:27] Ryan_Lane: It's the same. [19:26:46] our package, when installing, tries to start the daemon, which fails [19:41:56] Ryan_Lane, has 'gluster volume start/stop' been working for you? [19:42:02] Oops [19:42:58] hmmm, gluster fs full? [19:43:05] instances / 100% use [19:44:45] ottomata, where are you seeing that? [19:45:47] hm, selftest4 and selftest5 on reportcard project [19:45:50] i only looked on those too [19:46:09] 2.5G /var/log/glusterfs [19:46:28] -rw------- 1 root root 2.5G Apr 16 19:46 /var/log/glusterfs/home.log [19:46:38] m [19:46:46] /var/log/ isnt' mounted via gluster [19:46:47] i deleted that file on selftest4 [19:46:50] ok [19:46:53] haha [19:46:54] so you just have runaway logfiles, it's a local problem. [19:47:04] I think I fixed log rotation but… maybe that patch hasn't merged yet. [19:47:05] uhh, but i don't have a /var mounted [19:47:10] just / [19:47:12] but yeah [19:47:31] /dev/vda1 3.8G 3.8G 0 100% / [19:51:08] Hello, I'd like to run my toolserver IRC bot on Wikimedia Labs. Can I join the Bots project? [19:52:41] Jelte: You can! Probably petan or coren can help [19:52:51] hey [19:53:07] Jelte yes, you can join both tools and bots [19:53:11] oh projects share space on / [19:53:12] ? [19:53:21] ottomata no [19:54:24] uhhhhhh why my / all used up in reportcard, hmmm [19:54:35] petanb|bnc-fu: What is the best one for running an IRC bot? [19:55:15] Jelte bots is supposed to be staging in future and tools production [19:55:25] we might change the names, I hope [19:55:29] anyway [19:55:35] ottomata: du -h / [19:55:50] yeah, there is a huge mount on /public [19:55:56] i'm du-ing all over the place to find it :) [19:55:58] I can give you access to bots and tools right now... bots is a nice place to play on and tools is rather production like [19:56:30] petanb|bnc-fu: That would be nice [19:58:53] !log bots added Jelte to bots and tools [19:59:10] Logged the message, Master [19:59:29] !botsdocs [19:59:29] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [19:59:31] !toolsdocs [19:59:31] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help [19:59:42] here is some reading if u were bored :P [20:00:34] petanb|bnc-fu: Thank you [20:00:40] yw [20:00:47] andrewbogott: can you give me admin rights on project testlabs? [20:00:55] i'd like to create instances there to test this puppet::self stuff [20:01:19] yep [20:01:41] danke! [20:02:14] Jelte do you need sql db? [20:02:29] petanb|bnc-fu: Yes, please [20:02:37] well, you don't really need to ask me for that it's self maintained :P [20:02:45] so you can create as many db's as you need [20:03:08] on bots it's quite simple, not sure how it works on tools atm, but you can actually access bots sql server from tools project as well [20:04:03] you can find out how to create / access db's in documentation of bots, on tools you need to have a tool account first... [20:05:17] ok. I will first try to run my bot in the bots project. [20:05:21] ok [20:05:43] running it in tools is quite more complicated atm but once it works on bots it should be easy to move it [20:06:16] btw in bots there is a server which you probably need [20:06:22] @labs-resolve ibnr [20:06:22] I don't know this instance - aren't you are looking for: I-00000635 (bots-ibnr1), [20:06:24] this one [20:06:34] that is for irc bots because it has low load [20:06:51] other boxes are very loaded and may not be good for irc bots because they would lag horribly [20:07:04] hi petan [20:07:08] hi [20:07:35] i still wonder why cron/fcron doesn't work for me on bots-4, can you help? [20:07:42] yes [20:07:57] but TBH you should stop using bots-4 as soon as you can :P [20:07:59] because we moved to SGE ages ago [20:08:26] but that doesn't really mean we are going to kill bots-4 any time soon [20:08:26] mh, ok [20:08:36] what is your problem with cron? [20:09:16] it does not run any processes for me [20:09:35] :/ [20:09:39] let me check [20:10:23] what is your id [20:10:25] unix [20:10:29] or username [20:10:30] whatever [20:10:35] gifti [20:10:39] ok [20:11:30] additionally i can't do sudo [20:11:34] are you sure that crontab isn't broken? it looks weird to me [20:12:10] yes sudo was disabled too :/ as we are merging with tools project, it might be eventually reenabled in future but we need to find out a secure way [20:12:12] how could i tell? [20:12:39] ah, ok, thanks [20:12:46] 0 22,23 * * * [ $(date +%-H) = 0 ] && aft-adt-sg.tcl $ [20:12:48] so, are we really merging? … wow [20:12:50] why so many spaces [20:13:09] giftpflanze not exactly. bots project is going to stay as a staging area for tools project [20:13:27] ah [20:13:27] so bots will become a testing platform, but configured in similar way [20:13:32] that's current plan [20:13:54] see /data/project/giftbot/crontab-4 for the layout [20:15:46] petanb|bnc-fu, giftpflanze: If that cron line is supposed to trigger on 0 CET/CEST, "TZ=:Europe/Berlin" must be prefixed to "date". [20:16:09] environment is not enough? [20:16:22] giftpflanze: Ah, okay, that should work as well. [20:16:45] giftpflanze your crontab works - if these 2 commands you have there work, I can;t tell [20:16:57] I just inserted a test job in there and it got launched [20:17:05] i see [20:17:24] that's good news :D [20:17:27] heh [20:22:06] novainstance broken for others too? [20:22:06] https://wikitech.wikimedia.org/wiki/Special:NovaInstance [20:24:41] works to me [20:24:49] hmmm me too now hmm [20:26:23] agh, so, andrewbogott, the thing where the initial puppet.conf wasn't correct just happened to me again [20:26:26] can you explain that to me? [20:26:51] Previously you couldn't log into an instance during the first puppet run. [20:26:52] server is not set in /etc/puppet/puppet.conf [20:27:00] So you never saw what it looked like before puppet finished. [20:27:14] I predict that if you tail the syslog you'll see that puppet is still running [20:27:32] yeah i noticed that once or twice, and i thought that was the problem too [20:27:36] i'm watching, but i don't htink it is [20:27:39] ps doesn't show it running either [20:28:08] i got this notification email that this instance was ready 17 minutes ago [20:28:11] and just logged in [20:28:26] what instance? [20:32:34] testlabs-otto0 [20:32:40] andrewbogott: ^ [20:32:54] i can fix this if I manually set server=virt0.wikimedia.org [20:34:20] Ah, so it has the same symptom as a system with ::self installed [20:34:47] I don't know why it's doing that. There must be a race in the puppet setup someplace :( [20:36:20] yeah, but this is different [20:36:28] this is pre ::self stuff [20:36:37] and, i'm pretty sure the ::self problem is different [20:36:51] the ::self problem is that puppet generate cert was being run before the compile puppet.conf exec [20:36:57] so the cert creation failed [20:37:06] Right -- I suspect that it's the same kind of problem, but a different example of it. [20:37:16] and then since there was no cert, and the ca changed, puppet couldn't ever run again, so it couldn't fix itself [20:37:24] coren, you're using labstore3 and labstore4, right? [20:37:33] in this case…puppet.conf doesn't have server set at all [20:37:44] is this the default puppet.conf that ships with the puppet packag? [20:42:38] ottomata: I don't know. Probably if you start a new instance it won't have the same problem. [20:43:23] yeah i just created a new one [20:43:30] what's weird is this happened to me on friday too [20:43:39] with two temp instances I created in the reportcard project (they are deleted now) [20:44:01] Coren, ping? [20:44:11] andrewbogott: semipong. [20:44:26] andrewbogott: Yes, 3 and 4. [20:44:51] OK then… Ryan_Lane, it looks like gluster is still trying to use labstore3. Various things are breaking as a result. [20:45:17] 'gluster peer status' gets me "State: Peer in Cluster (Connected)" for 3 [20:45:24] but Disconnected for 4 [20:45:27] o_O [20:45:38] I can disconnect it but… would like to know what's happening first [20:45:57] is gluster turned off and uninstalled for labstore3? [20:46:07] Ryan_Lane: The package is purged, even. [20:46:13] Hm. [20:46:22] There's still a glusterfs-client though. [20:46:24] andrewbogott: specifically what is breaking? [20:46:24] I guess that means it won't hurt for me to disconnect it :) [20:46:49] Coren: you *sure* it's gone? root 11118 1 0 Apr15 ? 00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid [20:46:57] looks like its running to me [20:47:00] Any time I try to start or stop a volume it fails. [20:47:08] p glusterfs-server - clustered file-system (server package) [20:47:08] p glusterfs-server:i386 - clustered file-system (server package) [20:47:13] Hostname: labstore3.pmtpa.wmnet [20:47:13] Uuid: 881081a3-677d-47ae-a735-dba2dc48f4a2 [20:47:13] State: Peer in Cluster (Connected) [20:47:19] the service is still runnin [20:47:19] o_O! [20:47:21] *runing [20:47:23] *running [20:47:26] *oashgoasgouhgrs [20:47:58] Se package, she is broo-kenne! [20:48:20] andrewbogott: is it fixed now? [20:48:41] stupid glusterfs [20:48:43] * Coren says evil things about a .deb that won't even stop its service when *purged* [20:48:58]