[00:00:35] how do I use it :-) [00:01:00] I need to merge… [00:01:31] I wonder if it should be called role::lamp::labs [00:03:00] nano [00:06:04] should I just include role::labs-lamp [00:06:22] I've not used roles befire [00:07:16] It's not quite ready yet, hang on... [00:07:32] ok [00:07:34] I wait [00:08:17] And, on your existing instance it won't do anything useful anyway… since you're using puppetmaster::self you only have access to local manifests. [00:08:23] And probably have lamp already working, right? [00:08:31] no [00:08:43] now php seems broken [00:09:07] oh… uhoh, my class may have the same problem :) [00:09:36] we will see... [00:11:16] Oh! You know, since puppetmaster::self is broken in the upstream (that's what ottomata and I were talking about)… I'd recommend against updating your puppet tree. [00:11:53] looks like I dont have any php modules [00:12:49] I run "sudo puppetd -tv" [00:13:05] does that update the tree or would I have to use git ? [00:15:00] puppetd -tv updates your instance to correspond with the current puppet tree. [00:15:24] If you are developing you want a local puppet tree that doesn't spontaenously change itself, which is what you have :) [00:15:56] So, you have the php binary present but not libraries that you'd expect? [00:16:18] I'm not sure [00:16:33] I see that there is a php def in the files directory [00:17:27] I had installed wamp+moodle on my dev machine - it took about 15 minutes [00:17:32] this is taking ages [00:19:46] andrewbogott: it looks I don't have a php module or manifest for it just the class defs. [00:22:15] Oren_Bochman, try adding the webserver::php5 package [00:22:22] um… class, I mean. [00:22:37] using the web ui ? [00:23:28] yeah [00:28:58] which my sql should I use [00:30:17] ? [00:30:26] What do you mean? [00:35:20] what is the difference between a class and a role [00:35:59] a role is a kind of class. [00:36:32] We're trying to refactor so that everything is contained within roles, at which point that 'configure instance' page will only show roles as options. [00:36:32] But, not there yet. [00:36:51] So the list on that page is a mishmash… many of the options include other options, kind of tangled up. [00:40:10] ok [00:40:28] I can't reboot the instance for some reason [00:41:04] How are you trying to reboot? [00:41:12] from the web ui [00:41:12] (And, I might ask why you want to...) [00:41:49] I applied the changes to php [00:42:27] well, wait, I'm looking at your instance now... [00:42:37] Did you have role::mediawiki-install::labs selected before? [00:43:01] erm I don't think so [00:43:12] but you do now? [00:43:20] no Idea [00:43:42] ok. So… you're working on the instance he-moodle, right? [00:43:51] yes [00:44:08] Right now it has role::mediawiki-install::labs selected, and a hostname entered for the instance (one that won't work.) [00:44:11] Did you do that? If not, who did? [00:44:49] I did [00:44:56] just now [00:45:29] I figured if media wiki works then i'd have a working lamp [00:45:57] But you don't actually want mediawiki, do you? [00:46:05] but I don't need it on this machine [00:46:12] I want it on a second instance [00:46:39] I'm working on intergating MW and Moodle [00:47:00] so I don't need it here [00:47:11] So, I think you need to pick one strategy per instance and stick with it. That mediawiki role will install mediawiki and everything it needs, but might conflict with other things you already have installed. [00:47:29] ok [00:47:48] Anyway… to answer your earlier question… to sync your instance with puppet changes or a new set of classes, you want to 'sudo puppetd -tv'. [00:47:53] That causes puppet to refresh. [00:48:00] It automatically refreshes every 30 minutes otherwise. [00:48:07] Rebooting won't affect puppet's timeline one way or another. [00:48:20] ok [00:48:52] I had webserver::php5-mysql before, since I now use webserver::php5 which option should I use for mysql ? [00:49:07] I just ran a test and for some reason you need both. [00:49:16] The names are stupid, that's why I'm working on the 'lamp' role :) [00:49:18] role::labs-mysql-server? [00:49:32] both ! [00:50:50] For lamp you'll want three classes: "role::labs-mysql-server", "webserver::php5-mysql", "webserver::php5" [00:54:18] I get err: Failed to apply catalog: Cannot alias Package[mysql_client] to ["mysql-client"] at /etc/puppet/modules/mysql/manifests/init.pp:22; resource ["Package", "mysql-client"] already defined at /etc/puppet/manifests/mysql.pp:513 [00:56:02] also all my history is gone :-( [00:57:17] I'm not sure what the error is about. It could be conflicting with things in the site.pp that you edited. [00:58:19] I'm testing this [00:59:17] one thing that puzzles me is where in the site.pp should the definition from the lab console be if at all [00:59:38] Yeah, I don't know how they interact. I'm surprised that site.pp does anything at all. [01:00:40] site.pp is full of nodes that should not be in labs AFAIK [01:02:05] Right -- I use site.pp for production systems and ldap for labs instances. I'm not sure what happens when people mess with site.pp in labs. [01:02:38] In the tutorials I was in - we were told to use site,pp [01:02:42] In the tutorials I was in - we were told to use site.pp [01:03:08] and that it should be cleaned once stuff gets moved into modules [01:04:22] and they showed how to parametrize the site.pp using parameters we add from the console [01:05:47] ok I took out all the lamp stuff from th site.pp [01:08:27] it fails with err: /Stage[main]/Generic::Apparmor::Service/Service[apparmor]: Could not evaluate: Could not find init script for 'apparmor' [01:09:07] hmm [01:09:32] I think it is an issue with the moodle getting a bad dbase pass [01:11:38] so I toss it out also - and check if lamp works [01:15:11] nope something there is broken [01:15:30] same error as before [01:16:04] The apparmor thing might be a red herring, I'm seeing it too. [01:16:14] I need to go in a moment, though. [01:16:34] I'd advise against starting a new instance until the problem with puppetmaster::self is sorted out. You can check with ottomota for the status on that. [01:16:43] ok [01:16:44] ottomata? Something like that :/ [01:17:04] yea [01:17:35] one last question - any idea about the mysql pass [01:17:46] I think I asked you this before [01:17:58] should I try an empty one ? [01:20:10] anyhow I still have no lamp [01:20:22] looks like apache is broken [01:20:35] based on wget [01:20:54] I'm pretty sure it's empty. [01:21:08] Are you doing wget locally or from another instance? [03:18:37] [bz] (REOPENED - created by: Legoktm, priority: Unprioritized - normal) [Bug 47278] Unable to log in with api.php on wikitech.wikimedia.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=47278 [05:36:04] hello, there is an angoing problem with Jenkins bot see https://integration.wikimedia.org/ci/job/mediawiki-core-merge/11824/console [08:16:41] Hi! Are there any problems to labs? I can't access to bastion [08:27:05] mauro742: bastion.wmflabs.org works for me [08:30:44] I've added a new public key but ssh says "Permission denied (publickey)" [08:31:53] @requests-on [08:31:56] Requests were enabled [08:32:01] @requests [08:32:02] Warning: There are 3 users waiting for shell requests, displaying last 3: Carretasu(waiting 0.541635983333333 minutes) Fbstj(waiting 0.499802066666667 minutes) Soul Train(waiting 0.344840116666667 minutes) [08:32:20] um [08:35:18] petan: Could you also add me to the bastion project, please? [08:35:23] oh [08:35:24] sure [08:35:59] @labs-user Jelte [08:35:59] Jelte is member of 2 projects: Bots, Tools, [08:36:10] hm, I can't do that for some reason :/ [08:36:15] andrewbogott_afk ^ [08:36:18] paravoid? [08:36:30] I don't really have much access over bastion [08:37:35] petan: ok, thanks for trying [08:43:23] @requests [08:43:24] Warning: There are 3 users waiting for shell, displaying last 3: Carretasu (waiting 2 minutes) Fbstj (waiting 2 minutes) Soul Train (waiting 2 minutes) [08:45:45] Warning: There are 3 users waiting for shell, displaying last 3: Carretasu (waiting 0 minutes) Fbstj (waiting 0 minutes) Soul Train (waiting 0 minutes) [08:46:23] jeremyb_ any idea for customization until I load this to wm-bot [08:48:10] @requests [08:48:10] Warning: There are 3 users waiting for shell, displaying last 3: Carretasu (waiting 3 minutes) Fbstj (waiting 3 minutes) Soul Train (waiting 3 minutes) [08:48:15] mm [08:56:06] @requests [08:56:08] There are no shell requests waiting [08:56:11] @requests [08:56:12] There are no shell requests waiting [08:58:11] @requests [08:58:12] Warning: There are 3 users waiting for shell, displaying last 3: Asitaka (waiting 0 minutes) Jonathanischoice (waiting 0 minutes) Parent5446 (waiting 0 minutes) [08:58:30] @requests [08:58:31] Warning: There are 4 users waiting for shell, displaying last 4: Asitaka (waiting 0 minutes) Jonathanischoice (waiting 0 minutes) Parent5446 (waiting 0 minutes) Wrightonlookn' (waiting 0 minutes) [08:58:58] Warning: There are 4 users waiting for shell, displaying last 4: Asitaka (waiting 0 minutes) Jonathanischoice (waiting 0 minutes) Parent5446 (waiting 0 minutes) Wrightonlookn' (waiting 0 minutes) [09:00:34] @requests [09:00:34] Warning: There are 2 users waiting for shell, displaying last 2: Asitaka (waiting 2 minutes) Jonathanischoice (waiting 0 minutes) [09:11:24] why is my gid ( 550/ svn)? [09:11:34] liangent who knows... [09:11:36] everyone has it [09:12:18] Warning: There are 1 users waiting for shell, displaying last 1: Asitaka (waiting 14 minutes) [09:12:18] petan: so it's sure that everyone is using it? [09:12:28] ? [09:12:36] I mean it's default gid [09:12:38] for everyone [09:13:35] petan: so I can make my home drwx-----x to prevent ordinary users from accessing it? [09:13:47] !log bots petrb: deploying requests to /mnt/share/wmib/ on -1 [09:13:50] Logged the message, Master [09:13:51] that's what I do on toolserver [09:13:59] liangent of course you can do that [09:14:01] where all users are in the group 'users' [09:14:47] petan: but this solution fails when, for example, some people who registered at a different time have a different gid, say 551 [09:14:54] @lm requests.bin [09:15:04] then they falls to 'others' so they have x now [09:15:29] liangent I don't know, do other users have different gid now? [09:15:39] I am not a labs admin I am just telling you what I know [09:15:48] !ping [09:15:49] pong [09:16:02] petan: I'm not sure and the group name 'svn' looks strange [09:19:04] I know, but I can't really change it... [09:19:08] @requests-on [09:19:08] Requests were enabled [09:19:13] @requests [09:19:13] Warning: There are 1 users waiting for shell, displaying last 1: Asitaka (waiting 0 minutes) [09:19:33] Warning: There are 1 users waiting for shell, displaying last 1: Asitaka (waiting 0 minutes) [09:19:40] I think you need to wait for Ryan [09:21:36] @requests [09:21:36] Warning: There are 1 users waiting for shell, displaying last 1: Asitaka (waiting 2 minutes) [09:21:42] meh [09:21:58] @requests [09:21:58] There are no shell requests waiting [09:23:02] petan: what's this? [09:26:59] liangent check wikitech [09:27:01] mail [09:31:20] petan: got it [10:00:15] hi [10:01:19] hi anyone knows if puppet master self is now ok ? [10:10:23] or what err: /Stage[main]/Generic::Apparmor::Service/Service[apparmor]: Could not evaluate: Could not find init script for 'apparmor' is about ? [10:18:06] !log integration rebooting -jobbuilder : can't ssh to it [10:18:08] Logged the message, Master [10:19:07] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 0 minutes) [10:32:40] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 13 minutes) [10:46:07] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 27 minutes) [10:59:35] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 40 minutes) [11:13:02] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 53 minutes) [11:26:30] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 67 minutes) [11:40:03] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 80 minutes) [11:41:09] well, there [11:41:17] thank you, wm-bot [11:53:36] Warning: There are 1 users waiting for shell, displaying last 1: Marcoil (waiting 94 minutes) [12:01:00] oh, I needed to mark it as done [12:01:01] ok [12:20:02] Warning: There are 1 users waiting for shell, displaying last 1: Edswing (waiting 0 minutes) [12:33:34] Warning: There are 1 users waiting for shell, displaying last 1: Edswing (waiting 13 minutes) [12:47:13] Warning: There are 1 users waiting for shell, displaying last 1: Edswing (waiting 27 minutes) [13:34:15] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 47249] puppetmaster::self unusuable - https://bugzilla.wikimedia.org/show_bug.cgi?id=47249 [14:09:35] @requests [14:09:36] There are no shell requests waiting [14:35:15] ^demon: Is it possible to sync Gerrit and GitHub? [14:35:26] <^demon> Sync how? [14:35:59] ^demon: So a change merged in Gerrit would be pushed to GitHub, too? [14:36:13] <^demon> We already do that for all repositories. [14:36:26] ^demon: Oh, I does not know that [14:36:52] <^demon> https://github.com/wikimedia/ [14:37:02] <^demon> operations-puppet last updated 3 minutes ago. [14:37:36] Do this work in both directions? [14:38:26] <^demon> Not yet. That part's a bit harder. [14:38:36] <^demon> But we would definitely like to git stuff like pull requests into gerrit. [14:38:59] ^demon: Thank you for the infos :-) [14:39:02] <^demon> yw. [15:08:56] hi [15:09:20] having some problems setting the mysql password [15:09:44] sudo mysqladmin -u root -h localhost password 'newPass' [15:10:09] fails with access denied for user 'root'@'localhost' (using password: NO)' [15:10:14] any ideas [15:29:37] hello, I want to join the Bots project on Labs with my FischBot [15:29:46] what I have to do? [15:30:09] petan or Coren would be the best contact points [15:31:40] hello [15:31:40] petan, Coren: are you here? [15:31:48] hello petan [15:31:52] hi [15:33:30] petan: like you see, I want to join the Bots project. I operate the FischBot on Wikidata which adds claims to person items [15:34:22] ok [15:34:31] what is your wikitech name [15:34:37] @labs-user Pyfisch [15:34:37] That user is not a member of any project [15:35:07] yes, my name is Pyfisch [15:36:34] ok I added you to bots and tools projects, but I couldn't add you to bastion no idea why [15:36:35] !log wikidata-dev wikidata-testrepo checkout latest Diff and DataValues versions [15:36:37] Logged the message, Master [15:36:40] you can try to login now [15:36:47] if it doesn't work we need to poke someone [15:36:53] !toolsdocs [15:36:53] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help [15:36:56] !botsdocs [15:36:56] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [15:36:57] !access [15:36:57] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [15:37:05] Pyfisch some reading :P [15:37:41] !log wikidata-dev wikidata-testclient checkout latest Diff and DataValues [15:37:43] Logged the message, Master [15:37:57] logging in to what? [15:38:11] to https://gerrit.wikimedia.org/ ? [15:38:42] !log wikidata-dev wikidata-testrepo commented out "$wgWBSettings['entityNamespaces'][CONTENT_MODEL_WIKIBASE_QUERY]" in /var/lib/git/operations/puppet/modules/wikidata_singlenode/templates/wikidata-repo-requires.php [15:38:43] Logged the message, Master [15:40:58] Jelte, still there? What is your username on wikitech? [15:41:19] petan: where I should try to login? [15:41:42] I am logged in at wikitools and gerrit.wikimedia.org [15:42:43] bastion [15:43:11] Oren_Bochman: puppetmaster::self is fixed, although you should now probably use role::puppet::self instead. [15:43:29] thanks [15:43:42] Oren_Bochman: Also, the apparmor thing can be fixed with 'sudo apt-get install apparmor'. [15:43:57] It's a regression which only happens on new instances at the moment. [15:43:58] I [15:44:13] I'm trying a puppet free install [15:44:21] in a new instance [15:44:36] ottomata, I would like to update the docs for role::puppet::self. Is there any change in usage from puppetmaster::self? [15:47:33] no, but last I talked to someone (Ryan) we said we'd leave it as is for a few days, and let me use it, before we publicize it [15:47:34] just to make sure it works [15:47:39] acutally, q for you: [15:47:53] do you think I should make role::puppet::self use a hostname for the puppetmaster by default, rather than localhost? [15:48:01] localhost is the orig puppetmaster::self way of doing things [15:48:13] but using the node's hostname will work the same way [15:48:18] and allow other nodes to use it as a puppetmaster [15:48:44] Warning: There are 1 users waiting for shell, displaying last 1: Akkking (waiting 0 minutes) [15:48:49] Probably fine to make it use the hostname then; it won't change the behavior for single-instance cases. [15:49:19] Although… hm, maybe 'self' isn't accurate for that use case... [15:49:25] perhaps role::puppetmaster::local [15:49:37] yeah, self was kind of historical, so I kept it [15:49:59] Well, your name says puppet and not puppetmaster, so it is already extremely inaccruate :) [15:50:04] well, sorta [15:50:09] i like how that is transparent, [15:50:32] you use the same role for all the nodes, but the manifest figures out which one should be the master if $::puppetmaster == $::fqdn [15:50:47] Oh, I see. Hm. [15:50:58] role::puppet::local is fine [15:51:03] with me [15:51:20] Eh, I dunno, you may have convinced me. [15:51:26] puppet::local::master and puppet::local::client make a lot of sense too [15:51:37] no, i mean, i don't want to name the role 'puppemaster', i like having it just as puppet [15:51:42] but the name 'self' i don't really like that much [15:51:46] The clearest would be to have two roles, role::puppetmaster::labs and role::puppetclient::labs [15:51:52] i could leave puppetmaster::self as is, and use localhost [15:52:07] I like local as well [15:52:11] I like moving it into roles. Someday I'd like to banish all non-role classes from the gui. [15:52:16] yeah [15:52:17] and then have role::puppet::local [15:52:33] Except then it's not local, right? [15:52:35] which would include ither puppet::local::master or puppet::local::client just like it does now [15:52:36] * andrewbogott bikesheds! [15:52:38] haha [15:52:50] uh, depends on what you mean by local I guess [15:52:50] though local-repository might spell things out a bit more [15:52:50] local to labs intsances? [15:52:56] The domain selection dropdown that used to be shown on the user login page on labsconsole (only value was 'labs'), was that made by the LdapAuthentication extension? [15:53:11] role::puppet::labsinstance [15:53:15] petan: error message: http://justpaste.it/2feu [15:53:18] role::puppet::labs::instance [15:53:19] idunno [15:53:44] Can still see it with $( '#mw-user-domain-section' ).css( 'display', 'table-row' ) [15:53:59] We already have some roles that end in ::labs to indicate they are for labs instances. role::mediawiki-install-latest::labs, role::lamp::labs [15:54:15] yeah, but that gets confused with things like puppetmaster::labs [15:54:21] which is not run on a labs instnace [15:54:33] Ugh, yeah. [15:54:36] but, i like role::puppet::labs [15:54:39] aside from that [15:54:43] I guess we need a big official naming scheme [15:55:04] yeah the whole file needs a cleanup, but I wanted to only mess with things that I had time to work on, ya know? [15:55:09] I'm pretty sure that Ryan requested that I use ::labs for the mediawiki role. [15:55:20] So, I still vote for puppet::local::master or puppet::local::client [15:55:21] and that is meant to be run on instances? [15:55:27] Yeah. [15:55:37] Also make sure there are header comments so that they classes get documented... [15:55:42] (I haven't looked, maybe you did that already) [15:55:49] i think they are pretty well documented...? [15:56:15] Krenair: I don't know without digging through code. [15:56:18] # [15:56:18] # Below are classes used to configure self hosted puppet [15:56:18] # on labs instances. role::puppet::self (in puppet.pp) [15:56:18] # is the recommended class to use. Please use it to [15:56:19] # include these classes. [15:56:19] # [15:56:19] ... [15:56:28] Pyfisch sorry busy atm, checking... [15:56:57] yep, looks like you documented already. [15:57:04] Pyfisch that may be related to you not being in bastion project [15:57:15] Pyfisch which needs to be solved by andrewbogott or someone like that [15:57:38] Seems to be part of AuthPlugin which is overridden by LdapAuthentication [15:58:02] ok andrewbogott is here I will ask him [15:58:09] petan: For what it's worth… the process of giving someone shell permissions automatically adds them to bastion. So having someone not in bastion is a pretty unusual case these days. [15:58:21] andrewbogott: I guess let's ask Ryan_Lane and see what he thinks [15:58:40] Pyfisch, can you tell me what the trouble is? [15:59:35] andrewbogott: I can't login to bastion [16:00:02] Have you been able to, in the past? [16:00:40] no I'm new [16:00:56] Krenair: The log bots log into wikitech with the api… let's see if they are broken. [16:01:06] !log testlabs I am checking to see if I can still log. [16:01:08] Logged the message, dummy [16:01:24] Krenair: works [16:01:55] Pyfisch: OK. Are you generally familiar with using ssh? What OS are you on? [16:01:58] And, have you uploaded keys? [16:02:01] (Many questions!) [16:02:12] is bastion down ? [16:02:17] Warning: There are 1 users waiting for shell, displaying last 1: Akkking (waiting 13 minutes) [16:03:35] andrewbogott: os: windows, keys are uploaded to gerrit.wikiedia.org and wikitech wiki. I'm not very familiar with ssh [16:03:57] Oren_Bochman: Why do you ask? petan, can you attach to bastion just now? [16:04:14] andrewbogott I can't attach to bastion anyone it tell me "error" [16:04:18] I got kicked out [16:04:18] but I'm back now [16:04:22] but it is slowww [16:04:25] actually failed to add to bastion [16:04:38] ok… Pyfisch, stay tuned, we may be having a general issue. [16:04:47] Oren_Bochman, can you do a 'df' and see if any of the volumes are full? [16:08:23] petan, bastion looks more-or-less OK to me. What exactly is happening? [16:09:29] andrewbogott I have no idea, from my side I just know I can't add users to project [16:09:39] he seems to have problems connecting [16:09:49] hi Coren [16:10:09] petan, language problems… my question is: can you log in to bastion? [16:10:21] petan, I think the reason you can't add users is because they are already added. [16:10:27] aha [16:10:37] ok andrewbogott I am by default connecting to bastion3 and that one works to me [16:11:00] bastion1 works too [16:11:03] ok, cool. So, back to Pyfisch, I think probably this is just a config issue. [16:11:10] Pyfisch, you are using putty, right? [16:11:20] !access | Pyfisch, did you actually read this [16:11:20] Pyfisch, did you actually read this: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [16:11:29] Have you gone through the incredibly complicated guide to using putty and pageant? [16:11:45] I don't use Windows so am not so good with Windows tech support. [16:11:52] neither I am :o [16:12:59] andrewbogott: I am using git bash [16:13:27] Pyfisch, did you upload keys to gerrit or to wikitech or both? [16:13:39] both [16:13:44] Ok, good. [16:13:59] So… we are going to have to find someone who knows how to use ssh keys with git bash :( [16:14:23] Well, I guess the other gut-check question is: you uploaded your public keys and not your private ones, right? [16:14:43] I hope so :P [16:15:22] Pyfisch: You'd be surprised at how often that turns out to be the problem :/ [16:15:55] Warning: There are 1 users waiting for shell, displaying last 1: Akkking (waiting 27 minutes) [16:16:05] do I need to add a network rule to be able to wget files from the web ? [16:16:12] andrewbogott: thats what I uploaded: [16:16:14] ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtmxK3WtBUXrSu93qihqx52/aeRpQevf63GeYNFuXZB47BIj4DY2FMZ7wjMlkANiosdDd3QTgjuaL0c+otV3wiB5AfcYjOnihvH7D3Ai/GvlYAri53GjPC5mgG/OgXAjjruBpIfgcNyMmrF5/HneIUOIIzHr9KmgLIMvi+gJ52QzZadY5Q0dEHUdkNsGGAUj7GJPA+Zt+hSx2nul3V5WuR4xgMnXe9CDOTwU1PQTOcwDHRObnSm8knNDg0VVfL6T1u20BN9kl8iwLT7rlK5ef0pEnYUks5Yxyj2CRAbBwlzkEQ+0D5ATj2glGfjRCp/10gVCQqIzQeMtiGxSgqWEeFQ== pyfisch@googlemail. [16:16:15] petan: That's pretty cool, btw :) [16:16:15] com [16:16:34] andrewbogott ty :) [16:16:36] Oren_Bochman: Are you trying to wget from your instance or /on/ your instance? [16:16:51] Pyfisch it's generaly a bad idea to post your keys anywhere, despite this is public one... [16:16:55] on my instance from the web [16:17:15] Oren_Bochman outgoing requests from an instance should work fine. [16:19:53] petan: thats an unused key ;) [16:20:52] Pyfisch, there's a bit about git bash here: http://stackoverflow.com/questions/5727555/remember-password-git-bash-under-windows Most likely you've done all this though... [16:24:32] andrewbogott: https://wikitech.wikimedia.org/wiki/Access#Using_agent_forwarding I have done it like here, I think both ways are equal [16:25:19] Pyfisch: Agent forwarding is actually about how to get /from/ bastion to other instances. [16:25:38] Although if your key is in the right place for forwarding it should work for ssh as well. [16:25:52] Ryan_Lane, do you know anything about 'git bash' or know anyone who is using it with labs? [16:27:13] I don't know what git bash is :) [16:27:24] :-) [16:27:32] andrewbogott: I can also install putty [16:27:59] Ryan_Lane: It's a shell emulator for windows. Now and then people show up here trying to use it as an ssh client but I can't recall if anyone has ever succeeded. [16:28:44] ah [16:28:46] Pyfisch: I don't know how to use putty either, but I can at least testify that it has worked for people in the past. [16:29:28] Warning: There are 1 users waiting for shell, displaying last 1: Akkking (waiting 40 minutes) [16:29:28] we have docs on putty and winscp [16:29:42] https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP [16:29:53] https://wikitech.wikimedia.org/wiki/Help:Putty [16:31:09] Ryan_Lane I implemented new stuff [16:31:12] @requests [16:31:12] There are no shell requests waiting [16:31:23] ^ :D [16:35:30] Ryan_Lane, is anything interesting or surprising happening at the summit? [16:35:38] Other than keystone becoming ever more abstract? [16:35:44] NSA talking about their openstack use: https://www.openstack.org/home/Video/ [16:36:52] otherwise mostly expected things [16:37:17] Oh, we've met that guy! IIRC he declined to name his employer at the time. [16:37:23] yep [16:38:02] Good look for a spook. [16:38:19] :D [16:38:20] indeed [16:50:05] ok im using putty now [16:50:19] whats meant with "Type “ssh ”. " [16:50:39] Is my username? [16:51:21] where are you reading that? [16:51:41] would be the name of the… instance… you are trying to connect to. [16:52:47] "instance" is labs-speak for virtual machine [16:53:16] here: https://wikitech.wikimedia.org/wiki/Help:Putty [16:53:32] but I dont now whats my instance ;-) [16:54:13] or is there something like a "default" [16:54:51] Im away for half an hour [16:56:23] Pyfisch: You'll want to log into bastion first, then log into your instances from there. Have you managed to ssh into bastion yeT? [16:57:39] Warning: There are 1 users waiting for shell, displaying last 1: Robotto7831a (waiting 0 minutes) [17:11:19] Warning: There are 1 users waiting for shell, displaying last 1: Robotto7831a (waiting 13 minutes) [17:13:51] re [17:28:08] can somebody help me logging in with putty? [17:30:12] hey apmon [17:30:24] I saw your work on maps varnish [17:31:11] I don't think that was me [17:31:20] lol:) [17:32:07] perhaps we should start using the labs-log facility to coordinate who does what... ;-) [17:32:38] !log maps Who set up maps-varnish? [17:32:39] Logged the message, Master [17:33:31] Do you know where the log actually is? e.g. is there a URL for it? [17:34:08] https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL [17:34:49] aude, awjr, paravoid - was that someone of you?:) [17:34:59] eh? [17:35:21] awjr, did you set up maps-varnish? [17:35:25] that *might* have been me MaxSem [17:35:48] iirc i was starting to try and get varnish configured in the local puppet but didn't quite finish [17:38:00] awjr, why it used the frontend/backend distinction? its purpose is rather closer to bits which doesn't have it [17:38:24] MaxSem: i was modeling it off mobile varnish (i think at faidon's suggestion) [17:39:07] mhm [17:39:58] but we've also since kinda changed how frontend/backend works for mobile varnish... [17:39:59] on the other hand, some tiles should be much more popular than other [17:40:10] i think that was the idea [17:40:11] did we? [17:40:31] i think only in that the TTLs were changed [17:40:55] let me see if i can dig up my notes [17:41:32] I don't think we need to mess with TTL on maps [17:43:03] TTL for the object in cache, or for the HTTP header passed on to the client? [17:45:30] object in cache [17:45:37] varnish [17:46:00] it's based on TTL from reponse headers, but can be overridden [17:47:17] MaxSem: im not finding my notes :-/ [17:48:54] err: Could not request certificate: getaddrinfo: Name or service not known [17:48:54] andrewbogott_afk ^^^^ i have reopened the puppet cert bug https://bugzilla.wikimedia.org/show_bug.cgi?id=47249 :-D [17:49:27] and it looks like we didn't capture this on the etherpad either, MaxSem :( paravoid might remember better than i, but i'm pretty sure we wanted a smaller frontend cache for handling the super popular tiles [17:49:44] yeah, I agree with you now [17:50:04] mod_tile has a bunch of heuristics to determin the TTL (http header) it sends out. (Trying to guess how likely it is that a tile will change in the future) [17:50:14] mhm [17:50:42] but if we can get the proper expiry information to the caches with HTCP, then those heuristics aren't as important [17:51:07] is it possible to cramp tiles into different varnish storages based on zoom level? [17:51:58] If you have a load balancer in front that redirects the requests to the apropriate varnish, it should be possible [17:52:38] or do you mean different storages on the same varnish box (which have different properties)? [17:53:26] the latter [17:54:09] looks like that would require different instances [17:54:10] PITA [17:56:12] Btw, is there something I can currently do, that moves things forward? (Setting up varnish is probaly not one of them... ;-)) [17:56:33] Warning: There are 1 users waiting for shell, displaying last 1: Exbe (waiting 0 minutes) [17:59:08] apmon, does the OSM stack currently support sharing multiple PG servers betwen multiple renderd's? [17:59:39] * aude waves [17:59:49] hey aude [18:00:09] no, I don't think so. [18:00:24] mmm, can LVS do that? [18:00:30] It is one PG server per renderd. Unless you have a PG load balancer in front of it [18:01:33] There are a number of Postgres specifgic tools that can handle that [18:01:46] btw, i'll be adding multichill to the maps project and making an instance to have map warper on labs [18:01:58] not specific to osm but maps related [18:02:16] we would like to georeference old maps in commons [18:07:23] MaxSem: sorry stopped paying attn to the channel - im pretty sure in varnish you can send different resources to different places based on pretty much… anything [18:07:41] do the file names indicate zoom level? [18:07:48] yes [18:08:07] yeah then i imagine that would be relatively straightforward [18:08:34] and zoom level is also present in the request URI right? [18:09:13] awjr: The URLs look like http://a.www.toolserver.org/tiles/osm/0/0/0.png [18:09:30] oh right, i remember someone expalining this to me in copehagen [18:09:42] yeah, that should be no problem then. [18:09:49] tiles is just a static part of the URL. osm is the style sheet, then it is z (zoom level) x and y coordinates [18:10:01] yah [18:10:06] apmon, who are you IRL? [18:10:06] Warning: There are 1 users waiting for shell, displaying last 1: Exbe (waiting 13 minutes) [18:10:14] Kai [18:10:17] ahha!!! [18:10:24] good to e-see you :) [18:10:25] awjr: Are you Arthur? [18:10:28] yep [18:10:32] hehe [18:10:41] in fact, i think you had previously explained the URL structure to me :) [18:10:46] well possible :-) [18:23:40] Warning: There are 1 users waiting for shell, displaying last 1: Exbe (waiting 27 minutes) [18:25:01] !legoktm [18:25:02] shh [18:25:17] Coren: Have you already created a puppet class for mariadb? [18:26:14] MaxSem: You could use either PGpool or PGBouncer if you want a single renderd to balance over multiple SQL servers [18:26:40] paravoid said that he doesn't like either of them [18:26:40] but simply assigning a sql server to each renderd seems sufficient initially [18:28:45] Imho, I would suggest: setting up two postgresql instances using postgresql's built-in streaming replication [18:29:00] then have each renderd configured to connect to one db [18:29:20] if the db server fails for a short time, then performance will simply be degraded for that time [18:29:42] if it fails for longer, you can reconfigure renderd to connect to a working db server [18:33:04] !seen Coren [18:33:12] @seen Coren [18:33:12] Jan_Luca: Coren is in here, right now [18:34:32] petan: Have you already created a puppet class for mariadb? [18:36:44] !log wikiversity-sandbox [18:36:45] Message missing. Nothing logged. [18:36:56] !log wikiversity-sandbox Updated wikiversity-sandbox-frontend [18:36:59] Logged the message, Master [18:37:41] !log wikiversity-sandbox Create new instance wikiversity-sandbox-db1 for a seperated MySQL/MariaDB-Server [18:37:42] Logged the message, Master [18:46:33] !log Updated and rebooted the new instance wikiversity-sandbox-db1 [18:46:34] Updated is not a valid project. [18:46:43] !log wikiversity-sandbox Updated and rebooted the new instance wikiversity-sandbox-db1 [18:46:45] Logged the message, Master [18:46:57] @requests [18:46:58] There are no shell requests waiting [18:47:34] petan|fu: Have you created some puppet for mariadb for bots? [18:47:44] Jan_Luca nope [18:47:53] and TBH I don't even like that idea [18:48:06] I think mysql servers deserve to be hand configured and optimized [18:48:25] rather than making some universal configuration which works but suck [18:49:54] ottomata, an attempt to explain & name future roles: https://wikitech.wikimedia.org/wiki/Puppet_usage#Roles [18:51:16] ha, nice, i like it…except I kinda disagree with the ::labs|::production difference, i think that should be handled via puppet environments or whatever [18:51:23] but I am in ameeting right now and cannot debate! [19:06:14] andrewbogott: There seems to be some problems with service groups: When I try to create one I get the message "Failed to create service group." but the group appears in the list on Special:NovaProject [19:06:53] and when I try to delete a group I get the message "Failed to delete service group." and nothing happens [19:10:02] Jan_Luca: Ok… I can look shortly. What project are you using? [19:10:18] andrewbogott: wikiversity-sandbox [19:11:32] hmm, I have no keys in prefs on wikitech. how come I can login into labs?:) [19:12:53] ottomata: andrewbogott: so hmm sorry but new labs instance have a broken puppet.conf :-D [19:13:09] and i have no f**** idea how to fix it :( [19:13:43] hashar: I have just created a new instance. What are your problems? I missed some standard-packages [19:14:19] the /etc/puppet/puppet.conf is missing the certificate server :) [19:14:24] https://bugzilla.wikimedia.org/show_bug.cgi?id=47249 [19:17:26] hashar, the problem you're having doesn't have anything to do with puppetmaster::self, does it? [19:20:02] andrewbogott: nop that is a fresh instance [19:20:11] andrewbogott: I should had eopened another bug, sorry for the confusion :( [19:20:17] hashar: OK, so, probably could use a different bug. [19:20:27] likely, want me to fill another bug? [19:20:37] I am fast copy paster, I don't mind :-] [19:20:37] When I see the behavior you're seeing, it's just because I have logged into an instance before the initial puppet run has completed. [19:20:42] Is that what you're seeing? [19:21:12] well the 1st puppet run might locked because it does not have the proper conf [19:21:35] I will look at the log file :D [19:21:58] instance is deployment-cache-upload-test9.pmtpa.wmflabs i-000006c7 [19:23:05] puppet-agent[1127]: Certificate Request fingerprint (md5): 25:69:AD:AF:BE:86:D8:C7:DE:5E:25:53:53:2A:CC:AB [19:23:06] puppet-agent[1127]: Did not receive certificate (repeated 5 times) [19:23:09] and it cache it [19:23:38] Yeah, looks like that instance is seriously broken. Is that happening to you every time? [19:23:59] log for the first session : http://pastebin.com/DA4wWuk1 [19:37:43] hashar, can you run a test for me? Build a new instance, then log into it but /don't/ run puppet. Instead, tail the syslog and watch until it reports that the initial puppet run is finished. [19:37:47] And then try puppetd -tv [19:38:10] can't you do it directly ? :D [19:38:23] I can, but it works when I do it. [19:38:27] :) [19:38:42] created deployment-andrew.pmtpa.wmflabs [19:41:55] puppet running [19:42:41] that instance has a correct puppet.conf [19:44:27] Finished catalog run in 175.58 seconds [19:44:59] that works now [19:45:09] I have no idea what might have gone wrong with the other creations [19:54:47] andrewbogott: seems to work now [19:55:15] Yeah, there's a real problem here, I just don't know how to reproduce it on purpose. [19:55:32] I wonder if somehow running puppet inside a puppet run causes it? But even that doesn't break every time. [19:56:09] puppet should lock on a run [19:56:21] yep, it should. [20:00:09] Coren, if you're around can you meet me in wikimedia-e3? [20:01:28] andrewbogott: at least I got an instance created. [20:01:32] will check puppetmaster::self now [20:05:47] andrewbogott: that worked :-] [20:06:13] Third time's the charm :( [20:09:39] is cpu temperature 90 C normal :D [20:10:11] poor laptop [20:11:17] I hope it isn't actually on your lap [20:15:00] Jan_Luca: I am now looking at your service-group problem. I see the same bad behavior... [20:17:43] andrewbogott yes it is [20:17:48] brb [20:24:18] ottomata: what's going on with puppetmaster::self? [20:24:28] is this related to new instance images? [20:24:36] or is this also a problem on old instances? [20:24:44] * hashar jenkins jobs are on hold for an emergency fix sorry. [20:25:12] what I know is: [20:25:26] sometimes, on new instance creation (without applying puppetmaster self), the puppet.conf file is whacky [20:25:32] like, missing a server line [20:25:41] hm [20:25:48] the bootstrapping changed for that... [20:25:57] but actually, that might be different than what hashar is experiencing [20:25:59] that's supposed to get added by cloud init [20:26:18] i'm not sure [20:26:53] I'm going to make that bootstrapping better. I may put a patch into openstack nova for this [20:27:08] right now the instance has to do an ldap lookup to figure out its project [20:27:18] which is likely causing this issue [20:30:08] andrewbogott: Is this a problem of my project or a general one? [20:30:18] I can't tell yet. [20:30:29] I mean, service groups are definitely working in tools. [20:30:52] Maybe you could test another project? [20:36:57] andrewbogott: My problem is not very important at moment, I only wanted to test the feature. Should I fill a bug for it so you could look at it when you have time? [20:40:14] Sure, you can file a bug. [20:40:24] it's something to do with group id conflicts [20:40:45] [bz] (NEW - created by: Jan Luca, priority: Unprioritized - major) [Bug 47336] Errors when creating service groups - https://bugzilla.wikimedia.org/show_bug.cgi?id=47336 [20:41:24] andrewbogott: I found another bug: The groups are shown in the project filter [20:41:40] Hm… that one is fixed but maybe not merged. [20:42:34] Jan_Luca: there, how's that? [20:43:24] andrewbogott: ? [20:43:31] Is the fliter better? [20:44:13] no [20:45:10] andrewbogott: I tested the groups with the centralauth-project, too. There are the same bugs [20:49:42] hah. the toggles don't work for the projects or regions [20:49:49] that's annoying [20:50:11] I should just remove that. it's confusing and I don't think anyone uses it [22:11:20] apmon & awjr, I think I've set up maps-varnish [22:11:30] oo! [22:11:35] now it guru-meditates on tileservers:) [22:11:48] lol [22:11:54] yay? [22:14:11] MaxSem: nice [22:14:14] Warning: There are 1 users waiting for shell, displaying last 1: Jean-Frédéric (waiting 0 minutes) [22:14:31] So what is the next step? [22:14:42] se up tileservers [22:17:16] Are the varnish configs in some gerrit puppet ticket? [22:18:40] the main OSM commit [22:20:11] * apmon goes and looks [22:21:05] I haven't committed a small fix to it [22:21:35] [bz] (NEW - created by: Chris McMahon, priority: Unprioritized - normal) [Bug 47339] beta cluster: wrong MessagesEn.php file? - https://bugzilla.wikimedia.org/show_bug.cgi?id=47339 [22:27:42] Warning: There are 1 users waiting for shell, displaying last 1: Jean-Frédéric (waiting 13 minutes) [22:28:56] MaxSem: It is mainly the scaling that needs still considering in setting up the tileserver? [22:29:17] as your puppet scripts set up a single tileserver fairly fine, from what I have seen [22:30:30] yes, we need to set up 2 tileservers using ceph and a shared PG [22:32:35] Would you set up ceph through puppet, or rather not, as those won't be used again in production? [22:35:18] wouldn't you need 3 servers for ceph? [22:36:30] you need to host 3 components, all 3 can be on the same machine in a minimum testing configuration [22:36:44] ah. for testing [22:36:45] right [22:37:34] and ap/mon meant that we need to set up 2 servers using ceph, we will definitely create a separate instance for it [22:38:54] * Ryan_Lane nods [22:39:09] I thought you may have been talking about production, so I thought I'd ask ;) [22:39:32] nah, there's a different channel for prod [22:39:36] :) [22:40:04] and OSM will use our existing ceph cluster [22:40:08] Do you need to set up the mds part of ceph? (or was that not one of the 3 components?) [22:41:41] mmm [22:41:44] dunno:) [22:42:05] you definitely know it better than me [22:42:21] but I started learning just now!:P [22:42:37] *building an instance* [22:43:08] Well, I just installed a minimal version of ceph on my laptop to have something libceph could talk to [22:45:35] but I guess that is sufficient for the purposes here as well, as the production ceph is already running, so no need to worry about that [23:08:57] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=674702 edit summary: [-1158] [23:10:02] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=674703 edit summary: [-2] /* Documents */ [23:38:48] Krenair, can I clear the changes in w1 on nova-precise2? [23:38:56] (If they are yours? branchname 'api'.) [23:39:09] I don't recognise that branch name, no [23:39:36] It's 'Add API action for instance reboots' [23:39:37] so not mine [23:39:47] Authored by root... hm. [23:40:20] I'll save the changes [23:41:42] `who` says only we are online on that instance, so someone's left uncommitted changes and logged off again -.- [23:44:17] dammit, everything is broken [23:54:00] What is that bastion-restricted instance? All I know is that it's restricted to ops, but what does it do that's special?