[00:17:34] RECOVERY dpkg-check is now: OK on wikidata-dev-3 i-0000021d output: All packages OK [00:18:44] RECOVERY Current Load is now: OK on wikidata-dev-3 i-0000021d output: OK - load average: 0.11, 0.09, 0.03 [00:19:24] RECOVERY Current Users is now: OK on wikidata-dev-3 i-0000021d output: USERS OK - 0 users currently logged in [00:20:44] RECOVERY Free ram is now: OK on wikidata-dev-3 i-0000021d output: OK: 74% free memory [00:21:54] RECOVERY Total Processes is now: OK on wikidata-dev-3 i-0000021d output: PROCS OK: 90 processes [01:04:14] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [01:08:14] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [01:16:56] Ryan_Lane: could you merge my bugzilla changes for puppet? [01:17:06] link? [01:17:14] 1s [01:17:20] or add me as a reviewer [01:18:25] done [01:23:22] oh, this is in the production branch? [01:23:27] you need to use the test branch for labs [01:23:36] hexmode: I reviewed it [01:23:37] :) [01:23:49] tyvm [01:24:34] Ryan_Lane: abandoning, switching to test branch [01:30:29] Ryan_Lane: ok, obviously I'm a git n00b. Where is all this documented? [01:31:02] on mediawiki's site somewhere [01:31:06] git checkout test [01:31:40] Then to reuse your existing change, git cherry-pick 09bd35a9ac49c8d48fdee1e9620e0ae0ecd9e4b5 [02:49:24] RECOVERY Current Users is now: OK on upload-wizard i-0000021c output: USERS OK - 0 users currently logged in [02:50:34] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 93% free memory [02:50:54] RECOVERY Disk Space is now: OK on upload-wizard i-0000021c output: DISK OK [02:51:54] RECOVERY Total Processes is now: OK on upload-wizard i-0000021c output: PROCS OK: 75 processes [02:53:44] RECOVERY Current Load is now: OK on upload-wizard i-0000021c output: OK - load average: 0.03, 0.36, 0.27 [03:44:39] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:48:59] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [03:57:10] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [04:02:34] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [04:04:39] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [04:08:59] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:09:39] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:13:59] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:17:19] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:22:30] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 3% free memory [04:27:10] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:27:30] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [05:30:10] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours [09:52:57] petan|wk: hello :) [09:53:10] petan|wk: is there any project name for the pre deployment cluster ? [09:53:27] wmflabs being too generic [09:53:30] 'beta' same [09:53:38] deployement-prep is too long :-] [10:00:41] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-3 i-0000021d output: DPKG CRITICAL dpkg reports broken packages [10:05:20] mutante: ping pong :-D [10:05:41] mutante: any idea about a name for the preproduction cluster / deployment tester / beta etc ? [10:23:44] PROBLEM Current Load is now: CRITICAL on wikidata-dev-4 i-0000021f output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:24:24] PROBLEM Current Users is now: CRITICAL on wikidata-dev-4 i-0000021f output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:25:04] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-4 i-0000021f output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:25:44] PROBLEM Free ram is now: CRITICAL on wikidata-dev-4 i-0000021f output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:26:54] PROBLEM Total Processes is now: CRITICAL on wikidata-dev-4 i-0000021f output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:27:34] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-4 i-0000021f output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:34:20] hey there [10:35:19] i'd like to use the vcsrepo type in my manifest. [10:37:07] normally i'd install it with "puppet module install vcsrepo" [10:37:09] unfortunately, the puppet that is included with the labs instances is too old and doesn't have the module sub-command. [10:37:09] hints? [10:58:04] PROBLEM host: wikidata-dev-4 is DOWN address: i-0000021f check_ping: Invalid hostname/address - i-0000021f [11:01:50] hashar: yes it's deployment cluster or deployment test site [11:01:59] or just deployment-prep [11:02:07] deployment-prep is name of project in labs [11:02:24] beta.wmflabs.org and deployment.wmflabs.org are dns [11:02:34] ;) [11:03:35] that is confusing :-D [11:03:44] PROBLEM Current Load is now: CRITICAL on wikidata-dev-3 i-00000222 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:03:49] how is that confusing the purpose is to test [11:04:02] it's test of sw before it's deployed to prod :-) [11:04:07] so deployment site [11:04:24] PROBLEM Current Users is now: CRITICAL on wikidata-dev-3 i-00000222 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:04:42] like a site where we install the trunk mediawiki so that community of wikimedia projects can test if it's going to work with the project [11:05:04] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-3 i-00000222 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:05:07] I can't think if any better name [11:05:14] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [11:05:44] PROBLEM Free ram is now: CRITICAL on wikidata-dev-3 i-00000222 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:06:54] PROBLEM Total Processes is now: CRITICAL on wikidata-dev-3 i-00000222 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:07:34] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-3 i-00000222 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:07:47] I've created an instance on my project Upload-wizard, I can see it in "Manage Instances", but why can't I see it in my project page? [11:08:33] Why the "Instances for this project" section of the page "Nova Resource:Upload-wizard" is still empty? [11:09:14] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [11:10:52] Shujenchang: you better use Special:NovaInstance [11:11:07] !instance [11:11:07] https://labsconsole.wikimedia.org/wiki/Help:Instances [11:11:52] !project deployment [11:11:52] https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment [11:11:54] yay [11:12:16] !project deployment-prep | hashar [11:12:16] hashar: https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment-prep [11:50:15] !project deployment-prep | hashar [11:50:16] hashar: https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment-prep [11:50:24] documentation :P [11:52:41] sorry went out for lunch [11:59:44] I've created an instance on my project Upload-wizard, I can see it in "Manage Instances", but why can't I see it in my project page? Why the "Instances for this project" section of the page "Nova Resource:Upload-wizard" is still empty? (My computer was crashed just then, and I rebooted it, so I may not recieve replies about the question I asked just then) [12:04:51] Shujenchang: https://labsconsole.wikimedia.org/wiki/Nova Resource:Upload-wizard?action=purge [12:05:00] Shujenchang: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Upload-wizard?action=purge [12:05:53] thx for your help~ [12:14:43] and how to config the Puppet if I want to install a mediawiki and upload-wizard extension, what puppets should be checked? [12:18:15] !log deployment-prep petrb: updated git and commited all changes [12:18:16] Logged the message, Master [12:18:27] !log deployment-prep petrb: moved the log file storage to gluster [12:18:29] Logged the message, Master [12:19:03] Shujenchang: apache-php5-mysql [12:19:11] mediawiki is not puppetized so far [12:19:49] hello mutante [12:20:08] webserver::apache2 and webserver::php5-mysql? [12:20:26] Shujenchang: only one [12:20:33] mutante, any update on wikistats? I'd love to add a bunch of wikis to the tables :) [12:20:36] I think that php5 mysql works [12:21:22] should I checked something under "database"? [12:21:43] hashar: is there any way to debug why it loads so slow? [12:21:51] it ? [12:21:53] Shujenchang: I don't think so [12:21:56] hashar: mediawiki [12:22:18] need to debug it to find out the root cause [12:22:19] every page on deployment site loads slow [12:22:19] Nemo_bis: yea, there have been updates, actually i was on it almost the whole weekend, trying to make the unified update script to work for all. there are issues with letting a system user run cron jobs right now..opened a bug and talked to ryan [12:22:25] I see [12:22:41] hashar: in fact I believe that mediawiki itself is pretty slow, because I never managed to install it anywhere for it to be fast [12:22:48] Nemo_bis: in which form do you have the new wikis right now? [12:22:57] all servers I ever installed it on even on fast ones it was running slow [12:23:49] while sw like word press runs 200 times faster :) [12:23:51] I will look at it [12:24:02] meanwhile [12:24:06] you mean you will fix the mediawiki so that it's faster :P [12:24:10] nice [12:24:11] :D [12:24:12] do you guys use puppet ? [12:24:17] for? [12:24:20] sometimes we do [12:24:29] to install packages on the tests servers [12:24:37] we use it for apache [12:24:39] that's all [12:24:51] against the `test` branch? [12:24:59] everything else is installed by hand [12:25:05] on deployment site [12:25:16] yes it's test branch [12:25:20] whole labs [12:25:21] which is like … a very bad practice :-] [12:25:22] hashar: yea, test branch if you want to install packages on instances [12:25:32] hashar: it's only working practice [12:25:45] I can't wait 2 weeks for someone from ops to merge every little change [12:26:05] if I did, deployment site wouldn't even exist, we would be probably at very beginnig [12:26:24] mutante, a list of 7400 api.php [12:26:40] I can push in ops/puppet test branch [12:26:44] for example I wanted Ryan to puppetize squid from prod, I was waiting few months, then I decided to do it myself [12:26:44] well I could last time I checked [12:26:53] mutante, https://code.google.com/p/wikiteam/source/browse/trunk#trunk%2Fbatchdownload%2Flists [12:27:22] and yes, mutante, I saw something moving, that's why I asked :) [12:27:27] hashar: that doesn't change the fact that we can't and we definitely wanted the deployment site to work sooner than in few years :-) [12:27:45] mutante, any news from Wikia? I see they have some sweet central APIs with lists of active wikis and such [12:27:49] that explains why things were done by hand [12:28:00] also in beginning of labs it wasn't even possible to use puppet [12:28:03] unless you were ops [12:28:06] Nemo_bis: wow! nice . well for the "wikistats admin" thing that would be too many anyways.. it would just allow adding one by one.. so just mysql import [12:28:18] mutante, I see [12:28:38] Nemo_bis: no, i've been meaning to contact somebody at wikia for the data [12:28:39] those are all verified APIs, although some give DB errors on particular requests (!) [12:28:42] ok [12:29:02] mutante, do you know who to contact? [12:29:56] Nemo_bis: not a specific "best person to ask" yet.. but there should be good contacts between wmf and wikia [12:30:09] yep, more or less [12:31:26] interesting how people run wikis on IP addresses without DNS names [12:32:16] heh [12:32:31] there's even a mirror of Wikipedia on a test wiki now full of spam, for what it's worth [12:32:56] (I removed it from the lists, though) [12:33:20] Nemo_bis: i'll just import these to db with a manual import for now [12:33:29] mutante, sweet [12:33:34] Nemo_bis: and then run an update and we can look at the HTTP response codes [12:33:48] mutante, I didn't check if there are duplicates though [12:33:55] PROBLEM HTTP is now: CRITICAL on upload-wizard i-0000021c output: CRITICAL - Socket timeout after 10 seconds [12:34:09] Nemo_bis: don't worry, database won't allow duplicates in the URL to the api [12:34:28] mutante, yes, but you might still have some of those in the old Special:Statistics raw form [12:34:32] or not? [12:34:58] well, i can't guarantee but i have an "conversion" script [12:35:04] that tries to use new API URLs [12:35:10] and if that works it changes them [12:35:13] wow [12:35:33] Nemo_bis: what list you talk about? [12:35:44] petan|wk, https://code.google.com/p/wikiteam/source/browse/trunk#trunk%2Fbatchdownload%2Flists [12:36:06] I'm archiving all those wikis (well, not alone, but I did 1000 of them so far) [12:36:10] right [12:36:15] whatever the purpose is [12:36:50] petan|wk, the purpose is http://archive.org/details/wikiteam [12:37:19] it's trying to find all mediawiki installations on the internet [12:37:37] hm.. [12:38:10] https://code.google.com/p/wikiteam/wiki/TaskForce [12:38:45] http://www.cs.brown.edu/~pavlo/mediawiki/ already did it in 2009, we're now archiving all the survived wikis and then we'll try to copy him and run a crawler to find all the new ones and archive them too [12:38:52] petan|wk: I only checked webserver::php5-mysql, and run the command 'sudo puppetd -tv', but when the command was finished, it seemed php and mysql is not installed. [12:39:10] highlight + bold seems a very nasty thing [12:39:31] Shujenchang: that sounds weird [12:39:35] it should work [12:39:39] Nemo_bis: ah, well, for these, we don't have names yet, but i figured anyways we should ask the API itself for the Wiki name anyways [12:39:43] can you insert me to project? [12:39:43] It is said "The program 'php' is currently not installed. To run 'php' please ask your administrator to install the package 'php5-cli'" and "The program 'mysql' can be found in the following packages: [12:39:44] * mysql-client-core-5.1 [12:39:44] * mysql-cluster-client-5.1" [12:39:55] Nemo_bis: need to make one API call to get wiki name for all ... [12:40:07] Shujenchang: I will look in that, but I don't have access to that project [12:40:27] mutante, ? [12:40:29] * Nemo_bis afk now [12:40:37] petan|wk: sure,wait a moment~ [12:40:46] What's your username on labs? [12:40:50] petan|wk: I think we should add puppet class apaches::service on the Apaches [12:40:58] Nemo_bis: all we have in the lists are API URLs, the db wants an URL and a wiki name currently. we can retrieve the name from the API though [12:40:58] really? [12:41:03] that should resolve lot of issue [12:41:07] hashar: feel free to do that [12:41:09] Nemo_bis: ok, i'll show you later .. will import [12:41:13] such as including php5-apc a opcode accelerator [12:41:14] mutante, yeah, sounds silly to require both [12:41:20] hashar: don't forget to log it [12:41:20] mutante, thanksù [12:41:20] of course [12:41:21] hashar: log everything :-) [12:41:32] petan|wk: can you make me a sysadmin on the project ? [12:41:35] sure [12:41:58] petan|wk: What's your username on labs? [12:42:15] done [12:42:19] Shujenchang: Petrb [12:42:19] petan or petan|wk? [12:43:00] petan|wk: danke [12:43:26] 04/24/2012 - 12:43:25 - Creating a home directory for petrb at /export/home/upload-wizard/petrb [12:44:05] petan|wk: I've added you [12:44:26] 04/24/2012 - 12:44:25 - Updating keys for petrb [12:44:47] ok [12:45:25] petan|wk: thx~ [12:49:05] Reedy: could you comment on this: https://gerrit.wikimedia.org/r/#patch,sidebyside,5553,1,ProofreadPage_body.php [12:49:28] Reedy: or could you point me to someone who knows about extending the parser [12:49:52] Shujenchang: it seems to work [12:50:14] probably some problem in puppet [12:50:26] but apache server is surely running there [12:51:17] but php's not, I run the command "php", It is said "The program 'php' is currently not installed. To run 'php' please ask your administrator to install the package 'php5-cli'" [12:51:26] yes that's cli [12:51:38] php-cli is not a part of apache:php5:mysql [12:52:10] and mysql seems not be installed too [12:52:12] in fact I don't know if there is any puppet class for that [12:52:30] yes mysql server isn't puppetized, it installed only php modules for mysql [12:52:49] mysql server needs to be installed separately [12:52:58] Can command "apt-get" be used on instance? [12:53:02] yes [12:53:36] I see, and I'll use "apt-get" to install them? [12:53:45] ok [12:55:45] I've created an instance on my project Upload-wizard, I can see it in "Manage Instances", but why can't I see it in my project page? Why the "Instances for this project" section of the page "Nova Resource:Upload-wizard" is still empty? (My computer was crashed just then, and I rebooted it, so I may not recieve replies about the question I asked just then) [12:56:15] sudo apt-get install php5 [12:56:16] [sudo] password for shujenchang: [12:56:16] Reading package lists... Done [12:56:16] Building dependency tree [12:56:16] Reading state information... Done [12:56:16] The following NEW packages will be installed: [12:56:16] php5 [12:56:17] 0 upgraded, 1 newly installed, 0 to remove and 44 not upgraded. [12:56:17] 1 not fully installed or removed. [12:56:18] Need to get 1,108 B of archives. [12:56:18] After this operation, 20.5 kB of additional disk space will be used. [12:56:19] Get:1 http://ubuntu.wikimedia.org/ubuntu/ oneiric-updates/main php5 all 5.3.6-13ubuntu3.6 [1,108 B] [12:56:19] Fetched 1,108 B in 0s (53.0 kB/s) [12:56:20] Selecting previously deselected package php5. [12:56:45] It seemed install errored? What's wrong? [12:56:46] :o [12:56:58] it seems ok to me [12:57:33] php5 is already the newest version. [12:59:14] But why it is said "Errors were encountered while processing: ganglia-monitor"? [13:00:10] no idea [13:00:18] it must be something else broken [13:00:25] but apache works [13:02:18] and is php works on my instance? [13:02:24] mutante: automount on deployment-web2 is broken [13:02:25] petan|wk: Shujenchang : Ryan just added the generic mysql server class to be ready for use [13:02:40] ok [13:02:43] that's new to me [13:02:50] and besides that there is mariadb as alternative [13:03:10] I would recommend mysql for development since that's what most of people use [13:03:19] mutante: what is the name of class? [13:03:43] petan|wk: how to see the info of php? [13:04:06] Shujenchang: I think there is some function built in php [13:04:28] Shujenchang: generic::mysql::server [13:04:40] mutante: do you have any idea how we could add new puppet class in NovaInstance configuration screen? [13:04:51] mutante: can you check why deployment-web2 sucks so much :O [13:05:16] petan|wk: it is binary compatible though and the lead developer is former guy from mysql. it is free as opposed to Oracle licensing being unclear [13:05:36] wait a moment oracle licensing isn't clear? [13:05:41] how can we use it then [13:05:54] I thought it was GPL [13:06:08] http://en.wikipedia.org/wiki/MariaDB [13:06:33] see on labs-l [Labs-l] Problems when setting up an instance of Wikidata [13:08:05] in the beginning there was a "rather this than mysql on labs", but it isn't finally decided [13:08:34] ok [13:08:45] anyway I still would rather use same as we have on prod [13:08:59] are we switching to Maria there? :P [13:09:01] hashar: which one were you looking for? [13:09:11] sudo puppetd -tv [13:09:11] info: Loading facts in projectgid [13:09:11] info: Loading facts in default_gateway [13:09:11] info: Loading facts in projectgid [13:09:11] info: Loading facts in default_gateway [13:09:11] Could not retrieve project_gid: private method `chomp' called for nil:NilClass [13:09:12] Could not retrieve project_gid: private method `chomp' called for nil:NilClass [13:09:12] Could not retrieve project_gid: private method `chomp' called for nil:NilClass [13:09:13] Could not retrieve project_gid: private method `chomp' called for nil:NilClass [13:09:13] Could not retrieve project_gid: private method `chomp' called for nil:NilClass [13:09:14] Could not retrieve project_gid: private method `chomp' called for nil:NilClass [13:09:14] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class generic::mysql::server for i-0000021c.pmtpa.wmflabs on node i-0000021c.pmtpa.wmflabs [13:09:15] warning: Not using cache on failed catalog [13:09:15] err: Could not retrieve catalog; skipping run [13:09:20] What's wrong? [13:09:34] hm... [13:09:47] Could not find class generic::mysql::server for i-0000021c.pmtpa.wmflabs on node i-0000021c.pmtpa.wmflabs? [13:09:50] yes [13:10:06] I think that it wasn't merged to puppet [13:10:16] "could not retrieve project_gid" sounds like a bug [13:10:17] Ryan probably forgot to do that [13:10:37] yes that's another problem [13:10:41] meh [13:10:51] please open bugs in BZ :p [13:10:52] mutante: apaches::service [13:11:03] !bz [13:11:03] https://bugzilla.wikimedia.org/$1 [13:11:06] mutante: looks like that class is the perfect one to setup the various deployement hosts [13:11:07] meh [13:11:13] err the various web deployement hosts [13:11:33] maybe I have to install mysql manually via apt-get too [13:13:11] Shujenchang: the wikidata guys just had that installed. are you on the labs-l mailing list ? would probably be best to reply there as well [13:13:27] hashar: looking ...brb [13:13:41] looks like I can add group per project [13:13:54] mutante: yes, I'm on the mailing list [13:13:56] [Manage Puppet Groups] [13:15:14] Shujenchang: reply here, see last sentence by Ryan http://lists.wikimedia.org/pipermail/labs-l/2012-April/000176.html [13:15:14] mutante: yeah I can add new puppet classes :-D [13:15:54] !report is to report new bug open: https://bugzilla.wikimedia.org/enter_bug.cgi [13:15:54] Key was added! [13:16:34] hashar: ah, so also classes that have not been in there at all? and done? [13:16:59] well I have managed to add a project specific puppet class [13:17:01] trying right now [13:17:05] should be in Apache group [13:17:43] !log added apache::service on deployment-web host [13:17:43] added is not a valid project. [13:17:48] grrr [13:17:54] well, project specific sounds good enough, right [13:18:32] hashar: yeah, if in doubt i would log it to bastion project, abusing that as "meta" or something :) [13:18:51] !log deployment-prep added apache::service on deployment-web host [13:18:53] Logged the message, Master [13:18:56] uhm.. in this case, probably to deployment-prep [13:18:57] yep [13:18:58] we need a shorter project name [13:19:10] something like `tentacle` `jailbreak` or whatever [13:19:38] so how can I run puppet update now ? :-D is there a nice sudo rule ? [13:19:44] sounds good for project names, but i think hostnames should be project-001 or something and not just random names [13:20:02] I was referring to the project name [13:20:39] hashar: sudo puppetd -tv ? [13:20:50] ask for password [13:20:54] hashar: given the brand new changes to sudo i did not try yet [13:21:01] but there is always "sudo su -" :p [13:21:21] I must not be allowed to use sudo on sep-prep [13:21:22] hashar: hmm, Shujenchang was just able to do that [13:21:38] ? [13:21:54] Shujenchang: you just ran "sudo puppetd -tv" and did not have permission issues [13:22:25] permission issues? [13:23:22] hashar: [Labs-l] Per-project sudo, managed via labsconsole, you saw that yet? pretty new [13:23:47] yup I see that [13:24:19] deployment-prep has ALL ALL , and the default [13:24:26] though sudo ask me for a password [13:24:41] isn't it because I am not allowed to "puppetd -tv" ? [13:24:53] oh.. [13:25:00] no,just type your labs password [13:25:24] oh of course [13:25:32] mutante: you mean I shoud change "ALL ALL"? [13:25:50] works! [13:25:52] Shujenchang: no, you are fine, i was just giving hashar an example it worked for you. dont worry [13:25:59] hashar: k:) [13:26:06] :D [13:27:08] next question, when login on deployment-web PS1 show the instance name i-0000000217 [13:27:22] can't I make it to show the deployement-web instead ? [13:27:44] no :( not that easy [13:28:05] there has been discussion on that as well [13:28:17] like when we wanted the "nice" hostnames in nagios [13:28:34] I will just skip that for now so [13:28:46] I'm going to have my dinner, see u later~ [13:29:39] hashar: well, you can change your prompt by setting PS1= in login script, but this was about the real hostname :) [13:30:23] maybe LDAP could be used to lookup the instance-name given the instance-ID [13:30:38] Ryan wants us to file bugs for all of that ;) [13:33:11] it is not that much important anyway [13:33:44] another more serious issue is the devops group [13:34:19] it has a local gid of 1075 [13:34:35] and members seems to be manually managed in /etc/group directly [13:34:52] also, it is not the devops primary group which cause all kind of funny issues for me [13:35:12] namely, I can not write a file owned by someone:devops -rw-rw---- [13:35:30] vim think I can not write to it and open it in read only mode [13:35:37] hashar: similar, just filed this https://bugzilla.wikimedia.org/show_bug.cgi?id=36206 [13:35:41] then when writing it, it ends up as : hashar:svn -rw-rw---- [13:37:45] how are the account created on deployment-prep project anyway? [13:39:26] hashar: human users via labsconsole, putting them into the project- primary group, system users should be created via puppet but i think they aren't (and would NOT be in the project- group and you can't add them via puppet).. and i don't think puppet is used for this ..but should [13:40:25] ohh I did not know labsconsole could let us add / create users [13:40:27] checking [13:41:50] hashar: well if you add a user to a project it creates the homedir and puts the key there on next puppet run [13:43:07] bbiaw, importing wikis [14:14:47] I'm back [14:20:30] bah [14:30:47] mutante: here is my blocking bug https://bugzilla.wikimedia.org/show_bug.cgi?id=36218 :D [14:34:55] petan|wk: did you ever get help for https://bugzilla.wikimedia.org/show_bug.cgi?id=36135 localization on labs commons? [14:36:11] I've already installed php and mysql, how can I install Mediawiki? [14:37:34] hashar: good, best way to go i think since the whole gid/LDAP thing is changing and being worked on [14:37:49] no [14:37:53] chrismcmahon: I didn't [14:38:13] I don't know how many people from ops should I insert so that someone would tell me [14:39:38] mutante: one thing I don't understand is that when I add an entry in /etc/passwd it is not overriden [14:40:07] hashar: why should it be [14:40:25] because nsswitch.conf has : passwd: files ldap [14:40:35] so to me libc should read the files first and then ldap [14:40:37] local services are using passwd [14:40:45] you can create account using that too [14:40:58] hashar: it can do "first files, then ldap" but still not remove anything from files [14:41:27] well it does not lookup in /etc/passwd [14:41:28] :-( [14:41:38] what are you trying to do [14:41:40] oh, "overridden", got you now [14:41:55] my user has a primary GID of 550 (svn) [14:42:07] that you need to change in LDAP I guess [14:42:17] since most files are shared through depops (1075) I am screwed [14:42:30] hashar: best would be to create a bz ticket and assign it to Ryan he's the only person who could fix it [14:42:33] I mean, which path on my instance should be used to installed Mediawiki? [14:42:44] then I probably don't want to change the GID in LDAP since my production account would be affected and receive a different / non existent gid [14:42:45] hashar: forwarding another mail to you [14:43:05] hashar: I have 550 group too [14:43:07] petan|wk: yeah I am sure Ryan will find an answer [14:43:18] hashar: it doesn't matter what group you have, it will work [14:43:20] so I loggued bug https://bugzilla.wikimedia.org/36218 [14:43:23] you are member of deopops [14:43:28] depops [14:43:34] but still, creating a "hashar" account in /etc/passwd should override my GID [14:43:49] did u logout since then? [14:44:13] petan|wk: when editing a file that belong to someone:depops -rw-rw---- , vim opens it in readonly mode cause it find out that it does not belong to me [14:44:34] ok that's weird [14:44:35] petan|wk: I can force write though but then the file belong to hashar:svn -rw-rw---- and other people can not read it anymore [14:44:35] sec [14:44:41] are you using dbdump instance to do that? [14:44:55] hashar: that's not possible [14:44:58] i-000000d2 [14:45:08] you can't create any file with any other group than depops [14:45:14] which is stored on nfs [14:45:36] because nfs is running background script which changes all files to belong to depops [14:45:52] so if you create such a file, it will automaticaly change the group within 1 hour [14:46:23] yeah that is a hack [14:46:36] it is going to cause various headaches on long term [14:46:43] petrb@deployment-dbdump:~$ groups hashar [14:46:43] hashar : svn depops wmf project-testswarm project-bastion project-deployment-prep project-jenkins [14:46:53] you are member of depops [14:46:59] no idea why vi doesn't let you write [14:47:15] cause it is not my primary GID :-] [14:47:22] my primary GID is svn [14:47:25] and it let me [14:47:33] another solution would be to name 550 'depops' :-] [14:47:41] why it works to me? [14:47:50] I have no idea :-/ [14:48:14] anyway, the thing I don't understand is why creating a user in /etc/passwd does not override ldap settings [14:48:27] looks like that is how we do it on fenari [14:48:32] there are so many things on labs I would like to understand, hashar [14:48:33] :D [14:48:38] ;-] [14:49:10] !Ryan [14:49:11] man of all answers ever (but there are others :)) [14:56:56] How can I access the website (/var/www) on my instance? Should I require a DNS Domains first? [14:58:22] !proxy [14:58:27] @search proxy [14:58:27] Results (found 1): socks-proxy, [14:58:44] @search tunnel [14:58:44] Results (found 1): putty, [15:00:41] You mean I need to use the basion as a proxy, and visit my instance via socks-proxy? [15:02:30] ssh shujenchang@bastion.wmflabs.org -D 8080? [15:03:07] yes [15:09:38] Firefox can't establish a connection to the server at 10.4.0.37:8080? [15:09:56] What's wrong? [15:10:16] Firefox can't establish a connection to the server at 10.4.0.37. [15:14:16] localhost:8080 [15:31:06] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours [15:35:49] petan|wk: how can I write to mkdir -p /usr/local/apache/common-local/docroot ? [15:36:01] mount is: deployment-nfs-memc:/mnt/export/apache on /usr/local/apache type nfs (rw,addr=10.4.0.58) [15:36:34] but when trying to create a file in /usr/local/apache it gives me a Permission denied :/ [15:37:16] hashar: huh [15:37:20] what you want to do? [15:37:45] mkdir -p /usr/local/apache/common-local/docroot/default [15:37:45] ;) [15:38:03] it works to me [15:38:05] done [15:38:14] don't do that as root :P [15:38:36] I have no idea what went wrong [15:39:19] anyway the deployment-web host is now a Wikimedia application server (wikimedia-task-appserver). [15:39:26] no idea if it works though :-] [15:40:18] why? :D [15:40:27] ok I need to go, don't forget to !log [15:40:31] would [15:40:37] I didn't see any line so far [15:40:37] :P [15:40:38] !log deployment-prep [15:40:38] Message missing. Nothing logged. [15:40:52] !log deployment-prep I told hashar to log stuff, if he won't, slap him [15:40:53] been busy with the depops group :-] [15:40:54] Logged the message, Master [15:41:01] ok [15:41:18] be back in 2h [15:41:38] !log deployment-prep made deployement-web host a wikimedia-task-appserver , add to create some apache2 configuration placeholder. Apache2 does launch but it is not working though (timeout) [15:41:39] Logged the message, Master [15:42:43] !log deployment-prep deployement-web host does work! :-] [15:42:44] Logged the message, Master [15:42:46] \o/ [15:45:06] mutante: so to get the hostname to show, one could use the $INSTANCENAME env variable [15:47:16] petan|wk: ^ that's what you did for Nagios? [15:52:12] is out I guess [15:56:13] 04/24/2012 - 15:56:13 - Updating keys for shujenchang [15:56:24] 04/24/2012 - 15:56:23 - Updating keys for shujenchang [15:57:13] 04/24/2012 - 15:57:13 - Updating keys for shujenchang [16:01:28] see you later tonight maybe [17:27:13] PROBLEM Puppet freshness is now: CRITICAL on nginx-dev1 i-000000f0 output: Puppet has not run in last 20 hours [17:33:36] btw, http://www.openssl.org/news/secadv_20120419.txt [17:33:44] ECHAN [17:55:53] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [18:15:52] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 3% free memory [18:20:53] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 94% free memory [18:42:13] PROBLEM Current Load is now: WARNING on bots-sql3 i-000000b4 output: WARNING - load average: 5.96, 5.77, 5.23 [18:54:13] !log deployment-prep wrap -dbdump motd to 80 chars [18:54:15] Logged the message, Master [18:57:14] is there any internal web service to log a message with labs-morebots ? [19:02:13] RECOVERY Current Load is now: OK on bots-sql3 i-000000b4 output: OK - load average: 1.62, 3.31, 4.42 [19:24:53] hashar: yes [19:24:57] log [19:27:29] hashar: can u check b 36135 [19:27:37] !bz 36135 [19:27:37] https://bugzilla.wikimedia.org/36135 [19:27:40] hello [19:27:45] finishing commit [19:27:48] ok [19:32:08] done [19:33:47] petan|wk: I don't know anything about the cache system [19:34:00] it is a serious huge pile of messy spaghettis [19:34:05] someone do know? [19:34:09] there is several layers of caches [19:34:15] and all kind of interesting issues with it [19:34:22] #mediawiki-i18n maybe [19:34:28] I think lasagna is the more appropriate Italian food metaphor here [19:34:35] it will also depends on how you have the cache setup I guess [19:34:36] is there a single person in operation team who knows how to do that :D [19:34:44] because they must run it frequently [19:35:19] I am pretty sure nobody knows anything about it beside NikeRabbit and maybe Roan [19:35:22] It's devs that do it [19:35:34] I'm new around here but from what I heard it doesn't seem to be something that we ops do.- [19:35:39] RoanKattouw: devs? [19:35:43] er, s/.-$//, too much lag [19:35:46] Devs with shell access specifically [19:35:48] they run stuff on prod? [19:35:52] aha [19:35:54] paravoid: welcome new op :-] [19:35:54] interestomg [19:36:15] Code deployments are done by devs with shell access, not by ops [19:36:26] aah [19:36:30] I should be able to answer most cache-related questions for that reason [19:36:37] so what is actually done by people in ops :D [19:36:40] \O/ [19:36:51] As I said on the bug, there's rebuildLocalisationCache.php which is probably what you need to fix that bug [19:36:53] RoanKattouw: just answer that bug [19:37:01] he did already [19:37:12] RoanKattouw: that is fine but there is a wiki farm living in 1 folder [19:37:21] the script is overwriting caches of other wikis [19:37:27] ops are handling the hardware and most of the low level software (configuring apache / checking performances / scaling hard drive …) [19:37:36] apergos told me that on prod the structure of cache is different [19:37:45] and overall they offer interesting stuff such as the labs :-D [19:37:58] files are prefixed per wiki [19:38:33] hashar: I work as op too, just for another company [19:38:54] In prod we share the l10ncache between all wikis [19:38:55] so I have an idea :D just it would be me who would run some script on prod [19:39:02] devs would just assist with that [19:39:10] And we use a weird-ass hack to make sure all extensions are included [19:39:27] RoanKattouw: that hack is what we need [19:39:47] RoanKattouw: actually I was about to make it, but I was like. someone from ops would see it and told me it's ass hack [19:39:56] or else some more reasonable hack that won't kill everything if it fails [19:40:47] The production hack is kind of elaborate and terrible [19:40:51] oh, hi chrismcmahon [19:41:00] I'm Faidon [19:41:07] We have wmf-config/extension-list [19:41:10] hi :) I've been following along [19:41:16] ok so that's reason why I didn't make it was that I thought someone from operation would tell me it's horrible hack [19:41:22] extension-list looks like: [19:41:22] irc is much better, isn't it? :) [19:41:24] $IP/extensions/AbuseFilter/AbuseFilter.php [19:41:26] $IP/extensions/AntiBot/AntiBot.php [19:41:26] and now I see that's exactly what you have in prod [19:41:27] $IP/extensions/AntiSpoof/AntiSpoof.php [19:41:47] Then there's the mergeMessageFileList.php maintenance script, which produces ExtensionMessages-1.20wmf1.php [19:42:00] ah [19:42:07] This file looks roughly like this: [19:42:09] $wgExtensionMessagesFiles = array ( [19:42:11] 'PagedTiffHandler' => "$IP/extensions/PagedTiffHandler/PagedTiffHandler.i18n.php", [19:42:12] 'PagedTiffHandlerMagic' => "$IP/extensions/PagedTiffHandler/PagedTiffHandler.i18n.magic.php", [19:42:13] RoanKattouw: could you somehow get these scripts to labs [19:42:14] 'Timeline' => "$IP/extensions/timeline/Timeline.i18n.php", [19:42:37] Then at the END of CommonSettings.php, we have: [19:42:40] given that we use same extensions it should work on labs [19:42:41] require( "$wmfConfigDir/ExtensionMessages-$wmfExtendedVersionNumber.php" ); [19:42:51] The scripts are all there [19:42:52] yes we have this too [19:42:56] but we had to comment it out [19:43:03] mergeMessageFileList.php is in MediaWiki [19:43:16] Or possibly in the WikimediaMaintenance extension [19:44:12] hm... [19:44:13] ok [19:44:18] we should probably make a list of things that need to get done in deployment-prep [19:44:23] for both ops and devs [19:44:26] Yeah [19:44:31] that's what I try in bz [19:44:33] of course, this would assume we actually have a dev working in it [19:44:36] I made a lot of tickets ther [19:44:43] which we do not, right now [19:45:11] for example one ticket where I wanted to get a config of squid :) [19:45:20] heh [19:45:31] I've never seen this ticket. what's this you speak of? [19:45:39] nevermind [19:45:47] I'm kidding ;) [19:45:47] I think someone actually fixed it [19:45:50] dunno who [19:45:55] o.O [19:45:55] people don't log [19:45:57] someone did? [19:46:00] yes [19:46:09] it's the same as in production? [19:46:11] config of squid was completely changed by someone [19:46:14] I can't compare it [19:46:15] hm [19:46:17] weird [19:46:24] but it works much better now [19:46:49] there's one reason I haven't done this yet [19:46:52] no idea who it was there are only logs from me [19:46:54] there's no way to keep them in synch [19:47:04] between production and labs [19:47:14] yes I wanted to sync CommonsSettings and Initialise too [19:47:19] hashar: any idea how to do that? [19:47:22] that would be nice. [19:47:25] Put wmf-config in a git repo [19:47:32] I've been wanting to do that forever [19:47:39] on prod? [19:47:52] RoanKattouw: some things need to be disabled in labs [19:47:53] But 1) no time and 2) it needs restructuring to put the private stuff in a separate dir and include some stuff from common/ like *.dblist [19:48:02] but labs doesn't have identical config [19:48:05] anti-vandalism-network stuff, for instance [19:48:10] Can't they have override files? [19:48:18] how would that possibly work? [19:48:28] RoanKattouw: that's what I would like to know how to do [19:48:42] we need to split the cluster specific settings into a separate file [19:49:00] Cluster-specific as in "have to disable in labs"? [19:49:02] indeed if we restructured the config on prod it would be simpler [19:49:04] then we don't need to worry about merge conflicts [19:49:06] Cause really the entire config is WMF-specific [19:49:07] having one huge file [19:49:08] is problem [19:49:17] RoanKattouw: not most of it [19:49:42] there are few lines different right now [19:49:46] most of config is same [19:49:52] By WMF-specific I don't mean specific to our cluster as opposed to labs, I mean specific to our site as opposed to some other wiki [19:49:55] for example the dns is different we are on wmflabs.org [19:50:14] etc [19:50:17] <^demon> Everyone should remember that config mgmt is getting overhauled this summer anyway :) [19:50:22] Right, yeah, that kinda stuff [19:50:24] Oh, right! [19:50:30] yes, but we need to have this kind of stuff working now [19:50:31] we have some wikis like deployment.wikimedia that doesn't exist on prod [19:50:54] RoanKattouw petan|wk sort of the immediate question is whether we can bring labs beta Commons back to being usable [19:51:04] I hope yes [19:51:18] <^demon> Ryan_Lane: I know, I'm just saying nobody should put a huge amount of effort into this beyond "making it work" [19:51:27] <^demon> Since it'll all be scrapped for big-picture unicorns later. [19:51:29] I mean I am sure yes [19:51:31] petan|wk: A workaround you could use is to have a separate $wgCacheDirectory for every wiki, and turn off manualRecache [19:51:35] chrismcmahon: by tommorow it should be ok [19:51:39] I've been hearing that for ages now [19:51:45] since the prototype days [19:52:01] thanks petan|wk let me know if I can help in any way [19:52:25] <^demon> Ryan_Lane: It's on the roadmap for after I finish git migration stuff :) [19:52:52] chrismcmahon: actually I can fix commons now [19:53:07] beware [19:53:21] beware that one of the apache runs using wikimedia::appserver task [19:53:41] I am not sure yet what is the beta infrastructure but that might cause issue [19:53:59] That shouldn't be a problem [19:54:25] we *should* be using that stuff [19:54:29] yup [19:54:38] that is the first thing I have done :-D [19:55:21] what is the overall n-tiers architecture anyway? client -> 1 squid -> 1 LVS -> 7 apaches -> 1 memcached -> 1 MySQL DB ? [19:55:28] hashar: can you make all apaches similar or same :D [19:55:30] would be great to have an overview on a wiki page somewhere [19:55:42] hashar: also keep in mind that /etc/apache2 is on nfs [19:55:45] <^demon> So apparently when you want to merge the same fix to several branches, you can just re-use the same change-id and it will group them? [19:55:53] so if u change config on one of them, you change them all [19:55:54] <^demon> Is this actually true? How did we not know this? [19:55:59] petan|wk: yeah should be all about testing the first instance then add the wikimedia::appserver to all the apache instances [19:56:37] ^demon: aren't you reusing the same change-id whenever you use git cherry-pick? :-] [19:57:14] petan|wk: I think we will want to drop /etc/apache2 and NFS all together eventually [19:57:33] anyway I got a conf call [19:57:34] brb [19:57:34] hm... [19:57:36] ok [20:30:04] petan|wk: back around :-D [20:39:29] ok [21:00:16] petan|wk: well had another phone call and I am now heading bed :-/ [21:00:22] see you tomorrow [21:02:09] ok [21:06:14] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [21:10:14] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [22:16:17] paravoid: Do you have a guess as to when per-project puppet branches will be a reality? If it's close then I'll put all of my puppet tinkering on hold. [22:17:03] it's on my immediate goals but there are some semi-complicated issues to be resolved [22:17:31] ok. Let me know if there are any subtasks that I can take on. [22:33:58] * jorm sighs. YAGDIC. [22:35:10] !initial-account | jorm [22:35:17] where's that damn bot? :( [22:35:22] !initial-account | jorm [22:35:27] -_- [22:35:39] * jorm raises an eyebrow. [22:39:35] Ryan_Lane: bot is right here [22:39:38] !ping [22:39:38] pong [22:39:43] you are asking wrong [22:39:47] I am? [22:40:02] @search know [22:40:02] Results (found 1): sal, [22:40:08] @search I want to [22:40:08] Results (found 1): whatIwant, [22:40:12] heh [22:40:21] @search account [22:40:21] Results (found 4): credentials, account, account-questions, accountreq, [22:40:30] where's the damn keyword? [22:40:36] !account-questions [22:40:36] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [22:41:18] not that [22:41:25] bah [22:41:27] I'm dumb [22:41:32] !initial-login | jorm [22:41:32] jorm: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [22:41:36] http://bots.wmflabs.org/~petrb/db/%23wikimedia-labs.htm [22:42:00] !db is http://bots.wmflabs.org/~petrb/db/%23wikimedia-labs.htm [22:42:00] Key was added! [22:53:02] petan, around? [22:56:26] Ryan_Lane: did mutante talk to you about creating a mailman project on labs? [22:56:40] Eww mailman [22:56:44] I think there was some discussion about this [22:57:04] I think the ops consensus was that we didn't want to maintain any changes [22:58:10] maintain changes? I was under the impression it was a one time thing for now Ryan_Lane to get the new listinfo pages globally across the mailing lists [23:15:22] Eloquence: hi [23:17:26] heya [23:17:36] looks like fancycaptcha on deploymentprep is acting up on account creation [23:17:50] oh [23:18:00] is it bad or right? [23:18:14] let me give you a URL once it stops stalling (it's still occasionally slow for me) [23:18:19] here: http://commons.wikimedia.beta.wmflabs.org/w/index.php?title=Special:UserLogin&type=signup&returnto=Main+Page [23:19:21] I see [23:21:23] !log deployment-prep petrb: disabled $wmgEnableCaptcha [23:21:24] Logged the message, Master [23:21:34] Eloquence: fixed [23:22:24] it's a temporary fix, we need to find out why it was broken [23:22:35] for now it shouldn't ask for any captcha [23:22:56] thanks! [23:23:35] it's still sloooow :( we did upgrade the web instances? [23:24:59] yes it's running large instances now, I asked hashar to help me debug why it's so slow [23:25:19] we didn't get much results so far [23:26:01] I thought it was IO but now I moved some stuff to gluster and it's not any better [23:26:12] sql server is idle most of time [23:26:18] only apaches are on fire [23:27:14] http://ganglia.wmflabs.org/latest/?c=deployment-prep&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [23:29:23] I need to go now, let me know in email if you needed anything else [23:29:27] petan|wk: feel free to raise any labs or ops-related issues with me [23:29:32] if I can help in any way. [23:29:54] ok, I will :-) [23:30:00] just not now [23:30:23] it's almost 2 am [23:30:38] good night petan [23:30:44] thanks :)