[00:16:10] csteipp: /srv/deployment/mediawiki/common/wikiversions.cdb [00:16:18] csteipp: that's missing on beta [00:16:22] any idea how that's generated? [00:16:43] Reedy would know for sure, but I think that's something we keep in synch manually [00:16:57] from wikiversions.dat [00:17:08] sync-wikiversions makes it and pushes to mediawiki-installation [00:18:12] from the .dat file? [00:18:20] ya [00:18:24] sync pushes both .dat and .cdb [00:18:28] but how is the .cdb generated? [00:18:47] and are we going to be checking that in locally or something? [00:18:48] $COMMONDIR/multiversion/refreshWikiversionsCDB [00:18:49] $COMMONDIR/multiversion/refreshWikiversionsCDB [00:19:01] /usr/local/bin/sync-wikiversions does the whole lot [00:19:02] I see it's on tin [00:19:05] was it checked in? [00:19:17] right, but we're getting rid of scap [00:19:33] Still essentially the same process (make, push, distribute) [00:19:39] the dat was before, I think I added the cdb on newdeploy [00:19:43] so we need to check it in locally [00:20:16] hmm, maybe I didn't [00:20:21] But yeah, if it's not already it does [00:20:37] what do you mean it does? [00:20:39] what does? [00:20:50] it does want checking in [00:20:54] heh [00:22:02] on tin it must be checked in [00:22:27] Invalid version dir '/srv/deployment/mediawiki/common/php-master' for wiki 'aawiki'. [00:22:27] it needs checking in? or it already is? :p [00:22:28] eh? [00:22:39] Because labs don't do the same thing [00:22:40] well, git status shows clean [00:22:41] and it's there [00:22:47] ……. [00:22:53] ok, you're not really being clear here [00:22:55] labs always uses master [00:23:02] it's not going to for now [00:23:24] until we add the deploment system to production beta will be running the same versions [00:24:09] https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=commitdiff;h=4b33c7ca13271336a26aa04cbd717d73bb9649f9 [00:24:13] Looks like antoine started [00:24:24] not sure why he only did 2 though [00:24:25] hm [00:24:34] ah, it uses a different file [00:24:49] Can you say hack? :D [00:25:01] well, I understand why [00:25:09] aye [00:25:21] Reedy: mind updating the labs one to point to the correct ones? [00:25:27] is it just a matter of copy/paste? [00:26:50] Why has antoine kept the php prefix? :/ [00:27:09] no clue :) [00:27:13] that's not correct [00:27:19] indeed [00:27:20] just replace all php-master with 1.21wmf7 in the file [00:27:24] * Reedy does [00:32:08] Done [00:33:16] thanks [00:34:44] So on the i10n stuff... it looks like mw-update-l10n updates the entire cache in one place, and then sync-l10nupdate-1 copies out the cache to the branch specific place on each apache [00:34:57] I'm guessing Brad didn't rewrite that piece yet? [00:35:04] I think he did [00:35:19] I think it's in his homedir on tin [00:35:26] the cron needs some work [00:35:39] because I need to put in logic for knowing when it can move past the fetch step [00:35:53] since it's automated [00:37:14] well, one thing that's fairly faster on labs is deployment [00:37:16] :) [00:37:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 24% free memory [00:37:32] though if it had 150 minions it would probably be much slower ;) [00:38:43] RECOVERY Free ram is now: OK on newprojectsfeed-bot.pmtpa.wmflabs 10.4.0.232 output: OK: 34% free memory [00:39:03] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 23% free memory [00:41:40] wikiversions.cdb has no version entry for `aawikibooks`. [00:41:43] -_- [00:42:22] indeed it does not [00:42:34] did the cdb get rebuilt? [00:42:44] yeah, it's missing in the .dat file too, though [00:43:12] where is that complaining? [00:43:16] on beta [00:43:30] first entry in the production .dat: aawikibooks 1.21wmf7 * [00:43:42] first entry for labs one: aawiki 1.21wmf7 [00:43:50] why the * in the production one? [00:44:00] I'm not sure, Aaron added it for some reason [00:44:11] I did hesitate when I made my changes [00:44:14] heh [00:44:29] maybe the labs one should be exactly like the production one for now? [00:44:33] https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=blob;f=wikiversions-labs.dat;h=a458491a92b207b5e80beb4b689bfa90563a4742;hb=HEAD [00:44:44] The one in master has none either [00:44:54] dewikivoyage php-master * [00:45:13] right at the end [00:45:22] hooray for consistency :D [00:45:32] I'd say for now let's make labs the same as production [00:45:35] Yeaaah [00:45:46] What was actually complaining about the lack of aawikibooks though? [00:45:51] yep [00:46:23] it's missing from the .dat file [00:47:08] And the script that was actually complaining? [00:47:18] ^ that's what I meant [00:47:33] /var/lib/git-deploy/dependencies/l10n [00:47:50] it's the new version of l10nupdate-quick [00:47:53] oh [00:48:00] I wonder if that's the wiki it's picking to use [00:48:05] well, not I wonder [00:48:15] Why aawikibooks when we use aawiki as our usual fallback? [00:48:19] mwVerDbSets=$($BINDIR/mwversionsinuse --withdb) [00:48:38] 1.21wmf7=aawikibooks 1.21wmf6=abwiki [00:49:36] is the current cdb built from production? [00:50:10] I'd imagine it's using the -labs.dat [00:51:01] since the cdb refresher complained about it when it had php-master in it [00:52:06] Array [00:52:06] ( [00:52:06] [ee_prototypewiki] => 101 [00:52:06] [labswiki] => 246 [00:52:06] [nnwikibooks] => 303 [00:52:06] ) [00:52:06] The above 3 wiki DBs are missing wikiversion rows. [00:52:15] reedy@tin:/srv/deployment/mediawiki/common$ multiversion/activeMWVersions --withdb [00:52:15] 1.21wmf7=aawiki 1.21wmf6=enwiktionary [00:52:15] that happens when I try to use the production one [00:52:28] weird. I wonder why its different in labs [00:52:32] ^ I just moved the labs.dat to the default .dat [00:52:44] rebuilt the cdb and then ran activeMWVersions again [00:53:18] multiversion/refreshWikiversionsCDB doesn't seem to take any account of whether we're on labs or not [00:53:24] unless getRealmSpecificFilename is suppose to.. [00:53:36] it must [00:54:04] ah, seems there's three wikis in beta that don't exist in production [00:54:53] PROBLEM host: ee-lwelling.pmtpa.wmflabs is DOWN address: 10.4.0.243 CRITICAL - Host Unreachable (10.4.0.243) [00:55:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [00:55:38] hm. even after I rebuild it, it still comes back with: 1.21wmf7=aawikibooks 1.21wmf6=abwiki [00:57:48] mv wikiversions.dat wikiversions-prod.dat && mv wikiversions-labs.dat wikiversions.dat && ./multiversion/refreshWikiversionsCDB && ./multiversion/activeMWVersions --withdb [00:58:10] ah [00:58:12] wikiversions.dat? [00:58:21] wait [00:58:24] inore me [00:58:34] yeah [00:59:32] ah [00:59:35] wtf [00:59:43] I don't get that [01:00:02] it works now? [01:00:25] If so, I'd say multiversion/MWRealm.php is brokeneded [01:00:25] yes, and I don't understand wy [01:01:29] Are /etc/wikimedia-realm and /etc/wikimedia-site correct? [01:01:46] yep [01:03:32] ah. no private settings file now [01:03:34] that's progress [01:05:05] reedy@deployment-bastion:/srv/deployment/mediawiki/common$ php -a [01:05:06] Interactive shell [01:05:06] php > require_once( 'multiversion/MWRealm.php' ); [01:05:06] php > echo getRealmSpecificFilename( 'wikiversions.dat' ); [01:05:06] wikiversions-labs.dat [01:05:08] :| [01:06:43] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 175 processes [01:07:06] php > echo getRealmSpecificFilename( '/srv/deployment/mediawiki/common/wikiversions.dat' ); [01:07:06] /srv/deployment/mediawiki/common/wikiversions-labs.dat [01:08:53] RECOVERY host: ee-lwelling.pmtpa.wmflabs is UP address: 10.4.0.243 PING OK - Packet loss = 0%, RTA = 11.65 ms [01:09:16] I guess it's going to be something daft [01:09:23] PROBLEM Total processes is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [01:09:25] Fatal error: require(): Failed opening required '/srv/deployment/mediawiki/common/l10n-1.21wmf7/ExtensionMessages.php' [01:09:34] I thought that was moved [01:09:44] oh [01:09:44] they were in wmf-config [01:09:45] it hasn't been built yet [01:10:11] hm [01:10:18] shouldn't it write that file out, and then use it? [01:10:53] PROBLEM Current Load is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [01:10:53] PROBLEM dpkg-check is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [01:10:57] presumably, that's what scap did [01:11:11] I thought it did on tin, too [01:11:31] They're certainly there on Tin [01:11:33] PROBLEM Current Users is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [01:11:43] yeah, it worked properly there [01:11:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes [01:11:54] maybe it was because it did a full build accidentally the first time [01:12:12] before the backported change that anomie added to core [01:12:13] PROBLEM Disk Space is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [01:13:03] PROBLEM Free ram is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [01:20:52] RECOVERY Current Load is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: OK - load average: 1.03, 1.14, 0.74 [01:20:52] RECOVERY dpkg-check is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: All packages OK [01:21:32] RECOVERY Current Users is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: USERS OK - 0 users currently logged in [01:22:12] RECOVERY Disk Space is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: DISK OK [01:23:02] RECOVERY Free ram is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: OK: 898% free memory [01:24:22] RECOVERY Total processes is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: PROCS OK: 84 processes [01:41:43] PROBLEM Free ram is now: WARNING on newprojectsfeed-bot.pmtpa.wmflabs 10.4.0.232 output: Warning: 19% free memory [01:55:02] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [02:50:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 4.11, 5.19, 5.06 [02:55:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.65, 4.90, 4.98 [03:03:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 6.26, 5.92, 5.41 [03:08:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 8.53, 7.95, 6.18 [03:19:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 5.80, 5.90, 5.40 [03:44:44] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 3.96, 4.42, 4.94 [03:52:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 4.47, 5.03, 5.12 [04:18:44] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 4.75, 4.64, 4.91 [04:34:31] !tunnel [04:34:31] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [04:40:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [04:41:43] RECOVERY Free ram is now: OK on newprojectsfeed-bot.pmtpa.wmflabs 10.4.0.232 output: OK: 34% free memory [04:58:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.51, 4.34, 4.92 [05:08:27] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [05:24:42] PROBLEM Free ram is now: WARNING on newprojectsfeed-bot.pmtpa.wmflabs 10.4.0.232 output: Warning: 19% free memory [06:14:43] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [06:21:54] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [06:25:46] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 20% free memory [06:30:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 151 processes [06:34:43] RECOVERY Free ram is now: OK on newprojectsfeed-bot.pmtpa.wmflabs 10.4.0.232 output: OK: 51% free memory [06:46:52] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 20% free memory [06:48:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 18% free memory [06:50:54] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [06:51:34] RECOVERY dpkg-check is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: All packages OK [06:55:13] RECOVERY dpkg-check is now: OK on testing-arky.pmtpa.wmflabs 10.4.0.45 output: All packages OK [08:38:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [08:38:52] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 22% free memory [08:46:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [09:02:42] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 154 processes [09:07:42] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 148 processes [09:26:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 18% free memory [09:47:42] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 156 processes [11:27:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.95, 4.81, 4.99 [11:38:42] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 158 processes [11:56:52] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 20% free memory [12:29:53] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 18% free memory [12:39:54] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 21% free memory [13:15:08] !log wikidata-dev wikidata-dev-8: client is still on wmf7, repo has been updated to 2013-01-14. Files WBpollForChanges_testclienten.continue and WBpollForChanges_testclienten.pid are now in /var/run and permissions had to be adapted for wikidata user to run pollForChanges. Katie also modified something else, see https://gerrit.wikimedia.org/r/#/c/44048/ [13:15:09] Logged the message, Master [13:17:13] PROBLEM Current Load is now: WARNING on bots-sql1.pmtpa.wmflabs 10.4.0.52 output: WARNING - load average: 5.09, 9.12, 5.11 [13:27:14] RECOVERY Current Load is now: OK on bots-sql1.pmtpa.wmflabs 10.4.0.52 output: OK - load average: 0.10, 2.69, 4.10 [13:27:54] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 18% free memory [14:19:53] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [14:19:53] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [14:23:53] RECOVERY host: gerrit-dev.pmtpa.wmflabs is UP address: 10.4.0.207 PING OK - Packet loss = 0%, RTA = 0.86 ms [14:23:53] RECOVERY host: gerrit-db.pmtpa.wmflabs is UP address: 10.4.0.47 PING OK - Packet loss = 0%, RTA = 0.62 ms [14:24:23] PROBLEM Total processes is now: CRITICAL on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: Connection refused by host [14:24:24] PROBLEM Total processes is now: CRITICAL on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: Connection refused by host [14:25:53] PROBLEM Current Load is now: CRITICAL on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: Connection refused by host [14:25:53] PROBLEM dpkg-check is now: CRITICAL on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: Connection refused by host [14:25:53] PROBLEM Current Load is now: CRITICAL on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: Connection refused by host [14:26:33] PROBLEM Current Users is now: CRITICAL on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: Connection refused by host [14:26:33] PROBLEM Current Users is now: CRITICAL on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: Connection refused by host [14:26:33] PROBLEM dpkg-check is now: CRITICAL on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: Connection refused by host [14:27:19] PROBLEM Disk Space is now: CRITICAL on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: Connection refused by host [14:27:19] PROBLEM Disk Space is now: CRITICAL on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: Connection refused by host [14:28:04] PROBLEM Free ram is now: CRITICAL on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: Connection refused by host [14:28:04] PROBLEM Free ram is now: CRITICAL on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: Connection refused by host [14:33:02] RECOVERY Free ram is now: OK on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: OK: 2293% free memory [14:33:02] RECOVERY Free ram is now: OK on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: OK: 641% free memory [14:34:22] RECOVERY Total processes is now: OK on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: PROCS OK: 84 processes [14:34:23] RECOVERY Total processes is now: OK on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: PROCS OK: 100 processes [14:35:53] RECOVERY Current Load is now: OK on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: OK - load average: 0.14, 0.70, 0.61 [14:35:53] RECOVERY dpkg-check is now: OK on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: All packages OK [14:35:53] RECOVERY Current Load is now: OK on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: OK - load average: 0.23, 0.75, 0.60 [14:36:33] RECOVERY Current Users is now: OK on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: USERS OK - 0 users currently logged in [14:36:33] RECOVERY Current Users is now: OK on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: USERS OK - 0 users currently logged in [14:36:33] RECOVERY dpkg-check is now: OK on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: All packages OK [14:37:13] RECOVERY Disk Space is now: OK on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: DISK OK [14:37:13] RECOVERY Disk Space is now: OK on gerrit-dev.pmtpa.wmflabs 10.4.0.207 output: DISK OK [14:48:53] PROBLEM dpkg-check is now: CRITICAL on gerrit-db.pmtpa.wmflabs 10.4.0.47 output: DPKG CRITICAL dpkg reports broken packages [15:52:07] !log integration running puppet on integration-jenkins2 to find out how bad it is right now. [15:52:09] Logged the message, Master [15:54:38] !log integration -jenkins2 : reset hard /usr/local/src/zuul . It had a failed merge. That should make puppet bring up the latest Zuul version. [15:54:39] Logged the message, Master [15:55:25] !log integration -jenkins2 manually updated /etc/zuul/wikimedia repo [15:55:26] Logged the message, Master [15:56:44] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 153 processes [16:01:44] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 149 processes [16:14:44] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 153 processes [16:37:54] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 21% free memory [16:50:33] !log bots petrb: inserting wikivoyage.org subdomains to wm-bot db [16:50:34] Logged the message, Master [17:00:47] @system-rm RC [17:00:47] Unloaded module RC [17:00:59] File not found modules/modules/wmib_rc.bin [17:01:03] Loaded module modules/wmib_rc.bin [17:01:15] @recentchanges+ en_wikivoyage [17:01:15] Wiki inserted [17:01:27] @RC+ en_wikivoyage W* [17:01:28] Inserted new item to feed of changes [17:04:36] @RC+ en_wikivoyage U* [17:04:36] Inserted new item to feed of changes [17:04:44] meh, not a lot of changes there... [17:05:50] either that or bot is broken [17:05:53] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 18% free memory [17:08:52] @RC+ en_wikivoyage W* [17:08:52] There is already this string in a list of watched items [17:09:01] meh [17:09:03] @RC- en_wikivoyage W* [17:09:03] Deleted item from feed [17:09:05] @RC- en_wikivoyage U* [17:09:05] Deleted item from feed [17:09:11] who cares... will fix tommorow [17:09:14] see ya all [17:50:43] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 162 processes [17:51:33] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 19% free memory [17:57:39] Morning all. I'm having difficulty creating a labs instance. Depending on which puppet groups I configure on it, I either get errors in the puppet run or I get access denied when I try to ssh in. Is there a known good combination? [17:58:20] lwelling: I believe people are encouraged to try and create an instance without any puppet groups first, then only install stuff on it after SSH is confirmed to work [17:58:26] Also, building an instance takes time, up to ~5 mins [17:58:39] As in, 5 minutes after it tells you it should be ready [17:59:25] Thanks Roan. It says it sends email when done, but I've not recieved one so I've looked at console output to guess [17:59:44] I've not tried ssh before adding to it, I'll try that now [18:04:52] PROBLEM host: eelwelling.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:06:04] what's the deal with the token field for the labs login page? [18:08:54] RECOVERY host: eelwelling.pmtpa.wmflabs is UP address: 10.4.1.3 PING OK - Packet loss = 0%, RTA = 1.24 ms [18:09:24] PROBLEM Total processes is now: CRITICAL on eelwelling.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [18:10:52] PROBLEM Current Load is now: CRITICAL on eelwelling.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [18:10:52] PROBLEM dpkg-check is now: CRITICAL on eelwelling.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [18:11:32] PROBLEM Current Users is now: CRITICAL on eelwelling.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [18:12:12] PROBLEM Disk Space is now: CRITICAL on eelwelling.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [18:13:06] PROBLEM Free ram is now: CRITICAL on eelwelling.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [18:15:44] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 150 processes [18:18:04] RECOVERY Free ram is now: OK on eelwelling.pmtpa.wmflabs 10.4.1.3 output: OK: 620% free memory [18:19:24] RECOVERY Total processes is now: OK on eelwelling.pmtpa.wmflabs 10.4.1.3 output: PROCS OK: 84 processes [18:20:54] RECOVERY Current Load is now: OK on eelwelling.pmtpa.wmflabs 10.4.1.3 output: OK - load average: 0.17, 0.71, 0.62 [18:20:54] RECOVERY dpkg-check is now: OK on eelwelling.pmtpa.wmflabs 10.4.1.3 output: All packages OK [18:21:34] RECOVERY Current Users is now: OK on eelwelling.pmtpa.wmflabs 10.4.1.3 output: USERS OK - 0 users currently logged in [18:22:14] RECOVERY Disk Space is now: OK on eelwelling.pmtpa.wmflabs 10.4.1.3 output: DISK OK [18:24:04] <^demon|lunch> Ryan_Lane: Are the testlabs/* branches on ops/puppet used for anything? Curious what the acls are for. [18:24:12] <^demon|lunch> Oh, looks like yes. Hm. [18:24:44] I don't think anyone actually uses them, but they are there so that people can make remote branches [18:46:28] <^demon> Ryan_Lane: Well, one of my fears about the upgrade is resolved. I was afraid the ldap group conversion was messy and we'd have acls to clean up. [19:06:29] lwelling, having better luck with with instance creation/config now? [19:21:43] anyone here? [19:22:28] * hashar look around [19:22:40] benestar: go ahead and ask :-D [19:22:43] someone might eventually answer [19:22:49] :) [19:22:52] if all fail, we have a labs mailing list too :-] [19:23:07] there seems to be a problem with one of the webservers [19:23:29] http://bots.wmflabs.org/~bene/items_by_cat.php [19:23:45] one time it works, the other time I get an 403 error [19:44:46] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes [19:46:29] benestar, are you subscribed to labs-l? [19:47:00] andrewbogott: where to subscribe? [19:47:15] benestar, no idea if it's connected, but there was a recent discussion there about spontaneously-changing permissions on some bots machines. [19:47:36] benestar, here maybe? https://lists.wikimedia.org/mailman/listinfo/labs-l [19:47:45] thanks [19:48:10] The thread in question starts here: http://lists.wikimedia.org/pipermail/labs-l/2013-January/000706.html [19:51:19] andrewbogott: could be the reason [19:51:37] the permission was wrong so I changed it [19:52:00] but even now, after changing, it does not work everytime [19:52:14] Is that because the permissions are changing back? [19:52:38] just checked [19:52:45] the permission should be right [19:53:03] maybe it is an issue of the servers? [19:53:07] So maybe something in an Apache config [19:53:27] Yeah, I have roughly no idea how bots is set up [19:53:58] maybe petan is around… or damianz? [19:56:47] andrewbogott: seems not [19:57:15] then your guess is as good as mine :( [19:57:22] * addshore waves [19:57:31] whats the problem? [19:57:34] hi [19:57:46] there is a problem at bots [19:57:51] D: [19:57:54] http://bots.wmflabs.org/~bene/items_by_cat.php [19:58:08] O_o [19:58:17] this is a tool written by me and I cannot call it [19:59:02] addshore: any idea? [19:59:16] let me have a look [19:59:37] I had to change the permissoins (they were changed anyhow) but even now it does not work :7 [19:59:56] !tunnel [19:59:56] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [20:02:06] have you tried setting permissions to 755? [20:04:13] hmm, even 777 doesnt make a difference.. [20:04:25] re: tunnel https://labsconsole.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option [20:04:29] bblunch [20:06:18] sorry benestar, no idea [20:06:38] addshore: :( [20:39:42] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 149 processes [20:40:52] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 22% free memory [20:41:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 23% free memory [20:41:32] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 34% free memory [20:49:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [21:00:13] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [21:04:28] https://plus.google.com/hangouts/_/b33de5d0cc90ed8a6cadb4b35ec53e30c22b0291?authuser=0&hl=en-US# [21:04:32] PROBLEM Free ram is now: CRITICAL on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: Critical: 5% free memory [21:04:40] https://plus.google.com/hangouts/_/b33de5d0cc90ed8a6cadb4b35ec53e30c22b0291?authuser=0&hl=en-US#https://plus.google.com/hangouts/_/b33de5d0cc90ed8a6cadbhttps://plus.google.com/hangouts/_/b33de5d0cc90ed8a6cadb4b35ec53e30c22b0291?authuser=0&hl=en-US#4b35ec53e30c22b0291?authuser=0&hl=en-US# [21:06:53] PROBLEM Current Load is now: WARNING on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: WARNING - load average: 5.08, 7.78, 5.60 [21:08:53] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 17% free memory [21:09:33] RECOVERY Free ram is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: OK: 407% free memory [21:11:53] RECOVERY Current Load is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: OK - load average: 0.65, 3.16, 4.20 [21:12:14] * Damianz pats andrewbogott [21:44:33] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 19% free memory [21:49:11] !log deployment prep renamed /data/project/apache/common-local to common-local.pre-git-deploy [21:49:11] deployment is not a valid project. [21:49:22] !log deployment-prep renamed /data/project/apache/common-local to common-local.pre-git-deploy [21:49:24] Logged the message, Master [21:49:38] !log deployment-prep ln -s /srv/deployment/mediawiki/common /data/project/apache/common-local [21:49:40] Logged the message, Master [21:51:03] andrewbogott hi [21:51:13] * andrewbogott waves [21:51:21] what did u need [21:51:54] benestar was having trouble with permissions on bots. I think he's gone now though... [21:52:01] ah [21:52:05] Oh, nope, he's still here! maybe. [21:52:08] petan: i am here [21:52:28] ok, what troubles did u have :D [21:52:36] petan: http://bots.wmflabs.org/~bene/items_by_cat.php [21:52:42] suddenly it did not work any more [21:53:04] I had to change the rights because they were changed anyhow [21:53:06] @labs-user Bene [21:53:06] Bene is member of 2 projects: Bastion, Bots, [21:53:30] benestar the permissions are changed by some script made by Damianz [21:53:41] well, it worked some times and other times not [21:53:43] hu? [21:53:56] why changes damianz the permissions? [21:54:10] because he wanted to chmod 000 users who are no longer in the project [21:54:18] so this script handles that automaticaly [21:54:20] it has bugs [21:54:46] doesn't have bugs [21:54:48]