[00:20:33] RECOVERY Free ram is now: OK on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: OK: 43% free memory [01:03:32] PROBLEM Free ram is now: WARNING on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Warning: 12% free memory [01:08:43] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 168 processes [01:13:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 91 processes [02:47:22] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 18% free memory [04:59:33] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [05:04:32] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [06:13:33] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [06:29:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 154 processes [06:32:22] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 47% free memory [06:47:32] PROBLEM Total processes is now: WARNING on bots-2.pmtpa.wmflabs 10.4.0.42 output: PROCS WARNING: 154 processes [06:49:52] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 150 processes [06:52:33] RECOVERY Total processes is now: OK on bots-2.pmtpa.wmflabs 10.4.0.42 output: PROCS OK: 144 processes [06:55:23] PROBLEM Free ram is now: WARNING on bots-2.pmtpa.wmflabs 10.4.0.42 output: Warning: 18% free memory [06:55:25] Ryan_Lane, if you're around (or if you get this later): has any progress been made on fixing the issue I was having with Gerrit? [06:56:31] It's been several days now. [07:15:22] RECOVERY Free ram is now: OK on bots-2.pmtpa.wmflabs 10.4.0.42 output: OK: 21% free memory [07:30:19] franny: did ^demon not handle it for you? [07:33:16] Does anyone know the project name for wikidata on wm-bot? [07:33:37] Ryan_Lane, doesn't look like it. [07:33:40] petan: ^ [07:33:42] PROBLEM Free ram is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Critical: 5% free memory [07:33:46] franny: let me try [07:33:49] I've tried wikidata, wikidata_wiki, wikidatawiki [07:33:56] wikidata-dev [07:34:25] Ryan_Lane: didnt work [07:34:39] what are you trying? [07:35:13] [11:34:12 PM] <+legoktm> @RC+ wikidata-dev User_talk:Legoktm [07:35:13] [11:34:14 PM] <+wm-bot> Unable to insert the string to the list because there is no such wiki site known by a bot, contact some developer with svn access in order to insert it [07:35:32] oh [07:35:33] wiki name? [07:35:36] I have no clue [07:35:42] wikidatawp? [07:36:05] didnt work :( [07:36:14] I have no clue what it is [07:36:28] is the full list in a git repo somewhere? [07:36:43] found it [07:37:40] (the repo, not the name) [07:38:00] franny: try logging into gerrit with username Fran McCrory [07:38:47] Same thing, Ryan_Lane. [07:38:52] ok [07:39:05] I have no idea, I'll just wait for petan... [07:40:36] franny: try again now [07:42:44] Same thing. [07:45:14] franny: one more time, while I'm tailing the logs? [07:46:12] Done. [07:46:20] ok. let me try to figure out what I need to fix [07:47:27] franny: ok, try now [07:49:29] Same thing. [07:49:35] yeah [07:56:03] I have to sleep right now; I'll be back tomorrow. Thank you for the help. [07:57:42] yw [08:34:38] hm [08:34:39] legoktm [08:34:45] hi [08:35:07] @labs-project-info wikidata [08:35:07] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 43 seconds ago [08:35:13] @labs-project-info wikidata-dev [08:35:13] The project Wikidata-dev has 8 instances and 18 members, description: Wikidata aims to create a free knowledge base about the world that can be read and edited by humans and machines alike. It will provide data in all the languages of the Wikimedia projects, and allow for the central access to data in a similar vein as Wikimedia Commons does for multimedia files. Wikidata is proposed as a new Wikimedia hosted and maintained project. [08:36:23] legoktm what you meant? [08:36:56] no, i want to @RC+ a page on wikidata, but i dont know the project name [08:37:16] like @RC+ wikidata User_talk:Legoktm wasn't working [08:37:25] aha [08:37:28] true [08:37:38] what is channel name on wmf irc? [08:38:43] i uh…no idea [08:39:46] it's not #www.wikidata [08:40:27] #incubator.wikimedia|https://incubator.wikimedia.org/w/index.php|incubator_wiki [08:40:41] this is what I need [08:40:42] :D [08:41:06] I suppose #en.wikidata [08:41:08] ? [08:41:11] let's try [08:41:19] nope [08:41:25] wikidata is one language [08:41:29] ooh [08:41:33] wikidatawiki? [08:41:44] or maybe just wikidata.wikimedia [08:41:59] @seen mutante [08:41:59] petan: mutante is in here, right now [08:42:04] mutante ping [08:42:52] #wikidata.wikipedia [08:42:56] petan: ^ [08:50:42] k [08:52:01] sec [08:52:06] @system-rm RC [08:52:06] Unloaded module RC [08:52:20] @feed-off [08:52:36] File not found modules/wmib_feed.bin [08:52:43] Loaded module modules/wmib_rc.bin [08:52:59] @recentchanges+ wikidata [08:53:00] Wiki inserted [08:53:10] legoktm should work now alias is wikidata [08:53:39] awesome, thanks :D [08:58:34] does it work? :D [08:58:46] @RC+ wikidata Wi* [08:58:47] Inserted new item to feed of changes [09:07:51] petan: yeah it did sorry [09:08:04] ok weird I don't see any changes though [09:08:29] @RC+ wikidata Q* [09:08:30] Inserted new item to feed of changes [09:08:31] Change on 12wikidata a page Q1644684 was modified, changed by Sk!dbot link https://www.wikidata.org/w/index.php?diff=3334870 edit summary: /* wbeditentity */ Bot: Item is a disambiguation page. Added description for: ru,fr,en,pt,ca,de,it,nb,fa,ms,fi,id,es, [09:08:34] @RC- wikidata Q* [09:08:35] Deleted item from feed [09:08:37] oh [09:08:39] ok [09:08:43] @RC- wikidata Wi* [09:08:43] Deleted item from feed [09:09:25] these Q* identifiers are not fortunate solution [09:09:31] very hard to remember [09:09:41] why we have numbers instead of human friendly names? [09:10:01] because human friendly names are language dependent [09:10:06] ok [09:10:07] numbers are more universal [09:10:32] also then you end up with Person (actor born March 1972) [10:32:04] @labs-project-users bast [10:32:04] I don't know this project, did you mean: Bastion? I can guarantee there is no such project matching this name unless it has been created less than 18 seconds ago [10:32:21] legoktm now it's more idiot proof :D [10:32:31] @labs-project-info wikidata [10:32:32] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 45 seconds ago [10:32:35] :D [10:32:44] mm [10:32:48] this should have suggest wikidata-dev [10:32:54] @labs-project-users wikidata [10:32:54] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 8 seconds ago [10:33:00] @labs-project-users wikidata-dev [10:33:00] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 14 seconds ago [10:33:06] interesting [10:37:57] @labs-project-users wikidata-dev [10:37:57] Following users are in this project: Aaron Schulz, Ataherivand, Aude, CSteipp, Daniel Kinzler, Demon, Hashar, Jens Ohlig, Ryan Lane, Novaadmin, Reedy, RobLa, Silke Meyer, Tim Starling, Denny Vrandecic, Anja Jentzsch, Andrew Bogott, Jeroen De Dauw, [10:38:45] * DanielK_WMDE blinks [10:38:54] just was fixing [10:43:30] * Damianz pats DanielK_WMDE [10:45:55] megaping [11:06:17] :) petan hehe [11:06:46] but it's fixed :P [11:06:51] now everyone can find your project [11:21:48] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623968 edit summary: [+35] /* Labs wide (not only bots / tools), but available for all projects */ [11:29:05] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623970 edit summary: [+158] /* Web */ formatting / + ref to JIRA [11:33:40] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623972 edit summary: [+259] /* Job-system */ + e.g. stable cron [11:36:59] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623975 edit summary: [+285] /* Web */ more status [11:39:43] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623976 edit summary: [+69] /* Backup */ + including config files [11:41:56] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623977 edit summary: [+49] /* Various */ + mail forwarding [11:48:12] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623981 edit summary: [+57] formatting / make 'done' visible [11:49:33] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623982 edit summary: [+58] + [[../Toolserver features needed in Tool Labs]] [11:53:52] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623984 edit summary: [+51] /* Web */ + server for tools and pages [11:57:22] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by DrTrigon link https://www.mediawiki.org/w/index.php?diff=623985 edit summary: [-37] formatting / make 'done' visible [13:11:26] !log bots ireas: Installed qt4-qmake and libqt4-dev on bots-4. [13:23:25] @labs-user Ireas [13:23:25] Ireas is member of 2 projects: Bastion, Bots, [13:23:35] never heard of that guy [13:23:39] * addshore waves at petan [13:23:47] that's you? :D [13:23:49] hi [13:23:52] no :P [13:24:17] paravoid if you had a moment can you insert me to webtools? [13:27:37] petan, Im guessing if i ftp to bastion, upload something to my home then ssh to bots-4 i should be able to see it? [13:29:12] @labs-user Addshore [13:29:12] Addshore is member of 4 projects: Bastion, Bots, Huggle, Hugglewa, [13:29:59] no [13:30:11] you need to scp to bastion, then scp from bastion to bots-4 [13:30:19] >.< [13:30:20] :< [13:30:20] that sucks a bit, I know [13:30:42] i just have a very weired moment, when i uploaded something to bots-4 and now it appears everywhere :P [13:30:44] we could actually have a public sftp server or something but I don't know if Ryan would like it [13:30:59] home is shared per project [13:31:06] ahhh [13:31:07] so $HOME in bots-1 is same as on bots-nr1 [13:31:13] okay :) [13:31:17] but bastion is a different project [13:31:23] per @labs-info bastion1 :P [13:31:26] makes sense :) [13:31:58] now, how to scp through an scp? [13:35:22] you can tunel a scp [13:35:24] through ssh [13:35:32] @search tunel [13:35:32] No results were found, remember, the bot is searching through content of keys and their names [13:35:36] @search tunnel [13:35:36] Results (Found 1): putty, [13:35:43] well, this sucks [13:36:14] ;p [13:36:21] okay, shame there is no easier way :0 [13:38:56] !tunnel is ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [13:38:56] Key was added [13:39:01] addshore ^ [13:39:31] [= [13:40:07] so you execute this, then you sftp to localhost:6000 and you directly upload to bots-1 [13:41:53] !-f is when you are forwarding some port using ssh you should use parameters -fN which bring ssh on background without spawning any remote shell [13:41:53] Key was added [13:43:09] addshore btw you need to have ssh installed for this :P if you are on windows... [13:43:24] yep i know :P [13:43:39] i have tunnels just using putty before, it will be a bit of a hastle but pah! [13:43:47] or ill go boot up a virtual machine :) [13:43:57] !msys is MSYS is a collection of GNU utilities such as bash, make, gawk and grep for windows http://www.mingw.org/wiki/MSYS [13:43:58] Key was added [13:44:37] addshore or install msys :D [13:44:41] better than virtual [13:44:47] :D [13:44:57] I <3 git in msys [13:45:03] it's like on linux [13:45:52] basically MSYS make your windows machine behave like if it had linux terminal with all common tools [13:49:08] :D [13:49:13] i have always wanted something like that xD [13:53:13] I have C:\BIN on my path and http://unxutils.sourceforge.net/ unpacked there, also nice [14:01:43] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 153 processes [14:16:43] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 147 processes [14:20:19] 01/04/2013 - 14:20:18 - Updating keys for demon at /export/keys/demon [14:42:41] Hello, can anyone help me with saving my SSH key on Labs? [14:53:01] Hello, can anyone help me with saving my SSH key on Labs? [14:53:20] russblau: I can try -- what error message are you running into? [14:53:43] RECOVERY Free ram is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK: 60% free memory [14:54:58] sumanah: "Failed to import keypair." I generated a keypair using Puttygen (following the instructions on [[Help:Putty]] religiously), and pasted into [[Special:NovaKey]], tried several times [14:57:45] russblau hi [14:57:57] you need to upload your public key [14:58:03] that is the one with .pub [14:58:20] !keys [14:58:20] http://bots.wmflabs.org/~petrb/db/ list of infobot keys [14:58:24] eh [14:58:26] !ssh [14:58:26] https://labsconsole.wikimedia.org/wiki/Help:SSH [14:58:27] petan: Yes, that's what I did; copied the entire text of the public key and pasted into the textbox on [[Special:NovaKey]] [14:58:53] is it rsa or dsa? [14:59:24] RSA, 2048 bits [14:59:46] ok that should be ok [14:59:52] what error you get [15:00:00] petan: "Failed to import keypair." [15:00:04] you need to copy paste it as you see it with all symbols [15:01:01] Ryan doesn't make descriptive error messages :P [15:01:45] petan: "symbols"? do id_rsa.pub files typically have stuff other than ASCII in them? :-) [15:01:48] As I said, I copied the entire text from the public key and pasted it into the textbox [15:02:00] sumanah yes [15:02:35] really?! mine is entirely in ASCII [15:04:12] I see that puttygen saves the public key in a different format than it displays it on the screen, so I tried copying and pasting the saved version; [[Special:NovaKey]] now says "The format of the SSH key has been automatically converted. Failed to import keypair." [15:04:48] blargh. [15:12:19] sumanah: FWIW, I was able to upload exactly the same publickey to gerrit without any problems [15:12:34] Sorry russblau I don't know what to tell you! [15:12:59] russblau: I suggest you email labs-l .... Ryan_Lane won't be available to help for a few hours, I think, but he does respond quickly to labs-l posts [15:13:58] !log wikidata-dev wikidata-dev-9 turned off memcached on repo and client again... [15:14:41] hey, bottibot! [15:14:56] Silke_WMDE there is no logbot [15:14:59] morebots [15:15:00] ok [15:15:01] :/ [15:15:05] dunno why [15:15:44] sumanah depends what you consider ASCII - I meant symbols like = petrb@i-0000005a [15:15:54] that all should be there [15:16:07] well, it may not need to be [15:16:08] petan: I am counting everything that's in the ASCII standard. [15:16:20] ok I consider = a symbol :) [15:16:39] ok! I should have been clearer :) [15:17:58] !mail | russblau [15:17:58] russblau: we have a mailing list labs-l@lists.wikimedia.org feel free to send a message there, don't forget to subscribe [15:18:05] that's the mailing list [15:18:44] this seems to be a bug in Ryan's extension [15:18:52] otherwise gerrit wouldn't like it as well [15:23:01] Hi guys! [15:23:23] What happened to the bots-3 instance? I has been down for about five days. [15:24:06] sumanah, petan: Thanks for trying. I've posted to labs-l [15:24:12] thanks russblau [15:24:14] Damianz, any idea? [15:24:23] dschwen it has been rebooted [15:24:30] because it OOM crashed [15:24:33] i cannot log in [15:24:39] let me check [15:24:43] maybe it's OOM again [15:24:46] and my bot on it hasn't been run in 5 days [15:24:52] in fact I would recommend you to use some other instance [15:25:00] bots-3 is loaded [15:25:19] Ok, no prob, but I need to ogin and clear my crontab [15:25:29] I've been using bots-3 for months now [15:25:44] sure but I can't login either, it seems that it's overloaded [15:25:46] so 3 is loaded, 1 is a crap install that should not be used [15:25:53] so ssh daemon doesn't respond quickly [15:25:58] can I use 2? [15:26:18] dschwen I recommend you either -4 in case you need root or bots-nr1 in case you don't [15:26:25] -2 is loaded as well [15:26:35] there is overview [15:26:37] !botsdocs [15:26:37] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [15:26:41] here [15:26:41] oooookay [15:27:11] if you type this you will see all current instances we have :P [15:27:16] @labs-resolve bots [15:27:16] I don't know this instance - aren't you are looking for: I-0000009c (bots-2), I-0000009e (bots-cb), I-000000a9 (bots-1), I-000000af (bots-sql2), I-000000b4 (bots-sql3), I-000000b5 (bots-sql1), I-000000e5 (bots-3), I-000000e8 (bots-4), I-0000015e (bots-labs), I-00000190 (bots-dev), [15:28:05] dschwen bots-4 should be OK if not I can create bots-5 [15:29:46] I don't necessarily need root. My project dir is set up and the bot does not need it [15:29:50] !log bots petrb: restarted adminbot on -labs [15:29:52] Logged the message, Master [15:30:12] dschwen ok then bots-nr1 is likely most stable instance we have :P [15:30:20] because almost no one has root there no one can break it [15:30:33] heh [15:30:33] I still need to clean my crontab on bots-3 [15:30:46] sure, I am checking what's wrong [15:31:00] !log bots petrb: rebooting bots-3 which died for some reason [15:31:01] Logged the message, Master [15:31:31] hmmmrg [15:31:34] I am pretty sure it was OOm again [15:31:38] Damianz [15:31:40] ?? [15:31:46] ? [15:31:52] why hmmmmmremrgrg [15:32:15] was it like hmmgrrrrr or more like mhmmmmm [15:32:18] Wondering why graylog2 hasn't flushed to elasticsaerch and has 888973 items in the queue [15:33:08] btw Damianz is it possible to resize PE size of VG? I believe it is but some other people believe it's not [15:33:09] :D [15:33:25] actually I did it on my own box just to test if it's possible [15:33:39] though I was successfull only changing it to 4M then 8M and back to 4M [15:33:42] online [15:33:53] depends [15:34:03] on HP-UX they say it's not possible [15:34:05] on debian it works [15:34:05] yes - if it's not actually allocted on disk and you're using ext* [15:34:17] though if you use weird filesystems that supports that, sure [15:34:30] I am using btrfs on my own pc it's allocated on disk and mounted as / [15:34:36] and I am still able to online resize it [15:34:44] sounds like sci-fi but I can do that [15:35:00] Tbh, I don't trust the filesystem enough to do it online, ever [15:35:20] I know with ext3 it's fussy because it has to de-journal its self then add the journal back effectivly [15:35:42] I am resizing PE not FS [15:35:51] physical extents [15:36:12] yeah but that's pointless without resizing the fs ontop [15:36:17] of course [15:36:27] but I was wondering if it's even technically possible [15:36:33] HP-UX people say it's not [15:36:35] also - if you downsize to smaller than the fs ontop you break it [15:36:43] pretty sure it is possible though [15:36:45] linux people say something weird [15:37:16] Damianz I resized to bigger, then to smaller and I didn't break anything :D [15:37:27] Linux is pretty insane - like you can inspect a running vms image file.... useful for getting version info/doing file scans as the host. [15:37:30] but btrfs support online shrink [15:38:44] I wonder if I disable fsync how much faster this caching server will go... seems to block the process for a sync far too often :( [15:39:15] Is the webtools project up and running now? [15:39:22] I am still thinking of buying like 100gb of ram and booting from huge ramdisk... [15:39:32] physical drives suck [15:39:39] heh [15:39:44] I like the ssd in my laptop [15:39:49] ssd is slow too [15:39:51] :D [15:39:56] ramdisk ftw [15:39:59] But for servers doing in memory caching syncing to disk is a huge performance killer [15:40:06] yup [15:40:30] Actually - all our virtual nodes boot off the network and have no local disks because they're slow. [15:41:03] RECOVERY Total processes is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 95 processes [15:41:03] RECOVERY Current Users is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: USERS OK - 0 users currently logged in [15:42:13] any admin here that can make me a member of the webtools project? [15:42:33] RECOVERY SSH is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:42:38] I have a webtool currently running on the tollserver, which I would like to port over [15:42:54] RECOVERY Free ram is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: OK: 1415% free memory [15:43:13] RECOVERY Current Load is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: OK - load average: 0.01, 0.03, 0.01 [15:43:23] RECOVERY dpkg-check is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: All packages OK [15:43:27] it is an IIP viewer for large images on commons (in use a lot and needs lots of storage to cache to multiresolution pyramids) [15:43:33] RECOVERY Disk Space is now: OK on bots-3.pmtpa.wmflabs 10.4.0.59 output: DISK OK [15:43:46] the conversion process spikes the load on the toolserver too much [15:44:00] bots-3 is back [15:44:05] it would be a good candidate for an independent instance within webtools [15:44:10] thanks petan [15:47:58] yes it died on OOM [15:49:49] Damianz 1415%? [15:49:53] that ram check is borked [15:50:01] I know [15:50:07] you know why? [15:50:10] nope [15:50:13] meh [15:50:14] I didn't write that one [15:50:19] why we don't use the former one [15:50:20] Not had time to fix it [15:50:27] Because it got deleted [15:50:31] o.o [15:50:37] it was in puppet [15:50:45] Yeah [15:50:48] can instances be moved between projects? [15:50:52] But when someone tried to merge diverged branches [15:50:55] It lost loads of stuff [15:51:00] dschwen in theory yes [15:51:02] So that version is from the prod branch of puppet [15:51:05] But it's broken [15:51:07] dschwen no interface for it so far [15:51:17] then screw the webtools project [15:51:27] I'm creating an instance in my own project [15:51:40] webtools has been stilborn for the past 3 months [15:52:04] an understandable choice [15:52:12] @labs-project-info webtools [15:52:12] The project Webtools has 1 instances and 5 members, description: A project for providing web-based tools to help Wikimedia projects. Tool authors can get access to this project to host their tools. (Note: This is somewhat similar to the original Toolserver) [15:52:47] ok this project need someone to make it live [15:53:23] I was hoping Platonides would be that person [15:53:30] As our company values poster says 'take ownership' tl;dr 'feel free to drive it' [15:53:32] but he has been missing for a little while (I emailed him recently) [15:53:43] this is frustrating as I contacted platonides the day after to project was created [15:53:50] asking to be added and offering help [15:53:53] is giftpflanze interested in taking the driver's seat on the webtools project? [15:54:13] dschwen: did he simply neglect to respond to you? [15:54:25] dschwen: That is frustrating and disappointing [15:54:47] he responed but never added me. He basically just said that he needs time to design it right [15:54:55] how long ago was that? [15:55:05] and would you mind fwding me that mail? sumanah at wikimedia [15:55:09] few days ago someone asked random op to be added to project and they did it [15:55:20] so I don't think it's restricted in any way [15:55:21] happened on IRC [15:55:24] yes [15:55:28] if you have logs that'd be great [15:55:35] me? [15:55:40] this channel is publicly logged [15:55:42] * Damianz orders sumanah a holiday to a forest [15:55:47] of course there is log [15:55:57] dschwen: did you contact Platonides publicly or privately? [15:56:02] if publicly, I can see whether petan [15:56:15] whether petan is right or whether it was one of the days when his log bot died [15:56:15] I don't remember, may have been PM [15:56:37] dschwen: here is my suggestion: email labs-l and ask Ryan to add you to the webtools project so you can run it. No one is running it right now, really, and so why shouldn't you. [15:56:45] I don't recall there was any other day beside that one [15:56:53] of like 30 min outage [15:57:12] I seem to recall more but then again I would not be able to name dates [15:57:42] logging bots are a kind of infrastructure and it only takes 1 failure for a piece of infrastructure to get a hard-to-remove reputation of "unreliable" [15:58:13] ok, let's see [15:58:15] Wikipedia must be very unreliable then [15:58:33] Damianz: what was that whole forest holiday thing about? [15:58:46] You wanted logs, they have logs in forests! [15:58:52] ah. [16:00:23] sumanah if you have any suggestion how to make it more reliable, I am listening :) [16:01:02] if you type /whois wm-bot wm-bot you should see when it connected [16:01:04] petan: I'm assuming that other IRC bots have built-in safeguards against the kinds of failures that you've run into. So look at those [16:01:47] believe me this bot has tons of safe guards, it was just an badly tested release of some binaries [16:01:56] * a [16:02:33] This is freenode, it splits betwean 0 and 500 times a day [16:02:42] irc sorta sucks not being a mesh [16:03:35] it can't be online 100% of times, freenode crashes some times [16:03:36] petan: so then I guess the suggestion would be "test more consistently" .... sounds like you know what the problem was [16:03:53] in fact this bot has typical higher uptime than most of freenode [16:03:58] * typically [16:04:12] separately from uptime is those weird permissions errors [16:04:30] yes that's another thing which I fixed like 2 days ago [16:04:38] now it should be more restricted [16:04:45] more? [16:04:51] so, fewer people can read the logs? [16:04:53] or you mean that http permission errors? [16:04:54] aah [16:05:04] I thought you meant that problem when someone removed all logs [16:05:52] that one is not fixed yet but now I know why it happens, mod_userdir in apache restrict folders to their owners, but bot's folders are owner but its unix user, not mine [16:06:01] so ~petrb is linked to folder which isn't mine [16:06:07] and apache tell you permission denied [16:06:17] I don't have many ideas how to fix it properly [16:06:20] I guess I'm wondering why your bot is running into this kind of stuff and mw-bot never seemed to [16:06:21] just some workarounds [16:06:42] but I don't know enough to speculate [16:07:04] mw-bot was far simpler and it didn't have alias - the proper url is http://bots.wmflabs.org/~wm-bot/logs with this one you will never have troubles [16:07:23] problem is that you were using that ~petrb alias which was kept for historical purposes [16:08:25] petan: then you should change the /topic links that are in all the logged channels [16:08:32] are they? [16:08:36] I thought I did o.o [16:09:10] !htmllogs [16:09:10] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [16:09:27] anyway someone keeps on putting this back to topic, dunno why [16:09:43] !logs [16:09:43] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [16:09:49] o.o [16:09:53] !unalias logs [16:10:51] !unalias logs [16:11:00] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.10.4.80 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [16:11:16] sometimes you need a manual for your own program... :/ [16:11:41] !logs unalias [16:11:41] Alias removed! [16:12:08] !logs is logs bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [16:12:08] Key was added [16:12:16] damn firefox [16:12:19] !logs del [16:12:20] Successfully removed logs [16:12:26] You could ust make a redirect under petrb [16:12:26] !logs is logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [16:12:26] Key was added [16:12:33] also damn why do I keep spelling your name peterb [16:12:33] Damianz how? [16:12:47] you can call me Peter I don't mind [16:12:50] :D [16:12:59] it's the same name just a different language version [16:13:22] oh that's good to know! I keep trying to say "Petr" but others say "Peter" [16:13:47] I don't really care, both is fine :) [16:13:51] rm logs; mkdir logs; echo 'Redirect 301 / http://bots.wmflabs.org/~wm-bot/logs/' > logs/.htaccess # might do it, or something similar [16:14:01] Can never remember the exact apache syntax... <3 nginx [16:14:20] mmm [16:14:25] sounds promising [16:15:44] btw sumanah I was thinking of realiable system to log channels which would consist of multiple bots on multiple freenode servers, but other than being overbotted - it still doesn't solve freenode outages, and there are more freenode outages than outages of my bot, that's for sure [16:15:52] ok [16:16:11] so it wouldn't really help [16:16:19] it will be never reliable :/ [16:16:24] It would be interesting to put irc logs in logstash [16:16:31] what is that [16:16:54] log management thing someone has an instance in labs that all the instances send their syslogs too [16:16:59] only reliable logs would be if IRC server would generate them itself and all servers were merging them into one huge log [16:17:05] the log deletions, the "you don't have permissions to read this directory" AND the bot outages have all contributed to my impression of this logging system as more unreliable than I am used to. But I accept that you're working to improve [16:18:59] bot could at least mark the places in logfiles where there was some outage, either from freenode side or from our [16:19:24] that would likely improve it [16:20:47] Time for newbie questions! :) Is it actually recommended to use the MW install.php script to get a MW? What's the status of that script? [16:21:01] Do you support wildcard domains? such as *.web.wmflabs.org �(all resolving to web.wmflabs.org)?� [16:21:05] I think it works [16:21:13] really the only way would be to have a bot per server clustered in a way where each one ack'd the message so when it split you'd get all emssages, in the right order [16:21:13] dschwen yes [16:21:19] good [16:21:40] Damianz that would be like 20 bots in a channel just to log it :P [16:21:46] yep [16:22:03] or you write a server and connect it to each host as a uline or w/e it's called [16:22:04] modern irc server support bot flag [16:22:07] then it sees all traffic [16:22:14] so you can just set up your client to hide them from user list [16:22:22] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [16:22:51] yes it's uline, but in fact, loading a plugin directly to irc server would be better [16:22:56] because uline can disconnect just as bot [16:23:08] yeah but that wouldn't solve split servers with convo [16:23:19] you would need to somehow merge them [16:23:20] Really the best would be to re-design the linking protocol as a tru mesh [16:23:32] that would be cool too [16:23:53] but slower [16:24:46] Hmmm I don't think adding acks would be too slow [16:24:49] * sumanah listens to the sound of yakshaving [16:25:05] you see sumanah we respect your desire :D but it would take a whole USA budget to implement a solution that fits all your needs :P [16:26:07] I doubt it. [16:26:18] that was some kind of joke [16:26:57] * sumanah smiles [16:27:03] Hey, wmf gets like 25million/year - they can totally fund opensource development :P [16:27:10] :D [16:59:52] Ryan_Lane: Did memcache break again on lc? [17:05:09] Damianz: it segfaults, then puppet restarts it [17:05:16] I have the backtrace this time, though [17:05:20] lovely [17:06:03] I'm assuming the key chain failure was you changing the config for the thing we talked about the other day, and the failure to login is memcache falling on its ass (mailing list last email) [17:06:42] it shouldn't be [17:07:13] hm [17:07:14] maybe it is [17:07:20] I can't import keys either [17:08:02] I havn't tried since I cracked my phone screen and need to find my paper tokens to setup my work phone for 2fa [17:08:47] well fuck. this backtrace is garbage [17:09:17] no symbols, of course [17:11:05] heh [17:11:20] wait, no [17:11:24] key import works for me [17:11:47] From what I can see that error is basically ldap_modify failed, then if it succeeded it deletes stuff from memcache [17:11:47] of course, I've never actually tried the putty key import [17:12:05] where do you see this? [17:12:21] oh [17:12:23] no idea, I closed that window ages ago :P [17:12:23] in the code [17:12:26] yeah [17:12:42] might be old code though heh, gerrit is too slow and my branch is a little old [17:12:46] well, it's a possibility that the putty key conversion is coming back with an empty key [17:12:47] s/branch/clone/ [17:15:21] 01/04/2013 - 17:15:21 - Updating keys for laner at /export/keys/laner [17:15:40] hmm, good point [17:16:06] though he said he's using the ssh-rsa version, which is the right one and shouldn't need converting [17:16:52] oh [17:16:57] I wonder if this is a brand new user [17:17:12] [04/Jan/2013:14:35:30 +0000] MODIFY REQ conn=22871 op=2 msgID=4 dn="" [17:17:19] [04/Jan/2013:14:35:30 +0000] MODIFY RES conn=22871 op=2 msgID=4 result=50 message="The entry cannot be modified due to insufficient access rights" etime=3 [17:18:05] I disabled the nova credentials check for ssh key uploads [17:18:24] but maybe lots of things are screwed up on initial login [17:19:03] yeah - I suggested logging out and in, but them memcache fell over [17:19:04] :D [17:19:08] yay new user experience [17:19:15] heh [17:20:08] well, this would be an instance of an already known bug [17:20:18] no god damn clue what's wrong with memcache [17:20:35] maybe I should switch to redis [17:21:50] yay redis [17:21:55] though I wish they would bring clustering out [17:30:14] petan: whats up [17:34:33] The sky [17:39:38] It makes me sad to say it, but God is awesome [17:40:02] Screws up less than supervisord anyway and is easier to configure condition based stuff per process [17:55:36] ping andrewbogott [17:57:08] howdy [17:57:27] :) I'm having problems with the role/labsmediawiki.pp file where I enter role_config_lines. [17:57:34] labs-morebots, you still there? [17:57:35] I am a logbot running on i-0000015e. [17:57:35] Messages are logged to labsconsole.wikimedia.org/wiki/Server_Admin_Log. [17:57:35] To log a message, type !log . [17:57:46] Was there a problem with the logbot earlier? [17:58:11] when I use a variable in role_config_lines it is not read. [17:58:27] Silke_WMDE: OK, let me get that code in front of me... [18:00:21] For $wgLBFactoryConf for example, I have things like user, password etc. This only works without variables. Which would be ok for a default user, but passwords... [18:01:09] Can you upload a paste or submit a patch or something so I can see what you're doing? [18:01:31] When you say 'variables' you mean you want a template variable substitution from a variable defined in a puppet manifest? [18:02:33] I want the role file to use things like $wgDBuser [18:05:02] Silke_WMDE: I just glanced at this so far, but it looks like you are single-quoting a lot of your includes. Puppet will not do variable subs within single quotes. [18:05:17] So instead of '"stringwithquotes"' you'd want "\"stringwithquotes\"" [18:05:43] does that explain what you're seeing? [18:05:48] oh [18:06:07] here is what I am working on: http://pastebin.com/w70gWdMH the load balancer foo [18:06:54] with the password explicitly written down, it works. :p [18:08:28] Your paste has the explicit password, right? Can you show me a paste with the var instead? [18:09:16] it said \'password\' => $wgDBpassword, [18:10:20] 01/04/2013 - 18:10:19 - Updating keys for platonides at /export/keys/platonides [18:10:44] same for $wgDBuser and $wgDBname [18:11:46] OK, but the whole line is double-quoted right? [18:11:52] Um… single-quoted I mean. [18:11:57] yes [18:12:04] So I stand by my conjecture that that is the problem :) [18:12:40] so you would go "$wgLBFactoryConf = array( blablah $wgDBname blah )" [18:12:40] But of course you have other variable names in there like $wgLBFactoryConf which you /don't/ want substituted... [18:12:57] Just a second, lemme paste. [18:14:16] well… wait, I'm wrong and/or confused. [18:14:31] You don't want wgDBpassword to get substituted by puppet, you want it substituted by mediawiki [18:14:38] right? [18:14:57] probably [18:15:06] Where is wgDBpassword defined? [18:15:16] * Damianz high fives andrewbogott then goes to burn his face off with chilli [18:16:02] i made puppet create one in mediawiki.pp, then it lands in /srv/mediawiki/orig/LocalSettings.php [18:16:29] * Silke_WMDE didn't like the no password solution [18:16:40] OK. So in LocalSettings.php you have a line up top like $wgDBpassword = "somerealpassword" [18:16:46] yes [18:16:53] And then elsewhere in LocalSettings.php you want to refer to $wgDBpassword as a variable. [18:17:16] in /srv/mediawiki/LocalSettings.php, yes [18:17:32] (which includes the other one) [18:17:39] And the part about $wgDBpassword = "somerealpassword" is getting into LocalSettings.php correctly? [18:17:47] yes [18:18:11] a different example: [18:18:31] for the dbuser name and db name I had variables [18:19:02] but in the end, my wiki tried to access the db "my_wiki" with user "wikiuser" [18:19:24] which are default, not at all, what we were using all the time ("testwiki" and "root") [18:19:27] Can you paste your whole localsettings.php for me? [18:19:42] the generated output? wait... [18:20:09] yep, the generated file. [18:21:38] '1+1=2 only in legacy system. In modern distributed database with eventual consistent is 1+1=1. ' < lol [18:24:15] andrewbogott: number one /srv/mediawiki/LocalSettings.php http://pastebin.com/CcpHfcHX [18:24:30] Hi, what is the procedure for getting added to the bots project? [18:24:58] hi danmichaelo - you can ask on labs-l or you can personally contact one of the people who runs it [18:25:00] Silke_WMDE: OK, but wgDBpassword doesn't appear in that :( [18:25:19] Platonides: ping re danmichaelo's question :-) also, how are you? haven't seen you in a bit [18:25:46] andrewbogott: thats in the other one that just triggered the pastebins spam protection... [18:26:02] ok, here we go: http://pastebin.com/HqqzpqrY [18:26:37] I feel spammy about sending an email to a public mailing list just to get myself added to a project [18:27:18] Tried contacting project owners a couple of times on IRC, but I haven't been successful [18:27:32] andrewbogott: Does it matter /where/ I add the line require_once( 'orig/LocalSettings.php' );? [18:27:57] so I'm a bit confused, but I guess it's because the project is new and everything is not sorted out yet [18:28:45] Silke_WMDE: Sorry, still behind… those are two different files generated by the same puppet run? And one includes the other? [18:29:12] :D That was your idea. [18:29:28] Yep, I know. Just, on account of pastebin I can't tell the names of the files [18:29:34] Hence can't follow the inclusion step. [18:30:31] The second paste is orig/LocalSettings.php, and the first is the puppetized LocalSettings.php, right? [18:30:36] yes [18:31:19] Damianz: I understand your worry but please know it's NOT spammy, that's how labs-l is right now [18:31:19] it *is* confusing [18:31:40] sumanah: Which worry? [18:31:49] sorry I meant danmichaelo [18:32:02] * Damianz holds danmichaelo by the scruff of the neck and points [18:32:02] Silke_WMDE, so it looks to me like orig/LocalSettings.php which defines the variable is being included after LocalSettings.php which refers to the variable. Do I have that right? [18:32:19] Eventually there will be a request queue for everything, but yes it's not spam for now [18:33:01] Silke_WMDE: Sorry, what I just wrote was very confusing. But the essence is: looks like the var is defined after the reference. [18:33:13] Do you agree? [18:33:33] danmichaelo: you have already dev access? [18:33:47] andrewbogott: yes, true [18:34:09] I'll try and change the order [18:34:12] yup, ok, I'll try labs-l :) [18:34:19] So /probably/ it won't break anything for us to include the orig/LocalSettings.php at the top of the puppet file. [18:34:34] Give that a try, and if it doesn't break anything then subit a patch with just that one change. [18:45:20] 01/04/2013 - 18:45:20 - Updating keys for russblau at /export/keys/russblau [18:47:01] yay, more people interested in webtools [18:47:47] +1 [18:48:20] Shame you need +2 to do anything +1 is just like going for the ride [18:48:28] +2 [18:48:29] :D [18:48:43] benestar: you have the rights to add people to bots project i think [18:48:57] really? [18:49:03] nice :) [18:49:14] see special:novaproject and try [18:49:21] if you're a member you can add members [18:49:31] * Damianz afk [18:49:31] danmichaelo: hi [18:49:50] giftpflanze: just add him to the list? [18:50:02] yes [18:51:12] oi, i'm admin on webtools :o [18:51:57] i am eating now [18:52:11] danmichaelo can ping me if i am back [18:53:21] !log bots gifti: test [18:53:23] Logged the message, Master [18:53:29] aha [18:53:38] so, that works [18:54:25] !log bots test [18:54:26] Logged the message, Master [18:54:42] is labs-morebots broken? [18:55:44] it looks like it's logging [18:55:51] no, it isn't [18:55:56] no? [18:56:00] it says it is [18:56:31] and it's logging to the resource page [18:56:37] it has bot flag … [18:59:03] ireas reported it not to work earlier [19:01:49] benstar: bon appétit, I'm here, but I might be away for short periods of time [19:02:15] benestar: it is [19:09:37] andrewbogott: looks good [19:09:47] great! [19:10:16] andrewbogott: I would also moved the script path further up [19:10:34] sounds reasonable [19:10:48] ok [19:13:29] andrewbogott: submitted to review [19:32:44] hi - in order to be added to a project it seems like I need to have the "loginviashell" right. How do I get it? [19:33:13] https://labsconsole.wikimedia.org/wiki/Special:FormEdit/Shell_Access_Request [19:33:20] skalman12: Usually you would make a request on the web interface, but I can do it right now. [19:33:31] skalman12, what's your username on labsconsole? [19:33:36] Skalman [19:34:04] ok, done. [19:34:15] andrewbogott: I want to try one other thing that I absolutely need: a customizable database name. I can't connect a Wikidata repo and client with the same database names. I gave it a first try today but that was rather a desaster. [19:34:26] thanks :) that was quick :) [19:34:43] Silke_WMDE: I don't immediately know how to do that, but it should be possible. [19:35:09] I will play a bit and get back to you [19:40:20] 01/04/2013 - 19:40:19 - Updating keys for skalman at /export/keys/skalman [19:41:51] Yossie: https://labsconsole.wikimedia.org/wiki/Special:FormEdit/Shell_Access_Request [19:44:32] PROBLEM Current Users is now: WARNING on wikidata-puppet-client2.pmtpa.wmflabs 10.4.1.14 output: USERS WARNING - 6 users currently logged in [19:51:40] andrewbogott: if you aren't swamped right now, could you take this bug? https://bugzilla.wikimedia.org/show_bug.cgi?id=43370 [19:52:26] hm. I guess shell justification would be another form on the login page, that'll be annoying. heh [19:52:38] (annoying to do) [19:54:07] does justification matter? [19:54:37] I mean -- we're trying to prevent sockpuppets and such, but having a 'why do you want shell' field doesn't much help with that. [19:54:47] Yossie: there you go.. shell justification is the topic right now anyways:) [19:55:50] Ryan, if I need a package like exuberant-ctags on the bots project, should I ask you or some other project owner to install it, or install it to my home dir myself? [19:56:28] andrewbogott: true [19:56:38] andrewbogott: can likely do away with justification [19:56:57] I'll skip it for now, we can add later if it turns out to matter somehow. [19:57:02] we likely only need to worry much about shell access if we have a user that's being a problem [19:57:32] danmichaelo: please request it from the bots owners [19:57:39] danmichaelo: we're trying to track dependencies [19:58:02] thanks for adding me to webtools! [19:58:13] I cannto create instances, can I? [19:59:09] do I have to be in the sysadmin or netadmin groups? [19:59:17] that would be useful [20:04:27] Ryan_lane: Ok, thx, but you are a project owner, no? [20:07:29] danmichaelo: technically since I have sysadmin/netadmin, yes [20:07:34] giftpflanze, thanks! I just sent another mail to the list, to generate a papertrail. [20:07:38] but I only have those to help with labs problems [20:07:48] dschwen: oh, you don't have those roles? [20:07:59] I don't see them [20:08:03] one sec [20:08:13] oh lord :P [20:08:38] dschwen: done [20:08:41] thanks [20:08:58] danmichaelo: I can install the package and document it [20:09:09] danmichaelo: which instance do you need it installed on? [20:10:11] awesome! btw. sorry for being fussy, but I'm quite excited about labs, and would like to get things working [20:10:17] indeed [20:10:23] I generally try to stay out of the management of projects ;) [20:10:30] dschwen: it did not do that [20:10:43] one of the boots-instances, boots-1 perhaps [20:10:47] it? [20:11:02] oh, got it [20:11:06] nevermind then [20:11:08] hm. I think people aren't supposed to use bots-1 [20:11:14] :D [20:11:33] The email sent by 'you' to the mailing list, saying that 'you' added me, confused me ;-) [20:11:54] @search bots [20:11:54] Results (Found 10): keys, bots, cs, db, bot, search, htmllogs, home, tunnel, logs, [20:12:00] so bots-2? [20:12:00] !bots [20:12:00] http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_a_bot_running_infrastructure proposal for bots [20:12:08] -_- [20:12:15] @labs-info [20:12:22] @labs-info bots-1 [20:12:23] [Name bots-1 doesn't exist but resolves to I-000000a9] I-000000a9 is Nova Instance with name: bots-1, host: virt8, IP: 10.4.0.48 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: bots, size of storage: 30 and with image ID: lucid-server-cloudimg-amd64.img [20:12:32] heh [20:12:33] i answered to danmichaelo [20:12:50] (this is why I avoid managing projects) [20:12:51] getting started with toolserver was much simpler, but yeah, labs is cool ;) [20:12:58] I honestly don't know which instance you should use [20:13:02] petan and Damianz would [20:13:32] k [20:13:43] i suggest bots-4 [20:13:47] I know this is documented somewhere [20:14:07] sure. use 4 for now [20:14:07] bots-[1-3] are … [20:14:23] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots#Servers [20:15:34] ah, thanks [20:15:59] I just install exuberant-ctags on it [20:16:22] I'm off to lunch. will be back soon. let me know if you need anything else [20:16:32] what's your bot name? [20:16:52] you should edit the doc I just linked to add your bot to the list of bots on bots-4 [20:16:55] ok, thanks! I have several working on Norwegian Wikipedia [20:17:18] !log bots installed exuberant-ctags on bots-4 for danmichaelo [20:17:19] Logged the message, Master [20:17:23] UKBot is the first I plan to move, updating results for a weekly contest [20:17:47] danmichaelo: when you make changes on bots-4 you should update the server admin log, using !log [20:17:51] @log [20:17:55] !logging [20:17:56] To log a message, use the following format: !log [20:18:06] what kind of changes? [20:18:19] well, if you add a bot and start running it, for instance [20:18:23] k [20:18:27] ok. back in a little bit [20:18:32] now i get it :D [20:23:11] hi [20:23:18] Ryan_Lane 2 questions [20:23:25] can you insert me to webtools? [20:23:31] second one I already forgot [20:28:43] PROBLEM Current Load is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: WARNING - load average: 6.14, 6.15, 5.49 [20:28:53] PROBLEM Current Load is now: CRITICAL on node.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [20:29:33] PROBLEM Current Users is now: CRITICAL on node.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [20:30:12] PROBLEM Disk Space is now: CRITICAL on node.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [20:30:52] PROBLEM Free ram is now: CRITICAL on node.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [20:32:22] PROBLEM Total processes is now: CRITICAL on node.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [20:33:12] PROBLEM dpkg-check is now: CRITICAL on node.pmtpa.wmflabs 10.4.1.63 output: Connection refused by host [20:35:52] RECOVERY Free ram is now: OK on node.pmtpa.wmflabs 10.4.1.63 output: OK: 277% free memory [20:37:22] RECOVERY Total processes is now: OK on node.pmtpa.wmflabs 10.4.1.63 output: PROCS OK: 84 processes [20:38:12] RECOVERY dpkg-check is now: OK on node.pmtpa.wmflabs 10.4.1.63 output: All packages OK [20:38:52] RECOVERY Current Load is now: OK on node.pmtpa.wmflabs 10.4.1.63 output: OK - load average: 0.08, 0.70, 0.58 [20:39:32] RECOVERY Current Users is now: OK on node.pmtpa.wmflabs 10.4.1.63 output: USERS OK - 0 users currently logged in [20:39:50] Naaarrghh, I'm trying to edit the project description for webtools, but after saving it is all gone and just says "No description" [20:40:12] RECOVERY Disk Space is now: OK on node.pmtpa.wmflabs 10.4.1.63 output: DISK OK [20:47:41] Aha, bug in labsconsole? If you have external links in the description and answer the captcha (right or wrong, does not matter) an empty page is saved. (Ryan-lane) [20:47:55] that was not a quote [20:48:06] but meant for ryan [20:50:40] hm, works if I edit here: https://labsconsole.wikimedia.org/w/index.php?title=Nova_Resource:Webtools/Documentation&action=edit [20:51:01] does not work here: https://labsconsole.wikimedia.org/wiki/Special:FormEdit/Nova_Project_Documentation/Nova_Resource:Webtools/Documentation [20:52:22] Ryan_Lane, what's going on i-00000563 / webtools-bastion ? [20:52:27] seems creation failed [20:54:32] RECOVERY Current Users is now: OK on wikidata-puppet-client2.pmtpa.wmflabs 10.4.1.14 output: USERS OK - 1 users currently logged in [21:04:39] Platonides can you insert me to webtools [21:12:02] Platonides are you around? [21:16:13] Platonides: is this a new instance? [21:16:39] ah, yep [21:16:46] let me check the error message [21:17:33] is platonides invisible or something? I don't see anyone except Ryan talking o.o [21:17:46] don't think so [21:18:30] ok when one of you have time can you insert me to webtools [21:20:23] @labs-info node [21:20:23] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 33 seconds ago [21:20:31] <^demon> Ryan_Lane: Where's the docs to turn an old ldap user into a labs user? [21:20:40] @labs-resolve node [21:20:40] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 50 seconds ago [21:20:51] ^demon: oh [21:20:59] ^demon: can you fix fran's account? [21:21:01] in gerrit [21:21:07] * sumanah also can't see Platonides  [21:21:09] I couldn't figure out how [21:21:15] that is, I can't see his lines, if he is speaking [21:21:24] Platonides.visable = True [21:21:27] i can [21:22:24] <^demon> Ryan_Lane: In gerrit, "Fran", uid = 381? [21:22:30] Also ^demon deserves cookies, gerrit is a pile of shit to deal with [21:22:32] yes [21:23:36] <^demon> Done. [21:23:41] <^demon> update account_external_ids set external_id = "username:Fran McCrory" where external_id = "username:fran"; [21:23:45] <^demon> fyi. [21:23:53] oh [21:23:55] is that how it works? [21:24:00] <^demon> account_external_ids holds the ldap links. [21:24:01] <^demon> Yep. [21:24:09] what about gerrit:? [21:24:42] <^demon> That's the sn, set on login. [21:24:52] <^demon> As is accounts.full_name [21:25:12] do they generally both need to be changed? [21:25:13] <^demon> account_external_ids holds a couple of things actually. Kind of fun ;-) [21:25:19] or just username? [21:25:32] <^demon> *Should* be able to just logout/login now. [21:25:37] <^demon> If not, we can play with it further. [21:26:18] Platonides: seems scheduling failed for the instance… logs don't really say way [21:26:30] nova's scheduler logs are terrible :( [21:26:39] s/logs// [21:26:46] yes, that too [21:26:53] we use a stupid scheduler, though [21:27:16] ah. found it [21:27:28] 2013-01-04 18:35:03 WARNING nova.scheduler.manager [req-f193002d-755b-47e0-8249-43cff3439d8f platonides webtools] Failed to schedule_run_instance: No valid host was found. Is the appropriate service running? [21:27:51] I'm betting all of the nova-compute services were listed as down due to periodic tasks [21:28:03] it's a race condition [21:28:09] an annoying one [21:28:15] it's pretty rare, thankfully [21:28:46] periodic tasks are being moved out of the service in grizzly, which will hopefully make this problem go away [21:29:21] Platonides: delete/recreate [21:32:41] petan: let me give you access [21:32:47] sorry was troubleshooting something [21:33:16] <^demon> Ryan_Lane: So, docs on converting a user? [21:33:24] oh [21:33:36] ^demon: on formey: add-labs-user —help [21:33:44] well [21:33:45] via sudo [21:33:57] <^demon> Ah, got it. [21:34:19] can we convert a percentage of users to klingons? [21:40:10] petan: added you [21:40:32] <^demon> Damianz: We don't do Klingon anymore. [21:40:40] sadtimes [22:05:43] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [22:33:42] RECOVERY Current Load is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: OK - load average: 4.03, 4.25, 4.93 [22:41:35] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=624219 edit summary: [+122] /* Labs wide (not only bots / tools), but available for all projects */ [22:44:06] I swaer that whole page is [22:45:20] hahaha [22:45:37] I really just want a requirements list [22:47:23] really I want a table with ts feature, equivilent labs feature, requires new feature (y/n), comments [22:48:50] though on the subject of munin [22:48:54] ganglia seems broken [22:49:02] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=624225 edit summary: [+592] /* Labs wide (not only bots / tools), but available for all projects */ [22:49:05] dunno who owns that now left mwf a while back [22:49:13] ganglia is? [22:49:15] lemme check [22:49:27] http://ganglia.wmflabs.org/ is not responding anyway [22:49:31] dunno if the aggregator is up [22:49:39] hm [22:50:39] are we done with this? #173: Replicate cluster environment in virt infrastructure for training, testing and planning [22:51:01] that instance won't let me in [22:51:04] beta sorta covers that I guess [22:51:10] it's immediately killing my ssh connection [22:51:13] mutante: yes [22:51:19] Damianz: yea, just not sure about the "sort of" part [22:51:22] beta is that [22:51:24] alright :) [22:51:57] re last edit, I think that goes against wikipedia's principle and is insanity [22:52:05] what does? [22:52:16] my stance, or his? [22:52:30] his [22:52:37] the founding principle is openness [22:52:43] indeed [22:52:44] not limited to if you're nice to people with the info [22:53:35] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=624227 edit summary: [+127] /* Labs wide (not only bots / tools), but available for all projects */ [22:53:48] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=624228 edit summary: [+89] /* Labs wide (not only bots / tools), but available for all projects */ [22:54:16] yep. we limit ourselves to open source software as much as possible in our cluster [22:54:34] the only think I can think ops uses that's closed source is the netapp [22:55:38] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=624230 edit summary: [+2] [22:56:05] Some of our network guys are against opensource, so I have to remind them how stuff would be different if totally propiatry heh [22:56:57] I can't imagine using closed source software [22:57:14] the support I get with open source is so, so much better than I've ever gotten with closed source [22:57:19] well [22:57:21] and Google [22:57:33] paravoid: google? [22:57:44] we use proprietary software for all of our emails, calendar and meetings [22:57:48] ah [22:57:49] right [22:58:05] we use a number of proprietary things on the office management side [22:58:17] I use os x as my desktop, too [22:58:27] I was talking about in the cluster, though [22:58:28] true [22:58:46] 9 non-free packages, 0.5% of 1964 installed packages. [22:58:51] that's my desktop [22:58:58] firmware, skype and sun java basically [22:59:02] and I rarely use skype :) [22:59:16] I have some non-free stuff on my laptop (though that's mostly encoder stuff for dvds etc heh) [22:59:23] RECOVERY SSH is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:59:24] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [23:00:04] Comercial support is expensive and quite often awful in my experience.... like bugs that would just get fixed in opensource are 'features' [23:01:32] RECOVERY Disk Space is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: DISK OK [23:02:22] RECOVERY Current Load is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK - load average: 0.04, 0.09, 0.05 [23:02:22] RECOVERY Total processes is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: PROCS OK: 236 processes [23:03:02] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 93% free memory [23:03:32] RECOVERY Current Users is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: USERS OK - 0 users currently logged in [23:08:15] petan, I did read you... :S [23:08:47] not adding more people today, though [23:08:49] good night [23:21:12] Ryan_Lane: nova-precise2 is refusing to let me log in (or even change my password). Can you suggest where to look to debug this? Log files, channels to turn on, etc? [23:21:58] Anyone in here have an opinion on cameras? [23:35:02] andrewbogott: the instance itself isn;t? [23:35:17] Ryan_Lane: No, mediawiki. [23:35:21] ah [23:35:27] do you have an account made in ldap? [23:35:50] Actually I have an idea what this might be -- back in 5. [23:50:51] Ryan_Lane, looks like it's finally fixed. Thanks. :) [23:50:52] RECOVERY dpkg-check is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: All packages OK [23:51:02] RECOVERY host: aggregator1.pmtpa.wmflabs is UP address: 10.4.0.79 PING OK - Packet loss = 0%, RTA = 0.73 ms [23:51:02] RECOVERY Disk Space is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: DISK OK [23:51:08] franny: great! ^demon fixed it [23:51:19] he showed me what he did as well, which is good [23:51:22] RECOVERY SSH is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:51:31] Looks like my SSH keys disappeared in the process. [23:51:41] Though I can paste them back in. [23:51:52] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 975% free memory [23:52:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 7.84, 7.54, 5.75 [23:52:52] RECOVERY Current Users is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: USERS OK - 0 users currently logged in [23:54:32] RECOVERY Current Load is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK - load average: 0.15, 0.10, 0.03 [23:59:32] Damianz: ganglia is back up [23:59:34] it needed to be rebooted [23:59:49] rebooting never solves issues :P [23:59:59] damn aggregator was down too so we lost stats