[01:27:58] andrewbogott_afk: Resting, but I'm at hand for emergencies. Whazza? [02:39:34] Change on 12mediawiki a page Wikimedia Labs was modified, changed by 64.40.54.180 link https://www.mediawiki.org/w/index.php?diff=685997 edit summary: [+35] /* Roadmap */ current roadmap [07:46:57] Coren do we need /dev/vdb for anything on -login? [07:47:11] I am wondering if we couldn't use it for something... [07:48:03] maybe we could make /tmp from it, so that there would be lower fragmentation of system fs and no risc that user accidentally fill in rootfs [07:48:18] mhm... [07:48:28] I will try that on -dev [08:00:01] !log tools petrb: making lvm from unused disk from /mnt on -login so that we can eventually use it somewhere if needed [08:00:03] Logged the message, Master [08:04:00] !log tools petrb: created a small swap on -login so that users can not bring it to OOM so easily and so that unused memory blocks can be swapined in order to use the remaining memory more effectively [08:04:01] Logged the message, Master [09:24:32] @notify binasher [09:24:32] You already requested this user to be watched [09:24:37] meh [09:24:55] addshore ping [09:31:20] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 48127] beta: text squid conf should be in puppet - https://bugzilla.wikimedia.org/show_bug.cgi?id=48127 [09:36:35] !log deployment-prep Adding ArielGlenn as a member/sysadmin [09:36:37] Logged the message, Master [10:22:03] Warning: There is 1 user waiting for shell: FFS2309 (waiting 0 minutes) [10:35:33] Warning: There is 1 user waiting for shell: FFS2309 (waiting 13 minutes) [10:40:52] [bz] (NEW - created by: Riley Huntley, priority: Unprioritized - normal) [Bug 47808] internal_api_error when running bot on en.wikipedia.beta.wmflabs.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=47808 [10:49:06] Warning: There is 1 user waiting for shell: FFS2309 (waiting 27 minutes) [11:02:36] Warning: There is 1 user waiting for shell: FFS2309 (waiting 40 minutes) [11:16:11] Warning: There is 1 user waiting for shell: FFS2309 (waiting 54 minutes) [11:29:41] Warning: There is 1 user waiting for shell: FFS2309 (waiting 67 minutes) [11:43:03] Warning: There is 1 user waiting for shell: FFS2309 (waiting 81 minutes) [11:56:08] 500 Internal Server Error? [11:56:29] Warning: There is 1 user waiting for shell: FFS2309 (waiting 94 minutes) [11:56:47] You are admin identified by name .*@wikipedia/Cyberpower678 [11:57:26] petan, Coren: I'm getting a 500 error [11:57:35] Cyberpower678 where [11:57:48] http://tools.wmflabs.org/cyberbot/onetimebots/chartarchive.php [11:58:06] It's supposed to intitialize the bot script. [11:58:56] there is problem with UID [11:58:59] of that file [11:59:07] Oh right. -.- [11:59:12] it needs to be owned by a bot [11:59:32] it's owned by you [11:59:42] do u want me to fix it? [12:03:55] Yes. [12:03:58] pleaze [12:04:43] done [12:05:25] Still 500ing [12:05:32] there are more errors with other files [12:05:42] like what? [12:05:42] do u want me to change ownership of all files in that folder? [12:05:48] Sure. [12:05:54] ok [12:05:58] Would I be able to do it myself? [12:06:03] with sudo [12:06:17] I don't know... [12:06:29] you use sudo right? [12:06:38] yes, anyway it's done [12:07:07] It seems to be doing something now. [12:09:55] Warning: There is 1 user waiting for shell: FFS2309 (waiting 108 minutes) [12:10:13] petan, I want all of my single use scripts to be accessible from Wikipedia. So I made a run link that will run it once. [12:16:07] aha [12:23:21] Warning: There is 1 user waiting for shell: FFS2309 (waiting 121 minutes) [12:36:47] Warning: There is 1 user waiting for shell: FFS2309 (waiting 134 minutes) [12:37:38] petan: Yeah, swap on -login isn't insane. :-) [12:39:31] we could as well create a separate /tmp fs, but it's not really important... depends on how much is it going to be used... I was just wondering how we could utilize that 20gb vdb [12:50:09] Warning: There is 1 user waiting for shell: FFS2309 (waiting 148 minutes) [15:15:32] hey Coren how are you today? [15:15:50] Well. [15:16:21] Elbow-deep in the dbs testing interaction between deletion, revdel, os and images. "Fun" [15:31:00] https://wikitech.wikimedia.org/wiki/Flask-stub might interest you [16:18:12] !log puppet-cleanup added Matanya [16:18:14] Logged the message, Master [16:18:28] thanks :) [16:34:21] !log Wikivoyage Set $wgServer to "http://voy-en.instance-proxy.wmflabs.org" in /var/www/srv/mediawiki/orig/LocalSettings.php [16:34:21] Wikivoyage is not a valid project. [16:34:35] !log wikivoyage Set $wgServer to "http://voy-en.instance-proxy.wmflabs.org" in /var/www/srv/mediawiki/orig/LocalSettings.php [16:34:36] Logged the message, Master [16:35:10] Thehelpfulone: Seems to be working now. [16:36:43] ah nice [16:41:41] Ryan_Lane, is 'The user account was not created, as we could not confirm its source' a well-defined failure, or is it just a generic account-creation-failed message? [16:42:00] I don't believe I've ever seen that error [16:42:34] :( [16:43:11] that message is nocookiesfornew [16:43:29] defined in SpecialUserlogin.php [16:43:42] it's a CSRF protection check [16:43:50] are a lot of people getting this error? [16:44:00] Nope, I'm just getting it on a puppetized install. [16:44:03] ah [16:44:13] how are you creating the account? [16:44:35] With the GUI, here: https://openstack-role-dev4.pmtpa.wmflabs/w/index.php?title=Special:UserLogin&returnto=Main+Page&returntoquery=type%3Dsignup&type=signup [16:44:57] It's very likely that the install is messed up, I was just hoping you would know specifics. [16:45:32] [bz] (NEW - created by: Nemo, priority: Unprioritized - normal) [Bug 48162] Beta main page has many links to 404'ing config files - https://bugzilla.wikimedia.org/show_bug.cgi?id=48162 [16:45:51] I would've expected ldap to be broken, but seems we aren't getting that far [16:46:25] LDAP? [16:46:37] it's not using ldap, right? [16:46:39] oh [16:46:46] right. this is for labsconsole [16:46:59] yeah, I'm expecting that to be the next broken thing [16:47:16] proxy is refusong connections? [16:47:21] oh [16:47:22] heh [16:47:43] I thought this was using instance proxy for some reason :D [16:48:32] is memcache running and configured properly? [16:48:53] check the port [16:49:05] I think we use the stupid non-standard port in production [16:49:08] I want to fix that [16:50:38] port looks right -- 11000 in both places [16:50:48] using 127.0.0.1? [16:50:53] yeah [16:51:20] It might be a legit cookie problem, actually, since I reuse hostnames sometimes. Trying to figure out how to selectively clear my cookies... [16:51:57] no, cause I'm getting the same issue [16:52:02] oh, ok. [16:52:02] it's because of the edittoken [16:52:13] it looks to me like I'm getting an empty edittoken [16:52:44] what's the site url set to? [16:53:39] $wgServer? [16:53:43] yeah [16:53:53] $wgServer = "//" . $_SERVER["SERVER_NAME"]; <- copied from nova-precise2 [16:54:04] comment that line out [16:54:23] though I couldn't see that being an issue [16:54:23] Um… things will be quite broken if I do that, I know from experience... [16:54:46] I seem to remember writing that line [16:54:51] Not sure if I put it on nova-precise2 though [16:54:55] * Ryan_Lane nods [16:54:56] Ryan_Lane, as you see, no static content [16:55:01] right [16:55:03] due to confusion with http vs https [16:55:26] are cookies being sent as secure? [16:55:59] $wgCookieSecure = true; [16:56:51] yep [16:57:23] use: $wgServer = "https://" . $_SERVER["SERVER_NAME"]; [16:58:56] same behavior, it seems [16:59:14] hm [16:59:36] ask in #mediawiki why your edit tokens may be broken [16:59:39] I can't think of a reason [17:00:25] ok -- thanks for looking. [17:00:33] btw, how are you seeing your edit token? [17:02:01] "editToken":"+\\" <- that's it? [17:02:42] yeah. [17:02:48] in the JS [17:02:55] I think it should also be a cookie, though [18:06:56] !cutelabs is http://grab.by/mhFS [18:06:56] Key was added [18:11:00] !forget cutelabs [18:11:07] mhhm [18:11:21] LOL [18:11:22] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.10.6.8 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [18:11:31] !cutelabs is https://wikitech.wikimedia.org/wiki/File:Labs_Tools_topology.jpg [18:11:31] This key already exist - remove it, if you want to change it [18:11:42] !forget is this is not a way to delete keys [18:11:42] Key was added [18:11:55] xD [18:16:02] petan: im not sure how I should change it more [18:16:09] i should probably add arrows [18:16:26] but i can see it getting messy :P [18:16:31] !cutelabs del [18:16:31] Successfully removed cutelabs [18:16:39] !cutelabs is https://wikitech.wikimedia.org/wiki/File:Labs_Tools_topology.jpg [18:16:39] Key was added [18:16:46] addshore: can you create a service account for you bot [18:16:48] on bots [18:16:59] I implemented that thing that we use on tools [18:17:00] mhhhm [18:17:04] okay [18:17:09] * addshore goes to check [18:17:13] right now you can run it either as you or as local-addbot [18:17:30] well, you will need to cp .my.cnf from your account [18:17:31] this is why I was thinking everything should be piled into one project [18:17:39] well it shouldn't [18:17:51] bots is supposed to be staging area of tools, so it needs to be separate [18:18:08] we can't let evil bots break the production [18:18:09] but it can be seperate without being in an entirly different project? [18:18:20] not easily [18:18:31] but it would be best if it was renamed or something [18:18:35] like tools-beta [18:18:37] or whatever [18:18:45] bots and tools is so different [18:19:06] I don't think it would be hard to create aliases in labs [18:21:18] addshore why is tool-login on different level than bots-login [18:21:20] :D [18:21:31] ah [18:21:32] tools-login has a public ip [18:21:32] I see [18:21:40] the cyan bit at the top is internet accessible [18:21:48] ok [18:22:00] btw you can ssh into tools-apache01 from external ips ;p [18:22:06] wait [18:22:10] bots-apache01 [18:22:18] balls its got a mistake in it xD [18:23:31] addshore: What's the purpose of the arrow from bots-sql01 -> tools-db? [18:23:42] (BTW, SVG rules :-).) [18:23:50] scfc_de: currently you can access botsdbs in tools [18:24:43] addshore: What's the host name? [18:24:51] @notify binasher [18:24:51] This user is now online in #wikimedia-dev so I will let you know when they show some activity (talk etc) [18:24:55] bots-bnr1? [18:25:05] wait [18:25:07] bots-bsql01 ;p [18:26:10] addshore: "*b*sql", I tried bots-sql01 :-). But shouldn't the arrow point the other way, then? [18:26:27] scfc_de: potentially :P [18:26:48] (Or from tools-login, etc.) [18:27:07] i feel it needs lots of arrows [18:27:46] ill mock one up with arrows and see how much of a mess it is :P [18:30:06] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 0 minutes) [18:30:31] On https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools, the "Preliminary documentation on running tools on this project can be found here." needs to be more promiment. Otherwise people will just see the topology, the SAL and list of instances, and give up :-). [18:31:20] mhhm http://grab.by/mhV2 [18:31:47] :-) [18:33:24] BTW, do you have an idea how to rephrase https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools/Help&diff=68367&oldid=68366 so that it is clear that users only have to give the access right (x), but not the read right (r)? [18:35:08] nope :P but Ill have a think :P [18:35:41] k [18:36:13] heh I like the number of people who actually run their bots on -login no wonder they are slow [18:36:19] kolossos if one of them [18:36:25] hah [18:36:32] disable cron on login [18:36:35] and at [18:36:42] I [18:36:53] cron.deny/cron.allow at.deny/cron.allow [18:36:56] err [18:36:58] at.allow [18:37:11] I am more likely to simply make a daemon that randomly kills off non-interactive processes. :-) [18:37:25] Coren: any details of the backend shared storage for projects? something I can turn into a pretty picture for the pretty picture? :P [18:37:47] also Coren tools and bots public ips need ilines on freenode ;p [18:37:49] addshore: It's an NFS server that's not an instance in labs itself. [18:38:34] addshore: Someone's already started the chat with the freenode people so we stop running out of allowed connections. [18:38:43] lovaly :) [18:39:12] Ryan_Lane, Coren: On Toolserver, people seem to like using screen for that, so "non-interactive" might be too narrow :-). [18:39:35] cron.allow/deny is easy [18:39:58] killing noninteractive processes may cause issues with sftp/scp [18:40:00] Ryan_Lane: Yeah, but cron remains useful/necessary. The rule should be "cron can only be used to submit jobs" [18:41:10] Yep, and that can easily be monitored by other users using top. [18:42:11] Ryan_Lane killing cron on login would be evil, people need it [18:42:17] for what? [18:42:27] submitting jobs? [18:42:28] Ryan_Lane how are you supposed to schedule execution of job at certain time without cron? [18:42:40] I have qsub in cron [18:42:46] * Ryan_Lane nods [18:42:59] like some bots are supposed to be run twice a day or evey day once etc [18:42:59] we should sanitize cron, then [18:43:41] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 13 minutes) [18:43:59] Coren I would prefer asking these people to run their bots properly rather than randomly kill them :D or you will get many more e-mails like "labs suck" [18:44:18] petan: ACK. [18:44:44] petan: That was a joke. :-) [18:44:49] ah :P [18:45:23] Hint: When a sysadmin starts talking about killing off random processes or deleting random files, they're just channeling the BOFH. [18:46:11] Coren Ryan_Lane took it seriously as well :P [18:46:11] * Coren carfully avoids hinting that petan is the PFY. :-) [18:46:30] whatever. shortcuts suck when there are too many [18:46:44] googling PFY would probably result in 100 different meanings [18:47:24] petan: That's missing from your culture if you don't know what I mean by heart. Start here: http://en.wikipedia.org/wiki/Bastard_Operator_From_Hell [18:48:47] Ryan_Lane: Something plausibly less evil; have something that walks crontabs, disable non jobs starts and emails? [18:49:13] wikipedia say that PFY is "Steven" that can't be me [18:49:22] but there is 1 Steven here [18:49:28] * petan looks at StevenW [18:50:22] Coren anything that disable things shouldn't be probably automatic... maybe if it found a suspicious line, it could log it somewhere [18:50:52] how do you recognize if it doesn't start a job? [18:50:55] anyone think this might look better? http://grab.by/mhYu [18:51:06] what if the cronned thing is a shell script [18:51:08] petan: It's arguable either way. I'd probably have at the very least a grace period of just notices. [18:51:11] which actually start the job [18:51:20] petan: Hm. Good point. [18:51:21] or even a binary? : [18:51:22] :D [18:51:31] because that is my case lol [18:51:47] I have scheduled a script which determines if the bot needs to start or not [18:52:00] it could be probably done in a different way but this seemed easiest for me [18:52:21] petan: It might. I can think of a few other ways, but they'd be annoyances of no benefit. [18:52:36] petan: I think I'll just add some monitoring instead, and apply larts liberally. [18:52:48] I think easiest would be just explain properly to people how to run bots [18:53:00] petan: what what? [18:53:17] StevenW are you pimply faced? [18:54:30] we are in a need of pimply faced youth as alternative to unicorn. We need a mascot in cases the unicorn is on holidays :> [18:55:55] http://grab.by/mhZk ? [18:57:10] addshore: I'm working on slides with neat little diagrams of tools for the Hackaton, I'll send it to you. It shows the relationship between the boxes and all. [18:57:13] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 27 minutes) [18:57:22] Coren: would be lovely :P [18:58:08] relation between the boxes - how romantic [19:00:34] what do you think of the most recent addition petan? http://grab.by/mi04 [19:01:11] bots-gs will probably stay for some time as master of grid [19:01:21] :O [19:01:25] because I don't know how to move master server from one box to other :D [19:02:38] petan: It's a nightmare. [19:02:54] ok that is why I probably keep it [19:03:55] so petan http://grab.by/mi0I ;p [19:04:22] ok :D [19:04:42] may as well get rid of ibnr1 now? [19:05:34] hmm depends... I don't really need it as long as there is -labs but I have had some plans with it [19:05:42] :P [19:05:54] we need to see how the interactive bots will work on shared server with heavy load bots [19:09:18] [bz] (ASSIGNED - created by: Antoine "hashar" Musso, priority: High - enhancement) [Bug 36648] replicate HTTPS architecture - https://bugzilla.wikimedia.org/show_bug.cgi?id=36648 [19:10:49] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 40 minutes) [19:14:07] !log wikivoyage Rebooted voy-en [19:14:08] Logged the message, Master [19:16:20] petan: sorry, can't help ;) [19:16:40] :P [19:24:19] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 54 minutes) [19:24:43] !sh [19:24:44] http://bit.ly/10eZZoa [19:27:25] lazy [petan [19:27:35] :D [19:27:49] addshore u don't have rights to grant shell? [19:27:56] howcome [19:28:21] I think that bot should have the rights so that I can just do !approve [19:28:23] :D [19:29:12] hah, because noone has ever given them to me and I dont ask for such things ;p it would just give me one more thing to do xD [19:37:53] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 67 minutes) [19:45:39] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/TODO was modified, changed by MPelletier (WMF) link https://www.mediawiki.org/w/index.php?diff=686379 edit summary: [+264] Trying visual editor. Will have to remove the nowiki by hand. [19:46:32] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/TODO was modified, changed by MPelletier (WMF) link https://www.mediawiki.org/w/index.php?diff=686380 edit summary: [-53] fix ve overparanoia [19:51:25] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 81 minutes) [19:51:38] addshore, you there? [19:51:43] yup [19:51:59] I'm assuming you ahven't looked at Peachy yet? [19:52:05] nope :/ [19:52:18] That's ok. [19:52:38] My development on Peachy has currently stopped at the moment. [19:53:44] addshore, All I did to Peachy since the second release was improve stability and decrease curl timeouts. [19:54:21] link me to a single file where i can see the majority of peachy functions for wp? :D [19:54:41] addshore, umm... [19:54:51] depends on what you want to do with it. [19:55:03] Images, pages, or users? [19:55:09] link me to the file that has the function to list all members of a cat ;p [19:55:24] is it on github yet? ;p [19:55:31] No [19:56:16] :< [19:56:23] is it anywhere online yet? :D [19:56:43] addshore, the function you are looking for is in Wiki.php [19:56:46] addshore, petan: I've made tools-dev public since it was already configured as a bastion. [19:57:19] Although, that's irrelevant since all files work as one. [19:57:32] addshore, [19:57:54] $site = Peachy::newWiki( ); [19:57:54] show me wiki.php! :D [19:58:46] Coren how public? aren't we going out of IP's? [19:58:46] $site->categorymembers( $category, $subcat = false, $namespace = null, $limit = 50); [19:59:09] Coren also, -dev isn't configured as submit host, for security reasons [19:59:15] !log tools made tools-dev.wmflabs.org public [19:59:15] Don't you have the downloaded copy addshore? [19:59:16] Logged the message, Master [19:59:47] Cyberpower678: i may have lost it somewhere :/ [19:59:55] /facepalm [19:59:56] petan: Still is a useful bastion. [20:00:01] ok [20:00:15] addshore you lost it because you aren't using pidgeon :D [20:00:24] the best irc client ever XD [20:00:34] (because I made it) [20:00:41] mwahaha [20:00:49] petan: dont worry, as soon as irccloud finishes beta ill likely use it all the time xD [20:00:57] addshore, http://tools.wmflabs.org/cyberbot/Peachy-Releases/Peachy-2-alpha-2.zip [20:01:09] my client is better than irccloud, it's just that peope don't know how to use it [20:01:36] addshore in combination with pidgeon services it offers more than irccloud, you don't think that I am really online for several weeks ;) [20:01:43] mmmmm, vanilla yogurt [20:01:57] it's just like quassel [20:02:15] petan, I thought you were an automaton [20:02:20] Damianz bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 2% free memory [20:02:25] Cyberpower678 sush [20:02:33] shush [20:02:38] or whatever you brittons say [20:02:39] :D [20:02:48] mmmm, sushi. [20:02:51] :) [20:02:51] petan: just shh ;p [20:02:56] aha [20:03:10] I never know if it's ssssssh or shhhhhhhh [20:03:11] addshore, you SSH. :p [20:03:12] :D [20:03:31] in our language there is a letter that actually makes the "sh" sound [20:03:40] show me! [20:03:47] The first is more like a snake hissing. [20:03:56] the second one is shushing. [20:04:04] Coren: What's tools-dev's purpose? [20:04:07] š [20:04:16] š == sh [20:04:19] addshore, get my link. [20:04:21] ? [20:04:22] compressed :P [20:04:23] :D [20:04:39] Cyberpower678: can I start a github project for it? :P [20:04:44] petan: It's fine [20:04:50] ššš == shh [20:04:54] addshore, no. Because your copy is outdated. [20:04:55] scfc_de: A place where one can build large packages, for instance, without impacting the "normal" -login bastion. It will double as a backup bastion at need. [20:04:55] Warning: There is 1 user waiting for shell: Christian Clerc (waiting 94 minutes) [20:05:01] compared to mine. [20:05:03] Damianz it's not :D [20:05:06] it's dying OOM [20:05:10] Cyberpower678: then put your on git ;p [20:05:17] it's not [20:05:23] I want to finish Peachy 2 first. [20:05:32] Coren: Thanks. BTW, was there any progress on the Python module packaging? [20:05:47] * Coren points at petan. [20:05:55] oh that [20:06:04] Petan was looking into it. Dunno what his success was. [20:06:11] well, I sadly didn't come out with any automated solution [20:06:39] 1 time conversion would suck as there will be new versions [20:06:47] Cyberpower678: my plan was to tell you what else I think your missing ;p [20:07:31] addshore, you could only tell me that once I'm done. [20:08:02] And besides, you can write custom plugins that seamlessly will integrate into the framework. [20:08:33] Cyberpower678: http://collabedit.com/8nn29 ;p [20:08:40] petan: Are the Debian rules (?) files available somewhere? [20:08:49] what you mean? [20:09:14] anyway I think it's easier and probably more reliable to use both apt and pip rather than using some home-made convertor [20:09:39] scfc_de you mean control files? for which package? [20:09:45] addshore, [20:09:47] chatting [20:09:57] petan: For whatever module you looked into packaging. [20:10:08] scfc_de mhm let me check [20:10:31] I was doing that on -dev but I think no, because I didn't even find out how to debug what pip is doing [20:11:01] in order to create a control file I would need to get all scripts that are executed during the installation and all paths of all new stuff [20:11:16] + dependencies [20:11:37] this isn't easily extracted from pip files [20:12:53] petan: I'd have thought that one would take the files from another package already in Ubuntu and used that as a template. [20:13:37]