[00:42:57] [bz] (NEW - created by: Marc A. Pelletier, priority: Unprioritized - normal) [Bug 48282] Initial image (lucid) fails puppet runs - https://bugzilla.wikimedia.org/show_bug.cgi?id=48282 [04:12:48] !log tools added -exec-03 and -exec-04. Moar power!!1! [04:12:51] Logged the message, Master [04:19:55] Righ then, who broke Icinga? :P [04:20:35] FastLizard4's question is CRITICAL: CRITICAL - no answer in 15 seconds. [04:20:55] :O [04:21:03] o noes [04:21:09] * FastLizard4 acknowledges the service problem [04:21:22] * FastLizard4 also suppresses notifications and turns on passive checking :P [04:25:20] Warning: There is 1 user waiting for shell: Morriswa (waiting 0 minutes) [04:38:43] Warning: There is 1 user waiting for shell: Morriswa (waiting 13 minutes) [04:52:04] Warning: There is 1 user waiting for shell: Morriswa (waiting 26 minutes) [05:05:26] Warning: There is 1 user waiting for shell: Morriswa (waiting 40 minutes) [05:18:47] Warning: There is 1 user waiting for shell: Morriswa (waiting 53 minutes) [05:32:13] Warning: There is 1 user waiting for shell: Morriswa (waiting 67 minutes) [05:45:34] Warning: There is 1 user waiting for shell: Morriswa (waiting 80 minutes) [05:58:55] Warning: There is 1 user waiting for shell: Morriswa (waiting 93 minutes) [06:12:21] Warning: There is 1 user waiting for shell: Morriswa (waiting 107 minutes) [06:12:26] !log wikivoyage Edited /orig/LocalSettings.php: $wgStylePath = "/skins"; [06:12:28] Logged the message, Master [06:16:02] !log wikivoyage edited /srv/mediawiki/orig/LocalSettings.php: $wgScriptPath = "/w"; [06:16:03] Logged the message, Master [06:16:55] !log wikivoyage edited /srv/mediawiki/orig/LocalSettings.php: $wgSitename = "Voy-en"; [06:16:56] Logged the message, Master [06:17:23] !log wikivoyage edited /srv/mediawiki/orig/LocalSettings.php: $wgLogo = "https://wikitech.wikimedia.org/w/images/e/e6/Wikivoyage-lab-icon-152x161.png"; [06:17:24] Logged the message, Master [06:17:57] !log wikivoyage edited /srv/mediawiki/orig/LocalSettings.php: $wgFavicon = "$wgScriptPath/favicon.ico"; [06:17:57] Logged the message, Master [06:18:42] !log wikivoyage edited /srv/mediawiki/orig/LocalSettings.php: $wgMetaNamespace = "Voy-en"; [06:18:43] Logged the message, Master [06:20:03] !log wikivoyage uploaded favicon.ico to /srv/mediawiki/favicon.ico [06:20:04] Logged the message, Master [06:25:42] Warning: There is 1 user waiting for shell: Morriswa (waiting 120 minutes) [06:39:04] Warning: There is 1 user waiting for shell: Morriswa (waiting 133 minutes) [06:52:26] Warning: There is 1 user waiting for shell: Morriswa (waiting 147 minutes) [07:05:47] Warning: There is 1 user waiting for shell: Morriswa (waiting 160 minutes) [07:14:02] what does that error message mean? [07:14:35] rschen7754: the bot, you mean? [07:14:40] yeah [07:14:43] the waiting for shell [07:15:04] there's still a manual approval process for all new users that are requesting shell access to labs [07:15:17] this is supposed to help prevent people from falling through the cracks / getting ignored [07:15:53] during daytime PST you almost never hear it because people are generally quite responsive, but during the night time PST (i.e., now) it gets obnoxious [07:19:13] oh... [07:19:13] Warning: There is 1 user waiting for shell: Morriswa (waiting 174 minutes) [07:19:21] so an on-wiki request? [07:19:49] yeah, https://wikitech.wikimedia.org/wiki/Form:Shell_Access_Request [07:20:16] yeah i guess i'm a bit puzzled by that request… [07:21:35] rschen7754: erm, sorry -- should have linked to https://wikitech.wikimedia.org/wiki/Requesting_shell_access [07:21:54] yeah, seems more like an automatic request filed on account creation [07:32:35] Warning: There is 1 user waiting for shell: Morriswa (waiting 187 minutes) [07:45:56] Warning: There is 1 user waiting for shell: Morriswa (waiting 200 minutes) [07:59:18] Warning: There is 1 user waiting for shell: Morriswa (waiting 214 minutes) [08:01:54] !sh [08:01:54] http://bit.ly/10eZZoa [08:02:20] rschen7754 don't like it? request the permissions to sort out the requests, and fix it [08:02:49] petan: i have this channel on mute :) [08:02:58] doesn [08:03:05] 't seem to be, when you responded to me [08:03:06] i was just surprised that that particular user wanted shell access [08:03:15] well, it still highlights me [08:11:49] Coren do we really need more power? XD [08:11:59] Coren these 2 boxes were idle most of time [08:12:40] Warning: There is 1 user waiting for shell: Morriswa (waiting 227 minutes) [08:26:02] Warning: There is 1 user waiting for shell: Happy5214 (waiting 1 minutes) [08:39:23] Warning: There is 1 user waiting for shell: Happy5214 (waiting 14 minutes) [08:52:49] Warning: There is 1 user waiting for shell: Happy5214 (waiting 27 minutes) [09:06:19] Warning: There is 1 user waiting for shell: Happy5214 (waiting 41 minutes) [09:19:48] Warning: There is 1 user waiting for shell: Happy5214 (waiting 54 minutes) [09:33:14] Warning: There are 2 users waiting for shell, displaying last 2: Happy5214 (waiting 68 minutes) E27 11w corn bulb (waiting 8 minutes) [09:46:39] Warning: There are 2 users waiting for shell, displaying last 2: Happy5214 (waiting 81 minutes) E27 11w corn bulb (waiting 21 minutes) [10:00:01] Warning: There are 2 users waiting for shell, displaying last 2: Happy5214 (waiting 94 minutes) E27 11w corn bulb (waiting 34 minutes) [10:13:22] Warning: There are 2 users waiting for shell, displaying last 2: Happy5214 (waiting 108 minutes) E27 11w corn bulb (waiting 48 minutes) [10:26:43] Warning: There are 2 users waiting for shell, displaying last 2: Happy5214 (waiting 121 minutes) E27 11w corn bulb (waiting 61 minutes) [10:40:13] Warning: There are 2 users waiting for shell, displaying last 2: Happy5214 (waiting 135 minutes) E27 11w corn bulb (waiting 75 minutes) [11:07:43] [bz] (NEW - created by: Niklas Laxström, priority: High - major) [Bug 48203] Purging does not work on deployment-prep / beta labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=48203 [12:21:57] @notify Ryan_Lane [12:21:57] I will notify you, when I see Ryan_Lane around here [14:02:35] petan: i now know what the best thing about irc cloud is ;p [14:02:46] what is it [14:03:00] it has put the above gazillion joins and quits all in one line :P [14:03:07] huh? [14:03:21] what exactly you mean [14:03:25] http://grab.by/mnRq [14:03:46] http://grab.by/mnRs [14:03:47] lol [14:03:59] this wouldn't be really that hard to implement in my client [14:04:04] do it then ;p [14:04:09] but it's againsted its philosophy so it would need to be extension [14:04:13] its literally the best ever feature [14:04:44] hmm [14:09:36] anyone knows if and how a labs instance's disk space can be increased? [14:09:56] mlitn tricky thing... [14:10:07] you should ask ryan [14:10:45] alright, will do - thanks :) [14:54:00] scfc_de, what's up? [14:59:38] andrewbogott: How to "officially" use instance-proxy with mediawiki_singlenode. http://meta.wikimedia.org/wiki/User_talk:Rogerhc#Wikivoyage_on_Labs for some background (it seems to be fixed now). Basically, earlier, I set $wgServer to voy-en.instance-proxy in orig/LocalSettings, but then your change https://gerrit.wikimedia.org/r/#/c/62416/ overrode $wgServer in ./LocalSettings. What's the proper way to use instance-proxy with [14:59:38] mediawiki_singlenode? [15:00:50] scfc_de, what happens right now if you just visit .instance-proxy.wmflabs.org? Does it redirect? [15:02:45] Warning: There is 1 user waiting for shell: Alessio (waiting 0 minutes) [15:04:43] andrewbogott: Yes, at the moment it is working perfectly. But the process to get there was more trial and error, and it might probably break with the next Puppet change :-). So I'm asking what *you* as the manifest author thinks how mediawiki_singlenodes via instance-proxy should be configured, so that it may be futureproof. [15:05:08] (And having no opinion would be fine as well :-).) [15:06:54] I'm not sure. Certainly the idea of the instance proxy is that it will just work. But, let me check a couple of things. [15:13:36] scfc_de: OK, I'm still not sure :) I does indeed look like $::labs_mediawiki_hostname is now moot and not used anywhere, and I'm a but surprised that that works. [15:14:00] But… if things work properly for you without a tunnel set up, then probably all is well :) I'll try a few more experiments but probably the answer to your question is just 'access via the proxy'. [15:16:12] Warning: There is 1 user waiting for shell: Alessio (waiting 13 minutes) [15:18:21] andrewbogott: Thanks. [15:19:11] scfc_de: I don't understand enough about MW config to understand why it works to have the server set so generically. But, it seems like an improvement if everything is working for you :) [15:22:19] andrewbogott: Well, also the Apache config and of course instance-proxy need to be considered, so I think of course it would be nice if the manifest had some option "use_instanceproxy" to spare other projects the trouble of figuring it out, but it's very low priority for me. [15:22:44] Did you have to change things in the apache config? [15:24:22] No, I mean when designing a manifest one needs to think of how MediaWiki config, Apache config and instance-proxy interact. Sorry for the confusion. [17:38:55] [bz] (NEW - created by: Pau Giner, priority: Unprioritized - normal) [Bug 48299] Interlanguage links do not work inside labs environment - https://bugzilla.wikimedia.org/show_bug.cgi?id=48299 [17:43:28] hello. [17:45:45] Coren: hm. maybe I'll yank ganglia-monitor from the image [17:46:00] it's not configured when it starts anyway and puppet will install it [17:46:19] Ryan_Lane: Indeed it will. It seems to confuse the heck out of the initial run. [17:46:34] it's absurd that it doesn't install properly [17:47:05] [bz] (ASSIGNED - created by: Antoine "hashar" Musso, priority: High - enhancement) [Bug 47827] beta: Setup a Wikidata wiki - https://bugzilla.wikimedia.org/show_bug.cgi?id=47827 [18:17:11] andrewbogott: seems the same issue on openid-wiki2 is happening on nova-precise2 [18:17:33] Ryan_Lane: On phone, I'll check shortly [18:17:36] one thing I noticed on nova-precise2 was that it had a version of php upstream [18:17:42] rather than the one from our repo [18:18:09] Is this the right place to ask q's about Wikipedia admin procedure? (I'm waiting for resolution on a TfD discussion; I'd thought those usually got close after 7 days.) [18:18:18] wrong channel [18:18:23] likely #wikimedia-tech? [18:18:27] or #wikipedia ? [18:18:35] probably #wikipedia [18:19:08] thanks. will try that one. [18:20:15] yw [18:49:06] Ryan_Lane: Ok, back -- I don't think I ever quite understood the issues with openid-wiki2; what are the symptoms? [18:49:56] /usr/bin/php: error while loading shared libraries: libasn1.so.8: failed to map segment from shared object: Cannot allocate memory [18:50:57] Oh yeah, that :( [18:52:21] The only things set to => latest in that manifest are php-apc and imagemagick. Is it possible that php-apc upgraded and the error is caused by a version mismatch? [18:53:09] I've tried removing it [18:53:38] of course, all packages upgrade in labs [18:55:02] this stuff in /var/log/apt/history.log is you troubleshooting, right? [18:55:51] yep [19:03:00] Sorry, I'm still behind… what are you doing to produce that error, and where are the errors getting logged? [19:03:17] (I'm running php things on nova-precise2 without any trouble) [19:08:42] what's memory_limit in php.ini? [19:11:13] ori-l: I've tried making it quite higher [19:11:19] 2-3 times higher [19:12:14] hm… I can't log into openid-wiki2 at all :( [19:13:45] gluster is such a piece of crap [19:14:04] we really need to put some time into switching everything over to NFS [19:14:19] maybe we should just schedule an outage of /home and /data/project [19:16:27] Ryan_Lane: Gluster related to the php thing or just a similtaneous annoyance? [19:16:39] Oh, probably preventing my login I expect [19:16:48] coincidence [19:19:22] Ryan_Lane: OK, can you tell me how to produce/detect this failure? I want to see if it's happening on other similar instances. [19:19:35] it's when I try to use the openid extension [19:19:54] just try logging in with google [19:20:32] Coren: wow [19:20:40] you finaly decided to enable swap? [19:20:42] wow? [19:20:50] petan: Not for the reason you think. :-) [19:20:52] 20gb on each node [19:20:57] which reason then? [19:20:57] Oh, OK, that's not a thing I ever use [19:21:05] maybe the gluster stuff is related [19:21:14] because it's surely not working either [19:22:54] I purged and reinstalled gluster [19:22:56] it's working now [19:22:56] Coren? [19:23:10] petan|ic: ? [19:23:12] openid isn't :) [19:23:17] so, unrelated [19:23:35] :o [19:24:46] Coren: what reason? [19:24:56] if not for a reason I think [19:25:48] petan|ic: Workaround silly platforms that allocate ridiculous amounts of memory it doesn't use. I /still/ don't allow overcommit. :-) [19:25:53] Ryan_Lane, sorry, I'm still slow… I don't see openid running on w or w3 on nova-precise2. and w2 fails for a seemingly unrelated reason What am I missing? [19:26:04] I disabled it earlier ;) [19:26:13] it's in the settings file [19:26:25] Coren: ok what is that workaround? how does it work? [19:27:14] Coren: what about just allowing bigger h_vmem than 8gb? on 8gb when it was on 100% there was some 400mb of real memory used [19:27:16] ah [19:27:27] petan: Keep overcommit disabled in the kernel, but enable swap so that vmem is higher. In practice, it means the swap should/can never be used, but a cbreak() that never touches the pages will "allocate" thinking it has swap. [19:27:39] aha [19:27:48] so not h_vmem is 28gb? [19:27:51] * now [19:28:26] ah yes... ok [19:28:28] nice [19:28:36] petan: You still miss the point, though. If the h_vmem is fully allocated, then that memory /could/ go away; which is why you can't/shouldn't overcommit it. [19:28:59] petan: So, even with 8G, maybe 400M were used, but those processes /could/ have used all 8G [19:29:03] yes it /could/ but I bet my shoes it never will [19:29:13] Probably. Hence the padding I added. :-) [19:29:19] Coren: Mind naming the "silly platforms"? :-) [19:29:39] scfc_de: Mono is the worst offender, with python a close contender when you import a lot of stuff. [19:29:50] Coren: if in mono or python you were getting close to 8gb of real memory, the interpretor allocate more virtual memory, and crash [19:30:22] so.. no worries [19:30:35] scfc_de: The python hello world may be a one-liner; but it still takes nearly 1G of vmem. :-) [19:30:45] Coren: are you sure that java isn't worst? :P [19:31:01] petan: Java is a glutten, but if you tell it use X ram, it'll use X. [19:31:05] anyway mono on /my/ server eats 20mb of vmem on complicated application [19:31:06] glutton* [19:31:17] mono on labs eat 800mb for simple app [19:31:39] I think it is something with labs repository [19:32:53] labs repository? [19:32:56] we don't have mono in t [19:32:59] it [19:33:16] Coren: Hmmm. On Toolserver, Python rarely takes more than 100 MByte for most of my scripts. There are some processes that take over 1 GByte, but that usually means that someone tries to reimplement SQL in Python and the memory is spent for data. [19:33:18] it would be ubuntu's repo ;) [19:34:11] scfc_de: of /actual/ memory; it still allocates a *huge* arena. [19:34:27] Ryan_Lane: could you add me to the tools project? I'm interested in trying to move a bot there [19:34:50] Coren is the tools master ;) [19:35:10] !coren [19:35:10] The toolmeister: http://www.mediawiki.org/wiki/User:MPelletier_(WMF) [19:35:14] See? ^^ [19:35:20] :D [19:35:34] danmichaelo: What's your wikitech username? [19:35:44] same [19:35:54] (as here) [19:36:11] danmichaelo: {{done}} [19:37:47] thx! [19:39:33] 2 weeks from hackaton and I managed to loose my ID card [19:41:28] Coren: That probably means I need to adjust "-l h_vmem" in my crontab :-). [19:44:22] Coren: Do you know from the top of your head which resource specifier in time(1) gives virtual memory used? qacct seems to be disabled on Tools. [19:45:20] scfc_de: qstat -F h_vmem for all queues [19:45:24] scfc_de: You can use 'qstat -j ' to get detailed information while the job is running; otherwise time -v will give it to you. [19:45:51] qacct will require some extra trickery with NFS, which I don't have time to do now. [19:46:31] Ryan_Lane: I was checking jabber today and I must say I find it ultimately complicated, irc really beats it in simplicity [19:46:49] setting up a jabber server is pain in the ass [19:47:09] yes, irc is simple, totally insecure and just simply a terrible protocol [19:47:21] I disagree it's totally insecure [19:47:28] you can always use ssl [19:47:47] there's no authentication built into the protocol [19:48:01] irc protocol isn't best, but keep in mind it's very old, xml didn't exist in time when irc was invented [19:48:06] of course there is [19:48:09] no. there isn't [19:48:12] there's ident [19:48:18] ancient servers were using it [19:48:20] which is a bullshit form of identification [19:48:24] meaning of ident was different in past [19:48:45] there's no otr [19:48:49] otr? [19:48:54] off the record [19:49:00] what you mean? [19:49:09] there's no offline message support [19:49:34] like being able to read the backlog of messages from time when you weren't in channel? [19:49:38] otr is person to person encryption [19:49:40] there are workarounds to enable such a thing [19:49:58] yes, yes, workarounds. jabber has all of this stuff [19:50:19] ejabberd is relatively easy to install/configure [19:50:28] Coren: But the information at the end of the job run would be interesting, not some intermediate value. "/usr/bin/time -v true" gives only average values for memory except for RSS. [19:50:37] it's easy to embed into web apps [19:50:41] you can use a chat client with it [19:50:50] scfc_de: You can cheat a bit and ssh to tools-master to use qacct [19:50:53] but jabber is not so popular, I guess the reason is how much complicated it is [19:51:01] are you shitting me? [19:51:06] scfc_de: That trick will go away once the NFS is done. [19:51:15] why so many open source projects use irc then? [19:51:17] google, microsoft, yahoo, facebook all use it [19:51:23] yes corporations [19:51:36] large web sites, that actually do communications well ;) [19:51:36] but open source world stay on irc... there must be a reason [19:51:58] nerds already have irc [19:52:09] normal people have no clue, but they all have chat clients [19:52:14] <^demon> We like our irc and changing things is hard :) [19:52:19] well, if you like it so much why don't we set up a community maintained jabber server on labs for wikimedia purposes [19:52:20] that's basically it [19:52:51] you don't necessarily need to force people to use something else, you can offer it and if they like it, they would switch to it [19:52:51] I'm not doing anything like that, unless we choose to use something like that on the sites :) [19:53:22] I'm just saying that embedding IRC into web apps is the wrong approach [19:53:24] well, first you need to set it up and let people try it, then you can start discussions about using it [19:53:38] it suck, but that is a wikimedia way [19:53:46] !log wikivoyage voy-en, set $wgLogo to "https://wikitech.wikimedia.org/w/images/d/dc/Wikivoyage-lab-icon-v2-152x161.png" in /srv/mediawiki/orig/LocalSettings.php [19:53:47] Logged the message, Master [20:03:32] ok. new image fixes ganglia puppet issue. still not sure why lucid image doesn't allow immediate login [20:05:55] Coren: Just discovered: "qstat -u local-dbreps -s z" shows "recently finished" jobs as well. In "qacct -j 1148", "maxvmem" is the value you want for "jsub -mem", isn't it? [20:06:27] scfc_de: Cool, I didn't know that! Also, yes, "maxvmem" is what you want. [20:06:44] Ryan_Lane: why would Google index some of the articles with the https protocol? [20:07:08] there's one wiki that is https only [20:07:18] uz, I think? [20:07:34] no, i mean on enwiki [20:07:45] otherwise maybe because of people linking to it that way? [20:08:12] but don't we have a canonical link in the pages? just like that .m. is also not indexed? [20:08:32] what do you mean? [20:09:13] Fun for the whole family: http://tools.wmflabs.org/?status [20:09:42] Coren: why not special views in ganglia? :) [20:09:52] http://ganglia.wmflabs.org/latest/ [20:10:07] we don't allow google to index the .m. articles separately, they are indexed as the regular desktop URL [20:10:08] http://ganglia.wmflabs.org/latest/?c=syslog-collection&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [20:10:25] is it a good idea to preserve the internal IP when recreating an instance? [20:10:28] Ryan_Lane: Because ganglia is teh suxx0rz at showing enumerations of stuff. :-) [20:10:36] MaxSem: not much choice there [20:10:39] Ryan_Lane: Although the usage should probably be metered. [20:10:45] to date about 1600 instances have been created [20:10:54] Coren: memory is virtual or resident? [20:11:03] Coren: used or allocated? [20:11:09] Coren: including swap? [20:11:12] petan: Actuall resident usage [20:11:21] physical ram only? [20:11:32] http://ganglia.wmflabs.org/latest/?c=tools&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [20:11:36] petan: That number is indicative of actual use, not vmem. So yeah, only RSS and only physical [20:11:56] Coren: you can make custom views: http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&tab=v&vn=test [20:12:02] that show aggregations [20:12:10] Coren: it's nice [20:12:24] Ryan_Lane: Yes, but those views are not very good at showing /what/ is running, which is the point of ?status [20:12:44] ah. right [20:13:06] The stats are just nice. :-) [20:13:12] * Ryan_Lane nods [20:13:39] Coren: maybe some graphs? :P [20:13:46] addshore is person you need for that [20:13:55] Coren: Yes, very nice. I also like the direct links to the maintainers on http://tools.wmflabs.org/?list. [20:13:57] ? :P [20:14:41] Warning: There is 1 user waiting for shell: Sanja pavlovic (waiting 0 minutes) [20:14:47] Coren: why my bot isnt in ?list [20:15:06] What's its name? [20:15:10] afcbot [20:15:11] drdee: do they set noindex in the content for .m? do we have a different robot.txt for it? [20:15:26] drdee: I'm not sure if we can do either for https [20:15:35] maybe at the varnish layer? [20:15:44] petan: Because the web server isn't allowed to look at its home. :-) [20:15:52] I'm wondering how google would handle that... [20:16:11] whether it would count the page as noindex, or the page with the protocol [20:16:19] we could add rel=canonical link to a wiki apge [20:18:02] petan|ic: Because the web server isn't allowed to look at its home. :-) [20:19:06] drdee: for https the content is the same [20:19:22] drdee: we'd need to do this at the varnish/squid level, if we were to do it. [20:20:30] seems it can be done with a header [20:20:34] which is likely the best option [20:21:32] shall i send an email to Ops? [20:21:40] we'd need to watch out for https only wikis [20:21:45] like uz [20:21:50] yes, please do [20:22:04] reason is because community people are complaining about drop in page views since we stopped sending nginx traffic to udp2log [20:22:16] and this only affects https versions of articles [20:22:24] I really don't understand that [20:22:46] aren't they showing up in the squid logs? [20:22:46] what part? [20:22:47] yeahhhhh [20:22:52] that's what i always thought as well [20:22:57] but the drop is larger than 50% [20:23:11] there must be some other issue then [20:23:19] there's no way we're getting that much https traffic [20:23:43] i think it''s two issues: [20:23:51] 1) google indexing some articles using https [20:24:14] 2) nginx hits not routed to squid [20:24:38] or not logged at least obviously it's being served [20:24:38] a 50% drop is dramatic. [20:24:50] 1 sec [20:24:51] there's no way this is just related to https [20:24:58] is IPv6 being counted properly? [20:25:12] because that traffic increased heavily [20:25:15] check http://stats.grok.se/en/latest90/Schizophrenia [20:25:38] oh. you mean it's dropped 50% on specific articles? [20:25:41] yes [20:25:50] ah. that makes a lot more sense :) [20:26:00] ok, that's a possibility, then [20:26:01] and only those articles that are indexed with https in google [20:26:22] rel=canonical header seems like a good plan [20:26:38] we'll need to put some special cases in for https only wikis [20:26:42] we only have one right now, thankfully [20:28:15] Warning: There are 2 users waiting for shell, displaying last 2: Sanja pavlovic (waiting 13 minutes) BraneJ (waiting 10 minutes) [20:29:32] ok; drafting email [20:31:05] [bz] (RESOLVED - created by: Marc A. Pelletier, priority: Unprioritized - normal) [Bug 48282] Initial image (lucid) fails puppet runs - https://bugzilla.wikimedia.org/show_bug.cgi?id=48282 [20:34:55] Coren: there is something interesting, many people joining tools every day, but still only 13 bots running for weeks :P [20:34:55] anomie: Ping? [20:35:12] Coren: pong? [20:35:17] pang [20:35:32] petan|ic: Not entirely surprising; people are mostly experimenting with the setup right now, not running their tools yet. Also, many tools are just web-based. [20:35:37] Ryan_Lane: final question how would I recognize a redirected nginx ssl hit in the squid log? check the referer and make sure it starts with https? [20:35:49] X-Forward-Proto [20:35:53] anomie: Take a look at http://tools.wmflabs.org/?status [20:36:22] X-Forwarded-Proto, that is [20:36:23] Coren: I updated motd on -login [20:36:24] anomie: anomiebot reserves ~1G but is very reasonable in its actual usage. You could consider reducing the -mem request? [20:36:34] Coren: now it say "don't run your bot here" :D [20:36:59] drdee: it's possible that until we switch to varnish that we may not be able to fix this [20:37:13] why is that? [20:37:25] oh X-Forward-Proto [20:37:37] no, rel=canonical [20:37:47] or would we send the same on both http and https? [20:37:53] Coren: I'll try it. [20:37:57] petan|ic: When I get a bit of free time, I'm going to rework motd.d entirely so that we can do real messages of the day and stuff, and remove that crappy ubuntu verbosity only roots want to know. :-) [20:38:18] if we can always set that in the head, it's easy to solve in mediawiki [20:38:19] yes, rel=http://wiiki [20:38:32] i think we can fix it in mediawiki [20:38:48] ok i will show you my draft email soonish [20:38:49] <^demon> Coren: Doesn't everyone want to know that 3 packages need updating? ;-) [20:38:49] if we need to only set that in https pages, then we need to do it at the cache layer [20:38:51] Coren: that is actually very simple [20:39:02] Coren: just remove bunch of /etc/update-motd files :P [20:39:24] Coren: tools-exec-03 is missing JSON.pm. We had that problem before. [20:39:29] I most hate ** reboot required ** I remove that from all my servers [20:39:40] most useless information ever [20:39:51] <^demon> But, how will you know that your server requires a reboot? [20:40:07] anomie: Gah! I forgot to note it! At least now, exec hosts are done with puppet so that won't happen again. :-) [20:40:11] Coren: Could you add the explanation of the two "VMEM" values to http://tools.wmflabs.org/?status as well? At present, it's a bit confusing. [20:40:15] Coren: I also hate the command-not-found package which is right now installed everywhere [20:40:24] it slow down the shell incredibly :/ [20:41:03] scfc_de: If I find a good way of saying it. It's (vmem you are using / vmem you requested) with, possibly (peak vmem you've ever used if higher than currently) [20:41:45] Warning: There are 2 users waiting for shell, displaying last 2: Sanja pavlovic (waiting 27 minutes) BraneJ (waiting 24 minutes) [20:43:34] Ryan_Lane: what if we made that shell request optional on registration page, unchecked by default? [20:43:42] Ryan_Lane: all these spam bots wouldn't trigger it :> [20:43:50] heh [20:44:02] I guess only like 20% of current new users wouldn't ask for it [20:44:06] * would [20:44:18] anomie: Will be fixed shortly, just waiting on Jenkins. [20:44:31] or we could fix the bug [20:44:42] both would take dev time [20:44:42] which bug [20:44:52] hi, i'm trying to set up a bot on the tools project, I've created a service group and it looks fine… but I need the libxml2-dev package. Could some apt-get install libxml2-dev? [20:44:59]