[12:30:00] hey all, anyone online that can add JvGent to the gwtoolset user group on commons.wikimedia.beta.wmf ? [12:30:00] * YuviPanda waves [12:30:00] dan-nl: you mean onwiki, right? [12:30:00] hey YuviPanda [12:30:00] onwiki? [12:30:00] no, on beta there's a user right gwtoolset [12:30:01] dan-nl: yeah, I mean, on http://commons.wikimedia.beta.wmflabs.org/ [12:30:01] JvGent needs it so she can try out gwtoolset uploads on beta before she does so on production [12:30:01] right? [12:30:01] yes [12:30:01] dan-nl: sorry, looks like I've rights only on enwiki on betalabs, not commons [12:30:01] hey yuvi you have rights on beta [12:30:01] http://commons.wikimedia.beta.wmflabs.org/w/index.php?title=Special:ListUsers&group=bureaucrat [12:30:01] she only needs it there [12:30:01] it's only on betalabs that she needs it [12:30:01] dan-nl: yeah, but the problem is that it's not a unified account apparently, and I've no clue what password I used for that separate account [12:30:01] ah [12:30:01] and no email set either [12:30:01] let me see if I can remember the paswsword [12:30:01] oh boy [12:30:01] dan-nl: no, can't really figure out. sorry [12:30:01] I'll need to wait for someone with shell access to reset it for me [12:30:01] ok [12:30:01] thanks for trying [12:30:21] 3Wikimedia Labs / 3tools: Service group links are incorrect on http://tools.wmflabs.org/ - 10https://bugzilla.wikimedia.org/66614 (10Maarten Dammers) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier On http://tools.wmflabs.org/ you see links to add or remove members, for example https://wikitech.wikimed... [12:30:21] !log local-heritage Added Lokal Profil per request at [https://commons.wikimedia.org/w/index.php?title=User_talk:Multichill&oldid=126591871#Heritage_at_labs Commons] [12:30:21] Logged the message, Master [12:31:08] !ping [12:31:08] !pong [16:15:44] 3Wikimedia Labs / 3tools: Service group links are incorrect on http://tools.wmflabs.org/ - 10https://bugzilla.wikimedia.org/66614#c1 (10Tim Landscheidt) 5NEW>3RESO/DUP *** This bug has been marked as a duplicate of bug 63741 *** [16:15:44] 3Wikimedia Labs / 3tools: "add / remove maintainers" links at http://tools.wmflabs.org/ should link directly to management page - 10https://bugzilla.wikimedia.org/63741#c3 (10Tim Landscheidt) *** Bug 66614 has been marked as a duplicate of this bug. *** [16:47:56] 3Wikimedia Labs / 3tools: Provide filearchive table with fa_storage_key or, if it exists and is sufficiently indexed and populated, fa_sha1 for commonswiki - 10https://bugzilla.wikimedia.org/57697#c10 (10Rainer Rillke @commons.wikimedia) Thank you. What I, however observe is that fa_sha1-queries are notably... [17:11:55] i have dir-listing.activate = "enable" in my .lighttpd.conf, but http://tools.wmflabs.org/giftbot/ is still 404. what do i do wrong? [17:31:11] !ping [17:31:11] !pong [17:47:45] gifti: [17:47:46] $HTTP["url"] =~ "" { dir-listing.activate = "enable" [17:47:46] } [17:47:49] ^ try that [17:48:06] that seems to work for http://tools.wmflabs.org/jira-bugimport/svn/. [17:48:24] it works now [17:48:31] i had it without $HTTP… [17:48:48] the more i work with lighttpd the more i hate it [17:52:59] gifti: it's not worse than apache in terms of configuration [17:53:13] although maybe apache has a standard configuration that better suits you :-p [17:53:55] i don't think that i like that better, i never wrote configuration in it i think [17:54:09] * YuviPanda wonders if anyone can help him with an lighty url rewrite thing [17:54:35] valhallasw: isn't lighty config just lua? [17:54:46] YuviPanda: no, it's some DSL I think [17:54:51] lol [17:54:55] ah, yeah, I see the $ [17:54:57] hmm, sigh [18:33:25] multichill: thanks for doing the migration! :) [18:33:34] multichill: major pain points during it, if any come to mind? [18:34:16] YuviPanda: Several, uncommitted code, bugs, setup of git/gerrit, etc [18:34:44] multichill: hmm, specifically things toollabs can improve on? [18:36:07] Full log at https://wikitech.wikimedia.org/wiki/Nova_Resource:Local-heritage/SAL btw [18:36:16] multichill: looking [18:36:27] One thing I still had to look at was an easy logrotate service so I don't flood the homedir [18:37:53] ah, right. [18:38:01] let me file a bug for that. [18:38:14] I see that logrotate is installed, that might work [18:38:18] one of the things I've been thinking of adding as a service is a logstash service that makes toollogs indexed, searchable, etc [18:38:29] That would be nice yes [18:38:54] YuviPanda: For catbot I expose my logs at http://tools.wmflabs.org/catbot/logs/ [18:39:11] Would be nice if that would be something easy and shared so I can do it for the other tools too [18:39:32] multichill: yeah, plus it'll take care of spacing, etc as well [18:39:43] My logfile is already 1,4GB [18:39:47] and tool authors should be able to set it to public/private as they want (in case they have sensitive info there) [18:39:48] yikes [18:40:06] Exactly. It should work a bit like the archive bots we use [18:40:39] multichill: yeah. http://logstash.net/ is being currently trialled in production, and people seem to like it [18:41:04] multichill: it is also rather easy to write to, if you use any of the logging frameworks in any language (logging in python, IIRC) [18:41:13] and we should also be able to hook it up to SGE's .err, .output [18:42:20] multichill: an easier, shorter term solution would be to do logrotate, though. [18:42:30] Oh, that's nice. Does it do syslog too? I've been using https://code.google.com/p/php-syslog-ng/ for years, but that went closed source :-( [18:43:06] multichill: yeah, out of the box. [18:43:41] Cool. Does it contain libraries for explaining messages? [18:43:47] multichill: 'explaining'? [18:44:17] So I got this library of Cisco syslog messages. So if you hover over an error code, you get the Cisco manual of what's happening [18:44:34] That's pretty damn useful while digging through Syslog ;-) [18:45:24] multichill: aaah, right. not that I know of, though. I suppose we can hack that on in some form, but I don't think logstash supports it intrinsically [18:46:43] 3Wikimedia Labs / 3tools: Setup an easy to use logrotate based system for rotating tools logs - 10https://bugzilla.wikimedia.org/66623 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier Currently they can go *really* large, and there's no easy way to fix this. Having a way for tool authors... [18:46:53] multichill: ^ filed. [18:47:13] 3Wikimedia Labs / 3tools: Setup an easy to use logrotate based system for rotating tools logs - 10https://bugzilla.wikimedia.org/66623 (10Yuvi Panda) a:5Marc A. Pelletier>3Yuvi Panda [18:48:53] YuviPanda: Ah nice, I wonder if they have http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs/logmsgs.html in a machine-readable format [18:49:58] multichill: yeah, possible. also what UI thing were you using to read the logs? something cisco built? [18:50:25] php-syslog-ng has a simple, but effective web interface [18:51:20] ah, hmm. that's interesting, so I guess they were doing something that was standardized and not their own thing [18:52:11] It's just normal syslog format [18:53:02] I don't think I have any networking equipment that doesn't support syslog YuviPanda [18:54:08] multichill: no, talking about the extra bits (hovering over an error gives you docs) [18:55:05] Oh, yes, that is part of php-syslog-ng. They seem to have deleted all references to their source code, the fuckers [18:58:13] dicks, I gues [18:59:41] 3Wikimedia Labs: Request to access redacted webproxy logfiles of (Tool) Labs - 10https://bugzilla.wikimedia.org/59222#c14 (10Yuvi Panda) @metratron: Help would be appreciated! I've copied scrubbed-of-IPs sample log (with 1000 entries) to /shared/sample-nginx-log/cleaned-samplelog.log. If you can write a script... [18:59:49] multichill: do you know who metratron is? and if they're on IRC? [18:59:52] valhallasw: scfc_de ^ [19:06:22] YuviPanda: Found it at https://www.assembla.com/code/logzilla/subversion/nodes :-) [19:06:39] metratron doesn't ring a bell [19:06:55] multichill: is that the php version's last open thing? [19:07:00] or a new thing? [19:07:26] Seems to be the successor [19:07:38] hmm, right [19:08:10] Looking at https://www.assembla.com/code/logzilla/subversion/nodes/606/snmp/mibs it's quite infrastructure oriented [19:09:15] YuviPanda: metatron = Hedonil. [19:09:29] (I think.) [19:09:29] ah [19:09:36] that does make sense [19:10:18] multichill: http://rashidkpc.github.io/Kibana/ is the interface that WMF is using [19:14:26] 3Wikimedia Labs: Request to access redacted webproxy logfiles of (Tool) Labs - 10https://bugzilla.wikimedia.org/59222#c15 (10Yuvi Panda) Hopefully the 1000 log entries are enough. I can provide a larger sample if needed. [19:16:58] YuviPanda: I haven't looked at log tools for a while. Cool stuff! [19:17:08] multichill: :) yeah [19:17:26] multichill: in fact, I'm tempted to just setup a node, and tools that are sure their logs are all public an ship there now. [19:18:10] That would be nice [19:18:20] And while you're at it, start collecting netflow :P [19:18:41] multichill: let me file a bug for that. ACLs are not easy with logstash yet, I think, so private logs should probably remain private [19:19:05] multichill: :P we already are, there's a direct pipe to the NSA :) [19:20:14] Kibana looks definitely cool. [19:20:24] You sure? I thought the WMF wasn't collecting that. [19:20:42] :P [19:20:48] AFAIK the connection between EU and USA is still not encrypted so the NSA probably already has all our passwords anyway :P [19:20:53] multichill: there's also a tap on the office network, for example [19:20:53] :) [19:20:55] multichill: haha [19:21:07] Netflow != a tap [19:22:01] pff, https://en.wikipedia.org/wiki/NetFlow could use some clean up [19:23:23] I'm pretty sure if the NSA wants access, they don't go to such lengths, but serve a subpoena in Montgomery Street and WMF will hand over whatever they are legally obliged to like every good citizen. [19:24:39] scfc_de: maybe. we do aggressively limit the things we keep, though [19:25:20] scfc_de: re: logrotate, I'm thinking I'll have a script that crons every day, looks for logrotate.conf in each tool's directory, and then fires off logrotate jobs on to the grid. Thoughts? [19:26:20] !log local-heritage I sent out the Toolserver will die email. http://lists.wikimedia.org/pipermail/labs-l/2014-June/002672.html . I plan to drop the database p_erfgoed_p on the 21st. [19:26:22] Logged the message, Master [19:27:31] YuviPanda: Oh, I meant the logrotate of the redacted nginx log (because that needs to be synced with the log rotation of the unredacted log). [19:28:11] scfc_de: I'm talking about different things, yeah :) [19:28:15] I'm not sure if it is technically possible to logrotate access.log and error.log as the latter is also stdout/stderr of SGE. [19:28:26] !log local-heritage Did the [https://www.wikidata.org/wiki/Wikidata_talk:Cultural_heritage_task_force#Rijksmonumenten_import first steps to import the data to Wikidata]. I wonder when we can deprecate the monument database [19:28:28] Logged the message, Master [19:28:37] So you would need to restart the webservice properly (= shutdown & start). [19:29:28] scfc_de: ah, hmm. [19:30:12] scfc_de: hmm, actually - isn't access.log and error.log lighty generated? [19:30:18] if so why can't we just specify the filenames there? [19:30:50] Re "aggressively limit ...", the privacy policy doesn't commit to anything :-), and the three months for checkuser are ... long if I look at that here in Germany the balance between six months or two years data retention for telcos to ease the prosecution of organized crime/pedophiles/etc. tips in favour of the innocent :-). [19:31:17] YuviPanda: No, error.log is *also* the stdout/stderr of the webservice. Let me look where that's defined. [19:32:05] YuviPanda: $(which webservice) => line 56 ("webservice start"): "if qsub -e $home/error.log -o $home/error.log -i /dev/null -q "webgrid-$server" -l h_vmem=4g -b y -N "$server-$tool" /usr/local/bin/tool-$server >/dev/null 2>&1 ; then" [19:32:08] scfc_de: meh, you're right. I just realized I'd need to send SIGHUP or something to lighty [19:33:20] What could be interesting is to logrotate on webservice start; if tools' logs grow out of bounds, you would just have to ask the tool authors to "webservice restart". [19:34:07] scfc_de: coment on https://bugzilla.wikimedia.org/show_bug.cgi?id=66623? [19:35:38] BTW, re big tools directories, Damianz: Does cluebot really need 762 GBytes? :-) [19:35:51] YuviPanda: Will do (even though technically I believe that is a duplicate :-)). [19:36:04] scfc_de: :) [19:36:11] 3Wikimedia Labs: Request to access redacted webproxy logfiles of (Tool) Labs - 10https://bugzilla.wikimedia.org/59222#c16 (10Yuvi Panda) And I guess the format would be: 1. toolname 2. url 3. hits 4. bytes I wonder if we should actually augment this with other stats, such as: 5. error responses (non-200) 6.... [19:36:53] scfc_de: idk if I told, you I did a scan of all directories in /data/project in past [19:37:08] there is a number of tools that uses several hundreds of GB each [19:37:13] YuviPanda: https://bugzilla.wikimedia.org/show_bug.cgi?id=46471 (not totally equal). [19:37:46] petan: That's how I knew that :-). Otherwise I wouldn't have a clue :-) [19:40:31] scfc_de: I wonder if we can deploy http://developer.rackspace.com/blog/introducing-loggerfs.html [19:43:20] YuviPanda: Haven't looked in detail, but error.log is essentially a "dump" and not something structured. I don't know if that'd fit an "aggregator". [19:43:38] scfc_de: well, anything that can fit in a line by line thing can go into logstash, and I see the issue with dumps... [19:44:18] multichill: what does heritage use for logging? [19:44:39] Just stdout and sterror [19:44:46] hmm, bah [19:45:09] that'll be hard to ship to logstash [19:46:30] If we add some piping to move access.log (and other structured logs) to logstash, the size of the remaining error.log (and other logs) would remain small, I think. [19:46:39] true [19:46:51] scfc_de: I'm thinking of setting up tools-logstash-test and seeing how it goes [19:46:57] scfc_de: and deleting tools-syslog [19:47:53] YuviPanda: +1. [19:48:31] scfc_de: heh, someone already deleted syslog [19:48:39] scfc_de: hmm, any idea what tools-shadow does? [19:49:11] That's the SGE shadow master. It's unused, but Coren plans to set it up after the move to eqiad :-). [19:49:30] scfc_de: I guess now it's the move to dallas! :) [19:49:45] we are supposed to have tools in two DCs when that happens [19:49:54] should be, uh, interesting :) [19:50:30] Sounds more scary to me :-). [19:50:57] scfc_de: :D we'll have to have some form of replication for everything across [19:51:01] tools-db, redis, mongo... [19:51:02] YuviPanda: https://bugzilla.wikimedia.org/61190 ("Retire tools-shadow"). [19:51:02] SGE [19:51:22] ... and what do we get for that? [19:51:38] scfc_de: redundancy? and more resources? [19:51:41] 3Wikimedia Labs / 3tools: Retire tools-shadow - 10https://bugzilla.wikimedia.org/61190#c4 (10Yuvi Panda) So, now that we are in eqiad... [19:53:00] I've joked in the past that eqiad is on the final approach to a runway at Washington, but I think the risk of a catastrophic failure is smaller than getting all that replication up and running without disturbing someone :-). [19:53:14] scfc_de: :D [19:53:22] scfc_de: it's a year away at least anyway [19:53:52] scfc_de: we could also do something else, like move all non tools to one DC and keep one DC dedicated to just tools [19:54:52] Does Labs need that much rackspace?! [19:55:06] no, but since we're getting a new one anyway... [19:59:37] scfc_de: can you help me with an apache rewrite rule? on labs, but not tools related... [19:59:40] (ok if you don't have the time now) [19:59:47] it's fucking simple, which makes it all the more infuriating [20:00:06] rewrite everything that is '/s/' to '/wiki/Special:UrlRedirector/' [20:00:11] the obvious solution is giving me nothing [20:01:35] I'm not an expert :-); what's your present "solution"? [20:02:48] scfc_de: RewriteRule ^/s/.*$ /w/index.php [20:03:07] scfc_de: RewriteRule ^/s/(.*)$ /wiki/Special:UrlRedirect/$1 [20:03:08] actually [20:03:10] teh second one [20:03:11] is what I had [20:05:30] That looks fine to me and reminds me that I hate Apache rewrites because there is no real way to debug them :-). [20:06:25] exactly [20:06:30] I am getting 404s and nothing else [20:06:30] grr [20:06:35] I'll poke at it tomorrow [20:06:56] http://stackoverflow.com/questions/9632852/how-to-debug-apache-mod-rewrite suggests "RewriteLog "/var/log/apache2/rewrite.log"", "RewriteLogLevel 3". [20:07:18] (Or "LogLevel alert rewrite:trace6" for 2.4+.) [20:08:40] scfc_de: aha! that's more useful [20:11:19] scfc_de: hmm, split uri=/w/index.php?title=Special:UrlRedirect/1 -> uri=/w/index.php, args=title=Special:UrlRedirect/1 [20:11:23] so I guess it is being redirected [20:24:12] scfc_de: woot, works now :)