[00:00:15] i keep getting questions on talk pages if "my" projects are abandoned or not [00:00:26] but 90% of the time i am only member of them because i made them [00:00:30] after requests from others [00:00:33] so .. i don't know [00:00:51] i should get in the habit of removing myself right after creating [00:00:54] just remove yourself after creation [00:00:59] ... [00:01:08] yea, also, i kept adding myself before i had my key in root keys [00:01:27] on more bad habit to kill [00:01:59] yea, don't merge stuff and don't be member of projects :p [00:02:26] http://serverfault.com/questions/84685/early-signs-of-a-bad-sysadmin/84717#84717 [00:06:45] not sure which part you mean [00:07:56] just fun reading [01:45:30] 3Wikimedia Labs / 3deployment-prep (beta): no log in deployment-bastion:/data/project/logs from "503 server unavailable" on beta labs - 10https://bugzilla.wikimedia.org/72275#c7 (10Bryan Davis) Debugging some... On udplog.eqiad.wmflabs `sudo tcpdump udp port 8420` does not show any log packets being receive... [01:46:42] is there "projectstorage.eqiad.wmnet" just like their was projectstorage.pmtpa.wmnet ? [01:46:49] there [01:47:30] seems not [02:00:53] Any reason my not particularly memory intensive cgi script is throwing memory errors on tools-webgrid-02? [02:28:57] w930913: ran into the same issue earlier [02:52:19] I think we're simply running out of resources on the webgrid. Ima add a new node tomorrow and reduce overcommit a midge. [10:34:13] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-02.diskspace._var.byte_avail.value (37.50%) [11:02:13] !log cleaned out coredumps from tools-webgrid-02 [11:02:13] cleaned is not a valid project. [11:12:50] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [11:36:21] Coren: my job appears running in qstat [11:36:29] but it doesn't produce output anymore [11:36:44] and I can't find the process on the exec nodee [12:13:57] petan: hi [12:14:23] job 5625621 appears running in qstat but produces no more output [12:14:29] and I can't find its process [13:05:52] petan: ^ ? [13:06:02] liangent: here [13:06:08] let me check [13:06:44] petrb@tools-dev:~$ qs | grep 5625621 [13:06:45] 5625621 0.30255 php_cleanu tools.liange r 11/13/2014 05:55:53 task@tools-exec-06.eqiad.wmfla 1 [13:08:48] indeed, there is no process running on that box [13:08:53] which belongs to that tool [13:09:30] !admindocs [13:09:36] Results (Found 3): morebots, labs-morebots, hyperon, [13:09:36] @search admin [13:09:45] Results (Found 159): morebots, labs-home-wm, labs-nagios-wm, labs-morebots, gerrit-wm, wiki, labs, extension, wm-bot, gerrit, revision, monitor, alert, bz, os-change, instancelist, instance-json, amend, info, keys, blueprint-dns, bots, rt, pxe, group, pathconflict, terminology, etherpad, osm-bug, manage-projects, rights, quilt, labs-project, openstack-manager, wikitech, load, load-all, wl, ssh, start, link, socks-proxy, labsconf, resource, security, project-discuss, putty, git, port-forwarding, report, db, instance, bot, pl, projects, accountreq, nagios, puppetmaster::self, addresses, initial-login, gerritsearch, deployment-beta-docs-1, sudo-policies, svn, forwarding, labsconsole, sudo-policy, puppetmasterself, search, gitweb, htmllogs, mobile-cache, botsdocs, labswiki, google, requests, single-node-mediawiki, coren, tooldocs, wmflabs, shellrequests, rq, labs-l, msys-git, pypi, rb, proxy, venue, airport-centre, replicateddb, stats, add, pythonguy, pythonwalkthrough, unicorn, enwp, tools-bug, sudo, bible, docs, status, searchlog, tools-web, tools-admin, logsearch, StoneB, wm-bot2, wm-bot3, labstore3, pastebin, awstats, access, reboot, petan, nocloakonjoin, filemoves.py, beta, wm-bot4, wikitech-putty, toolsbeta, todo, tdb, mediawiki-instance, toolsvslabs, cloak, mob, logs, sumanah, dependency, mediawiki, jenkins, icinga, labs-putty, sal, newweb, self, lighttpd, xy, somethingtorelax, gc, explain, queue, ident, accessproblems, bug, tools-request, jira, migration, toolsmigration, relay, rofl, labsdocs, fingerprints, bastion, mysql, pywikibugs, hook, hackaton, tools-status, [13:09:45] @search http [13:09:57] Results (Found 5): bot, botsdocs, tools-admin, reboot, todo, [13:09:57] @search Documenta [13:10:01] !tools-admin [13:10:01] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin [13:11:48] btw tools-dev ssh fingerprint has changed? [13:12:03] o.O etrb@tools-master:~$ qacct -j 5625621 [13:12:04] error: job id 5625621 not found [13:12:11] yes it changed ages ago [13:12:17] when we migrated datacenter [13:12:40] hmm I'm always using tools-login [13:12:50] liangent: That may have been caused by the job DB corruption we had, but that was a couple weeks ago. [13:13:34] liangent: 'qdel -f' allows removal of a stale entry; but it might need an admin. Try? [13:13:36] Coren: the problematic job started at 11/13/2014 05:55:53 [13:14:27] liangent: ... that's... really odd. Don't touch anything while I look? [13:14:55] well I typed a qdel just now [13:15:06] and it stuck at state=dr [13:15:21] liangent: Unsurprisingly, if the job doesn't really exist. [13:21:47] Coren: still looking at it? [13:22:29] Yep; give me a minute please. [13:23:30] Ah, *there* is the issue. the sge_execd is dead [13:24:40] I probably need to add an alert for that. [13:25:18] liangent: gridengine should now have noticed your job is gone. [13:25:50] Coren: good it's gone [13:28:56] * Coren checks for execd health everywhere. [13:35:58] Ah, there were in fact two execds down. I wonder what's up with that. [13:36:49] It's not immediately noticable as this doesn't actually prevent jobs from running, only from having their statuses updated. [13:37:16] thus block new -once jobs from running [13:37:20] that's how I found it [13:37:27] * Coren nods. [13:37:47] It's inevitable that someone would end up being bitten by it; I'm still annoyed it took this long. [13:38:01] * Coren adds monitoring. [13:38:41] It's also a bit annoying that the grid master would not complain loudly about some of its nodes not reporting in anymore. [14:44:28] does anyone know if https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000236.eqiad.wmflabs aka protorel-tests in project puppet-cleanup is still in use? or who created it? trying to find out if I can delete it. [14:45:34] jzerebecki: I can check the creator, just a second... [14:47:23] Hm, it was created by user 'novaadmin' [14:47:26] That doesn't tell us much. [14:48:16] jzerebecki: but that project was marked 'mothballed' in march and the instance hasn't been touched since then. So I'd say it's fair game for deletion. [14:48:56] ok thx, i'll delete it [14:50:40] jzerebecki: btw, are you asking because you got an email from Dinu? She was going to start nagging people about these old instances :) [14:54:24] andrewbogott: yes, asked because she asked on my talk page if that whole project was abandoned. thought i still want to use the project. [14:54:42] great, glad that is underway. [14:54:57] And, of course it's fine for you to keep the project -- just trying to clean up the unwanted stuff [14:55:11] As long as the project has a running instance in it it won't be in any danger :) [14:58:58] andrewbogott: but now i deleted the last one :-/ . btw. the two old ones in https://wikitech.wikimedia.org/wiki/Nova_Resource:Nginx might not also not be needed anymore? [14:59:52] jzerebecki: For puppet-cleanup, make a note on the project documentation page and also here: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress [15:00:00] And I'll most likely remember not to delete it :) [15:00:04] I'll look at the nginx stuff [15:06:16] ok done. thx. [15:50:02] anomie, you around? [15:52:40] Cyberpower678: yes [15:53:16] So about the speed of queries from page and revision_userindex... [15:56:15] anomie, ^^ [15:57:39] So why is it so slow, when it used to be so fast? [15:58:28] Cyberpower678: When I was messing with it, I noticed it was much faster on subsequent queries over the same rows, which makes me suspect cold pages. But I'm not really an expert. I have no idea why it used to be fast. [15:58:58] * Cyberpower678 goes to Bugzilla [16:07:02] 3Wikimedia Labs / 3tools: DB replication results are slow. Act as if there are no indexes. - 10https://bugzilla.wikimedia.org/73420 (10Cyberpower678) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier Lately, the DB has required several tens of seconds to supply the results of a query that took about 2 s... [16:07:09] anomie, ^^ [16:09:41] Cyberpower678: "acts as if there are no indexes" is probably too strong. Did you actually try an unindexed query that has to scan the whole revision table for comparison? [16:10:52] The last time I did that, the query ran for 634 seconds. Archive did have an index problem, and had similar load times. [16:13:15] 3Wikimedia Labs / 3tools: DB replication results are slow. Act as if there are no indexes. - 10https://bugzilla.wikimedia.org/73420 (10Cyberpower678) p:5Unprio>3Normal [16:17:47] Cyberpower678: Since 18s is nowhere near 634s, "acts as if there are no indexes" is blatantly wrong. [16:18:30] Clarification. Missing indexes in archive resulted in a similar query time as in it does now in revision. [16:20:58] Cyberpower678: revision has somewhere around 12 times as many rows, so that's not a very good comparison. [16:21:26] True. I'll change it in a sec. [16:22:17] 3Wikimedia Labs / 3tools: DB replication results are slow. - 10https://bugzilla.wikimedia.org/73420 (10Cyberpower678) [16:24:08] anomie, can you riddle me this. Why are placeholder images a sillouette of Psyduck? [16:24:15] Isn't that a copyvio? [16:24:27] On Phabricator that is. [16:26:54] Cyberpower678: I have no idea. [17:27:05] Hi! I'm new here. I'm working on a project and wondering if it would be possible to download *just* the IP addresses associated with both anonymous and logged-in edits to wikipedia. [17:28:56] I'm hoping to visualize which university campuses contribute the most to the wikipedia project. Do the Wikimedia APIs support such an export? [17:30:37] nicholas: IPs of logged-in users are considered private information, so we almost certainly cannot provide those. IPs of anonymous edits should be available, although I don't know exactly where they accumulate. [17:32:33] nicholas: It's probably worth lurking here for a while in hopes of getting a different and/or better answer :) [17:33:52] So I am unable to manage instances or web proxies of a project I'm an admin for. [17:33:58] I set the filter but nothing shows up [17:34:12] even though I know for instance I have an active web proxy that is working presently [17:34:35] this has happened before and I got removed and then readded as a project admin [17:34:36] nicholas: try asking in #wikimedia-research [17:35:02] and that seemed to have worked although might have been a coincidence. [17:35:12] anyway i need to make a new instance and a new proxy and I can't :/ [17:38:40] andrew & chris : ok will do ! thanks :) [17:39:09] mvolz: try logging out and logging in again? [17:39:13] mvolz: We're still having session bugs of some sort… log [17:39:18] yeah, what legoktm said :( [17:39:43] I am too discouraged by this bug to work on it today. It bites me frequently, though [17:44:30] * hashar looks at the Horizon [17:50:56] !log logstash Added anomie as a project admin [17:50:59] Logged the message, Master [17:51:30] anomie: I haven't touched things in that project for quite a while so it may be a mess :( [17:51:52] anomie: If you need help with it just shout [17:53:01] bd808: A quick primer on how to actually test something? [17:53:11] heh. hope? [17:53:32] I never setup a wiki to feed data into that cluster I don't think [17:53:40] * bd808 looks around [17:54:36] anomie: I was mostly using it as a place to test the puppet config and to make sure the rules compiled [17:55:51] It runs a self-hosted puppet master on logstash-deploy that you can use to mess with puppet config. [17:56:28] logstash-dev runs logstash, elasticsearch and kibana [17:57:55] For wha tyou are doing we probably need to setup third node running labs-vagrant and figure out how to have it send logs into the logstash-dev udp2log endpoint [18:01:29] bd808: At this point, I'm thinking just hacking up a logstash configuration that reads from stdin and prints to stdout with some filters in between and see that my "clone" filter is actually spitting things out sanely. [18:01:36] !log logstash Updated kibana to latest master [18:01:40] Logged the message, Master [18:02:08] anomie: cool. Yeah just do that on logstash-deploy then [18:02:20] let's disable puppet there so it doesn't mess with you [18:03:55] bd808: I'm not seeing the 'logstash' binary on logstash-deploy? [18:04:06] !log logstash disabled puppet on logstash-dev [18:04:07] Logged the message, Master [18:04:12] anomie: Opps I meant logstash-dev [18:04:36] Puppet is off there now [18:04:59] you can just add config to /etc/logstash/conf.d [18:07:08] stdout and stderr for the logstash process are logged to /var/log/logstash [18:10:48] bd808: I'm just doing "/opt/logstash/bin/logstash -f ~/test.cnf < ~/test.log" on the command line. Should work well enough, I think. [18:11:02] cool [18:23:38] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-04.diskspace._var.byte_avail.value (33.33%) [18:26:32] !log tools cleaned out core dumps on tools-webgrid [18:26:34] Logged the message, Master [18:36:58] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [18:45:50] YuviPanda: I think we want to turn core dumps off entirely on grid nodes, actually. [18:46:05] Coren: yeah, hiera [18:46:12] Coren: need to have a 'enable dumps' and turn it off on tools [18:48:37] Coren: I'll get that change done now [20:02:52] !log deployment-prep Cherry-picking https://gerrit.wikimedia.org/r/#/c/173336/ for testing in logstash [20:02:56] Logged the message, Master [20:21:48] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-webgrid-01.diskspace._var.byte_avail.value (33.33%) [20:23:09] !log tools cleared out coredumps on tools-webgrid-01 to free up space [20:23:12] Logged the message, Master [20:33:05] YuviPanda: find -name "core" | xargs rm? :P [20:33:08] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [20:33:17] valhallasw: :) they're always in /var/tmp/core [20:33:20] valhallasw: and that's problematic [20:33:26] valhallasw: that change to put them there was merged a few days ago [20:33:31] valhallasw: which caused the avalanche of such errors [20:34:06] YuviPanda: really? I think I've seen coredumps around in working directories :/ [20:34:15] but maybe that wasn't on tools. Not quite sure. [20:35:10] valhallasw: yeah, this change happened very very recently [22:00:11] Coren: did you set ulimit -c on tools instances? [22:00:17] -c is 0 but there are still coredumps. [22:00:39] That'd only apply to new processes. [22:01:02] no I mean, did you set it recently? [22:01:08] More specifically, new process groups. So anything continuous that currently forks() then crashes would still leave coredumps. [22:01:11] I am wondering if it was 0 all this time and core dumps were still happening [22:02:50] quick question Coren the wmf version of trusty is the x64 Server version correct? [22:03:00] Betacommand: Aye. [22:03:13] Coren: thanks [22:03:15] YuviPanda: Hm, no, I was working on something else so it wasn't me. Odd. [22:03:31] Coren: indeed, so now I'm confused as to how they happened at all. [22:03:46] Coren: Getting ready to migrate my webserver to linux [22:04:00] so thanks for the info [22:04:44] YuviPanda: The hard limit is "unlimited"; perhaps something is raising the ulimit? Python, maybe? [22:04:59] YuviPanda: Do you know what core dumps? [22:05:08] nope, I deleted them all without preserving, sadly [22:05:14] but has been happening about twice a day now [22:05:20] perhaps lighty was raising them? [22:05:27] but I think scfe_de cleaned some up from exec nodes too [22:05:35] Coren: they were definitely not just one tool [22:05:44] Possibly. Of course, the obvious solution is to simply set the hard limit at 0 [22:06:25] yeah, I could put that in /etc/profile.d [22:06:37] hmm, I wonder if SGE launched jobs would pick that up [22:06:57] I could also put it in /etc/security/limits.conf but that file is a bit of a PITA to manage atm [22:07:39] I'm pretty sure jobs never source login scripts; and only continuous jobs are pretty much guaranteed to source bashrc [22:07:49] So there aren't that many alternatives. [22:07:58] But figuring out what raises its ulimit would help. [22:08:08] yeah [22:08:15] e.g.: lighty might do it when you're in verbose, etc. [22:08:24] Which should be easy to turn off. [22:08:42] Or Python does it, or PHP (both of which are common to many tools) [22:09:07] Coren: http://redmine.lighttpd.net/projects/1/wiki/Server_core-filesDetails [22:09:28] modules/toollabs/files/lighttpd-starter:server.core-files = "disable" [22:09:36] so it's not lighty [22:09:37] So probably not lighty [22:09:56] so probably the fcgi processes it's running [22:10:02] should've preserved some of the dumps, grrr [22:10:22] Meh, we'll get the next one I expect. [22:10:40] yeah, but I don't want to be checking and clearing things out through the weekend :) [22:11:06] I'll keep an eye out for the next one and gdb it to see what creates it. [22:11:17] hmm, ok! [22:11:24] I'll probably be on a bus tomorrow. [22:11:49] Squeezing out the last bits of bohemian lifestyle while you can? :-) [22:11:49] Coren: there's a task, https://phabricator.wikimedia.org/T1259 [22:12:16] Coren: yeah, was in the himalayas, now heading to a beach hacking place, then back to himalayas for the 'winter' :) [22:19:50] YuviPanda: I'm looking forward to your upcoming memoir [22:20:12] chrismcmahon: :D I just realized I haven't been in one city/village for more than a month for about 16 months now almost [22:20:55] YuviPanda: and you weren't exactly living a retired life before that :-) [22:21:04] chrismcmahon: heh :) [22:21:31] chrismcmahon: it's fun, and I'm slightly worried how I'll adapt to 'live in one place' in SF. [22:21:55] chrismcmahon: although, I move in end March, May is French Hackathon, I'd like to spend some time in UK in Jun/Jul after, then right after is Mexico... [22:22:01] chrismcmahon: so this pattern might not actually end [22:23:10] YuviPanda: yep. and after a time in SF, you might go elsewhere. I know I've had a blast bopping around the American West since I sold my house in Colorado. It can be a nice way to live. [22:23:26] chrismcmahon: yeah, indeed. [22:23:48] chrismcmahon: plus Rent in SF is crazy. I live fairly like a king/US Congressman here in India, and Rent alone in SF would be almost double of what I spend for everything here... [22:24:27] YuviPanda: Likewise. I'm renting a super-nice house in Tucson for the price of a 1BR apartment in SF. [22:24:43] I wonder what H1B has to say about that, tho [22:26:41] YuviPanda: when I was traveling a lot, I had a PO Box in a town in California where I knew I would be passing through every month. Might be an option. [22:26:58] hmm, true [22:27:06] let's see. I still have a few months [22:27:38] The Foundation doesn't have the budget to get me to move to SF. :-) [22:27:58] Coren: me neither [22:28:34] heh [22:28:40] And if they paid me enough that I'd be willing to move to SF, I'd put that money in a telepresence 'bot instead and /not/ move to SF. :-) [22:28:57] Coren: chrismcmahon I don't know if I'll want to live in SF forever, but I consider it just a part of my 'move around and live everywhere!' thing. [22:29:00] except slightly longer [22:30:37] Coren: hmm, do we have any way to just raise the hard limit? [22:30:54] other than /etc/security/limits.conf? [22:31:17] You mean lower? I don't think there is, at least no guaranteed way. [22:32:23] err, yeah, lower [22:32:31] what was the status quo before? [22:32:37] they just dumped core on PWD? [22:33:24] That's a kernel tunable. [22:33:33] right [22:33:36] so there's a patch now [22:33:39] that's setting it to /var/tmp [22:33:41] which is why we have this [22:33:45] I wonder what the default was [22:33:46] * YuviPanda checks [22:33:49] (And, btw, it's 'cwd' - 'pwd' is the command that prints it.) [22:33:56] gah, yes. [22:36:35] Coren: this is also why we should have bigger /vars, btw :) [22:36:44] one single core dump can fill it up now [22:37:24] (03PS1) 10BearND: Update SDK download/update scripts for new location [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/173440 [22:37:26] (03PS1) 10BearND: Update gitignore [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/173441 [22:37:29] Having crap fill up a filesystem is not a sign that you should increase the size of the filesystem, but that you should stop filling it with crap. :-) [22:37:46] Making it bigger just means we'll have to clean more of it less often. [22:38:12] YuviPanda: yeah, I finally got to push the changes to gerrit ^ [22:38:15] bearND: :D [22:38:23] YuviPanda: would you like to review? [22:39:02] (03CR) 10Yuvipanda: [C: 032 V: 032] Update SDK download/update scripts for new location [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/173440 (owner: 10BearND) [22:39:13] (03CR) 10Yuvipanda: [C: 032 V: 032] Update gitignore [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/173441 (owner: 10BearND) [22:39:14] bearND: done [22:39:37] Coren: tch tch, I'd rather clean them up much less often than have to keep getting interrupted by it *this* often [22:39:49] Coren: plus prod has big /var, and ours is *really* tiny [22:39:58] so things that work great in prod, like this /var/tmp change, suck here. [22:40:21] YuviPanda: That's because prod uses a pre-2004 filesystem layout that filles /var with crap. See point (a) above. :-) [22:40:38] well, I've assigned https://phabricator.wikimedia.org/T1259 to you, Coren :) [22:40:44] Heh. :-) [22:41:29] :) [22:42:18] YuviPanda: thanks a bunch! Had to do some git acrobatic to get it push from labs. Also had to copy a commit-msg hook file from my box. [22:42:53] bearND: git review -s sets up the commit hook, I think [22:43:09] YuviPanda: there was no git review install IIRC [22:43:19] bearND: 'sudo pip install git-review' [22:43:37] YuviPanda: doh! [22:44:12] apt-cache show git-review [22:44:27] YuviPanda: ok, installed it now [22:44:31] :) [22:45:43] mutante: not sure what you're trying to tell me [22:46:11] bearND: you can use 'sudo apt-get install git-review' too [22:46:13] instead of pip [22:46:44] ah, ok [22:47:03] thanks, guys! [22:47:35] i guess i was trying to see even better than using pip is just using apt [22:47:40] as long as the version is new enough [22:47:41] to be usable [22:48:01] we kind of frown on pip because it's a second way to install software, when we already manage packages [22:48:09] but sometimes you need it because packages are old [22:48:09] andrewbogott: is there docs on resetting 2FA? The one on wikitech seems to want me to input current 2FA token, which I can't [22:48:20] * YuviPanda tch tchs [22:48:35] YuviPanda: https://wikitech.wikimedia.org/wiki/Password_reset, at the bottom [22:48:56] andrewbogott: yeah, better to do it myself [22:49:26] YuviPanda: of course there's not a user-friendly GUI for it, because if you could reset your own 2fa that… would defeat the purpose :) [22:49:46] andrewbogott: well, I just did it on Google :) it asks you for your password instead [22:49:59] That doesn't make sense… the password is the first factor [22:50:14] So if you only need one factor then why…? [22:50:27] I think when I've done that with google there was an email involved, or a text , or something [22:50:50] andrewbogott: nope, you are already logged in, and *then* you want to change it, and it asks you for your password again [22:50:55] mutante: I see, I'm using mostly OS X lately, hence was leaning towards pip since it platform independent [22:51:03] Ah, sure, if you're already logged in... [22:51:07] hm, that sort of makes sense. [22:51:34] andrewbogott: yeah, so in our reset screen instead of verifying token we should ask for password [22:51:41] bearND: i see, yea, it's fine, we just wouldn't want to use it in prod [22:52:00] yeah, but it's labs, so it's ok :) [22:52:14] unless you want to test close to prod :) [22:52:37] * YuviPanda wonders if/when mutante will get past his illusion of 'labs is just this thing we use to test prod' [22:52:38] :) [22:53:08] mutante: YuviPanda: ok, learned something new today. Thanks. :) [22:53:35] andrewbogott: https://wikitech.wikimedia.org/wiki/Password_reset doesn't tell me on which machine all of this happens :) [22:53:37] andrewbogott: virt1000? [22:53:46] yeah, virt1000 [22:53:50] probably a good idea to add that :) [22:54:32] andrewbogott: ok, so passwords have changed as well now, so I can't actually get into mysql [22:54:56] really? I think if you're root on virt1000 it just works [22:55:00] if not, I will look [22:55:23] andrewbogott: yup, am on virt1000 [22:55:29] andrewbogott: aha, bam [22:55:39] andrewbogott: docs said mysql, you need 'sql' [22:55:40] * YuviPanda edits [22:55:49] thanks [22:56:21] woah [22:56:25] I'm user id 49 on wikitech?!?! [22:56:25] wt [22:58:12] andrewbogott: done, thanks! [23:30:06] https://gerrit.wikimedia.org/r/#/c/173456/2/modules/lvs/manifests/configuration.pp [23:30:09] ^ is there LVS in eqiad labs? [23:30:17] Hi, do you know if I can use mwparserfromhell to match the content of a page between {{foo}} and {{bar}} ? [23:32:43] Earwig: ^ [23:33:41] alkamid: uhh... you could do it, yes... but if they're not in the same "context" in the page, that would get complicated [23:33:54] like, if you had == {{foo}} == \n {{bar}}, that would be difficult [23:34:38] well [23:34:54] Earwig, thanks. I'll stick to my custom regexps then, at least for this task [23:35:42] but could you elaborate on "context"? [23:36:28] okay, so mwparserfromhell stores individual elements of wikicode as a list of nodes [23:36:54] yup - I got that far (: [23:36:58] so when you have the text "{{foo}}{{bar}}", that would be stored as a list of the two templates "{{foo}}" and "{{bar}}" [23:37:11] but if you have "{{foo|{{bar}}}}{{baz}}" [23:37:21] then it's still a list of two templates [23:37:26] but the first template has a template inside of it [23:37:38] so {{bar}} and {{baz}} are not stored in the same list, i.e., not in the same context [23:38:45] {{bar}} is inside of {{foo}}, so if you wanted to get the content between... say, {{bar}} and {{baz}}, you'd end up with "}}", which doesn't really make sense [23:38:58] and if you wanted mwparserfromhell to do that, it couldn't [23:40:14] hmm, I need to rebuild the shinken package [23:41:00] but why your first example illustrates a problem with the context? " == {{foo}} == \n {{bar}}" [23:41:19] because the {{foo}} is contained within a heading node [23:41:27] oh, right [23:45:54] Earwig, so if they are within the same context, I should just search the list for my templates and join the elements between them? [23:46:14] I mean, mwparserfromhell does not have a dedicated method for this? [23:46:16] right; you can use code.index("{{foo}}") for that (or whatever the template node is called) [23:46:29] no, no dedicated method because I never thought of a use case for this :) [23:47:22] haha, if you're interested, on some wikis (like pl.wiktionary) pages are divided in sections with templates [23:47:32] like "definitions", "etymology", "translations" etc. [23:47:42] so... i = code.index(first_template), j = code.index(second_template), and then code.nodes[i:j] gives you a list of the nodes that you want [23:47:48] ah. [23:50:12] would it be faster than re.search(r'{{foo}}(.*?)\n{{bar}}', re.DOTALL) ? [23:50:40] re.search(r'{{foo}}(.*?){{bar}}', re.DOTALL) [23:54:50] alkamid: regex is fairly fast in general, and you don't need to parse the whole page, so I would guess yes [23:54:54] but you're using python anyway, so I wouldn't be too concerned about speed [23:55:12] sorry, I meant to say I would guess no (i.e. that regex is faster) [23:55:41] well, since I'm parsing whole dumps, I'm concerned about speed (: [23:56:24] I would suggest the regex approach here simply because it's easier and you don't have to worry about the nodes-inside-of-nodes thing [23:56:41] okay, thanks [23:56:49] unless you're doing other stuff with mwparserfromhell, in which case it might make sense