[00:01:16] I don't have any upcoming [00:01:20] we'll have the gerrit one coming up [00:01:58] * Danny_B|backup is looking for practical workshop/hackathon in europe [00:02:00] 07/27/2012 - 00:02:00 - Changing group ownership of /export/keys/mwalker to 550; was set to 500 [00:02:06] but we don't have the date yet, so, I'll leave it out [00:02:23] 07/27/2012 - 00:02:22 - Updating keys for mwalker at /export/keys/mwalker [00:04:43] The gerrit one isn't open to the public anyway. [00:04:56] oh [00:05:16] Well, Erik's talking about maybe having some call-in thing...but it's not like a tutorial or workshop. [00:05:27] It's "bunch of people getting together to talk about how we want gerrit to improve" [00:05:30] sure [00:12:42] so, that's why they don't want public to attend [00:12:58] they fear being overwhelmed by users telling them that it sucks ;) [01:12:23] 07/27/2012 - 01:12:23 - Updating keys for jhsoby at /export/keys/jhsoby [08:46:02] !log deployment-prep migrating jobrunner06 to use the /data/project for uploads [08:46:04] Logged the message, Master [09:51:12] !log deployment-prep {{bug|38749}} jobrunner06 : removed /usr/local/apache and /mnt/upload6 empty dir. Downgraded PHP manually. Rerunning puppet. [09:51:13] Logged the message, Master [12:42:04] paravoid: I got deployment-jobrunner06 dead somehow :/ It does not report any console output despite two reboots [12:42:05] https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=deployment-prep&instanceid=i-0000031d [12:42:07] :D [12:48:03] hashar: looking [12:49:46] paravoid: I guess after wikidiff is sorted out :) [12:52:11] hmmmm [12:52:14] stuck at grub [12:52:34] and that happened to patrick yesterday too [12:52:59] that's strange [12:54:16] paravoid: ahhhh [12:54:20] paravoid: so I got the root cause [12:54:32] paravoid: I did a diet upgrade which pushed a new linux-virtual package [12:54:55] debconf asked about grubs/menu.lst or something and whether I wanted to keep the local version (default) or use the package provided one [12:55:00] I pressed the default (keep local) [12:55:02] that must be it [12:55:09] sorry should have told you about that message before :/ [12:55:32] there is similar issue when updating some PAM component where we want to keep the local version IIRC [12:55:39] while grub it seems we want the package one [12:55:51] paravoid: I can just recreate the instance if it is too long to fix [13:01:22] <^demon> paravoid: Since you're the debian guru, I'm wondering if you could help me put together a package. I want to replace gitweb (it sucks) on the gerrit.wm.o host with gitlist (much prettier, demo: http://git.gofedora.com/). It's just a simple tar extraction + apache config. [13:02:34] hashar: don't worry :) I just want to track this and fix it for anyone else too [13:02:42] hashar: can I reboot the machine? [13:02:45] paravoid: sure [13:02:50] paravoid: it is not doing anything [13:02:57] I am migrating it to use /data/project so go ahead :-D [13:02:59] oh? it didn't came up [13:03:04] hm. [13:03:10] paravoid: it can die too. It is fully puppetized [13:08:46] hashar: should be okay [13:08:53] there were NFS mounts in /etc/fstab [13:08:57] boot was halting [13:09:07] then ubuntu has this thing where when it halts it stops grub the next time or something [13:09:14] I commented them out, everything seems okay now [13:10:24] I should have added some mount { deployment-nfs-mc * ensure=>unmounted } [13:10:32] paravoid: thanks!! [13:11:40] !log deployment-prep removing all deployment-nfs-memc entries from /etc/fstab [13:11:41] Logged the message, Master [13:14:56] !log deployment-prep restarted job runner on jobrunner06 [13:14:57] Logged the message, Master [13:31:36] paravoid: can you please ping me when you have time to discuss about my ugly hack to get syslog-ng on dbdump ? https://gerrit.wikimedia.org/r/#/c/16661/ :-D [13:32:18] I have time but I remain reluctant [13:34:27] my position is that it is not uglier than the existing one and has no impact on production I see [13:34:45] not sure how we could fix that up properly though [13:35:21] but maybe puppet has a way to remove a class form a node after it got included [13:36:24] no it doesn't [13:36:28] and for a good reason :) [13:37:07] or we could create a base::remote-syslog::disabled class that would ensure rsyslog package is absent [14:08:35] paravoid: ping ^^^^ :D [14:59:28] paravoid: so what do we do with the hack ? [16:18:10] is there a repository for puppet we should use? [16:19:18] !puppetmaster [16:19:18] https://labsconsole.wikimedia.org/wiki/Help:SelfHostedPuppet [16:36:31] I am out, see you on monday [18:03:13] Hi. Can anyone help me sort out a port forwarding issue? [18:03:33] I hav followed the guide in: https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_services_using_port_forwarding [18:03:33] . [18:03:46] But, everytime I get a timeout error. My local machine does not run any firewall. [18:03:53] Socks proxy method is not working too. [18:04:08] So, I'm stuck now. [18:10:17] what OS, what did you do? [18:12:47] OS is Windows 7 (unfortunately) but I'm using cygwin for ssh. [18:13:48] I typed like this on the cygterm: ssh adminxor@bastion.wmflabs.org -L 8080:i-0000035e:80 [18:14:17] i-0000035 is my instance. [18:14:57] is the instance running something on port 80 ? [18:15:10] Yes, apache2 as usual. [18:15:34] hm... channel 2: open failed: connect failed: Connection timed out [18:15:56] Exactly. [18:16:17] is there some security rule precenting access to that vm from bastion? [18:17:00] your problem is not in the forwarding, but in accessing from bastion [18:17:07] ~$ wget http://i-0000035e [18:17:07] --2012-07-27 18:16:34-- http://i-0000035e/ [18:17:07] Resolving i-0000035e... 10.4.0.223 [18:17:07] Connecting to i-0000035e|10.4.0.223|:80... failed: Connection timed out. [18:18:59] I think so. But, how do I check them? [18:19:07] I did "netstat -tulpn | grep ESTAB" to see the established connection, no entry for port 80. [18:19:08] Tried to do a traceroute as well, but the packet gets blocked even before I reach the instance. [18:19:29] at labsconsole [18:19:42] see the security rules of the project [18:20:06] I'm not sure if you can switch them for a running instance, though [18:23:21] Okay. So I go to Manage Security->Add New Security Rule ...that way? [18:24:12] modifying rules on a group that's on an instance will affect the instance [18:24:28] the only thing that doesn't work is adding a new group to an already running instance [18:24:31] !security | adminxor [18:24:31] adminxor: https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [18:24:59] ok. gotta go [18:26:11] Thanks. [18:58:40] Added the new security rule, rebooted the instance. Working smooth now. Thanks Platonides, Ryan_Lane [19:32:47] Ryan_Lane: You must be busy. I hope you remember the Syslog-collection project we talked about. [19:32:48] If you could point to http://i-0000035e/loganalyzer and check, it would be great. It's nothing fancy yet. Just a start, but I would like to have your view on this. [19:33:02] cool [19:33:27] hm [19:33:30] not working for me [19:33:36] do you have a firewall rule open? [19:33:36] oh [19:33:37] wait [19:33:39] ignore me [19:34:51] adminxor: is this open source? [19:35:12] Yes of course. It's from the same community as rsyslog [19:35:17] ah [19:35:18] cool [19:38:16] You talked about saved searches. [19:38:17] Can you please explain a little? Maybe I can come up with something. [19:57:33] adminxor: basically, we have mediawiki logging errors and such [19:57:39] there are certain errors that we often grep for [19:57:58] it would be nice to have a search that we can just click on for that specific error type [19:58:31] so, we can create the search in the search box, then say "save this search as 'fatal mediawiki errors'" or something like that [19:59:09] Okay. Is it possible to get a sample error string? [20:21:16] adminxor: a coworker is telling me he used this at another place and it doesn't scale :( [20:22:53] adminxor: hm. RoanKattouw would be able to give you one [20:23:05] adminxor: but really, any search string that could be saved would work [20:23:46] the idea is that I have a really awesome search, and I want to share it with others [20:23:53] or that I just want to save it for myself [20:24:03] so, private and public saved searches would be cool [20:24:05] j^: I think -bastion is dead. I never managed to connect to it [20:24:18] j^: I opened a bug to get petr to look at it and eventually rebuild it [20:24:25] j^: can you connect there? [20:24:28] j^: /data/project is an automount on every instance in the project [20:24:42] just ls /data/project and it'll appear [20:27:30] grmblblbl [20:27:34] ssh deployment-bastion.pmtpa.wmflabs [20:27:40] permission denied (publickey). [20:27:53] Ryan_Lane: Okay, looks like I need to dig some php codes. [20:27:55] Also, if you could tell me in which areas thi loanalyzer does not scale, maybe I could find something better. [20:28:23] this loganalyzer ... oopss [20:28:26] j^: also I did the /data/project migration yesterday but completely forgot about notifying everyone about it :( [20:28:32] j^: might have broken some instances [20:28:39] adminxor: he said when they used it that it was slow as hell after 3 months [20:28:52] and they removed everything but the routers and switches [20:28:58] adminxor: maybe try logstash [20:29:17] !log deployment-prep j: deployment-bastion: removing all deployment-nfs-memc entries from /etc/fstab [20:29:20] Logged the message, Master [20:29:29] j^: great :-D [20:29:47] j^: you need to then amount /usr/local/apache (if it is there) and /mnt/upload6 and /mnt/thumbs [20:29:48] Cool. Thanks. [20:29:54] j^: then delete the local directories [20:30:24] j^: the puppet classes are nfs::upload::labs and nfs::apaches::labs but they are usually installed via a role class already [20:30:29] the most unfortunate thing about logstash is that it's jruby [20:30:41] Ryan_Lane: I got a permission denied on deployment-bastion host. I just don't understand it :/ [20:30:50] the plus about jruby is that it's easier to install and manage [20:31:26] hashar: try now [20:31:43] it says accepted public key [20:31:44] Ryan_Lane: oohhhh you are the magic guy :-D [20:31:49] I didn't do anything [20:32:01] all I did was tail the auth..log [20:32:12] that fixed it :-]]]]]]]]]]]]]] [20:32:40] heh [20:32:44] somehow doubtful :) [20:32:52] we need a bastionhost::labs class :D [20:34:00] my bug report close comment is: "Ryan fixed my connection issue by using tail -f auth.log. He is so great that by just looking at an issue it magically fix." [20:35:02] j^: looks like you magically fixed -bastion :-) [20:40:16] heh [20:40:22] Ryan_Lane: if you get any spare time this afternoon, would you mind commenting on https://gerrit.wikimedia.org/r/#/c/16661/ ? it is to have -dbdump set up using syslog-ng just like nfs1 / nfs2 [20:40:44] paravoid & mark did not like it cause it is a hack :D [20:41:24] I don't like this [20:41:34] me neither [20:41:52] I'd like less labs specific stuff in the manifests [20:41:55] but the current state is already a hack [20:41:58] especially instance specific [20:42:05] fully agree [20:42:11] but I have no idea how to properly fix that [20:42:29] and I am not willing to rewrite the whole exception properly :-) [20:42:46] so I ended up sending an easy hack that get the stuff done (patchset 1) [20:42:58] patchset 2 is another hacked approach i have no idea if it work [20:43:20] I haven't managed to simulate a puppet run on an arbitrary node on my local machine [20:43:46] i wanted to do something like having puppet to generate catalog for node nfs1.pmtpa.wmnet [20:51:39] Ryan_Lane: could you add your mood on https://gerrit.wikimedia.org/r/#/c/16661/ please? Just "I don't like how it uses an instance name" would be fine [20:51:54] then next week I will try to get a solution with Faidon and mark [20:51:57] :) [20:52:51] j^: did you get a job runner to run the web video transcode jobs ? [20:53:54] indeed [20:54:26] j^: i lost the track but did you get the debian package + shell script merged in puppet ? [20:55:01] so does video02 use puppetmaster:self ? [20:55:56] https://gerrit.wikimedia.org/r/#/c/16501/ ? [20:56:31] Ryan_Lane: thanks !!!! :-] Have a nice week-end [20:57:01] thanks :) [21:01:31] hashar, do you care if I rebuild (again) the captcha using the official list ? [21:01:47] Platonides: what do you mean by official list ? [21:02:06] the words used by wmf [21:02:12] sure that will be great [21:02:13] Platonides: please make sure to delete the old captcha and eventually you might want to change the secret key [21:02:21] Platonides: and please do log the command you run :-D [21:02:36] this is what I did the last time: [21:02:41] installed python-imaging and wamerican on deployment-dbdump [21:02:41] cat /usr/share/dict/american-english | tr '[A-Z]' '[a-z]' | grep -E '^.{4,5}$' | grep -vE '(.)\1$' | grep -vE '^(.)\1' | LAN [21:02:41] G=C grep '^[a-z]*$' | grep -v ^f > WORDS.txt [21:02:41] time python captcha.py --output /mnt/upload/private/captcha --wordlist WORDS.txt --key 8U-W#L@ZXz --font /usr/share/fonts/tru [21:02:41] etype/ttf-dejavu/DejaVuSans.ttf --count 200 --dirs 3 [21:02:58] ohhh [21:03:12] I just used /usr/share/dict/american-english with a --count of 1000 [21:03:19] I saw [21:03:27] that was the missing secret key, too :P [21:03:31] hehe [21:03:41] so do whatever you want :-) [21:03:49] I don't know which font is better [21:03:54] but make sure to log the full captcha.py command (excluding the secret key) [21:03:58] it probably doesn't matter [21:04:18] yeah probably not [21:04:25] just pick whatever font you like :-] [21:04:33] not that there are a lot of them... [21:04:52] AS LONG AS IT IS NOT MS COMIC SANS !!!!!!!!!! [21:04:57] I may try adding some per-wiki captchas, now that that thread came again to live [21:05:00] xD [21:05:01] or I will ban you ;-) [21:05:09] it's tempting to do [21:05:19] http://hyperkind808.files.wordpress.com/2010/07/340x_no_comic_sans_please_thumb.jpg?w=369&h=473 < [21:05:20] what about using comic sans for simplewiki? [21:05:21] yeah per wiki would be great [21:05:37] would need a patch in ConfirmEdit to be able to react on a per $wgLang basis [21:05:37] xDD [21:05:43] oh [21:05:58] that would be a VERY NICE addition [21:06:07] I was thinking in basing the folder path in the db name [21:06:11] as done for other bits [21:06:12] and I feel beta is exactly the place where we can conduct some tests [21:06:22] na use the lang [21:06:37] cause I as a french user would expect french captcha even if I am login to the hebrew wiki :-) [21:06:46] I found a quote by Tim on these internationalised captchas "It's possible, but it would only solve half the problem -- the other [21:06:46] half being that captchas fundamentally suck. [21:06:49] Damianz: exactly :) [21:07:11] I swear some captchas are easier to solve by bots than me [21:08:02] heh, I made a javascript to solve the mathcaptcha for me [21:08:13] it got boring very fast [21:13:33] j^: https://gerrit.wikimedia.org/r/#/c/16501/ seems good overall [21:13:43] j^: would love to have the virtual limit to be a JR change [21:13:54] j^: err a JR env variable and a puppet class parameter [21:14:09] j^: trivial fixes, you get some trailing whitespace. You can check them easily using git diff --check [21:14:35] the virtual memory limit I know that Tim talked about, would need to remember to ask him about it [21:14:52] it is fine to me anyway since the transcode jobs will be on dedicated hardware [21:14:59] so I did not add a comment [21:15:37] j^: overall nice change :-] Thanks for puppetizing all that mess [21:15:57] Platonides: ohh make sure to test the captcha [21:16:06] Platonides: you have some logs in /home/wikipedia/logs/captcha.log [21:17:37] Platonides: also if you work on per lang captcha, make sure to catch with the i18n team. You will need to select specific fonts per scripts [21:17:52] Platonides: and hebrew / arabic / hindi / thai etc captchas might not play well [21:18:20] Platonides: definitely contact Siebrand about it. I am sure he will just LOVE the idea and will probably be able to provide you some resources from his team provided he got any [21:18:35] j^: o [21:18:53] j^: so just leave the code around for later and have the limit disabed [21:18:53] m [21:19:12] maybe "ulimit -v 0" cancel it so the default could be just 0 [21:19:39] fine [21:23:55] could get upstream involved too [21:24:01] they might be interested in crash reports [21:24:11] anyway, I am out for now , will be back on monday [21:24:33] and thanks Jan for all the stuff you did to get TMH polished :-))) [21:33:37] !log deployment-prep installed in deployment-bastion the packages joe, python-imaging and wamerican [21:33:38] Logged the message, Master [21:33:47] or so I hope... [21:34:00] dpkg isn't too happy [21:42:47] !log deployment-prep removed generated captcha files [21:42:48] Logged the message, Master [21:48:45] aargh! [21:50:26] silly apt-get removed some packages ecause they were "orphaned" and now complains that there are packages with unmet dependancies [21:51:50] wtf?? /usr/local/apache is missing? [21:52:08] why did apt-get remove wikimedia-task-appserver ? [21:53:36] Platonides: This used to happen in production all the time [21:53:57] how nice [21:54:10] do you have the recipe for fixing it? [21:54:19] cookies [21:54:23] looks like I will need to manually reinstall all the packages... [21:54:40] re-run puppet? [21:55:38] at least we succeeded in making a copy of production environment [21:55:40] bugs included [21:56:44] !log deployment-prep apt-get removed wikimedia-task-appserver in deployment-bastion :( [21:56:45] Logged the message, Master [21:57:11] "dependency problems, but removing anyway as you requested" [21:57:20] he could have asked me for confirmation! [21:57:55] damn [21:58:32] probably [21:58:59] it seems to have removed the docroot folder from nfs [21:59:05] yeah [21:59:26] symlink /data/project/apache to /usr/local/apache < hasher's email [22:00:58] * Platonides is looking for the manual puppet command [22:01:08] puppetd -tv [22:01:44] ah, puppetd [22:01:57] puppet -tv was giving Could not parse options: ambiguous option: -tv [22:02:07] Yeah... [22:02:14] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: File[/mnt/thumbs] is already defined in file /etc/puppet/manifests/nfs.pp at line 117; cannot redefine at /etc/puppet/manifests/nfs.pp:152 on node i-000002bd.pmtpa.wmflabs [22:02:14] warning: Not using cache on failed catalog [22:02:14] err: Could not retrieve catalog; skipping run [22:02:20] puppet vs puppetd is annoying, it's like puppet ca via puppetca [22:03:47] there's no /etc/puppet/manifests/nfs.pp on this machine [22:05:33] some puppet wizard here? [22:05:51] actually, the data is still there [22:06:02] Just logging in to look atm, not sure this is on self [22:07:18] recreating the symlink should be simple enough [22:07:38] but if it was tracked by wikimedia-task-server, then that will produce a conflict when trying to reinstall [22:09:37] hmm seems this is using the normal server [22:11:32] now it is managing to install them [22:11:38] !puppetmaster::self [22:11:39] https://labsconsole.wikimedia.org/wiki/Help:SelfHostedPuppet [22:12:13] yeah that seems to be a broken manifests file =/ [22:15:58] finally [22:16:18] I managed to apparently restore it [22:16:32] hmm... http://commons.wikimedia.beta.wmflabs.org/ still broken [22:16:40] define restore? [22:16:58] get those removed packages reinstalled and configured [22:17:02] ah [22:17:28] also, /usr/local/apache is now there again [22:17:38] I just sorta fixed the symlink [22:17:42] the docroot is missing though apparently [22:17:44] oh, you did it? [22:18:21] is it possible that the package performed a rm -tf /usr/local/apache somehow crossing the symlink? [22:18:35] possibly [22:18:41] docroot not existing is why everything is down [22:18:46] moving the latst backup over atm [22:18:55] files seem to be present [22:19:26] /usr/local/apache/common/docroot/default doesnt :( [22:20:27] Lets assume common-local is a valid version... suppose I only need the rough structure can update as long as the configs/db stuff is there [22:20:58] common-local is what was being used [22:21:09] hmm [22:21:14] maybe it just broke the symlink then [22:21:41] that's unlikely [22:22:00] !log deployment-prep DocumentRoot got wiped, broken all the sites - fixed the broken symlink on bastion for /data/project/apache to /usr/local/apache. Also fixed common-local to common symlink. [22:22:01] Logged the message, Master [22:22:16] Well adding a symlink from common-local to common (the doc root) brought stuff back [22:22:27] that symlink was there [22:22:39] wikimedia-tasksever readded it [22:22:46] /usr/local/apache/common -> /usr/local/apache/common-local? [22:22:53] It wasn't showing on any of the apache boxes =/ [22:22:54] yes [22:23:25] but next time it was only the new symlink [22:23:34] interesting [22:23:54] Thumbnails seem broken, maybe they are 'suppose' to be... not so huge on the mw setup [22:24:02] I thought that reconfiguring the package had changed it into the /usr/local/apache/ symlink [22:24:43] I guess you removed the /usr/local/apache/ folder to create the symlink? [22:24:55] yeah [22:24:59] that explains it [22:25:06] wikimedia-task-server adds such symlink [22:25:12] AFAIK that's how it's suppose to be since we killed nfs (finally) [22:25:14] but expected /usr/local/apache to be there [22:25:20] ah [22:25:21] yeah [22:25:43] Should fix the nfs manifest, maybe hasher has a patchset open for it [22:25:48] in which case it would have been added in the right place [22:26:05] 'xterm-color': unknown terminal type. < bleh so clear doesn't work [22:26:06] that also explains why it was done [22:26:14] or so I think [22:26:22] it removed the common -> common-local symlink [22:26:32] then though that was the only member of the folder [22:26:39] so it tried to remove the "folder" [22:26:42] which was a symlink [22:26:47] yay symlinks [22:27:17] We totally should just kill apache in /usr/local, use packages and set the doc roots directly to project storage. Would save a bunch of disk seeks. [22:27:27] Have issues breaking from production (and sanity) there though. [22:27:50] those symlinks are crazy, too [22:28:30] Hmm dunno why thumbnails are broken, the images are there [22:28:42] wfStreamThumb() The source file for the specified thumbnail does not exist. < win [22:29:01] wfStreamThumb() The source file for the specified thumbnail does not exist. [22:29:11] some other missing symlink I guess [22:29:29] I have no freaking idea *which* symlink thumbnails use, lets find the config [22:31:02] Surley multiple sites could be handled in extensions over symlinks [22:34:45] /usr/share/perl5/Debconf/FrontEnd/Dialog.pm is full of \0s [22:34:52] this doesn't look any good [22:36:00] we should probably rebuild this vm [22:36:14] might be an idea [22:39:34] files at /usr/share/perl5/Debconf/FrontEnd/ being nulls look like storage corruption [22:39:53] and incidentally explain why apt-get was getting problems configuring the packages [22:40:02] Might be a broken vm from migrations [22:40:18] I thought deployment-bastion was new [22:41:57] mwstore://local-backend/local-public/2/26/Angela_Merkel_IMG_4162_edit.jpg < seriously, wtf is mwstore:// [22:43:00] a pseudo-protocol [22:45:03] Hmm [22:45:18] Assuming the folders are named logically it seems there is actually just no thumbnails [22:49:23] !log deployment-prep Thumbnails are currently broken - thumbs folder seems empty, images are there as expected - beta quirk or need rebuilding? [22:49:24] Logged the message, Master [22:50:39] !log deployment-prep Filesystem corruption signs in deployment-bastion, most debconf backends /usr/share/perl5/Debconf/FrontEnd are zeroed files. This explains some of the earlier apt-get problems, and maybe also bug 38747 (aka. magic tail -f fix) [22:50:41] Logged the message, Master [22:53:29] it's not a "new" vm [22:53:40] created: 2 June 2012 09:14:18 [22:54:22] I'll regenerate the captchas from deployment-dbdump [22:58:03] $ mkdir /data/project/upload6/captcha [22:58:03] mkdir: cannot create directory `/data/project/upload6/captcha': Permission denied [22:58:08] drwxrwxr-x 10 apache depops 8192 2012-07-17 21:29 upload6/ [22:58:15] $ sudo adduser platonides depops [22:58:15] The user `platonides' is already a member of `depops'. [22:59:25] lol [22:59:41] shouldn't apache create the dir if it dosn't exist [22:59:58] I had to make depops my primary group [23:00:00] weird [23:00:02] Or do we just have a static set of captchas.. which would defeat the point of them [23:01:34] !log deployment-prep Running time python captcha.py --output /mnt/upload6/private/captcha --font /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf --count 500 --dirs 3 --key "$(grep -Po '(?<=wmgCaptchaSecret = (["'"'"'])).*(?=\1)' /data/project/apache/common-local/wmf-config/PrivateSettings.php)" --wordlist <( < /usr/share/dict/american-english tr '[A-Z]' '[a-z]' | grep -E '^.{4,5}$' | grep -vE '(.)\1$' | grep -vE '^(.)\1' | LANG=C gre [23:01:35] Logged the message, Master [23:01:38] it is static [23:01:48] the apaches randomly load from a large pool [23:02:11] I hope it's *huge* [23:02:24]