[07:30:22] heya [07:38:41] petan: I'm disappointed I can't have a pywikipedia conversion bot shell account ;p [07:38:56] just create a service group [07:39:03] It's a gerrit bot anyway ;-) [07:39:10] huh? [07:39:19] but it still can be a service group [07:39:32] no, because it needs it's own SSH keys [07:39:47] but service groups can have own ssh keys as well [07:39:57] just create .ssh folder in its home [07:40:09] gerrit has it's own SSH key store ;-) [07:40:27] gerrit bot as in 'interacts with gerrit' [07:40:55] ok, shell access granted... what can I do... [07:41:01] nooooooooo [07:41:05] that's not what I meant :P [07:41:11] it was a joke -- I don't need shell access [07:41:22] so what do you need? [07:42:00] it's just an account to push changes to gerrit. it does everything it has to do, and it doesn't need labs access for that - it's just that labs and gerrit use the same LDAP information [07:42:46] sorry for the confusion [07:42:47] ok [08:02:19] hi :) [08:17:45] hello [11:43:15] Coren, are you there? [11:56:08] Coren: the connection limit stuff is really urgent, as we are required by the EU folks to have our tools published to go into the final evaluation phase. do you have any way of reaching asher? a phone number? [11:56:45] Coren: you're probably still asleep, once you read this, please try to contact asher if you have a phone number or anything. [11:59:18] JohannesK_WMDE is the limit on local db working now or it's still low? [11:59:24] I am wondering if what I did was correct or not [11:59:37] petan, i think it worked for the tools-db [11:59:40] let me check again [11:59:43] ok [12:01:10] petan: max_user_connections shows as 0 for both render and render-tests. so it seems to have worked. [12:01:31] o.O [12:01:39] I raised it to 80 and it's... 0 [12:01:52] mysql is weird [12:02:48] petan: yes, the *variable* shows as 0 but i did this: [12:02:59] local-render-tests@tools-login:~$ for i in $(seq 1 100); do ((sleep 10; echo quit) | mysql --defaults-file=replica.my.cnf -h tools-db) & done [12:02:59] [1] 28028 [12:03:00] [2] 28029 [12:03:00] [...] [12:03:09] and then: [12:03:11] ERROR 1226 (42000): User 'p50380g50454' has exceeded the 'max_user_connections' resource (current value: 80) [12:03:19] interesting [12:03:23] yep [12:03:56] ah, ok. the global max_user_connections is 0, but the local one is 80. [12:04:26] ok [12:04:28] good [12:05:04] templatetiger eats 28gb in sql... [12:05:51] Templatetiger extracts all templates from dumps. It intends to analyse the values contained in the templates and represent them in new ways, apply filters and do other useful stuff. [12:05:58] what a vague explanation :D] [12:06:17] represent them in new ways, apply filters and do useful stuff :D :D [12:07:48] uhhh... yeah... [12:08:14] EAT MOAR RAMZ and do... interesting stuff [12:08:16] ;) [12:09:25] petan: is there nobody other than asher who can raise our limit to the replicas? [12:09:49] we're kind of... sitting on hot coal, if there is such an expression in english :D [12:09:50] I think not... :/ [12:10:06] nobody I know of [12:10:24] http://www.dict.cc/?s=auf+hei%C3%9Fen+kohlen <-- this [12:10:40] JohannesK_WMDE: Maybe try #wikimedia-operations? [12:11:06] in more than one sense, because it's rather hot also ;) [12:11:17] ok, i'll just ask there also [12:12:22] BTW, what's the connection limit on the replicas for "normal" users? [12:12:39] 10 o.O very low if you ask me [12:13:51] on the toolserver, there is no limit for MMPs, scfc_de, so this was unexpected... [12:26:12] JohannesK_WMDE: so please fill in a bug against Wikimedia Labs > Infrastructure [12:26:23] that will send a mail to all people that can help [12:26:42] Coren should be active soon, he is Canada eastern coast iirc [12:27:21] and you can try finding out where that parameter is in operations/puppet.git and submit a change for it :-] [12:33:16] hashar I don't think it's in gerrit [12:33:25] hashar this parameter is in mysql.user table [12:34:32] regardless, that requires a bug report :-] [12:41:09] hashar: https://bugzilla.wikimedia.org/show_bug.cgi?id=49872 [12:42:40] JohannesK_WMDE: great [12:42:45] and that is in puppet apparently : modules/mysql_multi_instance/manifests/instance.pp: 'max_user_connections' => 10, [12:43:58] hashar: well, i don't know if it helps because asher doesn't seem to be in the list, and i was told he was the one he could change it... i sent him an email before though [12:44:16] hashar: oh, this is in puppet? would that mean that Coren *can* change it? [12:45:25] yup [12:45:35] https://bugzilla.wikimedia.org/show_bug.cgi?id=49872#c1 :-] [12:45:42] that needs a bit of tweaking to one of the puppet class [12:46:14] hashar: that's interesting [12:46:36] petan: can you change that file? see above link [12:48:37] you will need someone from ops to review / merge it anyway [12:48:42] so better to let them handle it [12:48:51] I have left some hint on comment 2 https://bugzilla.wikimedia.org/show_bug.cgi?id=49872#c2 [12:49:03] Coren can definitely fix it :) [12:49:55] thanks hashar [12:50:06] sorry can't help further more :-] [12:50:48] JohannesK_WMDE which file [12:51:07] aha [12:51:21] yes I can change it but I think waiting for Coren is a better idea :> [12:51:52] Also I would assume that in Puppet, the limit is for *all* users -- not sure if we want that. [12:52:06] indeed [12:52:13] JohannesK_WMDE: You don't have a working setup at Toolserver as plan B? [12:52:19] + I think it will only affect new users, not existing [12:52:51] scfc_de: no, it's currently not working properly on toolserver because of several... issues. we really want to move. [12:53:50] petan: i think we do want to raise the limit for all users. but this is something admins should talk and agree about of course. still, there must be some way to increase it for a specific user, for now [12:55:15] scfc_de: and once we are in the evaluation phase, we can't move. so we have to do it now. and almost everything works. this is the last blocker. [12:58:33] JohannesK_WMDE: Reading the docs, it seems to be possible to set max_connections per user, and I guess petan proved that on tools-db :-). But it looks like WMF needs a DBA in non-US timezones, perhaps someone from Asia. [12:58:55] or Europe ;) [12:59:12] YuviPanda is from Asia and I can reach him just as easily as Coren (very hard) [12:59:20] hmm? [12:59:25] so having another one from Asia would make a change :P [12:59:28] * wouldn't [12:59:35] YuviPanda sush, we know you are just a bot [12:59:50] THAT WAS SUPPOSED TO BE BETWEEN US! [12:59:56] ew [12:59:58] :/ [13:00:07] Now I'll tell everyone you're actually from Mars [13:00:14] they all know [13:00:24] I thought you told only me :( [13:00:25] hmpf [13:00:30] * YuviPanda goes off to read puppet instead [13:01:16] lol [13:01:30] if it is a bot, it's a good one ;) [13:01:52] he [13:01:54] h [13:01:57] almost as good as wm-bot [13:01:59] for the record, I'm not a bot :D [13:02:12] YuviPanda: yes, that's what you want us to believe! [13:02:36] yeah, I know that whatever I said will also be exactly what a bot would... [13:02:40] you should take the turing test :p [13:02:54] * YuviPanda freezes in shock [13:03:15] (http://xkcd.com/632/) [13:03:17] for the record, i wasn't serious ;) [13:03:32] hehe, I know. [13:03:48] I was just trying to play into that XKCD [13:03:51] hehe [13:04:43] you're going to be the first fully automatic db admin [13:05:54] heh [13:06:01] I don't think they'll let me anywhere near admining DBs [13:06:04] @notify Coren [13:06:04] This user is now online in #wikimedia-labs. I'll let you know when they show some activity (talk, etc.) [13:06:16] andrewbogott ping [13:06:25] andrewbogott: I need 1 IP for toolsbeta [13:06:37] YuviPanda: that's rude. just because you're a bot! [13:06:47] yeah, botists [13:06:58] petan: OK, just a minute... [13:07:10] thanks [13:11:17] petan, done [13:11:37] :O [13:12:05] Failed to allocate new public IP address. [13:13:44] petan, how many do you need, total? [13:13:49] 1 [13:14:30] hm, should work then [13:14:35] I wonder why the quota page is broken [13:14:56] maybe I need to logout? [13:15:23] :\ [13:16:16] try now? [13:18:36] petan, try now? [13:32:24] * Coren oinks. [13:36:09] hi Coren. [13:36:29] seen this? https://bugzilla.wikimedia.org/show_bug.cgi?id=49872 [13:37:38] Hadn't. Ima make the change, but that'll require a DB restart (i.e.: outage) [13:37:46] hi Coren :) [13:37:48] So I'll schedule one. [13:38:24] fale: Hello. [13:39:34] andrewbogott works [13:39:34] Coren: good. can you get it done till tomorrow? [13:39:50] Coren are you /sure/ [13:40:00] Coren I changed this option on tools-db, without restart [13:40:09] and it works [13:41:55] petan: Oh? Just a SIGHUP suffices? [13:42:13] Sometimes mysql is so odd in what it decides needs a restart vs what needs a kick. [13:42:25] no, Coren I changed this using the grant [13:42:30] it's in mysql docs [13:42:34] yes, it worked for tools-db [13:42:53] something like grant usage on blabla with max_user_connections 80 [13:42:54] http://dev.mysql.com/doc/refman/5.0/en/grant.html <-- Coren [13:43:45] OR update mysql.user set max_user_connections=80; flush privileges [13:45:19] JohannesK_WMDE: 32 suffices? [13:48:48] Coren: any news on bugzilla and gerrit? [13:50:13] Coren: no, i don't think so. how about 512. [13:51:30] we won't use that many normally, but when we publish it, some more people might just click the links. [13:53:17] 512 is a *lot*' [13:53:35] If you use that constantly, you and I are going to have /words/. :-) [13:53:43] several connections per tool, several tools per account because the toolkit is a kind of show case app thing for render [13:54:07] like i said, not constantly. that's why it's called *maximum*, not *mean* ;) [13:54:30] (and no, we can't split the toolkit up to separate accounts) [13:55:06] and btw, on toolserver, there was no limit at all for MMPs [13:56:03] 10 is very low, even if you only use one connection at all times, you reach that quickly when several people are using it. the limit should be raised for all tools. [13:59:23] btw Coren, um, didn't you say yesterday that only asher could change the limit? [14:01:24] Coren, ping [14:01:57] JohannesK_WMDE: I said yesterday that only Asher knew whether it was okay that we just should. :-) [14:02:00] Cyberpower678: Yes? [14:02:12] Can I tunnel my PHP debugger into labs? [14:02:26] Cyberpower678: I see no reason why not, as a rule. [14:02:41] Can I get some connection specs? [14:03:05] Cyberpower678: ? What do you need exactly? [14:03:11] Like the Tunnel Listening Address [14:03:50] Coren: good, that worked. thanks. [14:03:55] is that for all users btw? [14:04:50] JohannesK_WMDE: Yes, it seemed a little silly to special case accounts, and it was a promise of management nightmares to come. :-) [14:05:00] yeah, ok [14:05:16] Coren, I need host, port, and Tunnel Listening address [14:05:48] Cyberpower678: ... you're the one who picks those dude. [14:06:03] Cyberpower678: Tunnel through SSH [14:06:33] I know host and port, but I've never tunneled before. [14:06:37] Oh. [14:06:46] It's fairly easy. You using openssh or putty? [14:07:00] Actually, I'm using PHPed [14:07:14] The debugger has tunneling abilities. [14:07:35] btw, another thing Coren: yesterday i had some processes hanging on the webserver which were waiting for sql connections. it would be nice if i could log in to tools-webserver-01, so i can just kill those processes and i don't have to bother admins the next time something like that happens. [14:07:46] think you can give me ssh access to the webserver? [14:07:55] Hm. I don't know that I can help you then, since I don't know that bit of software. Presumably, it uses openssh tunnels internally [14:08:24] Coren, I think so. The documentation hasn't proven helpful yet. [14:08:51] The debugger essentially runs through the tunneled server. [14:09:12] I need this so I can test scripts that require DB access. [14:09:47] Cyberpower678: Google will have to be your friend since I'm not familiar with it; I could have told you how to tunnel through most ssh clients though. :-) [14:10:19] Coren, then give me the instructions for the clients you do know. I can probably adapt it for my software. [14:10:54] http://tools.wmflabs.org/xtools/blame/index.php?article=Pokemon+%28anime%29&lang=en&wiki=wikipedia&text=Pokemon grrr. [14:11:11] Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 56100 bytes) in /data/project/xtools/Peachy/Includes/Wiki.php on line 588 [14:12:27] * Cyberpower678 needs to work on some resource conservation regarding X!'s tools. [14:13:06] Coren, ? [14:13:30] Cyberpower678: Could you wait a moment, please? I'll be right with you. [14:13:39] * Coren doesn't have unlimited bandwidth. :-) [14:13:40] Coren, ok. [14:13:55] Coren, I thought you were a robot? [14:13:59] :p [14:14:12] Cyberpower678: https://wiki.toolserver.org/view/Database_access#External_access [14:15:22] scfc_de, umm...no [14:15:22] I want labs. [14:15:22] Not toolserver. [14:18:10] Cyberpower678: the method to create tunnels is the same [14:18:33] But the connections are different [14:19:06] *cough [14:25:35] addshore, can you lend me a hand? [14:26:55] doing? :0 [14:27:09] Look at the simple renames page. [14:27:16] ANd fufill my request. [14:28:49] addshore, ^ [14:29:06] mhhm *looking* [14:30:23] heh, is that account totally dead? [14:30:29] Yes. [14:31:16] I was talking to the person on IRC during this attempt to create that account. There is no password or email linked to it, because the Visual Editor broke the signup process. [14:32:55] Cyberpower678: done [14:33:08] thanks. :) [14:35:03] np [14:36:38] @notify Coren [14:36:38] I'll let you know when I see Coren around here [17:21:21] What a wonderful day. [17:21:53] fale: I now know why I never got your email: my hosted server, which serves as my IRC bouncer and mail server (inter alia) was dying. [17:22:14] I learned it when my bouncer died and I couldn't ssh back into it to fix it. [18:02:39] Coren: :D [18:02:55] Coren: it's good I cheked with you :) [18:14:23] Coren: could I get qt4-qmake on tools-dev? :-) [18:17:20] valhallasw: Sure. [18:29:43] Coren: care to kill pid 22479 on tool-login [18:30:13] firstly it shouldnt be running on login, secondly its spamming wikidata with edits from the local ip :) [18:31:18] addshore: Indeed. I murdered it with prejudice. [18:31:28] :) why thankyou :) [18:32:03] * Coren fires up an email to hasard-sj [18:32:39] hehe :P [18:33:03] Heh [18:33:06] I already memo'd him. [18:33:39] Then the point will be reinforced all the more. :-) [18:34:46] Hm. [18:35:28] :D [18:36:05] * Coren is annoyed at the number of detached screen sessions. [18:36:45] chrismcmahon: hey :) [18:36:58] hi hashar [18:37:08] chrismcmahon: we are logging fatal errors on beta now!!!! They are avail able in /data/project/logs/fatal.log [18:37:22] chrismcmahon: annnd we display the WMF error page on a fatal instead of a blank page [18:37:28] that will make it easier to find out the root cause [18:37:35] thanks hashar [18:37:49] I eventually got over my laziness and started looking at how we handled fatal in prod [18:37:59] basically: we wrote a PHP extension :) [18:38:42] chrismcmahon: and I am also updating the project status page at https://www.mediawiki.org/wiki/Beta_cluster/status#2013-06-15 :) [18:38:52] instead of using !log there [18:39:46] hashar: beta is getting more and more useful, thanks. now to get VisualEditor running there is the next thing I think [18:40:29] Coren: whats the range of ips labs instances can come from? :) 10.4.0.0 and upwards? :P [18:41:46] hashar: PHP extension? go wash your mouth! ;-) (is it published anywhere? it sounds like something a lot of people might be interested in) [18:42:32] addshore: Ostensibly, anywhere from 10.0.0.0/12 though that might be more restricted than that in practice. [18:42:47] okeydokey :) [18:43:21] chrismcmahon: I think Roan worked on bringing VE and Parsoid on beta. But we had to disable it in production so I guess it is disabled in beta as well [18:43:44] chrismcmahon: also the VisualEditor is not updating since May 28th, that is a bug in the repository mediawiki/extensions.git . Need to ping Chad about it [18:44:01] hashar: VE is enabled in test2wiki and mw.o [18:46:04] chrismcmahon: you could ask them to enable it on beta [18:46:10] still have to get it updating properly though [19:08:00] its spamming wikidata with edits from the local ip :) [19:08:02] What? how? [19:08:26] it wasnt logged in [19:08:33] yeah, but local ip? [19:08:44] its within the wikimedia network [19:09:32] ... still, local ip? [19:09:49] yes :P what else is it going to show? :P [19:10:19] the public IP that would be shown on any other non-wikimedia site? [19:10:42] it doesnt have a public ip [19:11:01] I imagine to other sites it will go through some proxy somewhere [19:11:07] then the proxy ip [19:11:21] but it doesnt go through the proxy to get to wikipedia :P [19:11:45] why not? [19:14:08] well it goes straight from the instance through the squid to the wiki servers [19:14:54] it never leaves the internal network, and the squid simply uses the requesting instances ip [19:15:19] Squid should be made to use the proxy IP instead then. [19:17:18] Either way labs IPs really shouldn't be shown on-wiki, it's difficult for people to work out what the address actually points to unless they know about labs/wikimedia's internal network [19:17:55] hmmm [19:18:36] I partly agree it might be nice to force all edits from anywhere on tools to come through a single IP [19:18:58] on all labs projects [19:19:41] all labs projects edit through a single ip.? [19:19:53] how would you find out where things where going wrong? [19:22:36] toolserver didn't come through a single ip, it had a range or ips [19:27:44] New review: coren; "LGM" [labs/toollabs] (master); V: 2 - https://gerrit.wikimedia.org/r/69157 [19:27:44] Change merged: coren; [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/69157 [19:29:29] addshore: Why not softblock the whole range? [19:29:40] we are going to :) [19:29:44] addshore: Tools shouldn't be editing anonymously at all anyways. [19:56:27] fale: Please enter a .description in your tool's home; I'll be using it as a basis for the bugzilla component. [20:07:06] Coren: I don't think 10.0.0.0/12 is complete, enwiki has seen bot edits from 10.64.0.123 and 10.64.0.126 in the past. Maybe 10.0.0.0/12 and 10.64.0.0/12? Or just 10.0.0.0/8? [20:07:41] 10.64.0.0/12 is eqiad (another datacenter) so it wouldn't be labs. [20:08:00] 10.0.0.0/8 should be safe enough anyways -- it's all internal. [20:09:44] i thought we couldn't do rangeblocks that big...? [20:10:03] Heh. "Range blocks larger than /16 are not allowed". Later I'll have to run a script to do 256 /16 blocks. [20:10:19] or, we could temporarily set the limit higher [20:10:23] just to make the block [20:12:45] anomie: zOMG adminbotz! [20:13:02] legoktm: Script-assisted ~~editing~~ blocking. q: [20:13:09] * Jasper_Deng was going to contact anomie about how he indeffed a school IPv6 range [20:13:14] :P [20:13:16] last year [20:14:41] Jasper_Deng: Which range is that? [20:18:14] * anomie notes that school block was by someone else with a similar name [20:23:20] !log deployment-prep VisualEditor self updated on beta, it was stuck due to a misconfiguration in gerrit {{bug|49846}} [20:23:23] Logged the message, Master [20:23:32] chrismcmahon: VisualEditor is now up to date (though it might still be disabled) [20:23:51] nice, thanks hashar [20:24:18] [20:24:18] grmblbl [20:39:47] petan: can I please have global abuse filter manager for beta wikis? [20:39:51] so I can manage the global filters [20:39:52] Coren: done :) [20:40:42] hashar ^ [20:41:48] chrismcmahon^ [20:42:47] Jasper_Deng: probably :) [20:42:58] * Jasper_Deng already has global sysop on that cluster [20:43:08] Technically I could do it, but I don't know about the policy, so... [20:43:20] Jasper_Deng: I have created a specific group to grant abusefilter rights [20:43:22] let me find out [20:43:33] Steinsplitter already has it [20:43:37] jepp^^ [20:43:42] Krenair: you can't, it's a global group and you're not a global steward [20:43:46] i hav blocked today a lot of long term spambots [20:43:57] Yes I can Jasper_Deng [20:44:06] Jasper_Deng: done :) [20:44:06] via the db perhaps [20:44:07] Don't have to be a global steward [20:44:22] oh, developer [20:44:22] yes. [20:44:45] the groups are probably a bit messy [20:45:10] !log deployment-prep Jasper Deng joined in AbuseFilter manager group :) [20:45:13] Logged the message, Master [20:45:31] hashar developer is meant for developers, it gives enough for them to do anything, they just need to know how mediawiki works :P [20:45:50] Steinsplitter: Jasper_Deng : if you guys managed to prevents most of the spam, a lot of people will be happier :) [20:46:03] petan: ahh [20:46:09] yeah [20:46:20] having an automatic block enabled would also be good [20:46:26] like on MediaWiki.org [20:46:29] petan: well since we implemented the bureaucrat/ steward roles, I have no idea how mw permissions works. [20:46:50] bureaucrat is a local right [20:46:53] afaik developer just give you rights to change any permissions anywhere [20:46:55] and ther ar a lot of long-term-spambots. I hav blocked som accounts who spammed over months. [20:47:17] To confuse things, steward is both local and global. [20:51:42] petan, hashar: any chance we could enable automatic local blockage? [20:52:05] Jasper_Deng I hate that feature, but if hashar is not against... idc [20:52:22] because it's very useful for keeping spam out of mw.org [20:52:46] if you find out how to enable that, sure :) [20:52:52] it's a config change [20:53:16] will be happy to merge it [20:53:17] :) [20:53:17] or you just give the global abuse filter managers the abusefilter-modify-restricted permission [20:53:26] ah that too [20:54:27] a /bug/ I've noticed, though, is that there's a mismatch between what the global Special:AbuseLog on deployment says and what the local log says (and the local log is the correct one) [20:54:40] 22:39, June 19, 2013: User:ConcepcionCoffey (en.wikibooks.beta.wmflabs.org) triggered filter 12, performing the action "edit" on User:Keieopwee. Actions taken: Warn; Filter description: More Chinese spambots (details | examine | adjust visibility) [20:54:48] Jasper_Deng: seems abusefilter-modify-restricted is checked [20:55:00] Jasper_Deng: the Global_Abusefilter_Manager group has all the abusefilter rights checked [20:55:02] the former user (ConcpcionCoffey) isn't registered on the target wiki [20:55:04] Hm. I wonder if global abusefilters were enabled on deployment-prep... [20:55:13] Krenair: they are [20:55:22] Krenair: i fixed them a few days ago with Steinsplitter :) [20:55:25] ... and you've just been discussing it. oops. [20:55:26] hashar: then $wgAbuseFilterRestrictedActions needs it [20:55:46] Krenair: the AbuseFilterCentralDB was pointing to 'metawiki' which does not exist on beta, that is 'labswiki' instead [20:56:09] Krenair: imho global abusefilter is the best solution to protect the beta cluster vor spam [20:56:12] ($wgAbuseFilterRestrictedActions excludes block by default) [20:56:31] * Jasper_Deng wonders if Krenair is going to investigate the bug [20:56:52] about to haed to bed [20:57:24] so one would have to create a abusefilter-labs.php that would include abusefilter.php (the one from production) [20:57:27] then do the overrid [20:57:49] yeah [20:57:58] Jasper_Deng, can you PM me the two mismatched log entries? [20:58:11] and wmf-config/CommonSettings.php: include( "$wmfConfigDir/abusefilter.php" ); [20:58:11] needs to be changed to use the getRealmSpecificFilename() wrapper. Aka include( getRealmSpecificFilename( "$wmfConfigDir/abusefilter.php" ) ); [20:58:21] and maybe edit docroot/noc/createTxtFileSymlinks.sh: wmf-config/abusefilter.php as well :-] [20:58:37] Jasper_Deng: feel free to copy paste the text above and fill a bug with it :) [20:59:07] will see. OK, so we need two things.... 1, set up automatic blockage, 2, troubleshoot the logs issue. [20:59:38] if you fill bugs against Wikimedia Labs > deployment-prep (beta) , I am on cc and do read notifications [20:59:56] for patches in gerrit, feel free to add me as a reviewer ( hashar is my name there) [21:00:07] will be more than happy to review any bug / patches [21:10:11] Getting an odd error trying to invoke any php script using jsub .. 'libgcc_s.so.1 must be installed for pthread_cancel to work'. Am I being silly ? [21:11:08] tb3: I've seen that error in the past. It means you're exceeding your memory limit. [21:11:58] Ta .. will fiddle with memory limits an see. [21:26:04] New patchset: coren; "Add tools.wmflabs.org website contents" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/69793 [21:31:59] fale: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Tool%20Labs%20tools [21:32:18] fale: https://gerrit.wikimedia.org/r/#/admin/projects/?filter=lists [21:33:33] New review: coren; "Just a dump of the current contents." [labs/toollabs] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/69793 [21:33:33] Change merged: coren; [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/69793 [22:43:41] Is it possible for somebody to get a crontab off of toolserver for me? [22:49:11] bgwhite: er, ask in -toolserver? [22:54:09] Coren: are you sure you killed Hazard-Bot? [22:54:15] it's still editing logged out [22:54:23] * Jasper_Deng is considering putting a soft block on that IP [22:55:15] Jasper_Deng: its probably on a crontab [22:55:22] * Jasper_Deng figured [22:55:26] Jasper_Deng: There are also cron jobs [22:55:41] * Coren disables them [22:57:31]