[00:16:53] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c4 (10Ori Livneh) Yes, probably. Are these files declared in Puppet? [00:23:16] andrewbogott: around? [00:38:08] 3Wikimedia Labs / 3deployment-prep (beta): Automate updating the puppet checkout - 10https://bugzilla.wikimedia.org/66683#c2 (10Tim Landscheidt) Note that deployment-prep is special in its use of role::puppet::self in that it always uses HEAD (AFAIK). Typically, there will be local changes and/or bisects, s... [01:27:53] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c5 (10Matthew Flaschen) (In reply to Ori Livneh from comment #4) > Yes, probably. Are these files declared in Puppet? Nope. [02:27:08] 3Wikimedia Labs / 3tools: tools-webproxy's / is full - 10https://bugzilla.wikimedia.org/58141#c5 (10Tim Landscheidt) 5NEW>3RESO/FIX (In reply to Yuvi Panda from comment #4) > Any update on this? Doesn't applying the 'biglogs' puppet class 'fix' the > issue anyway? Uh. Everything I wrote above is from b... [05:23:12] my fresh instance says command puppetd not found on giving sudo puppetd -tv [05:23:20] is there a problem ? [05:24:21] I think puppet 3 has a different command [05:25:08] oh. somebody would've updated here https://wikitech.wikimedia.org/wiki/Help:Instances#Configuring_Instances [06:39:24] 3Wikimedia Labs / 3(other): "Deprecated" error on devwiki/pagemigrationwiki (http://pagemigration.wmflabs.org/wiki/Special:RecentChanges) - 10https://bugzilla.wikimedia.org/66040#c3 (10Southparkfan) Added depends on [06:39:55] Better I marked it as a duplicate -,- [08:15:52] 3Wikimedia Labs / 3Infrastructure: Create a cxserver user in labs LDAP - 10https://bugzilla.wikimedia.org/66575#c7 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX The cxserver user and groups have been created in labs LDAP with identifiers 10060. On deployment-cxserver01.eqiad.wmflabs I have manually deleted... [08:59:06] My cronfile is rejected: /usr/bin/crontab: crontabs/tmp.g5mmXG: No space left on device [08:59:26] rillke: hmm, looking [09:00:00] I became my tool, cd'ed to the directory the file is in and did crontab cronfile [09:00:17] same happens when using xcrontab, btw. [09:00:39] rillke: looks like some host is out of space [09:00:47] usually it is someone not cleaning up /tmp [09:00:50] let me go look [09:01:43] can I $ rm -r /tmp ? [09:03:17] rillke: no [09:03:26] rillke: because lighty for some stupid reason keeps some socket files there [09:03:39] rillke: and we have no way of figuring out *which* device either [09:03:40] err [09:03:45] which system [09:04:25] rillke: -submit has /var full [09:05:44] rillke: meh, I don't have root on -submit :| [09:05:45] wtf [09:07:58] YuviPanda, should I file a bug? [09:08:07] rillke: yup. tools-submit /var is full [09:08:15] gah, https://wikitech.wikimedia.org/wiki/Special:NovaSudoer is erroring out [09:08:17] stupid wikitech [09:13:49] rillke: file a bug, I'll comment with what I know [09:17:21] https://bugzilla.wikimedia.org/show_bug.cgi?id=66772 [09:17:25] 3Wikimedia Labs / 3tools: cronfile rejected: /usr/bin/crontab: crontabs/tmp.g5mmXG: No space left on device - 10https://bugzilla.wikimedia.org/66772 (10Rainer Rillke @commons.wikimedia) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier I deleted my cronjob today to make way for an extended version. $ be... [09:17:43] rillke: should work now, akosiaris had root and is cleaning up [09:18:23] rillke: can you try again? [09:18:30] thanks, works [09:19:14] rillke: yw. needs to be fixed 'for real' soon [09:19:22] 3Wikimedia Labs / 3tools: cronfile rejected: /usr/bin/crontab: crontabs/tmp.g5mmXG: No space left on device - 10https://bugzilla.wikimedia.org/66772#c1 (10Rainer Rillke @commons.wikimedia) 5NEW>3RESO/FIX (11:04:14 AM) YuviPanda: rillke: -submit has /var full .... (11:17:32 AM) YuviPanda: rillke: should w... [09:21:37] 3Wikimedia Labs / 3tools: cronfile rejected: /usr/bin/crontab: crontabs/tmp.g5mmXG: No space left on device - 10https://bugzilla.wikimedia.org/66772#c2 (10Yuvi Panda) 5RESO/FIX>3REOP Temp. fixed, but should run out of space again soon. Looking into applying the biglogs class as a fix. [09:31:23] rillke: cron submitting might not work for a few mins now, we're fixing the problem for good [10:11:39] rillke: should be fixed now [10:12:49] 3Wikimedia Labs / 3tools: cronfile rejected: /usr/bin/crontab: crontabs/tmp.g5mmXG: No space left on device - 10https://bugzilla.wikimedia.org/66772#c3 (10Yuvi Panda) 5REOP>3RESO/FIX added biglogs class, and akosiaris did the appropriate moves, and everything is alright for now. [10:20:14] than you, yuvi [10:20:47] rillke: yw. we're now investigating why me nor scfe_de have root on the tools-submit host so we will be able to better handle this next time [10:29:34] 3Wikimedia Labs / 3tools: locale.setlocale(locale.LC_ALL, "ca_AD") doesn't work - 10https://bugzilla.wikimedia.org/62269#c6 (10Yuvi Panda) Has been merged, should be present in the nodes in about 30minutes. Can you test and verify after that? Thanks! [10:29:34] 3Wikimedia Labs / 3tools: generate language package for ca_ES.utf8 - 10https://bugzilla.wikimedia.org/66721#c6 (10Yuvi Panda) Has been merged, should be present in the nodes in about 30minutes. Can you test and verify after that? Thanks! [10:41:00] Hi there, I want to add a new feature to an API call, is this the right channel to ask for help? [10:46:28] ineditable: to a mediawiki API call? no. try #wikimedia-dev or #mediawiki [10:46:58] Yep, ok thank you! [12:05:34] 3Wikimedia Labs / 3tools: generate language package for ca_ES.utf8 - 10https://bugzilla.wikimedia.org/66721#c7 (10Igor Remolar @ Betxí) It works fine. >>> import locale,time;locale.setlocale(locale.LC_ALL, "ca_ES.UTF-8@valencia");[time.strftime("%B", time.strptime("01-%i-2014" % i, "%d-%m-%Y")) for i in ran... [12:06:49] 3Wikimedia Labs / 3tools: generate language package for ca_ES.utf8 - 10https://bugzilla.wikimedia.org/66721#c8 (10Yuvi Panda) 5PATC>3RESO/FIX :D Glad this could be fixed! (And no need to apologize for 'poor English' :) ) [12:35:21] @notify scfe_de [12:35:21] I'll let you know when I see scfe_de around here [13:07:48] 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c5 (10merl) 5RESO/FIX>3REOP globalimagelinks is still missing [13:16:28] whats mongo.creds.json ? [13:17:23] Betacommand: aaah. do ignore/delete it. I thought I had cleaned them all up [13:18:00] Betacommand: I was setting up a mongodb service for toollabs for people to use, and that would've had the credentials per tool. but hit some serious bugs in mongodb so decided to not do it [13:23:04] 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c6 (10merl) Also filearchive, filearchive_userindex, user_daily_contribs, user_properties and betafeatures_user_counts are missing compared to s4.labsdb.commonswiki_p [13:42:03] 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~daniel/WikiSense/* to Tool Labs and provide redirect - 10https://bugzilla.wikimedia.org/58869#c4 (10metatron) I can offer a working replacement. eg. https://tools.wmflabs.org/xtools/articleinfo/?lang=de&wiki=wikipedia&page=Berlin&style=new -> sec... [14:07:19] 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10merl) [14:07:19] 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683 (10merl) [14:12:50] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier I don't see a reason anyone would want to use the unindexed 'revision' vs 'revision_userindex'. We can rename 'revision' to revision_no... [14:18:48] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c1 (10Aaron Halfaker) Agreed. I don't see any good reason why the non-indexed thing behind "revision" should occupy such a privileged name. Is it actually useful for querying? If we can keep "revision_... [14:34:14] scfc_de: hey! tools-submit sudo got fixed [14:34:15] scfc_de: should both have sudo there now :) missing cn= entry in /etc/sudo-ldap.conf [14:34:34] YuviPanda: i though in revision_noindex revision with suppressed usernames but not hidden revsion are not available [14:34:59] Merlissimo: there's no such thing as revision_noindex right now :) [14:35:29] Merlissimo: right, so with usernames suppressed but unhidden revisions [14:35:44] Merlissimo: I think having that in revision_noindex would be ok, with revision_userindex being renamed revision [14:36:35] i think revision_userindex should be default yes, but som people need really all revisions (for counting, statistik, ...) [14:36:48] Merlissimo: indeed, and those people can use revision_noindex [14:38:40] YuviPanda: Ah, great, thanks! [14:38:51] scfc_de: :) [14:38:51] And +1 for unifying on revision. [14:39:00] scfc_de: do +1 on the bug :) [14:39:03] Merlissimo: you too! :) [14:40:34] 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10merl) [14:40:34] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c2 (10merl) According to doc revisions with suppresed username (by os, steward) )are currently only available from revision and not revision_noindex. But i agree that the indexed version should be the de... [14:41:18] 3Wikimedia Labs / 3tools: Rename revision_userindex to revision - 10https://bugzilla.wikimedia.org/66786#c3 (10Yuvi Panda) (In reply to merl from comment #2) > According to doc revisions with suppresed username (by os, steward) )are > currently only available from revision and not revision_noindex. I think... [14:41:26] voting has been unfortunately disabled on bugzilla [14:41:47] Merlissimo: commenting is a viable alternative [14:44:16] but Coren should look at https://bugzilla.wikimedia.org/show_bug.cgi?id=59683#c5 first. i really need globalimagelinks for migrationg from ts [14:44:32] Merlissimo: yeah. coren's on vacation till next week tho [14:45:05] toolserver is dead at the end of next week [14:45:10] Silke_WMDE: ^^ [14:46:40] Coren is on vacation? [14:46:40] * Merlissimo hates bugs that are marked as fixed without solving the reported problem [14:46:58] Silke_WMDE: honeymoon, I think. [14:47:05] he got married [14:47:26] Silke_WMDE: Merlissimo he's back on 25th [14:47:35] Oh. [14:49:00] YuviPanda: thx [14:50:37] Merlissimo: yw [14:50:40] * YuviPanda wades away for food [14:50:49] 3Wikimedia Labs / 3tools: locale.setlocale(locale.LC_ALL, "ca_AD") doesn't work - 10https://bugzilla.wikimedia.org/62269#c7 (10Yuvi Panda) 5PATC>3RESO/FIX Verified in the other bug. [14:56:37] andrewbogott_afk: scfc_de YuviPanda Can any of you resolve this issue? https://bugzilla.wikimedia.org/show_bug.cgi?id=59683#c5 [15:00:18] hi ! I made a new instance in wikitech labs - box3verp.wmflabs -- I enabled the role::mediawiki -- and gave puppet apply ( puppetd -tv doens't work it seem s ) and its showing nothing [15:03:59] YuviPanda: Sure about 25th? I have 28th in mind. [15:04:56] scfc_de: > I'm going to be off-grid from tomorrow (Jun 13) up to June 25. [15:05:40] YuviPanda: k [15:06:50] Silke_WMDE: I could blindly copy the patch, but neither check that it works nor deploy it. I think the easiest way is to assign the bug to Sean Pringle who should know what to do. [15:07:33] 3Wikimedia Labs / 3deployment-prep (beta): Automate updating the puppet checkout - 10https://bugzilla.wikimedia.org/66683#c3 (10Bryan Davis) (In reply to Tim Landscheidt from comment #2) > Note that deployment-prep is special in its use of role::puppet::self in > that it always uses HEAD (AFAIK). We are act... [15:07:40] scfc_de: Silke_WMDE yeah, poking springle seems to be the best bet. Also because of the previous password leak, I'm slightly wary of touching that myself [15:45:28] ok scfc_de and YuviPanda|zzz Thanks! [15:47:35] 3Wikimedia Labs / 3tools: Add some of the missing tables in commonswiki_f_p - 10https://bugzilla.wikimedia.org/59683#c7 (10Silke Meyer (WMDE)) a:5Marc A. Pelletier>3Sean Pringle Hi Sean, I'm assigning this to you because Coren isn't available, this has been reopened and is needed for the Toolserver migra... [16:37:48] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c6 (10Andrew Bogott) So the new puppet module installs virthosts and also removes any virthosts that aren't currently installed? That appeals to the obsessive/compulsive in m... [16:39:04] 3Wikimedia Labs: Puppet is erasing growthdoc.wmflabs.org and proveit.wmflabs.org sites-enabled - 10https://bugzilla.wikimedia.org/66751#c7 (10Andrew Bogott) Or, heck, just check $realm and skip the deletion step on labs. [16:55:21] !log deployment-prep Setup hourly cron as user bd808 on deployment-salt to test automatic update of puppet repo using ~bd808/git-sync-upstream script [16:55:23] Logged the message, Master [17:02:04] 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~erwin85/relatedchanges.php to Tool Labs - 10https://bugzilla.wikimedia.org/60872#c5 (10Helder) As reported on https://pt.wikipedia.org/w/index.php?diff=39171644&oldid=39162195&uselang=en the tool seems to be displaying edits made to Wikidata items... [17:03:35] 3Wikimedia Labs / 3deployment-prep (beta): Automate updating the puppet checkout - 10https://bugzilla.wikimedia.org/66683#c4 (10Bryan Davis) a:3Bryan Davis I have setup a temporary cron job on deployment-salt as user bd808 to update the operations/puppet.git checkout using ~/bd808/git-sync-upstream. It run... [17:25:48] 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~erwin85/relatedchanges.php to Tool Labs - 10https://bugzilla.wikimedia.org/60872#c6 (10Andre Koopal) I know the issue, at the moment I don't have time to look into this in detail, if I make time I first want to look at why commons is not connecting... [17:40:20] Hi, I have a problem [17:41:16] !ask [17:41:17] Hi, how can we help you? Just ask your question. [17:43:09] When I run my tool in bash, it run normally, but when I run in jsub it return exit code 139 [17:44:41] Miguel2706: if it's over 128, it should mean it received signal 11 [17:44:43] 128 +11 [17:45:03] Yes, and it finish [17:45:16] SIGSEGV 11 Core Segmentation Fault [17:45:35] that's no good, but i'm afraid i dont have the answer why [17:45:49] so it's a segfault [17:45:56] This run with mono [17:46:09] Miguel2706: you probably don't have enough memory allocated for it [17:46:18] Miguel2706: try -mem 1G [17:46:33] Ok [17:49:47] Now it says error 1 [17:50:15] scfc_de: ^ any ideas? [17:51:19] I tryied to run with -sync y, and it says this [17:56:14] Miguel2706: maybe increase the limit even further? -mem 4G [18:04:08] YuviPanda: No additional ideas; SIGSEGV sounds strange for OOM, but Mono might do strange things as well. [18:04:16] yeah [18:05:13] Yeah, it works with -mem 1024, but now the problem is that I am running a Irc bot, but it says too many global connections [18:06:05] Miguel2706: aaah, I think there's a bug for that. [18:06:59] (sorry for my bad english:D) [18:07:20] Miguel2706: https://bugzilla.wikimedia.org/show_bug.cgi?id=64802 [18:07:51] Ok, I am reading... [18:08:21] you should add a comment there that you have the same issue and say which instance your bot is on [18:08:32] then hopefully somebody nags freenode [18:08:36] and they raise the limit [18:10:28] Ok, thanks :) [18:15:06] 3Wikimedia Labs / 3tools: IRC: Too many user connections (global) - 10https://bugzilla.wikimedia.org/64802#c8 (10Miguel Peláez) I, I have the same problem, my last instance was named intelirc, and it die when the server was disconnected :/ [18:56:28] Hey, this channel not have the +t flag! [18:59:01] Miguel2706: that's on purpose [18:59:37] Miguel2706: that allows anyone to update the topic if e.g. there's downtime [19:01:18] Ok [21:41:47] Anyone around who knows beta well? [21:50:55] Krenair: I know some things about beta well. WHat's up? [21:55:20] bd808, beta got some fatal exception that didn't occur on master - https://bugzilla.wikimedia.org/show_bug.cgi?id=66792 [21:55:37] * bd808 looks [21:58:53] Krenair: Didn't occur on master in what environment? The line that is failing seems to have a lot of chained method calls so it's hard for me to guess which element may have ended up being null. [21:59:27] "getDisplayTitle() on a non-object" [21:59:38] getParserOutput returned a non-object, probably false [21:59:51] It's supposed to do this when the revision ID is not found [21:59:57] We're not specifying any revision ID... [22:02:55] Krenair: I'll amend my "knowledge about beta" statement to apply to the hosts there and how they are configured. :) Eleven months on the job and my MW internals knowledge is still sadly lacking for helping debug things like this quickly. [22:48:12] Is anyone around? [22:48:28] Not sure what's causing this issue: http://tools.wmflabs.org/mono/test [22:50:35] looks [22:51:23] mono log in as the tool and run : webservice start [22:53:00] Betacommand: with SSH right? [22:53:14] yeah [22:53:47] Mono: your using a tool, and not your account correct? [22:54:32] it's definitely on tools, I set it up last year and I got in with SFTP [22:55:29] Mono: a tool != on tools\ [22:55:41] oh [22:55:56] Mono: I have the account betacommand [22:56:28] but the tool is betacommand-dev [22:57:10] Mono: let me see if I can find the docs [22:57:18] My account is Mono and my tool is mono [22:57:18] !tools [22:57:18] There are multiple keys, refine your input: tools-admin, toolsbeta, tools-bug, tools-equiad, toolsmigration, tools-request, tools-status, toolsvslabs, tools-web, [22:57:23] it's been a year though [22:58:08] Mono: then you should just need to start your webservice [23:00:03] Betacommand: where do I find the RSA host key for tools-login.wmflabs.org? [23:00:12] mine is out of date [23:02:18] Hey I have a problem, when I run "become intelirc", it says "sudo: sorry, a password is required to run sudo", and I do not run sudo [23:03:56] Mono: check wikitech [23:04:12] Miguel2706: one sec [23:04:18] Oki [23:04:39] Betacommand: is that in my preferences? [23:05:29] Miguel2706: http://lists.wikimedia.org/pipermail/labs-l/2014-March/002196.html [23:05:36] Mono: no [23:06:34] Mono: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [23:07:04] Thanks so much Betacommand :) [23:09:17] I'm getting "The fingerprint for the RSA key sent by the remote host is 2b:b2:5d:48:43:f0:27:8a:c1:ab:06:6b:f9:3c:b6:57." in the terminal but how do I actually get the key not the fingerprint for my known_hosts file? [23:11:40] Betacommand: ping ^^ :) [23:12:07] Mono: why do you need it? [23:13:23] Terminal (OS X) is complaining about a man-in-the-middle attack and preventing the connection [23:15:41] It says to update the known hosts file with the changed key [23:16:16] Mono: just remove the old entry and reconnect [23:27:58] bd808, do the beta hosts keep the debug log stored anywhere? [23:29:15] Krenair: Yes. Logs are in /data/project/logs on deployment-bastion (or actually any deployment-prep host) [23:29:33] They are also in logstash at https://logstash-beta.wmflabs.org [23:29:43] Which file? [23:30:25] Krenair: web.log is the catch all file [23:45:55] bd808, everything put through wfDebug should go to web.log right? [23:48:19] Krenair: I think so. Either there or in cli.log depending on if the command came from apache or a job runner or other command line invocation. [23:48:28] apache [23:48:43] Yeah. It should end up in web.log then [23:50:04] So I made https://gerrit.wikimedia.org/r/#/c/140591/1..2/ApiVisualEditorEdit.php [23:50:18] It's definitely on the deployment bastion: [23:50:21] krenair@deployment-bastion:/data/project/logs$ grep Debug /a/common/php-master/extensions/VisualEditor/ApiVisualEditorEdit.php [23:50:21] wfDebug( '[VE] ApiVisualEditorEdit - parserOutput was false' ); [23:50:45] I can see that 'displayTitleHtml' is not getting set in the response [23:50:58] Yet I also can't find '[VE] ApiVisualEditorEdit - parserOutput was false' in web.log [23:52:02] Krenair: enwiki-d80b922e: 1.2558 23.0M [VE] ApiVisualEditorEdit - parserOutput was false [23:52:45] krenair@deployment-bastion:/data/project/logs$ grep "enwiki-d80b922e: 1.2558 23.0M [VE] ApiVisualEditorEdit - parserOutput was false" web.log [23:52:46] krenair@deployment-bastion:/data/project/logs$ [23:52:49] There are 4 matches for 'parserOutput' in web.log [23:53:23] * bd808 nods (right dir) [23:54:20] Krenair: http://p.defau.lt/?2kvuobuGQ6bN5_3dou3_kA [23:57:22] bd808, huh. thanks [23:57:30] np