[00:04:59] oh great, you truncated all my files :( [00:05:08] fuck [04:45:29] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Phabricator, 15User-bd808: Create a Conduit API method to lookup Policy information - https://phabricator.wikimedia.org/T137004#2356612 (10bd808) [04:57:57] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-faq-throttle-wait-title/en]] needs PLURAL - https://phabricator.wikimedia.org/T136027#2320688 (10Niharika) >>! In T136027#2356285, @MusikAnimal wrote: > Done. I am trying to get better at this, thank you for working with me! @MusikAnimal Was this fix made... [06:53:07] 10Tool-Labs-tools-Other: TextCatDemo 500/503 errors - https://phabricator.wikimedia.org/T137085#2356731 (10Nemo_bis) [08:31:58] 06Labs, 10DBA: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2341449 (10jcrespo) After seeing many cases like this, I can conclude that replication to labs breaks whenever there is a page move, an archival or an undeletion. I have not yet clear why, but... [08:33:54] 06Labs, 10DBA: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2356836 (10jcrespo) Of course, I can fix individual reports, although we should setup a more convenient way than a ticket per row with problems. [09:23:00] chasemp: you deleted all of (well, most of) my data :( how can i have it back? [09:23:37] and why did you empty files that were not out files but something like out-... [10:53:46] (03PS1) 10Jean-Frédéric: [FAILING] Add test ensuring correct usage of generateRegistrantUrl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) [10:54:33] (03CR) 10jenkins-bot: [V: 04-1] [FAILING] Add test ensuring correct usage of generateRegistrantUrl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) (owner: 10Jean-Frédéric) [11:38:33] 10Tool-Labs-tools-Other, 10Phabricator, 15User-bd808: Stashbot shouldn't subscribe itself to tasks - https://phabricator.wikimedia.org/T135790#2357155 (10Danny_B) Thanks to both! [12:00:30] 10Tool-Labs-tools-Other: TextCatDemo 500/503 errors - https://phabricator.wikimedia.org/T137085#2357198 (10TJones) There is text under the query box explaining, among other things, that you can hit to submit your query. I haven't exhaustively tested all browsers. It works on Chrome, Firefox, and Safari o... [13:12:12] (03CR) 10Lokal Profil: [C: 04-1] "Typo in docstring otherwise good." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) (owner: 10Jean-Frédéric) [13:35:08] (03CR) 10Lokal Profil: "Looks good. I'm just trying to get docker-compose up and running to actually test it." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) (owner: 10Jean-Frédéric) [13:48:26] (03CR) 10Lokal Profil: "OK. So I get `gzip: /docker-entrypoint-initdb.d/monuments_db.sql.gz: Permission denied` when running "docker-compose up" (first time or af" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) (owner: 10Jean-Frédéric) [14:08:37] (03PS2) 10Jean-Frédéric: Remove illegal uses of generateRegistrantUrl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) [14:12:43] (03PS3) 10Jean-Frédéric: Remove illegal uses of generateRegistrantUrl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) [14:14:27] (03CR) 10Jean-Frédéric: "Sorry Lokal Profil − had missed your comment before submitting a new patch. Comments addressed in latest patch" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) (owner: 10Jean-Frédéric) [15:00:58] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-faq-throttle-wait-title/en]] needs PLURAL - https://phabricator.wikimedia.org/T136027#2357547 (10MusikAnimal) @Niharika on [[ https://github.com/MusikAnimal/pageviews/commit/84f05b301629b2b0842a7dcd70df3d5ca2ffcdc7#diff-3c39d4ed69f4c5a996c0270b36a3fdb8R20... [15:06:31] 06Labs: Puppet stale on labtestweb2001 - https://phabricator.wikimedia.org/T136611#2357562 (10Andrew) 05Open>03Resolved [15:09:43] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-faq-throttle-wait-title/en]] needs PLURAL - https://phabricator.wikimedia.org/T136027#2357590 (10Niharika) @MusikAnimal Ah. This ticket wasn't associated with our projects, hence I thought this fix was for a different project. :) [15:10:19] (03CR) 10Jean-Frédéric: "> OK. So I get `gzip: /docker-entrypoint-initdb.d/monuments_db.sql.gz: Permission denied` when running "docker-compose up" (first time or " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) (owner: 10Jean-Frédéric) [15:12:03] It's been a few days now, any update on labsdb1001? [15:33:34] ShoeMaker: is that https://phabricator.wikimedia.org/T132431 ? [15:34:23] andrewbogott: the topic for the channel says "Status: labsdb1001 down, investigation ongoing." [15:34:30] Oh, that :) [15:34:39] Not sure if there is a specific tracker for it or not. [15:34:41] I'm not involved but I see other people working on it in another channel [15:35:45] (03CR) 10Lokal Profil: "Not running as sudo but as member of docker group (got same issue when runnign as sudo though)." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) (owner: 10Jean-Frédéric) [15:38:48] (03CR) 10Lokal Profil: [C: 032] "Re: merging tactic" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) (owner: 10Jean-Frédéric) [15:40:38] (03CR) 10Jean-Frédéric: "> Not running as sudo but as member of docker group (got same issue when runnign as sudo though)." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) (owner: 10Jean-Frédéric) [15:40:56] (03Merged) 10jenkins-bot: Remove illegal uses of generateRegistrantUrl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292908 (https://phabricator.wikimedia.org/T137096) (owner: 10Jean-Frédéric) [15:46:30] !log tools.heritage Deployed latest from Git: b77b7c2 (T137096) [15:46:31] T137096: Several countries use `generateRegistrantUrl` converter without defining a 'registrantUrlBase' - https://phabricator.wikimedia.org/T137096 [15:46:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [15:55:37] 10Labs-Other-Projects, 06Discovery, 06Maps: Receiving warning emails about failed puppet runs - multiple instances - https://phabricator.wikimedia.org/T135942#2357786 (10notconfusing) @yuvipanda do you have any suggestions? [15:56:11] andrewbogott: ^ can you take a look at this? [15:56:21] ssssure [15:59:20] ShoeMaker: I'm told that whatever issues labsdb1001 is having shouldn't really matter to users, so I removed the comment from the topic. (It was down, briefly, a few days ago) [16:00:39] Okay, because I'm still getting an email from the Cron Daemon every five minutes from xTools complaining about it... [16:02:12] 10Labs-Other-Projects, 06Discovery, 06Maps: Receiving warning emails about failed puppet runs - multiple instances - https://phabricator.wikimedia.org/T135942#2357803 (10Andrew) a:03Andrew It looks to me like this is hanging because of /public/dumps issues. We've been facing similar issues over the last f... [16:02:49] ShoeMaker: can you open a ticket about that? [16:03:59] I'm checking with musikanimal in the #wikimedia-xtools channel to see if he knows if it is in fact related to the db or something else new that was added. [16:04:09] If he doesn't know, I'll put in a ticket. [16:05:14] ShoeMaker: I don't think email you're getting is related, or shouldn't be [16:06:43] great, let me know if there turns out to be something I can do [16:06:50] Will do. :) [16:11:45] (03PS2) 10Lokal Profil: Ensure all monuments_config entries are used [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292533 (https://phabricator.wikimedia.org/T136704) [16:14:31] (03CR) 10Lokal Profil: "Possibly give this a day or two to see if anyone objects in the phabricator task." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292533 (https://phabricator.wikimedia.org/T136704) (owner: 10Lokal Profil) [16:17:56] (03CR) 10Lokal Profil: "Which OS are you running? My quick googling (https://github.com/docker/docker/issues/1295) suggested created directories might behave diff" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/291198 (https://phabricator.wikimedia.org/T136351) (owner: 10Jean-Frédéric) [16:19:09] Hi anyone, I'm interested in having the Libertine package avaiable in LaTeX from inside tool-labs, what should I do? [16:20:10] jem: if there are standard packages available upstream for debian and ubuntu then it shouldn't be hard to do. Just open a phab task in the 'labs' project [16:20:26] Yes, it seems it's standard [16:21:09] jem: if you're ambitious you can submit a puppet patch, otherwise I'll do it :) [16:21:29] That seems a little too much for me :) [16:21:46] I can open a phab task, but it seems it can take months in there [16:21:55] (From previous experiences) [16:22:51] jem: create a phab and then bug me about it [16:22:56] Ok [16:23:00] Thanks, andrewbogott [16:23:24] Getting on it [16:28:26] 06Labs, 10Tool-Labs, 06Operations, 10netops: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2357927 (10Andrew) 05Open>03Resolved Update: the bitninja people seem to be wrong about everything. I'm closi... [16:28:40] 06Labs: Add libertine package for LaTeX - https://phabricator.wikimedia.org/T137121#2357929 (10-jem-) [16:29:00] andrewbogott: Ready, https://phabricator.wikimedia.org/T137121 [16:29:57] 06Labs, 10Tool-Labs, 06Operations, 10netops: 'German Wikipedia Broken Weblinks Bot' is ill-behaved and in danger of getting all of Labs blacklisted - https://phabricator.wikimedia.org/T136829#2357942 (10MoritzMuehlenhoff) FTR, I received the same vague reply as Andrew, seems mostly auto-generated... [16:41:02] Mmmm, maybe I should have linked https://www.ctan.org/tex-archive/fonts/libertine/, but probably it's obvious enough... [16:47:54] jem: it looks to me like that package was already installed… are you seeing otherwise? [16:51:20] andrewbogott: Well... when I try to use it I get "package not found" [16:51:31] But not when I use charter instead [16:51:40] jem: dpkg —list | grep libertine [16:51:43] it's there, right? [16:52:05] ! LaTeX Error: File `libertine.sty' not found. [16:52:05] hello :) [16:53:07] I'm trying to make dpkg work... [16:53:42] 06Labs, 13Patch-For-Review: Add libertine package for LaTeX - https://phabricator.wikimedia.org/T137121#2357929 (10Andrew) The package fonts-linuxlibertine was already installed everywhere thanks to mediawiki::packages::fonts. So we need more info about what's actually failing/missing here. [16:53:49] Ah, double dash :) [16:55:00] andrewbogott: Yes, I see it, but maybe it's just a general system font package and not the LaTeX package [16:55:36] jem: I don't think I know enough to know the difference between those things. [16:56:28] Hm [16:57:35] Ok, thanks anyway, andrewbogott [16:57:58] jem: I'm happy to install new packages but we need to figure out what the actual problem is first :) [16:58:04] Of course [16:58:36] Probably it's included in some of the LaTeX packages [17:00:03] 06Labs, 13Patch-For-Review: Add libertine package for LaTeX - https://phabricator.wikimedia.org/T137121#2357929 (10MoritzMuehlenhoff) libertine.sty comes from texlive-fonts-extra , not fonts-linuxlibertine' [17:00:22] Hmm, it seems it's https://packages.debian.org/sid/texlive-fonts-extra [17:01:00] That one is not installed, indeed [17:01:41] andrewbogott: Could you check that one out? [17:01:48] yes, looking [17:01:52] Thanks :) [17:07:32] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: tools.merlbot stopped working - https://phabricator.wikimedia.org/T135006#2358097 (10bd808) [17:07:34] 10Tool-Labs-tools-Other, 07Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#2358096 (10bd808) [17:43:23] jem: try now? [17:55:35] andrewbogott: Working great! Thanks :) [18:22:10] 06Labs, 13Patch-For-Review: Add libertine package for LaTeX - https://phabricator.wikimedia.org/T137121#2358350 (10-jem-) 05Open>03Resolved a:03Andrew Checked and working, thanks! And sorry for the Debian package vs. LaTeX package confusion, I should have explained better. [18:49:57] How can xdebug be disabled for my application on Tool Labs? [18:50:14] It's severely impacting the performance of my app. [18:51:40] Have you measured it's xdebug that caused it? [18:52:16] In any case, you can override php.ini using .user.ini, or by overriding the full lighttpd configuration [19:13:09] 06Labs, 06Discovery, 06Operations, 10hardware-requests: rack/upgrade/setup/install/deploy relforge100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T136708#2358627 (10RobH) a:05RobH>03Cmjohnson assigned to @Cmjohnson for the disk installations/onsite part. Once done feel free to assign back to m... [20:07:20] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2358771 (10tom29739) [20:25:28] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2358771 (10valhallasw) As I mentioned on IRC: * Have you //measured// that it's xdebug that causes the slowdown? * you can override php.ini using .user.ini * you can override the full lighttpd configuration if that is... [21:53:07] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2358771 (10Legoktm) FWIW I think it's well known that xdebug does slow down requests and isn't intended for production environments (e.g. https://getcomposer.org/doc/articles/troubleshooting.md#xdebug-impact-on-composer) [22:08:25] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2359347 (10Matthewrbowker) a:03Matthewrbowker >>! In T137146#2358842, @valhallasw wrote: > As I mentioned on IRC: > > * Have you //measured// that it's xdebug that causes the slowdown? > * you can override php.ini u... [22:08:59] Hello all. [22:09:10] How do I configure my labs vm instance to send e-mail properly? [22:09:42] Is this documented on a wikipage somepalce? [22:10:37] hey dapatrick [22:10:51] bd808 might know [22:10:51] depends on how you define 'properly', I guess :D they are all setup with some default config that lets 'mail' work [22:10:59] (mail from the commandline) [22:11:02] Exim is running, but echo -n "Test" | mail -s Test dpatrick@wikimedia.org results in errors in the log. [22:11:17] hmm, that should work. let me take a look [22:11:22] dapatrick: which instance is this? [22:11:41] jobs.security-tools.eqiad [22:11:52] mediawiki-vagrant traps all the email if you are using that [22:12:03] I don't have that role enabled. [22:19:23] it is also trusty [22:19:56] Is it a problem that it is trusty? [22:21:25] dapatrick: no, but we were aware of some vague mail problems in jessie [22:21:27] none in trusty [22:21:29] so it's new etc [22:21:41] I see. [22:35:56] dapatrick: it looks like it is trying to talk to the prod mail server... [22:36:04] I'll look through puppet changes to see what happened [22:36:07] dapatrick: can you file a bug? [22:36:22] Okay. [22:37:57] dapatrick: oh, interesting. my mail did actually make it [22:38:54] dapatrick: I sent you a mail as well just now, with subject 'Testing mail server'. [22:39:03] 06Labs: Outgoing mail not working on new labs instance - https://phabricator.wikimedia.org/T137160#2359435 (10dpatrick) [22:39:52] yuvipanda: I don't think yours made it. [22:39:59] The log shows a failure. [22:40:28] dapatrick: yeah, I think it seems to work fine for @gmail and fail for wikimedia addresses [22:41:46] Yep, I was able to send externally as well. [22:42:13] 06Labs: Outgoing mail to wikimedia.org not working on new labs instance - https://phabricator.wikimedia.org/T137160#2359458 (10yuvipanda) [22:43:28] 06Labs: Outgoing mail to wikimedia.org not working on new labs instance - https://phabricator.wikimedia.org/T137160#2359435 (10yuvipanda) It seems to work fine for non wikimedia.org mail addresses: ``` 2016-06-06 22:26:49 1bA2yv-0000bk-RQ "root@jobs.security-tools.eqiad.wmflabs" from env-from rewritten as "root... [22:43:41] Hmm. [22:43:43] Local mail is blackholed. [22:43:45] Is that expected? [22:44:00] I just tried to send to dpatrick. [22:44:06] let's say we're both discovering this together, so nothing is expected :) [22:44:19] Okay. [22:44:21] :) [22:44:42] (I'm just catching up on mail, which was an area I never used to touch) [22:53:18] dapatrick: mailx -r 'yuvipanda@gmail.com' yuvipanda@wikimedia.org [22:53:20] works [22:53:46] 06Labs: Outgoing mail to wikimedia.org not working on new labs instance - https://phabricator.wikimedia.org/T137160#2359495 (10yuvipanda) ``` mailx -r 'yuvipanda@gmail.com' yuvipanda@wikimedia.org ``` works well and delivers it fine. [22:57:42] yuvipanda: That's a useful workaround. I'll make note of that. But that's not how cron sends mail. [22:57:49] yep [22:58:15] Local transmission is also broken from cron. Should I file that as a separate phab ticket? [22:58:54] dapatrick: ya [22:58:55] *yes [23:02:36] 06Labs: Local mail delivery not working on new labs instance - https://phabricator.wikimedia.org/T137165#2359548 (10dpatrick) [23:13:57] Hey folks. [23:14:07] I'm trying to figure out how to get 160GB on my xlarge instance [23:14:13] I can only see 20GB for / [23:14:38] 06Labs, 10Labs-Infrastructure: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#2359625 (10faidon) [23:15:01] halfak: sorry, it's the wrong phase of the moon. Your extra space is in another castle. [23:15:08] Nooooo [23:15:09] :D [23:15:15] srsly tho [23:15:43] halfak: in wikitech, in configure instance, you can select the 'role::labs::lvm::srv' option. that'll provision a 140G /srv [23:15:52] halfak: alternative is to use lvm commandline tools yourself. [23:16:51] * halfak configures and runs puppet [23:16:51] :) [23:17:27] It worked! [23:17:35] <3 yuvipanda [23:18:11] halfak: :D yw! [23:20:05] 06Labs: Local mail delivery not working on new labs instance - https://phabricator.wikimedia.org/T137165#2359548 (10faidon) This is because of 0295b935034c685936f5a79c273696df6bb4521c. We intentionally blackhole mails to root@, as there is no infrastructure support at the moment to direct root@ to the instance o... [23:29:19] 06Labs: Local mail delivery not working on new labs instance - https://phabricator.wikimedia.org/T137165#2359686 (10dpatrick) @faidon: Local mail to dpatrick is also blackholed. Is that expected behavior based on this configuration change? [23:32:54] 06Labs: Local mail delivery not working on new labs instance - https://phabricator.wikimedia.org/T137165#2359688 (10faidon) Yes, indirectly. Like production, the configuration rewrites all individual accounts to root@ (the last line of the exim config), which is then rewritten to :blackhole: based on the change... [23:46:14] 06Labs, 06Community-Tech, 15User-bd808: Create project Google-api-proxy - https://phabricator.wikimedia.org/T136862#2359723 (10DannyH) [23:48:00] 06Labs: Local mail delivery not working on new labs instance - https://phabricator.wikimedia.org/T137165#2359731 (10faidon) 05Open>03declined I'll decline — the current situation is working as intended; the problem is real, though, but the right solution is being tracked indirectly by T41785. If you disagree... [23:54:09] 06Labs: Outgoing mail to wikimedia.org not working on new labs instance - https://phabricator.wikimedia.org/T137160#2359786 (10faidon) On 0295b935034c685936f5a79c273696df6bb4521c I mentioned: > […] > On a later step, Labs could get its own mail relays entirely; this split config would allow for this. > > For mai...