[16:57:23] 10Wikimedia-Labs-wikitech-interface, 6operations: distribution upgrade for wikitech-static instance - https://phabricator.wikimedia.org/T94585#1167483 (10Dzahn) [17:20:36] 10Wikimedia-Labs-wikistats: MediaWiki - wiki registry - https://phabricator.wikimedia.org/T39062#1167540 (10Dzahn) should just make a simple HTML form that sends me a mail and be done with it? @RobiH do you have any wikis to add? [17:21:02] plese add .bigbrotherrc and restart https://tools.wmflabs.org/meetbot/ and https://tools.wmflabs.org/wdq2sparql/ [17:21:10] 10Wikimedia-Labs-wikistats: MediaWiki - wiki registry - https://phabricator.wikimedia.org/T39062#1167543 (10Dzahn) a:5Dzahn>3None [17:22:31] Coren, YuviPanda ^^? [17:22:54] 10Wikimedia-Labs-wikistats: deploy a replacement for the old "wikistats admin" (WSA) script - https://phabricator.wikimedia.org/T38287#1167551 (10Dzahn) Did RobiH migrate from BZ to phab? It seems not, which is unfortunate. Since i did deploy a replacement for WSA but i doubt anyone uses it, i'm tempted to clos... [17:22:59] jzerebecki: Give me a minute [17:25:18] jzerebecki: {{done}} [17:25:33] thx [17:31:41] 10Tool-Labs: Tool Labs: jsub starts multiple instances of tasks declared as "once" - https://phabricator.wikimedia.org/T62862#1167560 (10Krinkle) Indeed. Just spotted that dbbot-wm on IRC had three concurrent incarnations running. Presumably caused by the NFS maintenance earlier this week? ``` $ qstat job-ID p... [17:46:00] 10Wikimedia-Labs-wikitech-interface, 6operations: distribution upgrade for wikitech-static instance - https://phabricator.wikimedia.org/T94585#1167602 (10Dzahn) 10:29 < andrewbogott> mutante: regarding wikitech-static, that’s a VM. So I’m sure it would be better to build a fresh one rather than try to upgrade... [18:26:35] 10Wikimedia-Labs-wikitech-interface, 6operations: distribution upgrade for wikitech-static instance - https://phabricator.wikimedia.org/T94585#1167742 (10Dzahn) quote from https://wikitech.wikimedia.org/wiki/Wikitech-static "A (non-puppetized) cron on wikitech-static runs /usr/local/sbin/import-wikitech.sh... [18:49:29] wikibugs is not sending some updates. E.g. I made a comment in a task, but it didn't appear on IRC, https://phabricator.wikimedia.org/T94502#1167801 [18:50:01] I don't see anything related in https://phabricator.wikimedia.org/tag/wikibugs/ so I'll file a new bug... [18:53:48] 10Wikibugs: Wikibugs not sending some new comments to IRC - https://phabricator.wikimedia.org/T94598#1167840 (10Quiddity) 3NEW [18:53:59] There ya go. :) [18:54:23] legoktm, val[tab]damnit, YuviPanda ^ [18:58:19] hi [18:59:43] When did tools-login.wmflabs.org SSH public key change? [18:59:52] vvv: a while ago, there was an email... [18:59:57] vvv: it moved to trusty as well [19:00:00] Hm, email [19:00:10] That sounds like there's a mailing list [19:00:28] vvv: https://lists.wikimedia.org/mailman/listinfo/labs-l [19:00:41] Oh, that one [19:00:52] Is there a list similar to toolserver-announce? [19:00:52] . [19:01:23] vvv: https://lists.wikimedia.org/mailman/listinfo/labs-announce but it was only recently created, we haven’t been using it much yet [19:12:53] YuviPanda: disappointing :p [19:13:07] JohnFLewis: :P In my defense, I moved continents [19:13:44] YuviPanda: excuses! [19:24:39] 6Labs: Comprehensive monitoring / alerting for labstore* instances - https://phabricator.wikimedia.org/T94606#1167986 (10yuvipanda) 3NEW [19:30:27] 6Labs, 3ToolLabs-Goals-Q4: Test labstore switchover - https://phabricator.wikimedia.org/T94607#1168005 (10coren) 3NEW [19:30:51] 6Labs: Create a simple checklist to follow for announcing / doing planned maintenance (on labs) - https://phabricator.wikimedia.org/T94608#1168013 (10yuvipanda) 3NEW [19:32:51] Anyone want to do a quick query for me? I want a list of categories on enwiki with the string "Needed-Class" in them. [19:35:11] 6Labs, 3ToolLabs-Goals-Q4: Reinstall labstore1001 with Jessie - https://phabricator.wikimedia.org/T94609#1168033 (10coren) [19:36:18] 6Labs, 3ToolLabs-Goals-Q4: Test labstore switchover - https://phabricator.wikimedia.org/T94607#1168038 (10coren) [19:36:19] 6Labs, 3ToolLabs-Goals-Q4: Reinstall labstore1001 with Jessie - https://phabricator.wikimedia.org/T94609#1168026 (10coren) [19:37:11] 6Labs, 5Patch-For-Review: Process for user backups - https://phabricator.wikimedia.org/T85608#1168048 (10coren) [19:37:11] 6Labs, 3ToolLabs-Q4-Sprint-1: Sync up the new labs NFS project filesystem with the live one - https://phabricator.wikimedia.org/T93792#1168046 (10coren) 5Open>3Resolved This is complete, with some pain. [19:37:21] harej: http://pastebin.com/iMzBRL07 [19:37:34] thanks! [19:48:29] 6Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: dhclient overwrites /etc/resolv.conf - https://phabricator.wikimedia.org/T93691#1168076 (10Dzahn) ``` modules/labs_vmbuilder/files/postinst.sh: rm -f /etc/resolv.conf ``` ``` modules/base/manifests/resolving.pp: file { '/etc/resolv.conf': modules/base/m... [19:50:34] 6Labs: Create scripts to help stagger restarts of labs VMs by different criteria - https://phabricator.wikimedia.org/T94613#1168086 (10yuvipanda) 3NEW [20:16:30] 6Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: dhclient overwrites /etc/resolv.conf - https://phabricator.wikimedia.org/T93691#1168123 (10Andrew) I have some faith in that last patch -- hopefully no one will find it too offensive. [20:24:48] I can't do ls in tools-login (tools-bastion-01) and my web tool is down, is this known/continuation of yesterday's outage? [20:28:51] 6Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: dhclient overwrites /etc/resolv.conf - https://phabricator.wikimedia.org/T93691#1168159 (10scfc) We could disable it like you did in that patch with `nodnsupdate`, but I don't //like// plastering yet another layer over `dhclient`, `resolvconf`, and what else Ubun... [20:35:53] Coren: ^ (from jem) [20:36:00] I'm still at lunch [20:36:22] jem: Is your login session dating from before the switch? [20:38:52] Yes, coren [20:39:13] Ok, I reconnect :) [20:39:21] jem: You'll have to log off then back in again. Interruption of existing sessions is an expected effect. :-) [20:40:18] Hi.. could someone guide me about connecting to the Wiki db from an external app, running on a different server? Is it possible? [20:40:56] sam_: You mean outside of Labs? [20:41:17] Yeah, e.g. running from my own local server? [20:42:00] Basically, I would like to have the resultset of my queries in my application [20:43:06] right now, I can do this: "sql enwiki 'select *...' > filename.txt" , then download the file, and then process it [20:43:18] sam_: You can't generally connect directly (at least, not efficiently); you'll have to write a backend here on labs that does the queries. The simplest way to do this is to write a small web application that provides an API of sorts for you to use. It's fairly simple to do, for instance, if you respond with json or somesuch. [20:43:22] i am sure there is a better way to do it... i hope? [20:43:32] Ok, reconnected and webservice restarted, everything Ok [20:43:46] For occasional queries, you can tunnel a mysql connection over SSH. [20:43:48] Thanks, Coren [20:44:09] that makes sense. [20:47:03] Coren, could you tunnel mysql over ssh when you are using it in an application? [20:47:16] sorry if newbie question - new to all this. [20:47:45] sam_: Well, it's not newbie question because it's not trivial to do if you're not used to it. :-) But newbie questions are okay too. [20:47:58] i am currently using it over ssh to create the file from a terminal, and then sftping the file down to my application. [20:48:22] Your application would need to start ssh and turn on port forwarding instead; and connect to the (local) port provided by ssh. [20:48:39] Coren: O_O :D [20:53:19] Coren: if I setup a server on bastion for API access, would it stay up over a months? Or would it be shut down by some garbage colllection-like process? [20:53:56] "server on bastion" sounds like defeats the purpose of bastion, imho [20:54:19] mutante: Misunderstanding; a webserver would run on the grid. :-) [20:55:10] sam_: The relevant documentation for a web service is at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web [20:55:17] ah:) [20:55:26] sam_: And the service that would restart it is at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Bigbrother [20:55:34] I am sure I am messing up the vocabulary... [20:55:37] thanks! [21:32:06] there was an environmental variable set on instances that contained the project, what was the name again? [21:32:52] got it, $INSTANCEPROJECT [21:48:37] 10Tool-Labs, 3ToolLabs-Goals-Q4, 3ToolLabs-Q4-Sprint-1: Add more Trusty exec nodes - https://phabricator.wikimedia.org/T94304#1168470 (10yuvipanda) @scfc if you look at https://tools.wmflabs.org/?status, all other exec nodes have 'free vmem' that's quite a bit more than physical RAM, while total VMEM availab... [21:50:00] Coren: andrewbogott I’m going to copy the etherpads onto wikitech and mail out. any objections? [21:50:03] YuviPanda: Can you check the nginx configs on project-proxy in regards to https://phabricator.wikimedia.org/T94413 [21:50:15] YuviPanda: sounds good [21:52:12] 6Labs, 6Phabricator: Phab-02 sending old stylesheet copies - https://phabricator.wikimedia.org/T94413#1168477 (10yuvipanda) So sendfile is turned on, however since it is just proxying things I do not know if that's actually causing this issue or not. [21:53:46] 6Labs, 6Phabricator: Phab-02 sending old stylesheet copies - https://phabricator.wikimedia.org/T94413#1168478 (10yuvipanda) (I don't mind turning it off if needed since shouldn't affect our usecase, but I am not sure if that'll help you) [21:56:12] 6Labs, 6Phabricator: Phab-02 sending old stylesheet copies - https://phabricator.wikimedia.org/T94413#1168487 (10Negative24) >>! In T94413#1168478, @yuvipanda wrote: > (I don't mind turning it off if needed since shouldn't affect our usecase, but I am not sure if that'll help you) Do you mind turning it off f... [21:56:15] 6Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: dhclient overwrites /etc/resolv.conf - https://phabricator.wikimedia.org/T93691#1168488 (10Andrew) I'm open to any suggestions -- I've spent all day trying to do it the 'right way' cooperating with dhclient and I'm done. [21:56:25] YuviPanda: None (sorry, just back from Dinner) [21:56:35] timezones still screwing with me [21:56:50] I'm -4 atm [21:57:10] yeah, I just don’t have an intuitive feel for any of it at all. I will in a while I hope [21:57:49] YuviPanda: This may or may not work. That's what testing is for [21:58:01] Negative24: probably won’t be able to get to it atm, sorry. [21:58:12] YuviPanda: that's fine [22:05:31] Coren: this did put a damper on our ‘sprint’ :) [22:06:25] YuviPanda: Meh; it's a day lost. With luck (knock on wood) I'll make good headway on the idmap thing anyways. [22:06:33] true [22:06:33] Besides, it *is* a closed task now. :-) [22:06:53] heh. however, the fact that we don’t know the root cause of it makes me feel ugh [22:06:53] still [22:07:27] I'm keeping a window open to look at the graph; if it recurs, at least, I'll be in a good position to grab some hard data. [22:08:37] The original fail isn't worrying me so much as the mysteriously breaking rsync 6 hours in. [22:08:47] Coren: yeah, that’s what worries me too [22:09:56] Also I need to resume tracking down that 5m cron job that's so network-heavy. [22:10:25] Catching one cron job out of at least 90 on that schedule lasting less than a minute in the noise is... "fun" detective work. [22:11:05] Coren: can’t you use iftop to see which host the traffic is coming from, and then see which jobs ran there, and narrow that down? [22:11:21] and a few times of doing that should narrow it down I guess... [22:11:40] YuviPanda: I have; I've got about half of the jobs eliminated already. [22:11:53] :D cool. [22:12:23] But that it comes from cron makes it extra fun; they start in various combinations on various exec nodes. :-) [22:12:57] Once I'm down to a half dozen plausible culprits, I'll add instrumentation to the jobs themselves. [22:15:36] Coren: I’m trying to setup a ‘checklist’ + a template phab task… https://etherpad.wikimedia.org/p/labs-maint-checklist [22:15:39] andrewbogott: ^ [22:19:05] 10Tool-Labs, 3ToolLabs-Goals-Q4, 3ToolLabs-Q4-Sprint-1: Add more Trusty exec nodes - https://phabricator.wikimedia.org/T94304#1168579 (10scfc) Hmmm. The "old" exec nodes have 2 GBytes of swap compared to 0.5 GByte for the new ones, but that can't be the difference. Looking at `www/content/status.php` led t... [22:22:37] 6Labs: Create a simple checklist to follow for announcing / doing planned maintenance (on labs) - https://phabricator.wikimedia.org/T94608#1168617 (10yuvipanda) WIP in https://etherpad.wikimedia.org/p/labs-maint-checklist [22:22:44] 6Labs: Create a simple checklist to follow for announcing / doing planned maintenance (on labs) - https://phabricator.wikimedia.org/T94608#1168618 (10yuvipanda) a:3yuvipanda [22:54:39] 10Tool-Labs, 3ToolLabs-Goals-Q4, 3ToolLabs-Q4-Sprint-1: Add more Trusty exec nodes - https://phabricator.wikimedia.org/T94304#1168730 (10coren) They come from the /node/ configuration; (qconf -se) [23:53:03] 10Tool-Labs, 3ToolLabs-Goals-Q4, 3ToolLabs-Q4-Sprint-1: Add more Trusty exec nodes - https://phabricator.wikimedia.org/T94304#1168808 (10scfc) Indeed (another one of those SGE idiosyncrasies that I'll probably never understand :-)); is there a "formula" for the conversion physical memory => virtual memory, o... [23:55:35] 6Labs, 6operations: Make morebots run on a production host - https://phabricator.wikimedia.org/T94638#1168809 (10yuvipanda) 3NEW