[06:29:04] is there a checklist for what I can do and what I can request, about the outage? [06:29:55] eg. request for retrieving files which were edited recently, then enable crontabs afterwards? [06:32:35] liangent: https://lists.wikimedia.org/pipermail/labs-l/2015-June/003831.html ? [06:41:09] zhuyifei1999: so, can I ask for "Recovery of modifications made after that date is potentially possible" mentioned in https://lists.wikimedia.org/pipermail/labs-l/2015-June/003829.html now? [06:43:18] ping Coren or YuviPanda for that [06:43:54] or andrewbogott [06:45:13] how about wildcat instances? i still can't ssh there, though the instance is still running [06:52:21] Danny_B|webgate: https://lists.wikimedia.org/pipermail/labs-l/2015-June/003824.html ? [10:22:11] I will let you know when I see YuviPanda around here [10:22:11] @notify YuviPanda [11:36:13] 6Labs: Unable to ssh to dwl - https://phabricator.wikimedia.org/T103245#1385829 (10yuvipanda) I just tried and I'm able to ssh to it. Are you using an appropriate ProxyCommand setup? What error are you seeing? Can you paste the output of ssh -v? [11:41:16] liangent: we do not know yet - fsck is still in progres... [11:52:52] 6Labs: Unable to ssh to dwl - https://phabricator.wikimedia.org/T103245#1385832 (10Giftpflanze) P817 - .ssh/config error: Permission denied (publickey). ssh_exchange_identification: Connection closed by remote host P818 - output of ssh -v [12:33:38] I thought that we had an SVG editor wmflabs [12:35:56] brion: didn't you have an SVG editor on labs somewhere? If you think that you still do, it is not at an active url [12:36:45] hells bells now something appears [13:25:00] * issyl0 reads the /topic and nods... good luck! [13:52:35] 6Labs, 10Tool-Labs: mcrypt not enabled on trusty - https://phabricator.wikimedia.org/T103061#1385878 (10Danmichaelo) [13:52:37] 6Labs, 10Tool-Labs: Tool Labs: Install php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1385879 (10Danmichaelo) [13:54:40] 6Labs, 10Tool-Labs: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1385881 (10Danmichaelo) [13:55:27] 6Labs, 10Tool-Labs: mcrypt not enabled on trusty - https://phabricator.wikimedia.org/T103061#1385885 (10Danmichaelo) [13:55:29] 6Labs, 10Tool-Labs: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1253572 (10Danmichaelo) [13:55:30] 10Tool-Labs-tools-Other: Croptool does not work on php 5.4 - https://phabricator.wikimedia.org/T103059#1385884 (10Danmichaelo) [13:56:18] 10Tool-Labs-tools-Other: Croptool does not work on php 5.4 - https://phabricator.wikimedia.org/T103059#1381423 (10Danmichaelo) [14:04:46] 6Labs: Kill NFS in scrumbugz project - https://phabricator.wikimedia.org/T102704#1385901 (10Christopher) I think that phab08 may not recoverable because it seems to be on a different network for some reason. It has the IP address of 10.68.17.0. The other project instances are on 10.68.16.* Anyway, no big deal... [14:17:59] 6Labs, 10Tool-Labs, 3ToolLabs-Goals-Q4: Switchover Labs NFS server to labstore1002 - https://phabricator.wikimedia.org/T97219#1385918 (10coren) 5Open>3Resolved This was made done, willy nilly, by the very forced emergency switch to 1002. [14:19:48] 6Labs, 10Labs-Infrastructure, 6operations, 3Labs-Sprint-100, and 2 others: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1385920 (10coren) The filesystem crashed caused us to... improvise around this plan a great deal. All but one project has been switched to a r... [14:22:51] 6Labs, 10incident-20150422-LabsOutage: Labs: investigate alternatives to maps' storage requirements - https://phabricator.wikimedia.org/T103264#1385921 (10coren) 3NEW [14:26:36] 6Labs, 10Incident-20150617-LabsNFSOutage: Labs: investigate alternatives to maps' storage requirements - https://phabricator.wikimedia.org/T103264#1385931 (10coren) [14:29:42] 6Labs, 10Incident-20150617-LabsNFSOutage: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1385936 (10coren) 3NEW [14:32:15] 6Labs: Labs: Reinstall labstore1001 with Jessie - https://phabricator.wikimedia.org/T103266#1385945 (10coren) 3NEW [14:34:01] 6Labs, 6operations, 10ops-codfw: Labs: Install the new RAID controller in labstore2001 and test - https://phabricator.wikimedia.org/T103267#1385952 (10coren) 3NEW [14:35:27] 6Labs: Labs: Reinstall labstore1001 with Jessie - https://phabricator.wikimedia.org/T103266#1385962 (10coren) [14:35:57] Coren: can you respond on the list about the maps project? [14:36:41] YuviPanda: I was hoping we'd have time to sit down and figure out a plan first, but I suppose an update saying "we have to sit down and figure out a plan first" also makes sense. [14:39:23] Coren: do communicate status of rsync / that maps project isn't being copied over, etc. [15:03:12] Hi every body. Ihave a problem with crontab on toollabs [15:04:49] i need some help !! [15:05:13] wahrani: hi. what is the issue? [15:06:56] when i use crontab to run a php script, t get an error and 3 files ( core + cron-tools.wahrani-1.out + cron-tools.wahrani-1.err) [15:08:24] content of cron-tools.wahrani-1.err : libgcc_s.so.1 must be installed for pthread_cancel to work [15:08:31] wahrani: unfortunately I've to leave now, but you can try emailing labs-l mailing list to see if someone might have a better answer? you can also file a bug with a more detailed description at phabricator.wikimedia.org? [15:08:47] wahrani: try increasing the value to -mem in jsub [15:08:50] to 1G or higher [15:08:55] it might be just running out of memory [15:29:20] @YuviPanda thank you very much, increasing the memory is the solution. what is the default memory size allocated by jsub ? [15:30:22] 256MB i think [15:33:44] ok [15:37:33] 6Labs, 10Incident-20150617-LabsNFSOutage: Recover files for project liangent-php - https://phabricator.wikimedia.org/T103268#1385989 (10liangent) 3NEW [16:03:04] 6Labs: Unable to ssh to dwl - https://phabricator.wikimedia.org/T103245#1386009 (10scfc) Can you `ssh` to `bastion.wmflabs.org`? That host's key changed recently and the error message if that host is used in a `ProxyCommand` is not obvious. [16:04:39] 6Labs: Unable to ssh to dwl - https://phabricator.wikimedia.org/T103245#1386010 (10Giftpflanze) Yes, I can. [17:05:17] 6Labs, 6Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1386072 (10scfc) 3NEW [18:01:09] 6Labs, 6Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1386107 (10scfc) If I understood http://permalink.gmane.org/gmane.org.wikimedia.labs.announce/49 correctly, the #Maps project is shut down until the NFS r... [18:12:18] 6Labs, 6Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1386130 (10MaxSem) Note tht reinstating these domains have ceased making sense after all wikis were made HTTPS-only. As a result of that, a single subdoma... [19:27:36] any news on the data recovery? [19:39:14] 10Quarry: "Your query is currently executing" showed when one is looking at running querry of other user - https://phabricator.wikimedia.org/T103275#1386185 (10Utar) 3NEW [19:54:28] YuviPanda: WDQ a bit overloaded? Seems to be lagging about 2 hours [20:12:09] 10Quarry, 7I18n: "Your query is currently executing" showed when one is looking at running querry of other user - https://phabricator.wikimedia.org/T103275#1386215 (10matej_suchanek) [20:23:49] 6Labs, 6Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1386225 (10Strainu) >>! In T103272#1386107, @scfc wrote: > If I understood http://permalink.gmane.org/gmane.org.wikimedia.labs.announce/49 correctly, the... [23:18:52] 6Labs, 6Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1386373 (10scfc) AFAIUI, http://wiki.openstreetmap.org/wiki/Tile_usage_policy explicitly prohibits switching to OSM tile servers without their prior appro... [23:50:02] hello [23:50:39] how can I stop bigbrother? it's trying to restart a deleted job [23:50:50] 2015-06-21 23:48:00 warn: job '' failed to start 2015-06-21 23:48:00 info: Restarting job '' [23:51:14] you cannot stop big brother. he is omnipresent. you should love big brother. [23:51:31] (joke) [23:51:36] he's sending me emails every couple of minutes :( [23:52:14] the job failed, I deleted the entry from .bigbrotherrc [23:52:17] but it didn't stop