[00:09:56] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1469484 (10bd808) 3NEW [00:59:02] 6Labs: Ask Github nicely to whitelist Tool Labs - https://phabricator.wikimedia.org/T100196#1469569 (10Sitic) [00:59:04] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1469570 (10Sitic) [00:59:51] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1469572 (10yuvipanda) We clearly need someone with the word 'Manager' in their title to ask GitHub. [01:00:58] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1469573 (10Jdlrobson) @abaso this would be a great use of twitter :) [01:01:23] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1469575 (10Jdlrobson) [01:05:22] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1469583 (10bd808) >>! In T106452#1469572, @yuvipanda wrote: > We clearly need someone with the word 'Manager' in their title to ask GitHub. What a... [01:58:47] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1469655 (10Ricordisamoa) [02:49:08] 6Labs, 10Tool-Labs: webservice uses default, non-working mail address - https://phabricator.wikimedia.org/T106462#1469709 (10scfc) 3NEW [03:00:25] 6Labs, 10Tool-Labs, 10pywikibot-core, 7Easy: Pywikibot : Fix scripts broken by toolserver.org to labs migration - https://phabricator.wikimedia.org/T78462#1469723 (10jayvdb) @daniel declared WikiSense dead at T60869#1365653 . [03:10:59] 6Labs, 10Tool-Labs, 6Commons, 10pywikibot-core, 7Easy: Pywikibot : Fix scripts broken by toolserver.org to labs migration - https://phabricator.wikimedia.org/T78462#1469732 (10jayvdb) As they are all #Commons related, I've asked onwiki: https://commons.wikimedia.org/wiki/Commons:Village_pump#Replacements... [03:11:21] 6Labs, 10Tool-Labs, 6Commons, 10pywikibot-core, 7Easy: Pywikibot : Fix Commons scripts broken by toolserver.org to labs migration - https://phabricator.wikimedia.org/T78462#1469735 (10jayvdb) [03:32:03] 6Labs, 10Tool-Labs, 10Wikimedia-Git-or-Gerrit: git clone operations/mediawiki-config fail: recursion detected in die_errno handler - https://phabricator.wikimedia.org/T106393#1469770 (10zhuyifei1999) Hmm. Tested again on another labs instance, might be just an issue with tool labs: ``` zhuyifei1999@commonsar... [03:32:50] 6Labs, 10Tool-Labs, 10Wikimedia-Git-or-Gerrit: git clone operations/mediawiki-config on tool labs fail: recursion detected in die_errno handler - https://phabricator.wikimedia.org/T106393#1469773 (10zhuyifei1999) [03:35:54] 6Labs, 10Tool-Labs, 10Wikimedia-Git-or-Gerrit: git clone operations/mediawiki-config on tool labs fail: recursion detected in die_errno handler - https://phabricator.wikimedia.org/T106393#1469780 (10yuvipanda) Was the other labs instance NFS as well? [03:41:47] 6Labs, 10Tool-Labs, 10Wikimedia-Git-or-Gerrit: git clone operations/mediawiki-config on tool labs fail: recursion detected in die_errno handler - https://phabricator.wikimedia.org/T106393#1469787 (10zhuyifei1999) No, commonsarchive-test doesn't use nfs. Cloning simple repos like mediawiki/exiensions/example... [05:33:32] 6Labs, 10Tool-Labs: Possible Tool Labs database corruption in enwiki_p - https://phabricator.wikimedia.org/T106470#1469836 (10MZMcBride) 3NEW [05:38:56] 6Labs, 10Tool-Labs, 7Database: Possible Tool Labs database corruption in enwiki_p - https://phabricator.wikimedia.org/T106470#1469844 (10Legoktm) [06:27:07] 6Labs, 10Tool-Labs, 10Wikimania-Hackathon-2015, 3Labs-Sprint-106: Conduct a Tool Labs Workshop in Wikimania hackathon - https://phabricator.wikimedia.org/T91061#1469873 (10yuvipanda) 5Open>3Resolved [06:37:18] 6Labs, 10Labs-Infrastructure, 3ToolLabs-Goals-Q4: Limit NFS bandwith per-instance - https://phabricator.wikimedia.org/T98048#1469890 (10yuvipanda) I'm inclined to close this as WONTFIX because of migrations to RAID10 and other NFS fixes and no responses in quite a long time. [06:42:13] 6Labs: Make continuous backups of NFS data to codfw - https://phabricator.wikimedia.org/T106474#1469894 (10yuvipanda) 3NEW [06:44:21] 6Labs: Evaluate a 'cluster solution' for use on Tool Labs - https://phabricator.wikimedia.org/T106475#1469900 (10yuvipanda) 3NEW [06:53:42] 6Labs: Setup monitoring and reporting for disk space usage of each project on NFS - https://phabricator.wikimedia.org/T106476#1469909 (10yuvipanda) 3NEW [08:02:40] 6Labs, 10Tool-Labs, 3Labs-Sprint-101, 3Labs-Sprint-102, and 3 others: Puppetize toolserver.org redirect configuration - https://phabricator.wikimedia.org/T85165#1469934 (10yuvipanda) @Coren This doesn't seem to be applied anywhere? [08:11:39] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Q4-Sprint-2, 3Labs-Sprint-100, and 3 others: Disable LDAP and enable admin puppet module on labstore100[12] - https://phabricator.wikimedia.org/T95559#1469946 (10yuvipanda) [08:11:40] 6Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-4, 3ToolLabs-Goals-Q4: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1469948 (10yuvipanda) [08:11:54] 6Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-4, 3ToolLabs-Goals-Q4: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1001152 (10yuvipanda) Merged in T95559. I heard @faidon had found a clever solution for this. [08:13:30] 6Labs, 10Tool-Labs, 3Labs-Sprint-101, 3Labs-Sprint-102, and 3 others: Puppetize toolserver.org redirect configuration - https://phabricator.wikimedia.org/T85165#1469953 (10yuvipanda) And it should be a module as well. [08:14:18] 6Labs, 3Labs-Q4-Sprint-2, 3ToolLabs-Goals-Q4: Remove dependencies on LDAP from labstore100[12] - https://phabricator.wikimedia.org/T95558#1469954 (10yuvipanda) Second one is gone, first one going soon. [08:14:50] 6Labs, 10Tool-Labs: Add monitoring to toolserver.org redirects - https://phabricator.wikimedia.org/T94022#1469955 (10yuvipanda) [08:18:50] 6Labs, 10Beta-Cluster, 6operations, 7Monitoring: Setup (simple) catchpoint monitoring and metrics for enwiki betacluster just like production - https://phabricator.wikimedia.org/T97865#1469958 (10hashar) 5Open>3declined a:3hashar From a reply I made to ops-l: > I thought Catchpoint to be super cheap... [08:43:19] 6Labs, 6operations: bond0 connection on labstore1001 is unpuppetized - https://phabricator.wikimedia.org/T92622#1469979 (10yuvipanda) No bonds on that anymore, afaik - the system was re-installed. Should we mark this as invalid? The bond isn't really needed now either, is it? [08:44:42] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103, 3Labs-Sprint-104: Labs: Make a new backup of the Labs storage to codfw - https://phabricator.wikimedia.org/T103356#1469983 (10yuvipanda) 5Open>3Resolved a:3yuvipanda (said copy is done) [08:47:20] 6Labs: Ensure that labstore machine is 'known good' hardware - https://phabricator.wikimedia.org/T106479#1469992 (10yuvipanda) 3NEW [08:47:59] 6Labs: Ensure that labstore machine is 'known good' hardware - https://phabricator.wikimedia.org/T106479#1469992 (10yuvipanda) [08:49:31] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad: labstore1002 issues while trying to reboot - https://phabricator.wikimedia.org/T98183#1470001 (10yuvipanda) Any updates on this? It's currently the main labstore server - is it considered reliable now? Did we swap out any hardware? [08:49:45] 6Labs: Ensure that labstore machine is 'known good' hardware - https://phabricator.wikimedia.org/T106479#1470003 (10yuvipanda) [08:50:06] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad, 3Labs-Sprint-102: Locate and assign some MD1200 shelves for proper testing of labstore1002 - https://phabricator.wikimedia.org/T101741#1470006 (10yuvipanda) Did this happen? @Coren updates? [08:52:14] 6Labs: Identify services labs provides - https://phabricator.wikimedia.org/T105721#1470007 (10yuvipanda) [08:53:19] 6Labs, 3ToolLabs-Goals-Q4: Fix documentation & puppetization for labs NFS - https://phabricator.wikimedia.org/T88723#1470008 (10yuvipanda) @Coren updates? [09:01:44] 6Labs, 6operations: Investigate heavy NFS users and see if they can move IO to local storage - https://phabricator.wikimedia.org/T96065#1470013 (10yuvipanda) [09:01:46] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1470015 (10yuvipanda) [09:02:08] 6Labs: Map Warper - No space left on device - https://phabricator.wikimedia.org/T102414#1470017 (10yuvipanda) Was this fixed? [09:02:27] 6Labs, 6operations, 3Labs-Sprint-101: Make Labs NFS alerts paging - https://phabricator.wikimedia.org/T101650#1470018 (10yuvipanda) Can this be marked resolved? [09:03:09] 6Labs, 5Patch-For-Review: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#1470023 (10yuvipanda) Updates on this? I feel like this is a bit obsolete now and there are other tasks that this should be merged into. [09:03:26] 6Labs: Return NFS to its standard ports - https://phabricator.wikimedia.org/T103052#1470025 (10yuvipanda) @Coren Thoughts? [09:04:59] 6Labs: start-nfs script warning message is scary and wrong - https://phabricator.wikimedia.org/T101742#1470030 (10yuvipanda) [09:05:54] 6Labs, 3ToolLabs-Goals-Q4: Allow labstores to hot or warm swap in case of failure - https://phabricator.wikimedia.org/T93589#1470031 (10yuvipanda) @mark @coren are we still going to do this? Afaik attempts at this might have been responsible for parts of the big NFS outage, and we decided to not have both mach... [09:06:32] 6Labs, 10Labs-Infrastructure: Some instances don't get automatic nfs exports for a long time - https://phabricator.wikimedia.org/T105024#1470032 (10yuvipanda) [09:06:34] 6Labs, 10Labs-Infrastructure: nfs-exports-daemon hangs, prevents new instances from accessing nfs - https://phabricator.wikimedia.org/T106076#1470034 (10yuvipanda) [09:06:47] 6Labs, 6operations: Recover home folders and /data/project from wikimetrics1 - https://phabricator.wikimedia.org/T103530#1470036 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [09:07:14] 6Labs, 6operations, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: labstore has multiple unpuppetized files/scripts/configs - https://phabricator.wikimedia.org/T102478#1470040 (10yuvipanda) [09:07:15] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1470039 (10yuvipanda) [09:07:32] 6Labs, 10Labs-Infrastructure: Weird state of /data/project for dumps (semi-missing files) - https://phabricator.wikimedia.org/T87224#1470042 (10yuvipanda) Is this still happening? [09:08:18] 6Labs, 10Labs-Infrastructure: "Stale file handle" for /public/dumps/ - https://phabricator.wikimedia.org/T87013#1470044 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Should be gone now, I hope - because we no longer mount that by default, and the machine hosting that mount has also gotten better over time.... [09:10:25] 6Labs: Labs homedirs owned by root for new projects - https://phabricator.wikimedia.org/T100478#1470048 (10yuvipanda) p:5High>3Normal Not sure, but new projects should not have NFS anymore and old projects should get NFS taken away over time :) [10:07:33] PROBLEM - Puppet failure on tools-static-01 is CRITICAL 100.00% of data above the critical threshold [0.0] [12:27:31] 6Labs, 10Labs-Infrastructure: Weird state of /data/project for dumps (semi-missing files) - https://phabricator.wikimedia.org/T87224#1470546 (10Nemo_bis) Yes, though the server was restarted in the meanwhile. Deleting the directory is not a problem, if that's the right thing to do. I don't know how to check wh... [13:04:39] 6Labs, 10Tool-Labs, 7Database: Possible Tool Labs database corruption in enwiki_p - https://phabricator.wikimedia.org/T106470#1470643 (10jcrespo) [13:04:42] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1470642 (10jcrespo) [13:06:03] 6Labs, 10Tool-Labs, 7Database: Possible Tool Labs database missing rows in enwiki_p - https://phabricator.wikimedia.org/T106470#1470650 (10jcrespo) [13:16:46] 6Labs, 10Tool-Labs, 7Database: Possible Tool Labs database missing rows in enwiki_p - https://phabricator.wikimedia.org/T106470#1470677 (10jcrespo) Updating description, as "database corruption" would make the problem worse (data impossible to access). "Data integrity" issues are probably the way to better d... [13:40:43] 6Labs, 10Tool-Labs, 7Database: Tool Labs enwiki_p replicated database missing rows - https://phabricator.wikimedia.org/T106470#1470705 (10MZMcBride) [13:44:21] 6Labs, 10Tool-Labs, 7Database: Tool Labs enwiki_p replicated database missing rows - https://phabricator.wikimedia.org/T106470#1470715 (10MZMcBride) >>! In T106470#1470677, @jcrespo wrote: > Updating description, as "database corruption" would make the problem worse (data impossible to access). "Data integri... [14:03:24] 6Labs, 10Tool-Labs, 7Database: Tool Labs enwiki_p replicated database missing rows - https://phabricator.wikimedia.org/T106470#1470739 (10jcrespo) > How do we fix the labsdb* hosts? Is a re-import required? I am doing an informal check to see how badly it is, but only counting the rows takes 20 minutes (it... [15:37:51] 6Labs, 10Tool-Labs: Make tools-mail route mail for @tools-*.pmtpa.wmflabs correctly - https://phabricator.wikimedia.org/T63484#1470913 (10scfc) Recent example courtesy of T106462: ``` Received: from tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs ([10.68.18.45] ident=Debian-exim) by mail.tools.wmflabs.org wi... [15:38:58] (03PS1) 10Sitic: Fix i18n message interpolation for tags [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226309 [15:40:52] (03CR) 10Sitic: [C: 032 V: 032] Fix i18n message interpolation for tags [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226309 (owner: 10Sitic) [15:43:07] 6Labs, 10Tool-Labs: Make tools-mail route mail for @tools-*.pmtpa.wmflabs correctly - https://phabricator.wikimedia.org/T63484#1470927 (10scfc) And `/etc/mailname` on `tools-bastion-01` is (correctly): ``` scfc@tools-bastion-01:~$ cat /etc/mailname tools-bastion-01.tools.eqiad.wmflabs scfc@tools-bastion-01:~... [15:55:34] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1470967 (10bd808) Here's an example of the rate limit being exhausted: ``` $ date; curl -sD - https://api.github.com/rate_limit | grep '^X-RateLim... [16:36:57] 10MediaWiki-extensions-OpenStackManager, 10Librarization, 10MediaWiki-extensions-Translate: Bring in spyc for OpenStackManager and Translate via composer - https://phabricator.wikimedia.org/T75945#1471245 (10Nikerabbit) Just for FYI phpyaml is now the recommended library for Translate. [16:47:15] 6Labs, 10Labs-Infrastructure, 7Composer, 7Upstream: Composer activity from Labs hosts can be rate limited by GitHub - https://phabricator.wikimedia.org/T106452#1471326 (10dr0ptp4kt) >>! In T106452#1469573, @Jdlrobson wrote: > @dr0ptp4kt this would be a great use of twitter :) @Jdlrobson, got some proposed... [17:09:44] 6Labs, 6operations: virbr0 interface present in some virt hosts - https://phabricator.wikimedia.org/T83732#1471530 (10fgiunchedi) this is still happening on `labvirt1007` ``` root@palladium:~# salt -b 10 'virt*' cmd.run 'ip a l | grep virbr' Executing run on ['virt1006.eqiad.wmnet', 'virt1002.eqiad.wmnet', '... [17:09:47] 6Labs, 6operations: virbr0 interface present in some virt hosts - https://phabricator.wikimedia.org/T83732#1471533 (10fgiunchedi) [17:50:51] 6Labs: Labs homedirs owned by root for new projects - https://phabricator.wikimedia.org/T100478#1471726 (10Gage) 5Open>3Resolved a:3Gage Yeah I haven't seen recurrence of this so I'm closing the ticket. Thanks. [18:07:29] anyone able to help me understand why mediawiki has not installed and being shown on http://mfrontend-browser-tests.wmflabs.org/ ? [18:07:37] vagrant provision seems to be throwing lots of errors [18:44:21] 6Labs, 10Labs-Infrastructure: nfs-exports-daemon hangs, prevents new instances from accessing nfs - https://phabricator.wikimedia.org/T106076#1471942 (10yuvipanda) So what we should do is: # set a timeout for the requests # Make systemd restart this a fixed number of times # Write an icinga check for this run... [19:39:03] 6Labs, 10Labs-Infrastructure, 6operations: rename holmium to labdns1002 - https://phabricator.wikimedia.org/T106303#1472227 (10RobH) [19:39:04] 6Labs, 10Labs-Infrastructure: holmium is a spof - https://phabricator.wikimedia.org/T106142#1472228 (10RobH) [19:39:06] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1472224 (10RobH) 5Open>3Resolved a:3RobH Allocating wmf4575 as labdns1001; will create the tasks for installation and link them. [19:41:11] 6Labs, 10Labs-Infrastructure, 6operations: install/setup labdns1001 - https://phabricator.wikimedia.org/T106584#1472236 (10RobH) 3NEW a:3RobH [19:45:06] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1472283 (10mark) Is that all that box will do, backup dns for Labs? [19:46:34] 6Labs, 10Labs-Infrastructure, 6operations: rename holmium to labdns1002 - https://phabricator.wikimedia.org/T106303#1472291 (10RobH) [19:46:34] 6Labs, 10Labs-Infrastructure: holmium is a spof - https://phabricator.wikimedia.org/T106142#1472292 (10RobH) [19:46:37] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1472289 (10RobH) 5Resolved>3Open I misread this allocation. I thought it was a replacement, not a backup. This needs more discussion, as Mark's question demonstrates.... [19:48:25] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1472298 (10RobH) Also, this sits in a public vlan, unlike other labs boxes. Could a simple labs dns system exist in a ganeti vm or does mixing those two stacks seem horrible? [19:50:00] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-107, 5Patch-For-Review: nfs-exports-daemon hangs, prevents new instances from accessing nfs - https://phabricator.wikimedia.org/T106076#1472306 (10yuvipanda) [19:50:46] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-107, 5Patch-For-Review: nfs-exports-daemon hangs, prevents new instances from accessing nfs - https://phabricator.wikimedia.org/T106076#1458401 (10yuvipanda) Verified that the timeout does kill the process, which then gets restarted by systemd. [19:58:14] 6Labs: Build proper monitoring for making sure that processes that need to run only once on one labstore only are running only once on one labstore only - https://phabricator.wikimedia.org/T106590#1472375 (10yuvipanda) 3NEW [19:58:24] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-107, 5Patch-For-Review: nfs-exports-daemon hangs, prevents new instances from accessing nfs - https://phabricator.wikimedia.org/T106076#1472382 (10yuvipanda) Actually, just letting this be for now - monitoring needs more thought, filed T106590 for that. [20:03:33] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1472428 (10RobH) [20:03:36] 6Labs, 10Labs-Infrastructure, 6operations: install/setup labdns1001 - https://phabricator.wikimedia.org/T106584#1472427 (10RobH) [20:04:05] 6Labs, 10Labs-Infrastructure, 6operations: install/setup labdns1001 - https://phabricator.wikimedia.org/T106584#1472236 (10RobH) [20:15:29] jdlrobson> anyone able to help me understand why mediawiki has not installed and being shown on http://mfrontend-browser-tests.wmflabs.org/ ? < bd808 YuviPanda ? [20:16:03] jdlrobson: are you sure you selected ubuntu trusty when creating the instance this time? [20:16:15] i'm not sure about anything.. i went with the defaults :-/ [20:16:17] "Apache2 Debian Default Page" -- no [20:16:20] yeah, no [20:16:41] so where do i set that? [20:16:44] https://wikitech.wikimedia.org/wiki/Help:Labs-vagrant [20:16:47] step 2 [20:16:55] ah image type [20:17:09] is there any reason jessie is the default? [20:17:19] (that a default is given at all) [20:17:39] it's the new default for prod and hence the new default for labs, and eventually hopefully labs-vagrant will work on jessie too. [20:17:57] I don't know how much proportion of labs usage is labs-vagrant tho [20:17:58] YuviPanda: patches welcome [20:18:06] k as an fyi some kind of help/boilerplate on that page would be useful e.g. what's suitable for my need [20:18:14] ok 3rd time lucky :) [20:18:22] jdlrobson: the labs vagrant document page is pretty detailed if you actually follow it :P [20:18:36] bd808: indeed, but lots of things to do before we can even start doing that (HHVM packages, etc) [20:19:07] YuviPanda: *nod* Last time I asked hhvm on jessie wasn't on anyone's radar [20:19:23] bd808: indeed, so I guess current status quo is what's going to be around for a while. [20:19:36] bd808: I wonder if we shoudl just do a 'bail early with a message' bit to labs-vagrant code [20:20:14] That might not be a bad idea. I have that in the replacement patch to use Vagrant for real [20:20:25] bd808: oh, right. we should merge that too [20:20:38] that would be nice :) [20:20:55] bd808: looking at it now [20:21:05] I need a root STAT [20:21:15] Like right now. [20:21:23] This is an emergency. [20:21:46] YuviPanda, ^^^ [20:21:57] go on Cyberpower678 [20:22:00] what is happening? [20:22:21] So I accidentally deleted APPER's wikihistory project. cnf file and everything. [20:22:33] Can you roll back the wikihistory project? [20:22:40] on toollabs? [20:22:59] YuviPanda, ^ [20:23:05] name of tool? [20:23:10] wikihistory [20:23:29] Cyberpower678: when did you delete it? when was the data from? [20:23:40] YuviPanda: what i mean is it's not linked to from the page where you actually do the action [20:23:46] I deleted it now. [20:23:53] Or like 1 minute ago. [20:24:16] I don't think there's been any updates to the code for several weeks now. [20:24:34] jdlrobson: indeed, unfortunately there's no ongoing development of the OpenStackManager extension, and it's being slowly killed over the next year. so the UI is going to suck for a while [20:24:38] YuviPanda, if you could reinstate the most recent backup, that would be great. [20:24:52] Cyberpower678: alright, let me poke around [20:25:31] YuviPanda, wikihistory is crying now. Bigbrother is complaining it can't start jobs now. [20:25:42] YuviPanda, thanks. :-) [20:26:16] 6Labs: Map Warper - No space left on device - https://phabricator.wikimedia.org/T102414#1472513 (10Artturimatias) Just tested and adding maps seems to work again. [20:26:41] Cyberpower678: I'll note that accidental user deletion recovery is not something we support officially - in this case it works, but in the general case, try not to delete things you will need. [20:27:22] I meant to delete a different directory. [20:27:58] I wanted to delete /data/project/xtools/public_html/wikihistory, and ended up wiping /data/project/wikihistory. :/ [20:28:39] okay third time round i can't even ssh in.. o_O [20:29:14] YuviPanda, How long will the process take? [20:30:10] ssh mf-browser-tests.eqiad.wmflabs > YuviPanda are you able to ssh in there? [20:32:37] jdlrobson: ssh to ssh mf-browser-tests.eqiad.wmflabs works for me [20:33:12] bd808: it's complaining about my key [20:33:17] ssh gather-browser-tests.eqiad.wmflabs works fine though [20:33:45] Cyberpower678: I've started the copy, I"ll let you know when it's done [20:34:10] bd808: oh wait i know why [20:34:14] i've used this name before [20:34:21] i'll delete the key and... [20:34:36] yup back in [20:34:39] yeah, not whining about your key but about the host key having changed right? [20:35:35] alias ssh-rmkey='ssh-keygen -R' [20:35:38] super handy [20:36:21] bd808: I was going to bikeshed on name of https://gerrit.wikimedia.org/r/#/c/193665/21 to be role::vagrant::lxc or something like that instead of so many underscores, but you can shush me and I'll merge and we can rename later if you want. [20:37:22] its not vagrant, it's mediawiki-vagrant [20:37:24] Cyberpower678: it has plenty of tiny files - going to take a while [20:37:36] Okay [20:37:37] we could drop the _lxc [20:37:46] bd808: sure! that'll work too. [20:37:52] role::mediawiki::vagrant? [20:38:06] shrug [20:38:18] do we have role::mediawiki? [20:38:32] feels like random namespacing [20:38:40] yeah, it's a bit random [20:38:45] role::labs::vagrant::mediawiki? [20:38:53] eg YuviPanda doesn't like underscore today [20:38:57] :P [20:39:05] It just feels like it should be :: [20:39:14] but as I said, you can tell me to shush - it's your project, etc [20:39:41] how about role::labs::mediawiki_vagrant ? [20:39:56] sure! [20:39:58] that works [20:40:09] I'll amend then [20:40:15] cool, I'll merge afte [20:40:16] r [20:42:47] Cyberpower678: plenty of .wha files. This is going to take a long long time, I think. [20:43:02] YuviPanda, urg. [20:43:15] Can you give me an estimate on how long? [20:43:21] And how many files? [20:46:52] Cyberpower678: no. I've stopped the scp and am going to do a tarred scp - should be faster. I'll let you know when the tar file is done - that'll give us estimates [20:48:10] tar cf - . | (ssh somewhere; cd foo; tar xvf -) [20:48:41] actually I don't think that works quite like that [20:49:15] there's a slightly more intense version of that yeah [20:49:19] with the useful pv tool int he middle [20:50:17] pv++ [20:50:23] (i like it :) [20:54:28] Cyberpower678: ok, so it's 28G [20:54:35] that's going to take... a while. [20:54:45] it's bigger than /tmp so I'll have to move things around a bit [20:54:46] :O [20:55:06] How the hell did I delete that so quickly? [20:55:53] YuviPanda: wow I hadn't noticed until just now how much you cleaned up the global groups for puppet. Nice job! [20:56:22] bd808: :D I think we should move the labs-vagrant stuff into something called mediawiki instead of 'roles' [20:56:32] Cyberpower678: well, a lot of the 'data' folder is still there [20:57:54] 6Labs, 6operations: lvm 'others20150715' snapshot full on labstore1001 - https://phabricator.wikimedia.org/T106601#1472660 (10yuvipanda) 3NEW [20:59:51] Cyberpower678: is it ok if I copy everything except the 'data/' directory? [20:59:55] YuviPanda: {{done}} [[WP:BOLD]] [21:00:02] bd808: sweet [21:02:17] YuviPanda: Did you put role::mediawiki-install::labs into project specific groups? I see 25 hosts that have it enabled still [21:02:59] bd808: I removed it - so it's enabled on all the hosts currently have it enabled, but can't be enabled for new hosts. that also means it can't be disabled from old hosts either without admin action [21:03:06] which... I need to figure something about. [21:03:18] the puppet code for that has been broken for months [21:03:26] so I'm tempted to just remove it from the instances and notify owners [21:03:37] or the other way around [21:03:48] well it's easy to add roles for a specific project but probably not too many people know that [21:03:55] YuviPanda, should be [21:04:06] like much of OSM it is dark magic [21:04:59] bd808: true [21:05:06] bd808: I'll write up a deprecation document maybe [21:08:09] Cyberpower678: it should be in wikihistory/recovered folder now [21:08:14] you can use the 'take' command to take ownership of that [21:09:22] didn't copy the data directory [21:09:27] I have to go take a break now [21:11:06] YuviPanda, How do I transfer it out of recovered [21:12:14] Cyberpower678: once you `take` it to change the ownership to the tool you should be able to just `mv` the files [21:40:26] bd808: thanks :) [21:40:36] bd808: also the lxc patch needs manual rebasing before merge :( [21:42:06] YuviPanda: k. I can do that [21:42:11] thanks [21:44:45] YuviPanda: {{done}} [21:45:20] bd808: \o/ merged [21:45:22] thank you :) [21:45:30] now to write docs :( [21:45:51] https://vagrant-lxc-demo.wmflabs.org/wiki/Main_Page [21:45:55] \o/ nice! [21:46:03] nice [21:46:07] I wonder if we can clusterify these [21:46:21] I guess the biggest problem will be 'how do we get people shell' and 'how/where does this persist' [21:46:53] oh you mean run the lxc on a grid of some kind? [21:48:19] YuviPanda: if you're still in a puppet reviewing mood https://gerrit.wikimedia.org/r/#/c/158016/ [21:48:20] :D [21:48:28] bd808: ya [21:48:46] marxarelli: not going to be touch varnish unfortunately :( sorry! [21:48:56] doh. [21:49:03] both alex and bblack are on vacation atm [21:49:34] yeah, i think that's why it hasn't gotten any love [21:49:49] it can wait, as long as the varnish config doesn't drift too much [22:01:42] marxarelli: yeah :) [22:02:05] Cyberpower678: were you able to recover the stuff you needed properly? have you informed apper / other tools members too, so they aren't confused? [22:02:33] YuviPanda, I don't know how to use the mv command. [22:02:38] Can you help me? [22:07:27] Cyberpower678: it's a fairly common Linux command, I'd suggest googling for answers? [22:07:37] Cyberpower678: http://lmgtfy.com/?q=mv&l=1 [22:07:44] YuviPanda, I've never used it. [22:07:57] Google? [22:21:57] Cyberpower678: did you figure it out? do you want me to do anything else or are you ok? [22:22:34] YuviPanda, I see how to move individual files, but not the directory as a whole. [22:22:42] exactly the same way [22:22:44] mv file file [22:22:49] mv file file2 [22:22:52] mv dir dir2 [22:23:26] Cyberpower678: You should download https://github.com/gleitz/howdoi [22:23:29] Cyberpower678: ^ what Reedy said. what exactly were you doing? [22:23:50] SigmaWP: howdoi install howdoi [22:23:55] :D [22:24:05] mv /data/project/wikihistory/recovered /data/project/wikihistory [22:24:28] YuviPanda, ^ [22:24:33] that probably won't work [22:24:33] I keep getting an erroe [22:24:44] mv /data/project/wikihistory/recovered/* /data/project/wikihistory [22:24:51] though [22:24:55] you probably should cp [22:25:11] * Cyberpower678 is N00B when it comes to linux. [22:25:16] Please speak english. :p [22:25:32] don't move [22:25:33] copy [22:26:15] Reedy, I want to move [22:26:31] Good for you [22:26:34] You're struggling with that [22:26:36] Copy it [22:26:38] Make sure it's right [22:26:42] Then delete the source [22:31:51] YuviPanda: I'm not aware that I'm using that much space. I'll check [22:32:05] physikerwelt1: not you as in you personally but the math project [22:32:37] do you see the user? [22:32:52] physikerwelt1: most of it is /data/project/cbpd [22:33:42] ok I see that's pubmed... there is quite a cool project that converts the open source pubmed subset to mediawiki [22:34:14] however I think we only need a sample of the dataset [22:34:33] I'll check back with my coworker [22:35:42] physikerwelt1: it's not a big deal if you need it at all :) [22:35:53] physikerwelt1: feel free to use the space - just wanted to let you know in case it was unexpected or accidental [22:38:10] YuviPanda: oh sorry I accidentally cc'ed you to the email I wrote to my coworker which was in german [22:38:20] physikerwelt1: :D I noticed :D [22:38:21] 'tis ok [22:38:39] 6Labs, 7Mobile, 3Mobile-App-Sprint-62-Android-Summer-Breeze, 3Wikipedia-Android-App: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1473187 (10yuvipanda) Done! http://android-builds.wmflabs.org/ Can you verify this works and then we can close the ticket? [22:42:14] YuviPanda, Reedy got the move working. Everything's up again. :-) [22:42:21] Thanks for your help [22:42:35] Cyberpower678: yw. do let the other maintainers know [22:42:48] I left APPER an apology email. [22:43:07] has anyone heard of the labs used as an anonymizing proxy? [22:43:14] (for legit reason) [22:51:12] yurik: I think people used TS for that [22:51:24] Reedy, TS? [22:51:27] tool server? [22:51:30] toolserver [22:51:31] yeah [22:51:37] do you know what's the status of that? [22:51:54] apparently wikivoyage uses maps from MapQuest without a proxy [22:53:05] Reedy, ^ [22:53:23] I reported that long ago [22:53:49] Should be immediately disabled as clearly illegal but nobody wanted to act [22:53:55] https://meta.wikimedia.org/wiki/Wikivoyage/Lounge#Maps_loading_external_resources [22:59:37] Nemo_bis, thanks! didn't know someone else raised it as an issue. I spoke with James Heilman about maps at wikimania, he showed wikivoyage, i pressed the panic button, so now we are trying to figure out how to do it properly [23:05:57] Nemo_bis, seems like something was done there, according to the discussion. Yet when James showed it to me, the default loaded mapquest tiles (external) [23:06:59] 6Labs, 10Tool-Labs: toolsbeta-puppetmaster3 can't resolve hiera('labs_puppet_master') - https://phabricator.wikimedia.org/T106627#1473226 (10scfc) 3NEW [23:07:14] mapquest never really disappeared IIRC [23:07:48] from en.voy, I mean [23:08:36] Nemo_bis, it might be ok to keep it there (with the proper user warning), but it should never be shown by default [23:09:32] 6Labs, 10Tool-Labs: toolsbeta-puppetmaster3 can't resolve hiera('labs_puppet_master') - https://phabricator.wikimedia.org/T106627#1473238 (10scfc) a:3scfc Catch 22: `/etc/puppet/hiera.yaml` is set up by Puppet which cannot run which does not set up `/etc/puppet/hiera.yaml`. So I'll copy it manually and see... [23:13:04] 6Labs, 10Tool-Labs: toolsbeta-puppetmaster3 can't resolve hiera('labs_puppet_master') - https://phabricator.wikimedia.org/T106627#1473245 (10scfc) 5Open>3Resolved Copied `modules/puppetmaster/files/production.hiera.yaml` to `/etc/puppet/hiera.yaml`, stopped and started `puppetmaster` service, now `puppet a... [23:21:39] Nemo_bis, https://en.wikivoyage.org/wiki/Wikivoyage:Travellers%27_pub#Dynamic_maps [23:23:25] ok [23:36:34] 6Labs, 10Tool-Labs: toolsbeta-puppetmaster3 can't resolve hiera('labs_puppet_master') - https://phabricator.wikimedia.org/T106627#1473411 (10scfc) 5Resolved>3Open a:5scfc>3None