[04:57:41] 6Labs, 7Tracking: Create labs project for Reading department - https://phabricator.wikimedia.org/T101325#1344685 (10bd808) >>! In T101325#1344381, @yuvipanda wrote: > I think projects should be more specific and not be team based - that's something we've been trying to move away from in the past so we don't en... [06:30:11] 6Labs, 7database: Rebuild s6 and s7 on labsdb1002 - https://phabricator.wikimedia.org/T101567#1344840 (10Springle) fyi, https://lists.wikimedia.org/pipermail/labs-l/2015-June/003760.html If the box is taken down, keep the list in the loop. [07:00:36] (03CR) 10John Vandenberg: [C: 04-1] "Not needed anymore; we followed the advice given and added support for ipaddr" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209978 (https://phabricator.wikimedia.org/T86015) (owner: 10Merlijn van Deen) [07:01:23] (03Abandoned) 10Merlijn van Deen: Add python-ipaddress package [labs/toollabs] - 10https://gerrit.wikimedia.org/r/209978 (https://phabricator.wikimedia.org/T86015) (owner: 10Merlijn van Deen) [07:01:29] 10Tool-Labs, 10Pywikibot-compat-to-core, 10pywikibot-core, 5Patch-For-Review: Install all pywikibot python optional dependencies on tool labs - https://phabricator.wikimedia.org/T86015#1344847 (10jayvdb) [07:01:32] 10Tool-Labs, 10pywikibot-core, 5Patch-For-Review: Support Debian package python-ipaddr - https://phabricator.wikimedia.org/T100603#1344846 (10jayvdb) 5Open>3Resolved [07:02:21] 10Tool-Labs, 10Pywikibot-compat-to-core, 10pywikibot-core, 5Patch-For-Review: Install all pywikibot python optional dependencies on tool labs - https://phabricator.wikimedia.org/T86015#959861 (10jayvdb) >>! In T86015#1051029, @jayvdb wrote: > Now that T76286 is merged, another dependency for py2.6 and 2.7... [07:18:58] 10Tool-Labs, 10Pywikibot-compat-to-core, 10pywikibot-core, 5Patch-For-Review: Install all pywikibot python optional dependencies on tool labs - https://phabricator.wikimedia.org/T86015#1344855 (10jayvdb) [07:19:28] 10Tool-Labs, 10Pywikibot-compat-to-core, 10pywikibot-core, 5Patch-For-Review: Install all pywikibot python optional dependencies on tool labs - https://phabricator.wikimedia.org/T86015#959861 (10jayvdb) [07:45:56] 6Labs, 10Tool-Labs: Document labsdb replication set up - https://phabricator.wikimedia.org/T85868#1344861 (10jcrespo) I started doing this here: https://wikitech.wikimedia.org/wiki/MariaDB/Sanitarium_and_Labsdbs [08:10:38] 6Labs, 10Datasets-General-or-Unknown, 10Labs-Infrastructure, 10Wikidata, and 2 others: Add Wikidata json dumps to labs in /public/dumps - https://phabricator.wikimedia.org/T100885#1344896 (10ArielGlenn) ...waiting for folks to get legacy/symlink/etc stuff sorted out on the changeset and I'll be happy to me... [09:14:26] 10Tool-Labs: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#1345118 (10yuvipanda) [09:14:48] 6Labs, 10Tool-Labs, 7Documentation: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#1344508 (10yuvipanda) [09:15:30] 10Quarry: SQL String functions (like UCASE, UPPER etc) not working - https://phabricator.wikimedia.org/T100057#1345122 (10Aklapper) [09:15:34] 6Labs, 7Tracking: Create labs project for Reading department - https://phabricator.wikimedia.org/T101325#1345124 (10yuvipanda) @bd808 I think the solution is to reduce the time it takes to spin a new one up, and we've been better about it now than with the previous SMW (praise be its name!) process (which aver... [09:21:38] 10Tool-Labs, 3Labs-Sprint-101: Get rid of tools-trusty bastion - https://phabricator.wikimedia.org/T101094#1345131 (10yuvipanda) @legoktm yes, they were! T96472 [09:26:15] 6Labs, 10Tool-Labs, 7Documentation: Explicitly document policies for requesting new projects - https://phabricator.wikimedia.org/T101687#1345133 (10yuvipanda) 3NEW [09:27:40] 6Labs, 6operations: Expand list of people who can create new Labs project - https://phabricator.wikimedia.org/T101688#1345141 (10yuvipanda) 3NEW [09:28:33] 6Labs, 7Tracking: Create labs project for Reading department - https://phabricator.wikimedia.org/T101325#1345154 (10yuvipanda) Created T101688 and T101687 to help make the process smoother! [09:32:00] !log etcd ran sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on etcd01 for _joe_ [09:32:03] Logged the message, Master [09:43:31] 6Labs, 10Maps, 6Scrum-of-Scrums, 7Blocked-on-Operations: Upgrade postgres on labsdb1004 / 1005 to 9.4, and PostGis 2.1 - https://phabricator.wikimedia.org/T101233#1345174 (10yuvipanda) This also requires labsdb1005 to be upgraded since that's the postgres slave for this instance, but it's also the master f... [09:55:06] 6Labs, 10Maps, 6Scrum-of-Scrums, 7Blocked-on-Operations: Upgrade postgres on labsdb1004 / 1005 to 9.4, and PostGis 2.1 - https://phabricator.wikimedia.org/T101233#1345183 (10akosiaris) Trusty, which is the easy upgrade path, has postgis 2.1.2 and postgres 9.3. @maxsem, @yurik, are those sufficient ? Otherw... [10:18:45] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL 40.00% of data above the critical threshold [0.0] [10:20:53] 6Labs: LDAP failures in etcd01 - https://phabricator.wikimedia.org/T101689#1345197 (10yuvipanda) 3NEW [10:40:55] 6Labs, 10Tool-Labs, 7Documentation: Cleanup https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools - https://phabricator.wikimedia.org/T101690#1345219 (10yuvipanda) 3NEW [11:07:05] 6Labs, 6operations, 7network: permit syslog from labs to lithium - https://phabricator.wikimedia.org/T90695#1345250 (10akosiaris) 5Open>3Resolved a:3akosiaris Rules have been updated on cr{1,2} and now the packets flow through. Resolving [11:08:12] 6Labs, 6operations, 7network: permit syslog from labs subnet to lithium - https://phabricator.wikimedia.org/T90695#1345253 (10yuvipanda) [11:09:49] 6Labs, 6operations, 7network: permit syslog from labs hosts subnets to lithium - https://phabricator.wikimedia.org/T90695#1345256 (10akosiaris) [11:13:37] (03PS1) 10Sitic: Fix new page watchlist events [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/216633 [11:13:53] (03CR) 10Sitic: [C: 032 V: 032] Fix new page watchlist events [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/216633 (owner: 10Sitic) [11:17:24] (03PS1) 10Yuvipanda: Add Joe's key to labs root [labs/private] - 10https://gerrit.wikimedia.org/r/216634 [11:17:37] _joe_: ^ [11:18:14] (03CR) 10Giuseppe Lavagetto: [C: 031] Add Joe's key to labs root [labs/private] - 10https://gerrit.wikimedia.org/r/216634 (owner: 10Yuvipanda) [11:19:26] (03CR) 10Yuvipanda: [C: 032 V: 032] Add Joe's key to labs root [labs/private] - 10https://gerrit.wikimedia.org/r/216634 (owner: 10Yuvipanda) [11:26:11] 6Labs, 7database: Rebuild s6 and s7 on labsdb1002 - https://phabricator.wikimedia.org/T101567#1345296 (10jcrespo) a:3jcrespo [11:28:44] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK Less than 1.00% above the threshold [0.0] [12:02:09] 6Labs, 7database: Rebuild s6 and s7 on labsdb1002 - https://phabricator.wikimedia.org/T101567#1345329 (10jcrespo) This looks similar to https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1382333 but we are running the exact same kernel that supposedly patched this (checked on the changelog). I recommend a k... [12:41:38] 6Labs, 10Labs-Infrastructure, 6operations, 3Labs-Sprint-100: Rsync live labstore filesystem to local eqiad copy - https://phabricator.wikimedia.org/T101011#1345407 (10coren) [12:41:40] 6Labs, 10Labs-Infrastructure, 6operations, 3Labs-Sprint-100: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1345408 (10coren) [12:41:43] 6Labs, 10Labs-Infrastructure, 6operations, 3Labs-Sprint-100: Make a block-level copy of the codfw mirror of labstore1001 to eqiad - https://phabricator.wikimedia.org/T101010#1345405 (10coren) 5Open>3Resolved The copy is complete, and is mounted at the destination. A caveat worth nothing: since the sou... [12:44:31] 6Labs, 10Labs-Infrastructure, 6operations, 3Labs-Sprint-100, 3Labs-Sprint-101: Rsync live labstore filesystem to local eqiad copy - https://phabricator.wikimedia.org/T101011#1345425 (10coren) [12:44:53] 6Labs, 10Labs-Infrastructure, 6operations, 3Labs-Sprint-100, 3Labs-Sprint-101: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1345427 (10coren) [12:45:16] 6Labs, 10Tool-Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, and 3 others: Make sure tools-db is replicated somewhere - https://phabricator.wikimedia.org/T88718#1345428 (10coren) [13:10:28] 6Labs, 6operations: Make Labs NFS alerts paging - https://phabricator.wikimedia.org/T101650#1345486 (10yuvipanda) p:5Triage>3High [13:10:41] 6Labs, 6operations, 3Labs-Sprint-101: Make Labs NFS alerts paging - https://phabricator.wikimedia.org/T101650#1344262 (10yuvipanda) [13:14:52] Coren, YuviPanda, or anyone else with sysop on wikitech: It seems spammers have decided repeatedly posting spam on my user page on wikitech (https://wikitech.wikimedia.org/wiki/User:Anomie) would be a fun thing to do. Please delete the current content, copy https://meta.wikimedia.org/wiki/User:Anomie (no GlobalUserPage extension, I see), and protect. Thanks. [13:17:16] anomie: done, only autoconfirmed protection though. [13:17:24] anomie: I wonder if we can extend globaluserpage to wikitech [13:17:38] I think it might need SUL? [13:17:54] ah, right [13:17:55] it might [13:17:58] it probably does [13:18:07] we should do SULF2 :P [13:21:27] YuviPanda: Thanks, BTW. Although I'd have deleted the spam revision entirely (: [13:22:05] anomie: ah, oh well. [13:22:07] I blocked them too [13:31:44] 6Labs, 10Tool-Labs, 3Labs-Sprint-101, 3ToolLabs-Goals-Q4, 7Tracking: Move tools-shadow away from labvirt1004 - https://phabricator.wikimedia.org/T101636#1345508 (10yuvipanda) [13:32:23] 10Tool-Labs, 7Mail: kolossos@toolserver.org bouncing - https://phabricator.wikimedia.org/T101656#1345512 (10coren) 5Open>3Resolved a:3coren The exim4 mailserver (on relic.toolserver-legacy) died, probably because of the NFS outage. Restarting it fixed it. [13:32:43] 10Tool-Labs, 7Mail: kolossos@toolserver.org bouncing - https://phabricator.wikimedia.org/T101656#1345515 (10yuvipanda) This should be properly puppetized. [13:35:18] YuviPanda, Coren, I’m removing all use_dnsmasq refs from ldap. The next hour or so may be interesting. [13:36:49] Hm, wait, actually it’s probably smarted to disable that code in puppet so that it’s possible to back things out. [13:37:10] 10Tool-Labs, 3Labs-Sprint-101, 3ToolLabs-Goals-Q4: Puppetize toolserver.org redirect configuration - https://phabricator.wikimedia.org/T85165#1345522 (10yuvipanda) [13:37:24] andrewbogott: +1 [13:37:37] andrewbogott: also, self hosted puppetmaster. [13:37:38] gooood morning [13:38:13] andrewbogott: have you disabled the ability to upload images in OpenStack ? [13:38:18] seems it disappeared from horizon [13:38:23] hashar: yes [13:38:35] hashar: it’s been disabled for a long time though [13:38:53] I may be able to selectively enable it for you, open a phab task? [13:38:59] sure thing [13:40:40] https://phabricator.wikimedia.org/T101701 \O/ [13:40:42] [13intuition] 15amire80 opened pull request #48: Fix the description (06master...06patch-1) 02https://github.com/Krinkle/intuition/pull/48 [13:40:56] I have been out of nodepool for a while: / [13:48:33] hashar: we don’t have a very smart storage system for images, they’re just packed into a drive on the labs controller. So I’m afraid that if I open up that feature widely we’ll just fill up our drive immediately. The same goes for snapshots. [13:48:35] hence disabled [13:48:36] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL - Socket timeout after 10 seconds [13:48:36] ahhh [13:48:37] so I gotta be extremely careful [13:48:37] will double checks how many images nodepool send [13:48:38] iirc it is only a current image + a backup one [13:48:38] YuviPanda: https://gerrit.wikimedia.org/r/#/c/216673/1 pls [13:48:40] hashar: If it cleans up old ones that’d help a lot [13:48:41] yeah looking for a citation :-} [13:48:42] found one :-} [13:48:43] andrewbogott: ugh, NFS outage. [13:48:44] dammit [13:48:45] ssh is higly slow and defacto unusable for me [13:48:46] XuviPanda ping^^ [13:48:46] Steinsplitter: yup, we're aware and looking [13:48:52] ok :) [13:49:38] Steinsplitter: should be back up [14:00:13] Steinsplitter: well, spoke too soon. still in progress [14:00:13] i noticed :-D [14:00:16] [13intuition] 15Krinkle pushed 2 new commits to 06master: 02https://github.com/Krinkle/intuition/compare/07e75268a7d6...4bff4b2fc798 [14:00:17] 13intuition/06master 14701fec1 15Amir E. Aharoni: Fix the description... [14:00:17] 13intuition/06master 144bff4b2 15Timo Tijhof: Merge pull request #48 from amire80/patch-1... [14:00:17] hmm, it isn't responding for me and the SULWatcherbot went awol [14:00:17] stewardbot is still there [14:00:18] there are som issues atm [14:00:18] sDrewth: yeah, NFS outage in progress, should be back soon [14:00:18] okay, would have thought it would have been all tools, or no tools [14:00:18] sDrewth: it's 'tools that depend more on NFS' [14:00:19] sDrewith: maybe setting up a cron to automatically restart sulwatcher (or using bigbrother) [14:00:19] if it's a continuous job, gridengine should bring it back up automatically [14:00:19] Steinsplitter: wouldn't help with NFS outages [14:00:20] i know, :P but then after outage atomatically restart [14:00:20] Steinsplitter: yeah, gridengine's continuous flag should do that [14:00:21] ah, interestin [14:00:21] assiging a level of cluefulness that doesn't exist is not helpful ;-) [14:06:09] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL 33.33% of data above the critical threshold [0.0] [14:06:21] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 792581 bytes in 4.859 second response time [14:06:33] wheee [14:06:38] Steinsplitter: sDrewth things should be back up [14:10:34] they are indeed. thx [14:10:34] 6Labs, 3Labs-Q4-Sprint-3, 3Labs-Sprint-101: Labs: puppetize stripe_cache_size tweaks on labstores - https://phabricator.wikimedia.org/T96045#1345599 (10coren) 5Resolved>3Open Turns out puppet doesn't reliably run the script; this needs to be changed to cron. [14:10:34] 6Labs, 3Labs-Q4-Sprint-3, 3Labs-Sprint-101: Labs: puppetize stripe_cache_size tweaks on labstores - https://phabricator.wikimedia.org/T96045#1345603 (10coren)