[00:01:02] 6Labs, 10Tool-Labs, 7Epic: Convert all Labs tools to use cdnjs for static resources - https://phabricator.wikimedia.org/T103934#1403214 (10Ricordisamoa) [00:08:31] 6Labs, 10Tool-Labs, 7Epic: Convert all Labs tools to use cdnjs for static resources - https://phabricator.wikimedia.org/T103934#1403225 (10Ricordisamoa) [00:42:56] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403285 (10scfc) 3NEW [00:43:59] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403297 (10Krenair) I was running into this issue earlier... Can you delete your wikitech cookies and try again? [00:47:52] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403299 (10Krenair) (I should note that I was having this issue for several hours *before* syncing https://gerrit.wikimedia.org/r/#/c/220847/ - so I almost didn't sync it, but then found clearing my cookies re... [01:12:51] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403330 (10scfc) I have deleted the cookies for `wikitech.wikimedia.org` and was then able to log in again. [01:13:20] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403331 (10scfc) 5Open>3Resolved a:3Krenair [01:13:54] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403335 (10Krenair) Interesting... When I was speaking to @matanya earlier it became apparent that the issue did not affect everyone. But clearly it affects more than just me. I wonder if we can figure out how... [01:14:08] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403337 (10Krenair) 5Resolved>3Open p:5Unbreak!>3High a:5Krenair>3None [01:14:33] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1403285 (10Krenair) I'm going to leave this open at High priority until labs ops have had a chance to take a look. [01:42:05] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Vagrant, 10MediaWiki-Vagrant, 10wikitech.wikimedia.org: Update Vagrant role for Extension:OpenStackManager - https://phabricator.wikimedia.org/T103874#1403364 (10scfc) [01:44:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Merlijn van Deen was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=168009 edit summary: [02:22:18] 6Labs, 10Tool-Labs, 3Labs-Sprint-101, 3Labs-Sprint-102, and 3 others: Puppetize toolserver.org redirect configuration - https://phabricator.wikimedia.org/T85165#1403435 (10coren) This is done, for the web redirects; email config needs a bit of architecture to do right. [02:30:26] 6Labs, 7Database, 5Patch-For-Review: wikimania2016wiki not recognised by sql command - https://phabricator.wikimedia.org/T96638#1403449 (10Krenair) [03:02:41] 6Labs, 6operations, 7Database: Santitize recent wikis: wikimania 2016 and cn.wikimedia.org at labs dbs - https://phabricator.wikimedia.org/T100441#1403521 (10Krenair) 5Open>3Resolved Looks like this was done [03:02:52] 6Labs, 7Database, 5Patch-For-Review: wikimania2016wiki not recognised by sql command - https://phabricator.wikimedia.org/T96638#1403524 (10Krenair) 5Open>3Resolved [03:09:59] 6Labs, 10Labs-Infrastructure: add domain alias gomwiki.labsdb and lrcwiki.labsdb for s3.labsdb - https://phabricator.wikimedia.org/T103794#1403550 (10Krenair) [03:10:01] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1403548 (10Krenair) 5Open>3Resolved a:3Krenair [03:10:37] 10Tool-Labs-tools-Other, 7Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#1403554 (10Krenair) [03:10:40] 6Labs, 10Labs-Infrastructure: add domain alias gomwiki.labsdb and lrcwiki.labsdb for s3.labsdb - https://phabricator.wikimedia.org/T103794#1403551 (10Krenair) 5Open>3Resolved a:3Krenair Looks like that was done though, added hosts entries in https://gerrit.wikimedia.org/r/221045 [03:49:13] Could someone restart webservice tools.wmflabs.org/tmg? e.g. for https://tools.wmflabs.org/tmg/mimestat.php [06:20:05] 6Labs, 10Labs-Infrastructure: add domain alias gomwiki.labsdb and lrcwiki.labsdb for s3.labsdb - https://phabricator.wikimedia.org/T103794#1403695 (10jcrespo) [06:20:08] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1403693 (10jcrespo) 5Resolved>3Open Databases have not yet been sanitized. [06:20:27] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1403697 (10jcrespo) a:5Krenair>3jcrespo [06:24:36] 6Labs, 10wikitech.wikimedia.org, 7Documentation: Wikitech registration requires labs shell access - https://phabricator.wikimedia.org/T88092#1403702 (10Nemo_bis) [06:28:00] 6Labs, 10wikitech.wikimedia.org, 7Documentation: Wikitech registration requires labs shell access - https://phabricator.wikimedia.org/T88092#1403703 (10Nemo_bis) I don't see a good reason to hijack Tgr's report, which is about making Special:Userlogin smarter. The RT address is a separate issue, related to i... [07:34:01] 6Labs, 5Patch-For-Review: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1403753 (10MoritzMuehlenhoff) Frankly, I'm not comfortable with mosh in general (I wasn't aware it was used until now). SSH has been heavily scrutinised over the last 15 years, while mosh is mo... [07:52:22] 6Labs, 5Patch-For-Review: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1403871 (10valhallasw) The anecdocal evidence I can provide at the moment is 'a third of the people connected to tool labs are connected via mosh'. Currently, no-one is connected to bastion via... [08:03:32] 6Labs, 10Tool-Labs, 3Labs-Sprint-101, 3Labs-Sprint-102, and 3 others: Puppetize toolserver.org redirect configuration - https://phabricator.wikimedia.org/T85165#1403891 (10Ricordisamoa) >>! In T85165#1190669, @Ricordisamoa wrote: > I guess http://toolserver.org/~dartar/cite-o-meter/ should be redirected to... [08:12:08] 6Labs, 10Labs-Infrastructure: add domain alias gomwiki.labsdb and lrcwiki.labsdb for s3.labsdb - https://phabricator.wikimedia.org/T103794#1403901 (10jcrespo) [08:12:11] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1403899 (10jcrespo) 5Open>3Resolved Sanitization done. [08:30:34] 6Labs, 6operations, 3Labs-Sprint-102, 3Labs-Sprint-103, 5Patch-For-Review: Backport sshd with AuthorizedKeysCommand support to Ubuntu precise - https://phabricator.wikimedia.org/T102401#1403906 (10MoritzMuehlenhoff) 5Open>3Resolved The SSH backport has been installed across the fleet and all precise... [08:38:06] 10Tool-Labs-tools-Other, 7Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#1403925 (10jcrespo) [08:38:08] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1403926 (10jcrespo) [08:38:11] 6Labs, 10Labs-Infrastructure, 7Database: missing database entries at categorylinks table on dewiki db - https://phabricator.wikimedia.org/T72711#1403923 (10jcrespo) 5Open>3Invalid I've checked all the queries given by the user (note: one provides no results as that article does not exist) and I've found... [08:45:30] 6Labs, 10Wikimedia-Labs-General, 7Database: Discrepancy between enwiki_p.pagelinks on labs and production - https://phabricator.wikimedia.org/T73176#1403937 (10jcrespo) 5Open>3Resolved Test case cannot be reproduced anymore. [08:45:31] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1403939 (10jcrespo) [09:32:19] 6Labs, 6WMF-Legal: Toolserver relic files break attribution requirements for images - https://phabricator.wikimedia.org/T103965#1404001 (10Ricordisamoa) 3NEW [09:38:20] 6Labs, 6WMF-Legal: Provide an easy way for Tool Labs tools to expose their source code - https://phabricator.wikimedia.org/T102081#1404028 (10Ricordisamoa) I thought the easiest way for Tool Labs tools to expose their source code would have been a "Fork me on Gerrit" ribbon... [09:42:26] 6Labs, 10wikitech.wikimedia.org: Incorrectly attributed edits (from 2006) to me - https://phabricator.wikimedia.org/T59346#1404060 (10Nemo_bis) [09:43:40] 6Labs, 6WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#1404064 (10Ricordisamoa) What to do with OAuth keys? I'm afraid of adding co-maintainers to some of my tools because of the TOU... [09:43:43] 6Labs, 10Tool-Labs: Can't execute qstat on grid - https://phabricator.wikimedia.org/T103968#1404065 (10Steinsplitter) 3NEW [09:44:20] 6Labs, 10Tool-Labs: Can't execute qstat on grid - https://phabricator.wikimedia.org/T103968#1404073 (10Steinsplitter) [10:01:22] 6Labs, 6WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#1404104 (10valhallasw) Which part of the terms of use? If the OAuth TOU somehow effectively disallow multi-maintainer projects, that sounds like an issue with the OAuth TOU that needs to be... [10:06:24] 10Quarry: Show all published queries in profile - https://phabricator.wikimedia.org/T77948#1404110 (10Edgars2007) Temporary solution. In the "Recent queries" page add `?limit=5000` to page URL, so you (currently) get all queries. Then you can search for your username. Yes, it isn't a simple way, but at least it... [10:38:46] (03PS1) 10Ricordisamoa: Fix some HTML validation errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/221091 [10:42:09] !log abusefilter-global deleting per https://meta.wikimedia.org/wiki/User_talk:PiRSquared17#Abusefilter-global_project_still_active.3F [10:42:14] Logged the message, Master [10:54:16] (03CR) 10Ricordisamoa: ""3798 Errors, 6441 warning(s)"" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/221091 (owner: 10Ricordisamoa) [10:59:47] 6Labs, 10Tool-Labs, 7Epic: Convert all Labs tools to use cdnjs for static resources - https://phabricator.wikimedia.org/T103934#1404334 (10Ricordisamoa) [11:00:09] 6Labs, 10Tool-Labs, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1404335 (10Ricordisamoa) [11:06:07] (03PS1) 10Ricordisamoa: Load newer versions of jQuery and Bootstrap from cdnjs [labs/tools/translatemplate] - 10https://gerrit.wikimedia.org/r/221098 [11:08:15] (03CR) 10Ricordisamoa: [C: 032 V: 032] Load newer versions of jQuery and Bootstrap from cdnjs [labs/tools/translatemplate] - 10https://gerrit.wikimedia.org/r/221098 (owner: 10Ricordisamoa) [11:12:40] 6Labs, 10Tool-Labs, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1404392 (10Ricordisamoa) [11:21:36] 6Labs, 10Labs-Infrastructure, 7Database: Queries of commonswiki_p.filearchive for fa_sha1 are slow - https://phabricator.wikimedia.org/T71088#1404432 (10jcrespo) p:5Triage>3Normal I can confirm the issue: ``` MariaDB LABS localhost commonswiki_p > SELECT * FROM filearchive WHERE fa_sha1 = '0mpoldytyxspx... [11:22:55] 6Labs, 10Labs-Infrastructure, 7Database: rev_len should be available also for deleted revisions in database replicas - https://phabricator.wikimedia.org/T101631#1404439 (10jcrespo) p:5Triage>3Low [11:28:40] 6Labs: centralauth_p is missing tables - https://phabricator.wikimedia.org/T68533#1404455 (10jcrespo) This is either fixed or should be merged into T103011. [11:31:10] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1404472 (10Krenair) How were they available to labs users on labsdb1003 then? [11:34:48] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1404486 (10jcrespo) @Krenair Care to elaborate? Are you saying that they were available and they should not or the other way round? [11:47:32] Krenair: haha, I can't log into wikitech now either without clearing cookies... [11:52:03] 36 projects wit nfs! [11:58:04] YuviPanda, ugh. [11:58:06] YuviPanda, so it affected you as well? [11:58:08] Great.. I wonder what's wrong with it [11:58:37] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1404554 (10Krenair) Before we made the hosts change, I checked to see if I could connect to gomwiki_p and lrcwiki_p via `mysql --defaults-file=replica.my.cn... [11:59:04] Krenair: I restarted nutcracker to check. not it [12:00:19] I logged out and could log back in again [12:00:30] Are you still able to reproduce the issue? [12:01:42] Krenair: yes [12:01:48] Krenair: only in a browser with uncleaned cookies tho [12:01:56] Krenair: works fine on another browser [12:03:11] try now? [12:03:40] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1404563 (10yuvipanda) [12:03:42] 6Labs: Investigate and disable NFS in the ttmserver project - https://phabricator.wikimedia.org/T103840#1404561 (10yuvipanda) 5Open>3Resolved Done [12:03:44] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1404564 (10Aklapper) [12:04:19] Krenair: works now! [12:04:24] YuviPanda, interesting, ok [12:04:30] Maybe this is related to my change then [12:04:33] Krenair: what did you do? [12:04:58] Part of the wikitech config clearup yesterday was to remove the wgCookieDomain variable that I thought was unnecessary [12:05:16] production wikis don't set it [12:05:27] browsers should assume wikitech.wikimedia.org [12:05:30] But it doesn't make sense [12:05:50] I was having this issue in between the revert after the HHVM issue, and the re-application of the patch [12:06:23] YuviPanda, would you mind trying again, so I'm sure this isn't a fluke of some sort? [12:07:27] I almost didn't re-apply the patch because I wasn't going to be able to test it with login being seemingly broken for me (but not for all others) [12:11:05] Krenair: sure [12:11:43] Krenair: yup, not logging me in nmow [12:11:51] ok [12:12:00] I'll revert the cookiedomain change properly in a sec [12:12:10] Krenair: cool! [12:12:32] atm it's live-hacked into the config on silver [12:16:05] 6Labs, 7Database, 5Patch-For-Review: Add Wikipedia Northern Luri and Wikipedia Goan Konkani to labs replicas - https://phabricator.wikimedia.org/T102647#1404575 (10jcrespo) @krenair Yes, @coren may have added the wikis and perform the second-pass filtering/privilege control at labs side. As we do not trust... [12:17:47] 6Labs, 10wikitech.wikimedia.org, 7Documentation: Wikitech registration requires labs shell access - https://phabricator.wikimedia.org/T88092#1404579 (10Krenair) @Nemo_bis: I don't see a reason to ignore my comments. [12:18:05] 6Labs, 10wikitech.wikimedia.org, 7Documentation: Update wikitech customised shell account name registration instructions - https://phabricator.wikimedia.org/T88092#1404580 (10Krenair) [12:24:41] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1404595 (10Krenair) 5Open>3Resolved a:3Krenair I have no idea why but that appeared to fix it for Yuvi. Someone please reopen if you run into it aga... [12:35:32] YuviPanda, is the puppet patch for https://phabricator.wikimedia.org/T103595 not applied on shinken yet or something? [12:44:19] 10Tool-Labs-tools-Other, 10Phragile, 6TCB-Team: Deploy Phragile on tool-labs - https://phabricator.wikimedia.org/T100192#1404631 (10Tobi_WMDE_SW) 5Open>3Resolved [12:49:31] 6Labs, 3Labs-Sprint-101, 3Labs-Sprint-102, 5Patch-For-Review: Kill off virt1000 - https://phabricator.wikimedia.org/T102005#1404649 (10ArielGlenn) [12:49:33] 6Labs, 3Labs-Sprint-101, 3Labs-Sprint-102: Sort out remaining virt1000 salt minions - https://phabricator.wikimedia.org/T103010#1404647 (10ArielGlenn) 5Resolved>3Open It looks like some were missed somehow. I see these instances cephtest-1.cephtest.eqiad.wmflabs cephtest-3.cephtest.eqiad.wmflabs cephte... [13:23:43] 6Labs: Investigate and remove NFS mounts in the snuggle project - https://phabricator.wikimedia.org/T102680#1404723 (10yuvipanda) Ok, looks like I can just copy halfak/projects onto /srv and modify symlinks and everything should work. I'm not doing anything atm, however - just assessing how much work this will be. [13:24:21] (03PS1) 10Sitic: Last revision only as default [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/221114 [13:24:37] (03CR) 10Sitic: [C: 032 V: 032] Last revision only as default [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/221114 (owner: 10Sitic) [13:31:23] 6Labs, 6WMF-Legal: Provide an easy way for Tool Labs tools to expose their source code - https://phabricator.wikimedia.org/T102081#1404736 (10yuvipanda) So problems with Gerrit are: # No self-serve repo creation / forking # The UI is... somewhat new user hostile # The workflow is optimized for large projects,... [13:31:59] 6Labs, 6WMF-Legal: Provide an easy way for Tool Labs tools to expose their source code - https://phabricator.wikimedia.org/T102081#1404741 (10yuvipanda) Also, 4. Some people just do not care about version control - consider it too much pain for too little gain. They are wrong, of course (:P) but we should acc... [13:41:19] YuviPanda: we should give them an SVN server! [13:41:21] *runs* [13:41:27] lol [13:41:39] (we could actually do that with phab :-p) [13:42:01] I think YuviPanda is AFK or something [13:43:21] valhallasw: :P [13:43:27] valhallasw: I don't know if phab will have self serve repo creation [13:43:32] Krenair: I'm always AFK!!!!! [13:43:35] (and not afk) [13:45:36] YuviPanda, is the puppet patch for https://phabricator.wikimedia.org/T103595 not applied on shinken yet or something? [13:48:02] Krenair: oh, looking [13:50:54] ah, right [13:50:58] shinken shows this alert: [13:51:14] http://shinken.wmflabs.org/service/shinken-01/Puppet%20staleness [13:51:26] CRITICAL: 100.00% of data above the critical threshold [43200.0] [13:53:41] Krenair: haha [13:53:42] yeah [13:53:45] not sure why [13:54:27] you restarted it or fixed puppet? [13:54:54] Krenair: I just did a manual puppet run [13:55:15] not sure why puppet hasn't been running [14:19:54] 6Labs, 10wikitech.wikimedia.org: Replace SMW/SRF/SF with wikidata + lua - https://phabricator.wikimedia.org/T53642#1404810 (10yuvipanda) Awww, such an old ticket :) We should still get rid of it and replace it with horizon and other tools. [14:19:58] ImportError: No module named feedparser [14:20:10] i am wondering why there module missing on grid? [14:20:20] 6Labs, 10wikitech.wikimedia.org: Get rid of SMW/SRF/SF - https://phabricator.wikimedia.org/T53642#1404812 (10yuvipanda) p:5Low>3Normal [14:22:12] ping yuvipanda ? [14:22:32] hey Steinsplitter [14:23:01] Steinsplitter: I guess that module is not installed? [14:23:06] it is so frustrating to schoudle jobs on grid. it is always extra work :( [14:23:07] it is [14:23:12] when i execute in in shell works [14:23:32] well, it shouldn't be [14:23:35] someone must've installed it by hand [14:23:58] oh, interesting, [14:25:25] Steinsplitter: so it seems that calibre is installed and that brings in python-feedparser as an option but only on ubuntu trusty and not ubuntu precise [14:25:41] Steinsplitter: so your problems should go away if you pass '-l release=trusty' to your jsub command [14:25:54] valhallasw: ^ we need to find a solution to this at some point :( [14:26:05] thanks [14:26:11] tools-bastion* being trusty and default jsub going to precise. [14:27:44] works, thanks [14:28:15] Krenair: https://gerrit.wikimedia.org/r/#/c/220635/ ? I think we can remove some more semanticness with that one [14:28:40] will take a look later [14:28:46] 6Labs, 10wikitech.wikimedia.org: Get rid of SMW/SRF/SF - https://phabricator.wikimedia.org/T53642#1404851 (10yuvipanda) Places it is still being used for: # Request access to toollabs # Run SMW queries for finding out which instances have which roles and what not # Request access to shell rights (this is goin... [14:29:24] Krenair: thanks [14:29:38] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:29:42] PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:30:12] PROBLEM - Puppet failure on tools-redis-01 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:31:00] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:31:08] PROBLEM - Puppet failure on tools-checker-01 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:31:20] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:32:36] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:32:58] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:33:12] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL 44.44% of data above the critical threshold [0.0] [14:33:22] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:34:38] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:34:39] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:34:44] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:34:49] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:34:51] PROBLEM - Puppet failure on tools-exec-1205 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:34:57] PROBLEM - Puppet failure on tools-static-02 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:34:57] PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:34:59] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:35:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL 37.50% of data above the critical threshold [0.0] [14:35:18] PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:35:28] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 44.44% of data above the critical threshold [0.0] [14:35:31] PROBLEM - Puppet failure on tools-redis-02 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:35:31] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL 30.00% of data above the critical threshold [0.0] [14:35:35] PROBLEM - Puppet failure on tools-static-01 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:35:43] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:35:51] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:36:07] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL 33.33% of data above the critical threshold [0.0] [14:36:14] PROBLEM - Puppet failure on tools-submit is CRITICAL 55.56% of data above the critical threshold [0.0] [14:36:22] PROBLEM - Puppet failure on tools-exec-1408 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:37:20] PROBLEM - Puppet failure on tools-services-02 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:37:40] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:37:54] PROBLEM - Puppet failure on tools-exec-1203 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:38:00] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:38:02] PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:38:24] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:39:58] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL 30.00% of data above the critical threshold [0.0] [14:40:12] PROBLEM - Puppet failure on tools-exec-1404 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:40:54] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:41:00] PROBLEM - Puppet failure on tools-exec-1402 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:41:00] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:41:18] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:41:25] 6Labs, 10wikitech.wikimedia.org: Get rid of SMW/SRF/SF - https://phabricator.wikimedia.org/T53642#1404879 (10Yaron_Koren) I should note that Semantic Forms, as of six months ago, no longer requires the presence of SMW, so another option is to uninstall SMW and SRF while keeping SF. (Not that I'm recommending t... [14:41:48] twentyafterfour: now phab-03 doesn't have the log file for phd daemons [14:42:56] PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:42:58] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:43:19] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:43:56] YuviPanda: orite [14:44:33] puppet failures unrelated to labs, being looked at [14:44:46] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:44:52] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:45:02] PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:45:21] PROBLEM - Puppet failure on tools-mail is CRITICAL 22.22% of data above the critical threshold [0.0] [14:45:29] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:45:29] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:45:35] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:45:41] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:45:41] PROBLEM - Puppet failure on tools-shadow is CRITICAL 30.00% of data above the critical threshold [0.0] [14:45:43] PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:46:17] PROBLEM - Puppet failure on tools-exec-1401 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:46:23] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL 50.00% of data above the critical threshold [0.0] [14:46:37] PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 40.00% of data above the critical threshold [0.0] [14:46:37] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:46:41] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL 40.00% of data above the critical threshold [0.0] [14:47:15] PROBLEM - Puppet failure on tools-webproxy-01 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:47:42] 6Labs, 10wikitech.wikimedia.org: Get rid of SMW/SRF/SF - https://phabricator.wikimedia.org/T53642#1404888 (10scfc) I use it inter alia for generating lists for `pdsh` so that I can execute commands on all Tools or Toolsbeta instances. [14:47:51] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:47:55] PROBLEM - Puppet failure on tools-master is CRITICAL 20.00% of data above the critical threshold [0.0] [14:47:57] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1405 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:49:39] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:49:59] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:50:00] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL 40.00% of data above the critical threshold [0.0] [14:50:12] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL 70.00% of data above the critical threshold [0.0] [14:50:29] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:50:40] PROBLEM - Puppet failure on tools-services-01 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:51:22] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL 55.56% of data above the critical threshold [0.0] [14:51:36] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:51:39] 6Labs, 10wikitech.wikimedia.org: Get rid of SMW/SRF/SF - https://phabricator.wikimedia.org/T53642#1404916 (10yuvipanda) @scfc I use https://github.com/yuvipanda/personal-wiki/blob/master/project-dsh-generator.py instead. There is https://github.com/yuvipanda/personal-wiki/blob/master/tools-dsh-generator.py as... [14:52:46] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL 50.00% of data above the critical threshold [0.0] [15:06:06] 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1404982 (10yuvipanda) 3NEW [15:09:46] 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1405003 (10Yaron_Koren) I probably don't need to be subscribed to this ticket, but since I'm here: what do you mean by "confusing"? [15:10:32] 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1405011 (10yuvipanda) [15:10:44] 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1404982 (10yuvipanda) Edited to clarify. [15:12:33] YuviPanda, I'm not convinced by https://phabricator.wikimedia.org/T101517#1349065 [15:12:48] It's the host that's down [15:12:50] Not the service [15:12:56] oh, hmm [15:12:57] that's right [15:13:03] I should find the ping command and run that one [15:13:10] generic-host has this: check_command check_ping!500,20%!2000,100% [15:13:15] according to modules/shinken/files/templates.cfg [15:13:47] let me try that one [15:13:59] nagios has a check_ping_4 [15:14:07] as well as a check_ping [15:15:09] RECOVERY - Puppet failure on tools-exec-1407 is OK Less than 1.00% above the threshold [0.0] [15:15:29] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK Less than 1.00% above the threshold [0.0] [15:15:40] RECOVERY - Puppet failure on tools-services-01 is OK Less than 1.00% above the threshold [0.0] [15:16:14] Krenair: connect(3, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "2620:0:861:2:208:80:154:136", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [15:16:25] AF_INET6 [15:16:34] yup [15:16:36] RECOVERY - Puppet failure on tools-precise-dev is OK Less than 1.00% above the threshold [0.0] [15:16:36] with the ipv6 address [15:16:37] I'm guessing that's the problem [15:16:37] although a bit before that [15:16:47] does it attempt AF_INET? [15:16:55] I killed ircecho [15:17:03] it does [15:17:03] connect(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("208.80.154.136")}, 16) = 0 [15:17:09] and that... fails? [15:17:14] https://www.irccloud.com/pastebin/p3PFerv3/ [15:17:22] it does succeed [15:18:12] so... why does it try ipv6 as well? [15:18:22] and then fail because it could only get there via v4? [15:19:36] weird [15:19:41] yeah [15:19:45] anyway, shall we make it check v4 only? [15:19:46] I guess check_ping_4 will fix it [15:19:47] yeah [15:20:00] also labs can't speak to ipv6 networks even? [15:21:05] silver.wikimedia.org has IPv6 address 2620:0:861:2:208:80:154:136 [15:21:06] krenair@tools-bastion-01:~$ ping 2620:0:861:2:208:80:154:136 [15:21:07] ping: unknown host 2620:0:861:2:208:80:154:136 [15:21:09] apparently not [15:21:31] or would it be ping6 [15:21:39] well, that returns connect: Network is unreachable [15:21:40] so [15:21:44] YuviPanda, nope. [15:23:22] 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1405066 (10Yaron_Koren) Alright. :) By "interface", do you mean the mechanism for storing the data, or for querying it? If it's the latter, one possibly... [15:26:03] YuviPanda, so.. config, is it possible to override generic-host's check_command on an individual host basis? [15:30:51] hmm... [15:30:57] I think that should work. will upload a patch [15:31:38] Krenair: no, but we can just make it do ipv4 only checks for labs [15:31:43] Krenair: labs has no support anyway [15:32:56] so just change check_ping to check_ping_4 in the generic-host definition? [15:33:17] Krenair: ya [15:34:01] 6Labs, 10Tool-Labs: tools-bastion-01 has the wrong key for tools-bastion-02 in cache - https://phabricator.wikimedia.org/T103999#1405109 (10valhallasw) 3NEW [15:34:47] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK Less than 1.00% above the threshold [0.0] [15:34:55] RECOVERY - Puppet failure on tools-exec-1403 is OK Less than 1.00% above the threshold [0.0] [15:35:01] RECOVERY - Puppet failure on tools-exec-1212 is OK Less than 1.00% above the threshold [0.0] [15:35:19] RECOVERY - Puppet failure on tools-mail is OK Less than 1.00% above the threshold [0.0] [15:35:30] RECOVERY - Puppet failure on tools-exec-1216 is OK Less than 1.00% above the threshold [0.0] [15:35:30] RECOVERY - Puppet failure on tools-exec-1207 is OK Less than 1.00% above the threshold [0.0] [15:35:36] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0] [15:35:42] RECOVERY - Puppet failure on tools-shadow is OK Less than 1.00% above the threshold [0.0] [15:35:43] RECOVERY - Puppet failure on tools-exec-1202 is OK Less than 1.00% above the threshold [0.0] [15:35:45] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK Less than 1.00% above the threshold [0.0] [15:36:37] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK Less than 1.00% above the threshold [0.0] [15:36:39] RECOVERY - Puppet failure on tools-exec-1405 is OK Less than 1.00% above the threshold [0.0] [15:38:39] 6Labs, 3Labs-Sprint-101, 3Labs-Sprint-102: Sort out remaining virt1000 salt minions - https://phabricator.wikimedia.org/T103010#1405143 (10Andrew) Did I get all of them this time? [15:40:23] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-103: Instances without a shared NFS storage suffers from a 3 minutes boot delay - https://phabricator.wikimedia.org/T102544#1405146 (10yuvipanda) a:5yuvipanda>3None [15:40:33] andrewbogott: think you'll have time today for https://phabricator.wikimedia.org/T102544? [15:41:17] I’ll look at it… but I probably don’t want to install new images just for that. [15:43:54] 6Labs: Identify (and potentially) help move mwoffliner off NFS - https://phabricator.wikimedia.org/T102682#1405167 (10yuvipanda) 5Open>3Resolved And all other NFS mounts have been disabled. [15:43:56] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1405169 (10yuvipanda) [15:44:54] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 7Shinken: shinken thinks wikitech and wikitech-static are down. They aren't. - https://phabricator.wikimedia.org/T101517#1405178 (10Krenair) a:5yuvipanda>3Krenair [15:45:22] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1360119 (10yuvipanda) [15:45:24] 6Labs: Disable NFS from performance project - https://phabricator.wikimedia.org/T103824#1405180 (10yuvipanda) 5Open>3Resolved Done. [15:45:57] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 7Shinken: shinken thinks wikitech and wikitech-static are down. They aren't. - https://phabricator.wikimedia.org/T101517#1405186 (10Krenair) 5Open>3Resolved It's a host issue, not service. The generic-host check is for ping, and https://www.irccloud.c... [15:48:56] andrewbogott: alright, although it's fairly annoying :( [15:49:32] it increases new instance creation by... quite a large number [15:49:44] and more than 100 projects now have no trace of NFS on them (I'm down to 34 projects now) [15:50:50] wgOpenStackManagerRemoveUserFromBastionProjectOnShellDisable [15:51:09] Krenair: :D [15:52:22] hmm [15:52:25] YuviPanda: also it’s more-or-less impossible to test until the export daemon is running again. [15:52:30] this logic would not work in a CA-like setup [15:52:41] andrewbogott: fair enough actually [15:53:02] Krenair: yeah but a lot of things won't work in a CA like setup... [16:01:16] Krenair: updated [16:06:36] YuviPanda, approved, yay for one less wikitech bug [16:08:38] Krenair: <3 no swat until monday tho [16:08:46] 6Labs, 12Analytics-Backlog, 10Labs-Infrastructure: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#1405247 (10ggellerman) p:5Triage>3Low [16:30:02] 6Labs, 12Analytics-Backlog, 10Labs-Infrastructure: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#1405313 (10Milimetric) Dear @Spage: we can't commit to supporting a production or labs instance of piwik which would help with this. Using Event Logging from labs might be an o... [16:39:44] Coren|Away: is the toolserver-legacy project dependent on NFS at all? [16:52:30] Coren|Away, YuviPanda, remind me where the exports for a given NFS volume are defined? [16:52:41] andrewbogott: /etc/exports.d I think? [16:53:26] YuviPanda: yeah, that’s it. thanks [16:56:06] YuviPanda: do I have to run something to refresh? [16:56:20] andrewbogott: I don't know. I think you've to edit it by hand right now? [16:56:21] where is labs puppet hosted? I have a task to add a puppet manifest for shiny (server that runs searchdata.wmflabs.org)...although i'm not yet sure that a puppet manifest in labs puppet is the right answer...but maybe [16:56:26] (the git repo) [16:56:34] YuviPanda: yes, edited, but that’s not enough. [16:56:59] ebernhardson: https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet [16:57:38] andrewbogott: oh its the regular prod puppet? yea thats probably not a good place for this :) [16:57:58] ebernhardson: yes. Some passwords and keys and such come from a different repo [16:59:29] YuviPanda: seems to be /usr/local/sbin/sync-exports [17:04:24] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Isolation, 3Labs-Sprint-103: Instances without a shared NFS storage suffers from a 3 minutes boot delay - https://phabricator.wikimedia.org/T102544#1405408 (10hashar) + #ci-isolation so I get it on my radar. That is not needed for that project though. [17:05:34] YuviPanda: my project, utrs-primary I can't ssh into again with the latest NFS failure. can we get that fixed again please? [17:06:11] YuviPanda: I get ~120ms average, but with 450ms peaks at home as RTTs [17:11:43] hey i thought i'd throw this out if anyone else's use case for mosh is like mine, not as much lag problems as just wanting a persistent tmux: http://www.harding.motd.ca/autossh/ [17:12:00] it seems to Just Work across the bastion [17:21:45] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Isolation, 3Labs-Sprint-103, 5Patch-For-Review: Instances without a shared NFS storage suffers from a 3 minutes boot delay - https://phabricator.wikimedia.org/T102544#1405470 (10Andrew) a:3Andrew [17:33:05] andrewbogott: around? [17:33:32] Izhidez: what’s up? [17:34:23] I just looked through my logs of what was wrong with my instance last time, it appears you were able to fix it. Would you be willing to do it again? I have the same issue and logs from before [17:34:56] what instance/project? [17:35:09] UTRS / instance: utrs-primary [17:35:26] https://www.irccloud.com/pastebin/PMfK3OEm/ [17:35:38] thats what you said was wrong last time ^ [17:35:46] I can't ssh into the instace [17:35:47] and when you say ‘the same issue' [17:35:57] just ssh? [17:36:14] yes, my public key is denied [17:36:25] but the tool works fine [17:37:25] it’s not the same issue, something is broken with puppet [17:37:25] (i'm going to take some time over the next week to make a new instance and push everything over there so it's a clean start [17:37:27] but i’m looking [17:38:21] ya you noted that puppet not running was not a good thing last time. [17:39:00] “Could not find class role::labsnfs::client” [17:44:33] Hi, I am all of a sudden unable to access my instance: http://drmf.wmflabs.org/wiki/Main_Page [17:44:39] Does anybody know what might be happening? [17:44:59] andrewbogott: is that something I need to fix? cause I can't find it in configure instance [17:45:27] Yes, I removed it already [17:46:56] wow, your instance doesn’t have lsb_release. /that’s/ not going to work [17:47:34] hence why I'm going to port it to a brand new instance. fresh start everything [17:47:44] but without ssh access, I can't do that [17:49:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/MaxSem was created, changed by MaxSem link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/MaxSem edit summary: Created page with "{{Tools Access Request |Justification=test |Completed=false |User Name=MaxSem }}" [17:50:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/MaxSem was modified, changed by MaxSem link https://wikitech.wikimedia.org/w/index.php?diff=168091 edit summary: {{delete}} [17:52:55] Izhidez: try now? [17:53:01] k, standby [17:53:33] Nevermind, it is back again. [17:53:45] yep, in. Should I try and port to a new instance, or is this sufficent? [17:53:56] thanks also [17:54:47] Izhidez: that instance will probably keep breaking — best to move everything to Jessie. [17:55:03] ok, i'll try and do that this week. [18:07:25] 6Labs, 5Patch-For-Review: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1405655 (10Andrew) ok, I've added the project-id metadata setting to every instance. [18:08:24] 6Labs, 5Patch-For-Review: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1405664 (10Andrew) [18:08:25] 6Labs: Add project_id to instance metadata during instance creation - https://phabricator.wikimedia.org/T102832#1405662 (10Andrew) 5Open>3Resolved This is done by the sink/ldap hook. [18:10:08] 6Labs, 5Patch-For-Review: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#1405672 (10Andrew) [18:10:09] 6Labs, 5Patch-For-Review: Move to a new dns scheme for labs: hostname.projectname.eqiad.wmflabs - https://phabricator.wikimedia.org/T93087#1405670 (10Andrew) 5Open>3Resolved a:3Andrew [18:10:19] 6Labs, 5Patch-For-Review: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#1405677 (10Andrew) 5Open>3Resolved a:3Andrew [18:11:10] 6Labs, 5Patch-For-Review: Set up designate-dashboard - https://phabricator.wikimedia.org/T93089#1405684 (10Andrew) [18:11:11] 6Labs, 5Patch-For-Review: Clarify public/private role for holmium (aka labs-ns2) - https://phabricator.wikimedia.org/T93639#1405683 (10Andrew) [18:11:12] 6Labs, 5Patch-For-Review: Use Designate for public/floating labs IPs - https://phabricator.wikimedia.org/T93088#1405685 (10Andrew) [18:11:14] 6Labs, 5Patch-For-Review: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#987813 (10Andrew) [18:15:07] 6Labs, 5Patch-For-Review: Set up designate-dashboard - https://phabricator.wikimedia.org/T93089#1405695 (10Andrew) It should be available stock in Kilo, so best to wait for that. [18:18:14] YuviPanda: if you are ok with https://gerrit.wikimedia.org/r/#/c/220157/2/modules/puppetmaster/templates/git-sync-upstream.erb then we can turn on auto-update by default. [18:18:51] andrewbogott: am out ATM I'll be back in a few hours [18:18:56] ok [18:18:57] But yeah feel free to merge that [18:19:05] ok! [18:19:06] It shouldn't destroy Anything [18:19:22] And puppetmasters that don't currently auto update don't get that anyway [18:19:30] Do test it tho - I haven't [18:19:32] Brb [18:28:25] i created a new Precise instance yesterday: [18:28:26] N: Ignoring 'apt.conf.d' in directory '/etc/apt/apt.conf.d/' as it is not a regular file [18:28:52] /etc/apt/apt.conf.d/apt.conf.d/ contains similar but different stuff from /etc/apt/apt.conf.d/ [18:29:13] not using self-hosted puppetmaster [18:31:38] jgage: are things failing or is that just a puppet warning? [18:32:30] it's a warning from apt-get update/upgrade [18:32:42] they still work, they're just ignoring that subdir and its contents [18:33:28] andrewbogott: btw have you tested self-hosted puppetmaster on trusty or jessie? i've been sticking with precise for puppetmasters because that's what the prod ones are. [18:33:37] jgage: ok, make a phab task? [18:33:41] k [18:33:50] jgage: I haven’t tested much. It should work on Trusty [18:33:58] cool ok, i'll try it [18:35:15] * andrewbogott -> lunch [18:40:49] 6Labs, 6operations: update star.wmflabs.org cert from sha1 to sha256 - https://phabricator.wikimedia.org/T104017#1405726 (10RobH) 3NEW [18:41:16] 6Labs, 6operations: update star.wmflabs.org cert from sha1 to sha256 - https://phabricator.wikimedia.org/T104017#1405734 (10RobH) [18:43:15] 6Labs, 6operations: update star.wmflabs.org cert from sha1 to sha256 - https://phabricator.wikimedia.org/T104017#1405726 (10RobH) there also seems to be two wmflabs certificate files in the repo: star.wmflabs.crt star.wmflabs.org.crt Now, rapidssl also happens to have two different SHA1 hashed certificates f... [18:43:54] 6Labs: Nested ".d" dirs in /etc/apt/ - https://phabricator.wikimedia.org/T104019#1405744 (10Gage) 3NEW [18:46:14] 6Labs: Nested ".d" dirs in /etc/apt/ - https://phabricator.wikimedia.org/T104019#1405754 (10Gage) [18:52:13] 6Labs, 6operations, 5Patch-For-Review: update star.wmflabs.org cert from sha1 to sha256 - https://phabricator.wikimedia.org/T104017#1405767 (10RobH) once the cert is updated and in place, someone should kick this task back to me for rapidssl cert revocation of the older sha1 certs. [19:02:57] 6Labs, 6operations, 5Patch-For-Review: update star.wmflabs.org cert from sha1 to sha256 - https://phabricator.wikimedia.org/T104017#1405807 (10RobH) p:5Triage>3High [19:03:40] 6Labs, 6operations, 5Patch-For-Review: update star.wmflabs.org cert from sha1 to sha256 - https://phabricator.wikimedia.org/T104017#1405808 (10RobH) a:3yuvipanda Chatted with Yuvi in IRC. This seems like it would be something he would handle, so I'll assign it to him for now. If incorrect, he can just ki... [19:29:43] (03PS1) 10Legoktm: Stop adding comments [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/221180 (https://phabricator.wikimedia.org/T100945) [19:30:19] valhallasw: ^ [19:31:06] (03CR) 10Merlijn van Deen: [C: 032] Stop adding comments [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/221180 (https://phabricator.wikimedia.org/T100945) (owner: 10Legoktm) [19:31:27] (03CR) 10Jforrester: Stop adding comments (031 comment) [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/221180 (https://phabricator.wikimedia.org/T100945) (owner: 10Legoktm) [19:32:01] (03CR) 10Legoktm: [V: 032] Stop adding comments (031 comment) [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/221180 (https://phabricator.wikimedia.org/T100945) (owner: 10Legoktm) [19:33:04] legoktm: :-P [22:06:34] (03PS1) 10Gage: Add Gage's key to labs root [labs/private] - 10https://gerrit.wikimedia.org/r/221300 [22:39:15] 6Labs, 5Patch-For-Review: Nested ".d" dirs in /etc/apt/ - https://phabricator.wikimedia.org/T104019#1406284 (10Gage) I'm surprised to report that this is still happening on instances created after the patch was merged. I tried twice. [23:27:01] 6Labs, 5Patch-For-Review: Nested ".d" dirs in /etc/apt/ - https://phabricator.wikimedia.org/T104019#1406360 (10scfc) The patch only changed the script that builds the base images. So someone will need to build new images and deploy them; then you will see that it has been resolved (probably). [23:59:48] 6Labs, 6Discovery, 10Maps: Investigate and reduce NFS use in maps-team project - https://phabricator.wikimedia.org/T103757#1406404 (10MaxSem) Please keep /data/project, the rest can go. Data storage is still useful for sharing dowloaded dumps between instances.