[00:00:31] RECOVERY - Puppet failure on tools-bastion-01 is OK Less than 1.00% above the threshold [0.0] [00:01:03] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1308395 (10scfc) @yuvipanda, you added `sbt` to `toollabs::dev_environ` with d90368777c67f40c35e59cd7f5a5b898dbff89cb: ``` Toollabs: Add sbt to dev environments Scala Build Tool, for anyone who wants to build... [01:19:22] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1308450 (10scfc) I have moved `/data/project/.system/deb-trusty/python3-statsd_3.0.1-1_all.deb` and `/data/project/.system/deb-{precise,trusty}/python-websocket-client_0.23.0-1_all.deb` to `~tools.admin/archived-package... [01:19:39] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1308451 (10scfc) [02:20:10] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1308470 (10scfc) I have also moved `/data/project/.system/deb-{precise,trusty}/python-backports.ssl-match-hostname_3.4.0.2-1_all.deb` to `~tools.admin/archived-packages/` as it was not referenced by Puppet manifests. @... [02:20:27] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1308471 (10scfc) [02:24:24] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL - Socket timeout after 10 seconds [02:29:18] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 782916 bytes in 2.575 second response time [02:35:02] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1308488 (10scfc) [03:27:25] Why am I not allowed to ssh into the tools-exec instances ('Permission denied (publickey)') [06:28:48] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL 20.00% of data above the critical threshold [0.0] [06:58:51] RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK Less than 1.00% above the threshold [0.0] [08:18:55] !log tools.wikibugs valhallasw: Deployed 82b0b9f487ece85a40595b80f3f690554743e472 Ignore Forrestbot wb2-phab, wb2-irc [08:19:00] Logged the message, Master [08:28:29] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:28:31] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:30:10] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:30:32] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL 100.00% of data above the critical threshold [0.0] [08:31:20] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:31:36] PROBLEM - Puppet failure on tools-master is CRITICAL 100.00% of data above the critical threshold [0.0] [08:31:50] PROBLEM - Puppet failure on tools-mail is CRITICAL 100.00% of data above the critical threshold [0.0] [08:31:56] PROBLEM - Puppet failure on tools-mailrelay-01 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:32:44] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:33:24] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:33:34] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1308833 (10yuvipanda) Yes, python-sh was used for one of my own tools (Such A Bot) but it uses a virtenv now so can be killed. [08:33:40] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:34:50] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:38:14] YuviKTM: ^^^^ [08:38:21] STUFF IS BREAKING :OOOOO [08:39:19] PROBLEM - Puppet failure on tools-shadow is CRITICAL 100.00% of data above the critical threshold [0.0] [08:39:34] valhallasw: it's the gridengine package [08:39:45] valhallasw: it's just the daily shinken alert for things that are broken [08:39:49] aaaah [08:39:55] it re-alerts after 1 day [08:40:27] ah, hence the 100% [08:41:33] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:43:21] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL 100.00% of data above the critical threshold [0.0] [08:44:21] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL 100.00% of data above the critical threshold [0.0] [08:44:37] PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 100.00% of data above the critical threshold [0.0] [08:45:07] PROBLEM - Puppet failure on tools-submit is CRITICAL 100.00% of data above the critical threshold [0.0] [08:46:01] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL 100.00% of data above the critical threshold [0.0] [09:03:56] 10Tool-Labs: Install xvfb on exec nodes - https://phabricator.wikimedia.org/T100268#1308977 (10coren) 3NEW [09:06:58] YuviPanda: ^^ [09:40:00] 10Tool-Labs: PHP sessions should use Redis with per-tool prefix - https://phabricator.wikimedia.org/T100272#1309143 (10Mattflaschen) 3NEW [09:49:35] 6Labs: Provide a way to have homedirs on a separate LVM volume instead of NFS - https://phabricator.wikimedia.org/T100274#1309204 (10yuvipanda) 3NEW [11:04:39] [13intuition] 15siebrand pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/480145f7e075fc679906fb08ee97bf14c58f53f1 [11:04:40] 13intuition/06master 14480145f 15Siebrand Mazeland: Localisation updates from https://translatewiki.net. [11:11:41] 10Tool-Labs, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-05-05: Add wb_changes_subscription and wbc_entity_usage to labs db replication - https://phabricator.wikimedia.org/T98748#1309467 (10Tobi_WMDE_SW) [11:39:51] coren, yuvipanda: all the actual project storage is still on labstore1001, right? [11:39:59] andrewbogott: yes [11:40:03] andrewbogott: Aye [11:40:13] ok, lemme archive the projects I deleted :) [11:40:26] Coren: I found another 1.2T log file filled with php notices only :) soon NFS usage will be under 10T [11:40:29] hey, there are lots [11:40:31] I'll do an audit slowly [11:40:39] Kill, Destroy, Exterminate. [11:42:50] Coren: :) [11:48:51] yuvipanda: ok, the deleted-project cleanup is finished; there are 76G in ‘orphan-volumes,’ that’s the compressed remnants of deleted projects. [12:03:49] andrewbogott: sweet :) [12:07:02] Hi, what's the best way to analyse the current and past content of all pages in a category? https://meta.wikimedia.org/wiki/WikiXRay_Python_parser looks useful, but can I use it on Tool Labs? [12:11:05] Someone asked me about this problem via email. Is there anyone familiar with analytics here who is willing to help? I'll refer the asker to you. [12:15:32] Zhaofeng_Li, current and past content of all pages *currently* in a category? [12:16:58] Krenair: Yes, and the asker wants to analyse all revisions of a page, and Special:Export doesn't meet his needs. [12:19:13] I guess a dump with slightly outdated content is acceptable as well. [13:30:41] 6Labs, 10LabsDB-Auditor, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, and 8 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1310001 (10Aklapper) Anybody up to make decisions on T89270#1245165 ? If not this might just be "works enough for me" and get closed. [13:37:34] 10Tool-Labs, 10Wikimedia-Hackathon-2015: Tool-labs meeting agenda for Lyon Hackathon - https://phabricator.wikimedia.org/T98912#1310038 (10valhallasw) [14:12:08] 6Labs: point puppet.eqiad.wmflabs to virt1000.wikimedia.org in labs dns - https://phabricator.wikimedia.org/T100317#1310224 (10Andrew) 3NEW a:3Andrew [14:44:33] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:46:37] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:51:03] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1310314 (10yuvipanda) python-sh is gone [15:01:35] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0] [15:14:32] RECOVERY - Puppet failure on tools-webproxy-02 is OK Less than 1.00% above the threshold [0.0] [15:17:03] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1310356 (10scfc) [15:18:50] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1097938 (10scfc) @yuvipanda, what about `sbt`? [15:19:14] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1310359 (10scfc) [15:40:56] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1310423 (10valhallasw) [15:57:33] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1310451 (10scfc) [16:26:04] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1310522 (10scfc) [16:39:37] (03PS1) 10Sitic: Added option to only show latest change per page [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/213587 (https://phabricator.wikimedia.org/T100159) [16:40:01] (03CR) 10Sitic: [C: 032 V: 032] Added option to only show latest change per page [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/213587 (https://phabricator.wikimedia.org/T100159) (owner: 10Sitic) [16:43:34] (03PS1) 10Rjain: basic setup for the application [labs/tools/flow-oauth-demo] - 10https://gerrit.wikimedia.org/r/213590 [18:28:16] !ping [18:28:16] !pong [18:37:48] 10Tool-Labs: Reduce amount of Tools-local packages - https://phabricator.wikimedia.org/T91874#1310729 (10scfc) [19:23:01] legoktm: could you add confirmed to Negative24-testbot? [19:23:53] or make me a 'crat [19:24:09] I may have to fiddle with a few other things [19:25:07] on testwiki [19:31:47] or some other 'crat on testwiki? [19:38:01] what wiki is this? [19:38:17] oh, testwiki is only in prod, not beta [19:38:17] ok [19:38:24] strange channel choice [19:38:58] right? [19:39:11] Negative24? [19:39:32] Negative24: yea oh wait wrong channel [19:39:43] I'm on so many channels [19:40:01] and they're cutoff to #wikimedia***** [19:40:17] Krenair: ^ [19:40:51] I can't remember whether http://test.wikimedia.beta.wmflabs.org/ is testwiki or testwikimedia [19:41:05] is that the one you meant? or https://test.wikipedia.org/ ? [19:42:07] Krenair: test.wikipedia.org [19:42:16] the first one is the beta cluster [19:42:55] done, Negative24 [19:43:28] Krenair: thanks [19:54:54] PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL 20.00% of data above the critical threshold [0.0] [19:58:03] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL 40.00% of data above the critical threshold [0.0] [20:03:13] YuviPanda, Coren, all of these alerts are me messing with dnsmasq [20:03:16] I think [20:11:32] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 20.00% of data above the critical threshold [0.0] [20:12:24] PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL 50.00% of data above the critical threshold [0.0] [20:13:38] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL 30.00% of data above the critical threshold [0.0] [20:20:11] 6Labs: point puppet.eqiad.wmflabs to virt1000.wikimedia.org in labs dns - https://phabricator.wikimedia.org/T100317#1310973 (10Andrew) This requires a change for dnsmasq /and/ in powerdns. The dnsmasq change will be puppetized; for the powerdns change I'll just insert a record straight via the api. Once we dep... [20:23:03] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK Less than 1.00% above the threshold [0.0] [20:24:53] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK Less than 1.00% above the threshold [0.0] [20:37:25] RECOVERY - Puppet failure on tools-exec-1206 is OK Less than 1.00% above the threshold [0.0] [20:41:38] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0] [20:43:34] RECOVERY - Puppet failure on tools-exec-1409 is OK Less than 1.00% above the threshold [0.0]