[00:14:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:56:52] (03PS1) 10BryanDavis: Add link to OATH documentation on wikitech [labs/striker] - 10https://gerrit.wikimedia.org/r/333321 [00:58:21] (03PS2) 10BryanDavis: Add link to OATH documentation on wikitech [labs/striker] - 10https://gerrit.wikimedia.org/r/333321 [00:58:58] (03PS3) 10BryanDavis: Add link to OATH documentation on wikitech [labs/striker] - 10https://gerrit.wikimedia.org/r/333321 [01:10:05] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:11:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jack who built the house was created, changed by Jack who built the house link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Jack_who_built_the_house edit summary: Created page with "{{Tools Access Request |Justification=Hello, I have an "engineer" flag on Russian Wikipedia (engineers have editinterface & editprotected rights, among others), I have many in..." [02:16:19] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jack who built the house was modified, changed by Jack who built the house link https://wikitech.wikimedia.org/w/index.php?diff=1364990 edit summary: [02:35:07] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jack who built the house was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1365003 edit summary: [02:37:43] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:06:16] 06Labs, 06Operations, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2757207 (10scfc) +1. The last time this bugged me I thought maybe the Puppet roles were re-read from the filesystem each time, looked at the source (`modules/openstack/files/liberty/horizon/puppet... [03:42:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:54:37] 06Labs, 06Operations, 10netops: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2957555 (10faidon) [04:41:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:40:58] RECOVERY - Free space - all mounts on tools-exec-1220 is OK: OK: tools.tools-exec-1220.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [06:44:08] RECOVERY - Free space - all mounts on tools-exec-1221 is OK: OK: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [07:46:03] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:37:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:41:43] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:16:41] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [11:22:50] !log fa-wp deploying 96a481a into tofawiki [11:22:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Fa-wp/SAL [12:01:58] PROBLEM - Free space - all mounts on tools-exec-1220 is CRITICAL: CRITICAL: tools.tools-exec-1220.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-1220.diskspace.root.byte_percentfree (<44.44%) [12:14:02] somone familiar with addwiki? [12:42:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:50:10] PROBLEM - Free space - all mounts on tools-exec-1221 is CRITICAL: CRITICAL: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-1221.diskspace.root.byte_percentfree (<11.11%) [13:12:47] bd808: are you online? [13:32:52] [Sat Jan 21 13:30:20 2017] /mnt/nfs/labstore-secondary-tools-project/commons-delinquent/rmcatmoves.php: not an executable file <-- anyone a idea how to fix this? it stucks on grd engine. sigh. [13:49:05] resolved [14:09:04] bd808: is there anything in place that ensures that a tool can be adopted/usurpated when the maintainer is absent and doesn't run on tool labs but another labs project? [14:13:51] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2958049 (10Samtar) [14:13:54] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-LTA-Knowledgebase: tools.lta missing replica.my.cnf - https://phabricator.wikimedia.org/T155317#2958048 (10Samtar) [15:07:18] PROBLEM - Free space - all mounts on tools-worker-1014 is CRITICAL: CRITICAL: tools.tools-worker-1014.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found)tools.tools-worker-1014.diskspace.root.byte_percentfree (<11.11%) [16:56:40] Hi, it seems on gerrit-test3, it dosent have the home dir now [16:56:46] ssh in results in [16:56:47] Could not chdir to home directory /home/paladox: No such file or directory [16:56:53] chasemp yuvipanda ^^ [17:28:53] 10Tool-Labs-tools-LTA-Knowledgebase: Write contributing.md - https://phabricator.wikimedia.org/T155901#2958242 (10Samtar) [17:52:51] 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2958301 (10Krenair) [19:08:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:11:22] paladox: i'm looking [19:13:42] paladox: looks like it was one of the affected instances, but somehow missed out in out list of nodes to recover. recovery script running now [19:13:57] Oh thanks @madhuvishy :) [19:28:33] paladox: check now? some executable permissions may be lost but hopefully all your files should be there [19:28:50] Hi, yep, my /home/paladox is back :), thanks [19:28:57] okay great [20:13:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:42] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:50:23] 10Tool-Labs-tools-Wikidata-Periodic-Table, 10Wikidata, 07Documentation: Improve documentation of Wikidata periodic table - https://phabricator.wikimedia.org/T99847#2958479 (10Ricordisamoa) [20:50:53] (03PS1) 10Ricordisamoa: [WIP] README.rst and Sphinx documentation [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/333367 (https://phabricator.wikimedia.org/T99847) [20:52:13] annika: at the moment the right to fork and takeover policies are specific to Tool Labs. legoktm suggested that we expand them to all of Labs early in the discussion, but I wanted to keep a more narrow scope until we see how the process actually works. [20:52:27] mhm [20:52:39] I think if things go well in the first few months I would be open to a vote to expand to all of Labs [20:53:10] and really anyone who was willing to put in the work to advertise and call a voite could push for that at any time :) [21:09:03] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:14:53] (03PS2) 10Ricordisamoa: [WIP] README.rst and Sphinx documentation [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/333367 (https://phabricator.wikimedia.org/T99847) [21:17:42] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [21:17:58] (03CR) 10Ricordisamoa: "PS2 fixes a typo in docs/conf.py" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/333367 (https://phabricator.wikimedia.org/T99847) (owner: 10Ricordisamoa) [21:19:24] (03PS3) 10Ricordisamoa: [WIP] README.rst and Sphinx documentation [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/333367 (https://phabricator.wikimedia.org/T99847) [21:21:20] (03CR) 10Ricordisamoa: "PS3 adds the favicon via the new docs/_templates/layout.html" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/333367 (https://phabricator.wikimedia.org/T99847) (owner: 10Ricordisamoa) [21:37:54] (03CR) 10Ricordisamoa: [C: 04-1] [WIP] README.rst and Sphinx documentation (033 comments) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/333367 (https://phabricator.wikimedia.org/T99847) (owner: 10Ricordisamoa) [22:14:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:16:28] (03CR) 10Lokal Profil: [C: 032] Fix name of CommonsCat field in fr_(fr) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/332824 (owner: 10Jean-Frédéric) [22:17:47] (03Merged) 10jenkins-bot: Fix name of CommonsCat field in fr_(fr) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/332824 (owner: 10Jean-Frédéric) [22:20:37] (03CR) 10jenkins-bot: Fix name of CommonsCat field in fr_(fr) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/332824 (owner: 10Jean-Frédéric) [22:38:40] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:39:41] (03CR) 10Ricordisamoa: [C: 04-1] [WIP] README.rst and Sphinx documentation (031 comment) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/333367 (https://phabricator.wikimedia.org/T99847) (owner: 10Ricordisamoa) [23:10:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:18:43] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [23:28:37] 06Labs, 10Horizon, 06Operations, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2958596 (10bd808)