[00:09:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0]  
[00:40:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[01:05:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[01:05:59] <Cyberpower678>	 YuviPanda, ping
[01:07:08] <Cyberpower678>	 YuviPanda, if you're there, could you add me to the wikiviewstats project on toollabs?
[01:32:46] <wikibugs>	 3translatewiki.net, Wikimedia-Labs-General: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#990531 (10scfc) 208.80.155.255 doesn't look like a dedicated public IP, so it is probably from an instance without one.  I don't think that all network traffic is logged, s...
[01:59:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy-jessie is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:17:30] <Cyberpower678>	 YuviPanda, ping
[02:30:18] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<66.67%)  
[03:24:57] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[03:39:06] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[03:49:57] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:50:19] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[04:04:05] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]  
[04:13:25] <Cyberpower678>	 FastLizard4, do you know anyone here that is a toollabs admin that is currently available?
[05:00:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[05:05:29] <wikibugs>	 3translatewiki.net, Wikimedia-Labs-General: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#990627 (10Nikerabbit) >>! In T87352#990531, @scfc wrote: > 208.80.155.255 doesn't look like a dedicated public IP, so it is probably from an instance without one.  I don't...
[05:08:26] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[05:30:57] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[05:38:25] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:01:12] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]  
[06:26:13] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:37:14] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[06:39:35] <wikibugs>	 3translatewiki.net, Wikimedia-Labs-General: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#990679 (10Nemo_bis) I don't know how up to date it is, but https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools/Overview&oldid=141799#Sun_Grid_Engine_.28SGE...
[07:02:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[07:07:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:16:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[07:27:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:32:56] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[07:46:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:57:57] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:59:35] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[08:00:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[08:24:32] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:30:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:56:15] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[09:26:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[09:37:57] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[09:40:32] <wikibugs>	 3Tool-Labs: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#990833 (10Nosy) guess the db is called s51892_toolserverdb_p (with _p at the end for public access)
[09:41:39] <shinken-wm>	 PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[09:45:44] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[09:52:16] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<44.44%)  
[09:53:06] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[10:02:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:07:15] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[10:11:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:13:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0]  
[10:20:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:23:02] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:25:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[10:33:12] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[10:38:52] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:46:49] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[10:50:43] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:58:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:01:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[11:13:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[11:16:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:17:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[11:20:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[11:22:12] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[11:22:12] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[11:24:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[11:31:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:33:16] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[11:42:13] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:42:57] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:44:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:47:20] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:49:20] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:50:51] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[12:07:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[12:21:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[12:28:18] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[12:37:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0]  
[12:46:07] <shinken-wm>	 RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0]  
[12:46:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[13:07:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[13:07:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[13:21:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:22:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[13:27:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[13:29:17] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[13:30:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[13:32:38] <shinken-wm>	 RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:33:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:34:53] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]  
[13:40:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[13:40:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[13:42:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[13:47:09] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:55:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:57:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:59:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:59:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:00:56] <shinken-wm>	 PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[14:07:44] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:10:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:11:00] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[14:15:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]  
[14:15:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:17:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[14:21:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[14:33:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[14:34:00] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[14:37:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:40:54] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:41:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[14:45:36] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:50:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:51:46] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0]  
[14:53:19] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[14:59:14] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[15:03:55] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[15:04:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[15:11:53] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[15:23:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[15:24:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[15:34:15] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%)  
[15:44:18] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[16:06:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[16:18:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[16:34:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[16:34:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[16:41:59] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[16:53:40] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[16:59:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[17:05:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[17:41:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[18:03:44] <wikibugs>	 3Labs-Team: irc bouncer project - https://phabricator.wikimedia.org/T87442#991216 (10mmodell) 3NEW
[18:04:40] <Cyberpower678>	 Krenair, are you a projectadmin of the tools project?
[18:06:25] <Krenair>	 no
[18:06:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[18:10:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[18:15:48] <guillom>	 Is there an archive of tools from the Toolserver somewhere?
[18:20:18] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[18:25:13] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[18:28:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[18:33:53] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[18:40:52] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0]  
[18:45:15] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[18:50:17] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<22.22%)  
[18:51:21] <wikibugs>	 3Wikimedia-Labs-General, translatewiki.net: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#991373 (10scfc) That list contains inter alia all Precise webservers, so it narrows the pool "down" to all tools that had a webservice running and did not explicitly reques...
[18:53:56] <Cyberpower678>	 MusikAnimal, ping
[18:54:04] <MusikAnimal>	 what's up
[18:54:14] <Cyberpower678>	 I need your admin bit. :p
[18:54:19] <MusikAnimal>	 sure
[18:54:49] <Cyberpower678>	 When you updated the links to the edit counter, the contributions footer still has the old timing out one.
[18:55:05] <Cyberpower678>	 I also moved articleinfo to it's new branch.
[18:55:15] <Cyberpower678>	 So feel free to update links to those too.
[18:55:17] <MusikAnimal>	 yeah, I wanted to update the contributions footer but not sure where that lives
[18:55:35] <Cyberpower678>	 MediaWiki:Sp-contribution-footer I think
[18:55:39] <MusikAnimal>	 yay to articleinfo! that one certianly has high demand as well
[18:56:38] <Cyberpower678>	 And it works perfectly. :D
[18:57:05] <MusikAnimal>	 A+++++++
[18:57:39] <YuviPanda>	 !Log tools.xtools restarted webservice, was getting blank page (reported by quiddity)
[18:57:46] <labs-morebots>	 Logged the message, Master
[18:58:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[18:58:17] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[19:01:32] <MusikAnimal>	 Cyberpower678: update footer with new edit counter, and dropdowns gadget with new articleinfo. Is articleinfo linked in any footers?
[19:01:37] <MusikAnimal>	 *updated
[19:01:58] <MusikAnimal>	 ah, on the page history at the top
[19:02:03] <YuviPanda>	 Cyberpower678: MusikAnimal hmm, tools.wmflabs.org/xtools gives me a blank page
[19:02:09] <Cyberpower678>	 MusikAnimal, view history has a link to the tool.
[19:02:30] <MusikAnimal>	 that URL is redirecting to http://tools.wmflabs.org/xtools/articleinfo/?
[19:02:37] <YuviPanda>	 yeah
[19:02:38] <YuviPanda>	 it is
[19:02:59] <MusikAnimal>	 need update that redirect and you're good I think
[19:03:46] <Cyberpower678>	 Which should be redirecting to https://tool.wmflabs.org/xtools-articleinfo/
[19:03:52] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0]  
[19:04:12] <Cyberpower678>	 YuviPanda, are you a projectadmin on tools
[19:04:22] <YuviPanda>	 I am
[19:04:30] <YuviPanda>	 but we don’t have a process for taking over tools yet, sadly
[19:04:31] <Cyberpower678>	 Can you add me to Wikiviewstats?
[19:04:37] <YuviPanda>	 I’ll work with Coren today to figure that out?
[19:04:40] <YuviPanda>	 I know it’s frustrating, sorry
[19:04:52] <Cyberpower678>	 Coren said he was going to 21 days ago.
[19:04:59] <MusikAnimal>	 Cyberpower678: any idea where the page history template lives?
[19:05:00] <Cyberpower678>	 But apparently forgot.
[19:05:13] <YuviPanda>	 Cyberpower678: I’ll poke him and luis (legal) today
[19:05:22] <MusikAnimal>	 actually found it
[19:05:24] <MusikAnimal>	 MediaWiki:Pageinfo-footer
[19:05:25] <Cyberpower678>	 YuviPanda, thanks.
[19:05:43] <Cyberpower678>	 MusikAnimal, MediaWiki:Pageinfo-footer
[19:06:19] <MusikAnimal>	 actually that's not it
[19:06:29] <Cyberpower678>	 Oh wait.
[19:06:30] <MusikAnimal>	 I don't think that one is being used anymore
[19:07:29] <Cyberpower678>	 MusikAnimal, MediaWiki:Histlegend
[19:07:42] <MusikAnimal>	 there we go
[19:08:21] <MusikAnimal>	 updated
[19:08:30] <MusikAnimal>	 awesome stuff! things are moving along well, glad to see it
[19:08:57] <MusikAnimal>	 I suppose having all the tools fragmented like that is less than ideal, but it sure is having a grand effect on performance
[19:10:15] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK  
[19:10:31] <Cyberpower678>	 I'm keeping the bulk stored in xtools, for maintainability.  As long as the branched webservice is the one loading this.
[19:10:46] <MusikAnimal>	 I think you got the two big ones moved out
[19:10:51] <YuviPanda>	 andrewbogott: do you know why we can’t make nova use something not dnsmasq for DNS? We could still use it for DHCP
[19:11:16] <MusikAnimal>	 Anyway, I've got to sign off for the weekend. Today's my birthday actually, heading to Puerto Rico!
[19:11:28] <Cyberpower678>	 MusikAnimal, happy birthday. 
[19:11:35] <MusikAnimal>	 thank you :)
[19:12:14] <Cyberpower678>	 MusikAnimal, one more request.  Maybe a site notice on enwiki alerting to the changed location of the edit counter.  There are likely several dozen links to the edit counter.
[19:12:52] <MusikAnimal>	 hmm, like the ones that show up at Special:Watchlist? Never done that before
[19:13:03] <Cyberpower678>	 Yep.
[19:13:35] <MusikAnimal>	 seems like we'd need consensus or something for that, no? or is this safe enough you think?
[19:13:53] <MusikAnimal>	 also what would you like it to say, the copy
[19:13:53] <MusikAnimal>	 I'm not that great with words
[19:14:30] <Cyberpower678>	 MusikAnimal, Hmm.  Considering that the entire site is affected by this change, I would deem it appropriate.  The edit counter is one of the top most used tools in existence.
[19:15:27] <Cyberpower678>	 YuviPanda, thoughts? ^
[19:15:36] <MusikAnimal>	 I'd have to agree
[19:16:06] <YuviPanda>	 this seems to be an onwiki thing, I don’t know enough about enwiki conventions to have an opinion, sorry
[19:16:35] <Cyberpower678>	 YuviPanda, tsk tsk. :p
[19:17:09] <MusikAnimal>	 Well unfortunately I don't have the time. I've got to get ready. Somebody will help I'm sure, and for the record you have my endorsement
[19:17:13] <MusikAnimal>	 Have a lovely weekend, all!
[19:22:23] <andrewbogott>	 YuviPanda: I think it's possible to use something else, but I haven't looked in to it much.
[19:22:33] <andrewbogott>	 Things are mostly fixed since you changed ttl, right?
[19:22:50] <YuviPanda>	 andrewbogott: nope, have been fucked up for the last couple of days
[19:22:52] <andrewbogott>	 oh, seems not :(
[19:23:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]  
[19:23:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[19:27:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[19:32:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[19:47:47] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[19:48:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[19:51:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[19:57:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0]  
[19:57:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[20:00:16] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]  
[20:12:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0]  
[20:13:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[20:21:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[20:25:19] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[20:31:26] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[20:43:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[20:56:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[20:59:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[21:01:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[21:08:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]  
[21:25:23] <wikibugs>	 3Wikimedia-Labs-General, translatewiki.net: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#991778 (10Nemo_bis) Well, knowing it came from a tool with a webservice would already be an improvement. There are only a few dozens or hundreds, as opposed to thousands us...
[21:28:54] <Nemo_bis>	 Does someone have ideas why a git clone from gitorious might be three times slower on a labs instance than on another? https://paste.debian.net/142043/
[21:29:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[21:31:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[21:33:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0]  
[21:34:44] <valhallasw`cloud>	 Nemo_bis: I see 44 sec vs 40s vs 33s in user time, and system time seems to be somewhat random (43s and 11s on the same host), so I'm not sure what is actually the relevant measurement
[21:35:10] <valhallasw`cloud>	 Nemo_bis: dumps-stats and ttmserver-mediawiki01 are different 'sizes' (m1.xlarge vs m1.medium)
[21:35:35] <valhallasw`cloud>	 which might have an effect on the allocated computing time
[21:42:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[21:43:09] <Nemo_bis>	 valhallasw`cloud: I'm talking of wall clock time; those clones were not CPU bound
[21:43:58] <valhallasw`cloud>	 Nemo_bis: ahh, sorry, I missed that number.
[21:45:22] <Nemo_bis>	 My fault, I should have said explicitly :)
[21:46:36] <valhallasw`cloud>	 Nemo_bis: strangely enough, the transfer time seems to be longer than the actual wall clock time for the last one (239.09 MiB / 450.00 KiB/s = 9 mins)
[21:47:14] <valhallasw`cloud>	 but I guess the transfer speed is just the last x kb
[21:48:34] <Nemo_bis>	 yes, and it oscillated considerably
[21:49:35] <Nemo_bis>	 I doubt network characteristics can cause such big differences within labs, so the most obvious candidate is IO. But it didn't look busy
[21:50:24] <Nemo_bis>	 Silly me, I should have tested on /data/scratch from both instances
[21:50:56] <wikibugs>	 3Tool-Labs: Installation of rlwrap to trusty.tools.wmflabs.org - https://phabricator.wikimedia.org/T87368#991866 (10Giftpflanze)
[21:52:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[21:55:59] <wikibugs>	 3Tool-Labs-tools-Erwin's-tools: Migrate https://toolserver.org/~erwin85/delete.php to Tool Labs - https://phabricator.wikimedia.org/T62877#991883 (10Vogone) Anyone working on this right now? I personally miss this tool quite a lot.
[22:02:32] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[22:11:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[22:18:16] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[22:19:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[22:21:10] <valhallasw`cloud>	 Nemo_bis: oh, that's the other thing I noticed, yes, different paths. One last thing to check might be the git version
[22:21:15] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0]  
[22:22:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[22:26:23] <Nemo_bis>	 1.9.1 in bot labs ones; and on lakka (the fastest) it's 1.7.10.4 
[22:28:15] <andrewbogott>	 !log publicdata rm'ing shared files
[22:28:22] <labs-morebots>	 Logged the message, dummy
[22:30:39] <wikibugs>	 3Wikidata, Tool-Labs: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#991957 (10Springle) Unsure if your #1 and #2 are both on labsdb replicas, or if #2 is some other db like tools-db or a custom instance.  If #1 and #2 are both connections to the labsdb production repli...
[22:33:38] <andrewbogott>	 YuviPanda: I built a medium sized jessie image and, predictably, puppet is working for me.  Did I misunderstand your issue?  instance  testlabs-test-debian-med
[22:34:33] <wikibugs>	 3Wikidata, Tool-Labs: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#991972 (10coren) It's pretty clear that any tool that aims for reliability necessarily //must// have the ability to reconnect if the connection is dropped between transactions.  It's not clear that val...
[22:41:55] <shinken-wm>	 RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0]  
[22:42:49] <andrewbogott>	 !log publicdata deleting project
[22:42:52] <labs-morebots>	 Logged the message, dummy
[22:43:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[22:44:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[22:45:01] <T13|mobile>	 O.o labs-morebots us trolling now?
[22:46:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[22:46:23] <Mjbmr>	 what the hell?
[22:58:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[22:58:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]  
[23:03:50] <andrewbogott>	 YuviPanda: https://wikitech.wikimedia.org/wiki/OpenStack#Building_new_images
[23:04:35] <YuviPanda>	 andrewbogott: thanks!
[23:06:59] <andrewbogott>	 YuviPanda: nope :(
[23:10:07] <wikibugs>	 3Tool-Labs: Installation of rlwrap to trusty.tools.wmflabs.org - https://phabricator.wikimedia.org/T87368#992048 (10yuvipanda) 5Open>3Resolved a:3yuvipanda
[23:14:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[23:23:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[23:28:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[23:29:18] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[23:39:21] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[23:51:15] <wikibugs>	 3Wikimedia-Labs-wikistats: Add 300 000 wikia wikis to stats table - https://phabricator.wikimedia.org/T38291#992295 (10Dzahn) @Nemo_bis it might take a couple weeks to run that update. we might first have to change the update script to do multiple requests at once. maybe using curl_multi ?  http://php.net/manual...
[23:54:21] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[23:59:32] <wikibugs>	 3Wikimedia-Labs-wikistats: Add 300 000 wikia wikis to stats table - https://phabricator.wikimedia.org/T38291#992344 (10Nemo_bis) >>! In T38291#992295, @Dzahn wrote: > @Nemo_bis it might take a couple weeks to run that update. we might first have to change the update script to do multiple requests at once. maybe...