[00:23:31] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [00:34:33] 3Labs-Team: SMTP server on toolserver.org not responding - https://phabricator.wikimedia.org/T87119#984254 (10scfc) 3NEW [00:39:35] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [02:24:33] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [02:55:36] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:00:52] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:45:29] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [04:16:33] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:39:23] 3Wikimedia-Labs-Infrastructure: "Stale file handle" for /public/dumps/ - https://phabricator.wikimedia.org/T87013#984428 (10scfc) This also applies to tools-exec-10 (but apparently not any other Tools hosts): ``` scfc@tools-exec-10:~$ sudo puppet agent -tv Info: Retrieving plugin Info: Loading facts in /var/lib... [04:42:45] 3Tool-Labs: Install valgrind on bastions - https://phabricator.wikimedia.org/T87117#984436 (10scfc) p:5Triage>3Normal a:3scfc [04:46:31] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [05:17:51] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:17:51] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:32:38] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:32:42] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:32:44] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:32:48] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:32:48] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:32:49] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:32:50] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:35:05] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:36:17] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [05:36:24] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:37:34] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [06:45:00] PROBLEM - Puppet staleness on tools-redis is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [07:32:39] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:43] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:45] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:47] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:47] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:49] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:52:55] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:36:21] wikitech down :/ [09:12:42] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [09:13:34] Steinsplitter: works again? [09:14:01] yes [09:15:48] yeah I fixed it [09:16:19] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [09:16:23] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [09:22:33] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [09:22:53] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [09:22:54] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [09:25:08] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [09:27:58] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [10:39:29] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:59:20] 3Wikimedia-Labs-Infrastructure: "Stale file handle" for /public/dumps/ - https://phabricator.wikimedia.org/T87013#984580 (10Nemo_bis) @Hydriz had to reboot the instance to see the directory again. [11:24:28] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [11:56:31] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:26:32] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [12:37:28] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:02:34] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [13:18:34] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:43:35] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [14:08:32] hi, is there a replacement for toolserver.namespace on toolslabs? [14:11:46] Steinsplitter: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database [14:11:55] Steinsplitter: under 'metadata database' [14:12:40] thx [14:12:50] 3Tool-Labs: Provide wiki metadata in the databases similar to toolserver.wiki - https://phabricator.wikimedia.org/T50626#984636 (10valhallasw) [14:18:54] valhallasw`cloud: i can't find a lot of things there, but no table with namespaces? [14:19:07] Steinsplitter: no, you have to ask the api for that [14:19:23] sorry, missed that you were looking for namespaces [14:19:26] *sigh* ok :/ [14:19:51] Steinsplitter: the argument is that it's as fast to ask the api, and it's guaranteed to be up to date [14:20:22] and that joining on a namespace table slows stuff down [14:21:11] I still don't get why that would be the case, as that's basically what a database was built to do [14:22:18] valhallasw`cloud: the more joins that are made the more lookups that are required and thus the slower the performance [14:24:15] its ugly to work with api [14:29:04] Betacommand: and now the join is done in people's code. That just moves where the energy is spent. [14:29:53] valhallasw`cloud: true, but it takes the resources off the shared database hosts [14:30:00] that's true. [14:32:01] does a bug exists about the ns thing? [14:33:42] Steinsplitter: utfs :P https://phabricator.wikimedia.org/T50625 [14:34:13] apparently there is actually a toolserverdb_p now [14:37:55] yepp :) s51892_toolserverdb [14:39:30] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:41:59] MariaDB [s51892_toolserverdb]> SELECT * FROM namespace LIMIT 1; [14:42:00] ERROR 1142 (42000): SELECT command denied to user 'u2916'@'10.68.16.8' for table 'namespace' [14:42:04] meh! :P [14:47:36] 3Tool-Labs: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#984645 (10Steinsplitter) >>! In T50625#488761, @coren wrote: > Issues with the data should be reported to the toolserverdb tool maintainers (https://tools.wmflabs.org/?list#tool... [15:18:28] 3Wikidata, Tool-Labs: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#984664 (10Multichill) I noticed quite a few of my bots stopped working somewhere start of December last year. All crashed with "MySQL server has gone away". I'm pretty sure it's the 60 seconds idle lim... [15:18:40] 3Wikidata, Tool-Labs: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#984665 (10Multichill) p:5High>3Unbreak! [15:44:31] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:35] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:00:00] 3Tool-Labs: OSM: bw_mapnik renders highway=raceway and boundary=protected_area in original colors instead of greyscale - https://phabricator.wikimedia.org/T86932#984698 (10Aklapper) p:5Triage>3Volunteer? [16:00:19] 3Tool-Labs: WIWOSM not working in Wikipedias - https://phabricator.wikimedia.org/T87038#984701 (10Aklapper) p:5Triage>3High [16:26:31] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:17] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%) [17:32:14] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [18:06:28] is something up with labs? my connections lag extremely [18:15:21] my connection is fine [18:15:48] meh [18:17:31] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:02:33] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [19:18:28] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:43:33] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [19:59:35] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [20:24:31] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:20] For some reason I can not log in to the web interface at https://wikitech.wikimedia.org [20:37:27] is that a known problem? [20:56:32] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:21:30] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [21:21:58] Hi! I just have a very simple doubt. I want to do a SQL consultation to Tool Labs and it doesn't work because of syntax. Anyone could tell me how fix it? [21:22:15] select * from categorylinks cl_to='Persone_legate_all'Universit_degli_Studi_"Guglielmo_Marconi"'; [21:23:17] the problem is that I have these "" around Guglielmo_Marconi. [21:57:32] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:22:35] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0]