[00:29:11] (03PS1) 10Jean-Frédéric: Refactor extractWikilink [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/240937 [00:29:40] (03CR) 10Jean-Frédéric: [C: 032] Refactor extractWikilink [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/240937 (owner: 10Jean-Frédéric) [00:43:30] (03Merged) 10jenkins-bot: Refactor extractWikilink [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/240937 (owner: 10Jean-Frédéric) [03:07:24] gifti: Potentially, unless you prevent them from being downloadable. You may want to set the ulimit to prevent core dumps unless you are actively debugging. [07:08:37] 6Labs, 10Labs-Infrastructure: Redirect https://toolserver.org/~magnus/ - https://phabricator.wikimedia.org/T113696#1673644 (10Nemo_bis) 3NEW [08:34:02] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/240981 [08:35:19] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/240981 (owner: 10Hashar) [09:08:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mabandalone was created, changed by Mabandalone link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Mabandalone edit summary: Created page with "{{Tools Access Request |Justification=I want to make some improvements to the xtools gadget (adding more info to bar like top 10 authors) |Completed=false |User Name=Mabandalo..." [09:50:58] (03PS1) 10Hashar: Add py34 to tox envlist [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/241006 [09:51:45] (03CR) 10Jforrester: [C: 032] Add py34 to tox envlist [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/241006 (owner: 10Hashar) [09:51:57] (03Merged) 10jenkins-bot: Add py34 to tox envlist [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/241006 (owner: 10Hashar) [09:52:00] \O/ [09:53:59] hashar: :-) [09:54:13] James_F: and that job is running on Nodepool instances! [09:54:15] hashar: I have py35 and not py34 locally, so… [09:54:22] James_F: are you back in London or just not sleeping ? [09:54:36] hashar: I am back in London for the next two days. :-) [10:03:27] 10Tool-Labs-tools-Other, 6Commons, 6Community-Tech: [AOI] Create a new DerivativeFX after the Toolserver shutdown - https://phabricator.wikimedia.org/T110409#1674021 (10El_Grafo) [10:03:30] 10Tool-Labs-tools-Other, 7Tracking: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#1674020 (10El_Grafo) [11:30:47] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 896193 bytes in 2.278 second response time [11:32:11] <_joe_> up, up! [11:32:12] <_joe_> :) [11:32:36] thx ;) [11:33:11] PROBLEM - Puppet failure on tools-precise-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:33:18] PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:34:18] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:34:18] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:22] PROBLEM - Puppet failure on tools-k8s-bastion-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:22] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:35:22] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:23] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:23] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:30] PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:30] PROBLEM - Puppet failure on tools-exec-1205 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:31] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:31] PROBLEM - Puppet failure on tools-mailrelay-02 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0] [11:35:31] PROBLEM - Puppet failure on tools-packages is CRITICAL: CRITICAL: 83.33% of data above the critical threshold [0.0] [11:35:35] PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:35] PROBLEM - Puppet failure on tools-services-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:40] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:41] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:41] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:48] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:48] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:49] PROBLEM - Puppet failure on tools-web-static-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:49] PROBLEM - Puppet failure on tools-web-static-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:51] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:51] PROBLEM - Puppet failure on tools-checker-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:51] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:52] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:55] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:55] PROBLEM - Puppet failure on tools-exec-1221 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:35:56] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:56] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:35:59] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:01] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:02] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:02] PROBLEM - Puppet failure on tools-exec-1203 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:02] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:04] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:36:08] PROBLEM - Puppet failure on tools-exec-1408 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:08] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:08] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:09] PROBLEM - Puppet failure on tools-exec-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:10] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:16] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:17] PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:17] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:36:56] PROBLEM - Puppet failure on tools-redis-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:07] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:07] PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:09] PROBLEM - Puppet failure on tools-worker-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:21] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:27] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:27] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:37] PROBLEM - Puppet failure on tools-exec-1401 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:37] PROBLEM - Puppet failure on tools-k8s-master-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:38] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:39] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:43] PROBLEM - Puppet failure on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:44] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:51] PROBLEM - Puppet failure on tools-services-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:53] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [11:37:56] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:58] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:04] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:04] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:10] PROBLEM - Puppet failure on tools-redis-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:10] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:10] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:14] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:14] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:14] PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:16] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:20] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:20] PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:38:21] PROBLEM - Puppet failure on tools-exec-1220 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:47:00] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mabandalone was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=184806 edit summary: [11:49:15] RECOVERY - Puppet failure on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:51] RECOVERY - Puppet failure on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:20] RECOVERY - Puppet failure on tools-k8s-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:24] RECOVERY - Puppet failure on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:51] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:41] RECOVERY - Puppet failure on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [12:04:01] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:04:19] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [12:05:26] RECOVERY - Puppet failure on tools-packages is OK: OK: Less than 1.00% above the threshold [0.0] [12:11:05] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [12:16:27] PROBLEM - Puppet failure on tools-webproxy-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:21:52] 6Labs, 10Tool-Labs-tools-meetbot, 10Labs-Infrastructure: restore Meetbot logs from around 2015-06 lost in NFS outage - https://phabricator.wikimedia.org/T113000#1674276 (10TTO) [12:26:46] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:29:01] yuvipanda: good morning, are you aware of any ongoing outage on labs / openstack ? [12:32:10] hashar: there was a network outage but it should be resolved now [12:32:19] ah thanks [12:32:25] might explain some oddities I have seen in the log [12:33:44] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:35:14] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [12:35:22] oh [12:35:23] OperationalError: (OperationalError) (2006, "MySQL server has gone away (error(32, 'Broken pipe'))") None None [12:40:16] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:41:16] PROBLEM - Puppet failure on tools-k8s-bastion-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:41:22] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:41:24] PROBLEM - Puppet failure on tools-exec-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:41:26] PROBLEM - Puppet failure on tools-packages is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:41:49] PROBLEM - Puppet failure on tools-checker-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:41:50] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:42:07] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:47:50] RECOVERY - Puppet failure on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:51:50] RECOVERY - Puppet failure on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:51:56] RECOVERY - Puppet failure on tools-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:55:47] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [12:56:23] RECOVERY - Puppet failure on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [12:56:23] RECOVERY - Puppet failure on tools-k8s-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:57:05] RECOVERY - Puppet failure on tools-worker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:02:28] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [13:02:28] RECOVERY - Puppet failure on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [13:03:45] RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:18] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:20] RECOVERY - Puppet failure on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:20] RECOVERY - Puppet failure on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:20] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:28] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:28] RECOVERY - Puppet failure on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:38] RECOVERY - Puppet failure on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:42] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:48] RECOVERY - Puppet failure on tools-web-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:52] RECOVERY - Puppet failure on tools-web-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:54] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:54] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:00] RECOVERY - Puppet failure on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:03] RECOVERY - Puppet failure on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:07] RECOVERY - Puppet failure on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:11] RECOVERY - Puppet failure on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:11] RECOVERY - Puppet failure on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:14] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:27] RECOVERY - Puppet failure on tools-webproxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:29] RECOVERY - Puppet failure on tools-packages is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:43] RECOVERY - Puppet failure on tools-webproxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:06:50] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [13:07:07] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:00] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:02] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:10] RECOVERY - Puppet failure on tools-redis-02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:10] RECOVERY - Puppet failure on tools-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:12] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:18] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [13:10:18] RECOVERY - Puppet failure on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [13:10:26] RECOVERY - Puppet failure on tools-mailrelay-02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:10:52] RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [13:11:04] RECOVERY - Puppet failure on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [13:11:26] RECOVERY - Puppet failure on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [13:12:08] RECOVERY - Puppet failure on tools-exec-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [13:12:38] RECOVERY - Puppet failure on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [13:12:42] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [13:12:52] RECOVERY - Puppet failure on tools-exec-1211 is OK: OK: Less than 1.00% above the threshold [0.0] [13:14:00] RECOVERY - Puppet failure on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [13:15:22] RECOVERY - Puppet failure on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [13:15:36] RECOVERY - Puppet failure on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:15:50] RECOVERY - Puppet failure on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [13:15:58] RECOVERY - Puppet failure on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [13:18:12] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [13:18:12] RECOVERY - Puppet failure on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [13:18:14] RECOVERY - Puppet failure on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [13:20:38] RECOVERY - Puppet failure on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [13:22:06] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [13:22:25] Coren: I’m still getting a torrent of diamond complains; do you know where those are running or what the deal is? [13:22:36] RECOVERY - Puppet failure on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:22:37] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [13:23:06] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [13:23:10] andrewbogott: Which ones? [13:23:21] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [13:23:28] Coren: you don’t have 1000 emails from diamond in your inbox? [13:23:48] here’s one: "tools-services-02 : Sep 25 11:25:00 : diamond : unable to resolve host tools-services-02" [13:23:58] So presumably that isn’t running /on/ tools-services-02... [13:25:40] RECOVERY - Puppet failure on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [13:25:44] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [13:25:47] I haven't seen any since 1h ago, roughly. [13:25:55] They stopped at 8:25 my time [13:25:56] RECOVERY - Puppet failure on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [13:26:02] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [13:26:06] hm [13:26:08] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [13:26:14] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [13:26:14] RECOVERY - Puppet failure on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [13:26:29] Oh, wait, I'm still getting a couple - but not very fast [13:26:39] I’m still getting them but maybe it’s a delayed effect, since their timestamp is from a couple of hours ago [13:27:12] Ah - your mail might just have gotten flooded/clogged and they are being throttled. gmail? [13:27:24] andrewbogott: the poor openstack is unable to delete instances in tenant contintcloud :\ [13:27:29] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [13:27:29] yeah, that might be it [13:27:36] hashar: I’m working on it still [13:27:42] RECOVERY - Puppet failure on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [13:27:45] ok ok [13:28:15] andrewbogott: if it can help, Horizon shows them in the 'deleting' state but that is not recorded in the instances action logs [13:28:18] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [13:28:19] RECOVERY - Puppet failure on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [13:31:05] RECOVERY - Puppet failure on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [13:33:08] RECOVERY - Puppet failure on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [13:33:12] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [13:33:14] RECOVERY - Puppet failure on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [13:33:18] RECOVERY - Puppet failure on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [14:14:17] 6Labs, 10wikitech.wikimedia.org: Stop using an older release of SMW and get back to tracking their master branch - https://phabricator.wikimedia.org/T75940#1674568 (10JanZerebecki) [14:24:39] 10Tool-Labs-tools-Other: [AG] [Bug] Internet Explorer: Wron height of input field - https://phabricator.wikimedia.org/T113590#1674602 (10Aklapper) [15:17:30] 6Labs, 10Tool-Labs: Remove dependency of toollabs::checker on toollabs::submit and shut down bigbrother on tools-checker-01/tools-checker-02 - https://phabricator.wikimedia.org/T113744#1674775 (10scfc) 3NEW a:3scfc [15:41:28] (03PS1) 10Addshore: Report analytics/limn-wikidata-data to wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/241060 [16:16:42] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:44:49] 6Labs, 10Tool-Labs: SMTP service on toolserver.org down - https://phabricator.wikimedia.org/T113756#1675114 (10scfc) 3NEW [16:56:40] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:00] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-114: Ironic on Labs - https://phabricator.wikimedia.org/T110556#1675454 (10Andrew) https://etherpad.wikimedia.org/p/labs-ironic [17:40:07] On wikitech, https://wikitech.wikimedia.org/wiki/Special:UserLogin/signup has no title. Why did Ryan Lane blank the message https://wikitech.wikimedia.org/wiki/MediaWiki:Createaccount in March 2013 [17:47:22] yuvipanda: can haz another OAuth approval? https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/c683af90c0ed69e3a4dc855eaa52d9e9 [17:47:37] ragesosss: am in a meeting can do once done [17:48:09] (I got my ssh config working yesterday, btw. For some reason, on Debian it didn't like applying the wildcard, so I put all the options into the same stanza for each host) [17:48:16] yuvipanda: thanks. no rush. [18:04:56] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 3Labs-Sprint-115: Can't delete rule in default security group - https://phabricator.wikimedia.org/T112492#1675605 (10jcrespo) Done. [18:05:25] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 3Labs-Sprint-115: Can't delete rule in default security group - https://phabricator.wikimedia.org/T112492#1675606 (10Andrew) 5Open>3Resolved and fixed! [18:07:06] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1675620 (10yuvipanda) Ok, so this requires: # Port 80 on nobelium be available from labs instances, for instances to be able to query this through the proxy. # Port 9200 on nobel... [18:15:14] yuvipanda, andrewbogott : FYI I improved https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Quick_start , https://wikitech.wikimedia.org/wiki/Help:Tool_Labs?type=revision&diff=185040&oldid=175503 [18:15:57] thanks, spagewmf! [18:22:25] andrewbogott: np. It says "create your Labs wiki account (you will get Bastion SSH access after that)." What does that mean, is it relevant? AIUI you don't need to be in the bastion group to have a tool labs account or to `ssh login.tools.wmflabs`, so can I remove this confusing "WTF is a bastion!?!" aside? [18:24:16] spagewmf: yes please do [18:25:05] Didn't that used to be relevant? [18:25:12] bastion group membership was separate? [18:27:06] didn't matter for toollabs users anyway [18:27:12] since they are logging in through tools.wmflabs.org [18:32:47] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 3Labs-Sprint-115: Can't delete security group rules after OpenStack upgrade - https://phabricator.wikimedia.org/T112492#1675763 (10Krenair) [18:35:54] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 3Labs-Sprint-115: Can't delete security group rules after OpenStack upgrade - https://phabricator.wikimedia.org/T112492#1675767 (10Krenair) >Successfully removed rule. Thanks @Andrew and @Jcrespo! [18:36:38] spagewmf: yeah, ok to remove that bit [18:38:15] andrewbogott: done. BTW you're not listed in https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Contact [18:38:49] spagewmf: I think that’s ok with me for now, I’m not the best with the tools-specific stuff [18:51:46] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-114: Ironic on Labs - https://phabricator.wikimedia.org/T110556#1675821 (10chasemp) p:5Triage>3Normal [19:00:22] 6Labs, 6Discovery, 7Elasticsearch: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1675840 (10EBernhardson) If we want to keep things limited, it should be safe to only allow jobrunners + terbium. Writes are always processed in a job. I would like terbium to al... [19:07:51] Krenair: FWIW 36 tools members are not bastion members, according to Special:NovaProject. You can have one without the other [19:08:05] oh, right [19:08:15] because tools is a special snowflake project that gets it's own bastion, right? [19:09:49] spagewmf: what that's weird [19:09:54] and shouldn't be happening [19:10:03] unless it's an artifact of something from a long time ago [19:11:09] yuvipanda: why is it weird? Do you need to be a member of bastion to use Tool Labs? [19:11:48] no but 1. you need to have been granted the 'shell' right to login and 2. granting the 'shell' right automatically makes you a member of bastion [19:12:33] it didn't always automatically make you a member of bastion though, did it [19:12:33] ? [19:12:49] shell right always did [19:13:01] we just did away with needing the shell right to be manually granted [19:13:03] yuvipanda: OK. Here are the 36. https://phabricator.wikimedia.org/P2095 , apologies if I goofed in my shell-fu [19:13:12] well, 'always' as in as long as I can remember [19:13:51] maybe it's the more recent ones rather than the old ones? [19:14:11] or do they also not have shell rights? [19:15:26] these look older [20:55:59] for doing meta query on meta_p.wiki, selecting all wikis with family=wikipedia and lang=en I see some test wikis.is there a nice flag in the database that tell it is not a "real" wiki but a test wiki? [21:03:16] Well, there aren't many test wikis [21:03:21] testwiki, test2wiki and testwikidatawiki [21:05:53] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-114: Ironic on Labs - https://phabricator.wikimedia.org/T110556#1676539 (10chasemp) >>! In T110556#1675454, @Andrew wrote: > https://etherpad.wikimedia.org/p/labs-ironic Just because I don't trust etherpad ```= Links = * http://lists.openstack.org/pipermail/opens... [21:15:10] PROBLEM - Puppet staleness on tools-worker-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [21:39:16] is there a index on user_properties_anon.up_property? It takes forever for querying it on enwiki_p (select up_property, count(*) from user_properties_anon where up_property like 'gadget-%' and up_value=1 group by up_property ) [21:40:24] (finally finished...) [21:56:47] eranroz, nope - it's a BLOB so you can't fully index it [23:18:54] (03Abandoned) 10Awight: Write the "blacklist" config variable as an array [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/119924 (owner: 10Awight) [23:54:27] yuvipanda, around? [23:56:03] Krenair: kind of [23:56:07] Just heading to the office [23:56:52] yuvipanda, I was wondering where would be the best place to put test instances of icinga and shinken