[00:51:47] (03CR) 10Dzahn: [C: 031] Support more approved wikimedia cloaks to run the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320931 (owner: 10Paladox) [01:42:58] 10Tool-Labs-tools-Pageviews: Add ability to see pageviews of pages that translcude files in a given category - https://phabricator.wikimedia.org/T150507#2788161 (10MusikAnimal) [01:50:39] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:03:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rtrust was created, changed by Rtrust link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Rtrust edit summary: Created page with "{{Tools Access Request |Justification=I plan to use the Tools project to allow users to be able to have statistics based on their contributions. |Completed=false |User Name=Rt..." [02:30:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:23] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:00:22] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:48:00] 06Labs, 06Operations, 13Patch-For-Review: 4.4-series kernel vs. iptables - https://phabricator.wikimedia.org/T142388#2788708 (10MoritzMuehlenhoff) 05Open>03Resolved This has been fixed, all labvirt systems are running Linux 4.4 for a while now. [12:50:13] 06Labs, 07LDAP, 13Patch-For-Review: Restore ldaplist -l passwd - https://phabricator.wikimedia.org/T122595#2788715 (10MoritzMuehlenhoff) a:05MoritzMuehlenhoff>03None [14:25:54] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2788886 (10chasemp) Are these https://gerrit.wikimedia.org/r/#/c/320804/2/modules/role/templates/labsdb/maintain-views.yam... [16:43:22] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:43:48] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:44:10] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:44:16] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:44:42] PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:45:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:45:19] PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:45:43] PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:45:54] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:45:58] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:45:58] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:46:14] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:46:24] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:46:58] PROBLEM - Puppet run on tools-exec-1418 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:46:59] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:47:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:47:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:47:21] PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:47:31] PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:47:45] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:47:53] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:47:55] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:47:57] PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:48:09] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:48:17] PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:48:26] PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:48:28] PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [16:48:40] PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:49:06] PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:49:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:49:54] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:50:01] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:50:15] PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:51:13] PROBLEM - Puppet run on tools-exec-1420 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:52:09] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:52:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:52:17] PROBLEM - Puppet run on tools-exec-1207 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:52:26] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:52:26] PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:52:31] yeah so that's clearly me ^ :) [16:52:33] fix incoming [16:52:42] PROBLEM - Puppet run on tools-exec-1209 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:53:39] 06Labs, 10Labs-Infrastructure, 10DBA, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2789064 (10chasemp) Where I'm at after thinking on it for a bit: * We have the ecosystem to effectively standup HAProxy here (monitoring, operating... [16:54:10] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:54:12] PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:54:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:54:36] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:54:40] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:54:47] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:54:59] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:55:23] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:55:39] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:55:41] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [16:55:48] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:55:54] PROBLEM - Puppet run on tools-worker-1005 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:56:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:56:07] PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:57:01] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:57:09] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:57:37] PROBLEM - Puppet run on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:57:43] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:58:13] PROBLEM - Puppet run on tools-exec-1205 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:58:17] PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:58:29] PROBLEM - Puppet run on tools-static-10 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:59:01] PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:59:31] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:00:17] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:00:22] PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:00:24] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:00:30] PROBLEM - Puppet run on tools-exec-1204 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:00:40] PROBLEM - Puppet run on tools-worker-1022 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:00:50] PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:00:54] PROBLEM - Puppet run on tools-exec-1211 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:01:02] PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:01:10] PROBLEM - Puppet run on tools-exec-1419 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:01:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:01:16] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:01:20] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:02:07] in theory this starts recovering pretty soon here [17:02:50] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:03:00] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:03:10] PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:04:28] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:04:55] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:04:59] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:09:32] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154#2789074 (10madhuvishy) [17:10:05] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154#2652289 (10madhuvishy) [17:10:15] i iz broke teh wikiz [17:12:46] That fix anytime soon? [17:13:15] should wikitech have village stocks? [17:16:45] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154#2789082 (10madhuvishy) [17:20:42] RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [17:21:58] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:14] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:20] RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:28] RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:40] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:56] 06Labs, 10Horizon: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789088 (10Matthewrbowker) [17:23:22] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:24] RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:28] RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:50] 06Labs, 10Horizon, 07Upstream: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789102 (10Krenair) Should be filed on Launchpad, not Wikimedia Phabricator [17:23:50] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:17] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:36] 06Labs, 10Horizon, 07Upstream: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789106 (10Matthewrbowker) Huh, I thought Horizon was developed in-house. Do you have a link to this "launchpad"? [17:24:41] RECOVERY - Puppet run on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:17] hooray [17:25:19] RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:53] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:59] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:59] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:13] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:14] RECOVERY - Puppet run on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:23] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:45] 06Labs, 10Horizon, 07Upstream: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789108 (10Krenair) No, Horizon is an Openstack project, not Wikimedia. We have some code to extend it a bit. It's just called 'horizon' there: https://launchpad.net... [17:26:57] RECOVERY - Puppet run on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:10] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:52] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:52] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:58] RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:12] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:20] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:40] RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:06] RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:12] RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:18] 06Labs, 10Horizon, 07Upstream: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789112 (10Krenair) I changed the #Horizon description to make this clear [17:29:56] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:03] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:03] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:11] 06Labs, 10Horizon, 07Upstream: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789113 (10Matthewrbowker) I learned something new, thank you for the information. This bug was fixed back in June, in Horizon version 10.0.0.0b1. See: https://bug... [17:30:15] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:31:09] RECOVERY - Puppet run on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:17] RECOVERY - Puppet run on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:24] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:26] RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:40] 06Labs, 10Horizon, 07Upstream: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789088 (10Krenair) Looks like that is included in Newton: https://releases.openstack.org/newton/index.html [17:32:42] RECOVERY - Puppet run on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [17:33:14] RECOVERY - Puppet run on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [17:33:14] RECOVERY - Puppet run on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [17:33:18] 06Labs: (eventually) Upgrade Labs to OpenStack Newton - https://phabricator.wikimedia.org/T145920#2789118 (10Krenair) [17:33:19] 06Labs, 10Horizon, 07Upstream: Mark "Description" as a required field when creating security groups - https://phabricator.wikimedia.org/T150543#2789117 (10Krenair) [17:33:30] RECOVERY - Puppet run on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:02] RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:32] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:36] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:39] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:47] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:19] RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:23] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:33] RECOVERY - Puppet run on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:39] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:39] RECOVERY - Puppet run on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:41] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:47] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:57] RECOVERY - Puppet run on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:09] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:10] RECOVERY - Puppet run on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:18] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:22] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:02] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:12] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:36] RECOVERY - Puppet run on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:36] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154#2789124 (10chasemp) [17:37:44] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:12] RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:30] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:33] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:58] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:23] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:51] RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:55] RECOVERY - Puppet run on tools-exec-1211 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:03] RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:47] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:43:01] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [17:44:56] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [17:46:08] Hey, yuvipanda when you're around (I know you may be on holiday) can I speak with you about T53434 ? [17:46:08] T53434: Implement a system to montior tools on tool-labs - https://phabricator.wikimedia.org/T53434 [18:13:00] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Install libenchant1c2a in the toollabs-python2-base docker image - https://phabricator.wikimedia.org/T143449#2789164 (10tom29739) The interim solution of using a compiled wheel of pyenchant no longer works because the wheel at http://tools-docker-builder-01.eqiad.wmfla... [18:18:43] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2789166 (10AlexMonk-WMF) >>! In T148561#2788886, @chasemp wrote: > Are these https://gerrit.wikimedia.org/r/#/c/320804/2/m... [19:13:46] 06Labs, 10Labs-Infrastructure, 10DBA, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2789262 (10chasemp) sure, thanks, I understand. I meant something more like: is this a table to be exposed on a single wi... [19:29:04] !log tools Disabling puppet across tools to dual mount tools share from labstore-secondary T146154 [19:29:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:29:08] T146154: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154 [19:37:08] 06Labs, 10DBA: Reduce watchlist_count threshold - https://phabricator.wikimedia.org/T150548#2789278 (10Dispenser) [19:41:58] (03PS8) 10Paladox: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294 [19:57:12] Hey, another maintainer started the task. How do I stop it? It tells me I do not have access. [19:57:20] It is editing while logged out [20:07:34] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:14:48] ^ working on it [20:15:39] PROBLEM - Host tools-puppetmaster-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.61) [20:18:02] !log tools Rolling out dual mount of tools share across all hosts T146154 [20:18:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:18:06] T146154: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154 [20:23:11] (03PS1) 10Paladox: Fix Cannot read property 'match' of undefined [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321020 [20:24:17] (03CR) 10Zppix: [C: 032] Fix Cannot read property 'match' of undefined [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321020 (owner: 10Paladox) [20:24:57] 06Labs, 10Labs-Infrastructure, 05Continuous-Integration-Scaling: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2789326 (10hashar) [20:25:10] (03Merged) 10jenkins-bot: Fix Cannot read property 'match' of undefined [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321020 (owner: 10Paladox) [20:27:33] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [20:39:16] Krenair? [20:49:18] !log tools Dual mount of tools share complete. Puppet reenabled across tools hosts. T146154 [20:49:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:49:23] T146154: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] - https://phabricator.wikimedia.org/T146154 [21:10:15] (03PS1) 10Paladox: Send the irc messages to all irc channels if you restart or force-restart [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321025 [21:11:25] (03CR) 10Paladox: [C: 031] "I tested this on the test bot and it worked :)" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321025 (owner: 10Paladox) [21:12:14] (03PS2) 10Paladox: Support more approved wikimedia cloaks to run the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320931 [21:24:32] (03CR) 10Zppix: [C: 032] Support more approved wikimedia cloaks to run the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320931 (owner: 10Paladox) [21:24:35] (03CR) 10Zppix: [C: 032] Send the irc messages to all irc channels if you restart or force-restart [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321025 (owner: 10Paladox) [21:24:58] (03Merged) 10jenkins-bot: Support more approved wikimedia cloaks to run the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320931 (owner: 10Paladox) [21:25:12] (03Merged) 10jenkins-bot: Send the irc messages to all irc channels if you restart or force-restart [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321025 (owner: 10Paladox) [21:53:35] 06Labs, 10Labs-Infrastructure, 10DBA, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2766141 (10bd808) Having to keep two authentication systems in sync seems like something that will break and consume a lot of operational debugging t... [22:20:59] 06Labs, 10DBA: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789420 (10fgiunchedi) [22:59:30] 06Labs, 06Operations, 10wikitech.wikimedia.org: Can't login wikitech - https://phabricator.wikimedia.org/T144805#2610832 (10Shoichi) Months ago, I can log in,but it also happen to me. I don't know rember I had set two-factor authentication or not. Can someone help me? >_< My account there is also "shoichi" [23:01:56] (03PS1) 10Paladox: Revert "Send the irc messages to all irc channels if you restart or force-restart" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321085 [23:02:02] (03CR) 10Paladox: [C: 032] Revert "Send the irc messages to all irc channels if you restart or force-restart" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321085 (owner: 10Paladox) [23:02:43] (03Merged) 10jenkins-bot: Revert "Send the irc messages to all irc channels if you restart or force-restart" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321085 (owner: 10Paladox) [23:03:25] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 - https://phabricator.wikimedia.org/T149870#2789503 (10chasemp) Two things we have noticed that need to happen to close this: 1) we need to use flock or something to ensure one job... [23:08:33] (03CR) 10Paladox: "recheck" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321085 (owner: 10Paladox) [23:16:19] 06Labs, 10Tool-Labs-tools-Other, 10DBA: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789559 (10Volans) Just to clarify, this is because of row-based replication, for each 100MB relay log there was ~93k updated rows for the table `s51370__isbntool.nr... [23:18:11] (03CR) 10Paladox: "recheck" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321085 (owner: 10Paladox) [23:22:36] (03CR) 10Paladox: "recheck" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/321085 (owner: 10Paladox) [23:23:51] 06Labs, 10Tool-Labs-tools-Other, 10DBA: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789567 (10Volans) To the `isbn` tool group: Looks like you have some issue in the queries you're performing because I can see that you're updating always the same ~... [23:54:26] if anyone knows how to locate the user responsible for ^ that'd be appreciated! [23:56:51] i do [23:56:59] not [23:57:28] godog: you want contact info for the isbn tool? [23:57:41] godog qchris__ ? [23:58:13] https://tools.wmflabs.org/?tool=isbn -- "G" and "Ireas"