[00:16:15] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2874948 (10Ykhwong) Sorry, but I still don't see the ko_KR.EUC-KR on the login.tools.wmflabs.org. [00:59:42] yuvipanda: Union does work, and wrote a little tool that expands it - example https://quarry.wmflabs.org/query/5530 and https://quarry.wmflabs.org/query/5531 - thoughts :) ? [01:00:47] only problem is that it seems limited to ~250-300 wikis, so can do all wikipedias but not all wikis. Presumably there is a limit somewhere to how long the sql text is allowed to be [01:13:48] paladox: sorry by the time you pinged me I already went off [01:22:39] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:45:28] PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:57:43] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:03:48] PROBLEM - Free space - all mounts on tools-worker-1017 is CRITICAL: CRITICAL: tools.tools-worker-1017.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found)tools.tools-worker-1017.diskspace.root.byte_percentfree (<100.00%) [02:05:27] 10Labs-project-Wikistats: all kinds of mixed issues with miraheze table (was: allthetropes is not updating on wikistats) - https://phabricator.wikimedia.org/T146712#2875124 (10NDKilla) @Dzahn MariaDB [metawiki]> select * from cw_wikis where wiki_dbname="reriawiki" or wiki_dbname="pnpwiki" or wiki_dbname="ronuos... [03:12:50] 06Labs, 10Labs-Infrastructure: labtestnet2001 - disk space - nova-api.log needs rotation - https://phabricator.wikimedia.org/T153279#2875187 (10Dzahn) [03:13:17] 06Labs, 10Labs-Infrastructure, 06Operations: labtestnet2001 - disk space - nova-api.log needs rotation - https://phabricator.wikimedia.org/T153279#2875202 (10Dzahn) [03:17:49] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2875217 (10Dzahn) ``` dzahn@tools-bastion-03:~$ sudo locale-gen ko_KR.EUC-KR Generating locales... ko_KR.EUC-KR... done Generation complete. dzahn@tools-bas... [03:33:15] 06Labs, 10Tool-Labs: webgrid-lighttpd queues kill OOM jobs with SIGKILL leaving php-cgi processes behind - https://phabricator.wikimedia.org/T153281#2875220 (10scfc) [03:38:36] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2875234 (10Ykhwong) Thank you, Dzahn. It's working fine now. I appreciate your quick support. [03:38:48] 06Labs, 10Tool-Labs: webgrid-lighttpd queues kill OOM jobs with SIGKILL leaving php-cgi processes behind - https://phabricator.wikimedia.org/T153281#2875235 (10scfc) [03:49:09] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2875238 (10Dzahn) 05Open>03Resolved @Ykhwong Glad to hear it works and i could help. I'm calling the ticket resolved then :) [04:22:47] 06Labs, 06Discovery, 06Operations, 03Interactive-Sprint, 06Maps (Maps-data): PostgreSQL query planner bug on labsdb1006 - https://phabricator.wikimedia.org/T145599#2875258 (10Yurik) [05:53:51] PROBLEM - Free space - all mounts on tools-exec-gift is CRITICAL: CRITICAL: tools.tools-exec-gift.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-gift.diskspace.root.byte_percentfree (<50.00%) [06:32:17] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:07:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:15] valhallasw`vecto: you might find https://bugzilla.redhat.com/show_bug.cgi?id=1404918 interesting [07:42:39] (03PS6) 10Hashar: Replace git.wikimedia.org links with diffusion [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) (owner: 10Paladox) [07:45:46] (03CR) 10Hashar: [C: 032] "No clue how it is updated on tools labs though." [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) (owner: 10Paladox) [07:53:00] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:53:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:28:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:33:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:24:06] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Alicia Fagerving (WMSE) was created, changed by Alicia Fagerving (WMSE) link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Alicia_Fagerving_(WMSE) edit summary: Created page with "{{Tools Access Request |Justification=I'm working on the Connected Open Heritage project, migrating data from the WLM database to Wikidata, and thus would like to get access t..." [09:26:03] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2875731 (10yuvipanda) From a 'requirements' pov - that is, things that we need to build/consider in our current en... [10:11:13] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2875862 (10yuvipanda) Operational concerns - these are things that will make the life of the people who are 'takin... [10:43:50] RECOVERY - Free space - all mounts on tools-exec-gift is OK: OK: tools.tools-exec-gift.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [10:45:53] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:07:04] 06Labs, 10Tool-Labs, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2876092 (10Urbanecm) [12:25:53] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [12:33:49] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2876135 (10yuvipanda) Community concerns - does the PaaS have a big enough community to sustain itself over time?... [13:09:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:10:30] How does one run a nodejs on kubectl? [13:21:00] (03CR) 10Zppix: [C: 032] Remove hacks, fix logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/323712 (owner: 10MarkTraceur) [13:25:08] (03Merged) 10jenkins-bot: Remove hacks, fix logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/323712 (owner: 10MarkTraceur) [13:29:45] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2328978 (10chasemp) A lot good thoughts here. I don't think either of the two contenders I know of can satisfy it... [13:39:07] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2876274 (10yuvipanda) @chasemp yup, added :) Yeah, I think we need to distill these into a set of criteria we can... [13:44:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:44:17] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:44:29] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:44:42] PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:44:44] PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:45:10] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:45:14] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:45:20] PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:45:52] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:45:58] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:46:02] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:46:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:46:41] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:46:54] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:47:21] PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:47:51] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:47:53] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:49:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:49:14] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:49:17] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:49:17] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:49:21] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:49:23] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:49:55] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:49:58] PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:49:59] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:50:05] PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:50:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:50:26] PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:50:28] PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:50:57] PROBLEM - Puppet run on tools-exec-1418 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:52:10] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:52:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:52:14] PROBLEM - Puppet run on tools-exec-1420 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:52:19] PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:52:33] PROBLEM - Puppet run on tools-docker-registry-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:53:02] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:53:15] PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:53:17] PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:53:40] PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:53:46] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:54:10] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:54:25] ^ we know [13:55:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:55:26] PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:55:39] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:55:39] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:56:03] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:56:55] PROBLEM - Puppet run on tools-worker-1005 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:57:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:57:23] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:57:32] PROBLEM - Puppet run on tools-static-10 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:57:43] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:57:48] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:58:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:58:16] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:58:37] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:58:39] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:59:00] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:59:10] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:59:12] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:59:20] PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:59:30] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:59:32] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:00:12] PROBLEM - Puppet run on tools-exec-1419 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:00:18] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:00:20] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:00:31] PROBLEM - Puppet run on tools-exec-1204 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:00:36] PROBLEM - Puppet run on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:00:40] PROBLEM - Puppet run on tools-worker-1022 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:01:02] PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:01:55] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:02:11] PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:02:20] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:02:21] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:02:29] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:02:31] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:02:41] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:02:47] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:02:50] PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:03:06] PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:03:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:03:35] PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:03:42] PROBLEM - Puppet run on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:03:56] PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:03:59] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [14:04:59] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:05:06] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:05:07] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:05:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:05:34] PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:05:42] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:05:43] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:05:43] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:05:51] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:55] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:06:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:06:47] PROBLEM - Puppet run on tools-worker-1025 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:07:13] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:08:09] PROBLEM - Puppet run on tools-worker-1023 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:08:30] PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:08:49] PROBLEM - Puppet run on tools-worker-1013 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:09:53] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:10:09] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:10:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:10:35] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:10:39] PROBLEM - Puppet run on tools-exec-1417 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:10:40] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:12:27] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:12:37] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:15:44] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:18:54] RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:20:40] RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:20:59] RECOVERY - Puppet run on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [14:21:53] RECOVERY - Puppet run on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:11] RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:21] RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:23] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:29] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:31] RECOVERY - Puppet run on tools-docker-registry-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:49] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:49] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:17] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:17] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:29] RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:35] RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:37] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:06] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:12] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:24] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:24] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:30] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:42] RECOVERY - Puppet run on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:44] RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:54] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:58] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:58] RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:06] RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:10] RECOVERY - Puppet run on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:14] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:22] RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:22] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:24] RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:26] RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:28] RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:30] RECOVERY - Puppet run on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:34] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:41] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:41] RECOVERY - Puppet run on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:41] RECOVERY - Puppet run on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:41] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:55] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:59] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:03] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:03] RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:43] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:47] RECOVERY - Puppet run on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:55] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:55] RECOVERY - Puppet run on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:07] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:09] RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:09] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:13] RECOVERY - Puppet run on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:15] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:17] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:21] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:23] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:29] RECOVERY - Puppet run on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:52] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:00] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:04] RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:08] RECOVERY - Puppet run on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:10] RECOVERY - Puppet run on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:10] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:12] RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:40] RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:40] RECOVERY - Puppet run on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:46] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:52] RECOVERY - Puppet run on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:58] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:10] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:12] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:18] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:20] RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:30] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:34] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:54] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:05] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:07] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:07] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:19] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:21] RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:31] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:35] RECOVERY - Puppet run on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:39] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:43] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:03] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:40] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:42] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:42] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:50] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:50] RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:40] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [14:34:58] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [14:55:58] !log tools reboot tools-services-01 [14:56:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:00:08] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2876498 (10yuvipanda) Integration - we already have a bunch of stuff we use in our infrastructure, so whatever we... [15:03:05] 06Labs, 10Tool-Labs: ssh from tools-puppetmaster-02 and tools-bastion-03 to tools-services-01 times out - https://phabricator.wikimedia.org/T152695#2876505 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Yeah, definitely looks like fallout from... something? Not sure what exactly. I've rebooted the box whi... [15:04:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:23:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:23:51] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:25:31] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:26:10] 06Labs, 10Labs-Infrastructure, 06Operations: labtestnet2001 - disk space - nova-api.log needs rotation - https://phabricator.wikimedia.org/T153279#2875187 (10AlexMonk-WMF) So this file has a lot of `/usr/lib/python2.7/dist-packages/nova/context.py:181: DeprecationWarning: Using function/method 'oslo_utils.ti... [15:26:14] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:26:32] PROBLEM - Puppet run on tools-exec-1204 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:26:40] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:26:56] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:27:02] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:27:30] marktraceur hi, didnt we have problems with https://gerrit.wikimedia.org/r/#/c/323712/2/src/relay.js ? [15:27:54] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:27:56] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:28:03] I thought ^^ created mutiple bots. [15:28:10] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:28:14] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:28:22] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:28:48] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:29:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:29:36] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:30:10] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:30:43] PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:31:19] PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:32:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:32:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:32:47] PROBLEM - Puppet run on tools-worker-1025 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:32:55] PROBLEM - Puppet run on tools-worker-1005 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:33:17] PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:33:29] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:33:59] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:34:07] PROBLEM - Puppet run on tools-worker-1023 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:34:13] PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:34:17] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:34:47] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:34:53] PROBLEM - Puppet run on tools-worker-1013 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:34:59] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:35:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:35:21] PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:35:41] PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:35:57] PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:35:59] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:36:07] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:36:19] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:36:23] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:36:27] PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:36:29] PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:36:32] PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:40:17] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:40:53] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:45:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:49:08] (am investigating these straggleres too) [15:51:42] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:53:14] PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:53:16] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:53:45] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:54:57] PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:00:03] PROBLEM - Puppet run on tools-puppetmaster-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:01:39] PROBLEM - Puppet run on tools-worker-1022 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:01:54] PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:03:22] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:04:34] PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:06:46] PROBLEM - SSH on tools-worker-1017 is CRITICAL: Connection refused [16:11:47] RECOVERY - SSH on tools-worker-1017 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [16:12:44] paladox: What does the log say about the duplicate bots? [16:12:53] Let me check [16:13:02] Seems like the thing to do [16:14:10] error: prefix=barjavel.freenode.net, server=barjavel.freenode.net, command=err_notregistered, rawCommand=451, commandType=error, args=[*, You have not registered] [16:14:10] (node) warning: possible EventEmitter memory leak detected. 11 whois listeners added. Use emitter.setMaxListeners() to increase limit. [16:14:16] marktraceur im getting ^^ now [16:14:53] paladox: Interesting. There's only one call to whois. [16:14:55] RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:59] error: prefix=asimov.freenode.net, server=asimov.freenode.net, command=err_nosuchnick, rawCommand=401, commandType=error, args=[grrrit-wm12, grrrit-wm11, No such nick/channel] [16:15:01] RECOVERY - Puppet run on tools-puppetmaster-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:08] marktraceur ^^ im getting that too [16:15:18] I don't think that's relevant but we'll check [16:15:23] ok [16:16:42] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:50] marktraceur this https://phabricator.wikimedia.org/P4626 is the full log [16:17:42] http://stackoverflow.com/questions/9768444/possible-eventemitter-memory-leak-detected [16:18:16] paladox: Yeah, I understand what the warning means. [16:18:16] RECOVERY - Puppet run on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [16:18:22] Ok [16:19:57] paladox: I suspect it has something to do with our listeners not getting cleaned up on the reconnect, or something [16:20:30] But then again, maybe not... [16:21:02] ok [16:21:25] Frankly I'm a little baffled, but I'm sure we can find the issue [16:22:00] paladox: As a sanity check, can you remind me why we need to restart the IRC connection? [16:22:14] I would much rather just restart the SSH connection and not have to worry about IRC [16:23:14] and we want it to get changed back to grrrit-wm. [16:23:53] paladox: Which is why we have the "nick" command, right? [16:24:15] oh yep [16:24:28] the nick command dosent seem to be working now and i doint know why [16:24:33] RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:26:51] We'd better figure it out then. [16:27:02] ok [16:33:16] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:36:56] RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:46] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:48] RECOVERY - Free space - all mounts on tools-worker-1017 is OK: OK: tools.tools-worker-1017.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found) [16:44:03] RECOVERY - Puppet staleness on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [3600.0] [16:47:44] 06Labs, 10Labs-Infrastructure, 06Operations: labtestnet2001 - disk space - nova-api.log needs rotation - https://phabricator.wikimedia.org/T153279#2877061 (10Dzahn) Yes, checked on labnet1001 and it also occurs there. A _lot_ of the exact deprecation warning as above. Here the log is even 13GB, it just didn'... [16:48:04] 06Labs, 10Labs-Infrastructure, 06Operations: labnet/ labtestnet2001 - disk space - nova-api.log needs rotation - https://phabricator.wikimedia.org/T153279#2877063 (10Dzahn) [16:49:24] 06Labs, 10Labs-Infrastructure, 06Operations: labnet/ labtestnet2001 - disk space - nova-api.log needs rotation - https://phabricator.wikimedia.org/T153279#2875187 (10Dzahn) Looks like this is where the oslo_utils get imported: modules/openstack/files/liberty/nova/virt-libvirt-driver:from oslo_utils import t... [16:52:20] 06Labs, 10Labs-Infrastructure, 06Operations: labnet/ labtestnet2001 - disk space - nova-api.log needs rotation - https://phabricator.wikimedia.org/T153279#2877085 (10AlexMonk-WMF) yeah but virt-libvirt-driver doesn't use strtime, does it? this file (nova/contex.py) would come from the nova package [17:00:29] RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [17:02:55] RECOVERY - Puppet run on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:22] ok, the kubeernetes worker stuff is better now [17:04:01] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:47] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:19] RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:18] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:22] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:40] RECOVERY - Puppet run on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [17:08:30] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [17:09:58] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [17:12:46] RECOVERY - Puppet run on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [17:13:20] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:14:11] RECOVERY - Puppet run on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [17:16:31] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:51] RECOVERY - Puppet run on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:17] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:54] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:18] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:42] RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:56] RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:58] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:26] RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:28] RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:18] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:13] RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:02] valhallasw`vecto: There's 2 wikibugs again [17:40:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:30:42] !log tools rebooted tools-exec-1212 to 1216 [18:30:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:41:14] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [18:41:52] !log tools reboot tools-exec-1217 to 1221 [18:41:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:42:54] RECOVERY - Puppet run on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:42] RECOVERY - Puppet run on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:46] !log tools reboot tools-exec-gift [18:45:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:46:40] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [18:46:55] RECOVERY - Puppet run on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [18:49:02] !log tools reboot tools-webgrid-lighttpd-12[01-05] [18:49:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:52:00] !log tools reboot tools-exec-1204 [18:52:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:53:49] RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:41] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2877724 (10scfc) I'd like to repeat my statement from the introduction of Kubernetes: To me, it makes much more se... [18:56:06] RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:02] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:56] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [18:58:22] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [18:58:54] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [19:03:09] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [19:03:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [19:03:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:37] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:54] PROBLEM - Host tools-exec-1204 is DOWN: CRITICAL - Host Unreachable (10.68.17.88) [19:09:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [19:09:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Alicia Fagerving (WMSE) was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1160534 edit summary: [19:18:14] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [19:20:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [19:20:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [19:20:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [19:21:19] RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:03] !log tools.paste migrate to kubernetes, seems to work fine [19:52:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paste/SAL [19:52:59] valhallasw`cloud: can I shut down the dpkg webservice? [19:55:34] Yes [19:55:49] valhallasw`cloud: ok [19:55:56] Something something packagebuilder [19:57:12] But I haven't done anything in that area in ages [20:02:34] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#2877999 (10DannyH) [20:04:40] !log tools.media-reports move to k8s [20:04:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.media-reports/SAL [20:04:47] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2878004 (10bd808) >>! In T136265#2877724, @scfc wrote: > I'd like to repeat my statement from the introduction of... [20:06:24] !log tools.dpkg stop webservice, no activity [20:06:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dpkg/SAL [20:07:38] Harej: just curious, is tools.wmflabs.org/niosh a wiki? [20:08:04] oh, no, not a wiki. bunch of useful scripts [20:08:25] it seems to be an older copy of wikidata-todo? [20:09:36] !log tools.noclaims move webservice to kubernetes, is only serving static files [20:09:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.noclaims/SAL [20:10:47] yuvipanda: it's getting pretty close to where we have to reason about teh service ip range, maybe right after teh holidays? [20:11:02] chasemp: yeah, tha sounds good. [20:11:20] chasemp: we have about 70 new webservices to go before we need to do that tho [20:11:50] truly but we don't stop ppl from migrating themselves so at less than 20 slots or so it will feel weird [20:11:54] not emergency agreed [20:12:25] chasemp: yeah. [20:12:40] what in the heck was megacron? [20:13:27] chasemp: like megatron, I suppose? :D [20:13:31] heh [20:14:14] hmm, I don't see a wikipedia-readability namespace [20:14:15] * yuvipanda goes to investigate [20:23:15] !log tools.wikipedia-readability create namespace manually [20:23:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikipedia-readability/SAL [20:23:25] !log tools.wikipedia-readability move tool to k8s [20:23:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikipedia-readability/SAL [20:24:05] !log tools.wikidata-exports moved webservice to k8s backend, was only static files [20:24:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-exports/SAL [20:26:21] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265#2878060 (10yuvipanda) I pretty much agree with all the things @bd808 said, and most of the things that @scfc said... [20:27:01] chasemp: at some point I was considering making each user in PAWS a service. at that rate we'd have run out of IP space fast heh :D [20:27:24] true [20:28:38] !log tools.ircredirect move to k8s webservice backend, small amount of PHP code only. works fine on newer version [20:28:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ircredirect/SAL [20:51:00] !log tools.mediawiki-mirror moved webservice to k8s backend, is mostly just static [20:51:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mediawiki-mirror/SAL [20:54:57] yuvipanda: tools/niosh is a collection of stuff, including (a) a script to generate a wikidata gap report (b) a fork of quick statements (c) me screwing up forking quick statements [20:55:36] Harej: ah, ok. is it all PHP from the webservices? [20:57:26] am looking for things I can move simply to the kubernetes backend and php / static webservices are at the top of the list [20:58:16] the two main things are (a) a python script that generates a javascript page and (b) php stuff [21:08:05] !log tools.niosh move to kubernetes backend [21:08:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.niosh/SAL [21:09:00] Harej: ok, everything should continue working as they were before, but let me know if it isn't [21:14:10] (03CR) 10Paladox: [C: 032] Adding ##zppix-wikipedia [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327121 (owner: 10Zppix) [21:18:56] (03Merged) 10jenkins-bot: Adding ##zppix-wikipedia [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327121 (owner: 10Zppix) [21:19:01] !log tools.redirtest moved webservice to k8s backend, simple static files only [21:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.redirtest/SAL [21:25:54] (03Draft1) 10Paladox: Fix a missing : in zppix code [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327586 [21:26:42] (03CR) 10Zppix: [V: 032 C: 032] Fix a missing : in zppix code [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327586 (owner: 10Paladox) [21:27:10] (03PS1) 10Paladox: Revert "Fix a missing : in zppix code" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327587 [21:27:17] Zppix why did you merge an empty commit? [21:27:27] It wasnt for me [21:27:28] paladox: Why are you reverting an empty commit? [21:27:40] Not sure why i pressed the revert button. [21:27:51] Im a dumbass i looked at the wrong tab :P [21:27:55] lol [21:28:00] Thats why [21:28:55] (03PS2) 10Paladox: Fix a missing : in zppix code [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327587 [21:30:39] (03CR) 10Zppix: [V: 032 C: 032] Fix a missing : in zppix code [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327587 (owner: 10Paladox) [21:31:06] Zppix you should never force merge, c+2 is enough even though ci can be slow it can catch any errors. [21:31:19] Your adding a colon... [21:31:25] Yep [21:31:45] A colon wont break anything [21:32:09] Especially at the end of an yaml file [21:32:17] Yep [22:09:07] how can i run a nodejs command using kubectl [22:09:36] yuvipanda ^^ [22:40:03] Zppix: are you wanting to do something related to a webservice or do you really want to use kubectl directly? [22:40:34] Im trying to use kubectl for our test bot for grrrit-wm [22:40:36] webservice --backend kubernetes nodejs shell would give you a shell in a container with nodejs [22:40:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:00] running a bot with kubernetes is a bit more involved [22:41:11] The prod one uses it [22:41:23] you will need a deployment file that tells kubectl how to launch your container [22:41:30] How do i do that [22:42:17] if grrrit-wm is already setup for it you should be able to look at its config and copy it. I've really not looked at how that bot was setup. [22:42:27] I know it was an early test for k8s stuff [22:43:15] See the thing is is that test bot isnt the same dir that prod one is in its in a sep dir on that tool [22:43:30] And i know jackshit about setting kubectl up [22:47:40] Zppix: hmmm... looks like it may use a custom image even. I think the ~/grrrit.yaml file is the installed replication controller [22:48:12] See https://wikitech.wikimedia.org/wiki/Grrrit-wm [22:48:47] paladox: if that page described what was needed would you two be here asking me for help? ;) [22:49:06] Nope, i am not asking for help, just giving the link to Zppix. [22:49:27] didnt even realise he was talking about grrritwm until i saw it here. [22:49:43] to deploy changes you do [22:49:46] kubectl get pods [22:49:57] copy the pod from grrrit-wm-(pods) [22:50:14] kubectl delete pod grrrit-wm-(pods) [22:50:18] Zppix ^^ [22:50:39] No i want to make it so test bot runs on kubectl [22:50:45] doint forget the git pull you do in ~/lol* before doing that. [22:50:46] I know how to deploy [22:50:58] I'd say that the main bot is using this custom docker image -- https://phabricator.wikimedia.org/diffusion/TGRT/browse/master/Dockerfile [22:51:08] Oh, kubectl makes it hard to test changes [22:51:15] i rather do node src/relay.js [22:51:19] fine [22:51:21] and then sigterm [22:51:43] since i doint want to keep getting the pod number everytime i want to restart the test bot. [22:56:07] paladox: that's massively complicated... [22:56:23] yep. [22:57:15] And there's much simpler ways to achieve the same outcome. [22:58:06] oh [23:55:26] 06Labs, 10Tool-Labs, 13Patch-For-Review: install php5-readline on bastion and exec hosts - https://phabricator.wikimedia.org/T136519#2337526 (10Dzahn) @tools-exec-1401:~# Notice: /Stage[main]/Toollabs::Exec_environ/Package[php5-readline]/ensure: ensure changed 'purged' to 'latest' [23:55:34] 06Labs, 10Tool-Labs, 13Patch-For-Review: install php5-readline on bastion and exec hosts - https://phabricator.wikimedia.org/T136519#2878946 (10Dzahn) 05Open>03Resolved