[00:00:53] 3Wikimedia-Labs-wikistats: Add 300 000 wikia wikis to stats table - https://phabricator.wikimedia.org/T38291#992349 (10Nemo_bis) [00:08:36] 3Wikimedia-Labs-wikistats: Add 300 000 wikia wikis to stats table - https://phabricator.wikimedia.org/T38291#992356 (10Dzahn) i extracted from http://community.wikia.com/api.php?action=query&list=wkdomains&wkto=500000&format=json and using jq and grep and got 119675 wikis [00:12:09] Coren, did YuviPanda poke you yet? [00:21:18] Reedy: Hi, sorry for not waiting, thanks for your commit and advice, can you also run namespaceDupes please, I'm worried about missing pages. [00:23:26] 3Wikimedia-Labs-wikistats: Add 300 000 wikia wikis to stats table - https://phabricator.wikimedia.org/T38291#992383 (10Nemo_bis) >>! In T38291#992356, @Dzahn wrote: > i extracted from http://community.wikia.com/api.php?action=query&list=wkdomains&wkto=500000&format=json and using jq and grep and got 119675... [00:23:47] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:24:52] 3Wikimedia-Labs-wikistats: Add 300 000 wikia wikis to stats table - https://phabricator.wikimedia.org/T38291#992386 (10Dzahn) it also has duplicates. example: domainofheroes.wikia.com domainofheroes.wikia.com [00:30:57] Mjbmr: sure [00:31:11] Thank you so much [00:31:16] which wiki was it? :P [00:33:16] 3Wikimedia-Labs-wikistats: Add 300 000 wikia wikis to stats table - https://phabricator.wikimedia.org/T38291#992412 (10Dzahn) >>! In T38291#992383, @Nemo_bis wrote: > That's not right, you have to use at least three batches up to 1.5M and filter by wkactive=1. How? well, for now i deleted the horribly old list... [00:34:35] Reedy: fawikibooks [00:35:05] Mjbmr: All done now [00:35:19] Thanks :) [00:35:23] There were some moved [00:36:34] Yes, they're fine. [00:37:53] 3Wikimedia-Labs-General, translatewiki.net: Vulnerability scanning from Wikimedia Labs IP address - https://phabricator.wikimedia.org/T87352#992417 (10scfc) I meant to write "it would narrow". Because, as I said in comment T87352#990531, 208.80.155.255 is an instance without a public IP. So if the request came... [00:39:04] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:53:45] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [01:04:08] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [01:53:00] Coren: we have a tech task that is still referring users to unicorn.wmflabs.org, which i believe was jorm's test machine. How can i find out who are admins there? [01:54:40] ebernhardson: https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwreview [01:54:52] ebernhardson: It's only ops and jorm [01:55:57] ebernhardson: Opsen should be able to add you to the project though [01:56:16] There's a few more members than just admins [01:56:28] but probably a 3 day wait? I'll see brandon tomorrow anyways i suppose i'll just tweak it then [01:56:49] I don't see why there'd be a 3 day wait [01:57:12] hmm, i'll file an RT then [01:57:14] But getting him to add you wouldn't be the worst idea [01:57:29] *cough*phabricator*cough* [01:57:48] i like phab :P [01:58:10] If YuviPanda is at a laptop still, he might be able just add you [02:16:28] ebernhardson: no 3 day wait. I will add you when I am on a laptop [02:44:50] YuviPanda, did you manage to poke Coren yet? Sorry if I sound impatient. [05:06:56] !log bots restarted wm-bot services, unresponsive in irc [05:07:01] Logged the message, Master [06:53:55] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:57:20] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:10:57] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:18:51] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [07:27:16] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:00] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [08:38:45] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:00:36] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:08:47] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [09:13:52] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:25:19] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:25:31] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:29:52] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [09:40:00] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:41:20] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:55:17] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [09:58:52] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [09:59:50] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [10:01:57] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:05:01] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [10:06:25] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:22] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:32:00] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [10:33:57] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [10:37:06] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:38:00] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:44:48] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:49:55] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:52:22] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:53:02] PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:58:57] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:03] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:13] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:07:08] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [11:09:46] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [11:18:23] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:19:14] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:19:53] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [11:23:49] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:37:15] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:42:59] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:29] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:29] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:56:16] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:04:48] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:07:13] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [12:10:44] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:14:01] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:27:01] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:43:56] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [12:51:59] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [13:02:57] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:11:18] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:16:52] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:16:56] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:27:59] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [13:31:55] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:36:02] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:36:22] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [13:41:56] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [13:41:56] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:49] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:00:57] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [14:01:43] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:01:55] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:14:56] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:16:46] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:33] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:13] 3Tool-Labs-tools-Erwin's-tools: xwiki.php not working - https://phabricator.wikimedia.org/T86976#993125 (10MarcoAurelio) @Billinghurst: as you can see in T62878, code is already on Labs, "just" needs to be fixed to run there. [14:44:53] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [15:00:09] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:00:13] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:16:31] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:25:05] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:15] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:46:35] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:49] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:58:14] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [16:00:50] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:04:39] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:08:26] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:12:47] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:14] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:38] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [16:30:49] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:23] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:43] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:01:52] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:10:47] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [17:31:52] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:47] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:54:50] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:09:48] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:19:22] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:19:54] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [18:49:23] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:57] PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:36:19] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:50:55] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:57:34] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:59:57] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [20:20:55] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [20:21:15] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:27:36] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:28:16] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:33:59] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:58:12] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:03:56] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [21:09:45] 3Labs, Wikimedia-Labs-Infrastructure: Debian Jessie image for Labs - https://phabricator.wikimedia.org/T75592#993399 (10yuvipanda) Few issues persist, have filed blocking tasks. [21:20:46] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:25:40] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:50:41] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:47] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [22:03:59] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:04:43] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:29:00] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [22:34:42] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:37:10] 3Tool-Labs-tools-Erwin's-tools: xwiki.php not working - https://phabricator.wikimedia.org/T86976#993441 (10Billinghurst) We still need to ping on of the people with access to get them to throw the switches @thedj @akoopal @Cyberpower678 @MZMcBride [23:36:17] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]