[00:13:29] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [00:14:55] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.052 second response time https://wikitech.wikimedia.org/wiki/ORES [00:16:55] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [00:19:02] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.040 second response time https://wikitech.wikimedia.org/wiki/ORES [00:36:38] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [00:38:00] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.405 second response time https://wikitech.wikimedia.org/wiki/ORES [00:53:38] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [00:55:14] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 9.481 second response time https://wikitech.wikimedia.org/wiki/ORES [01:16:18] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [01:17:28] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.568 second response time https://wikitech.wikimedia.org/wiki/ORES [01:22:24] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [01:23:22] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 0.207 second response time https://wikitech.wikimedia.org/wiki/ORES [01:30:06] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [01:35:34] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 5.560 second response time https://wikitech.wikimedia.org/wiki/ORES [01:50:56] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [01:52:22] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.035 second response time https://wikitech.wikimedia.org/wiki/ORES [01:56:08] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [01:57:36] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.060 second response time https://wikitech.wikimedia.org/wiki/ORES [02:19:58] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.020 second response time https://wikitech.wikimedia.org/wiki/ORES [02:21:36] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.047 second response time https://wikitech.wikimedia.org/wiki/ORES [03:30:48] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.017 second response time https://wikitech.wikimedia.org/wiki/ORES [03:34:00] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.071 second response time https://wikitech.wikimedia.org/wiki/ORES [03:37:02] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [03:40:06] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 0.095 second response time https://wikitech.wikimedia.org/wiki/ORES [03:42:16] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [03:43:42] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.060 second response time https://wikitech.wikimedia.org/wiki/ORES [04:02:48] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [04:05:52] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.042 second response time https://wikitech.wikimedia.org/wiki/ORES [04:07:24] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [04:08:58] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 6.975 second response time https://wikitech.wikimedia.org/wiki/ORES [04:11:12] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [04:12:40] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.096 second response time https://wikitech.wikimedia.org/wiki/ORES [04:30:00] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [04:30:16] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [04:30:36] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [04:32:02] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.106 second response time https://wikitech.wikimedia.org/wiki/ORES [04:33:26] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 6.210 second response time https://wikitech.wikimedia.org/wiki/ORES [04:34:40] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.080 second response time https://wikitech.wikimedia.org/wiki/ORES [04:47:20] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [04:48:50] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 3.541 second response time https://wikitech.wikimedia.org/wiki/ORES [05:01:22] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [05:06:06] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.273 second response time https://wikitech.wikimedia.org/wiki/ORES [05:10:30] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [05:10:42] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [05:12:10] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.064 second response time https://wikitech.wikimedia.org/wiki/ORES [05:12:44] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [05:13:36] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.562 second response time https://wikitech.wikimedia.org/wiki/ORES [05:14:10] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.093 second response time https://wikitech.wikimedia.org/wiki/ORES [05:29:30] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [05:29:54] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [05:32:38] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.044 second response time https://wikitech.wikimedia.org/wiki/ORES [05:33:00] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 2.189 second response time https://wikitech.wikimedia.org/wiki/ORES [06:03:26] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [06:05:02] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 7.957 second response time https://wikitech.wikimedia.org/wiki/ORES [06:13:08] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [06:14:36] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.064 second response time https://wikitech.wikimedia.org/wiki/ORES [07:28:04] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [07:29:32] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 4.060 second response time https://wikitech.wikimedia.org/wiki/ORES [07:53:48] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [07:55:18] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.622 second response time https://wikitech.wikimedia.org/wiki/ORES [08:08:18] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:11:20] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.042 second response time https://wikitech.wikimedia.org/wiki/ORES [08:22:00] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:23:30] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 975 bytes in 2.748 second response time https://wikitech.wikimedia.org/wiki/ORES [08:33:54] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:36:58] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.099 second response time https://wikitech.wikimedia.org/wiki/ORES [08:39:44] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:42:00] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:42:24] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:42:56] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 7.892 second response time https://wikitech.wikimedia.org/wiki/ORES [08:43:56] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 2.590 second response time https://wikitech.wikimedia.org/wiki/ORES [08:45:10] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 4.979 second response time https://wikitech.wikimedia.org/wiki/ORES [09:01:26] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:03:00] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 7.334 second response time https://wikitech.wikimedia.org/wiki/ORES [09:08:48] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:11:02] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:11:02] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:11:26] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:12:00] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 7.912 second response time https://wikitech.wikimedia.org/wiki/ORES [09:12:32] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 3.278 second response time https://wikitech.wikimedia.org/wiki/ORES [09:12:32] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 3.280 second response time https://wikitech.wikimedia.org/wiki/ORES [09:12:54] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.038 second response time https://wikitech.wikimedia.org/wiki/ORES [09:13:17] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Implement hunspell dictionary for euwiki article quality model - https://phabricator.wikimedia.org/T223788 (10Theklan) Hello! I have some queue to start with the labelling campaign, but I will follow up as soon as possible. We have... [09:21:06] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:22:42] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 8.768 second response time https://wikitech.wikimedia.org/wiki/ORES [09:29:14] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:31:24] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:32:16] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.036 second response time https://wikitech.wikimedia.org/wiki/ORES [09:32:50] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.054 second response time https://wikitech.wikimedia.org/wiki/ORES [09:34:01] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Implement hunspell dictionary for euwiki article quality model - https://phabricator.wikimedia.org/T223788 (10Theklan) The article Lantanoide (https://eu.wikipedia.org/wiki/Lantanoide) is a good example: structurally perfect, but wit... [09:38:30] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:38:30] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:39:56] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.523 second response time https://wikitech.wikimedia.org/wiki/ORES [09:39:56] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.540 second response time https://wikitech.wikimedia.org/wiki/ORES [09:52:54] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:56:08] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 7.817 second response time https://wikitech.wikimedia.org/wiki/ORES [09:57:06] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [09:58:38] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 3.063 second response time https://wikitech.wikimedia.org/wiki/ORES [09:59:52] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:01:28] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 8.248 second response time https://wikitech.wikimedia.org/wiki/ORES [10:05:54] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:06:52] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:07:28] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 6.840 second response time https://wikitech.wikimedia.org/wiki/ORES [10:07:58] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:08:20] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 2.184 second response time https://wikitech.wikimedia.org/wiki/ORES [10:12:40] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.064 second response time https://wikitech.wikimedia.org/wiki/ORES [10:20:30] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:21:58] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.054 second response time https://wikitech.wikimedia.org/wiki/ORES [10:26:50] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.018 second response time https://wikitech.wikimedia.org/wiki/ORES [10:27:26] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:27:58] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:30:02] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 0.134 second response time https://wikitech.wikimedia.org/wiki/ORES [10:30:36] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 5.392 second response time https://wikitech.wikimedia.org/wiki/ORES [10:31:10] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 7.716 second response time https://wikitech.wikimedia.org/wiki/ORES [10:39:54] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:40:52] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:42:26] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 7.551 second response time https://wikitech.wikimedia.org/wiki/ORES [10:46:16] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:46:46] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:47:50] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 4.505 second response time https://wikitech.wikimedia.org/wiki/ORES [10:49:26] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 6.370 second response time https://wikitech.wikimedia.org/wiki/ORES [10:53:10] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 5.179 second response time https://wikitech.wikimedia.org/wiki/ORES [10:54:24] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:55:56] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 6.389 second response time https://wikitech.wikimedia.org/wiki/ORES [10:58:08] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:59:18] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:00:50] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:01:20] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 8.344 second response time https://wikitech.wikimedia.org/wiki/ORES [11:02:22] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.134 second response time https://wikitech.wikimedia.org/wiki/ORES [11:03:30] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:05:06] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 9.664 second response time https://wikitech.wikimedia.org/wiki/ORES [11:06:12] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:07:40] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 2.525 second response time https://wikitech.wikimedia.org/wiki/ORES [11:08:50] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 4.588 second response time https://wikitech.wikimedia.org/wiki/ORES [11:12:40] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:14:12] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 3.953 second response time https://wikitech.wikimedia.org/wiki/ORES [11:14:48] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:17:02] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:17:04] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:17:58] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 6.606 second response time https://wikitech.wikimedia.org/wiki/ORES [11:18:32] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.056 second response time https://wikitech.wikimedia.org/wiki/ORES [11:18:38] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 975 bytes in 9.352 second response time https://wikitech.wikimedia.org/wiki/ORES [11:22:18] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:22:50] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:23:28] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:24:58] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 3.291 second response time https://wikitech.wikimedia.org/wiki/ORES [11:25:26] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 3.250 second response time https://wikitech.wikimedia.org/wiki/ORES [11:25:54] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.051 second response time https://wikitech.wikimedia.org/wiki/ORES [11:30:22] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:31:28] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:33:53] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Implement hunspell dictionary for euwiki article quality model - https://phabricator.wikimedia.org/T223788 (10Theklan) I have finished the labelling campaing. There was a redirect in the list, so I said it was a stub, because I could... [11:34:06] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:42:38] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.218 second response time https://wikitech.wikimedia.org/wiki/ORES [11:42:48] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:43:08] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 3.796 second response time https://wikitech.wikimedia.org/wiki/ORES [11:43:36] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.071 second response time https://wikitech.wikimedia.org/wiki/ORES [11:44:20] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.383 second response time https://wikitech.wikimedia.org/wiki/ORES [11:51:14] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:52:48] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 7.879 second response time https://wikitech.wikimedia.org/wiki/ORES [11:56:36] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:57:40] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:58:52] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [11:59:40] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.189 second response time https://wikitech.wikimedia.org/wiki/ORES [12:00:44] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.769 second response time https://wikitech.wikimedia.org/wiki/ORES [12:03:38] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.100 second response time https://wikitech.wikimedia.org/wiki/ORES [12:10:04] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [12:10:30] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [12:11:30] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.059 second response time https://wikitech.wikimedia.org/wiki/ORES [12:13:34] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.085 second response time https://wikitech.wikimedia.org/wiki/ORES [12:14:56] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [12:16:24] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.597 second response time https://wikitech.wikimedia.org/wiki/ORES [12:16:30] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [12:17:32] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [12:19:02] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 3.965 second response time https://wikitech.wikimedia.org/wiki/ORES [12:21:10] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.048 second response time https://wikitech.wikimedia.org/wiki/ORES [12:21:46] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [12:24:50] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.047 second response time https://wikitech.wikimedia.org/wiki/ORES [12:59:18] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [13:00:20] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [13:00:54] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 8.509 second response time https://wikitech.wikimedia.org/wiki/ORES [13:05:00] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.514 second response time https://wikitech.wikimedia.org/wiki/ORES [13:05:47] Woah. OK. Look at all this! [13:05:59] So. This is crazy. [13:06:24] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [13:08:33] Looks like our ability to process requests is diminished. [13:09:36] I told icinga to shuttup. [13:09:44] So now to look into what's going on. [13:45:31] Something really weird is going on. [13:45:59] I just cleared out celery and restarted everything and we're still getting *pinned* on 3/4 of our celery workers. [13:46:03] There was no lull. [13:46:35] And AFAICT, there's not a queue of incoming requests that is filling up. [13:46:47] Something else is using a ton of CPU on these machines. [13:47:19] I have looked through the user agents. There's no clear indication of anyone. It's almost all just "-" [13:47:49] I suspect that celery is spinning on its own, so I'm going to try temporarily disabling the load balancer -- thus cutting outside traffic and monitoring what celery does. [13:50:34] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Implement hunspell dictionary for euwiki article quality model - https://phabricator.wikimedia.org/T223788 (10Halfak) Thanks Theklan. I looked through https://eu.wikipedia.org/wiki/Lantanoide but my stupid American monolingual eyes... [13:54:14] OK it looks like CPU usage on the workers has fallen, but we still have one worker process on each machine taking up 100% of a core. [13:54:44] This should be impossible. We are using signal based timeouts that should eliminate the possibility of a run-away celery process. But here it is happening. [14:00:55] 10ORES, 10Scoring-platform-team (Current): ORES celery workers maxing CPU in Cloud VPS - https://phabricator.wikimedia.org/T234926 (10Halfak) [14:02:03] 10ORES, 10Scoring-platform-team (Current): ORES celery workers maxing CPU in Cloud VPS - https://phabricator.wikimedia.org/T234926 (10Halfak) From IRC while I have been thinking through this: ` [08:08:32] Looks like our ability to process requests is diminished. [08:09:29] >chanserv< quiet #wikimedi... [14:06:18] OK I have CPU back to effectively zero on the all of our VMs. I'm going to let traffic back in. [14:12:07] And as soon as they came back online, workers-01,02,03 came right back to 100% CPU usage while worker-04 is floating at ~50%. [14:12:28] I am starting to think that workers-01,02,03 are all weird. And worker-04 is in a better state. [14:16:02] I'l going to try killing off one and rebuilding a VM. [14:21:40] o/ kevinbazira [14:21:48] How are you doing? Was my email helpful? [14:39:09] I guess not ^_^ [14:41:20] * halfak watches puppet agent tv [14:52:42] OK I started ores-worker-05 and it is starting to take traffic. [14:52:59] I'll check in later to see if it is maxing CPU again. [15:36:59] Looks like ores-worker-05 is maxing CPU too [15:37:54] ores-worker-02 isn't really taking on new tasks AFAICT [15:38:01] So restarting the service there. [15:39:49] We're still running at super low capacity. [16:00:19] Hey folks! Async time! [16:02:28] Y: Mostly worked on odds and ends. Was buried in emails for a while. I know a lot more about getting a US visa than I did before :) I got myself re-re-re-invited to the design review meetings. I also found some assets for pulling into my designs. I'm still working on a good workflow for that. I did a substantial writeup for kevinbazira's intro task around Basque dictionaries and reached out to our confederate, Theklan. [16:04:14] T: I've been dealing with ores.wmflabs.org having a serious issue. I haven't made heads or tails of it yet. I'm really starting to think that shutting down that cluster would be a good idea. But honestly, I'm probably going to learn something important that will have implications for prod. Once I finish this, I'll be working on getting OKRs into betterworks and continuing on Jade design stuff. [16:04:47] Oh! I forgot. One other thing I was working on was figuring out who to talk to about recommendations for MW UI stuff. I sent some emails and made some posts but no bites yet. I'll likely get more pushy today. [16:08:26] cool [16:08:31] Y: Started reviewing the revscoring session-orientation PR. Contacted tech support about laptop issue. Minor Jade cleanup. [16:08:31] T: Finish session-orientation review, OKR stuff, laptop followup and maybe start looking at the CentralID Jade stuff. [16:10:00] Hi, Rerunning historical revision scoring and working on other stuff. [16:10:54] 10Scoring-platform-team, 10drafttopic-modeling: Key-value extraction misses on Wikipedia:WikiProject Council/Directory/WikiProject template invocations - https://phabricator.wikimedia.org/T229401 (10dr0ptp4kt) For those still following along: I haven't forgotten about this. Looking at this a bit more, I think... [16:14:59] hey groceryheist! Thanks for jumping in :) [16:15:07] Working on the threshold data? [16:16:38] threshold data's fine [16:17:25] i had an issue with how I was merging scores after calling revscoring. [16:17:36] but yeah it's for the threshhold analysis [16:21:50] Gotcha. Cool. [17:40:29] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Refactor revscoring to handle session-orientation - https://phabricator.wikimedia.org/T231214 (10ACraze) @Halfak, everything looks great so far. I think you are good to go with applying `list_of_tree()` elsewhere. It seems well docum... [18:02:19] Thanks fo rthe review! [18:15:06] 10ORES, 10Scoring-platform-team (Current): ORES celery workers maxing CPU in Cloud VPS - https://phabricator.wikimedia.org/T234926 (10Halfak) After a bunch more digging in the logs, I'm finding some revids that never seem to finish scoring. E.g. https://ores.wmflabs.org/v3/scores/enwiki/559565916/damaging?fea... [18:37:33] Well. I am running into the deepest virtualenv issue I've seen in a while trying to replicate this on one of our nodes. [18:37:34] ugh [18:42:34] 10ORES, 10Scoring-platform-team (Current): ORES celery workers maxing CPU in Cloud VPS - https://phabricator.wikimedia.org/T234926 (10Halfak) OK I just replicated this on ores-worker-04: ` time revscoring score /srv/ores/config/submodules/editquality/models/enwiki.damaging.gradient_boosting.model --host https... [18:48:40] halfak: virtualenv == trash [18:48:50] i hate having to deal with that on toolforge to run my irc bot its so annoying xD [18:48:53] No way man. It's a godsend :P [18:49:22] It was really setuptools but I was experiencing it via virtualenv [18:49:38] I just started a fresh virtualenv and it's now fine. [18:49:54] I'm guessing the underlying issue was because I was trying to use another user's virtualenv. [21:20:11] 10Jade, 10Scoring-platform-team, 10Epic: Clean up naming conflicts around writing secondary schema data for Jade - https://phabricator.wikimedia.org/T235003 (10Halfak) [21:20:27] 10Jade, 10Scoring-platform-team (Current), 10Epic: Clean up naming conflicts around writing secondary schema data for Jade - https://phabricator.wikimedia.org/T235003 (10Halfak)