[10:10:48] 10ORES, 10Scoring-platform-team: ores-redis-02 is out of disk space. AOF file is too big - https://phabricator.wikimedia.org/T233831 (10akosiaris) @halfak, this is pretty much blog post material. Anyway I 'll try and summarize what I think happened and what we can do below = Intro ORES has a persistency mec... [11:08:27] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:08:33] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:11:54] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.072 second response time https://wikitech.wikimedia.org/wiki/ORES [11:12:00] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.045 second response time https://wikitech.wikimedia.org/wiki/ORES [11:14:20] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [11:15:26] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.056 second response time https://wikitech.wikimedia.org/wiki/ORES [11:16:18] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 355 bytes in 0.039 second response time https://wikitech.wikimedia.org/wiki/ORES [11:17:40] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:18:42] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 1.286 second response time https://wikitech.wikimedia.org/wiki/ORES [11:19:36] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:21:50] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1007 bytes in 0.081 second response time https://wikitech.wikimedia.org/wiki/ORES [11:21:59] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [11:23:14] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.108 second response time https://wikitech.wikimedia.org/wiki/ORES [11:27:02] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:31:24] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:31:42] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 355 bytes in 1.013 second response time https://wikitech.wikimedia.org/wiki/ORES [11:34:42] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.099 second response time https://wikitech.wikimedia.org/wiki/ORES [11:39:18] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:43:32] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.073 second response time https://wikitech.wikimedia.org/wiki/ORES [11:45:20] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.080 second response time https://wikitech.wikimedia.org/wiki/ORES [11:45:38] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.081 second response time https://wikitech.wikimedia.org/wiki/ORES [11:46:44] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 0.105 second response time https://wikitech.wikimedia.org/wiki/ORES [11:48:18] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:49:54] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.083 second response time https://wikitech.wikimedia.org/wiki/ORES [11:51:44] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [11:53:20] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 1.096 second response time https://wikitech.wikimedia.org/wiki/ORES [11:55:12] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:57:54] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [11:58:08] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [11:59:44] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.294 second response time https://wikitech.wikimedia.org/wiki/ORES [12:00:00] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.045 second response time https://wikitech.wikimedia.org/wiki/ORES [12:04:48] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [12:06:10] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [12:07:32] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.070 second response time https://wikitech.wikimedia.org/wiki/ORES [12:08:02] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.293 second response time https://wikitech.wikimedia.org/wiki/ORES [12:09:22] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.158 second response time https://wikitech.wikimedia.org/wiki/ORES [12:42:44] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 355 bytes in 0.007 second response time https://wikitech.wikimedia.org/wiki/ORES [12:45:56] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.287 second response time https://wikitech.wikimedia.org/wiki/ORES [17:16:56] afk for a bit [20:25:34] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [20:25:34] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [20:26:20] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [20:26:44] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [20:28:50] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.077 second response time https://wikitech.wikimedia.org/wiki/ORES [20:30:48] PROBLEM - ORES home page on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 355 bytes in 0.035 second response time https://wikitech.wikimedia.org/wiki/ORES [20:33:42] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [20:34:02] RECOVERY - ORES home page on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 420 bytes in 0.016 second response time https://wikitech.wikimedia.org/wiki/ORES [20:36:26] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.065 second response time https://wikitech.wikimedia.org/wiki/ORES [20:36:58] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.060 second response time https://wikitech.wikimedia.org/wiki/ORES [20:37:40] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.088 second response time https://wikitech.wikimedia.org/wiki/ORES [20:38:40] oof [20:38:45] wrong channel oops [20:40:10] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1007 bytes in 0.070 second response time https://wikitech.wikimedia.org/wiki/ORES [20:41:18] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [20:41:48] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [20:44:30] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 0.054 second response time https://wikitech.wikimedia.org/wiki/ORES [20:46:36] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [20:47:22] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [20:48:58] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.094 second response time https://wikitech.wikimedia.org/wiki/ORES [20:53:50] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [20:54:14] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [20:54:44] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.073 second response time https://wikitech.wikimedia.org/wiki/ORES [20:55:50] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.058 second response time https://wikitech.wikimedia.org/wiki/ORES [20:59:36] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [21:01:54] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.280 second response time https://wikitech.wikimedia.org/wiki/ORES [21:02:48] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.113 second response time https://wikitech.wikimedia.org/wiki/ORES [21:05:58] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.062 second response time https://wikitech.wikimedia.org/wiki/ORES [21:08:44] PROBLEM - ORES web node labs ores-web-04 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL - No data received from host https://wikitech.wikimedia.org/wiki/ORES [21:10:22] RECOVERY - ORES web node labs ores-web-04 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.056 second response time https://wikitech.wikimedia.org/wiki/ORES [21:22:04] PROBLEM - ORES web node labs ores-web-06 on ores.wmflabs.org is CRITICAL: CRITICAL - Cannot make SSL connection. https://wikitech.wikimedia.org/wiki/ORES [21:23:44] RECOVERY - ORES web node labs ores-web-06 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.982 second response time https://wikitech.wikimedia.org/wiki/ORES [22:27:03] o/ [22:27:05] Back in MN. [22:27:39] Got a little bit of hacking done on the flight. I want to take an opportunity for a (honestly pretty minor) refactor, but I need to outline it first. [22:27:48] Got the partial outline mostly done on the plane. [22:28:07] Gist of refactor is that the max() function is weird. [22:28:15] In that max(1,2,3) == 3 [22:28:22] And also max([1,2,3]) == 3 [22:28:37] But max([1,2,3], [2,3,4]) raises an exception. [22:28:48] It's really two different things happening with the same function. [22:30:52] The same thing doesn't make sense with other aggregators. E.g. sum(1,2,3) raises an type error (too many args), but sum([1,2,3]) == 6 [22:30:57] accraze, ^ [22:31:15] Anyway, I'm going to head out shortly. Need anything before I go? [22:55:33] halfak: added accraze to icinga2 anyway im gonna go eat dinner [22:55:41] YThank you! [22:57:10] sounds good halfak! nothing needed right now, im just powering through the Jade api module tests [23:31:59] Awesome. Godspeed :)