[17:44:16] o/ [18:00:30] halfak: o/ [19:19:19] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2760953 (10Halfak) Looks like some timeout error is squeaking past. See https://github.com/wiki-ai/ores/blob/master/ores/scoring_systems/celery_queue.py#L54 [19:19:21] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2760953 (10Halfak) Looks like some timeout error is squeaking past. See https://github.com/wiki-ai/ores/blob/master/ores/scoring_systems/celery_queue.py#L54 [19:19:47] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Easy, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2760955 (10Halfak) [19:19:48] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Easy, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2760955 (10Halfak) [19:22:18] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality, 07Easy: Scale up the number of observations for idwiki to 100k - https://phabricator.wikimedia.org/T147107#2760972 (10Halfak) [19:22:19] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality, 07Easy: Scale up the number of observations for idwiki to 100k - https://phabricator.wikimedia.org/T147107#2760972 (10Halfak) [19:23:05] 10Revision-Scoring-As-A-Service-Backlog, 06Collaboration-Team-Triage, 07Easy: Re-broadcast RCStream with ORES scores - https://phabricator.wikimedia.org/T106279#2760974 (10Halfak) [19:23:07] 10Revision-Scoring-As-A-Service-Backlog, 06Collaboration-Team-Triage, 07Easy: Re-broadcast RCStream with ORES scores - https://phabricator.wikimedia.org/T106279#2760974 (10Halfak) [19:34:58] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Easy, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2761022 (10Ladsgroup) a:05Ladsgroup>03None [19:35:00] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Easy, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2761022 (10Ladsgroup) a:05Ladsgroup>03None [20:12:02] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:32] PROBLEM - ORES home page on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:52] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:53] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:27] halfak: ^ This is expected [20:13:33] ores instances are being rebooted [20:33:43] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 1.673 second response time [20:33:44] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 0.862 second response time [20:34:23] RECOVERY - ORES home page on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 419 bytes in 0.115 second response time [20:34:53] RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 442 bytes in 0.943 second response time [21:19:08] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:55] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:30:45] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.282 second response time [21:32:15] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 1.290 second response time [21:38:52] Amir1, ^ ? [21:39:05] * halfak goes to look at -05 [21:39:20] I think that's expected too [21:39:33] The resets usually take lots of time [21:39:51] I think they started with ores-web-03 and now they are on ores-web-05 [21:39:55] halfak: ^ [21:40:13] Oh! [21:40:14] kk [21:40:20] * halfak gets back to his paper review [21:58:09] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:20] waaa [21:58:29] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:59] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 3492 bytes in 9.896 second response time [21:59:30] Looks like uwsgi got rebooted 10 seconds ago [21:59:38] *10 minutes [22:00:06] Redis is rebooting! [22:11:09] RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.860 second response time [22:11:19] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 4.310 second response time [22:11:49] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 441 bytes in 0.809 second response time [22:21:59] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Easy, 03Google-Code-In-2016, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2761653 (10Ladsgroup) [22:22:06] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Easy, 03Google-Code-In-2016, 15User-Ladsgroup: Quiet TimeoutError in celery logging - https://phabricator.wikimedia.org/T146681#2668149 (10Ladsgroup) I plan to mentor this in GCI [22:22:19] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality, 07Easy, 03Google-Code-In-2016: Scale up the number of observations for idwiki to 100k - https://phabricator.wikimedia.org/T147107#2761655 (10Ladsgroup) [22:22:23] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality, 07Easy, 03Google-Code-In-2016: Scale up the number of observations for idwiki to 100k - https://phabricator.wikimedia.org/T147107#2681280 (10Ladsgroup) I plan to mentor this in GCI [23:43:58] (03PS1) 10Mooeypoo: [wip] RecentChanges Dynamic Filters [extensions/ORES] - 10https://gerrit.wikimedia.org/r/319251 (https://phabricator.wikimedia.org/T144448) [23:44:53] (03CR) 10jenkins-bot: [V: 04-1] [wip] RecentChanges Dynamic Filters [extensions/ORES] - 10https://gerrit.wikimedia.org/r/319251 (https://phabricator.wikimedia.org/T144448) (owner: 10Mooeypoo)