[06:11:21] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [06:12:49] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 3.047 second response time https://wikitech.wikimedia.org/wiki/ORES [06:32:36] PROBLEM - check disk on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:32:38] PROBLEM - check users on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:33:15] PROBLEM - check load on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [06:35:40] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: connect to address 172.16.3.131 port 5666: Connection refusedconnect to host ores-web-01.ores.eqiad.wmflabs port 5666: Connection refused [07:24:37] RECOVERY - check disk on ORES-web01.Experimental is OK: DISK OK [07:24:38] RECOVERY - check users on ORES-web01.Experimental is OK: USERS OK - 1 users currently logged in [07:24:53] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:16] RECOVERY - check load on ORES-web01.Experimental is OK: OK - load average: 0.16, 0.13, 0.18 [07:33:04] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/532322 (owner: 10L10n-bot) [13:02:39] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [13:02:39] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [13:04:05] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.865 second response time https://wikitech.wikimedia.org/wiki/ORES [13:04:07] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 2.466 second response time https://wikitech.wikimedia.org/wiki/ORES [13:32:56] o/ [15:26:56] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Implement a MW API module for interacting with Jade entities - https://phabricator.wikimedia.org/T199834 (10ACraze) Hey @Halfak -- I'm cleaning up some existing code and seeing that talk pages were included for "Judgements". Is this still valid for... [16:01:19] 10Jade, 10Scoring-platform-team (Current), 10Documentation: Implement a MW API module for interacting with Jade entities - https://phabricator.wikimedia.org/T199834 (10Halfak) Talk pages should be included at the //Entity// level. [16:12:53] 10ORES, 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Use documentation builds as part of CI testing - https://phabricator.wikimedia.org/T231212 (10Halfak) [16:21:03] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [16:22:27] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.065 second response time https://wikitech.wikimedia.org/wiki/ORES [16:38:43] 10ORES, 10Scoring-platform-team (Current), 10Operations, 10serviceops: celery-ores-worker service failed on ores100[2,4,5] without any apparent reason or significant log - https://phabricator.wikimedia.org/T230917 (10Halfak) [16:39:35] 10Scoring-platform-team (Current): Refactor revscoring to handle session-orientation - https://phabricator.wikimedia.org/T231214 (10Halfak) [16:46:43] 10Scoring-platform-team (Current): Refactor revscoring to handle session-orientation - https://phabricator.wikimedia.org/T231214 (10Halfak) a:03Halfak [16:47:42] 10ORES, 10Scoring-platform-team (Current): Investigate Timeout for ORES - https://phabricator.wikimedia.org/T230381 (10Halfak) 05Open→03Resolved I investigated and didn't see anything unusual. Please re-open if you have more information. [17:01:39] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:02:45] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:03:49] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:08:25] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 1.500 second response time https://wikitech.wikimedia.org/wiki/ORES [17:08:51] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.495 second response time https://wikitech.wikimedia.org/wiki/ORES [17:09:27] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 8.466 second response time https://wikitech.wikimedia.org/wiki/ORES [17:42:21] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:43:47] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 1.274 second response time https://wikitech.wikimedia.org/wiki/ORES [17:49:23] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:49:51] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:50:19] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 4.316 second response time https://wikitech.wikimedia.org/wiki/ORES [17:50:45] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 3.374 second response time https://wikitech.wikimedia.org/wiki/ORES [17:57:09] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:57:39] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [17:59:25] Arg! [17:59:29] ^ hey halfak -- did we still want to move these notifications to another channel or something? [17:59:33] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 4.544 second response time https://wikitech.wikimedia.org/wiki/ORES [17:59:44] Yes. I think so. [18:01:19] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.240 second response time https://wikitech.wikimedia.org/wiki/ORES [18:01:30] I' [18:01:33] ll make a task [18:02:13] 10ORES, 10Scoring-platform-team (Current): Re-route labs-related icinga to #wikimedia-ores - https://phabricator.wikimedia.org/T231222 (10Halfak) [18:04:11] 10ORES, 10Scoring-platform-team (Current): Re-route labs-related icinga to #wikimedia-ores - https://phabricator.wikimedia.org/T231222 (10Halfak) I registered `#wikimedia-ores` and set a topic. [18:04:37] o/ paladox, do you know how we can get our labs-related icinga messages re-routed? [18:05:21] re-routed? [18:05:29] oh [18:05:33] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [18:05:34] you mean a new channel? [18:05:45] i'll do that later [18:05:52] about to head out for dinner [18:05:59] yes. thank you! [18:06:08] Have a nice meal. :) [18:06:27] thanks :) [18:06:55] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1007 bytes in 6.481 second response time https://wikitech.wikimedia.org/wiki/ORES [19:26:03] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [19:27:27] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.415 second response time https://wikitech.wikimedia.org/wiki/ORES [19:27:39] THANK YOU ICINGA YOU ARE VERY LOUD [19:28:06] I didn't know caps was on when I started, but it seemed appropriate so I left it. [19:51:54] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [19:55:08] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 975 bytes in 0.058 second response time https://wikitech.wikimedia.org/wiki/ORES [20:15:00] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [20:15:11] lol [20:16:24] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.407 second response time https://wikitech.wikimedia.org/wiki/ORES [20:40:59] dsaez, I just came across http://ruder.io/cross-lingual-embeddings/ [20:41:07] Looks like a really nice summary. [21:01:44] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [21:03:16] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 8.419 second response time https://wikitech.wikimedia.org/wiki/ORES [21:03:46] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [21:04:10] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [21:04:18] ^ Ahh. [21:08:44] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.537 second response time https://wikitech.wikimedia.org/wiki/ORES [21:09:52] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 1.793 second response time https://wikitech.wikimedia.org/wiki/ORES [21:36:16] accraze, finally, I have https://github.com/wikimedia/revscoring/pull/450 [21:36:47] I made some changes to DependentSet that required some changes across the package (mostly find and replace). [21:37:08] As you'll see, there are about 10 lines of code (more like 20 with spaces and stuff) that handle the tree-rewriting. [21:39:27] 10Scoring-platform-team, 10Community comms and outreach, 10CommRel-Specialists-Support (Jul-Sep-2019): Outreach campaign to raise awareness of Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Halfak) Thank you @mcruzWMF! I missed this in the storm of emails around Wikimania. I'll be reviewin... [21:43:13] nice halfak! i'll take a look [21:43:50] 10Scoring-platform-team, 10Community comms and outreach, 10CommRel-Specialists-Support (Jul-Sep-2019): Outreach campaign to raise awareness of Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Halfak) I just left a couple of notes. The plan generally looks good to me. My biggest question is a... [21:47:21] accraze, I'm just about to hit the road. Any last minute things for me to look at? E.g. could I review the docs work you have been doing? [21:48:03] ahh no, I've been in Jade-land for most of today [21:56:16] have a good race halfak! [21:57:00] yessir ^_^ [22:36:04] halAFK So you want all the icinga notif to go to the ores channel? [22:36:21] paladox, yes. For now that's good. Thank you for your help :) [22:36:27] ok [22:36:51] * halAFK could disappear at any moment :) [22:39:23] halAFK done [22:39:35] Amazing! Thank you [23:12:14] quick Q if you'e still around? [23:12:20] halfak^ [23:12:31] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/446068/1/wmf-config/InitialiseSettings-labs.php [23:12:41] do you know what wgUseOresUiEnabled means? [23:46:54] groceryhesit: it's a hook helper in the ores mediawiki extension [23:47:05] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ORES/+/master/includes/Hooks/Helpers.php#148 [23:47:13] ^ groceryheist [23:52:56] it's just a flag to show if ores is enabled or not