[00:08:06] 10MediaWiki-extensions-ORES, 10ORES, 10Scoring-platform-team, 10Growth-Team (Current Sprint): Non-overlapping threshholds in ORESModels on lvwiki - https://phabricator.wikimedia.org/T221871 (10Etonkovidova) 05Open→03Resolved [06:29:13] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.020 second response time https://wikitech.wikimedia.org/wiki/ORES [06:46:15] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.089 second response time https://wikitech.wikimedia.org/wiki/ORES [08:13:50] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:14:24] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [08:14:52] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 4.266 second response time https://wikitech.wikimedia.org/wiki/ORES [08:15:24] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 0.089 second response time https://wikitech.wikimedia.org/wiki/ORES [09:45:59] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/532926 (owner: 10L10n-bot) [13:49:41] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [13:51:15] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 7.536 second response time https://wikitech.wikimedia.org/wiki/ORES [14:02:53] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @bd808 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:46:21] that is a lie, it's starting in 15 minutes [14:52:37] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @bd808 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:41:41] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [15:41:45] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [15:42:49] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [15:47:29] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 8.831 second response time https://wikitech.wikimedia.org/wiki/ORES [15:47:59] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 7.915 second response time https://wikitech.wikimedia.org/wiki/ORES [15:49:27] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.917 second response time https://wikitech.wikimedia.org/wiki/ORES [16:05:21] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [16:05:23] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [16:08:25] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 4.740 second response time https://wikitech.wikimedia.org/wiki/ORES [16:10:05] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 5.751 second response time https://wikitech.wikimedia.org/wiki/ORES [16:43:05] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [16:44:35] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1011 bytes in 3.504 second response time https://wikitech.wikimedia.org/wiki/ORES [17:47:04] hay accraze I have another archaeology question [17:47:15] I'm trying to get scores from old versions of models [17:47:42] i'm trying to load up the submodules in mediawiki-services-ores-deploy [17:47:58] but for some of the commits I can't load the module. [17:48:05] git submodule update editquality fails [17:48:20] e.g. stderr: 'fatal: Not a valid commit name 7b1f46a98ff5ec96d62a9404cba18c7b4b137c0f' [17:48:43] maybe this commit was deleted from editquality or was on a branch? [17:49:33] hmm maybe, have you tried cd-ing into the submodule dir and the doing a git pull? [17:49:46] I recall having to do that a while back for something [17:51:26] seems like something happened around 2018-08-08 [17:51:48] I think that's around when we did our last deploy [17:52:20] huh, i see lots of commits after that [17:53:06] oh duh... i misread 2018->2019 [17:54:46] the only thing major difference I'm seeing with that commit is that there was a different repo remote (wikia-ai instead of wikimedia) [17:55:00] *wiki-ai [17:55:52] oh! that could be it [17:56:51] yeah that could explain why the submodule update wouldn't work [17:58:07] you might have to manually do a git checkout [18:11:01] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [18:11:05] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [18:12:35] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 5.765 second response time https://wikitech.wikimedia.org/wiki/ORES [18:17:23] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [18:18:47] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 3.449 second response time https://wikitech.wikimedia.org/wiki/ORES [18:18:57] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 9.893 second response time https://wikitech.wikimedia.org/wiki/ORES [18:23:39] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [18:25:07] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 3.216 second response time https://wikitech.wikimedia.org/wiki/ORES [18:46:14] accraze: hmm i'm getting "reference not in tree" [18:46:53] does the other remote still exist? [18:48:19] seems like no [18:48:48] nope, doesn't seem to exist anymore... [18:48:54] yikes [18:49:52] are you getting that error when you do "git checkout "? [18:49:53] the revision history seems to exist, but the shas changed [18:49:55] yeah [18:50:03] the hashes seem to have changed [18:50:20] wow that's really weird [18:55:11] they aren't in the phabricator repo either [18:57:39] i wonder if there was a rebase that happened around then that could have changed the hashes? [19:04:29] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [19:05:55] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 1.910 second response time https://wikitech.wikimedia.org/wiki/ORES [20:12:00] accraze & groceryheist! Aha! That is because we re-wrote history of git when we switched to LFS [20:12:02] Damn! [20:12:11] You can use timestamps. [20:12:18] but those hashes won't work anymore :( [21:11:35] 10Jade, 10Scoring-platform-team (Current): Fix jade role in mediawiki vagrant - https://phabricator.wikimedia.org/T231497 (10ACraze) [21:13:08] ahh ok that makes sense halfak, thanks! [21:15:45] 10Jade, 10Scoring-platform-team (Current): Fix jade role in mediawiki vagrant - https://phabricator.wikimedia.org/T231497 (10ACraze)