[05:21:41] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, 10Wikimedia-production-error: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10Legoktm) >>! In T225199#5267518, @Krinkle wrote: > The fall out... [05:39:48] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, 10Wikimedia-production-error: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10Legoktm) > Request ID: XN9fRgpAAEQAAFE8V0IAAABV (1.34.0-wmf.5)... [05:42:30] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, 10Wikimedia-production-error: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10Legoktm) It looks like it could be any job at all that fails ser... [06:34:37] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, and 2 others: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10Legoktm) OK, theoretically we'll be able to better pinpoint exactly which jobs are... [06:37:47] PROBLEM - check disk on ORES-web02.Experimental is CRITICAL: connect to address 172.16.6.234 port 5666: Connection refusedconnect to host ores-web-02.ores.eqiad.wmflabs port 5666: Connection refused [06:37:52] PROBLEM - check users on ORES-web02.Experimental is CRITICAL: connect to address 172.16.6.234 port 5666: Connection refusedconnect to host ores-web-02.ores.eqiad.wmflabs port 5666: Connection refused [06:39:11] PROBLEM - check load on ORES-web02.Experimental is CRITICAL: connect to address 172.16.6.234 port 5666: Connection refusedconnect to host ores-web-02.ores.eqiad.wmflabs port 5666: Connection refused [06:41:12] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: connect to address 172.16.6.234 port 5666: Connection refusedconnect to host ores-web-02.ores.eqiad.wmflabs port 5666: Connection refused [06:59:09] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. [07:15:47] RECOVERY - check disk on ORES-web02.Experimental is OK: DISK OK [07:15:49] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:15:52] RECOVERY - check users on ORES-web02.Experimental is OK: USERS OK - 1 users currently logged in [07:17:11] RECOVERY - check load on ORES-web02.Experimental is OK: OK - load average: 0.08, 0.07, 0.33 [07:23:41] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:15:57] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, and 2 others: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10ArielGlenn) ` 2019-07-19 09:32:42 [XTGOOQpAIDEAAKzYQLoAAABP] mw1328 dewiki 1.34.0-... [10:40:18] 10Scoring-platform-team (Research), 10Machine vision, 10artificial-intelligence: Implement NSFW image classifier using Open NSFW - https://phabricator.wikimedia.org/T214201 (10Tgr) open_nsfw is basically a (publicly available) snapshot of a neural network, not something we'd train ourselves, right? Which mea... [11:17:02] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, and 2 others: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10ArielGlenn) Hoping this is helpful, I looked at a couple of the few weird page tit... [14:58:34] (03PS1) 10DCausse: Normalize the user agent before sending FetchScoreJob [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524534 (https://phabricator.wikimedia.org/T225199) [15:07:45] (03PS2) 10DCausse: Normalize the user agent before sending FetchScoreJob [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524534 (https://phabricator.wikimedia.org/T225199) [15:58:59] o/ accraze [15:59:01] Good morning :) [15:59:26] o/ [15:59:35] hey halfak [16:00:14] Just wanted to remind you that I'll be around for standup today. :) [16:30:55] (03CR) 10Accraze: [C: 03+2] Normalize the user agent before sending FetchScoreJob [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524534 (https://phabricator.wikimedia.org/T225199) (owner: 10DCausse) [16:34:20] (03Merged) 10jenkins-bot: Normalize the user agent before sending FetchScoreJob [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524534 (https://phabricator.wikimedia.org/T225199) (owner: 10DCausse) [16:37:48] (03CR) 10jenkins-bot: Normalize the user agent before sending FetchScoreJob [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524534 (https://phabricator.wikimedia.org/T225199) (owner: 10DCausse) [17:13:02] I went looking for the blogpost I saw that said dict is slow [17:13:13] turns out it was using PyPy for the benchmark [17:13:40] apparently the timing for lookup is pretty similar between slots, attrs, and dict in Cython [17:55:38] that's pretty interesting groceryheist, I always just assumed slots were fastest [18:05:23] but it makes sense that they'd have similar lookups [19:09:50] slots are faster [19:09:59] not dramatically faster [19:10:03] well [19:10:09] depends on what "dramatically" means [19:10:16] https://stackoverflow.com/questions/472000/usage-of-slots [19:10:42] also you get space savings [19:11:37] makes sense, it's just object references vs an implicit dictionary [19:14:39] accraze: would you be willing to take some time monday to talk about abstractions for this project? [19:15:02] I'll take a stab at it today, but there will probably be room for improvement. [19:16:08] yeah for sure! [19:16:41] I'm pretty open on Monday, do you wanna pick a time? [19:21:01] does after standup work? or like 10ish? [19:21:20] yeah after standup sounds good [21:37:13] (03PS1) 10Accraze: Handle JobQueueError exception [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524604 (https://phabricator.wikimedia.org/T225199) [21:40:31] (03CR) 10jerkins-bot: [V: 04-1] Handle JobQueueError exception [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524604 (https://phabricator.wikimedia.org/T225199) (owner: 10Accraze) [21:41:00] (03CR) 10jerkins-bot: [V: 04-1] Handle JobQueueError exception [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524604 (https://phabricator.wikimedia.org/T225199) (owner: 10Accraze) [22:10:43] (03PS2) 10Accraze: Handle JobQueueError exception [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524604 (https://phabricator.wikimedia.org/T225199) [22:14:36] (03CR) 10Krinkle: [C: 03+1] Handle JobQueueError exception [extensions/ORES] - 10https://gerrit.wikimedia.org/r/524604 (https://phabricator.wikimedia.org/T225199) (owner: 10Accraze) [22:57:44] Hi everyone, we would like to invite people here to try WikiLoop Battlefield for labeling and reverting vandlism http://battlefield.wikiloop.org [23:00:03] in particular, this tool provide an api to get the list of marked revision http://battlefield.wikiloop.org/api/markedRev.csv to download them [23:00:25] http://battlefield.wikiloop.org/api/markedRevs.csv