[07:35:39] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10DBA, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Ensure ORES data violating constraints do not affect production - https://phabricator.wikimedia.org/T145356#2661490 (10jcrespo) Sorry, I checked and there is less than 1000 rows to delete per wiki; things should...
[09:44:53] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: No ORES jobs in the last 10 hours - https://phabricator.wikimedia.org/T146461#2661771 (10Ladsgroup)
[09:45:23] <grrrit-wm>	 (03PS1) 10Ladsgroup: No int typehinting [extensions/ORES] - 10https://gerrit.wikimedia.org/r/312491 (https://phabricator.wikimedia.org/T146461) 
[09:46:34] <grrrit-wm>	 (03CR) 10Thiemo Mättig (WMDE): [C: 032] No int typehinting [extensions/ORES] - 10https://gerrit.wikimedia.org/r/312491 (https://phabricator.wikimedia.org/T146461) (owner: 10Ladsgroup)
[09:47:35] <grrrit-wm>	 (03Merged) 10jenkins-bot: No int typehinting [extensions/ORES] - 10https://gerrit.wikimedia.org/r/312491 (https://phabricator.wikimedia.org/T146461) (owner: 10Ladsgroup)
[09:48:48] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup: No ORES jobs are running since deployment of 1.28.0-wmf.20 - https://phabricator.wikimedia.org/T146461#2661794 (10hashar)
[09:48:52] <grrrit-wm>	 (03PS1) 10Ladsgroup: No int typehinting [extensions/ORES] (wmf/1.28.0-wmf.20) - 10https://gerrit.wikimedia.org/r/312493 (https://phabricator.wikimedia.org/T146461) 
[09:49:29] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup: No ORES jobs are running since deployment of 1.28.0-wmf.20 - https://phabricator.wikimedia.org/T146461#2661798 (10hashar)
[09:50:22] <grrrit-wm>	 (03CR) 10Ladsgroup: [C: 032] No int typehinting [extensions/ORES] (wmf/1.28.0-wmf.20) - 10https://gerrit.wikimedia.org/r/312493 (https://phabricator.wikimedia.org/T146461) (owner: 10Ladsgroup)
[09:51:10] <grrrit-wm>	 (03Merged) 10jenkins-bot: No int typehinting [extensions/ORES] (wmf/1.28.0-wmf.20) - 10https://gerrit.wikimedia.org/r/312493 (https://phabricator.wikimedia.org/T146461) (owner: 10Ladsgroup)
[09:53:24] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup: No ORES jobs are running since deployment of 1.28.0-wmf.20 - https://phabricator.wikimedia.org/T146461#2661771 (10hashar) It is not detected because jobs are not monitored, the same happened last week T14609...
[10:00:50] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Chinese language utilities - https://phabricator.wikimedia.org/T109366#2661824 (10Cosine02) [https://meta.wikimedia.org/w/index.php?title=User_talk:Ladsgroup&diff=next&oldid=15909175 Asked a week earlier] but I don't know this task until now… I think the "...
[10:46:21] <grrrit-wm>	 (03CR) 10Thiemo Mättig (WMDE): "It seems this will collect an aweful lot of information, all in memory. I wonder if there is no way to let the SQL server let the majority" (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/312286 (https://phabricator.wikimedia.org/T145503) (owner: 10Ladsgroup)
[10:47:57] <grrrit-wm>	 (03CR) 10Thiemo Mättig (WMDE): [C: 04-1] "To make this more obvious: I believe there are two ideas mixed in this current code, and this will never delete anything. The GROUP BY ret" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/312286 (https://phabricator.wikimedia.org/T145503) (owner: 10Ladsgroup)
[11:06:56] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review, 15User-Ladsgroup, and 2 others: No ORES jobs are running since deployment of 1.28.0-wmf.20 - https://phabricator.wikimedia.org/T146461#2661771 (10Ladsgroup) 05Open>03Resolved
[15:54:05] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Implement new json-lines pattern in editquality - https://phabricator.wikimedia.org/T146410#2662623 (10Halfak)
[15:54:28] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Implement new json-lines pattern in editquality - https://phabricator.wikimedia.org/T146410#2660076 (10Halfak)
[15:54:45] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Implement new json-lines pattern in editquality - https://phabricator.wikimedia.org/T146410#2660076 (10Halfak)
[16:25:24] <halfak>	 My article quality script is resuming as planned!
[16:25:26] <halfak>	 \o/
[18:38:15] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Implement new json-lines pattern in editquality - https://phabricator.wikimedia.org/T146410#2663131 (10Halfak)
[20:27:03] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 15User-Ladsgroup: Revision not found error unformatted and not localized - https://phabricator.wikimedia.org/T139587#2663297 (10Halfak) a:03Ladsgroup
[20:27:26] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10rsaas-editquality, 15User-Ladsgroup: [Spec] Use `reverted` models in ORES review tool - https://phabricator.wikimedia.org/T146378#2663299 (10Halfak) a:03Ladsgroup
[20:27:30] <halfak>	 o/ Amir1 
[20:27:58] <Amir1>	 halfak: hey!
[20:28:04] <halfak>	 Per your earlier request, I've assigned a couple of tasks to you that I would really like you to take a look at.  Please feel free to remove yourself if you disagree :) 
[20:28:13] <Amir1>	 sure
[20:28:35] <Amir1>	 one thing, today I realized that our jobrunner has been failing all revisions in the last ten hours
[20:28:42] <Amir1>	 got it fixed ASAP
[20:28:50] <Amir1>	 incident report coming soon
[20:28:56] <halfak>	 Yeah.  I saw something about that when I logged on this morning, but I wasn't sure what was going on
[20:29:18] <Amir1>	 https://phabricator.wikimedia.org/T146461
[20:30:04] <halfak>	 Amir1, so we have ~10 hours of no scores for RC feeds?
[20:30:11] <Amir1>	 yup
[20:33:10] <halfak>	 Bummer
[20:33:19] <halfak>	 Does that recover automatically?
[20:39:41] <Amir1>	 nope, we merged a patch, back ported it, deployed it and then ran a maintenance script
[20:41:45] <halfak>	 Gotcha.  Cool.  So the problem is solved
[20:44:41] <icinga-wm>	 PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:44:48] <halfak>	 NO
[20:44:50] <halfak>	 WHY
[20:44:53] <icinga-wm>	 PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:45:01] <halfak>	 I'm like 4 layers of stack deep
[20:45:05] <halfak>	 Don't do this to me right now
[20:47:02] <icinga-wm>	 RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 1.112 second response time
[20:47:13] <icinga-wm>	 RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.122 second response time
[20:49:13] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Investigate memory leak in precached - https://phabricator.wikimedia.org/T146500#2663336 (10Halfak)
[20:49:17] <halfak>	 https://phabricator.wikimedia.org/T146500
[22:39:11] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements, 10rsaas-editquality, 03Collab-Team-Q1-July-Sep-2016: Research how to present ORES scores to users in a way that is understandable and meets their reviewing goals - https://phabricator.wikimedia.org/T146333#2663591 (10Catrope) Updated table: T1462...
[23:30:00] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements, 10rsaas-editquality, 03Collab-Team-Q1-July-Sep-2016: Research how to present ORES scores to users in a way that is understandable and meets their reviewing goals - https://phabricator.wikimedia.org/T146333#2663704 (10jmatazzoni) [[https://docs.go...