[06:47:38] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Using TensorFlow in ORES - https://phabricator.wikimedia.org/T161375#3130279 (10Ladsgroup) [06:47:51] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Epic: Using TensorFlow in ORES - https://phabricator.wikimedia.org/T161375#3130292 (10Ladsgroup) [06:48:40] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 15User-Ladsgroup: Initial assessment of TesnorFlow on ORES data - https://phabricator.wikimedia.org/T161376#3130293 (10Ladsgroup) [06:50:23] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Build TensorFlow models support in revscoring - https://phabricator.wikimedia.org/T161377#3130308 (10Ladsgroup) [06:52:09] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Refactor ORES to become scikit-learn agnostic (to support TensorFlow) - https://phabricator.wikimedia.org/T161378#3130321 (10Ladsgroup) [06:53:08] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10rsaas-editquality, 07Epic: Train/test TensorFlow for edit quality models - https://phabricator.wikimedia.org/T161379#3130334 (10Ladsgroup) [06:53:46] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train/test TensorFlow goodfaith/damaging in enwiki - https://phabricator.wikimedia.org/T161380#3130347 (10Ladsgroup) [06:54:03] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train/test TensorFlow goodfaith/damaging in fawiki - https://phabricator.wikimedia.org/T161381#3130360 (10Ladsgroup) [06:54:37] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 10rsaas-editquality: Train/test TensorFlow goodfaith/damaging models for wikidatawiki - https://phabricator.wikimedia.org/T161382#3130373 (10Ladsgroup) [06:54:54] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train/test TensorFlow goodfaith/damaging models for enwiki - https://phabricator.wikimedia.org/T161380#3130347 (10Ladsgroup) [06:55:03] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train/test TensorFlow goodfaith/damaging models for fawiki - https://phabricator.wikimedia.org/T161381#3130360 (10Ladsgroup) [06:56:12] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 07Epic: Deploy Tensorflow models in production - https://phabricator.wikimedia.org/T161383#3130388 (10Ladsgroup) [06:56:23] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Deploy Tensorflow models in production - https://phabricator.wikimedia.org/T161383#3130388 (10Ladsgroup) [07:12:07] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 15User-Ladsgroup: Initial assessment of TesnorFlow on ORES data - https://phabricator.wikimedia.org/T161376#3130402 (10Ladsgroup) https://github.com/Ladsgroup/TF-on-ORES I spent some time playing with this and here's the result: - Training on models is straig... [10:46:47] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10ORES, 15User-Ladsgroup: deployment-ores-redis /srv/ redis is too small (500MBytes) - https://phabricator.wikimedia.org/T160762#3130646 (10Ladsgroup) Okay. I migrated the redis server from deployment-ores-redis to deployment-ores-redis-01 whi... [10:47:25] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10ORES, 15User-Ladsgroup: deployment-ores-redis /srv/ redis is too small (500MBytes) - https://phabricator.wikimedia.org/T160762#3110103 (10Ladsgroup) 05Open>03Resolved [10:47:27] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10ORES, 13Patch-For-Review, 15User-Ladsgroup: Resurrect ores-beta with production roles - https://phabricator.wikimedia.org/T138445#3130651 (10Ladsgroup) [12:57:56] halfak: Hey, around? [16:55:20] halfak: ping [16:55:21] :D [23:11:34] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:34] RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 9.949 second response time [23:18:34] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:34] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:24:24] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 0.553 second response time [23:52:34] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:24] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 0.542 second response time