[07:26:02] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppet] [07:54:02] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:24:39] 10Scoring-platform-team (Current), 10Wikilabels, 10articlequality-modeling, 10User-Sebastian_Berlin-WMSE, and 2 others: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Sebastian_Berlin-WMSE) [12:17:38] 10Scoring-platform-team (Current), 10Wikilabels, 10articlequality-modeling, 10User-Sebastian_Berlin-WMSE, and 2 others: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Gilles) a:05Halfak→03Gilles [12:18:17] 10Scoring-platform-team (Current), 10Wikilabels, 10articlequality-modeling, 10User-Sebastian_Berlin-WMSE, and 2 others: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Gilles) I'm going to attempt doing this as a side project. Bear in mind that it's the first time I wo... [12:20:45] 10Scoring-platform-team (Current), 10Wikilabels, 10articlequality-modeling, 10User-Sebastian_Berlin-WMSE, and 2 others: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Gilles) p:05Triage→03Normal [12:56:43] 10Scoring-platform-team (Current), 10Wikilabels, 10articlequality-modeling, 10User-Sebastian_Berlin-WMSE, and 2 others: Build article quality model for svwiki - https://phabricator.wikimedia.org/T202202 (10Sebastian_Berlin-WMSE) Thanks a lot, @Gilles! [16:02:08] o/ [16:02:16] * halfak jumps from meeting to meeting [16:03:31] halfak: be careful that can result in injury [16:34:21] Zppix, :P [16:34:23] It's true! [16:34:30] Also it tends to result in not eating lunch [16:34:52] * Zppix gives halfak a subway gift card [16:50:47] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Simplify and modularize the Makefile template - https://phabricator.wikimedia.org/T190968 (10Halfak) @Ladsgroup, I'd love to have your notes on this one. [16:55:26] o/ harej [16:55:40] I scheduled a backlog grooming meeting for us later today. Will that work for you? [16:55:50] * harej is in a meeting technically for the next 5 minutes and then will walk to the officee [16:55:51] And, yse [16:55:58] My keyboard is lots of fun today. [18:00:30] Cool. [18:00:37] Management meeting done. On to lunch [18:00:52] congrats, did you manage the management meeting okay [18:52:43] halfak|Lunch, srrodlund: roan generously wrote some documentation on setting thresholds -- https://www.mediawiki.org/wiki/ORES/RCFilters -- would either of you be interested in discussing this at our meeting? If not, anything else to discuss? [18:59:43] harej, sure! [19:04:28] My usual conf rooms are occupied. Seeking an alternative. [19:05:55] kk I'm in my usual conference room :) [19:06:00] (My home office) [20:37:48] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 6521 bytes in 5.063 second response time [20:37:57] O_O [20:37:58] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 5687 bytes in 5.314 second response time [20:38:06] wait [20:38:07] What? [20:38:44] I was able to get a prediction. [20:38:58] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 973 bytes in 2.445 second response time [20:39:16] hmm [20:40:22] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 5687 bytes in 5.256 second response time [20:40:23] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 2.420 second response time [20:40:55] WMFLabs seems fine too. [20:41:21] The grafana looks fine. [20:41:29] I can't even see these 500s in the plots. [20:41:32] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 2.467 second response time [20:41:38] blip? [20:45:10] 10Scoring-platform-team, 10Analytics: [Discuss] ORES model development and deployment processes - https://phabricator.wikimedia.org/T216246 (10Halfak) Stat machines are not used to deploy to prod for #ores. [21:58:22] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/491624 (owner: 10L10n-bot) [22:00:04] halfak: will be a couple minutes late to backlog grooming [22:00:14] OK! Thanks for letting me know. [22:00:19] Ping here when you are ready OK? [22:06:53] halfak [22:20:20] 10ORES, 10Scoring-platform-team: How far is ORES in training? - https://phabricator.wikimedia.org/T207548 (10Halfak) 05Open→03Declined @TheDJ, let us know if this addresses your question. Please feel free to open if I have missed something. [22:21:16] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, and 2 others: Improve features for wikibase vandalism detection model - https://phabricator.wikimedia.org/T194737 (10Halfak) p:05Triage→03Normal [22:25:16] 10ORES, 10Scoring-platform-team, 10SEO: Improve search-ability of ORES stuff - https://phabricator.wikimedia.org/T179630 (10Halfak) p:05Triage→03Low [22:30:32] 10Scoring-platform-team, 10Operations, 10Release-Engineering-Team (Watching / External): Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10Halfak) p:05High→03Triage [22:30:44] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Halfak) p:05Normal→03High [22:30:51] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Halfak) p:05High→03Normal [22:35:18] 10ORES, 10Scoring-platform-team, 10Operations: [Epic] Deploy ORES in kubernetes cluster - https://phabricator.wikimedia.org/T182331 (10Halfak) p:05Low→03High [22:36:34] 10ORES, 10Scoring-platform-team, 10Operations: Investigate memory usage of ORES in kubernetes - https://phabricator.wikimedia.org/T210264 (10Halfak) [22:42:06] 10ORES, 10Scoring-platform-team, 10Operations: Build helm charts for ORES - https://phabricator.wikimedia.org/T210269 (10Halfak) Is this something that will be part of the common infrastructure for services in kubernetes? [22:42:11] 10ORES, 10Scoring-platform-team, 10Operations: Build helm charts for ORES - https://phabricator.wikimedia.org/T210269 (10Halfak) [22:44:04] 10Jade, 10Scoring-platform-team, 10DBA: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10Halfak) p:05Normal→03Low [22:44:58] 10Jade, 10Scoring-platform-team (Current), 10Accessibility, 10Epic: [Epic] JADE accessibility audit - https://phabricator.wikimedia.org/T209444 (10Halfak) [22:48:20] 10ORES, 10Scoring-platform-team, 10Operations, 10Performance: Stress test ORES/kubernetes (above 4.5k scores/second) - https://phabricator.wikimedia.org/T214054 (10Halfak) p:05Low→03High [22:52:01] harej, still alive? [23:29:05] halfak: interesting that we haven't gotten any traction with Vietnamese Wikipedia since apparently that is a current priority of the Growth team [23:43:21] (I just checked with Marshall and there is no urgent need for a Vietnamese model) [23:47:03] If they have contacts with viwiki, it'd be nice to share :) [23:59:34] halfak: do you remember what Johan and Keegan worked on, precisely? [23:59:50] I vaguely recall Keegan being recruited to consult on Jade, some time before I joined the team.