[05:58:35] PROBLEM - puppet on ORES-worker01.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:26:35] RECOVERY - puppet on ORES-worker01.experimental is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:55:42] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 2 minutes ago with 24 failures. Failed resources (up to 3 shown): Package[ntp],Service[systemd-timesyncd],Service[prometheus-node-exporter-ipmitool-sensor.timer],Service[prometheus-node-exporter-smartmon.timer] [07:20:36] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 3 minutes ago with 24 failures. Failed resources (up to 3 shown): Package[ntp],Service[systemd-timesyncd],Service[prometheus-node-exporter-ipmitool-sensor.timer],Service[prometheus-node-exporter-smartmon.timer] [07:23:42] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:46:35] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:09:04] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), 10Patch-For-Review, 10User-Ladsgroup: Fix oresBaseUrl config variable in frontend - https://phabricator.wikimedia.org/T219396 (10Ladsgroup) I need to set this variable in production after it got de... [12:11:01] 10ORES, 10Scoring-platform-team: ores icinga check for grafana alert returns 404 - https://phabricator.wikimedia.org/T221618 (10fgiunchedi) [13:00:34] 10ORES, 10Scoring-platform-team: ores icinga check for grafana alert returns 404 - https://phabricator.wikimedia.org/T221618 (10Halfak) This is probably due to some changes to dashboards that @Ladsgroup recently made. https://grafana.wikimedia.org/d/vAN_bQemz/ores-advanced-metrics is now the right board to... [13:07:27] 10ORES, 10Scoring-platform-team: ores icinga check for grafana alert returns 404 - https://phabricator.wikimedia.org/T221618 (10Halfak) I just manually added back two that I knew we had. 1. Alert on overloads above 5 per minute 2. Alert on CPU usage above 80% for 5 minutes [13:20:58] https://www.forbes.com/sites/kalevleetaru/2019/04/17/could-ai-create-a-super-wikipedia-from-all-of-recorded-history/amp/ [13:26:42] Seems like the author invents a problem (lack of connectivity between Wikipedia articles) and then proceeds to a solution. [13:27:11] But it's not clear at all that Wikipedia is too sparsely linked or that infoboxes don't generally exist for the most important content. [13:31:51] didn't you like the idea of replacng volunteers with ai? [13:33:49] oh wait, that's another recent article... [13:51:38] 10Jade, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 3 others: Introduce a new namespace for collaborative judgements about wiki entities - https://phabricator.wikimedia.org/T200297 (10jcrespo) @Harej My question is more like, is the summary still accurate about the result of the conversatio... [14:31:12] 10ORES, 10Scoring-platform-team, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, and 10 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Jdlrobson) In https://gerrit.wikimedia.org/r/#/c/mediawiki... [15:01:29] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Move fiwiki from custom to config-based Makfile - https://phabricator.wikimedia.org/T221640 (10Halfak) [15:01:47] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Move fiwiki from custom to config-based Makfile - https://phabricator.wikimedia.org/T221640 (10Halfak) p:05Triage→03Normal [15:30:15] o/ hare [15:30:31] I won't be in standup because of tech management meeting. [15:33:40] i'm here [15:33:56] Greetings [15:34:19] Idea: async standup on Tuesdays, done on IRC [15:34:29] I can ping y'all and you can leave updates For The Record so people can read later [15:34:45] okay [15:34:56] What do you think halfak? [15:35:18] Sure! I added some stuff already. But it's not complete. [15:35:44] I'm learning the complexities and asks for annual planning now so no brain cycles to spare. [15:37:22] good luck, my sympathies [15:37:33] Thank you. It's all pain. ^_^ [15:44:05] My update is... what I wrote on Current Work. https://etherpad.wikimedia.org/p/scoring_current_work [15:50:21] 10ORES, 10Scoring-platform-team: ores icinga check for grafana alert returns 404 - https://phabricator.wikimedia.org/T221618 (10Harej) p:05Triage→03Normal [15:51:22] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for Indonesian language Wikipedia - https://phabricator.wikimedia.org/T221542 (10Harej) p:05Triage→03Normal [15:55:02] halfak: now that the backlog is cleaned up, I would like to propose replacing our standing meeting at 4pm CDT with me blocking out time on my calendar to do the work each week (so as to ensure it gets done) and if there are tricky questions I can reach you on IRC. How does that sound? [15:55:32] That sounds great hare. I was going to make a similar suggestion. [15:55:50] It's still good for us to do regular backlog grooming together, but maybe every other week would be better. [15:56:02] Or even less frequently. [15:57:22] Sure. I think this grooming would focus on moving things between columns, rather than prioritizing [15:57:38] Like, getting things out of the "blocked on community" column, moving them to "Ready to Go," things like that [16:08:47] I entered my update [16:09:02] curious to see what folks think about the plots [16:10:10] 10ORES, 10Scoring-platform-team, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, and 10 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10hashar) >>! In T199939#5130664, @Jdlrobson wrote: > In htt... [16:11:42] 10ORES, 10Scoring-platform-team, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, and 10 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10hashar) Adding Minerva is T202030 [16:30:20] nick halfak|Lunch [16:30:30] Woops [16:30:46] hare, +1 to your last note [16:31:33] groceryheist, I'll hopefully find time to look at plots today. No promises. This annual planning cycle is the worst from a time consumption perspective. [16:31:52] halfak|Lunch: okay [16:34:03] Bah. Just looking while the microwave goes [16:34:14] Is "normal" everyone? Or non-newcomer/anon? [16:34:28] Arabic and Hebrew are weird. [16:40:02] normal is non-newcomer/anon [16:40:29] yeah about ar and he i'm going to investigate that closely while I check the work today [17:02:06] 10Scoring-platform-team, 10Wikilabels, 10cloud-services-team (Kanban): Move wikilabels DB to it's own instance with replica - https://phabricator.wikimedia.org/T221673 (10Bstorm) 05Open→03Stalled p:05Triage→03Normal [18:46:53] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10Analytics, 10Core Platform Team Backlog (Designing), and 3 others: ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10Pchelolo) a:05Ladsgroup→03Pchelolo [20:17:24] Looks like ORESModels is dead. [20:17:25] https://en.wikipedia.org/wiki/Special:ORESModels [20:17:44] groceryheist, check out [[mw:ORES/Thresholds]] [20:17:44] 10[1] 10https://www.mediawiki.org/wiki/ORES/Thresholds [20:17:52] Thanks AsimovBot. [20:18:00] You're not often useful. [20:18:21] haha [20:19:30] 10MediaWiki-extensions-ORES, 10Scoring-platform-team: ORESModels has a fatal error - https://phabricator.wikimedia.org/T221696 (10Halfak) [20:19:42] 10MediaWiki-extensions-ORES, 10Scoring-platform-team: ORESModels has a fatal error - https://phabricator.wikimedia.org/T221696 (10Halfak) p:05Triage→03High [20:21:53] i'd like to maybe use thresholds used in rcfilters [20:37:00] groceryheist, right. I want to get you that but it's the ORESModels thing I just filed a bug about :( [20:37:17] We can find it in the config, but I can't remember where to look. [20:38:39] 10MediaWiki-extensions-ORES, 10Scoring-platform-team: ORESModels has a fatal error - https://phabricator.wikimedia.org/T221696 (10Halfak) @Catrope, I figured you might know what's up. [20:40:32] oh crapola [20:46:51] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Wikimedia-production-error: ORESModels has a fatal error - https://phabricator.wikimedia.org/T221696 (10Krinkle) [21:39:28] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Wikimedia-production-error: ORESModels has a fatal error - https://phabricator.wikimedia.org/T221696 (10Reedy) [21:40:55] (03PS1) 10Reedy: Fix path to template dir [extensions/ORES] - 10https://gerrit.wikimedia.org/r/506007 (https://phabricator.wikimedia.org/T221696) [21:42:58] (03CR) 10Alex Monk: [C: 03+2] Fix path to template dir [extensions/ORES] - 10https://gerrit.wikimedia.org/r/506007 (https://phabricator.wikimedia.org/T221696) (owner: 10Reedy) [21:46:35] i'm just going to choose a threshhold where fpr = tpr [21:47:25] sorry where fnr = fpr [21:47:27] (03PS1) 10Reedy: Fix path to template dir [extensions/ORES] (wmf/1.34.0-wmf.1) - 10https://gerrit.wikimedia.org/r/506009 (https://phabricator.wikimedia.org/T221696) [21:47:42] (03CR) 10Reedy: [C: 03+2] Fix path to template dir [extensions/ORES] (wmf/1.34.0-wmf.1) - 10https://gerrit.wikimedia.org/r/506009 (https://phabricator.wikimedia.org/T221696) (owner: 10Reedy) [21:47:44] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Patch-For-Review, 10Wikimedia-production-error: ORESModels has a fatal error - https://phabricator.wikimedia.org/T221696 (10Krinkle) Culprit was {94634a01ae}. Thanks @Reedy and @Krenair for the quick action. [21:50:13] (03Merged) 10jenkins-bot: Fix path to template dir [extensions/ORES] - 10https://gerrit.wikimedia.org/r/506007 (https://phabricator.wikimedia.org/T221696) (owner: 10Reedy) [21:51:51] (03Merged) 10jenkins-bot: Fix path to template dir [extensions/ORES] (wmf/1.34.0-wmf.1) - 10https://gerrit.wikimedia.org/r/506009 (https://phabricator.wikimedia.org/T221696) (owner: 10Reedy) [21:52:11] (03CR) 10jenkins-bot: Fix path to template dir [extensions/ORES] - 10https://gerrit.wikimedia.org/r/506007 (https://phabricator.wikimedia.org/T221696) (owner: 10Reedy) [21:54:15] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Wikimedia-production-error: ORESModels has a fatal error - https://phabricator.wikimedia.org/T221696 (10Reedy) 05Open→03Resolved a:03Reedy [22:52:23] (03CR) 10jenkins-bot: Fix path to template dir [extensions/ORES] (wmf/1.34.0-wmf.1) - 10https://gerrit.wikimedia.org/r/506009 (https://phabricator.wikimedia.org/T221696) (owner: 10Reedy) [22:54:17] (03CR) 10jenkins-bot: Fix path to template dir [extensions/ORES] - 10https://gerrit.wikimedia.org/r/506007 (https://phabricator.wikimedia.org/T221696) (owner: 10Reedy)