[01:23:01] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Qualitative data collection for ores bias analysis - https://phabricator.wikimedia.org/T225441 (10Groceryheist) I'm making a list of people who helped with labeling campaigns for the different ores projects. @Le... [02:13:53] PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 111 MB (0% inode=91%); [04:33:57] PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 111 MB (0% inode=91%); [05:00:22] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 6 minutes ago with 26 failures. Failed resources (up to 3 shown): Package[ntp],Service[systemd-timesyncd],Service[prometheus-node-exporter-ipmitool-sensor.timer],Service[prometheus-node-exporter-smartmon.timer] [05:24:40] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:17] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:54:00] PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 112 MB (0% inode=91%); [07:46:17] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:56:43] 10ORES, 10Scoring-platform-team, 10Operations, 10serviceops: Migrate ORES Redis servers to Stretch/Buster - https://phabricator.wikimedia.org/T224569 (10ArielGlenn) p:05Triage→03Normal [09:14:03] PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 112 MB (0% inode=91%); [11:34:07] PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=91%); [12:43:58] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Qualitative data collection for ores bias analysis - https://phabricator.wikimedia.org/T225441 (10Lea_Lacroix_WMDE) Hey! I don't have a lot of information about who helped with the labeling campaigns, but I think... [13:54:10] PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=91%); [15:35:42] scoring async. i was sidetracked by tech engagement stuff yesterday; will resume work on jade. crucially, i got some nice documentation from carolyn about her work with wikipedia editing on mobile that i hope to incorporate into this work with Jade [15:38:06] yesterday I realized that the reverts I extracted were bad because of a change to the dumps schema [15:38:14] i patched mwxml to fix the issue [15:38:19] sent halfak a pull request [15:38:28] i also had a great meeting with Abbey [15:38:43] I think next steps on the qualitative side is to reach out to confederates [15:39:04] for some exploratory conversations [15:39:30] we talked about doing a screening survey as well, which also sounds like a good idea [15:39:44] I also trawled through phab looking for confederates [15:40:01] I didn't find some for all the wikis through phab [15:40:24] Do you think I could take a look into the wikilabels data? [15:47:33] that should be fine but I don't think that's my call [15:48:11] do you need a list of confederates to reach out to? [15:53:12] to find yeah if you have it handy that would be great [15:54:16] so today I'm going to finish that and post messages on the confederate's user talk pages [15:54:30] I'd also like a list of bot developers, but that is less urgent [15:55:28] I also think enough of the dumps are extracted that today I'll get to work identifying undos [15:58:21] https://docs.google.com/spreadsheets/d/1rWWUdBEasPzbbVch18rvjjW5APlfdy9MKFOoeE-c56Q/edit?usp=sharing [16:09:26] * harej grumbles about ACLs [16:11:10] groceryheist, you can look for bot accounts using the user_groups table and then looking for the maintainer information on the bot's user page. [16:11:34] I'm going to be working on hiring documents for our associate SWE position today and buttoning jawiki's models into our next ORES deployment. [16:12:24] Anyway, I think tech has big potential to make the case for extreme value to the org with minimal resources. So we have fertile material for developing a solid case against "whiners". [16:12:44] I think historically and culturally, Tech is also bad at politics and thus finds itself in a bad position often. [16:13:19] "Servant leadership" requires a different personality than power gathering requires. [16:13:30] Woops. wrong channel. [16:13:50] Nevermind all that ^_^. I'm thinking about org politics and tech dept :| [16:14:14] PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=91%); [17:25:40] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 4686 bytes in 0.027 second response time https://wikitech.wikimedia.org/wiki/ORES [17:26:00] Doing a deploy. Don't worry about ^ [17:28:46] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 4686 bytes in 0.023 second response time https://wikitech.wikimedia.org/wiki/ORES [17:28:55] Oh hush. [17:29:30] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 4686 bytes in 0.040 second response time https://wikitech.wikimedia.org/wiki/ORES [17:34:08] what channel is that servant leadership discussion in? because we need more of that imho [17:38:45] Ha. Agreed. :) [17:39:00] RECOVERY - check disk on ORES-worker01.experimental is OK: DISK OK [17:39:50] Yeah. So it looks like maybe we need to add a "git lfs prune" to our deployments. [17:48:03] Alright that seems easy enough. [17:48:15] I deleted an old core dump that was taking up 2.5GB [17:50:22] woah [17:50:24] nice find [17:51:20] Alright and the prune opened up another 1GB [17:51:28] \o/ [17:51:31] Simple enough. [17:52:45] sweet [18:03:14] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, 10Wikimedia-production-error: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10JTannerWMF) @Catrope will investigate [18:06:00] (03PS2) 10Halfak: Updates to srwiki, nlwiki, zhwiki, jawiki, and wheels [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/514963 (https://phabricator.wikimedia.org/T224484) [18:06:32] (03CR) 10Halfak: [V: 03+2 C: 03+2] "I tested this on ores-wmflabs-deploy and the config works as intended." [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/514963 (https://phabricator.wikimedia.org/T224484) (owner: 10Halfak) [18:25:51] (03PS1) 10Halfak: Adds jawiki damaging/goodfaith [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/516532 [18:26:05] (03CR) 10Halfak: [V: 03+2 C: 03+2] Adds jawiki damaging/goodfaith [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/516532 (owner: 10Halfak) [18:29:37] Looks like we have a version issue in the ORES deploy. [18:29:41] I'm looking into it. [18:46:01] (03PS1) 10Halfak: Fixes redis version issue and updates Flask [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/516533 [18:46:48] (03CR) 10Halfak: [V: 03+2 C: 03+2] Fixes redis version issue and updates Flask [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/516533 (owner: 10Halfak) [18:50:18] 10Scoring-platform-team, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Backlog), 10artificial-intelligence: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10thcipriani) Looks like master is showing a newer version: https://gerrit.wikimed... [18:53:50] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.162 second response time https://wikitech.wikimedia.org/wiki/ORES [18:55:36] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 5.782 second response time https://wikitech.wikimedia.org/wiki/ORES [18:57:38] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 0.108 second response time https://wikitech.wikimedia.org/wiki/ORES [19:00:05] (03PS1) 10Halfak: Updates requirements and wheels for redis 3.2.x [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/516535 [19:01:08] 10Scoring-platform-team, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Backlog), 10artificial-intelligence: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10Halfak) Right. I did a manual update with @20after4's help to get unblocked for... [19:01:35] (03CR) 10Halfak: [V: 03+2 C: 03+2] Updates requirements and wheels for redis 3.2.x [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/516535 (owner: 10Halfak) [19:10:17] \o/ We look good in beta. [19:10:25] So we're ready to deploy right away on Monday. [19:14:14] sweet [19:21:57] 10Scoring-platform-team, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Backlog), 10artificial-intelligence: Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10thcipriani) gotcha. My theory here is that there is some ssh timeout on the phab... [19:31:41] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Update ORES thresholds for nlwiki - https://phabricator.wikimedia.org/T225561 (10Halfak) [19:32:49] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Deploy ORES filters for zhwiki - https://phabricator.wikimedia.org/T225562 (10Halfak) [19:33:50] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Deploy ORES filters for jawiki - https://phabricator.wikimedia.org/T225563 (10Halfak) [19:34:56] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: ORES deployment: Early June - https://phabricator.wikimedia.org/T224484 (10Halfak) [19:36:06] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Re-train nlwiki models (better language assets) - https://phabricator.wikimedia.org/T224483 (10Halfak) [19:36:08] 10Scoring-platform-team (Current), 10editquality-modeling, 10Chinese-Sites, 10artificial-intelligence: Train/test zhwiki editquality models - https://phabricator.wikimedia.org/T224481 (10Halfak) [19:36:12] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Train/test edit quality models for jawiki - https://phabricator.wikimedia.org/T130288 (10Halfak) [19:36:14] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Serbian-Sites, 10artificial-intelligence: Investigate srwiki goodfaith model, why is it so bad? - https://phabricator.wikimedia.org/T199355 (10Halfak) [19:36:16] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: ORES deployment: Early June - https://phabricator.wikimedia.org/T224484 (10Halfak) [19:36:53] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: ORES deployment: Early June - https://phabricator.wikimedia.org/T224484 (10Halfak) [19:37:01] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: ORES deployment: Early June - https://phabricator.wikimedia.org/T224484 (10Halfak) [19:37:03] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Deploy ORES filters for jawiki - https://phabricator.wikimedia.org/T225563 (10Halfak) [19:37:21] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: ORES deployment: Early June - https://phabricator.wikimedia.org/T224484 (10Halfak) [19:37:23] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Deploy ORES filters for zhwiki - https://phabricator.wikimedia.org/T225562 (10Halfak) [19:37:45] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: ORES deployment: Early June - https://phabricator.wikimedia.org/T224484 (10Halfak) [19:37:47] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10editquality-modeling, 10artificial-intelligence: Update ORES thresholds for nlwiki - https://phabricator.wikimedia.org/T225561 (10Halfak) [19:37:51] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: ORES deployment: Early June - https://phabricator.wikimedia.org/T224484 (10Halfak) [19:38:14] 10Scoring-platform-team, 10Growth-Team: Update srwiki thresholds for goodfaith model - https://phabricator.wikimedia.org/T223273 (10Halfak) [21:01:11] 10Scoring-platform-team, 10Analytics, 10EventBus: Produce a ORES score event for HDFS - https://phabricator.wikimedia.org/T225210 (10Harej) p:05Triage→03Normal [21:01:38] 10ORES, 10Scoring-platform-team, 10Growth-Team, 10WMF-JobQueue, 10Wikimedia-production-error: Fatal error during RecentChange::notifyEdit (deferred update) from ORES/RecentChangeSaveHookHandler - https://phabricator.wikimedia.org/T225199 (10Harej) p:05Triage→03High [21:02:32] 10ORES, 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Fit models for revert prediction - https://phabricator.wikimedia.org/T224902 (10Harej) p:05Triage→03Normal [23:32:18] wow that was a lot of spam