[00:33:14] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 99 MB (0% inode=91%);
[02:53:18] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 99 MB (0% inode=91%);
[05:13:22] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 99 MB (0% inode=91%);
[07:33:26] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 112 MB (0% inode=91%);
[09:49:08] <wikibugs>	 (03CR) 10jenkins-bot: build: Updating mediawiki/mediawiki-phan-config to 0.6.1 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/516174 (owner: 10Libraryupgrader)
[09:53:29] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 112 MB (0% inode=91%);
[12:13:32] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 112 MB (0% inode=91%);
[13:44:13] <halfak>	 o/
[14:02:49] <wikibugs>	 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Serbian-Sites, 10artificial-intelligence: Investigate srwiki goodfaith model, why is it so bad? - https://phabricator.wikimedia.org/T199355 (10Halfak) It looks like we're going to get this deployed next week.  I'm aiming for Monda...
[14:33:36] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 112 MB (0% inode=91%);
[16:10:40] <wikibugs>	 10ORES, 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Fit models for revert prediction - https://phabricator.wikimedia.org/T224902 (10Halfak)
[16:16:49] <wikibugs>	 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Qualitative data collection for ores bias analysis - https://phabricator.wikimedia.org/T225441 (10Groceryheist)
[16:53:40] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 111 MB (0% inode=91%);
[19:13:43] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 111 MB (0% inode=91%);
[19:22:13] <groceryheist>	 halfak_: I'm getting errors from wmreverts
[19:22:35] <halfak_>	 Some errors are expected.  What kind of errors are you getting?
[19:24:01] <groceryheist>	 https://tools.wmflabs.org/paste/view/aca3f5df
[19:24:21] <groceryheist>	 None in user.id i think
[19:25:07] <groceryheist>	 so maybe an issue with mwtypes or we should allow it to pass through the exception?
[19:27:39] <groceryheist>	 this is from eswikibooks
[19:34:41] <halfak_>	 Oh weird!  You shouldn't have a user field at all if there's no user ID.  WTF
[19:35:01] <halfak>	 What the heck, XML DUMP.  YOU'RE DRUNK.
[19:36:04] <groceryheist>	 lol
[19:36:22] <groceryheist>	 I don't think this is the only one
[19:36:50] <groceryheist>	 i could go back and look at the last batch of dumps
[19:40:56] <groceryheist>	 yeah maybe it's a problem with the may dumps
[19:45:06] <groceryheist>	 * may dumps
[19:45:13] <groceryheist>	 should we make a bug on this?
[19:57:57] <groceryheist>	 well this is a setback. It doesn't totally block me from working on finding "undo" reverts, but it's going slow me down a bit.  
[20:06:30] <groceryheist>	 halfak: yeah does seem like an issue with the may dumps
[20:06:35] <groceryheist>	 hope it doesn't appear again in June
[20:07:25] <halfak>	 Ha.  Weird indeed.  I think we should handle this in mwxml if the dumps are where the weirdness is. 
[20:07:40] <halfak>	 Please do file a bug and try to beat me to making a patch :) 
[20:08:52] <groceryheist>	 ah so you think its easier or better to fix the parser rather than the dump generator?  
[20:09:35] <groceryheist>	 i've been hunting for the change in xmldumper
[20:13:56] <apergos>	 the user i being missing is a known issue
[20:14:17] <apergos>	 it has to do with the corresponding field in the actor table now being null
[20:14:54] <apergos>	 a fix for mw core was already merged and should have been deployed to all wikis
[20:15:30] <apergos>	 https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/512672/ 
[20:15:44] <apergos>	 https://phabricator.wikimedia.org/T224221
[20:16:12] <apergos>	 other code thta relies on uid 0 in the user table may now still get fooled by those nulls but the main code path for dumps will be ok
[20:16:21] <apergos>	 halfak:  groceryheist
[20:16:55] <halfak>	 Thanks apergos :) 
[20:16:58] <apergos>	 for now you should make your parser handle empty uids as a 0
[20:17:05] <apergos>	 yw
[20:17:30] <halfak>	 I'm so happy you're hanging out in this channel ^_^
[20:17:35] <apergos>	 stealth schema changes like these are hard to work around in advance
[20:17:39] <apergos>	 too many edge cases
[20:17:46] <apergos>	 anywys, happy I knew what this was
[20:18:11] <apergos>	 I'm gone again (was just doing one more peek in before disappearing for the evening)
[20:18:16] <apergos>	 have a good rest of your day!
[20:27:24] <halfak>	 you too!
[20:47:19] <groceryheist>	 apergos: thank you so much!
[20:47:26] <groceryheist>	 this was extremely helpful!
[21:33:46] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 111 MB (0% inode=91%);
[22:45:26] <groceryheist>	 halfak: sent a pull request
[23:53:49] <icinga2-wm>	 PROBLEM - check disk on ORES-worker01.experimental is CRITICAL: DISK CRITICAL - free space: / 111 MB (0% inode=91%);