[00:39:58] (03CR) 10Catrope: "I'm investigating why Special:RC isn't showing anything as having damaging/goodfaith scores on enwiki and this commit looks very suspiciou" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398381 (owner: 10Chad) [00:44:42] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Regression: [wmf.12 -regression] ORES damaging filters do not work - https://phabricator.wikimedia.org/T182936#3839185 (10Catrope) p:05Triage>03Unbreak! [00:47:28] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Regression: [wmf.12 -regression] ORES damaging filters do not work - https://phabricator.wikimedia.org/T182936#3839201 (10Catrope) I captured the main RC query from logstash: ```lang=sql SELECT... [00:48:07] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Regression: [wmf.12 -regression] ORES filters do not work - https://phabricator.wikimedia.org/T182936#3839202 (10Etonkovidova) [00:49:50] (03CR) 10Catrope: Join decomposition of ores_model table queries (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/395811 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [00:54:04] (03PS1) 10Catrope: Follow-up 36748ca523: fix broken join conditions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398397 (https://phabricator.wikimedia.org/T182936) [00:54:14] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Wikimedia-Incident: Extension:ORES caused MW train rollback - https://phabricator.wikimedia.org/T182921#3838777 (10jhsoby) Merge with T182936? [00:55:47] (03CR) 10jerkins-bot: [V: 04-1] Follow-up 36748ca523: fix broken join conditions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398397 (https://phabricator.wikimedia.org/T182936) (owner: 10Catrope) [01:00:08] (03PS2) 10Catrope: Follow-up 36748ca523: fix broken join conditions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398397 (https://phabricator.wikimedia.org/T182936) [01:01:44] (03CR) 10jerkins-bot: [V: 04-1] Follow-up 36748ca523: fix broken join conditions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398397 (https://phabricator.wikimedia.org/T182936) (owner: 10Catrope) [01:03:54] (03PS3) 10Catrope: Follow-up 36748ca523: fix broken join conditions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398397 (https://phabricator.wikimedia.org/T182936) [01:06:33] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Wikimedia-Incident: Extension:ORES caused MW train rollback - https://phabricator.wikimedia.org/T182921#3838777 (10Catrope) They're probably related but I'm not sure if they are. [01:10:01] (03CR) 10Bartosz Dziewoński: [C: 032] "Verified that the variables used in those conditions are always hardcoded strings, not user input. Looks sane." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398397 (https://phabricator.wikimedia.org/T182936) (owner: 10Catrope) [01:11:49] (03Merged) 10jenkins-bot: Follow-up 36748ca523: fix broken join conditions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398397 (https://phabricator.wikimedia.org/T182936) (owner: 10Catrope) [01:14:25] (03PS1) 10Catrope: Follow-up 36748ca523: fix broken join conditions [extensions/ORES] (wmf/1.31.0-wmf.12) - 10https://gerrit.wikimedia.org/r/398401 (https://phabricator.wikimedia.org/T182936) [01:14:34] (03CR) 10Catrope: [C: 032] Follow-up 36748ca523: fix broken join conditions [extensions/ORES] (wmf/1.31.0-wmf.12) - 10https://gerrit.wikimedia.org/r/398401 (https://phabricator.wikimedia.org/T182936) (owner: 10Catrope) [01:16:00] RoanKattouw: oooh it’s a field name and can’t be escaped :( [01:16:14] (03Merged) 10jenkins-bot: Follow-up 36748ca523: fix broken join conditions [extensions/ORES] (wmf/1.31.0-wmf.12) - 10https://gerrit.wikimedia.org/r/398401 (https://phabricator.wikimedia.org/T182936) (owner: 10Catrope) [01:16:57] awight: Exactly. Swatting the fix now [01:17:17] Elena had brought it to my attention earlier, but I didn't start looking until I was already SWATting, so I added this on [01:19:22] I’m doing the reviewer walk of shame right now... [01:19:39] Pretty crazy that the tests passed. [01:25:09] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review, 10Regression: [wmf.12 -regression] ORES filters do not work - https://phabricator.wikimedia.org/T182936#3839273 (10Catrope) Patch deployed, should be fixed now. [01:25:17] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review, 10Regression: [wmf.12 -regression] ORES filters do not work - https://phabricator.wikimedia.org/T182936#3839276 (10Catrope) a:03Catrope [01:25:28] awight: thats why i couldnt use ores on enwiki :/ [01:25:35] I thought it was me [01:29:09] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review, 10Regression: [wmf.12 -regression] ORES filters do not work - https://phabricator.wikimedia.org/T182936#3839285 (10Etonkovidova) Checked the fix in enwiki (wmf.12) - ORES filt... [01:29:20] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review, 10Regression: [wmf.12 -regression] ORES filters do not work - https://phabricator.wikimedia.org/T182936#3839286 (10Etonkovidova) 05Open>03Resolved [01:56:06] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter): Tests should have covered regression in T182936 - https://phabricator.wikimedia.org/T182942#3839304 (10awight) [06:50:37] PROBLEM - check load on ORES-web05.experimental is WARNING: WARNING - load average: 16.03, 5.37, 1.89 [06:51:34] RECOVERY - check load on ORES-web05.experimental is OK: OK - load average: 6.02, 4.43, 1.78 [09:14:40] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10User-Ladsgroup: Tests should have covered regression in T182936 - https://phabricator.wikimedia.org/T182942#3839697 (10Ladsgroup) a:03Ladsgroup [09:26:20] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Wikimedia-Incident: Extension:ORES caused MW train rollback - https://phabricator.wikimedia.org/T182921#3838777 (10Ladsgroup) All wikis are in wmf.12, there is no such error in fatalmonitor. I want to fix the issue but I have no idea what this ticket is... [14:19:06] o/ [14:19:24] I'm out sick again today. Will re assess this afternoon (my time) [14:19:38] So... Like 4 hours from now. I might be back [14:27:01] get well soon! [14:28:24] Get well soon! [14:43:58] :( [19:13:55] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 2 others: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3841366 (10thcipriani) [19:29:21] o/ [19:31:28] Hi [19:33:03] hey. I'm feeling a bit better, but still not 100%. I figure it's probably not hurting me to be around for a couple of hours. [19:33:31] Maybe you should take next week off Aaron [19:33:40] Speaking of which... codezee, do you need anything? [19:33:48] Zppix, yeah, I'm due. [19:34:10] I'll be taking the week of the 25-29th off because of WMF holiday. [19:34:18] Good [19:34:34] Take some asprin and have some soup or tea [19:34:57] (Cucumber water helps too i find) [19:38:45] halfak: not right now, i'll be wrapping up the time issue this weekend and look at the labels-config patch...don't think any blocker is expected :) [19:38:50] get well soon... [19:39:17] Zppix: I'm working on some Chicken Pho and it's really nice ^_^ [19:39:33] codezee, OK great. And thanks :) I'll do my best [19:41:29] If you come back next week and arent well i will be emailing your boss halfak [19:41:43] :P [19:41:47] Fair point. If it's next week and I'm not well, I'll be visiting a doc. [21:10:05] OK emails done and it seems like everyone has what they need so I'm going to go lay down again. [21:10:18] If I don't catch you before you leave today, have a good one! [21:10:47] Ok you too [21:10:52] Oh! And I expect to be online around 1600 UTC tomorrow for hacking on some ORES stuff. [21:11:01] Ill be here so if anyone comes by ill handle them [21:11:02] Hopefully I'll be feeling up to it! [21:11:13] Thanks for letting people know I'm AFK for sick, Zppix [21:11:24] I saw you let awight know yesterday and I appreciate that :) [21:11:51] No problem [21:12:18] Get better