[00:04:03] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 3 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3776445 (10awight) p:05Unbreak!>03High Reducing the priority, we need to reenable ORES on th... [00:05:44] 10Scoring-platform-team, 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776447 (10awight) [00:06:01] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776331 (10awight) [00:28:07] 10Scoring-platform-team: Contribution page: 'Hide probably good edits' displays entries that do not get ORES highlighted - https://phabricator.wikimedia.org/T176667#3776477 (10jmatazzoni) [00:28:31] 10Scoring-platform-team, 10Collaboration-Team-Triage: Contribution page: 'Hide probably good edits' displays entries that do not get ORES highlighted - https://phabricator.wikimedia.org/T176667#3633304 (10jmatazzoni) [00:52:29] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3776331 (10Legoktm) RecentChanges is one of the core features that basically can't be unavailabl... [00:58:29] (03PS1) 10Legoktm: Add missing GPL file headers [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392548 [01:00:05] (03CR) 10Krinkle: [C: 031] "LGTM per "license-name": "GPL-3.0+" in extension.json." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392548 (owner: 10Legoktm) [01:00:10] (03CR) 10Krinkle: [C: 032] Add missing GPL file headers [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392548 (owner: 10Legoktm) [01:01:57] (03Merged) 10jenkins-bot: Add missing GPL file headers [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392548 (owner: 10Legoktm) [01:03:40] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3776555 (10Legoktm) [01:13:52] (03PS1) 10Legoktm: Add 2 more GPL file headers [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392549 [01:13:54] (03PS1) 10Legoktm: Use Wikimedia\Rdbms\IDatabase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392550 [01:14:27] (03CR) 10Krinkle: [C: 032] Add 2 more GPL file headers [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392549 (owner: 10Legoktm) [01:14:41] (03CR) 10Krinkle: [C: 032] Use Wikimedia\Rdbms\IDatabase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392550 (owner: 10Legoktm) [01:15:26] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3776555 (10Catrope) It would be great if there was. We... [01:16:14] (03Merged) 10jenkins-bot: Add 2 more GPL file headers [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392549 (owner: 10Legoktm) [01:16:22] (03Merged) 10jenkins-bot: Use Wikimedia\Rdbms\IDatabase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392550 (owner: 10Legoktm) [01:20:05] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3776599 (10Legoktm) >>! In T181015#3776590, @Catrope w... [08:39:14] 10Scoring-platform-team (Current), 10DBA, 10MediaWiki-extensions-ORES, 10Blocked-on-schema-change, 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3777003 (10Marostegui) I have dropped the index on `enwiki`on db1089... [10:27:01] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, and 2 others: [regression] ORES filters are not available on French Wikipedia anymore - https://phabricator.wikimedia.org/T180115#3777263 (10Trizek-WMF) 05Resolved>03Open Re-openning.... [10:54:37] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, and 2 others: [regression] ORES filters are not available on French Wikipedia anymore - https://phabricator.wikimedia.org/T180115#3747154 (10Zppix) @trizek-wmf see T181006 we had to temp d... [11:00:08] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, and 2 others: [regression] ORES filters are not available on French Wikipedia anymore - https://phabricator.wikimedia.org/T180115#3777370 (10Ladsgroup) 05Open>03Resolved [11:00:29] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, and 2 others: [regression] ORES filters are not available on French Wikipedia anymore - https://phabricator.wikimedia.org/T180115#3747154 (10Ladsgroup) Sorry but that's another issue. I wi... [11:07:59] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, and 2 others: [regression] ORES filters are not available on French Wikipedia anymore - https://phabricator.wikimedia.org/T180115#3777409 (10Trizek-WMF) Thank you guys! [11:15:22] Amir1: good morning? [11:45:15] Zppix: hey [11:45:18] what's up [14:57:39] o/ [14:57:47] * halfak looks at ORES graphs [14:58:14] Looks like we had a 3 hour long event [14:58:27] https://grafana.wikimedia.org/dashboard/db/ores?panelId=23&fullscreen&orgId=1&from=1511250774443&to=1511276209577 [15:00:15] Everything looks OK now. [15:03:37] (03PS3) 10Petar.petkovic: [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) [15:04:42] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) (owner: 10Petar.petkovic) [15:07:45] (03PS4) 10Petar.petkovic: [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) [15:09:18] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Split WL and RC prefs for ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392452 (https://phabricator.wikimedia.org/T180866) (owner: 10Petar.petkovic) [15:17:23] Cross-posting: I'm trying to miss work today, and work Friday. [15:27:29] awight: everything ok? [15:27:45] Zppix: Thank you sir, may I have another! [15:28:10] halfak: What’s this about a 3-hour event? I looked at the graph but don’t understand how long the averaging window is. [15:28:33] It looks more like we had 3 50x errors. [15:29:51] That's possible. [15:29:59] Seems like we have a 3 hour window :) [15:30:02] lol [15:30:19] 3 hr window for what, sounds fun. [15:31:07] 3 hr window for counting up 500s [15:31:22] Need help? [15:31:34] nope [15:32:24] I needa get some closure on the error graphs. [15:32:25] alias(movingAverage(transformNull(scaleToSeconds(changeprop.internal-startup.ores-cache-1-retry_exec.count, 60)), 180) [15:33:26] reading… http://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.movingAverage [15:35:11] IMO that 180 was supposed to be 180 seconds, but it’s actually 180 data points. So what’s the X resolution...? [15:35:27] We should just say “3min” and not have to guess. [15:35:48] Anyone remember offhand which project has our checked-in grafana json? [15:36:04] Isnt it on github [15:36:16] Its grafana-dashboard iirc awight [15:36:28] Zppix: <3 thanks [15:36:35] Np [15:38:09] awight, I'm not sure that is maintained :S [15:38:32] 10-4, I think Amir1 and I just use it as a place to dump current code in case we need to roll back. [15:38:49] cool [15:38:55] awight: yes, that's correct [15:39:03] o/ [15:39:12] I didn't know that was still used [15:39:14] hey! [15:39:15] o/ Amir1 :) [15:42:32] Amir1: Um, once I make a change to a Grafana metric, how do I save it? [15:43:18] k I think I have it. [15:50:06] That was bothering the turds out of me! [15:50:18] 5xx and “scores errored” moving windows are fixed. [15:55:01] 10Scoring-platform-team (Current): Grafana has confusing or wrong scale for "scores errored" graph - https://phabricator.wikimedia.org/T175651#3778253 (10awight) [15:55:43] 10Scoring-platform-team (Current): Grafana has confusing or wrong scale for "scores errored" graph - https://phabricator.wikimedia.org/T175651#3599004 (10awight) 05Open>03Resolved [15:55:48] gtg [16:14:50] We need that incident report. [16:14:56] I wonder if awight started one. [16:16:35] halfak: i know he was talking about it last night [16:16:42] But im not 100% sure [16:17:07] Looks like awight hasn't been editing wikitech. [16:18:18] halfak: maybe hes looking at grafana to write the documentation? And is writing elsewhere who knows, he sounds like hes busy today though [16:24:09] (03PS1) 10Halfak: Parallelize scap but preserve serial restarts. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392671 [16:24:57] 10Scoring-platform-team (Current): Grafana has confusing or wrong scale for "scores errored" graph - https://phabricator.wikimedia.org/T175651#3778393 (10awight) @Halfak I see what you mean. The moving window was my biggest gripe, but we can reopen if the the graphs are hard to interpret the next time we need t... [16:25:10] 10Scoring-platform-team (Current), 10ORES: Parallelize scap for ORES - https://phabricator.wikimedia.org/T181067#3778394 (10Halfak) [16:25:20] 10Scoring-platform-team (Current), 10ORES: Parallelize scap for ORES - https://phabricator.wikimedia.org/T181067#3778407 (10Halfak) a:03Halfak https://gerrit.wikimedia.org/r/392671 [16:25:34] (03PS2) 10Halfak: Parallelize scap but preserve serial restarts. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392671 (https://phabricator.wikimedia.org/T181067) [16:25:56] halfak: Why keep restarts serial? [16:26:11] So the whole service doesn't go down at once. [16:26:14] Also, what do you think about the pip reinstall [16:26:22] awight, what? [16:26:22] :) that sounds solid. [16:26:55] so we pip purge then reinstall the wheels, and it happens during the last step of scap (I forget exactly what the logged line calls that) [16:27:11] It takes a non-zero amount of time. [16:27:30] Ideally, it would happen along with the fetch, so the venv would be inside of each source revision. [16:27:35] awight, should happen before the restart [16:27:44] +1 [16:27:52] Sounds cool to me [16:28:14] I think that’s correct—but it could be parallel, and could be instantaneous during rollback *if* we saved the venv at each rev [16:30:28] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Parallelize scap for ORES - https://phabricator.wikimedia.org/T181067#3778430 (10awight) [16:30:30] 10Scoring-platform-team (Current), 10Release-Engineering-Team (Watching / External): Write reports about why Ext:ORES is helping cause server 500s and alternatives to fix - https://phabricator.wikimedia.org/T181010#3778429 (10awight) [16:31:38] awight, I'm not sure we want to ever trust the venv unless we rebuilt it from scratch [16:32:05] awight, have you started hacking on an incident report? [16:32:17] I've got vcoleman asking about ORES this morning :/ [16:32:52] (03CR) 10Thcipriani: [C: 031] "Looks good!" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392671 (https://phabricator.wikimedia.org/T181067) (owner: 10Halfak) [16:34:16] halfak: I think a saved venv that was created for a previous version of production source is actually slightly more reliable than creating it from scratch. I’ve been seeing pip glitch out a significant number of times now. [16:34:36] Fair point. [16:35:06] Is the venv in the historical snapshot. Also, is this a new problem to fix or something we should do along with parallelization? [16:35:17] We should get vcoleman thinking about adversarial classification, if she wants to get involved 8D [16:35:54] This venv proposal is new—currently it’s parallel to the deployed dir, and is blown away & recreated for every deployment. Takes a few minutes per machine. [16:38:40] Should take a few minutes in total now [16:40:16] I like where this is going. [16:40:34] Yesterday we were at 43 minutes to deploy a new revision. [16:41:06] Wasnt it up to an hr at one point? [16:41:57] Zppix: That easily could have happened on some other occasion, since it would depend on net load. [16:56:26] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3776555 (10jmatazzoni) I see this is about hiding the... [17:04:47] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3778532 (10Legoktm) It might? I'm not sure tbh. We can... [17:12:29] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3778540 (10jmatazzoni) In T181015#3778532, @Legoktm wr... [17:16:53] halfak: I see that our promote check restarts celery, but not the uwsgi threads. Is there a good reason for that? [17:21:51] (03PS1) 10Awight: Remove unused scripts [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392681 [17:21:52] (03PS1) 10Awight: [WIP] Build venv into deployed source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 [17:26:40] awight, no idea. Amir1 might know. [17:27:02] Amir1: :D ? [17:29:39] Is there a reason we shouldnt? [17:33:03] We hate some services and would rather push them off a bridge. Others we restart for free. [17:38:31] 10Scoring-platform-team (Current), 10Operations, 10Release-Engineering-Team (Watching / External): Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3778593 (10awight) [17:40:42] (03PS2) 10Awight: Remove unused scripts [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392681 (https://phabricator.wikimedia.org/T181071) [17:40:44] (03PS2) 10Awight: [DNM] Build venv into deployed source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) [18:06:51] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3778680 (10Catrope) >>! In T181015#3778540, @jmatazzon... [18:07:40] Structured commons meeting at #wikimedia-office sry for late notif incase any are interested [18:38:43] 10Scoring-platform-team: Refactor scripts fetching text and other metadata - https://phabricator.wikimedia.org/T181074#3778789 (10Sumit) [18:58:10] halfak: I just got back from dinner [18:58:24] I think I know why but I need to check [19:03:55] Amir1, thanks for taking a look [19:03:57] (03CR) 10Ladsgroup: [V: 032 C: 032] "We were planning to separate web and worker nodes so it might be useful in the future but we can revive it if we decide to go to that dire" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392681 (https://phabricator.wikimedia.org/T181071) (owner: 10Awight) [19:04:24] (03CR) 10Ladsgroup: [V: 032 C: 032] Parallelize scap but preserve serial restarts. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392671 (https://phabricator.wikimedia.org/T181067) (owner: 10Halfak) [19:04:39] Amir1, looks like today is a wikidata day. Is that right. [19:04:41] *? [19:04:57] yup but I can borrow five minutes :D [19:04:58] (otherwise, I'd prioritize talking to you about JADE stuff) [19:05:12] OK cool. Thanks for using those 5 mins to look at scap [19:05:23] :) [19:06:02] I think I'm going to call it in this afternoon. This is the first chance I have for a break since getting back from my last trip. [19:06:11] I'll hang out for a bit to make sure awight has what he needs. [19:06:23] * halfak has so many "travel days" saved up. [19:11:07] halfak: quick question: I need everything to be lowercase so that we don't generate different hashes for different cases, but english.stopwords.revision.datasources.non_stopwords doesn't lowercase by itself, i made up - https://dpaste.de/7h6c [19:11:13] is there anything better than this? [19:11:47] read "stopwords" as "non_stopwords" [19:12:01] https://github.com/wiki-ai/revscoring/blob/master/revscoring/datasources/meta/mappers.py#L43 [19:12:41] oh, i was looking for sth like this, thanks.... [19:12:47] No problem [19:16:20] (03CR) 10Halfak: [C: 04-1] "add draftquality to omitted wheels" (031 comment) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/391562 (https://phabricator.wikimedia.org/T180496) (owner: 10Awight) [19:16:47] * halfak --> lunch [19:27:17] and here starts bow extraction on ores-misc.... [20:03:11] Amir1: whats the latest revscoring version? [20:03:23] 2.0.10 I gues [20:03:28] let me check pypi [20:03:43] Amir1: a major change happened across 2.x.x right? [20:03:58] https://pypi.python.org/pypi/revscoring [20:04:03] yup [20:04:09] semver.org [20:04:50] Amir1: is any of the utilities using the new one? i'm taking classifier params from https://github.com/wiki-ai/wikiclass/blob/master/config/classifiers.params.yaml and calling tune with right arguments but tune doesn't seem to recognize them [20:05:07] it can be [20:05:08] either i'm using wrong arguments or making a very silly error... :/ [20:05:12] I can't check atm [20:05:23] alright....i'll look around [20:05:23] but that happened for editquality [20:05:44] Amir1: so editquality is using the new revscoring in prod? [20:19:13] okay got it after digging into - https://github.com/wiki-ai/revscoring/pull/307/files [20:27:31] o/ [20:27:50] Ended up running some errands that were piling up while I was traveling. [20:28:31] codezee, the way you pass arguments to tune changed with 2.0.0 [20:29:12] halfak: yeah, I figured it out from the PR [20:29:25] fingers crossed for the results of the first run ... [20:30:33] halfak: apart from the refactoring what new was there in the above PR of August? I remember something being talked about thresholds and stuff...but don't remember exactly [20:31:36] codezee, mostly it was about how model information was stored. Given that fitness statistics are a big part of model information, this changed the tuning utility too. [20:32:11] oh, ok... [20:32:42] looks like tune took a list of labels in contrast to a singe one without any complains [20:35:13] interesting! Good :) [20:35:27] I bet the fitness statistics are going to be all wrong through [20:35:35] and by all wrong, I bet something is going to crash. [20:35:48] How do we even do multilabel fitness statistics? [20:41:02] one vs rest is something that can be employed here - http://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html#in-multi-label-settings [20:41:32] Hmm... we kind of already do that. [20:41:33] average precision-recall aren't that useful but per class precision-recall are useful as well as the confusion matrix [20:42:12] codezee, I wonder if positive should just be the presence of a label in the output set. [20:42:26] E.g. the prediction is a set [20:42:45] I don't think we handle that all in our classification statistics. [20:42:59] yes, so you mean if the set is ["Europe", "History"] so it'll be positive for these labels, right? [20:43:08] Right. [20:45:38] so if we follow the one vs rest strategy, we get the article rated by each classifier dedicated to a label, and whoever gives a +ve score, we add that label in prediction [20:45:49] isn't it? [20:46:04] Right. I think the real problem is confirming a "true positive" [20:46:39] E.g. if the prediction is a set, then a TP occurs when the target class C is in the prediction and in the label. [20:48:48] if we're taking one classifier per label, aren't we simply doing a binary classification with regards to that label? [20:49:15] Right. If we think of it that way, yes. But our estimator won't give us predictions that look like that. [20:49:19] * halfak gets an example. [20:50:27] https://github.com/wiki-ai/revscoring/blob/master/revscoring/scoring/statistics/classification/classification.py#L80 [20:51:25] On this line we'd want to have something like y_preds = [label in s[self.prediction_key for ...] [20:51:35] Messed up the brackets there [20:51:43] y_preds = [label in s[self.prediction_key] for ...] [20:51:53] This line https://github.com/wiki-ai/revscoring/blob/master/revscoring/scoring/statistics/classification/classification.py#L82 [20:52:13] y_trues=[label in l for s, l in score_labels] [20:56:24] oh, so because of labels changing from a single value --> a set, we have to change equality --> membership in the set... [20:57:30] right. [21:15:23] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3779375 (10jmatazzoni) In T181015#3778680, @Catrope wr... [21:19:18] 10Scoring-platform-team (Current), 10Edit-Review-Improvements-RC-Page, 10ORES, 10Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), and 2 others: Enable parts of ORES extension by default and manage impacts on the RC Page and the RC page Preferences t... - https://phabricator.wikimedia.org/T159763#3779382 [21:19:21] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10MW-1.29-release-notes: Manage ORES preferences on Watchlist (and Contributions) - https://phabricator.wikimedia.org/T160475#3779381 (10jmatazzoni) 05Open>03Resolved [21:33:16] Halfak was the incident doc written if so link pls i wanna read [21:33:36] awight is working on that [21:33:54] Sweet. [21:34:03] Hows your day been halfak [21:36:56] He needed to run away earlier. [21:37:12] Oh I'm good. I've been getting the last of travel backlogs taken care of :) [21:37:54] halfak: last night was sure interestig [21:39:59] Indeed it was. [21:40:06] Ended up with a few kicks in the face :/ [21:47:42] https://github.com/wiki-ai/jade/pull/4 [21:49:20] halfak: why what happened if u dont mind [21:49:44] (Feel free to pm) [21:50:03] Oh just people getting upset taking it out on Adam and me. It's OK. It happens. [21:50:17] Had our CTO send me some urgent and concerned messages this morning. [21:50:31] Though, I think she though that FRwiki and RUwiki were 100% offline [21:50:36] Which was far from the case. [21:51:07] 10Scoring-platform-team (Current), 10JADE, 10Epic: Implement basic path structure for JADE (judgements) - https://phabricator.wikimedia.org/T181098#3779510 (10Halfak) [21:51:19] 10Scoring-platform-team (Current), 10JADE, 10Epic: Implement basic path structure for JADE (judgements) - https://phabricator.wikimedia.org/T181098#3779524 (10Halfak) /judgements///// [21:51:51] 10Scoring-platform-team (Current), 10JADE, 10Epic: Implement basic path structure for JADE (judgements) - https://phabricator.wikimedia.org/T181098#3779525 (10Halfak) a:03Halfak [21:53:10] Halfak tbh i think everyone overreacted... crap happens stuff breaks... we are human. How many times have people broken projects and will continue to do so? Theres no amount of code review, CI tests, failsafes to prevent crap like this happening. Its life... people need to learn to cope without certain things be accessible for a few mins [21:54:04] Need me to merge anything review anything while im around halfak? [21:54:11] Agreed. It's good that we learn from these things and less good that we react aggressively. [21:54:32] Zppix, na. Though I am going to get srrodlund to look at your email in a few minutes [21:54:37] Ok [21:54:41] Ty [21:54:42] So we'll have a new version of it for your to review :) [21:54:55] Good, its long overdue imo [21:55:51] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train/test reverted model for Icelandic - https://phabricator.wikimedia.org/T181099#3779529 (10Halfak) [21:56:06] 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Icelandic - https://phabricator.wikimedia.org/T178524#3779543 (10Halfak) [21:56:25] 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Icelandic - https://phabricator.wikimedia.org/T178524#3694816 (10Halfak) Merged. next step is to build a basic reverted model. {T181099} [21:59:51] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Train/test reverted model for Icelandic - https://phabricator.wikimedia.org/T181099#3779550 (10Ladsgroup) a:03Ladsgroup [22:03:02] \o/ [22:03:02] <3 Amir1 [22:10:59] 10Scoring-platform-team (Current): Build mid-level WikiProject category training set - https://phabricator.wikimedia.org/T172321#3494777 (10Halfak) Reviewed and merged! Let's get a dataset uploaded to figshare and then call this "done"! [22:18:34] 10Scoring-platform-team: Could use some more disk space on ores-misc-01.ores-staging.eqiad.wmflabs:/srv - https://phabricator.wikimedia.org/T179592#3779602 (10Halfak) OK we've cleaned up all we can and yet we're still stuck with not enough space. So I think it might be time to build a new image and move thing... [22:31:06] * halfak clears off his todo items for the week [22:35:12] Letme when they start working on that email so i can work with then aswell halfak :) [22:36:28] Zppix, sent! srrodlund was feeling like getting it out. [22:36:42] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10MediaWiki-extensions-ORES, 10Technical-Debt: ORES extension should not modify $wgHiddenPrefs at runtime - https://phabricator.wikimedia.org/T181015#3779655 (10Petar.petkovic) 05Open>03Resolved a:03... [22:36:51] Halfak sweet [22:38:22] 10Scoring-platform-team (Current), 10ORES, 10Documentation: Announce the ORES FAQ - https://phabricator.wikimedia.org/T181103#3779666 (10Halfak) [22:38:37] 10Scoring-platform-team (Current), 10ORES, 10Documentation: Announce the ORES FAQ - https://phabricator.wikimedia.org/T181103#3779680 (10Halfak) a:03srodlund [22:40:27] 10Scoring-platform-team (Current), 10ORES, 10Documentation: Announce the ORES FAQ - https://phabricator.wikimedia.org/T181103#3779666 (10Zppix) "Co-assigning" myself [22:40:44] 10Scoring-platform-team (Current), 10ORES, 10Documentation, 10User-Zppix: Announce the ORES FAQ - https://phabricator.wikimedia.org/T181103#3779711 (10Zppix) [22:40:45] +1 [22:41:22] 10Scoring-platform-team (Current), 10ORES, 10Documentation, 10User-Zppix: Announce the ORES FAQ - https://phabricator.wikimedia.org/T181103#3779713 (10Halfak) https://lists.wikimedia.org/pipermail/ai/2017-November/000213.html [22:41:23] Phab should allow many assigees [23:44:58] 10Scoring-platform-team (Current), 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Former-Sprint-Board: ORES thresholds for Wikidata is too strict - https://phabricator.wikimedia.org/T180450#3779916 (10Jklamo) I agree with Sjoerddebruin, now it is second extreme, too much edits are marked. [23:52:44] SKYRAN: welcome how can we help you?