[00:15:01] awight: got few mins? [11:29:28] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 4 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3788257 (10MaxBioHazard) "Next week" is here, so we are waiting for ORES reenabling. [13:38:26] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 4 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3788592 (10Ladsgroup) My patch is merged and I will backport it today. Then we will reenable one... [13:50:22] (03CR) 10Awight: Warn instead of throwing error in case threshold is unparseable (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393373 (https://phabricator.wikimedia.org/T181191) (owner: 10Ladsgroup) [13:53:53] Amir1: ^ You have any strong opinions about WARNING vs ERROR? [13:54:04] (03PS2) 10Ladsgroup: Warn instead of throwing error in case threshold is unparseable [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393373 (https://phabricator.wikimedia.org/T181191) [13:54:09] hehe [13:54:13] (03CR) 10Ladsgroup: Warn instead of throwing error in case threshold is unparseable (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393373 (https://phabricator.wikimedia.org/T181191) (owner: 10Ladsgroup) [13:54:36] awight: Let's do it! [13:55:09] (03CR) 10Awight: [C: 032] "Awesome, I like the fix. It makes sense to get rid of the exceptions since we're still able to process the other thresholds." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393373 (https://phabricator.wikimedia.org/T181191) (owner: 10Ladsgroup) [13:55:56] awight: now it's time to backport the two patches [13:56:45] (03Merged) 10jenkins-bot: Warn instead of throwing error in case threshold is unparseable [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393373 (https://phabricator.wikimedia.org/T181191) (owner: 10Ladsgroup) [13:57:36] Amir1: I was going to bring up the re-enablement in the sync meeting, but we might as well figure it out now. [13:57:38] Seems like smoke testing on the beta cluster is good enough to be our only precaution before deploying to production? My confidence comes from the fact that we can reproduce the crash there now. [14:13:36] awight: yeah, it's fine for me [14:31:34] o/ [15:21:54] Halfak or awight could i have some assistance with figuring out why https://travis-ci.org/wiki-ai/ores/builds/307229129 is failing and howto fix [15:53:46] Zppix, I think that's an intermittent issue. Try restarting the build. [15:53:49] brb [15:53:52] relocating... [16:06:01] back [16:58:13] (03PS3) 10Awight: Put all the wheel tools into the Makefile [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/391562 (https://phabricator.wikimedia.org/T180496) [17:14:52] OK I'm changing locations for lunch. Back online in ~1.5 hours (after lunch) [17:29:34] (03PS4) 10Awight: Put all the wheel tools into the Makefile [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/391562 (https://phabricator.wikimedia.org/T180496) [17:35:59] (03PS1) 10Ladsgroup: Introduce ModelLookup interface and its SQL implementation [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393620 (https://phabricator.wikimedia.org/T181334) [17:36:38] 10Scoring-platform-team (Current), 10Patch-For-Review: Clean up ORES wheels Makefile - https://phabricator.wikimedia.org/T180496#3789464 (10awight) https://github.com/wiki-ai/ores-wmflabs-deploy/pull/92 [17:37:01] Amir1: halfak: ^ ready for re-review [17:37:21] awight: and this is for you: https://gerrit.wikimedia.org/r/#/c/393620/ [17:37:25] (03CR) 10jerkins-bot: [V: 04-1] Introduce ModelLookup interface and its SQL implementation [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393620 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [17:37:58] Amir1: Want to get this out today? [17:38:56] My work is continuous and take around one week or two (at least) [17:38:59] this is one step [17:39:32] Made a mess in the files, wait a sec [17:40:17] neat. [17:40:46] (03PS2) 10Ladsgroup: Introduce ModelLookup interface and its SQL implementation [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393620 (https://phabricator.wikimedia.org/T181334) [17:40:56] I was planning to deploy your extension fixes, but this seems like it should wait? [17:41:26] The extension fixes is something totally different, I think it'd be great if we can deploy those [17:42:36] +1 [17:42:59] I’ll wait to CR+2 this latest chain then, so that beta has what we’re actually planning to deploy. [17:43:54] cool [17:44:08] Is there anything I can help with that? [18:00:10] (03CR) 10Awight: "tiny suggestions" (035 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393620 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [18:05:53] Amir1: halfak: Important thing we haven’t discussed. Are we trying to roll the ORES service code forwards today, or just reenabling frwiki and ruwiki? [18:09:22] Wat. Why would en.beta RecentChanges not be showing ORES? [18:11:59] (03PS3) 10Ladsgroup: Introduce ModelLookup interface and its SQL implementation [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393620 (https://phabricator.wikimedia.org/T181334) [18:12:02] (03CR) 10Ladsgroup: Introduce ModelLookup interface and its SQL implementation (035 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393620 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [18:15:26] awight: also, Are you pushing the change in editquality repo? I need this for wikidata ASAP [18:28:26] Amir1: sure, we can do that. [18:28:36] I missed your earlier answer maybe—no other changes to ORES, right? [18:32:33] Amir1: oof, pulling recent editquality probably does require us to pull in the rest of the changes. [18:32:47] Can you make into a cherry-pick? [18:33:44] awight: what other changes have been made and not deployed? [18:34:00] We rolled back to an ancient ORES. [18:34:42] revscoring 1? [18:35:04] If that's the case, I trained the wikidata models based on the newest version of revscoring [18:35:30] it’s revscoring 2, but w/o the 500 fixes etc. [18:35:37] lemme get you the exact rev [18:36:14] Amir1: 82a13ae [18:36:27] So I need to retrain it :((((((((((((( [18:39:32] https://phabricator.wikimedia.org/P6379 [18:39:34] I’m not sure [18:39:57] Are we bold enough to push the new code, is the question... [18:40:43] Has it tested on beta? [18:41:13] grr BTW I think that new rollback speed improvements won’t help us, since the scap scripts are embedded in the source dir. [18:41:37] Let’s put this all on beta. Currently beta is the old code, so we can reproduce the latest nasty bug. [18:42:22] Ok restarting build halfak [18:45:38] wiki-ai/ores#844 (Pix1234-patch-1 - d0bb86c : Devin/Zppix): The build passed. https://travis-ci.org/wiki-ai/ores/builds/307229129 [18:46:05] Finally!! [18:49:52] awight: could you review https://github.com/wiki-ai/ores/pull/235 ? [18:54:21] Zppix: elegant! Please remind me again tomorrow morning though—I have a deployment and limited time this afternoon. I’ll smoke test that patch using a few environments. [18:54:44] K [18:59:36] (03PS1) 10Awight: Bump editquality submodule [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/393631 [19:02:00] Amir1: halfak|Mobile: ^ [19:02:05] Just about to setup at coffee shop. Thought I'd log in just in case [19:02:12] I don't have a log [19:02:19] (03CR) 10Ladsgroup: [V: 032 C: 032] Bump editquality submodule [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/393631 (owner: 10Awight) [19:02:23] o/ [19:02:59] What did you point me to? [19:03:06] halfak|Mobile: merged already :D [19:04:54] halfak|Mobile: Amir1 and I were talking about a little snafu that didn’t come up in this morning’s meeting, though…. It looks like we’ll be rolling the ORES service code forward rather than just making a config change. [19:04:56] And that’s scary. [19:05:12] I’m trying to put it on beta now. [19:05:17] Good point [19:05:21] I'll help test [19:06:58] halfak|Mobile: here’s another scary fact: beta wikipedia currently doesn’t show any ORES in S:RecentChanges, for any language. [19:07:16] I’m not cool with debugging and deploying a possibly hacked version in the next 2 hours... [19:07:32] There’d better be a really straightforward explanation… [19:08:58] OK we’re deployed to beta. [19:09:19] ORES is back on S:RC. /me shudders [19:11:14] halfak|Mobile: srrodlund: FYI, I can’t make the docs meeting today. Also, I suggest we move the time since it overlaps with a regular ORES deployment window… [19:12:45] There’s no frwiki on the beta cluster, so we can’t test that ATM. [19:14:13] I'm ok with moving docs meeting [19:15:06] I'm stealing corporate WiFi. Looks like it doesn't like irc [19:18:15] OK [19:18:19] IRC Cloud it is! [19:18:32] awight: so you saw no ORES before deploying new code to beta? [19:18:38] *service [19:18:48] halfak: exactly. It was terrifying. [19:19:02] Unfortunately, it doesn’t give us any data. [19:19:14] Hmm.. We'd confirmed that we saw ORES in S:RC before [19:19:19] Right! [19:19:27] I'd also confirmed that it totally crashed out [19:19:27] So it must have crashed or something? [19:19:33] With old server code [19:19:52] Wait... no that was good new server code that returned nulls [19:20:02] oops, we both using inaccurate terminology: yes, the S:RC page threw exceptions [19:20:04] oof that’s right [19:20:21] I can sort of reconstruct which version that was. [19:20:26] Right. I recorded that in a ticket. [19:20:57] https://phabricator.wikimedia.org/T181168 [19:21:04] dang, scap logs are still using arbitrary dates [19:22:02] The ticket has some versions and patchsets [19:22:06] Might be useful [19:23:27] It looks like we had 1eee380f deployed. That doesn’t appear in your comments, strangely. [19:25:48] Nope [19:26:06] I explicitly say that I deployed 5084251 :P [19:26:12] I probably gave you the wrong tree-ish at first… [19:26:13] anyway [19:26:21] "I deployed https://phabricator.wikimedia.org/source/ores-deploy/ @ 5084251" [19:26:21] Let’s figure out why it crashed. [19:26:23] lol [19:26:53] I think this was the demonstration of the failure to handle null. [19:27:15] And that if it is working now, that means that it is working on the version of ORES we tried to deploy last time (which is not HEAD of master now) [19:27:26] HEAD of master includes parallel scap. [19:28:33] +1 okay so maybe you did deploy 5084251, but then rm’ed the cache in order to play with parallel scap. [19:31:28] Well I can confirm that the service was 503’ing all day [19:31:33] I rm'd the cache because 5084251 wouldn't install. [19:31:36] I didn't do that [19:31:49] I didn't rm 5084251 at all. [19:32:01] Just got that service deploy to work as intended and moved on to MW testing. [19:33:50] Last HTTP 200 responses were at 2017-11-23T20:22:06 [19:36:21] There were no deployments around then. [19:37:00] OSError: [Errno 12] Cannot allocate memory [19:37:24] [2017-11-23 20:17:11,436: ERROR/MainProcess] Unrecoverable error: MemoryError('[Errno 12] Cannot allocate memory',) [19:37:30] File "/srv/deployment/ores/venv/lib/python3.4/site-packages/billiard/pool.py", line 1260, in maintain_pool [19:37:50] awight: maybe sca03 ran out of memory and killed one of ours [19:38:09] It’s not giving me confidence about promoting this version... [19:38:29] checking graphs around that time. [19:39:02] Where are the beta graphs... [19:40:09] sweet. https://grafana-labs.wikimedia.org/dashboard/db/ores-beta-cluster [19:41:26] Step change, https://grafana-labs.wikimedia.org/dashboard/db/ores-beta-cluster?orgId=1&from=1511379620201&to=1511552420202 [19:42:58] Looks like the OOM killing caused all of the wsgi queues to fill up. [19:43:16] celery dies and uwsgi starts waiting on celery until timeout for every request. [19:43:29] There’s just nothing to do after an OOM, the machine is hosed. [19:43:34] celery eventually fills up its queue and starts returning overload errors [19:43:38] Random OS-level things are usually broken,. [19:43:43] right [19:43:43] needs to be rebooted. [19:44:08] Is this the version of ORES that is using celery 4.1 [19:44:26] And splitting overlapping requests for different model-scores for the same revision? [19:45:07] No, this is Celery 3 still. celery 4 is all on branches. [19:45:16] AFAIK. [19:45:39] all the requirements.txt’s check out. [19:47:38] Nothing in the logs up to the OOM looks very suspicious: https://logstash-beta.wmflabs.org/goto/fee83e833db27cd9fab027180daca24b [19:48:06] The only interesting detail is that there are more wikidatawiki requests than usual, enough that we have to kill and respawn some workers. [19:48:36] Maybe we should stress test the beta ORES endpoint for wikidatawiki? [19:51:01] beta is not prepared to stress test [19:51:05] I imagine it will fall over. [19:51:12] We could stress wmflabs [19:51:32] yeah it would die horribly. [19:51:45] *sigh* we’re up against the wire [19:51:52] ores-staging is more beefy than ores-beta [19:52:12] We can re-enable ruwiki and frwiki without doing a service deployment. [19:52:14] In theory [19:52:23] Good point [19:52:25] They were working fine before our last service deployment. [19:52:37] As in the errors were not causing user-problems [19:53:02] halfak: oh hey you should be aware—your faster rollback code is great if we have to rollback to current or future versions, but won’t help us if we have to rollback to what’s currently on production :( [19:53:10] because the scap directory is included in the source. [19:54:00] hmm... good point [19:54:19] damn [19:55:48] So... I don't see any ORES on ru.wikipedia.beta [19:55:52] We could branch or somethign but [19:55:52] S:RC [19:55:54] oh wat [19:55:55] checking [19:56:02] Damn it nevermind [19:56:08] I swear it was... [19:56:10] oh hehe [19:56:11] I accidentally did prod [19:56:19] um [19:56:24] “did” == tested, I hope? [19:56:47] yep it’s there on beta. [19:57:15] yeah. Did = tested [19:57:35] * awight wipes pee from the floor [19:57:36] * halfak accidentallys prod all the time [19:57:41] lol [19:58:04] awight: do we have a 1:1 during the deploy window? [19:58:58] no it’s the hour before :) [19:59:14] We can chat about the “postmortem” lol [20:00:04] So, verdict is that beta is behaving perfectly. But we have no idea what caused the OOM and it might be the newest models. [20:00:33] I'm pretty skeptical it was the new models. [20:00:45] Models haven't changed recently. [20:00:51] Wikidatawiki is going to change though [20:00:56] if we want to do that. [20:01:05] wikidatawiki is kind of degraded quality right now. [20:08:18] halfak: https://phabricator.wikimedia.org/P6379 [20:17:25] https://wikidata.beta.wmflabs.org/wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&limit=50&days=7&urlversion=2 [20:17:34] awight: ^ [20:30:01] Halfak i got the build to pass I was wondering (i already asked adam he said he would but i wanted to ask if you incase you wanted to review it) if you wanted to review this, https://github.com/wiki-ai/ores/pull/235 [20:54:12] awight halfak I'm moving meeting to later this week. [20:57:29] srrodlund: TY! oh hey can you move it to a different time of day also? If you look at my calendar, we have this window at 21:00 UTC every day... [20:58:11] Amir1: Did you happen to backport your Ext:ORES patches? [20:58:36] awight NP. GCal's new UI is making my blowing my mind... [20:58:47] lol Looking forward to that [21:00:34] deployments! [21:00:43] just sent new code to ores-staging [21:01:48] https://ores-staging.wmflabs.org/v3/scores/ruwiki/?models=goodfaith&model_info=statistics.thresholds.false.%22maximum+recall+%40+precision+%3E%3D+0.15%22%7Cstatistics.thresholds.false.%22maximum+recall+%40+precision+%3E%3D+0.45%22%7Cstatistics.thresholds.false.%22maximum+recall+%40+precision+%3E%3D+0.75%22%7Cstatistics.thresholds.true.%22maximum+recall+%40+precision+%3E%3D+0.995%22&format=json [21:01:50] Works [21:02:01] Returns a null [21:02:26] Making the first config patch... [21:03:19] Ill buy the alcohol for this week [21:07:10] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10draftquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Collect all data for draftquality model in enwiki - https://phabricator.wikimedia.org/T179861#3790349 (10kaldari) Awesome! Subscribing @Nettrom. [21:09:00] Confirmed that wikidatawiki models are working correctly in context. [21:09:37] ores.wmflabs.org is running fresh code. [21:11:05] 10Scoring-platform-team, 10Wikimania-Hackathon-2017, 10Documentation, 10Easy, 10Google-Code-in-2017: [Wikimania doc sprint] docs on how to install ORES - https://phabricator.wikimedia.org/T170506#3790386 (10Aklapper) @Ladsgroup, @Halfak: If I was a new contributor who's never seen or heard of ORES, what... [21:12:44] Confirmed that ORES is disabled in wikidatawiki S:RC [21:13:22] Double-confimed [21:13:25] *r [21:15:39] (03PS1) 10Awight: Disable the filter if ORES says the threshold doesn't exist [extensions/ORES] (wmf/1.31.0-wmf.8) - 10https://gerrit.wikimedia.org/r/393659 (https://phabricator.wikimedia.org/T181006) [21:15:43] (03PS1) 10Awight: Warn instead of throwing error in case threshold is unparseable [extensions/ORES] (wmf/1.31.0-wmf.8) - 10https://gerrit.wikimedia.org/r/393660 (https://phabricator.wikimedia.org/T181191) [21:16:32] (03CR) 10Awight: [C: 032] Disable the filter if ORES says the threshold doesn't exist [extensions/ORES] (wmf/1.31.0-wmf.8) - 10https://gerrit.wikimedia.org/r/393659 (https://phabricator.wikimedia.org/T181006) (owner: 10Awight) [21:16:38] (03CR) 10Awight: [C: 032] Warn instead of throwing error in case threshold is unparseable [extensions/ORES] (wmf/1.31.0-wmf.8) - 10https://gerrit.wikimedia.org/r/393660 (https://phabricator.wikimedia.org/T181191) (owner: 10Awight) [21:16:38] * halfak watches with interest [21:17:03] * awight peeks from behind blindfold [21:18:05] (03Merged) 10jenkins-bot: Disable the filter if ORES says the threshold doesn't exist [extensions/ORES] (wmf/1.31.0-wmf.8) - 10https://gerrit.wikimedia.org/r/393659 (https://phabricator.wikimedia.org/T181006) (owner: 10Awight) [21:18:13] (03Merged) 10jenkins-bot: Warn instead of throwing error in case threshold is unparseable [extensions/ORES] (wmf/1.31.0-wmf.8) - 10https://gerrit.wikimedia.org/r/393660 (https://phabricator.wikimedia.org/T181191) (owner: 10Awight) [21:18:21] * Zppix fastens the barrier door closed [21:20:30] Awight gratz on the new scap time record [21:20:53] Zppix: LOL. the sleeping giant. [21:21:35] awight: scap had to much thanksgiving lol [21:24:40] Amir1: halfak: Want to smoke test ext-ORES functionality on enwiki etc? [21:24:50] prod? [21:24:51] I haven’t deployed new service code yet. [21:24:53] yes. [21:25:05] I see no ORES on ruwiki [21:25:10] correct. [21:25:20] It should be disabled on frwiki, ruwiki, and wikidata [21:26:15] enwiki looks good [21:26:34] Confirmed that not all edits are flagged and likely damaging edits are getting flagged. [21:27:02] ty [21:27:07] ok going for the service deploy now. [21:27:24] Goodfaith looks good too [21:27:32] * halfak will keep checking on enwiki [21:27:50] Oh—checking that the deployed ext-ORES code revision is correct [21:28:15] Need a hand? [21:28:20] Halfak [21:28:23] awight: will we need to clear the cached thresholds [21:28:38] good point—that can come last [21:28:41] Zppix: go to special recent changes on wikidata.org and try to use ORES [21:28:51] Kl [21:28:54] awight: might not notice an issue until we do that [21:29:18] Zppix: I expect it won't show up but that the page will otherwise work right now. [21:29:50] Halfak i would if i could filter for it [21:32:18] Zppix: give it a refresh every now and then and let me know if anything changes [21:32:35] Oo [21:32:37] OK* [21:33:08] I don't think it'll change at all FWIW, but if it does we want to know right away. [21:33:15] Ill check in 3 min intervals or so. [21:33:40] perfect [21:33:47] human icinga :) [21:34:21] Deploying service code to the canary, should take c. 10 min. [21:36:48] All still looks good :) [21:36:50] restarting the service... [21:37:48] All good here [21:37:50] ready to test (1/9 of the time) [21:39:38] looks good, continuing. [21:39:59] Confirmed [21:40:04] halfak: parallel scap output is strange [21:40:20] It seems to be fetching to 3 at a time, but not all in parallel [21:40:33] scap bug? [21:40:47] Amir1: halfak: JFYI so far, https://phabricator.wikimedia.org/P6381 [21:40:58] halfak: hehe I wouldn’t rule that out. [21:41:02] Wikidata didnt change [21:41:20] weird [21:41:34] Zppix: good. :) [21:41:35] Nothing should change. [21:41:48] Oh [21:41:53] Zppix: Nothing will change until I reenable the config…. Thanks for the note! [21:42:00] Enwiki continues to look good. [21:42:06] awight: in theory ;) [21:42:14] halfak: well thank you :P [21:42:15] ruwiki and frwiki also look good and have no chnage [21:42:16] * awight facepalms — I somehow limited to scb1* [21:42:33] I only re-read that command line 5 times [21:42:53] awight is assigned 3 readings of scap's manual and 4 recitations of the hacker's creed. [21:43:15] awight: not the worst thing, honestly [21:43:21] halfak: and an essay on both [21:43:35] Randomly generated essays are OK [21:43:36] Just slooow [21:43:59] (03CR) 10Krinkle: "Is this covered by tests?" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [21:44:05] halfak: i can see it already, a repo for automatic essays... [21:44:27] * halfak reverts some vandalism in enwiki [21:44:34] Might as well do something productive while I'm watching [21:44:45] Halfak atta boy [21:45:24] 10Scoring-platform-team, 10Scap, 10Release-Engineering-Team (Kanban): Need to make the number of cached revisions configurable - https://phabricator.wikimedia.org/T181176#3790492 (10mmodell) p:05Triage>03High a:03mmodell [21:45:37] 10Scoring-platform-team, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2): Need to make the number of cached revisions configurable - https://phabricator.wikimedia.org/T181176#3790495 (10mmodell) [21:48:49] Nothings exploded at wikidata [21:49:55] Confirmed enwiki, frwiki and ruwiki all remain unchanged and working [21:51:26] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert. [21:51:28] halfak: parallel scap FTW [21:52:11] Precached freaked out but then calmed down [21:52:32] Still nothing [21:52:43] The heck is happening with codfw [21:52:55] Stack traces mid-promote [21:53:01] awight: its being codfw [21:53:10] codfw is restarted now, let’s look... [21:53:35] awight: looks OK to me [21:53:44] Ok here [21:53:47] Had a derp for one time period and then recovered in the next [21:54:03] It did? [21:54:27] Yeah. Gave precaching some 500s [21:54:27] * awight smiles at narrower error averaging window [21:54:36] 10Scoring-platform-team, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2): Need to make the number of cached revisions configurable - https://phabricator.wikimedia.org/T181176#3790568 (10mmodell) Are you really seeing 10 old versions cached? As far as I can tell scap should only be... [21:54:43] 6 minutes until I have a chat with the boss. [21:54:54] Which boss o_O [21:55:02] CTO [21:55:03] hahaha well I might as well throw this config switch, what could possibly go wrong. [21:55:17] https://blog.wikimedia.org/2016/11/02/victoria-coleman-chief-technology-officer/ [21:55:25] Halfak ah [21:55:28] awight: great timing for everything to go wrong ;) [21:55:29] Have fun [21:55:42] halfak: Sort of amazing really [21:55:47] Flip it quick so that it doesn't start on fire in the call ;) [21:55:48] No pressure. [21:55:51] yeah man. [21:55:54] haha [21:55:57] And within our window even. [21:56:21] "How are you?" "Oh great. It's cold in MN, but ORES is on fire, so there's that." [21:56:39] lawl [21:56:50] halfak but it would be like special effects :P [21:57:08] Ill keep eye on wikidata [21:57:24] Ill alert if it decides to go take a lunch break [21:58:05] awight: Looks like both eqiad and codfw had a set of 500s one after the other [21:58:58] Reenabling. [22:00:08] Zppix: I need to go to this meeting. Can you check on a few more Urls for me? [22:00:12] https://en.wikipedia.org/w/index.php?damaging=maybebad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=30&enhanced=1&damaging__likelybad_color=c4&damaging__verylikelybad_color=c5&goodfaith__likelygood_color=c1&title=Special:RecentChanges&urlversion=2&uselang=en [22:00:16] Checking [22:00:17] ^ Should show ORES scores [22:00:22] I got it [22:00:23] https://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5_%D0%BF%D1%80%D0%B0%D0%B2%D0%BA%D0%B8&damaging=maybebad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=30&enhanced=1&damaging__likelybad_color=c4&damaging__verylikelybad_color=c5&goodfaith__likelygood_color=c1&urlversion=2&uselang=en [22:00:27] Doesn't but should soon [22:00:32] https://fr.wikipedia.org/w/index.php?title=Sp%C3%A9cial:Modifications_r%C3%A9centes&damaging=maybebad&hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=30&enhanced=1&damaging__likelybad_color=c4&damaging__verylikelybad_color=c5&goodfaith__likelygood_color=c1&urlversion=2&uselang=en [22:00:34] Dpesn [22:00:38] 't but should soon [22:01:15] Threshold caches cleared. [22:01:16] Ruwiki is ok [22:01:34] Enwiki is ok [22:01:37] Hi the **** five. [22:01:39] ORES shows up on ruwiki [22:01:49] Nothing highlighted yet [22:01:53] * awight goes to vomit lol [22:01:54] Frwiki is k [22:02:09] halfak mine does [22:02:35] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting. [22:04:13] 10Scoring-platform-team, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2): Need to make the number of cached revisions configurable - https://phabricator.wikimedia.org/T181176#3790606 (10mmodell) [22:05:09] Wikidata rc loads but i dont think ores has been given any edits for it score since it was enabled [22:05:18] Ill check regularly and shout if theres issues [22:05:23] awight: ^ [22:05:43] Zppix: There have been new edits, hmm... [22:05:56] awight: https://www.wikidata.org/wiki/Special:RecentChanges?damaging=likelygood&goodfaith=verylikelybad&hidebots=1&hidepageedits=1&hidecategorization=1&limit=50&days=7&damaging__likelygood_color=c3&goodfaith__verylikelybad_color=c2&urlversion=2 [22:06:19] Zppix: Looking healthy [22:06:49] Zppix: https://www.wikidata.org/wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&limit=50&days=7&damaging__likelygood_color=c1&damaging__likelybad_color=c2&damaging__verylikelybad_color=c3&urlversion=2 [22:07:03] Sou [22:07:04] Doh* [22:10:31] 10Scoring-platform-team, 10MediaWiki-Recent-changes, 10MediaWiki-Watchlist, 10ORES, and 4 others: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki - https://phabricator.wikimedia.org/T181006#3790640 (10awight) 05Open>03Resolved a:03awight ORES is reenabled on these wikis. [22:38:23] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: qunit tests for wikilabels - https://phabricator.wikimedia.org/T171083#3790710 (10Aklapper) @Ladsgroup: Thanks, however if I've never seen Wikilabels and don't know which "npm" you refer to (or what that is), and how to "integrate with t... [22:44:22] I'm back. Anything dead? [22:45:43] All looks good for me. :) [22:50:31] Good work awight :D [22:51:52] lol I got IRL’d also [22:52:18] (03CR) 10Ladsgroup: "No because I'm re-writing the whole class (and way more) anyway." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/392845 (https://phabricator.wikimedia.org/T181006) (owner: 10Ladsgroup) [22:53:29] halfak: Awesome. I’m out, see you tomorrow ! [22:53:37] o/