[00:00:03] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2513412 (10greg) What was {T130404} for then? [06:34:21] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Add support for Uzbek (https://uz.wikipedia.org) - https://phabricator.wikimedia.org/T119928#2514164 (10bmansurov) @Halfak, thanks for pinging. I feel bad for linking these but here we go: https://new.vk.com/topic-2907260_4551760 and http://www.yo... [10:11:50] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10revscoring, 07Documentation: Add MacOS instructions for installation to README - https://phabricator.wikimedia.org/T139355#2514729 (10schana) After the following, all but three tests pass: brew install aspell --with-all-languages brew install enchant... [10:26:23] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2513412 (10Ladsgroup) >>! In T141825#2513758, @greg wrote: > What was {T130404} for then? When I built the... [10:30:38] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2514813 (10Ladsgroup) a:03Ladsgroup [14:18:20] 06Revision-Scoring-As-A-Service, 10revscoring: Tamil language utilities - https://phabricator.wikimedia.org/T134105#2254643 (10Halfak) https://github.com/travis-ci/apt-package-whitelist/blob/master/ubuntu-precise Looks like we have "aspell-ta" in travis' Precise image. [14:19:08] 06Revision-Scoring-As-A-Service, 10revscoring: Tamil language utilities - https://phabricator.wikimedia.org/T134105#2515393 (10Halfak) https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Word_lists/ta contains the informal and badwords. It looks like we have regexes in the place of badwords... [14:23:59] 06Revision-Scoring-As-A-Service, 10revscoring: Tamil language utilities - https://phabricator.wikimedia.org/T134105#2515397 (10Halfak) 1. பூல் 1. பூலு 1. கூதி 1. தேவுடியாள் 1. தேவடியாள் 1. ஓத்த 1. ஓத்தா 1. சுன்னி 1. சுண்ணி 1. ஓல் 1. ஓழ் 1. ஓலு 1. ஓழு 1. ஓழி 1. ஒம்மால 1. சூத்து [14:28:40] 06Revision-Scoring-As-A-Service, 10revscoring: Tamil language utilities - https://phabricator.wikimedia.org/T134105#2515404 (10Halfak) @Shanmugamp7, we should have more informal words than just "பொட்டை". Is there a Tamil equivalent to "haha", "hello", "goodbye", "silly", "ain't", "awesome", "blah", etc.? You... [15:48:17] Amir1, is there a way that we could downgrade ORES extension errors to warnings when the API responds with an error. [15:48:28] This isn't really an "error" for the extension itself. [15:48:35] But more something we should raise a warning about. [15:49:03] halfak: yup and easily fixable [15:49:46] actually, when running the maintenance script it just logs the revision name [15:50:29] "revision name"? [15:50:38] rev_id [15:50:45] Gotcha. [15:51:20] I think that's what we want -- just a log of the fact than an error happened so we can review if we see a spike on gra(fana|phite) [15:52:29] halfak: but there are some caveats there too. Some times it gets an error due to time out or other reasons. If it sends warning it won't be retried [15:52:39] it's 0.3% now but it was 1% [15:53:14] we can configure a way to pick up some errors and send warning for others [15:53:16] Oh... interesting. We can't "warn and retry"? [15:53:22] (or vice versa) [15:53:25] Why is logging tied so strongly to behavior? [15:53:30] not sure. I should check the source code [15:54:38] It's a very complex system of job runners [15:54:54] Gotcha. This is just really a passing thought. [15:55:09] It would be nice if we could warn about "expected errors" [16:29:16] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Finding items that should be merged - https://phabricator.wikimedia.org/T127467#2044474 (10Esc3300) When first created, https://fr.wikipedia.org/wiki/Projet:Wikidata/Listes/Films_en_fran%C3%A7ais_sans_article... [16:42:42] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515793 (10greg) >>! In T141825#2514785, @Ladsgroup wrote: >>>! In T141825#2513758, @greg wrote: >> What was {T1304... [16:43:33] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515794 (10greg) >>! In T141825#2515793, @greg wrote: > That was my understanding of that original bug that I linke... [16:44:03] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515795 (10greg) [16:49:34] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515807 (10Ladsgroup) There were two bugs in the yesterday deployment. We couldn't... [16:54:41] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515821 (10Halfak) @greg, I'm not sure what you're looking for here. We obviously... [17:19:47] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2515946 (10greg) >>! In T141825#2515821, @Halfak wrote: > If you are asking how we... [17:28:01] 10Revision-Scoring-As-A-Service-Backlog, 06Research-and-Data: Impact of ORES on Wikidata: time-to-revert changes - https://phabricator.wikimedia.org/T141896#2515969 (10DarTar) [17:29:58] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 07Wikimedia-Incident: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2516001 (10greg) >>! In T141825#2515946, @greg wrote: > I think a separate task ab... [18:10:08] 10Revision-Scoring-As-A-Service-Backlog, 06Research-and-Data, 10Wikidata: Impact of ORES on Wikidata: time-to-revert changes - https://phabricator.wikimedia.org/T141896#2515969 (10Ladsgroup) [18:50:19] halfak: around? [18:50:21] https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=815996&oldid=815934 [18:51:59] with this you'll get ping every day for deploying new version of ores, if there is anything just show up in operations and tell that you want to deploy [18:56:30] Am now. Sorry was lunching [18:57:00] totally okay [18:57:40] I talked to Greg about deployment window. [18:57:54] it was the conclusion. We should only deploy in these hours [18:58:12] except urgent fixes (like what we had for nlwiki) [19:13:34] SOunds totally reasonable [19:13:34] :) [19:14:03] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 2675 bytes in 0.087 second response time [19:16:12] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 370 bytes in 0.595 second response time [19:32:55] I've got to run an errand. I'm meeting with the US Gov. to see if they'll let me join the Global Entry program [20:55:39] back! [21:26:15] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: Connection timed out [21:26:48] What the heck. nothing is really running on web-05 [21:27:24] PROBLEM - ORES home page on ores.wmflabs.org is CRITICAL: Connection timed out [21:27:54] Uh oh. Maybe this one is the LB [21:28:04] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: Connection timed out [21:28:25] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: Connection timed out [21:30:56] Web workers are up [21:31:01] load balancer seems to be running [21:44:24] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 369 bytes in 0.615 second response time [21:45:24] RECOVERY - ORES home page on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 419 bytes in 0.024 second response time [21:46:59] SPAM INCOMING [21:47:11] 06Revision-Scoring-As-A-Service, 10ORES: Update wmflabs deploy repo for new version of ORES - https://phabricator.wikimedia.org/T141377#2516924 (10Halfak) 05Open>03Resolved [21:47:13] 06Revision-Scoring-As-A-Service, 10ORES: precached: Merge multiple models per rev_id into a single request - https://phabricator.wikimedia.org/T141376#2516925 (10Halfak) 05Open>03Resolved [21:47:15] BTW, the issue was labs' proxy [21:47:16] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 05WMF-deploy-2016-08-02_(1.28.0-wmf.13): CI tests for the ORES extension - https://phabricator.wikimedia.org/T140455#2516926 (10Halfak) 05Open>03Resolved [21:47:18] 06Revision-Scoring-As-A-Service, 10ORES, 07Epic: [Epic] ORES refactor: Scoring structure - https://phabricator.wikimedia.org/T139408#2516928 (10Halfak) [21:47:18] Everything is back now [21:47:20] 06Revision-Scoring-As-A-Service, 10ORES: Don't load models into memory on web workers - https://phabricator.wikimedia.org/T139407#2516927 (10Halfak) 05Open>03Resolved [21:47:24] 06Revision-Scoring-As-A-Service, 10Wikilabels: Add wiki labels detail to deployment docs on wikitech - https://phabricator.wikimedia.org/T131768#2516931 (10Halfak) 05Open>03Resolved [21:47:25] * halfak speaks directly into the spam [21:47:26] 06Revision-Scoring-As-A-Service, 10ORES, 10Wikilabels, 07Documentation: Document maintenance tasks (restart something, deploy new versions, revert, etc...) - https://phabricator.wikimedia.org/T106271#2516932 (10Halfak) [21:47:28] 06Revision-Scoring-As-A-Service, 10ORES, 07Epic: [Epic] ORES refactor: Scoring structure - https://phabricator.wikimedia.org/T139408#2431247 (10Halfak) [21:47:30] 06Revision-Scoring-As-A-Service, 10ORES, 10revscoring: Score multiple models with the same cached dependencies - https://phabricator.wikimedia.org/T134606#2516933 (10Halfak) 05Open>03Resolved [21:47:32] 06Revision-Scoring-As-A-Service, 10ORES, 10Wikilabels, 07Documentation: Document maintenance tasks (restart something, deploy new versions, revert, etc...) - https://phabricator.wikimedia.org/T106271#1463432 (10Halfak) 05Open>03Resolved a:03Halfak [21:47:34] ...and gets swept away [21:48:11] 06Revision-Scoring-As-A-Service, 10ORES: Investigate web-05 downtime - https://phabricator.wikimedia.org/T141523#2516944 (10Halfak) I thought I'd caught some new downtime today, but it turns out it was labs' proxy. [21:49:55] RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 385 bytes in 1.081 second response time [21:50:05] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 368 bytes in 1.119 second response time [22:36:04] PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 2675 bytes in 0.135 second response time [22:36:05] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 2675 bytes in 0.087 second response time [22:37:20] lies [22:38:05] RECOVERY - ORES web node labs ores-web-05 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 370 bytes in 1.105 second response time [22:38:05] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 384 bytes in 0.553 second response time [22:59:34] wiki-ai/revscoring#762 (feature_vector - fa0196e : halfak): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/149331829 [23:00:45] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: Implement abstraction for Sparse Feature Vectors - https://phabricator.wikimedia.org/T132580#2517372 (10Halfak) I started looking at this. See some work here: https://github.com/wiki-ai/revscoring/pull/284 I think that we want a special "featurevector" t... [23:06:53] Hi, anyone around? I'm getting errors in the ORES API, along with periods of correct answers [23:07:20] For example https://ores.wmflabs.org/scores/eswiki/?revids=92662249&models=reverted right now [23:07:35] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 2675 bytes in 0.104 second response time [23:07:41] "code": "internal server error" etc. [23:08:03] Hi jem, can you copy the error body? [23:08:08] Yes, of course [23:08:34] It's too much maybe, I'll make a text file in Labs [23:09:11] I often use http://pastebin.ca/ [23:09:35] RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 369 bytes in 1.145 second response time [23:10:06] Now it's working again [23:10:15] https://tools.wmflabs.org/jembot/ores-error.txt [23:10:41] I guess that CRITICAL/OK error form icinga-wm has something to do [23:14:11] 10Revision-Scoring-As-A-Service-Backlog: [Investigate] Periodic redis related errors in wmflabs - https://phabricator.wikimedia.org/T141946#2517410 (10Halfak) [23:14:13] 10Revision-Scoring-As-A-Service-Backlog: [Investigate] Periodic redis related errors in wmflabs - https://phabricator.wikimedia.org/T141946#2517423 (10Halfak) [23:14:37] jem, we've had some intermittent issues today, but the earlier ones were related to the labs proxy [23:14:40] This looks different. [23:14:43] See https://phabricator.wikimedia.org/T141946 [23:15:07] Regretfully, I need to step away now. [23:15:35] Ok, thanks anyway, halfak [23:15:46] But I've filed a task for it. Please add any new notes you can there, OK? [23:16:02] Hopefully, this was just a random restart of our redis node [23:16:27] * halfak runs away [23:16:30] Ok [23:16:39] I'm adding some comments [23:20:24] 10Revision-Scoring-As-A-Service-Backlog: [Investigate] Periodic redis related errors in wmflabs - https://phabricator.wikimedia.org/T141946#2517452 (10-jem-) The problem has been happening for several days now, and it can last a few minutes or a few hours. As my patroller bot make continous use of the ORES API,... [23:28:19] Done [23:53:14] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 INTERNAL SERVER ERROR - 2675 bytes in 0.053 second response time [23:54:41] Hey Jem [23:55:02] I just realized that I should have directed you to ores.wikimedia.org [23:55:15] RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 367 bytes in 0.819 second response time [23:55:21] The production service that seems to be more stable right now [23:55:48] * halfak|Mobile goes back to his dinner