[07:00:08] <wikibugs>	 10Scoring-platform-team (Current), 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3725569 (10Marostegui) Please check: T179464 labsdb1001 has crashed and the storage looks totally broken, hard to say if it is...
[07:04:31] <wikibugs>	 10Scoring-platform-team (Current), 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3725571 (10Marostegui) We should consider labsdb1001 broken for good and decommission it -  we need to decide whether we want...
[08:29:40] <wikibugs>	 10Scoring-platform-team (Current), 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3725662 (10MoritzMuehlenhoff) >>! In T168584#3725571, @Marostegui wrote: > We should consider labsdb1001 broken for good and d...
[08:31:32] <wikibugs>	 10Scoring-platform-team (Current), 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3725667 (10Marostegui) We are aiming for 13th Dec to retire these two hosts: T142807 and https://wikitech.wikimedia.org/wiki/W...
[08:34:43] <wikibugs>	 10Scoring-platform-team (Current), 10DBA, 10Operations, 10cloud-services-team, 10Patch-For-Review: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3725668 (10MoritzMuehlenhoff) Let's just keep 1003 running w/o reboot then.
[12:54:01] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): ORES deploy submodule 504 - https://phabricator.wikimedia.org/T179336#3726164 (10mmodell) Phabricator has [[ https://secure.phabricator.com/T4369 | this really horrible bug (T4369)]]: you can't fetch git repositories larger than...
[12:54:01] <AsimovBot>	 10[1] 04https://meta.wikimedia.org/wiki/https://secure.phabricator.com/T4369
[13:17:18] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): ORES deploy submodule 504 - https://phabricator.wikimedia.org/T179336#3726180 (10mmodell) >>! In T179336#3721358, @thcipriani wrote: > We should be: >  > 1. Writing .gitmodules > 2. Calling `git submodule update --init` >  > Whic...
[13:59:20] <halfak|Mobile>	 o/
[13:59:26] <halfak|Mobile>	 Connecting from my kitchen
[14:06:39] <halfak>	 OK now in my office
[14:26:59] <travis-ci>	 wiki-ai/wikiclass#10 (master - 9e7660d : Aaron Halfaker): The build passed. https://travis-ci.org/wiki-ai/wikiclass/builds/295773000
[14:39:59] <Zppix>	 Hola!
[14:48:06] <halfak>	 o/ Zppix 
[14:48:24] <halfak>	 I'm runnign around this morning -- preparing for a trip to Seattle -- so I'll be mostly AFK. 
[14:48:30] <halfak>	 Got a deployment of wikilabels stuff done though :D 
[14:48:31] <Zppix>	 Sounds fun
[14:48:48] <Zppix>	 What you plan on doing im seattle?
[14:49:16] <halfak>	 Giving a lecture.  Hanging out with J-Mo and Nettrom
[14:49:33] <halfak>	 Mostly just working from Seattle for a few days :) 
[14:49:48] <halfak>	 Not sure if J-Mo set up some meetings for me or if I'm just giving a lecture to his class. 
[14:49:53] <Zppix>	 Have fun while your there too
[14:50:09] <Zppix>	 Go see the space needle or something
[14:51:54] <Amir1>	 halfak: o/ Today is wikidata/ores/IhaveNoIdea day
[14:52:05] <halfak>	 lol
[14:52:12] <halfak>	 o/ Amir1 
[14:52:16] <halfak>	 I'm not going to be very present today. 
[14:52:28] <halfak>	 Mostly prepping for travel and traveling. 
[14:52:42] <Amir1>	 regarding wikilabels, You did the deployment but I think we need to manually add the info_url to models
[14:52:46] <Amir1>	 *campaigns
[14:52:49] <halfak>	 I did it :) 
[14:52:52] <halfak>	 Check the SAL :D 
[14:53:00] <Amir1>	 that's very... fancy
[14:53:04] <halfak>	 \o/ 
[14:53:10] * halfak is very fancy
[14:53:22] * Zppix is also fancy
[14:54:15] <Amir1>	 http://labels.wmflabs.org/ui/enwiki/
[14:54:19] <Amir1>	 I don't see the link :D
[14:56:06] <Amir1>	 I need to manually add them to the database 
[14:56:11] <Amir1>	 hmm, okay
[14:58:03] <awight>	 o/
[14:58:12] <Zppix>	 Hi Adam!
[14:58:23] <Zppix>	 Im offically a year older yesterday
[14:59:03] <awight>	 k I still have two things that might prevent the revscoring 2 deployment: another scap glitch, and something with Extension:ORES being unable to parse some responses.
[14:59:19] <awight>	 Zppix: hey congrats!  You get the sr. citizen discount on the tram yet? Pp
[14:59:23] <awight>	 :p
[14:59:42] <awight>	 that’s all I look forward to these days.
[14:59:43] <Zppix>	 awight: in a smallish town like mine you dont have trams
[14:59:51] <Zppix>	 You have your own car
[14:59:56] <awight>	 Bart (San francisco subway) is punishingly expensive
[15:00:02] <awight>	 ah yes, I’m familiar with the type
[15:00:15] <awight>	 I just moved out of a town that had no sidewalks, that was absolutely killing me.
[15:01:27] <Amir1>	 awight: Regarding the ores extension glitch, I have some stuff to do but I can re-prioritize if you're busy with other things 
[15:02:00] <Zppix>	 awight: my town "has" some sidewalks but since im in a wheelchair they are a pardon my french, a bitch to use
[15:03:12] <awight>	 That’s annoying.  no curb cuts, I’m sure.
[15:03:40] <awight>	 That kind of stuff is so obviously a societal responsibility, I hate when people can’t imagine themselves in that position like tomorrow.
[15:05:07] <Zppix>	 awight: the curbs arent flat they like dive down then up again
[15:06:51] <awight>	 Amir1: I think it might be okay, i.e. just error responses.  I’ll dig around a bit more before committing to the deployment.
[15:14:45] <awight>	 halfak: ORES is timing out like crazy on the new cluster.
[15:20:03] <awight>	 halfak: Here’s a weird lead, https://docs.confluent.io/current/streams/developer-guide.html?highlight=state%20store#state-stores
[15:20:17] <awight>	 Validation the other way around.
[15:22:17] <awight>	 No timeouts at all on beta, though.  urgh, that makes for a difficult judgment call.
[15:22:42] <Zppix>	 Is it possible beta has that new cluster dont awight?
[15:23:00] <awight>	 beta is just a labs thing.  Not sure what you mean though?
[15:23:37] <Zppix>	 Is it possible the timeouts are due to something the old cluster has that new doesnt
[15:23:42] <Zppix>	 Is what i meant
[15:24:48] <awight>	 Amir1: halfak: Can we move the staff meeting, maybe to tomorrow morning?
[15:26:09] <awight>	 Zppix: Definitely a possibiltiy.  The new cluster isn’t being maintained yet, so lots of things could have gone wrong there.  However, beta isn’t a very accurate reproduction of the current production environment either,.
[15:26:20] <awight>	 Amir1: halfak: oops, it’s our backlog meeting that conflicts.
[15:26:27] <awight>	 Amir1: Feel like going to scrum of scrums for us?
[15:31:47] * awight taps the mic
[15:43:01] <wikibugs>	 (03PS1) 10Awight: New ORES codfw cluster isn't provisioned yet [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/387811 (https://phabricator.wikimedia.org/T165170)
[15:49:02] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Operations: Use external dsh group to list pooled ORES nodes - https://phabricator.wikimedia.org/T179501#3726795 (10awight)
[15:49:18] <wikibugs>	 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3726809 (10awight)
[15:49:22] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Operations: Use external dsh group to list pooled ORES nodes - https://phabricator.wikimedia.org/T179501#3726808 (10awight)
[15:55:15] <halfak>	 Hey sorry am on and off AFK getting ready for travel
[15:55:36] <halfak>	 awight, do you want to move the backlog meeting?
[15:56:07] <awight>	 halfak: Yeah, I think it’s better that I try to do this deployment.
[15:56:28] <halfak>	 awight, gotcha.  Sounds good. 
[15:56:49] <awight>	 There’s a good chance that something (celery change) is causing timeouts, but I was hoping to at least get this onto the canary to see how it behaves.
[15:57:14] <halfak>	 awight, I don't think canary is going to work for you 
[15:57:24] <halfak>	 You need to deploy to the whole cluster to test a change to celery. 
[15:57:37] <halfak>	 No guarantee you're getting the right celery node. 
[15:58:02] <awight>	 aww.  Right.
[15:58:15] <awight>	 Well, buckle up for a few potential minutes of downtime I suppose.
[16:00:15] <awight>	 ah there’s a timeout.  Mebbe I won’t even go there.
[16:00:54] <halfak>	 test in ores* should be useful.  Also we have beta for a reason :) 
[16:01:04] <halfak>	 awight, do you think the problem doesn't exist in beta?
[16:01:07] <awight>	 Yeah ores* is screwed
[16:01:16] <awight>	 That’s how it was looking, but now I see beta timing out too.
[16:01:37] <awight>	 My blasted local won’t reproduce this of course
[16:02:21] <halfak>	 Staff? 
[16:04:18] <halfak>	 Let's chat about this for a couple minutes in staff. 
[16:04:28] <awight>	 hargh brt
[16:14:11] <Amir1>	 halfak: I join right now
[16:38:19] <wikibugs>	 10Scoring-platform-team (Current), 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Compare coverage reports of migrating to pytest - https://phabricator.wikimedia.org/T179507#3726970 (10Ladsgroup)
[16:39:19] <wikibugs>	 10Scoring-platform-team (Current), 10articlequality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Article quality campaign for Persian Wikipedia - https://phabricator.wikimedia.org/T174684#3726985 (10Ladsgroup) a:03Ladsgroup
[16:42:31] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: id string in ORES does not include "features" - https://phabricator.wikimedia.org/T179509#3727002 (10Halfak)
[16:54:59] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Operations, 10Scap: Use external dsh group to list pooled ORES nodes - https://phabricator.wikimedia.org/T179501#3727071 (10Halfak) p:05Triage>03Low
[16:56:07] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3724435 (10Halfak) p:05Triage>03High
[16:56:55] <wikibugs>	 10Scoring-platform-team, 10Documentation, 10Easy: Mock JADE discussion page - https://phabricator.wikimedia.org/T179301#3727087 (10Halfak) p:05Triage>03Normal
[16:57:58] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Documentation, 10Easy: List ORES use cases - https://phabricator.wikimedia.org/T154440#3727091 (10Halfak) We have https://www.mediawiki.org/wiki/ORES/Applications
[16:58:06] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Documentation, 10Easy: List ORES use cases - https://phabricator.wikimedia.org/T154440#3727092 (10Halfak) 05Open>03Resolved
[16:58:08] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Documentation, 10Epic: [Epic] Clean up ORES service documentation - https://phabricator.wikimedia.org/T148974#3727093 (10Halfak)
[16:59:10] <wikibugs>	 10Scoring-platform-team, 10Research, 10Research-outreach: Organize a technical workshop for ORES - https://phabricator.wikimedia.org/T141310#3727098 (10Halfak) p:05Normal>03Lowest
[16:59:37] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Documentation, 10I18n: Make ORES documentation translatable - https://phabricator.wikimedia.org/T163786#3727102 (10Halfak)
[17:00:53] <wikibugs>	 10Scoring-platform-team, 10Wikimania-Hackathon-2017, 10Documentation, 10Easy: [Wikimania doc sprint] docs on how to install ORES - https://phabricator.wikimedia.org/T170506#3727109 (10Halfak)
[17:01:36] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Russian-Sites: Announce deployment of wp10 models to ruwiki community - https://phabricator.wikimedia.org/T138623#3727114 (10Halfak) Ping @putnik
[17:03:22] <wikibugs>	 10Scoring-platform-team, 10ORES, 10revscoring, 10artificial-intelligence: Build a sketch of basic ORES model building patterns - https://phabricator.wikimedia.org/T148692#3727117 (10Halfak) /What should get pulled over to MediaWiki and incorporated into the documentation there.
[17:03:32] <wikibugs>	 10Scoring-platform-team, 10ORES, 10revscoring, 10Easy, 10artificial-intelligence: Build a sketch of basic ORES model building patterns - https://phabricator.wikimedia.org/T148692#3727118 (10Halfak)
[17:09:22] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy: Error messages should not contain relative paths or error codes - https://phabricator.wikimedia.org/T175726#3727147 (10awight)
[17:09:45] <wikibugs>	 10Scoring-platform-team, 10Easy: revscoring model_info display should include target prediction value - https://phabricator.wikimedia.org/T177649#3727150 (10awight)
[17:10:11] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Easy: Cached thresholds should be invalidated for new model versions. - https://phabricator.wikimedia.org/T177421#3727156 (10awight)
[17:12:09] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Wikimedia-Logstash, 10Patch-For-Review, 10TestMe: Send celery logs and events to logstash - https://phabricator.wikimedia.org/T169586#3727164 (10awight)
[17:13:04] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Easy: Wikicode is not interpreted in system message - https://phabricator.wikimedia.org/T142406#3727168 (10awight)
[17:13:57] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Wikimedia-Incident: [Spec] CI tests for Wikilabels - https://phabricator.wikimedia.org/T137625#3727172 (10awight) 05Open>03Resolved
[17:15:08] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy: qunit tests for wikilabels - https://phabricator.wikimedia.org/T171083#3727187 (10awight)
[17:15:13] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy: pytests for wikilabels - https://phabricator.wikimedia.org/T171082#3727189 (10awight)
[17:16:36] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Easy: Wire statistics into test model included with our Vagrant role. - https://phabricator.wikimedia.org/T176914#3727193 (10awight)
[17:16:56] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy: Allow Wiki Labels API to list inactive campaigns - https://phabricator.wikimedia.org/T171768#3727195 (10awight)
[17:18:19] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy: Revisions that cannot be retrieved should be skipped - https://phabricator.wikimedia.org/T161102#3727200 (10awight)
[17:19:09] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy: Add dressed up JSON response for browser-based Wiki Labels API requests - https://phabricator.wikimedia.org/T171770#3727205 (10awight)
[17:19:51] <wikibugs>	 10Scoring-platform-team: Investigate using Travis-CI caching - https://phabricator.wikimedia.org/T173040#3727208 (10awight) 05Open>03declined
[17:19:59] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Easy: Wikicode is not interpreted in system message - https://phabricator.wikimedia.org/T142406#3727210 (10Legoktm)
[17:20:11] <wikibugs>	 10Scoring-platform-team, 10revscoring, 10Easy, 10artificial-intelligence: Use multithreading in test_model - https://phabricator.wikimedia.org/T169843#3727212 (10awight)
[17:20:28] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Easy: Wikicode is not interpreted in system message - https://phabricator.wikimedia.org/T142406#2533822 (10Legoktm) @MaxBioHazard in the future please avoid using external image hosting sites and just upload it to phabricator directly, thanks!
[17:20:31] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy, 10I18n: “Would you revert this edit?” in Revision Scoring should support GENDER - https://phabricator.wikimedia.org/T132204#3727216 (10awight)
[17:21:15] <wikibugs>	 10Scoring-platform-team, 10ORES: Meta ORES: UI/API for reviewing/refuting how ORES classifies you and your stuff - https://phabricator.wikimedia.org/T148700#3727220 (10awight)
[17:21:17] <wikibugs>	 10Scoring-platform-team, 10ORES: [Spike] [bike shed] Name the Meta ORES service - https://phabricator.wikimedia.org/T153146#3727219 (10awight) 05Open>03Resolved
[17:23:23] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Easy: Extend the GUI for ORES - https://phabricator.wikimedia.org/T107723#3727225 (10awight)
[17:23:51] <wikibugs>	 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: pytest for flask application of wikilabels - https://phabricator.wikimedia.org/T179015#3727228 (10awight)
[17:24:23] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Easy: Request raw features/datasources from ORES - https://phabricator.wikimedia.org/T133585#3727232 (10awight)
[17:31:52] <Amir1>	 afk
[17:34:59] <awight>	 Amir1: halfak|Mobile: You know what I wouldn’t mind doing?  Reverting the Celery 4 change so we can get revscoring 2 out the door.  We the classic expensive major upgrade problem at this point.
[18:06:54] <wikibugs>	 10Scoring-platform-team, 10ORES: Integrate ORES logging config into the app config scheme - https://phabricator.wikimedia.org/T179522#3727418 (10awight)
[18:15:11] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3727460 (10demon) p:05High>03Unbreak! This is really really spammy in production, raising to UBN!
[18:17:46] <wikibugs>	 10Scoring-platform-team, 10ORES: Consistent TimeoutErrors affecting ores-beta.wmflabs.org - https://phabricator.wikimedia.org/T179524#3727468 (10awight)
[18:18:26] <wikibugs>	 10Scoring-platform-team, 10ORES: Consistent TimeoutErrors affecting ores-beta.wmflabs.org - https://phabricator.wikimedia.org/T179524#3727487 (10awight)
[18:18:29] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180#3727486 (10awight)
[18:20:52] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3727491 (10awight) a:03awight
[18:21:03] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Consistent TimeoutErrors affecting ores-beta.wmflabs.org - https://phabricator.wikimedia.org/T179524#3727492 (10awight)
[18:21:15] <wikibugs>	 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3724435 (10awight)
[18:23:40] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Consistent TimeoutErrors affecting ores-beta.wmflabs.org - https://phabricator.wikimedia.org/T179524#3727499 (10awight)
[18:26:05] <wikibugs>	 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES: ORES extension failing to parse scoring response - https://phabricator.wikimedia.org/T179430#3727501 (10awight) Here are the MW loglines when this happens: ``` Expectation (masterConns <= 0) by MediaWiki::restInPeace not met (actual: 1): [connect...
[18:31:27] <halfak|Mobile>	 o/
[18:33:09] <halfak|Mobile>	 Snowing pretty hard here.  I wonder if my flight is going to be weird
[18:33:34] <paladox>	 heh
[18:35:52] <awight>	 Hopefully they don’t attempt it, I guess?
[18:36:36] <awight>	 I remember some pretty disconcerting stuff in the Hong Kong airport when I was taking off during the “super-typhoon”…. Including a display explaining how planes were designed to handle cross-winds :)
[18:50:09] <wikibugs>	 (03PS1) 10Awight: Always assume a v3 API response. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/387847 (https://phabricator.wikimedia.org/T179430)
[18:51:43] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Always assume a v3 API response. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/387847 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[18:53:23] <wikibugs>	 (03PS2) 10Awight: Always assume a v3 API response. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/387847 (https://phabricator.wikimedia.org/T179430)
[18:55:18] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Always assume a v3 API response. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/387847 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[18:57:04] <wikibugs>	 (03PS3) 10Awight: Always assume a v3 API response. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/387847 (https://phabricator.wikimedia.org/T179430)
[18:59:02] <awight>	 Amir1: halfak|Mobile: Anyone feel like kicking an UBN patch?  https://gerrit.wikimedia.org/r/387847
[19:00:55] <awight>	 Amir1: FYI, there’s some response to the nose->pytest news on wikitech-l
[19:01:21] <awight>	 legoktm: Any interest in pinch-reviewing an UBN fix?  https://gerrit.wikimedia.org/r/387847
[19:02:19] <awight>	 The patch is approximately equally ugly with the existing code, but should stop some annoying logspam and failed API?oresscores requests.
[19:03:43] <awight>	 RoanKattouw: ^ share the pain? :D
[19:04:00] <RoanKattouw>	 Looking
[19:04:07] <awight>	 ty
[19:04:36] <awight>	 It’s horrible that we’ve been unpacking API responses in class responsible for caching, but this doesn’t really tip the balance in that regard.
[19:05:54] <RoanKattouw>	 Hmm but now the unpacking is moving to the other side of the caching, right?
[19:06:05] <RoanKattouw>	 Could that break things for data that's already cached in the "wrong" format?
[19:06:36] <awight>	 I don’t think we ever cache the outer keys
[19:07:03] <awight>	 yeah, processRevision decouples us from that junk.
[19:11:49] <RoanKattouw>	 OK cool
[19:12:22] <wikibugs>	 (03CR) 10Catrope: [C: 032] Always assume a v3 API response. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/387847 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[19:13:10] * awight tips hat!
[19:13:48] <wikibugs>	 (03Merged) 10jenkins-bot: Always assume a v3 API response. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/387847 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[19:17:15] <wikibugs>	 (03PS1) 10Awight: Always assume a v3 API response. [extensions/ORES] (wmf/1.31.0-wmf.5) - 10https://gerrit.wikimedia.org/r/387851 (https://phabricator.wikimedia.org/T179430)
[19:17:26] <wikibugs>	 (03CR) 10Awight: [V: 032 C: 032] Always assume a v3 API response. [extensions/ORES] (wmf/1.31.0-wmf.5) - 10https://gerrit.wikimedia.org/r/387851 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[19:17:43] <wikibugs>	 (03PS1) 10Awight: Always assume a v3 API response. [extensions/ORES] (wmf/1.31.0-wmf.6) - 10https://gerrit.wikimedia.org/r/387852 (https://phabricator.wikimedia.org/T179430)
[19:17:52] <wikibugs>	 (03CR) 10Awight: [V: 032 C: 032] Always assume a v3 API response. [extensions/ORES] (wmf/1.31.0-wmf.6) - 10https://gerrit.wikimedia.org/r/387852 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[19:18:55] <wikibugs>	 (03Merged) 10jenkins-bot: Always assume a v3 API response. [extensions/ORES] (wmf/1.31.0-wmf.5) - 10https://gerrit.wikimedia.org/r/387851 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[19:19:48] <wikibugs>	 (03Merged) 10jenkins-bot: Always assume a v3 API response. [extensions/ORES] (wmf/1.31.0-wmf.6) - 10https://gerrit.wikimedia.org/r/387852 (https://phabricator.wikimedia.org/T179430) (owner: 10Awight)
[19:33:31] <Amir1>	 awight: I just got back (wrote afk above)
[19:33:42] <Amir1>	 but 1- I will answer in wikitech-l
[19:33:46] <awight>	 Amir1: Yes totally!  That was actually really helpful.
[19:33:59] <awight>	 I knew to ping reserve CR friends...
[19:34:09] <awight>	 Hadda get that fix out UBN style, but I think it went well.
[19:34:21] <Amir1>	 2- I'm okay with reverting celery change but I think halfak|Mobile needs to chime in too :D
[19:34:46] <Amir1>	 is there anything I can help now, just let me know :)
[19:35:53] <awight>	 I think we’re good!
[19:36:06] <awight>	 I’m going to stick my head in the TimeoutError until someone flushes
[19:37:10] <Amir1>	 okay :D
[19:37:19] <Amir1>	 and I have some stuff to do :D
[19:39:01] <awight>	 Amir1: I might have some instrumentation things if you’re still around later...
[20:05:58] <Amir1>	 awight: just ping me
[20:06:11] <awight>	 right on
[20:14:29] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Consistent TimeoutErrors affecting ores-beta.wmflabs.org - https://phabricator.wikimedia.org/T179524#3727831 (10awight) Here are debug logs of failed and succeeded requests, I'm combing through the wdiff now.  {F10557823}  {F10557824}  wdiff succeeded failed > wd  {F1...
[20:15:15] <awight>	 biab
[20:51:35] <awight>	 Amir1: holler if you want CR on any of that
[20:55:07] <travis-ci>	 wiki-ai/ores#801 (travis - 7db9fb3 : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/ores/builds/295949144
[20:55:21] <awight>	 Hey!
[20:55:31] <Zppix>	 Hey!
[20:55:51] <travis-ci>	 wiki-ai/editquality#6 (travis - a8b8b86 : Ladsgroup): The build was fixed. https://travis-ci.org/wiki-ai/editquality/builds/295947973
[20:56:09] <Amir1>	 awight: Do you have a min to review: https://github.com/wiki-ai/ores/pull/228 and https://github.com/wiki-ai/wikilabels/pull/209
[20:56:48] <awight>	 Zppix: / SOURCE
[20:56:56] <awight>	  /CTCP SOURCE
[20:57:00] <awight>	 ah ruined my own joke
[20:57:06] <awight>	 Amir1: will do
[20:57:20] <Amir1>	 https://github.com/wiki-ai/editquality/pull/105
[20:57:41] <awight>	 Amir1: Are you worried about the failing CI? https://travis-ci.org/wiki-ai/ores/builds/295949857?utm_source=github_status&utm_medium=notification
[20:58:00] <Amir1>	 Nah, It should not have anything to do with this
[20:58:09] <Amir1>	 I think it's a random failure
[20:58:37] <Zppix>	 awight: ??
[20:59:35] <awight>	 Amir1: Looks like it’s this line, https://travis-ci.org/wiki-ai/ores/builds/295949857#L1152
[20:59:46] <awight>	 But yeah, unrelated to the PR
[21:00:01] <Amir1>	 It's the line before acutally 
[21:00:04] <Amir1>	 *actually 
[21:00:37] <Amir1>	 awight: I think it's related to your work
[21:00:41] <Amir1>	 regarding timeout
[21:01:12] <awight>	 Very possible, it’s the same error message.
[21:01:17] <Zppix>	 I may go thru our repos and do some branch clean up, any objections?
[21:02:43] <awight>	 Zppix: Thank you!
[21:04:36] <Zppix>	 awight: the one liner branch on revs do you still need it?
[21:04:49] <awight>	 Zppix: link pls?
[21:05:03] <Zppix>	 awight: https://github.com/wiki-ai/revscoring/tree/oneliners
[21:05:56] <awight>	 Zppix: Wait, why was mwbase deleted?
[21:06:05] <Zppix>	 Its an old branch?
[21:06:07] <awight>	 I was imagining you were just going for the merged branches.
[21:06:07] <Zppix>	 Stale
[21:06:19] <Zppix>	 I can restore
[21:06:23] <awight>	 Nah let’s leave the stale branches, feel free to ping or something.
[21:06:40] <awight>	 Sorry we run an unpruned ship around here...
[21:06:45] <Zppix>	 Restored
[21:07:44] <Zppix>	 I should made it clear xD
[21:07:47] <Zppix>	 Its my fault
[21:08:10] <awight>	 hehe no I think it’s generally a good practice, but yeah probably best done by pinging the author first
[21:08:17] <awight>	 k lemme see that oneliners branch...
[21:09:03] <awight>	 hehe oneliners is probably still good
[21:09:41] <Zppix>	 Ill just do pr merged branches
[21:09:55] <awight>	 +1 for lower-hanging fruit to start with
[21:10:09] <awight>	 Maybe when everyone is around, we can prune as a group.
[21:11:09] <Zppix>	 Ok
[21:11:22] <Zppix>	 What about one liner branch awight ?
[21:11:31] <awight>	 that one just needs a rebase, it’s good.
[21:11:36] <Zppix>	 Ok
[21:14:55] <Zppix>	 Let me know if theres any other maintenance tasks i could help do awight
[21:18:54] <awight>	 Zppix: We did just preen the “Maintenance/cleanup” column in https://phabricator.wikimedia.org/project/view/1901/ , and things that we believe could be done independently and with reasonably limited effort are tagged with “Easy”.
[21:19:02] <awight>	 LMK if you see anything that looks fun.
[21:19:11] <Zppix>	 I meant like cleanup and stuff lol
[21:19:18] <Zppix>	 Ik that
[21:20:54] <awight>	 Zppix: sure!  For example, https://phabricator.wikimedia.org/T175726 is just going through the code and cleaning up error statements.
[21:21:46] <Zppix>	 Also im a mentor in GCI so is Amir1 so let me know if you come up with good gci tasks and ill be sure we get them added
[21:21:51] <Zppix>	 awight: ^
[21:22:03] <awight>	 Here’s the board filtered for “Easy”, https://phabricator.wikimedia.org/project/board/1901/query/vbEafoSDNeyj/
[21:22:24] <awight>	 Zppix: Great, thanks for taking on that responsibility!
[21:23:25] <Zppix>	 Its my first time so hopefully i survive :P
[21:35:38] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Consistent TimeoutErrors affecting ores-beta.wmflabs.org - https://phabricator.wikimedia.org/T179524#3728007 (10awight) I instrumented the four places we can raise this error, and we're hitting this one, a Celery exception from score_result.get, caught by _process_mis...
[22:03:45] <Zppix>	 Amir1: is The travis error on unrelated
[22:30:33] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Upgrade celery to 4.1.0 for ORES - https://phabricator.wikimedia.org/T178441#3728184 (10awight) @Halfak I'm having fun investigating T179524, but my instinct is currently to unblock Revscoring 2 by reverting this change, and trying to upgrade to...