[07:09:06] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2017): UX check RC Filters in beta (revscoring 2.0/thresholds release) - https://phabricator.wikimedia.org/T178395#3690748 (10Catrope) To clarify: @Etonkovidova's concern is that, because of the version... [07:39:43] 10Scoring-platform-team (Current), 10ORES, 10Services (watching): ORES internal server error for edit with many added links - https://phabricator.wikimedia.org/T179064#3714748 (10Pchelolo) [10:29:54] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: ORES service erroring, in a way that throws exceptions in Extension:ORES - https://phabricator.wikimedia.org/T179107#3713464 (10Ladsgroup) The URL still sends timeout error. [10:44:09] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 2 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715032 (10hoo) [10:44:21] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 2 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715052 (10hoo) [10:47:07] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715061 (10Ladsgroup) [11:33:57] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715115 (10Ladsgroup) @hoo and I sampled x-cache from wberequest dataset for the hour of 19 yesterday for two... [11:51:50] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715170 (10hoo) This has some potentially interesting patterns: `watchlist, recentchanges, contributions, log... [12:06:30] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715032 (10Marostegui) From those two masters's (s4 and s5) graphs, we can see that whatever happened, happene... [12:08:47] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715242 (10thiemowmde) p:05Triage>03High The tasks description talks about ongoing investigation. Is there... [12:12:30] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715253 (10hoo) p:05High>03Triage >>! In T179156#3715242, @thiemowmde wrote: > The tasks description talks... [12:34:35] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715319 (10BBlack) Copying this in from etherpad (this is less awful than 6 hours of raw IRC+SAL logs, but sti... [12:37:30] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715335 (10BBlack) My gut instinct remains what it was at the end of the log above. I think something in the... [12:47:50] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 3 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715381 (10BBlack) Unless anyone objects, I'd like to start with reverting our emergency `varnish max_connecti... [12:56:31] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715432 (10hoo) I think I found the root cuase now, seems it's actually related to the WikibaseQualityConstra... [13:00:10] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715446 (10BBlack) >>! In T179156#3715432, @hoo wrote: > I think I found the root cuase now, seems it's actu... [13:01:23] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715447 (10hoo) >>! In T179156#3715446, @BBlack wrote: > >>>! In T179156#3715432, @hoo wrote: >> I think I fo... [13:34:04] 10Scoring-platform-team, 10Research, 10Research-management: Figure out long term viability/funding for revscoring - https://phabricator.wikimedia.org/T106222#3715835 (10Halfak) Yup! [13:34:17] 10Scoring-platform-team (Current), 10Research, 10Research-management, 10Research-and-Data-2016-Q1: Revscoring tasks from Wikimania discussions - https://phabricator.wikimedia.org/T106398#3715838 (10Halfak) [13:34:19] 10Scoring-platform-team, 10Research, 10Research-management: Figure out long term viability/funding for revscoring - https://phabricator.wikimedia.org/T106222#3715836 (10Halfak) 05Open>03Resolved a:03Halfak [13:35:03] o/ Amir1 [13:40:01] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2017): UX check RC Filters in beta (revscoring 2.0/thresholds release) - https://phabricator.wikimedia.org/T178395#3715925 (10Halfak) Agreed. I believe that we have a maintenance script that is commonly... [14:07:27] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:07:57] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716009 (10Halfak) [14:08:21] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:08:32] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:09:57] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) See discussion of "datasets_p" and what we'd like to do with datasets like this one in T173511. My goal in fili... [14:11:19] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716024 (10Halfak) [14:12:27] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) [14:16:09] 10Scoring-platform-team (Current), 10DBA, 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) @diego, this is the task for loading that article quality dataset I showed you through #Quarry a few days ago.... [14:26:18] halfak: I just got back from the volunteer meeting [14:26:23] o/ [14:26:38] Great. Just finishing up my email/phab stuff. [14:26:47] Awesome! [14:27:03] Thanks for the merge, re. revscoring PR. I added the note you asked for [14:27:06] we've got some discussion with Catrope re. running the ORES ext maintenance scripts. [14:27:12] Saw that :) [14:27:24] I'll get the tasks moved around on the workboard shortly :) [14:27:45] okay, what was about? [14:27:46] :D [14:28:59] * halfak looks for the task. [14:33:22] Amir1, https://phabricator.wikimedia.org/T178395 [14:36:07] halfak: saw it, let's run the maintenance script, I would just work [14:36:24] halfak: also, the ores wasn't the reason for the outage [14:36:34] great. [14:36:39] it was a script in wikidata [14:36:41] The 5xx's yesterday? [14:39:05] halfak: yeah [14:39:18] We are working on the fix [14:39:35] Looks like the 5xx's stopped after a few hours yesterday [14:42:10] because we rollbacked to wmf.4 [14:42:16] like we disabled everything [15:03:31] 10Scoring-platform-team, 10ORES, 10Operations, 10Release-Engineering-Team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3716144 (10Lucas_Werkmeister_WMDE) > (Permalink: https://grafana.wikimedia.org/dashboard/db/wikidata-quality?p... [15:20:27] halfak: Amir1: I’m ready to do battle with wheels. [15:20:38] go on! [15:20:38] \o/ great! [15:20:49] * awight mounts steed [15:20:53] I'm almost done with my email backlog :'( [15:21:22] I need to be afk for some hours, will be back. Probably can't work at all today because of conferance stuff [15:21:44] Will work on other day for ORES [15:21:55] awight lol [15:22:00] doint forget the armour [15:22:02] heh [15:22:02] Amir1, I see. Can you respond on that one task re. maint scripts [15:22:17] That's the only thing that I think is critical [15:22:25] * awight slashes at “armour” for including a wheel-shaped letter. [15:22:26] Though, honestly, I could just confirm what we talked about earlier. [15:28:18] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716248 (10Marostegui) Not sure what is needed from us (DBAs) as we do not handle create statements (https://wikitech.wikimedia.org/wiki/Sch... [15:35:18] 10Scoring-platform-team (Current), 10ORES, 10Wikidata: Wikidata MediaWiki API timing out on ORES extractor requests - https://phabricator.wikimedia.org/T179112#3716276 (10awight) 05Open>03declined The API has recovered. [15:35:20] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: ORES service erroring, in a way that throws exceptions in Extension:ORES - https://phabricator.wikimedia.org/T179107#3716278 (10awight) [15:40:48] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: ORES service erroring, in a way that throws exceptions in Extension:ORES - https://phabricator.wikimedia.org/T179107#3716285 (10awight) I hear that Wikidata API performance has recovered, yet I can still get a timeout some of time, by using the u... [15:44:37] Who should I bug about coercing ores-misc-01, scb1002, and ores*.eqiad.wmnet to use the same platform? [15:44:58] awight, we have a task for moving ores* to stretch. I' [15:45:09] d then like to move ores-misc-01 to stretch and everything else [15:45:22] akosiaris, is tuned into the stretch for ORES* task [15:45:38] https://phabricator.wikimedia.org/T171851 [15:45:53] Think it’s worthwhile to upgrade scb* as well? I’d sort of prefer to not ask for that, since it’s not dedicated for ORES and we’d like to abandon anyway. [15:46:02] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, and 2 others: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3716304 (10Halfak) [15:46:04] 10Scoring-platform-team, 10ORES, 10Operations: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3716303 (10Halfak) [15:46:05] ty for the link [15:46:09] awight, +1 [15:46:11] kk [15:46:23] It would be nice if we could include stretch in a clean break from SCB [15:58:27] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10bd808) >>! In T179187#3716248, @Marostegui wrote: > Not sure what is needed from us (DBAs) as we do not handle create statements... [15:58:31] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716350 (10Halfak) I see. Who might create this DB on the replicas then? @bd808? [16:15:09] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716381 (10Halfak) I see what you're saying, @bd808. We're currently in a reduced capacity and have been for a while. I was under the impr... [16:15:56] awight, I added something to SOS that should have been there for a while. See like 45 of https://etherpad.wikimedia.org/p/scoring_staff [16:16:24] I understand that Cloud Services does not attend SOS, so I'm not sure this will effectively communicate our blocker to them, but I can't see anything else we can try. [16:16:39] My attempts of file-a-task-and-talk-about-it were unsuccessful. [16:17:12] If we raise it at SOS and Cloud doesn't have a rep, I'm guessing that ggellerman might go knocking on their door for us. [16:18:08] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716390 (10Halfak) [16:18:42] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3715994 (10Halfak) Removed from description but pasted here as a conversation point: > I'm [planning a workshop for IWSC](https://meta.wiki... [16:43:55] halfak: ah, that’s an important one. [16:44:03] Sorry, got IRL jacked by a friendly expat [16:55:13] 10Scoring-platform-team, 10ORES, 10Operations: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3478146 (10awight) @akosiaris We ran into a little glitch, that wheels created on ores-misc-01 aren't compatible with the new ores* cluster. I'm thinking we should reorder the d... [16:57:48] 10Scoring-platform-team (Current), 10ORES, 10Release-Engineering-Team, 10Patch-For-Review: Wheels built on ores-misc-01 are incompatible with ores* and scb* - https://phabricator.wikimedia.org/T179095#3716445 (10awight) [17:01:00] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716454 (10bd808) @Halfak is the `datasets_p` name important or would it be ok for your curated tables to live an a database named with the... [17:14:59] 10Scoring-platform-team (Current), 10ORES, 10Release-Engineering-Team, 10Patch-For-Review: Wheels built on ores-misc-01 are incompatible with ores* and scb* - https://phabricator.wikimedia.org/T179095#3716483 (10awight) Comparing environments: | Host | OS | Python | pip | | scb* | Jessie 8.6 | 3.4.2 | 1.5... [17:15:04] halfak: Wheels snafu solved. [17:15:15] Great! What was the issue? [17:16:43] the “manylinux” wheels can only be installed by pip >= 8.1 [17:16:51] So I’m just going to update pip on the target machines... [17:16:54] using a wheel. [17:17:24] Now to test this out on beta, and erase my tracks where I raise a stink about OS upgrades. [17:17:27] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716484 (10Halfak) Sure! That could work if that's easier to set up. If possible, I'd love to have a more intuitive DB name since we're... [17:18:17] Oh! Gotcha. [17:18:34] Would an alternative be to downgrade pip on ores-misc-01? [17:18:46] No, I tried that and for some reason it doesn’t work. [17:18:49] weird. [17:18:56] So pip creates wheels it can't install? [17:18:58] That was my first thought, cos I imagined it would be safer. [17:19:01] I… guess so. [17:19:07] really strange business. [17:19:45] agreed. [17:20:19] I just finished my emails. It seems like you have the wheel business figured out. Maybe I should get back to work on JADE stuff? [17:21:03] * halfak looks at workboard [17:21:11] That sounds great! [17:21:45] Oh I could finish off the fiwiki models. I have everything built. [17:21:47] Once this works, I’ll be able to get back to stress testing the new cluster, + Celery 4 [17:21:52] I'll just need to get the PR in place. [17:25:12] I also need to experiment with the scap script… Guess I’ll do that on a new cluster node. [17:25:54] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716520 (10bd808) @Halfak I think that having multiple users owning tables in a single db will end up being problematic from the server mana... [17:26:31] halfak: Feel free to CR https://git.io/vFkT7 and https://gerrit.wikimedia.org/r/386898 [17:28:38] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3716545 (10Halfak) I'm OK with owning my own DB, but note that I volunteered to manage curation of datasets for a shared DB. That has the b... [17:32:29] (03PS1) 10Awight: Revert "Blindly choose a timeout of 15 seconds" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386900 [17:32:31] (03PS1) 10Awight: Install pip 9 to allow us to use multilinux wheels [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386901 (https://phabricator.wikimedia.org/T179095) [17:32:36] halfak: And these two ^ [17:33:37] awight, adding a package to the frozen-requirements.txt means nothing and will get overwritten [17:33:45] Gotta add it to the requirements. :) [17:34:05] aha okay I was just using that as the source for building wheels, but I see whatchu mean [17:34:10] https://github.com/wiki-ai/ores-wmflabs-deploy/commit/5c3676e2904da173199fbe681f5c64102a0cc7f1 [17:34:21] woops. Accidentally submitted that direct to master :| [17:34:49] hehe I was gonna accuse you of being a pusher [17:35:15] I’m a little sad about wmflabs-deploy vs prod-deploy, maybe you can give me the history lesson one day. [17:35:37] (03CR) 10Halfak: [V: 032 C: 032] Revert "Blindly choose a timeout of 15 seconds" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386900 (owner: 10Awight) [17:35:38] I guess it’s just that there’s no scap on labs. [17:36:14] Sorry I forgot whatever you explained the last time I asked, c. July [17:36:38] (03CR) 10Halfak: [C: 04-1] "Please add to requirements.txt ala https://github.com/wiki-ai/ores-wmflabs-deploy/commit/5c3676e2904da173199fbe681f5c64102a0cc7f1" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386901 (https://phabricator.wikimedia.org/T179095) (owner: 10Awight) [17:37:35] awight, I think that we thought we'd spend more time deploying experimental models to labs than we actually have. [17:37:45] (03PS2) 10Awight: Install pip 9 to allow us to use multilinux wheels [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386901 (https://phabricator.wikimedia.org/T179095) [17:38:03] seems like a good use for branches... [17:41:16] We'd thought about that but it wasn't clear how we could do mutual modifications across branches and it was nice to keep the labs version in github where it's easier for others to work with us. [17:42:01] Cool, I’m capable of keeping the complaints to once every few months, as long as you don’t mind pointing out when I screw up :-0 [17:43:29] :D Maybe we can work out a better workflow some time soon. [17:44:13] just this one left to re-review, then I can do beta and ores1002… https://gerrit.wikimedia.org/r/386901 [17:44:28] I’ve totally been cheating and using the new cluster as a mega-beta btw. [17:44:35] It’s quite luxurious :) [17:51:12] Github should have a switch to prevent pushing to master by mistake... [17:53:34] halfak: ^ you have another minute? [17:59:38] Amir1: Are you online by chance? https://gerrit.wikimedia.org/r/#/c/386901/ [18:02:40] (03CR) 10Halfak: [V: 032 C: 032] Install pip 9 to allow us to use multilinux wheels [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386901 (https://phabricator.wikimedia.org/T179095) (owner: 10Awight) [18:02:43] {{done}} [18:02:43] How efficient, halfak! [18:02:45] o/ [18:06:10] 10Scoring-platform-team, 10Patch-For-Review: ORES deployment submodules should point to phabricator HTTPS repos. - https://phabricator.wikimedia.org/T179009#3710070 (10mmodell) @demon: fixing phabricator is going to be difficult. Git was never meant to be so huge :( [18:17:08] halfak: fyi, beta deployment is still fighting me, so I’m going straight to ores1002. [18:21:15] Nice, my sketchy scap shell change worked. [18:22:07] *yes* and the wheels installed [18:26:30] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Upgrade celery to 4.1.0 for ORES - https://phabricator.wikimedia.org/T178441#3692315 (10awight) I'm seeing a new error, http://0.0.0.0:8081/v3/scores/enwiki/123456/damaging ``` Traceback (most recent call last): File \"./ores/wsgi/routes/v3/s... [18:31:51] halfak: Nasty glitch seemingly related to Celery 4? ^ [18:32:06] I’ll dig into it but thought I would ask in case you’ve seen it before. [18:34:36] right. Thats new celery [18:38:02] I mean, yes this is using the new celery, but I don’t see anything in the release notes about the correlation_id key changing or going away. [18:38:31] Also, it’s vacillating between this correlation_id error and a timeout :( [18:38:49] Correlation_id appears! [18:43:38] * awight squints [18:56:24] halfak: random 20k entries from templatelinks table having "WikiProejct_%" template yielded very unbalanced results [18:56:31] i ran this - https://quarry.wmflabs.org/query/22495 [18:57:40] Bodies of water got about 4k, music, games and toys, religion roughly 2.5-3k and majority of them being zero or close to zero [19:00:38] What I don’t get is why these bugs, e.g. the correlation_id thing, are making it through CI. [19:02:05] i'm thinking something like https://dpaste.de/BZf0 , to get say 5k pages per mid-level wikiproject category by running a separate query for each of this mid-level entry supplying the list of individual wikiprojects associated with the mid-level entry in each query [19:10:44] awight: do you happen to know if we can connect to the mysql db in python as shown here? - http://paws-public.wmflabs.org/paws-public/User:Jtmorgan/NPP_delete_vs_deleted.ipynb [19:10:58] i'm mostly confused about the host part in connect_to_db function [19:11:42] codezee: hi! I haven’t tried it, but that makes sense. Maybe start by dumping os.environ? [19:12:30] awight: its on my local system so unless i configure MYSQL_HOST it will be non-existent [19:12:41] Ah, yeah that’s for sure. [19:12:49] Those would be set by PAWS [19:13:00] i;m just not sure if we can run this from local system or only labs? [19:13:21] Do you have the databases cloned to your local system? [19:13:54] awight: no, juggling with enwiki db would be a humongous task :P [19:13:59] *locally [19:14:10] hehe, well you could clone just a subset... [19:14:11] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Wheels built on ores-misc-01 are incompatible with ores* and scb* - https://phabricator.wikimedia.org/T179095#3716820 (10greg) [19:14:41] but yeah, I don’t think it would be pleasant to run mysql locally against labs dbs, some SSH tunneling would be involved. [19:15:17] awight: would running this on ores-compute provide necessary access? [19:15:37] J-Mo: might know, its his script, don't know if he's online [19:16:14] codezee lemme take a look [19:16:47] codezee: I think all labs machines can access the db replicas, yes [19:16:58] Maybe even run it within PAWS? [19:17:44] awight: whats PAWS? o.O [19:18:15] yes, codezee this should work as long as you have a labs MySQL account. https://wikitech.wikimedia.org/wiki/PAWS [19:18:31] https://paws.wmflabs.org/paws/hub/login [19:18:44] authenticates with your WM account via oauth [19:19:52] oh, i see, this should be helpful ,thanks! [19:22:18] np. don't trust the results of that notebook, btw codezee. The sampling criteria I used were flawed [19:23:50] J-Mo: i see, i was actually using the notebook for inspiration on the technical details of how to query the db using python and all, for a different use case [19:24:02] makes sense [19:24:09] O/ [19:25:09] J-Mo: i have some local data that my code on paws needs to use, i guess i can simply copy paste that in the notebook as static code, right? [19:26:17] depending on the size, you can upload the file directly via your Hub home directory. Or you can host the data somewhere open on the web and use wget to pull it in. [19:27:18] wget works if you create a new 'terminal' in PAWS (via the Home directory interface), and I think you can also invoke it directly in a Jupyter notebook by pre-pending a '!' before the wget command. like: !wget url_to_dataset [19:28:55] Personally, I’ve used CSV files in the same directory as the ipynb script [19:29:16] J-Mo: oh, anyways, it wasn't big so i have uploaded it, now i should be able to refer to it in code as if its in the current directory, right? [19:29:25] I'm not 100% sure what the limitations of PAWS are for file size uploads, or whether wget is fully supported in this Hub instance. madhuvishy probably knows more—she's maintaining PAWS, IIRC [19:29:34] codezee yes! [19:30:02] (note: madhuvishy looks like codezee doesn't need your help right now, sorry for the extraneous ping) [19:30:48] codezee: Here’s what I’ve done with CSVs, https://github.com/adamwight/2015_board_election_analysis/blob/master/Board%20Election%20analysis.ipynb [19:33:07] awight: interesting, the format is something i was about to use to draw a graph, so i can simply take this one :D [19:33:24] i mean the vertical bar graph thing [19:33:33] Code reuse FTW! [19:33:37] I’d be honored. [19:34:20] BTW, I recently played with bokeh for plotting and although it was kinda nice locally, it’s no good on Github cos the images aren’t cached. https://github.com/adamwight/thresholds_diagrams/blob/master/Thresholds%20diagrams.ipynb [19:34:34] Reduce reuse recycle xD [19:38:17] J-Mo: btw just curious what was the flaw in the sampling criteria of your notebook? since at one time i had given it a thorough read so it'd be interesting to know [19:56:08] halfak: So, the correlation_id thing only happens on real servers of course and not under mw-vagrant. [19:56:12] https://ores-beta.wmflabs.org/v3/scores/enwiki/123456 FYI [19:58:07] Maybe the testing revscorer doesn’t use the child task pattern? [19:59:23] I’m thinking of setting up the editquality repo under vagrant, halfak Amir1 does that sound like a good use of time? [20:06:58] awight, not sure how there's a difference. [20:07:05] I can do some testing though [20:09:18] OK found something that might be related, wheels only have Celery 3 for some reason. [20:09:20] what… the [20:10:47] Agreed. WTF [20:11:00] Oh! I have an idea for why that is. [20:11:25] Nope. Nevermind. I was wrong [20:11:33] Maybe pull new wheels into the submodule? [20:11:53] * awight facepalms all over [20:12:01] https://phabricator.wikimedia.org/source/ores-deploy-wheels/ [20:12:09] Looks like it has celery 4.1 [20:12:15] O_o that is good [20:23:18] I'm guessing there's a version mismatch that's at play here. [20:23:35] That's the thing that would make the most sense to me. [20:24:16] (03PS1) 10Awight: Update wheels to get pip9 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386954 [20:25:08] The deployed boxes only have Celery 3, I think I forgot to scap -r HEAD or something, after rev-locking to the stable version. [20:25:30] (03CR) 10Awight: [V: 032 C: 032] "Self-merging submodule update" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386954 (owner: 10Awight) [20:26:41] i'm getting 'u"User xxxxx has exceeded the max_user_connections' resource (current value: 10)" even though i'm having only one other parallel query in quarry [20:27:39] codezee: Maybe you have zombie queries? [20:27:50] awight: how do i figure that out? [20:28:00] Try logging into labs, opening mysql and running: show full processlist [20:28:20] awight: also, how can i kill this - https://quarry.wmflabs.org/query/22493 its a very inefficient one i ran by mistake and its been running since 3 days [20:28:31] That will only show the queries for the current user, so you might have to dig around to get the quarry user creds. [20:29:05] should be as easy as “kill ”, where query id is returned by the processlist statement. [20:29:19] awight: labs as in bastion host? [20:29:24] or some other server [20:29:37] codezee: Good question. I think any labs host. [20:29:43] They should all have access to the dbs [20:29:55] ores-compute, for example [20:31:14] awight: ores-compute-01 is working? i'm getting "channel 0: open failed: administratively prohibited: open failed" [20:31:38] codezee: oops, we probably decommissioned that. Try ores-misc-01.eqiad.wmflabs [20:31:56] ok [20:32:56] halfak: More version woes. mwtypes.User doesn’t exist... [20:33:06] We have mwtypes 0.2.0 installe [20:33:08] d [20:33:30] awight: can't do from ores-misc:( - ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) [20:33:58] halfak: Again… this really should have been caught by CI, I think we should spend a while renovating our tests. [20:34:16] codezee: You’ll need to specify the host, user + passwd [20:34:25] maybe dump os.environ from your PAWS script? [20:35:46] 10Scoring-platform-team, 10Patch-For-Review: ORES deployment submodules should point to phabricator HTTPS repos. - https://phabricator.wikimedia.org/T179009#3716895 (10demon) I mean we need to fix deploying large blobs (cf: git-lfs, git-fat, etc tasks) [20:37:43] halfak: So we need mwtypes 0.3.0 [20:38:40] Rather than just hack that in though, I’d like to know why it wasn’t automatically required. [20:41:02] editquality/requirements.txt asks for 0.2 [20:41:47] looks i i wasn't responsible for the exhaustion and it was using a common user id [20:42:48] codezee: ooh that makes sense [20:43:46] awight: although i went inside paws and looked around, it was as simple as simply doing show full processlist in quarry [20:43:59] :) I like it [20:44:27] I wish we were supporting quarry more, it’s a great tool [20:44:35] Especially for people without NDA access... [20:45:02] awight: yes, its a very useful thing to get things without the need to get your hands dirty [20:45:28] and paws is quite useful too for quick prototyping without worrying of all the setup [20:45:31] Or to get your hands dirty without the keys to the walled garden [20:45:59] :D [20:46:11] Yeah PAWS too, I love it but the few opportunities I’ve had to recommend it, it’s been down that day, cos it seems to not be officially supported. [20:46:40] oh, i was not aware of that [20:47:05] It’s been a labor of love. [20:47:49] halfak: harrgh, can you remind me how https://phabricator.wikimedia.org/source/editquality.git and git@github.com:wiki-ai/editquality.git are related? [20:49:07] phab mirrors from github [20:49:09] minor delay [20:51:18] don't know why query is taking too long today even with a LIMIT of 10! [20:54:29] one thing with paws is that when i'm running a query which takes time, i'm getting no indication if its running or not... [20:54:34] That mwtypes issue should throw an error. damn it. [20:54:47] or where's the code waiting [20:58:40] halfak: Hmm, I’m fine with it, but feel like we’re circumventing WMF’s security model by a bit. [20:59:05] awight, security signed off on the strategy [20:59:14] We do review the changes before merging to the gerrit repo that controls submodule deployment, so it’s reasonable. [20:59:15] -._o_.- [20:59:19] hehe [21:00:08] halfak: CR please, ^ [21:00:28] I’ll prepare the wheels and submodule stuff... [21:04:20] halfak: https://gerrit.wikimedia.org/r/386963 [21:04:34] * awight quietly curses linus [21:04:51] a thing has to be really bad to make me nostalgic for svn. [21:07:42] (03PS1) 10Awight: Bump to mwtypes==0.3.0 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386964 [21:07:51] halfak: and ^ if you would [21:09:51] awight upstream gerrit have made notedb the default for new install with https://gerrit-review.googlesource.com/#/c/gerrit/+/137111/ :) [21:09:59] and polygerrit has a new change header heh [21:10:46] Looks positive, https://gerrit-review.googlesource.com/Documentation/note-db.html [21:11:04] yep [21:11:14] i've been running it locally anyways [21:14:41] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Upgrade celery to 4.1.0 for ORES - https://phabricator.wikimedia.org/T178441#3716961 (10awight) Point of information, I botched the deployment and the above error was with Celery 3, so isn't related to this task. [21:18:12] (03CR) 10Awight: [V: 032 C: 032] "Self-merging submodule bump." [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386964 (owner: 10Awight) [21:20:11] awight, got distracted. Waiting on CR from me for anything? [21:20:28] hehe sorry, I went ahead and self-merged [21:20:36] just submodule bumps to get mwtypes-0.3 in there [21:22:37] Cool with me [21:23:24] [2017-10-27T21:22:37] FileNotFoundError: [Errno 2] No such file or directory: 'submodules/editquality/models/eswikibooks.reverted.gradient_boosting.model' [21:23:28] * awight shakes head [21:24:11] *where* did that config come from [21:25:01] Looks like another version mismatch. [21:25:11] What deploy repo are you working from? [21:25:50] https://gerrit.wikimedia.org/r/mediawiki/services/ores/deploy [21:26:26] The thing I want most of all is to be able to do this stuff on my local, without everyone having to watch my dirtbag hippie, trial-and-error-ass development process… [21:26:44] awight, I do it on my local o_O [21:26:49] :) [21:26:53] Also, it looks like you are running old config [21:27:06] I think this is the official config [21:27:08] Oh! Wait... no [21:27:11] Yes. Ha! [21:27:13] at least, git log -p origin/master sez so [21:27:31] So, we recently rebuild eswikibooks damaging and goodfaith -- then deleted reverted model [21:27:35] but when I bumped editquality to get my irrelevant requirements.txt change, I pulled in repo changes [21:27:38] yes, that. [21:27:42] So either downgrade editquality or advance the config :) [21:29:17] 10Scoring-platform-team (Current), 10Wikilabels, 10User-Ladsgroup: Add "info" URL to campaign data so that we can link to campaign page - https://phabricator.wikimedia.org/T139957#3716996 (10Halfak) Merged. Waiting on PR for UI components. [21:30:15] halfak: I’ll have a lot more questions about this, but can you give me the bare outline of how you run ores-prod locally? [21:30:33] python ores_wsgi.py in one term [21:30:40] python ores_celery.py in the other term [21:30:44] I have a local redis install [21:30:53] And 8 GB of memory [21:31:00] What I’ve tried is, create a fresh VM, new virtualenv, pip install wheels, and those scripts [21:31:07] Oh! I also have a 99-local.DONOTCOMMIT.yaml [21:31:11] kk I’ll chase that dream some more [21:31:12] LOL [21:31:16] do tell [21:31:17] * halfak gets that [21:31:41] * awight pretends that’s the only reason I failed [21:32:31] https://gist.github.com/halfak/3837cea786f84f46818ec98aa8e29d3c [21:33:46] (03PS1) 10Awight: Configuration for advanced edit quality models on eswikibooks [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386965 [21:33:57] * awight brandishes https://gerrit.wikimedia.org/r/386965, still wearing my blindfold [21:34:31] I solemnly vow to spend next week getting the production stuff running locally [21:34:41] (plus do actual work :p) [21:35:23] You showed me that config before. No idea which brainhole it passed through. [21:37:17] * awight attempts to espanicize “jalfak” [21:37:23] halfak: https://gerrit.wikimedia.org/r/386965 pls [21:39:21] halfak: for each WikiProject mid-level category i'm running a query like - https://quarry.wmflabs.org/query/22547 but they're taking too long and i have to run 15-20 of those, is there a graceful way to do that rather ran generating query for each and pasting in quarry? [21:39:45] by "graceful" i mean something in a script that we can leave for a day running on ores-misc and we have all the data [21:40:43] You could dump the titles into a table [21:41:01] awight: dump in a table meaning? [21:41:32] like, load the list of tl_title into a CSV, with a column for the category name [21:41:38] then load that CSV into a table [21:41:52] & possible could write a single query to pull each of the groups of sampled articles [21:42:40] the queries could push the results into another new table, “create table select” style. [21:43:00] that would be pretty amenable to generating the queries with a script. [21:43:06] 10Scoring-platform-team, 10ORES, 10Operations, 10Traffic, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3717006 (10greg) [21:45:17] awight: or i can pull say 500 articles corresponding to each tl_title and then later group these tl_title, article pairs in higher level categories in an offline post-processing step? [21:45:52] oh cool, that’s a great idea. I can’t say whether that would be more or less expensive, though. [21:46:20] awight: you mean expensive in terms of the amount of data being fetched? or post-processing? [21:48:39] awight: above in " groups of sampled articles" you meant the group as all articles corresponding to a single tl_title or all articles corresponding to any in that bunch of tl_titles? [21:50:28] looks like i have the result for music mid-level category \o/ [21:50:38] codezee: I was thinking in terms of the query itself. [21:50:38] codezee, having a look at your query now [21:51:03] lol @ "brainhole" [21:51:18] By “groups” I meant that the table I was imagining would have rows like (‘Music’, ‘WikiProject_Music’) [21:51:36] i see [21:51:38] (03CR) 10Halfak: [C: 04-1] "You'll need to add some lines in the bottom of the config to state where the model files exist." [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386965 (owner: 10Awight) [21:52:22] (03CR) 10Halfak: [V: 032 C: 032] "Oh wait! They are already there! Weird..." [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386965 (owner: 10Awight) [21:52:58] (03CR) 10Halfak: [C: 04-1] "Woops No they aren't. Was looking at eswiki. :/" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386965 (owner: 10Awight) [21:53:37] (03CR) 10Halfak: [C: 04-1] Configuration for advanced edit quality models on eswikibooks (031 comment) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386965 (owner: 10Awight) [21:53:47] (03PS2) 10Awight: Configuration for advanced edit quality models on eswikibooks [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386965 [21:54:37] nice work on hitting the brakes [21:54:55] :D [21:55:02] codezee, this should be faster: https://quarry.wmflabs.org/query/22549 [21:55:26] indexes on titles are compound with namespace -- which means in order to take advantage of fast-scans of titles, you need to also limit the namespaces. [21:55:37] My query finished in a few seconds :) [21:55:51] * halfak <3s query optimization work [21:55:56] halfak: oh i see, thats nice then :) [21:55:58] I find it so satisfying for some reason :) [21:56:21] OK. I need to hit the road in a few minutes. Anything for me to look at right now? [21:56:21] We need a mysql version of https://github.com/nvbn/thefuck [21:56:32] halfak: PS2 of that config change [21:56:32] lol [21:56:37] halfak: but the original thing remains that how can i move this to a script, right now i have to insert 15-20 queries like this in quarry and get results [21:57:10] halfak: nevermind, i think i can experiment with paws and see [21:57:22] if it doesn't work, i'll raise this in the sync up [21:57:29] codezee: If you had the titles in a table, then you’d be able to make this a join and group by category [21:57:35] maybe do it in one query, even. [21:57:36] codezee, good point. It would be good if we could do this in quarry, but I see how that is a pain. [21:58:06] most importantly this approach doesn't fit with ours of having an automatic pipeline in Makefiles [21:58:48] Agreed. [21:58:48] where we just specify a command and bam! all the data gets fetched and processed [21:59:20] Right. We should be able to update our sample easily. [21:59:21] codezee: I see what you mean, cos you would like to programmatically create all the quarry queries. [21:59:29] awight: yes [22:00:12] sth like "for query in queries: cur.execute(query) ...." [22:00:13] Maybe have the user table with all the tl_titles, then a single quarry query to pull down into a file that we can post-process from the Makefile. [22:01:02] Oops, there’s the intermediate step of cutting the samples. [22:01:06] yes [22:02:02] anyways i'll think this through for some time, gtg for now its 3AM :P [22:02:08] ouch! [22:02:31] and then we can discuss in the sync up [22:02:36] halfak: Are we good on PS2 of https://gerrit.wikimedia.org/r/#/c/386965/ ? [22:03:09] (03CR) 10Halfak: [V: 032 C: 032] Configuration for advanced edit quality models on eswikibooks [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/386965 (owner: 10Awight) [22:03:12] ty [22:03:13] Yup. [22:03:18] Sorry. Got a bit distracted. [22:03:22] I’ll save anything else for tomorrow [22:03:24] erp [22:03:28] business-morrow [22:03:55] right on. I'll be around tomorrow AM (CDT) though [22:04:07] So feel free to ping me and I'll get to it then :) [22:04:13] OK I'm out of here. [22:04:24] codezee, awight: take care :) [22:04:24] You don’t even know how far I live from the interwebs... [22:04:46] I don't know that you're not just Barbaloot typing right now. [22:04:52] o/ [22:05:01] * awight sniffs. I miss that dog. [22:05:11] I sure deploy like Barbaloot would. [22:05:14] See ya [22:21:03] awight https://phabricator.wikimedia.org/T179212 :) [22:21:21] nice idea! [22:22:10] hashar's idea :) [22:42:43] 10Scoring-platform-team (Current), 10cloud-services-team: Reload monthly article quality dataset into wikireplica "datasets_p" - https://phabricator.wikimedia.org/T179187#3717046 (10bd808) >>! In T179187#3716545, @Halfak wrote: > Then again, maybe we can do curation at the step where we enable replication. E....