[00:39:15] 10Scoring-platform-team, 10Operations, 10Wikimedia-Logstash, 10monitoring, 10Wikimedia-Incident: Send celery and wsgi service logs to logstash - https://phabricator.wikimedia.org/T181630#3798762 (10Dzahn) [09:48:56] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017, 10Patch-For-Review: Introduce and create pytest for flask application of the wikilabels AI service - https://phabricator.wikimedia.org/T179015#3799122 (10Ladsgroup) For me, it's better to have them passing, I think setting up a databas... [09:54:58] (03CR) 10Ladsgroup: [C: 032] Protect Special:Contributions code from missing threshold [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394109 (https://phabricator.wikimedia.org/T179830) (owner: 10Awight) [10:02:55] (03Merged) 10jenkins-bot: Protect Special:Contributions code from missing threshold [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394109 (https://phabricator.wikimedia.org/T179830) (owner: 10Awight) [11:11:21] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:11] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 847 bytes in 0.520 second response time [11:22:31] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:31] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 848 bytes in 6.053 second response time [11:46:51] PROBLEM - ORES worker production on ores.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:42] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 849 bytes in 2.031 second response time [13:22:13] Looks like we had a blip 2 hours ago but we recovered quickly. [13:22:15] The rate of requests we are getting jumped [14:29:39] 10Scoring-platform-team (Current), 10Wikilabels, 10Easy, 10Google-Code-in-2017, 10Patch-For-Review: Introduce and create pytest for flask application of the wikilabels AI service - https://phabricator.wikimedia.org/T179015#3799822 (10Ladsgroup) [14:29:43] 10Scoring-platform-team (Current), 10Wikilabels, 10Easy, 10Google-Code-in-2017, 10Patch-For-Review: Introduce and create pytest for flask application of the wikilabels AI service - https://phabricator.wikimedia.org/T179015#3799823 (10Phantom42) I edited travis build process a bit, so now it contains post... [14:37:21] 10Scoring-platform-team, 10Wikilabels, 10Easy: pytests for wikilabels - https://phabricator.wikimedia.org/T171082#3799833 (10Phantom42) [14:37:24] 10Scoring-platform-team (Current), 10Wikilabels, 10Easy, 10Google-Code-in-2017, 10Patch-For-Review: Introduce and create pytest for flask application of the wikilabels AI service - https://phabricator.wikimedia.org/T179015#3799831 (10Phantom42) 05Open>03Resolved Pull request is merged, so I think we... [14:40:29] o/ [14:41:21] Hey Amir1. [14:41:31] halfak: hey [14:41:34] Thanks for handling that pytest for wikilabels PR :) [14:41:53] Today is not ORES day but I'm spending some time on ORES because I worked for four hours only yesterday [14:42:04] Some patches will come in soon [14:42:05] Thanks [14:42:09] :) [14:42:34] halfak: you are welcome, coverage on wikilabels has increased 40 f*** percent [14:42:41] :D [14:42:46] Thanks to that student [14:44:28] 10Scoring-platform-team (Current), 10Wikilabels, 10Easy, 10Google-Code-in-2017, 10Patch-For-Review: Introduce and create pytest for flask application of the wikilabels AI service - https://phabricator.wikimedia.org/T179015#3799846 (10Phantom42) I will work on implementing this functionality. [14:49:44] Lol [14:58:53] OK so we've wound up in the place where we need to pay github money in order to use git-lfs at the level we want to. [14:58:55] *sigh* [14:59:09] There's far less "free" git-lfs storage in github than there is repo storage. [14:59:48] We now have strict storage and bandwidth limits. [15:00:42] "Forking and pulling a repository counts against the parent repository's bandwidth limit." [15:00:46] https://help.github.com/articles/about-storage-and-bandwidth-usage/ [15:00:57] So someone can lock our repositories by creating forks [15:01:24] "If you use more than 1 GB of bandwidth per month without purchasing a data pack, Git LFS support is disabled on your account until the next month." [15:01:38] So git-lfs is currently disabled on wiki-ai [15:01:41] and my account [15:02:31] :| [15:03:11] I'm still big fan of having two repos, one shallow and one deep [15:04:02] Amir1, managing that sounds terrible [15:04:08] I just don't see how that makes any sense [15:04:44] How could we enforce that checksums make it to the deep repo? [15:05:05] How would we match changesets when checksums are repo-dependent? [15:05:25] If we can automate the whole thing, I think it would be super easy to maintain [15:06:16] I don't see how automation is cheap or how we could make it easy for others to work with [15:06:45] git-lfs is supported code. Our automation/repo-hack would be our obligation to maintain. [15:06:59] git-lfs is becoming a standard [15:07:27] *github* just has predatory pricing for use of their git-lfs server. [15:10:00] * halfak looks seriously at gitlab [15:19:56] (03PS1) 10Ladsgroup: Wire ModelLookup service using OresServices and remove methods from Cache.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) [15:20:47] Amir1, ^ [15:21:10] (03CR) 10jerkins-bot: [V: 04-1] Wire ModelLookup service using OresServices and remove methods from Cache.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [15:22:43] Amir1, do you know about any examples of deep/shallow repos? [15:22:51] Maybe there's some automation we can re-use? [15:29:40] * halfak experiments with gitlab lfs [15:30:21] paladox, do you know if gitlab is supported for diffusion mirroring? [15:30:35] I'm not sure what would be involved in that [15:30:36] halfak: diffusion can pull from anythig [15:30:47] Oh I was hoping that was the case :) [15:31:21] halfak: I'm not sure about gitlab but I heard lots of good things about it [15:31:36] I'm so amazed that github's solution is "let's lock this guy's account and charge him $5 per fork of his repos to let him back online." [15:31:39] Amir1, me too. So far I like it a lot. [15:31:59] I'm attempting an upload of editquality with lfs to gitlab right now [15:32:06] halfak: regarding the automation, I think it should be rather straightforward [15:34:03] Halfak im not opposed to move to gitlab would we still have orgs? [15:34:21] We'd still have orgs :) [15:34:45] (03PS2) 10Ladsgroup: Wire ModelLookup service using OresServices and remove methods from Cache.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) [15:34:48] Great. [15:36:20] (03CR) 10jerkins-bot: [V: 04-1] Wire ModelLookup service using OresServices and remove methods from Cache.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [15:38:22] Arg. My internet connection is hosed because of this upload. [15:38:41] I'm guessing my ISP has decided to throttle anyone who is doing large uploads [15:39:06] I might need to give them a call and remind them how much extra I pay for a decent upload speed and how I don't torrent stuff. [15:41:45] RIP [15:42:21] (03PS3) 10Ladsgroup: Wire ModelLookup service using OresServices and remove methods from Cache.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) [15:43:52] (03CR) 10jerkins-bot: [V: 04-1] Wire ModelLookup service using OresServices and remove methods from Cache.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [15:49:03] 1.64 GB/ 5.57GB [15:49:08] Arg. I'm going to go eat breakfast [15:49:13] back in a bit [15:50:03] (03PS4) 10Ladsgroup: Wire ModelLookup service using OresServices and remove methods from Cache.php [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) [16:09:48] 3.42 / 5.57GB [16:32:26] halfak hi sorry for late reply was at college today :). diffusion mirroring is basically git push --mirror so any url is supported. though it requires you to enter the username and password so needs to be over http / https [16:32:58] halfak my host dosen't throttle :). they have a aggreement with the uk gove on this. [16:33:06] OK no problem [16:33:37] Re. throttling, I'm not really sure what is going on. But I can tell you it's driving me batty. [16:33:37] Amir1: do you know if anyone might be installing anything on ores-misc? i'm getting "E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)" [16:33:44] when i try to install git-lfs [16:34:12] halfak what connection are you on? [16:34:15] im on fttc [16:34:18] fibre [16:34:19] fiber [16:34:54] It's coax to a local office. I usually get ~100mb down and 40mb up [16:35:05] heh [16:35:16] Currently uploading at 8mb (not MB) and everything is locked up. [16:35:23] Phone tether is letting me do other stuff. [16:35:49] OMG the big upload finished. [16:35:58] oh i see "apt-get install aspell-is" is stuck for 458:39 minutes :O [16:38:39] don't know who did - "/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install nginx-full" so skeptical on killing it [16:38:47] Amir1, ^ [16:40:30] I know he was working on that [16:40:46] codezee, if you do not hear back from him in 5 mins, I think you should kill it. [16:44:01] ok [17:19:53] halfak: codezee_ it's stuck and there is no way I can unstuck it, I'm already moving to ores-staging [17:20:22] Amir1, are you saying that ores-misc-01 is hosed? [17:25:55] I was hoping you would use ores-staging to find out if aspell-is was installable at all -- not that work would move onto ores-misc-01 entirely. [17:28:20] *out of [17:38:42] BTW: https://gitlab.com/wiki-ai/editquality [17:38:45] It works! [17:54:41] Halfak did you see my task i created last night? [18:04:26] T181693 [18:04:26] T181693: [ORES] Make setup.py only work when using Python 3+ - https://phabricator.wikimedia.org/T181693 [18:14:53] 10Scoring-platform-team, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: ORES overload incident, 2017-11-28 - https://phabricator.wikimedia.org/T181538#3800604 (10akosiaris) [18:14:56] 10Scoring-platform-team (Current), 10Operations, 10monitoring: Investigate scb1001 and scb1002 available memory graphs in Grafana - https://phabricator.wikimedia.org/T181544#3800601 (10akosiaris) 05Open>03Resolved a:03akosiaris I 've delete the wrong graph and added a new one based on prometheus. This... [18:20:45] Zppix, just had a look [18:32:56] And? [19:24:03] Is Awight out today aaron? Halfak [19:24:13] I was wondering the same thing. [19:24:24] Zppix, am working on something to share with you re that task [19:24:29] Ok [19:25:23]  [19:40:00] 10Scoring-platform-team, 10ORES, 10User-Zppix: [ORES] Make setup.py only work when using Python 3+ - https://phabricator.wikimedia.org/T181693#3798845 (10Halfak) I think we should use a multifaceted strategy. There's a few different install pathways. 1. The wheel in pypi (`pip install ores`) 2. A tar.gz f... [19:41:05] Zppix, ^ there you go. [19:41:10] I did a big analysis for you :D [19:41:21] I'll get the sample package online quick so you can look at it. [19:41:44] Ok [19:44:03] https://gitlab.com/halfak/py3only [19:44:31] 10Scoring-platform-team, 10ORES, 10User-Zppix: [ORES] Make setup.py only work when using Python 3+ - https://phabricator.wikimedia.org/T181693#3800869 (10Halfak) Here's the test package I made: https://gitlab.com/halfak/py3only [19:46:03] Ok halfak fyi i have gitlab acct same info as my github [19:46:31] We'll need to discuss making the jump to gitlab. Once it's confirmed, I'll make sure it all gets configured the same :) [19:47:01] Ok [19:49:40] halfak: how do you want me to go about adding this? Just add it to existing code? [19:50:03] Zppix, yeah, I think that'll do it. I'm looking into building a better utility for supporting this. [19:50:17] I can wait if you want halfak [19:50:33] Hmm... na. Go ahead. If we want to change something later, that's OK [19:50:39] Ack [19:53:50] Im going to do a new pr the old one was to messy for me to work sanely with lol [20:05:52] Fuck [20:05:55] How do you revert [20:06:55] Halfak ^^ [20:07:19] I do it now [20:07:33] Goddamnit github [20:07:45] Ty Amir1 [20:08:29] wiki-ai/ores#853 (master - 94a4513 : Devin/Zppix): The build was broken. https://travis-ci.org/wiki-ai/ores/builds/309691345 [20:08:43] I pushed the revert on master [20:08:54] Ty [20:11:38] wiki-ai/ores#854 (master - c2543ad : Amir Sarabadani): The build was fixed. https://travis-ci.org/wiki-ai/ores/builds/309692353 [20:16:05] wiki-ai/ores#856 (T181693 - 403797b : Devin/Zppix): The build was broken. https://travis-ci.org/wiki-ai/ores/builds/309694550 [20:20:40] halfak: after my little mishap (see above) i got the new pr based upon your suggestions [20:21:24] Amir1: if your interested ^ [20:32:08] (03CR) 10Legoktm: [C: 04-1] "Why is this inventing a new service system? We should register our services with MediaWikiServices via a ServiceWiring file and use that i" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394316 (https://phabricator.wikimedia.org/T181334) (owner: 10Ladsgroup) [21:17:13] (03PS1) 10Legoktm: [WIP] Use PoolCounter to rate limit requests to the ORES service [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394407 [21:17:51] (03CR) 10Legoktm: "I haven't had the chance to test this yet." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394407 (owner: 10Legoktm) [21:18:19] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Use PoolCounter to rate limit requests to the ORES service [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394407 (owner: 10Legoktm) [21:19:11] (03PS2) 10Legoktm: [WIP] Use PoolCounter to rate limit requests to the ORES service [extensions/ORES] - 10https://gerrit.wikimedia.org/r/394407 [21:33:48] ^ this bot is missing lots of notifs [21:54:15] Zppix, added notes [21:54:30] * Zppix looks [21:55:07] I just finished fully rebuilding all of our models on stat1005 [21:55:23] Amir1, when you start work in iswiki, you might consider working from stat1005 :) [21:55:33] It works great and has way more capacity. [21:55:35] coooool [21:55:51] So how do you want me to the init.py im confused [21:55:52] Halfak [21:57:17] 10Scoring-platform-team (Current), 10Operations, 10monitoring: Investigate scb1001 and scb1002 available memory graphs in Grafana - https://phabricator.wikimedia.org/T181544#3801343 (10Halfak) Thanks @akosiaris! [22:55:38] Halfak (i think you missed this msg) 3:55 PM So how do you want me to the init.py im confused [22:55:59] "how do you want me to" [22:56:05] what does that mean? :P [22:56:34] I mean you said to put the check somewhere and do something and i dont understand what you mean [22:57:25] At the very top of the file [22:57:41] And the #noqa or whatever? [22:57:55] Will tell flake8 to shut up [22:58:15] But where you want that? [22:59:54] At the top of the file [22:59:59] Kk [23:02:15] Check out how "noqa" works. [23:02:17] Google will help [23:02:18] :) [23:15:30] Can i just put #flake8: noqa on init.py and call it a day travis is giving me hell [23:15:45] nope [23:15:49] do it good :P [23:15:57] So put it on every line :( [23:16:00] you can run flake8 locally [23:16:02] no [23:16:03] :P [23:16:12] Its complaining about the imports [23:18:01] lfs is now enabled on mediawiki/services/ores/editquality [23:18:03] halfak ^^ [23:18:26] \o/ thanks [23:18:33] github locked my account ^_^ [23:18:40] I'll me switching to gitlab. [23:18:48] It'll be interesting to see what happens from github now. [23:19:54] Why are you locked? [23:20:10] Uploaded too much to git-lfs on github [23:20:21] Enough to get locked for a month! [23:20:25] Thats bs [23:22:04] yup :) [23:22:15] Happily it's easy to migrate with git :D [23:22:21] * halfak walks nextdoor to gitlab [23:23:07] Let me know if i can help you with anything then [23:23:54] Will do. :) [23:24:02] OK time for me to head out. [23:24:06] Have a good one folks o/ [23:24:34] o/ [23:39:24] wiki-ai/ores#874 (T181693 - 87820c4 : Devin/Zppix): The build was fixed. https://travis-ci.org/wiki-ai/ores/builds/309778522 [23:39:36] Finally