[00:00:29] awight: am in the office now [00:24:20] awight: \o/ [00:24:29] awight: also have you talked to whoever is 'upstream' for scikit-learn? [00:24:39] nah [00:25:03] we should try to do that [00:25:06] The half-finished python3 packaging I built off of was a third party, not the deb maintainer nor the author [00:25:10] yeah fo real [00:25:12] ah I see [00:25:13] yeah [00:25:15] ok I'll roll that ball a bit [00:25:20] we don't want to end up maintaining these forever [00:25:27] so must foister them off on other people asap :D [00:25:35] I think we're supposed to push towards the debian maintainer, right? [00:25:39] legoktm: btw, I finished up the final issues with the mwparserfromhell package [00:25:40] :D [00:27:26] https://travis-ci.org/wikimedia/pywikibot-wikibase/builds/80347468 [00:29:10] awight: what is 'kali' [00:29:37] Never heard of it before: https://www.kali.org/ [00:30:42] YuviPanda, does that mean we can drop the "fromhell"? [00:31:24] That's pretty convenient [00:31:27] Amir1, ^ [00:31:29] halAFK: that's upto earwig and not us, I think [00:31:39] I suppose that makes sense [00:32:02] awight: ah, I see. just random people who needed it packaged. [00:32:20] awight: Imma try get it upstream into debian as well [00:34:21] halfak: do you think mediawiki-utilities can be gone within a week or so? [00:34:25] from our dependencies? [00:34:30] YuviPanda, yes [00:34:37] halfak: also yes, you should talk to earwig about the 'fromhell' prefix if you want to :) [00:34:44] especially if it becomes essential for a deploy. [00:34:59] halfak: yeah, that'd be nice, esp. if it isn't too much hard work [00:35:51] Na. [00:36:40] ok [00:36:45] halfak: me and awight just chatted a while ago. [00:37:04] next steps are: I move all our packages to gerrit and then puppetize them, slowly setup second staging server that's all debian [00:37:17] and he is going to see if he can do some more work for Extension:ORES [00:37:33] What does "move all our packages to gerrit" mean? [00:37:59] halfak: the debian packages we built are now in github under awight's or my account [00:38:06] I'm moving the packages into gerrit [00:38:12] Oh! I see. [00:38:29] eventually our 'ideal' would be to get all these extra packages into debian [00:38:37] so they can be maintained by a larger pool of people than just the two of us [00:38:43] Yeah. I can see how that makes sense. [00:38:47] so I'm also doing minor cleanups wherever necessary [00:39:28] halfak: so eventually we'll need ORES and revscoring to be in gerrit as well, for deploy. I'm personally ok with them just being mirrors of the github repo, so development continues as it does now and we just have a mirror on gerrit [00:41:31] [00:50:58] YuviPanda, that would be great. [00:52:10] halfak: others might disagree, possibly, but let's stay out of that discussion as long as we can :) [00:52:40] Sounds like a good plan to me. [00:58:44] YuviPanda: My jobs in grid takes very long time to process mostly dump reading for bad words detection [00:58:49] Can you check? [01:02:09] Amir1: not at the moment, sorry. try profiling it to see what's going on? [01:02:30] qstat -j? [01:02:36] no, python profiling [01:02:38] cprofile [01:03:30] sure [01:03:52] ok [01:15:50] halfak: so... things hitting ORES / any of labs from gadgets are going to be broken now [01:15:56] our ssl cert expired and nobody noticed >_> [04:51:54] YuviPanda, nooo [04:51:59] lol [04:52:06] How long until it's back? [04:52:13] halfak: apparently 'next business day' [04:52:24] because that's how CAs operate..... [04:52:27] which TIL [04:57:56] Is there a report I can link to? [04:58:00] YuviPanda, ^ [04:58:15] halfak: yeah [04:58:25] halfak: don't have it handy atm, see /topic in -labs [04:58:34] Got it. Thanks [04:58:49] halfak: yw [04:59:06] halfak: I also commented on https://phabricator.wikimedia.org/T107493 [05:01:31] {{merged}} [05:02:17] halfak: woo, thanks. [05:02:31] halfak: do get rid of mw-utils too at some point. I'm not going to bother updating that package [05:02:33] I'll have to learn how to update the debs for mwapi/xml/types/etc. [05:02:40] yes [05:02:45] I plan to hack on that in the AM. [05:02:51] I don't expect it'll take much. [05:02:51] since I've used the git buildpackage layout [05:02:55] it should be somewhat trivial [05:02:59] and consist mostly of git commands [05:03:06] :D [05:03:09] assuming [05:03:12] we tag releases properly [05:03:15] which we should anyway [05:03:29] halfak: jsonable, for example - some versions were tagged a v while others didn't have the v prefix [05:04:11] Yeah. Sorry about that. Was still working out the norms. [05:04:21] 'tis ok [05:04:23] Also may have made a typo to two. [05:04:31] yeah thought so :) [05:04:37] let me know if / when those change [05:04:59] https://meta.wikimedia.org/wiki/Talk:Objective_Revision_Evaluation_Service#ORES_HTTPS_down._HTTP_still_up. [05:05:23] halfak: thanks [05:05:29] halfak: remember this causes all gadgets to not work [05:05:32] since wiki is now https only [05:05:36] and can't access http resources [05:05:38] Yup. [05:05:40] ok [05:05:45] I should go home from the office [05:05:49] shouldn't I [05:06:01] Yes. [05:06:06] * halfak --> bed [05:06:16] halfak: night [05:06:21] o/ [16:11:49] halfak: HTTPS back [16:11:57] \o/ [16:11:58] * halfak tests [16:12:18] * halfak posts [16:13:00] Our talk page is becoming a wall of shame. [16:13:01] https://meta.wikimedia.org/wiki/Talk:Objective_Revision_Evaluation_Service [17:36:38] halfak: check mail please [17:37:34] halfak oh? [17:37:43] I feel uite the opposite [17:37:53] clearly people are liking what we are doing [17:38:03] we could have two existece [17:38:09] one no one cares about with perfect uptime [17:38:19] we cannot know for sure if anyone cares [17:38:31] and other where people complain about problems proving our use [17:39:08] Regretfully our uptime issues have nothing to do with our usage rate. [17:39:18] But they do magnify my embarrassment :S [17:39:19] sure [17:39:23] Amir1, I don't see an email. [17:39:29] we are an alpha release at the moment [17:39:39] such problems are to be expected [17:39:41] in your gmail [17:39:41] Na. I'd say Beta since we are advocating use. [17:39:52] okay then beta [17:49:38] But still a fair point ToAruShiroiNeko_ [17:49:54] The last few things that YuviPanda and I did made our deploy process much cleaner and the system more stable. [17:50:05] A period of instability seems like it is inevitable. [17:50:13] http://www.theonion.com/article/atlantic-ocean-excited-move-beautiful-beachfront-m-51303 [18:08:20] I just got a ping somewhere and I can't figure out where [18:09:42] Goddamn. gerrit-wm got renamed to grrrit-wm so it missed my ping filter. [18:09:46] Typo? [18:10:29] It has been that way for more than a year now [18:10:43] Since I rewrote it [18:23:43] Ahh... I guess I had blocked its pings ineffectively a couple weeks ago. [18:57:29] halfak|Lunch: it's because gerrit always makes you go "grr" [18:58:03] "It's grrreat!" [19:07:03] o/ harej [19:07:13] do you know when we'll hear back about submissions to wikiconfUSA? [19:08:44] Soon [19:08:50] Yours is probably accepted [19:08:54] I need to schedule the travel :S [19:09:20] Not sure if you'll be involved next year, but that's my main feedback. I need to know AT LEAST a month in advance if I'm traveling. [19:09:41] But I do like late deadlines -- just need to make sure that there's that month. [19:10:19] Is there an organizing committee that I can reach out to about this? [19:12:35] Email kirill.lokshin@wikimediadc.org [19:12:49] Thanks [19:15:42] YuviPanda, all of our workers are online and processing new tasks. It seems like our last change made a substantial difference. [19:15:47] We'd usually lose a worker overnight. [19:15:49] :) [19:15:53] What change [19:16:13] No workers died in the night? Mine the copper faster!! [19:16:18] socket_timeout on redis [19:16:30] Both within celery and in our direct cache [19:16:51] awight, lol. WORK 'EM HARDER [19:18:06] halfak: yeah but we had one die since no [19:18:11] Oooh [19:18:13] Yes [19:18:14] dammit [19:18:18] I totally forgot we did that [19:18:22] Aaargh [19:19:05] halfak: yay :) [19:19:21] halfak: how is cpu util on the workers. [19:20:08] What's the tool labs utility you directed me to before? [19:20:15] * halfak needs to bookmark that [19:20:33] Tools.wmflabs.org/nagf [19:22:12] Yeah. [19:23:01] We had a weird spike between 1630 and 1720 yesterday. [19:23:10] UTC I assume [19:23:20] I wonder if that was compiling something [19:23:35] Oh wait. That's today [19:23:44] So, 4 hours ago? [19:24:21] -04 hit 100% very briefly at the end of this period. [19:25:10] I wonder if someone was just hitting the server for an analysis. [19:25:21] Either way, most of this "spike" was spent at 30% CPU [19:25:34] memory is nominal [19:36:31] halfak: cool [19:36:32] Yay