[01:16:25] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#2957372 (10Aklapper) Any actual advantages apart from "it looks different" which justify additional maintenance costs? [01:30:02] 10Gerrit, 07artificial-intelligence: Patch-wrangler -- suggests the best reviewers for a patch - https://phabricator.wikimedia.org/T155851#2957426 (10Aklapper) There's quite some research existing on the topic of proposing assignees for code review or bugs; I guess you're aware? :) [01:30:26] PROBLEM - Puppet run on deployment-puppetmaster02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:45:24] RECOVERY - Puppet run on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:56:25] PROBLEM - Puppet run on deployment-puppetmaster02 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [02:00:16] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:02:04] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:02:08] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:15:26] PROBLEM - Puppet run on deployment-mediawiki06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:26:25] RECOVERY - Puppet run on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:27:03] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:37:10] RECOVERY - Puppet run on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [02:40:17] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:50:24] RECOVERY - Puppet run on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [02:54:20] 10Gerrit, 07artificial-intelligence: Patch-wrangler -- suggests the best reviewers for a patch - https://phabricator.wikimedia.org/T155851#2956819 (10Legoktm) I wrote a while back to try and address this problem and I've found that it's pretty reliable/accurate for most o... [03:44:24] deployment-restbase01 is a Restbase labs (restbase) [03:44:25] deployment-restbase01 is a Cassandra server (role::cassandra) [03:44:25] The last Puppet run was at Fri Jan 20 00:06:48 UTC 2017 (1652 minutes ago). Puppet is disabled. reason not specified [03:46:38] -rw-r--r-- 1 root root 43 Jan 20 00:15 /var/lib/puppet/state/agent_disabled.lock [03:46:46] Jan 20 00:15:49 deployment-restbase01 sudo: mobrovac : TTY=unknown ; PWD=/home/mobrovac ; USER=root ; COMMAND=/usr/bin/puppet agent --disable [03:47:42] same for 02: Jan 20 00:15:59 deployment-restbase02 sudo: mobrovac : TTY=unknown ; PWD=/home/mobrovac ; USER=root ; COMMAND=/usr/bin/puppet agent --disable [04:16:42] RECOVERY - Puppet run on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:39:01] Yippee, build fixed! [06:39:01] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #245: 09FIXED in 1 hr 58 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/245/ [06:58:06] Yippee, build fixed! [06:58:06] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #245: 09FIXED in 2 hr 17 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/245/ [07:55:29] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#2957717 (10hashar) Yup Blue Ocean is definitely worth trying and might end up being in core to form Jenkins 3 (pure speculati... [11:36:23] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#2957888 (10Paladox) >>! In T155840#2957372, @Aklapper wrote: > Any actual advantages apart from "it looks different" which ju... [12:03:16] Project selenium-RelatedArticles » chrome,beta-mobile,Linux,contintLabsSlave && UbuntuTrusty build #282: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/282/ [15:39:27] 06Release-Engineering-Team, 10Phabricator, 10Wikimedia-Blog: Write a blog post on the up coming phabricator update on wmf - https://phabricator.wikimedia.org/T155896#2958113 (10Paladox) [20:39:49] (03PS1) 10Umherirrender: [ImageTweaks] Make unit test voting [integration/config] - 10https://gerrit.wikimedia.org/r/333366 [20:47:55] RECOVERY - Puppet staleness on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [3600.0] [21:08:16] RECOVERY - Puppet staleness on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [3600.0] [21:08:39] !log Puppet failures on deployment-restbase0[12] seem to be some sort of hang of the Puppet process itself. Run prints "Finished catalog run in 2n.nn seconds" but Puppet doesn't terminate for about a minute longer. The only state change logged is cassandra-metrics-collector service start. [21:08:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:10:12] bd808, https://www.youtube.com/watch?v=JwZiezIxCVU Every Utterance of "Some Kind Of..." on STAR TREK: Voyager [21:11:51] quiddity: you keep too many things in your head. it's like some kind of alien storage system [21:12:09] ;) [21:13:57] (I was in the middle of trying to set up a hexchat alert, for "some kind", when you wrote the above. It seems I still need to work out how spaces work, as it's triggering on just "some" now...) [21:14:17] so you need some kind of regex [21:14:25] to match "some kind" of pattern [21:14:30] in some kind of message [21:14:37] or hexchat does... [21:18:18] hmm, yeah, github issue says they only support single words, no fancy regex nor multiword strings. "It's meant for extra nicks. For custom highlight triggers, write a script." [21:18:23] (03PS1) 10Hashar: mw-phpunit.sh: prevents word splitting in JUNIT_DEST [integration/jenkins] - 10https://gerrit.wikimedia.org/r/333389 [21:19:40] (03CR) 10Hashar: [C: 032] mw-phpunit.sh: prevents word splitting in JUNIT_DEST [integration/jenkins] - 10https://gerrit.wikimedia.org/r/333389 (owner: 10Hashar) [21:20:15] !log integration: updating slave scripts for https://gerrit.wikimedia.org/r/#/c/333389/ [21:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:59] quiddity: yup. https://github.com/hexchat/hexchat/blob/241dd69b081779a77718763561a779cdffcaf11f/src/common/inbound.c#L282-L286 [21:21:25] (03PS2) 10Hashar: mw-phpunit.sh: prevents word splitting in JUNIT_DEST [integration/jenkins] - 10https://gerrit.wikimedia.org/r/333389 [21:21:42] (03CR) 10Hashar: [C: 032] mw-phpunit.sh: prevents word splitting in JUNIT_DEST [integration/jenkins] - 10https://gerrit.wikimedia.org/r/333389 (owner: 10Hashar) [21:22:02] so it's tokenizing the message body on space, comma, etc and then checking each token against the keyword list [21:22:37] compiled regex matches would probably be just as fast in practice [21:23:09] (03Merged) 10jenkins-bot: mw-phpunit.sh: prevents word splitting in JUNIT_DEST [integration/jenkins] - 10https://gerrit.wikimedia.org/r/333389 (owner: 10Hashar) [21:23:12] someday I will find a linux irc client that I like as much as Textual on OS X :/ [21:24:24] bd808, O.O, i feel bad for distracting you, but pleased to learn more! (Out of curiosity, how did you find your way to that chunk of code? just searching the repository for "alert" or some kind of thing?) [21:24:32] bd808: I gave up :) [21:24:39] can't beat Textual really [21:25:02] I have been using QT based "Quassel" with some success [21:25:28] I started from the irc_extra_hilight preference name. saw that mapped to hex_irc_extra_hilight in the config reader. searched for that and found the call to alert_match_text [21:25:28] maybe a gnome based irc client would be good enough. Else we should start writing our own! [21:25:55] hashar, I solved that set of problems by never trying any apple OSs, nor windows past win98. I can't miss what I don't know! [21:26:18] quiddity theres hexchat on mac [21:26:21] Mac OS X made me love computers again [21:26:35] quiddity: this is why FLOSS has such horrible UX ;) [21:26:35] though Debian Jessie is not too bad [21:26:44] lol at both. [21:26:57] the fonts on linux are a drama really [21:27:14] but i doint get any notifications because it is on a mac. You doint get the flashy icons on macs [21:27:15] and even to access my NAS it is a struggle on Debian :-/ [21:27:25] only the icon turns into a small number. [21:27:36] hashar: I've nearly fallen into the rabbit hole of writing an irc client 3 times now. I actually put in a couple of weekends on a browser based one before coming to my senses [21:27:42] so I end up working on mac most of the time, and having terminals/ssh open on the linux box :} [21:28:21] https://slack.com !!!!!!! [21:28:29] * bd808 trouts hashar [21:28:57] i plainly refuse to use slack myself [21:28:59] problem #1 is walled garden, problem #2 is one window per organization [21:29:23] mattermost is the floss alternative [21:29:26] I probably won't be able to hold out much longer. Kubernetes is all slack channels [21:29:31] I wasn't aware of the one window per org, that is annoying [21:29:33] I still miss the 3 rainbow trout, that I caught age ~11, that my dad forgot in the hotel freezer when we drove home. :< [21:30:13] they probably would have tasted bad. keep telling yourself that [21:30:37] * hashar fights with composer [21:31:02] hashar no need to have an internet connection with windows 10, you have bash and can have as many windows open that have ssh into it. though ssh is still bugy on windows. [21:33:36] hashar: if you want something more "fun" to do, we need to profile the Puppet integration that Andrew wrote for Horizon (django app) and figure out how to speed it up a lot. :) [21:34:07] paladox: well at home I don't need internet :} the mac and linux are on the same switch! ;D [21:34:15] oh [21:34:24] bd808: labs/striker is that ? [21:34:34] it's labs but not striker [21:34:42] you know what [21:34:46] horizon is the openstack management ui [21:35:00] the day the smart kids at MIT invent a system to clone our selves, I rush to them and "shut up and take my money" [21:35:03] so I can do moaaar things [21:35:06] yeah, just been a while since I profiled python/django [21:35:16] I would love to learn about django really :( [21:35:25] I found my old pc to be very slow, so i barly use it now. Though i still update windows on it. HDD shoud all be replaced with ssd's or flash storage. [21:35:34] might be worth shooting a word on labs/wikitech list for django devs? [21:35:45] akosiaris wrote Servermon which is used by ops and django based [21:35:50] else hm [21:35:56] python -m profile ? : D [21:35:57] hashar: well when you are "done" with CI let me know and I'll hire you to help with django stuff :) [21:36:10] word! [21:36:26] don't be in a rush though [21:36:40] there is no deadline [21:36:41] there is half a decade worth of work on CI / Deployment :} [21:36:44] * bd808 reads https://code.djangoproject.com/wiki/ProfilingDjango [21:37:04] * hashar figured out ones has to create a composer.local.json ... [21:42:21] I think django isn't designed for code efficiency [21:42:34] Platonides: a bold claim [21:42:58] * Platonides agrees [21:43:00] unlike PHP, at least it is not bugged by design [21:43:08] its ORM has some issues but generally I don't think the framework is flawed [21:43:22] that's just what I think [21:43:54] this particular code is just making too many expensive api calls and sending too much data back to the browser [21:44:29] I mostly want a profiler to look at it so I can actually measure improvements [21:44:39] and I'm happy to be corrected [21:44:55] it's possible that I just haven't properly understood it [21:48:56] damn [21:54:19] bd808: while you are around would labs cloud labs team is going to manage multiple k8s in addition of tools? [21:54:28] (feel free to skip responding :D ) [22:07:20] (03PS3) 10Hashar: (WIP) mediawiki-releases job [integration/config] - 10https://gerrit.wikimedia.org/r/333280 [22:07:29] hashar: there was a bit of discussion at all-hands about k8s outside of tools. No firm decisions yet though [22:08:28] I personally think that projects can and should run k8s if it makes sense, but I'm not sure that we have the staff to support yet another shared service in the Labs team [22:09:36] in theory running a k8s cluster is "easy" of course I'm sure that in practice there are lots of things that will make it tricky [22:13:22] bd808: yeah that is my understanding as well [22:14:20] one sure thing is we don't want to reinvent the wheel for each k8s cluster :} [22:14:39] the tools cluster is going to be a special snowflake [22:14:52] hopefully CI, beta, and prod can be pretty similar [22:16:02] some group could try to run a general purpose k8s cluster. I was kind of advocating that tech ops try that [22:19:06] I have yet to read k8s overview myself :( [22:19:22] hopefully we will have some kind of path/.plan by end of quarter [22:20:15] step 1) have joe and alex build you a cluster, step 2) install the jenkins k8s plugin, step 3) profit [22:20:26] ;D [22:21:35] that is probably the easiest path [22:21:44] so I don't even have to bother about the infra anymore [22:21:51] then I can do some Django profiling [22:26:09] hacking complete [22:26:16] bd808: have a good afternoon :} Going to rest [22:26:25] o/ [22:26:27] ;-) [23:08:27] 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Make deployment-prep puppetmaster more similar to Production puppetmaster - https://phabricator.wikimedia.org/T146627#2958585 (10Krenair) #2 is now done too. [23:09:26] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:19:27] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [23:21:41] 10Beta-Cluster-Infrastructure, 07Puppet: Set up puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#2958592 (10Krenair) a:03Krenair [23:28:39] 10Beta-Cluster-Infrastructure, 07Puppet: Set up puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#2958597 (10Krenair) I have a commit on -puppetmaster02 that does this, and it seems to mostly work. It seems to only include ecdsa keys, but it excludes hosts t...