[00:02:34] <bd808>	 !log Stopped udp2log ans started udp2log-mw on deployment-bastion
[00:02:36] <qa-morebots>	 Logged the message, Master
[00:09:13] <wikibugs>	 3Quality-Assurance, VisualEditor, VisualEditor-MediaWiki: Update language_screenshot test - https://phabricator.wikimedia.org/T89370#1038201 (10Jdforrester-WMF) p:5Triage>3Normal
[00:25:34] <marxarelli>	 chrismcmahon: i'd love your thoughts on https://gerrit.wikimedia.org/r/#/c/190586/ if you have a moment
[00:25:59] <marxarelli>	 got a pretty decent acceptance testing suite going for mw-vagrant
[00:27:02] <bd808>	 VAGRANT_I_KNOW_WHAT_IM_DOING_PLEASE_BE_QUIET
[00:27:04] <bd808>	 lol
[00:27:27] <marxarelli>	 bd808: haha, i did actually lol when i found that
[00:28:08] <marxarelli>	 already uncovered a bug pre commit with it, so it seems pretty useful
[00:28:47] <marxarelli>	 i'm not totally sure the `rsync` stuff is the best way to setup an isolated test environment, but i couldn't think of anything better
[00:29:38] <bd808>	 oh that is a bit icky
[00:29:59] <bd808>	 is it jut needed so that .settings.yaml doesn't get muged?
[00:30:04] <bd808>	 *munged
[00:30:10] <marxarelli>	 and other stuff, potentially
[00:30:15] <chrismcmahon>	 marxarelli: to pick a small nit, I'd like to see the input more explicit, i.e. instead of "config foo bar" something like "config vagrant_setting value_for_vagrant_setting" 
[00:30:31] <marxarelli>	 i've already hosed my settings.yaml at least once
[00:31:14] <bd808>	 shouldn't ruby be all magic and let you money patch the stuff that writes to disk?
[00:31:20] <bd808>	 *monkey
[00:31:29] <bd808>	 typing: it's hard yo
[00:31:31] <marxarelli>	 heh. yeah, but not from outside the process
[00:31:44] <marxarelli>	 i've actually done that via fakefs for the unit tests
[00:32:06] <marxarelli>	 but these are acceptance test, so i think it's ok/desirable to test on a real fs
[00:32:50] <bd808>	 *nod*
[00:33:23] <bd808>	 depending on which mw-v copy I ran it from the rsync could eat a lot of disk for me
[00:33:24] <marxarelli>	 if only all filesystems had snapshot capabilities :)
[00:33:47] <bd808>	 I'll have to remember to only run tests in my just-for-testing directory
[00:34:13] <bd808>	 (who am I kidding, I won't run tests)
[00:34:14] <marxarelli>	 it excludes everything that git does, so it shouldn't touch anything under mediawiki/ and friends
[00:34:17] <marxarelli>	 haha
[00:35:13] <bd808>	 ah. I see the flag to use .gitignore now. smrt
[00:35:21] <chrismcmahon>	 bd808: tests are for losers
[00:35:43] <marxarelli>	 fuck it. do it live!
[00:35:43] <bd808>	 real devs just hose prod and iterate from there
[00:35:51] <chrismcmahon>	 right on
[00:37:05] <marxarelli>	 bd808: i thought about doing a `git clone --single-branch` but that wouldn't pick up changes in the working directory
[00:37:36] <marxarelli>	 makes actual tdd difficult if you can't test pre commit
[00:37:44] <bd808>	 yeah. no bueno
[00:37:54] * bd808 shrugs
[00:38:03] <bd808>	 rsync will work until it doesn't
[00:38:12] <marxarelli>	 werd
[00:39:13] <marxarelli>	 if we can get it working with zuul during gate-and-submit that's most important i think
[00:39:41] <marxarelli>	 "there is no gatekeeper, only zuul"
[00:40:03] <bd808>	 hmmm... the copy there is not needed. maybe it should only do that part of the setup if there is an existing .setting.yml file?
[00:40:52] <marxarelli>	 yeah, that would work for `vagrant config` but i'd like to test other things that might modify hieradata, etc.
[00:41:08] <bd808>	 yeah. keep it simepl. caveman style
[00:41:10] <marxarelli>	 maybe a tag for destructive tests?
[00:41:50] <^demon|busy>	 We have @group database :p
[00:42:16] <marxarelli>	 i meant a cucumber tag
[00:42:42] <marxarelli>	 is `@group database` for phpunit?
[00:42:46] <^demon|busy>	 Yeah
[00:42:51] <marxarelli>	 ah, got it
[00:43:18] <^demon|busy>	 We use it + reflection to determine if we need to do database trickery for unit tests
[00:43:21] * bd808 goes back to trying to test a logging change
[00:43:24] <marxarelli>	 hmm, maybe i should just profile the tests with and without the rsync, and see what kind of difference we're talking
[00:43:25] <^demon|busy>	 It's all very sneaky and black magic and evil
[00:44:02] <bd808>	 marxarelli: premature optimization I'm thinking. do what you're doing and fix it when its a problem
[00:44:29] * bd808 stops driving from the back of the bus
[00:44:44] <marxarelli>	 bd808: yeah, you're right
[00:45:10] <marxarelli>	 acceptance tests of any color are just going to be slow
[00:45:39] <bd808>	 wait until you want to provision and verify a VM :)
[00:46:10] <marxarelli>	 oh god... yeah
[00:46:38] <marxarelli>	 mock vboxmanage? :/
[00:46:52] <marxarelli>	 no, if it's too hard, i'm not going to do it :)
[00:47:03] <marxarelli>	 so there, Testing. take that
[00:48:07] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<37.50%)  
[00:48:30] <^demon|busy>	 greg-g: I can't say all's quiet on the western front, but the central(notice) powers seem to have stopped their ariel bombardment :)
[00:48:37] <greg-g>	 :)
[00:48:46] <greg-g>	 they did put a good ole college effort into it
[00:49:14] <^demon|busy>	 aerial even
[00:49:14] <marxarelli>	 thanks a lot. now i have "under the sea" stuck in my head
[00:49:16] <^demon|busy>	 words are hard
[00:49:42] <bd808>	  /var is full of crap on deployment-bastion again. When are you folks going to rebuild that node?
[00:49:56] * greg-g looks at ^demon|busy and twentyafterfour :)
[00:50:02] * ^demon|busy just finished spending 3 days rebuilding -elastic* ones
[00:50:23] <marxarelli>	 why is /var so tiny?
[00:50:32] <bd808>	 hystericl rasins
[00:50:38] <greg-g>	 that
[00:51:01] <^demon|busy>	 I wonder what all will break when we do :)
[00:51:01] <marxarelli>	 1.3G in log
[00:51:02] <^demon|busy>	 :p
[00:51:05] <bd808>	 Coren decided nobody should have more than 2G of logs for some unknown reason
[00:51:21] <bd808>	 "it's an Ops thing"
[00:51:21] <^demon|busy>	 Nobody will ever need more than 2G of logs.
[00:51:39] <bd808>	 when I was a boy we had 8K of RAM and we liked it
[00:52:11] <twentyafterfour>	 and you walked up hill in the snow to get to the terminal to use said 8K?
[00:52:20] <^demon|busy>	 ...both ways
[00:52:21] <marxarelli>	 looks like most are in /var/log/account
[00:52:28] <marxarelli>	 and there are two uncompressed files
[00:52:28] <bd808>	 yup
[00:52:35] <bd808>	 which is also useless in labs
[00:52:37] <marxarelli>	 maybe we can tweak the logrotate conf?
[00:52:55] <bd808>	 its a cluster of stupid proportions
[00:52:55] <^demon|busy>	 I could try rebuilding it now.
[00:53:02] <^demon|busy>	 5pm on a friday
[00:53:05] <bd808>	 he
[00:53:10] <bd808>	 3 day weekend!
[00:53:20] <bd808>	 rebuild the whole thing by tuesday
[00:53:25] <bd808>	 on jessie
[00:53:32] <^demon|busy>	 Of which I was thinking of spending at least one day not doing work :p
[00:53:44] <bd808>	 with all of ops/puppet migrated to hiera
[00:53:56] <^demon|busy>	 I had this crazy idea of getting outdoors this weekend
[00:54:08] <^demon|busy>	 Nah, I'll cancel. hiera sounds more fun
[00:54:15] <marxarelli>	 it's also beer week!
[00:54:17] <twentyafterfour>	 so if the node were rebuilt it would now automatically get a bigger /var ?
[00:54:22] <bd808>	 yeah
[00:54:24] <^demon|busy>	 Yeah
[00:54:32] <twentyafterfour>	 I can do that
[00:54:33] <^demon|busy>	 all new vms get the new partition scheme
[00:55:03] <bd808>	 It will break literally everything
[00:55:16] <^demon|busy>	 Yep
[00:55:26] <twentyafterfour>	 that's the fun part
[00:55:26] <bd808>	 But it *should* all be puppetized
[00:55:29] <marxarelli>	 !log gzip'd /var/log/account/pacct.0 on deployment-bastion
[00:55:32] <qa-morebots>	 Logged the message, Master
[00:55:49] <marxarelli>	 that should hold us through the weekend, maybe
[00:55:50] <bd808>	 shuch a nice boy. too bad he lives in Oakland
[00:56:09] <marxarelli>	 Raaaaaaaaiiiiiderzzz
[00:56:15] * bd808 throws 208 gang signs
[00:56:46] <^demon|busy>	 We should get an east coast / west coast feud going
[00:56:54] * greg-g is getting scared
[00:57:06] <marxarelli>	 whoa dude. slow your roll, please
[00:57:11] <bd808>	 I'm already down with the SF vs the world feud
[00:57:32] <marxarelli>	 that's understandable. just leave the easy bay out of it
[00:58:20] <bd808>	 Bagging on California is a semi-pro sport in Idaho. "all tham durn hippies and libtards"
[00:58:38] <twentyafterfour>	 hahah
[00:58:43] <greg-g>	 and bd808 is sometimes too good at impersonating them
[00:58:47] <marxarelli>	 i'm sure the kkk up in redding loves that rhetoric
[00:58:52] <bd808>	 We'd go pro but we don't have the population base to support the team
[00:59:11] <^demon|busy>	 Idaho: at least we have more people than Wyoming
[00:59:24] <bd808>	 Idaho: thank god for mississippi
[00:59:34] <greg-g>	 alright yokels, I'mma gonna go start this weekend and dink around in my new RV
[01:00:02] <greg-g>	 (where "new" == "built in 1983, just bought from neighbor")
[01:00:16] <^demon|busy>	 You should drive that to work one day
[01:00:21] <marxarelli>	 greg-g: nice! heading up Idaho i imagine
[01:00:23] <^demon|busy>	 park right on new montgomery
[01:00:30] <greg-g>	 ^demon|busy: next All Hands, most definitely
[01:00:53] <bd808>	 https://en.wikipedia.org/wiki/List_of_U.S._states_by_income#States_ranked_by_per_capita_income
[01:01:00] <greg-g>	 screw $75/night shared bathroom hotels, I'll bring my own bed kthx
[01:01:10] <bd808>	 idaho #48, mississippi #50
[01:01:24] <^demon|busy>	 Virginia is 5th? Wow
[01:01:51] <^demon|busy>	 Eh, NoVa skews the stats probably
[01:03:18] * greg-g goes
[01:06:25] <marxarelli>	 yeah, i'm out as well. this is prime dog parking at our unofficial neighborhood dog park
[01:06:41] <marxarelli>	 have a good weekend, everybudy
[01:07:06] <bd808>	 0/
[01:08:10] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK  
[01:10:24] <^demon|busy>	 Krenair: fwiw our RL bug is the top thing in fatalmonitor now
[01:10:31] <^demon|busy>	 (now that the CN stuff is gone)
[01:11:10] <Krenair>	 ok
[01:11:15] <Krenair>	 (where is fatalmonitor?)
[01:11:53] <^demon|busy>	 `fatalmonitor` on fluorine. It's basically just piping hhvm.log to look fancy
[01:12:57] <Krenair>	 then a couple of CA ones
[01:13:21] <^demon|busy>	 Yeah.
[01:13:26] <^demon|busy>	 The API ones for CA have a task.
[01:13:30] <^demon|busy>	 I don't think the others do
[01:13:43] <Krenair>	 it can't handle the "repeated n times:" entries :(
[01:13:59] <^demon|busy>	 yeah
[01:14:21] <bd808>	 feel free to fix my sed stuff to account for that
[01:14:29] <Krenair>	 oh, it's a simple bash script
[01:14:38] <^demon|busy>	 yeah
[01:14:39] <Krenair>	 very fancy :p
[01:14:48] <bd808>	 I rewrote fatalmonitor to deal with the hhvm.log pretty recently
[01:14:49] <^demon|busy>	 Roan wrote it on fenari a long time ago
[01:14:59] <^demon|busy>	 It survived the move to eqiad and got a little nicer at some point
[01:15:12] <Krenair>	 bd808, well ideally it'd then go and repeat the rest of the error n times so it gets counted correctly, I think
[01:15:28] <Krenair>	 but probably not that important
[01:15:59] <bd808>	 *nod*
[01:16:25] <bd808>	 it should filter out the slow db junk too
[01:16:40] <legoktm>	 ^demon|busy: the API CA one also has a patch :)
[01:16:46] <^demon|busy>	 Ah yes, it does
[01:17:00] <Krenair>	 !link
[01:17:20] <^demon|busy>	 Wrong channel, master
[01:17:27] <legoktm>	 https://phabricator.wikimedia.org/T89295 and https://gerrit.wikimedia.org/r/190361
[01:19:06] <^demon|busy>	 We also merged a patch for Cirrus today that should hopefully shut up those parse errors on searching for strings of tabs
[01:19:15] <^demon|busy>	 (which a surprising number of people search for)
[01:20:43] <^demon|away>	 I'm out
[01:22:07] <Krenair>	       6 dumped: Segmentation fault
[01:26:00] <twentyafterfour>	 that recursion detected in requestcontext is something I looked into before and honestly can't understand what's happening in that file. that class is a mess and it seems like it shouldn't need to do it's own recursion detection at all.. anyone have insight into that?
[01:26:38] <twentyafterfour>	 specifically:      37 Recursion detected in RequestContext::getLanguage in /srv/mediawiki/php-1.25wmf16/includes/context/RequestContext.php on line 306
[01:33:45] <legoktm>	 twentyafterfour: https://phabricator.wikimedia.org/T43201...it's a very very deep hole
[01:33:48] <bd808>	 is that the intentional one
[01:34:06] <bd808>	 yeah. there be dragons there
[01:36:14] <twentyafterfour>	 I assumed there were dragons involved, otherwise there's not way it could be justified to keep such a mess around
[01:36:17] <bd808>	 something wants a message which causes the user object to unpack to find out the language but before that completes (loading the user) something else makes the parser want the gender which causes the user to start unpacking ...
[01:37:24] <twentyafterfour>	 is it async?
[01:38:22] <twentyafterfour>	 is it at all sensible for the request context to call out to any code that could call the request context in lovely recursive dragon-loving fashion?
[01:38:29] <bd808>	 nope. just tangled
[01:38:46] <bd808>	 it's related to globals mostly.
[01:38:55] <twentyafterfour>	 global dragons, the best kind
[01:38:57] <bd808>	 the global user is a stub object
[01:39:16] <legoktm>	 basically things want the langauge, which depends on the User's preference, but while we're initializing that, something needs the language resulting in recursion
[01:40:10] <twentyafterfour>	 so why is the path to load the user prefs so full of dragons?
[01:40:29] <twentyafterfour>	 it doesn't make sense to me that anything else could happen in the mean-time while loading user prefs
[01:40:48] <bd808>	 https://phabricator.wikimedia.org/T56193
[01:40:52] <twentyafterfour>	 ( I'm very unfamiliar with mediawiki still, if you can't tell)
[01:41:05] <bd808>	 an error which wants a localized error message I think
[01:41:21] <Krenair>	 um
[01:42:04] <Krenair>	 if I grep hhvm.log why do I get entries out that are clearly in the wrong order?
[01:42:16] <bd808>	 twentyafterfour: see https://phabricator.wikimedia.org/T56193#554201
[01:42:35] * twentyafterfour makes mandatory batshit comment + raises priority
[01:42:38] <bd808>	 Krenair: udp
[01:43:10] <Krenair>	 20-40 minute difference
[01:44:36] <bd808>	 Krenair: The transport for those messages is hhvm -> local syslog -> udp -> fluorine. local syslog could buffer as could syslog on fluorine
[01:45:11] <Krenair>	 ah
[02:02:11] <Krenair>	 bd808, I was just tracking down https://phabricator.wikimedia.org/T75471 myself - hadn't seen the open task & patch though
[02:02:39] <Krenair>	 select * from logging where log_type = 'block' and log_action = 'block' and log_params is not null and log_params not like '%\n%'; should reveal entries with this issue
[02:04:12] <Krenair>	 e.g. https://en.wikipedia.org/w/api.php?action=query&list=logevents&letype=block&leaction=block/block&leprop=details&lestart=20070114094342 causes it
[02:10:15] <bd808>	 Sam was/is really good about filing bugs for these things but not too concered about making sure somebody fixed them
[02:15:45] <bd808>	 Krenair: is there a bug for the SpecialMediaStatistics undefined index errors yet? I can't find one
[02:15:55] <Krenair>	 I haven't seen one
[02:16:16] <bd808>	 ok. https://phabricator.wikimedia.org/T89534?
[02:16:26] * bd808 looks to see if that one is obvious
[02:34:39] <wmf-insecte>	 Project beta-scap-eqiad build #41611: FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41611/
[02:54:52] <wmf-insecte>	 Yippee, build fixed!
[02:54:53] <wmf-insecte>	 Project beta-scap-eqiad build #41613: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41613/
[03:05:10] <wmf-insecte>	 Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #447: FAILURE in 9.6 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/447/
[03:06:08] <wmf-insecte>	 Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #411: FAILURE in 8.2 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/411/
[03:11:08] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #539: FAILURE in 7.9 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/539/
[03:18:13] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #121: FAILURE in 13 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/121/
[03:19:17] <wmf-insecte>	 Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #480: FAILURE in 16 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/480/
[03:20:06] <wmf-insecte>	 Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #451: FAILURE in 5.6 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/451/
[03:21:11] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #296: FAILURE in 9.9 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/296/
[03:25:11] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #312: FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/312/
[03:27:35] <wikibugs>	 3Wikimedia-Labs-General, Release-Engineering: Increase quota for deployment-prep (beta) project - https://phabricator.wikimedia.org/T73886#1038709 (10bd808) Current settings:     Cores: 113/120     RAM: 231424/262144     Floating IPs: 7/7     Instances: 43/45     Security Groups: 8/10  #releng can decide if they...
[03:28:06] <wmf-insecte>	 Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #403: FAILURE in 6.2 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/403/
[03:30:05] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #185: FAILURE in 5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/185/
[03:31:06] <wmf-insecte>	 Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #482: FAILURE in 5.2 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/482/
[03:31:09] <wikibugs>	 3operations, Beta-Cluster: Make www-data the web-serving user (is currently apache) - https://phabricator.wikimedia.org/T78076#1038715 (10bd808)
[03:33:21] <wmf-insecte>	 Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #442: FAILURE in 21 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/442/
[03:41:01] <wikibugs>	 3Beta-Cluster: wmf-config/CommonSettings.php SpecialVersionVersionUrl hook does not check for regex match before using - https://phabricator.wikimedia.org/T75759#1038744 (10Krinkle)
[03:41:08] <wmf-insecte>	 Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #407: FAILURE in 7.8 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/407/
[03:47:08] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #120: FAILURE in 7.4 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/120/
[03:47:09] <wmf-insecte>	 Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #515: FAILURE in 8.6 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/515/
[03:48:06] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #120: FAILURE in 5.5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/120/
[03:48:10] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #455: FAILURE in 5.6 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/455/
[03:53:06] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #185: FAILURE in 5.5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/185/
[03:54:05] <wmf-insecte>	 Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #338: FAILURE in 4.8 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/338/
[03:55:07] <wmf-insecte>	 Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #466: FAILURE in 6.2 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/466/
[03:57:11] <wmf-insecte>	 Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #460: FAILURE in 6.9 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/460/
[03:58:06] <wmf-insecte>	 Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #410: FAILURE in 5.7 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/410/
[03:58:49] <Krenair>	 I wonder if https://phabricator.wikimedia.org/T62324 would be useful
[04:02:39] <wikibugs>	 3Deployment-Systems: Selecting configuration files depending on the realm of the current (bastion) server isn't always sensible - https://phabricator.wikimedia.org/T46889#1038753 (10Krenair) Is this relevant to #Wikis-in-codfw / T86888?
[05:14:24] <wikibugs>	 3Quality-Assurance, MediaWiki-extensions-GettingStarted: GettingStarted browser tests are broken - https://phabricator.wikimedia.org/T89539#1038815 (10Mattflaschen) 3NEW
[05:14:43] <wikibugs>	 3Quality-Assurance, MediaWiki-extensions-GettingStarted: GettingStarted browser tests are broken - https://phabricator.wikimedia.org/T89539#1038822 (10Mattflaschen) p:5Triage>3High
[08:54:33] <wmf-insecte>	 Project beta-scap-eqiad build #41649: FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41649/
[09:14:44] <wmf-insecte>	 Yippee, build fixed!
[09:14:44] <wmf-insecte>	 Project beta-scap-eqiad build #41651: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41651/
[13:33:49] <wikibugs>	 3Continuous-Integration, MediaWiki-Vendor, Librarization: Unable to update libraries in MediaWiki core and vendor due to version mismatch and circular dependencies - https://phabricator.wikimedia.org/T88211#1039072 (10Krinkle) >>! In T88436#1013289, @Krinkle wrote: > I propose we stop using mediawiki-vendor for...
[15:14:42] <wmf-insecte>	 Project beta-scap-eqiad build #41687: FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41687/
[15:34:55] <wmf-insecte>	 Yippee, build fixed!
[15:34:56] <wmf-insecte>	 Project beta-scap-eqiad build #41689: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41689/
[16:20:45] <wikibugs>	 3Deployment-Systems: Selecting configuration files depending on the realm of the current (bastion) server isn't always sensible - https://phabricator.wikimedia.org/T46889#1039220 (10Dzahn) This is a duplicate of T82145 now, which is the original (and now imported) RT ticket.
[18:05:27] <wmf-insecte>	 Yippee, build fixed!
[18:05:28] <wmf-insecte>	 Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #448: FIXED in 27 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/448/
[18:06:47] <wmf-insecte>	 Yippee, build fixed!
[18:06:48] <wmf-insecte>	 Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #412: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/412/
[18:18:53] <wmf-insecte>	 Yippee, build fixed!
[18:18:54] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #122: FIXED in 52 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/122/
[18:29:29] <wmf-insecte>	 Yippee, build fixed!
[18:29:30] <wmf-insecte>	 Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #452: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/452/
[18:30:02] <wmf-insecte>	 Yippee, build fixed!
[18:30:02] <wmf-insecte>	 Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #481: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/481/
[18:39:36] <wmf-insecte>	 Yippee, build fixed!
[18:39:36] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #314: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/314/
[18:48:11] <wmf-insecte>	 Yippee, build fixed!
[18:48:12] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #540: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/540/
[18:49:53] <wmf-insecte>	 Yippee, build fixed!
[18:49:54] <wmf-insecte>	 Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #404: FIXED in 59 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/404/
[18:50:51] <wmf-insecte>	 Yippee, build fixed!
[18:50:52] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #186: FIXED in 58 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/186/
[18:59:29] <wmf-insecte>	 Yippee, build fixed!
[18:59:30] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #297: FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/297/
[19:04:04] <wmf-insecte>	 Yippee, build fixed!
[19:04:05] <wmf-insecte>	 Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #483: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/483/
[19:06:41] <wmf-insecte>	 Yippee, build fixed!
[19:06:42] <wmf-insecte>	 Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #443: FIXED in 2 min 36 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/443/
[19:07:51] <wmf-insecte>	 Yippee, build fixed!
[19:07:51] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #313: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/313/
[19:08:46] <wmf-insecte>	 Yippee, build fixed!
[19:08:47] <wmf-insecte>	 Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #408: FIXED in 53 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/408/
[19:12:42] <wmf-insecte>	 Yippee, build fixed!
[19:12:43] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #507: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/507/
[19:14:19] <wmf-insecte>	 Yippee, build fixed!
[19:14:19] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #121: FIXED in 45 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/121/
[19:31:49] <wmf-insecte>	 Yippee, build fixed!
[19:31:49] <wmf-insecte>	 Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #516: FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/516/
[19:32:46] <wmf-insecte>	 Yippee, build fixed!
[19:32:47] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #121: FIXED in 56 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/121/
[19:40:23] <shinken-wm>	 PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[20:01:49] <wmf-insecte>	 Yippee, build fixed!
[20:01:50] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #186: FIXED in 37 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/186/
[20:02:36] <wmf-insecte>	 Yippee, build fixed!
[20:02:37] <wmf-insecte>	 Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #339: FIXED in 46 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/339/
[20:03:05] <wmf-insecte>	 Yippee, build fixed!
[20:03:06] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #468: FIXED in 28 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/468/
[20:05:57] <wmf-insecte>	 Yippee, build fixed!
[20:05:58] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #456: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/456/
[20:06:00] <wmf-insecte>	 Yippee, build fixed!
[20:06:00] <wmf-insecte>	 Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #467: FIXED in 2 min 54 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/467/
[20:06:17] <wmf-insecte>	 Yippee, build fixed!
[20:06:17] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #466: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/466/
[20:09:08] <wmf-insecte>	 Yippee, build fixed!
[20:09:09] <wmf-insecte>	 Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #411: FIXED in 53 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/411/
[20:19:37] <wmf-insecte>	 Yippee, build fixed!
[20:19:37] <wmf-insecte>	 Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #461: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/461/
[20:27:21] <wmf-insecte>	 Yippee, build fixed!
[20:27:21] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #312: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/312/
[20:29:49] <wmf-insecte>	 Yippee, build fixed!
[20:29:49] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #467: FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/467/
[21:34:45] <wmf-insecte>	 Project beta-scap-eqiad build #41725: FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41725/
[21:39:12] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<50.00%)  
[21:54:57] <wmf-insecte>	 Yippee, build fixed!
[21:54:57] <wmf-insecte>	 Project beta-scap-eqiad build #41727: FIXED in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41727/
[22:39:37] <wikibugs>	 3Continuous-Integration: Prevent the addition of files with names that aren't supported on Windows - https://phabricator.wikimedia.org/T67140#1039552 (10Umherirrender) Just for the record: T89569, T67139
[23:22:49] <grrrit-wm>	 (03PS1) 10Hoo man: Add "cldr" to all Wikibase/Wikidata php tests [integration/config] - 10https://gerrit.wikimedia.org/r/190690 
[23:24:04] <grrrit-wm>	 (03CR) 10Hoo man: "See also https://gerrit.wikimedia.org/r/190684" [integration/config] - 10https://gerrit.wikimedia.org/r/190690 (owner: 10Hoo man)