[02:22:09] bd808: I don't think so. when I talked to hashar about it most recently, it was because our VMs aren't actually isolated, since they have full internet access, etc. [04:06:40] Yippee, build fixed! [04:06:40] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #116: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/116/ [04:18:05] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #116: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/116/ [04:18:29] Yippee, build fixed! [04:18:30] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #116: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/116/ [07:52:30] Yippee, build fixed! [07:52:31] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #128: 09FIXED in 6 min 31 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/128/ [08:05:10] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 588 bytes in 0.002 second response time [08:06:18] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 590 bytes in 0.003 second response time [08:20:54] 10Browser-Tests-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki, 13Patch-For-Review, 15User-zeljkofilipin: Fix font support on SauceLabs VE screenshots - https://phabricator.wikimedia.org/T141369#2571053 (10zeljkofilipin) [08:26:56] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2571093 (10hashar) [08:27:07] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: Beta-cluster web server fills up /var/log with Apache logs - https://phabricator.wikimedia.org/T75262#2571091 (10hashar) 05Open>03Resolved a:03hashar [08:30:24] (03PS22) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [08:37:23] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 13Patch-For-Review: rsync errors to beta cluster, inconsistent state after scap - https://phabricator.wikimedia.org/T71590#2571108 (10hashar) [08:37:26] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2571103 (10hashar) 05Open>03Resolved a:03hashar That was a transient issue due to labs instances having a `/var` o... [08:37:34] 10Beta-Cluster-Infrastructure: Diamond logstash monitor fills /var/log/apache2 access log - https://phabricator.wikimedia.org/T74175#749228 (10hashar) p:05Normal>03Low [08:43:23] Krenair: re T142288 let me know if/how I can help for deployment-ms* [08:44:06] godog, how easy is it to add and remove backend servers? [08:44:25] and what are they up to with all that cpu activity? [08:51:20] Krenair: adding/removing means changing the swift ring for deployment-prep in operations/software/swift-ring.git and push the new version to the beta puppet master [08:52:35] cpu activity seems mostly due to the container replicator, I'm assuming because there's many containers and it is scanning them for replication [09:02:02] I think we need a custom flavour added to allow for xlarge vcpus + at least large memory + small storage [09:02:45] did you set up the /srv/swift-storage/lv-a1 mount manually instead of using the puppet role? [09:47:25] Krenair: hello [09:47:44] Krenair: regarding new flavors, Andrew B. said it was quite trivial to add new ones. So maybe as easy as filling a task :} [09:48:12] the small storage, I am not sure it is that much needed. AFAIK the extended /dev/vdb disk is not allocated on disk until you start making use of it [09:48:21] CPUs are shared [09:48:48] the memory is fully allocated though and is the main limitation factor [09:53:12] hashar hi, im wondering if you could backport https://github.com/openstack-infra/jenkins-job-builder to https://phabricator.wikimedia.org/diffusion/CIJJ/ please? [09:53:42] paladox: hello. What do you need from upstream? :} [09:53:58] Nothing much that i could tell since they split all the files [09:54:06] so carnt tell what new things we get [09:54:38] but they added support for maven i think [09:57:27] it has already :D [09:57:36] will look at upgrading it over the week [09:57:43] ok thanks :) [09:57:48] quite busy catching up with the few thousands of mails I have received [09:57:57] lol [09:58:26] * paladox has over 9+ thousond unread emails [09:59:26] hashar gerrit was updated last week to gerrit 2.12.3, should fix the bug you filled :) [09:59:41] * paladox is now waiting for gerrit 2.12.4 to be released, then gerrit 2.13 [10:05:18] :) [10:08:33] * paladox is going to the caravan today :) [10:50:26] PROBLEM - Puppet staleness on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [12:02:44] hashar: https://integration.wikimedia.org/ci/job/debian-glue/543/console - Can you increase build timeout? [12:03:03] hashar: this packages takes about 2 hours 30 minutes to build in my local machine :) [12:03:49] kart_: hey :) I am just back from vacations and processing the few thousands emails I have received [12:04:11] I haven noticed the build timeout in some gerrit comment. No real idea as to how to implement a per repo/job timeout [12:04:21] but eventually we will find a way :D [12:04:52] hashar: cool. Enjoy 1000s of emails first. [12:04:56] yeah :D [12:07:26] 10Continuous-Integration-Config: Make debian-glue job timeout configurable - https://phabricator.wikimedia.org/T143546#2571328 (10hashar) [12:07:37] kart_: I have filled yet another task https://phabricator.wikimedia.org/T143546 :D [12:09:02] hashar: thanks. [12:49:20] * paladox went to Pizza Hut, now driving to the caravan (Lincolnshire) [12:52:52] 10Beta-Cluster-Infrastructure: Granting sysop & import rights for Jogo.obb@beta-dewiki - https://phabricator.wikimedia.org/T143548#2571392 (10Jogo.obb) [12:56:57] (03PS1) 10Phedenskog: Use WebPageTest.org key on WMF WebPageTest to support relay server [integration/config] - 10https://gerrit.wikimedia.org/r/305993 (https://phabricator.wikimedia.org/T142964) [14:17:59] 10Browser-Tests-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki, 13Patch-For-Review, 15User-zeljkofilipin: Fix font support on SauceLabs VE screenshots - https://phabricator.wikimedia.org/T141369#2571570 (10zeljkofilipin) Linux + Chrome [[ https://integration.wikimedia.org/ci/job/language-screen... [14:24:08] (03PS23) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [14:27:28] (03PS24) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [14:33:55] Yippee, build fixed! [14:33:55] Project selenium-WikiLove » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #121: 09FIXED in 1 min 54 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/121/ [14:35:44] Krenair: yeah I think lv-a1 isn't puppetized yet [14:36:12] Krenair: also I'm not very active here so I might not see messages without an highlight [14:46:30] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:48:45] hashar hi, this https://phabricator.wikimedia.org/D313 was my go at merging upstream Jenkins job builder into our repo [14:49:11] But failed since it didn't add the authors of the patches [14:49:48] paladox: I just rebase against whatever upstream commit we need [14:49:51] git remote update [14:50:01] Oh [14:50:10] something like: git checkout -b wikimedia-master wikimedia/master [14:50:11] git rebase openstack/master [14:50:19] regenerate config and compare [14:50:25] Oh [14:50:28] if all fine, I will push the new HEAD to our Gerrit @master [14:50:33] and announce it on QA list [14:50:44] Ok thanks [14:50:47] :) [14:50:47] remember me about it over the week and I will do [14:50:57] Ok [14:50:58] Lol [14:50:59] today / tomorrow are going to be busy for me though [14:51:04] Ok [14:51:31] I Tyree doing things when I was in Scotland [14:51:42] Bug in the end I lost a lot of mobile signal [14:51:49] Bug - but [15:04:10] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2571665 (10Mbrt) [15:08:14] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2571682 (10Mbrt) You can reply you say at id.wikipedia.org/wiki/Pembicaraan Pengguna:Murbaut :) [15:32:00] 10Continuous-Integration-Config: Add Python validation to operations/software repo - https://phabricator.wikimedia.org/T143559#2571747 (10Volans) [16:05:41] hi folks. could i get some help with CI config? https://phabricator.wikimedia.org/T143475 [16:05:46] that's "UploadWizard tests fail on the new ResourcesTest::testMissingMessages() due to loading messages from WikimediaMessages" [16:12:18] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T141551#2571972 (10hashar) Will handle it. Not sure when I will cut the branch though, most probably before the European SWAT window so maybe around 10am UTC (noon CEST) [16:27:07] (03PS1) 10Bartosz Dziewoński: Specify a CI dependency of UploadWizard on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/306011 (https://phabricator.wikimedia.org/T143475) [16:27:10] ok, i think i figured it out. ^ now, can someone deploy it? :D [16:27:45] hashar: can you? ^ or are you done for today already? [16:28:04] MatmaRex: we are in meeting right now [16:28:04] MatmaRex: you just need a zuul deploy? [16:28:21] (03CR) 10Hashar: [C: 031] Specify a CI dependency of UploadWizard on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/306011 (https://phabricator.wikimedia.org/T143475) (owner: 10Bartosz Dziewoński) [16:28:31] hashar: alright, no big hurry :) (i'd want it today though) [16:28:33] looks fine to me but can't babysit it right now :( [16:28:42] poke others in roughly half an hour [16:28:43] bd808: i need https://gerrit.wikimedia.org/r/306011 to go live. i don't know what needs to be done to make that happen. [16:28:54] hashar: thanks [16:30:19] MatmaRex: you need somebody to do these things -- https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Update_configuration -- I can do it in a couple of hours if nobody has had time [16:30:50] actually I can multi task it [16:30:58] (03CR) 10Hashar: [C: 032] Specify a CI dependency of UploadWizard on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/306011 (https://phabricator.wikimedia.org/T143475) (owner: 10Bartosz Dziewoński) [16:32:03] (03Merged) 10jenkins-bot: Specify a CI dependency of UploadWizard on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/306011 (https://phabricator.wikimedia.org/T143475) (owner: 10Bartosz Dziewoński) [16:32:13] woot [16:33:00] MatmaRex: done please recheck :} [16:33:58] doing [16:34:40] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Switch beta to use the proper wiki models for scoring (rather than "testwiki") - https://phabricator.wikimedia.org/T143567#2572046 (10Halfak) [16:35:30] thanks, that worked! [16:36:05] 10Continuous-Integration-Config, 06Multimedia, 10UploadWizard: UploadWizard tests fail on the new ResourcesTest::testMissingMessages() due to loading messages from WikimediaMessages - https://phabricator.wikimedia.org/T143475#2572071 (10matmarex) 05Open>03Resolved a:03matmarex Deployed by @hashar. [16:36:17] \O/ [16:46:00] 06Release-Engineering-Team, 10MediaWiki-Vagrant, 07Epic: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#2572114 (10bd808) p:05Normal>03High This is looking more important now that {T143536} has been opened by @Joe. [16:48:28] 06Release-Engineering-Team (Long-Lived-Branches), 03Scap3: Create `scap swat` command to automate patch merging & testing during a swat deployment - https://phabricator.wikimedia.org/T142880#2572135 (10mmodell) Made some progress here. Below is some ugly code that scrapes the [[ https://wikitech.wikimedia.org... [16:53:44] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Switch beta to use the proper wiki models for scoring (rather than "testwiki") - https://phabricator.wikimedia.org/T143567#2572157 (10Halfak) p:05Triage>03Normal [16:58:23] 06Release-Engineering-Team, 10MediaWiki-Vagrant, 07Epic: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#2572195 (10bd808) @ori do you have any time/interest in helping with a creating a roadmap for this? Some things we will need to do off the top of my head: * Find/cr... [17:27:34] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T140971#2572327 (10greg) 05Open>03Resolved [17:47:47] 06Release-Engineering-Team, 10MediaWiki-Vagrant, 07Epic: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#2572433 (10ori) I'm way overcommitted, especially since I had to substantially reduce my availability in the last couple of months. Sorry. I do think that ops should... [17:52:27] 06Release-Engineering-Team, 10MediaWiki-Vagrant, 07Epic: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#2572477 (10greg) >>! In T136429#2572433, @ori wrote: > I suggest @greg bring it up in the next weekly ops meeting. Not there, sadly. There are just only so many SF<... [18:19:05] (03PS1) 10Hashar: operations/software: add tox to experimental [integration/config] - 10https://gerrit.wikimedia.org/r/306031 (https://phabricator.wikimedia.org/T143559) [18:19:42] not sure where to ask about gerrit, but the new diffing interface seems to have a bug [18:19:58] https://gerrit.wikimedia.org/r/#/c/303339/22..23/oozie/mediawiki/edit_history/coordinator.properties shows changes, and https://gerrit.wikimedia.org/r/#/c/303339/22..24/oozie/mediawiki/edit_history/coordinator.properties should show at least those changes and more, but doesn;t [18:20:24] in other words, changes seem to get lost if you diff certain patchsets [18:20:51] I'm on Chrome 50.0.2661.102 (64-bit) on Ubuntu 14.04 [18:21:16] happy to file a bug if this is new and someone tells me where to file it [18:22:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, and 2 others: Config beta ORES extension to use the beta ORES service - https://phabricator.wikimedia.org/T141825#2572786 (10Ladsgroup) [18:22:50] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 06Revision-Scoring-As-A-Service, and 2 others: [Spike] Should we make a model for ores in beta? - https://phabricator.wikimedia.org/T141980#2572784 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup [18:23:09] (seems to work ok when coming in from the main patchset screen, but not switching between patchsets in the diff view) [18:27:42] milimetric: hrm that's weird. Could you file a phab ticket tagged with gerrit? there may be something known here, ostriches would know for sure. [18:27:58] sure, will do [18:28:00] Huh? [18:28:31] ostriches: https://gerrit.wikimedia.org/r/#/c/303339/22..23/oozie/mediawiki/edit_history/coordinator.properties vs https://gerrit.wikimedia.org/r/#/c/303339/22..24/oozie/mediawiki/edit_history/coordinator.properties [18:28:47] Was 22 and 24 the same patch basically? [18:29:09] Hmm doesn't look so [18:29:40] That's annoying. [18:39:21] ostriches: that's my bad [18:39:31] I missed it too, it was because that file got moved to a different directory [18:39:34] false alarm, sorry [18:40:03] it could probably say something like "file doesn't exist in this patch" on one side [18:40:11] No worries. Yeah the UI doesn't give you any indication that's the case [18:40:27] Could file a feature request upstream maybe [18:40:58] You'd think you'd just have a diff where it's all adds or removals, but yeah I see why it doesn't at least right now. [18:42:06] right, not sure it's worth bugging upstream, I'm fine with "user error" :) [18:53:43] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:18] ostriches: I think I fixed that bug that was giving you a bogus 'over quota' error when you created instances in 'staging'. Let me know if you hit it again. [18:54:22] well, 'fixed' :/ [18:55:19] Oh cool thanks. Possibly semi-related to nodepool thinking it's overquota'd at times? [19:03:33] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 44853 bytes in 1.375 second response time [19:32:17] 10Continuous-Integration-Config: Make debian-glue job timeout configurable - https://phabricator.wikimedia.org/T143546#2573024 (10hashar) The plugin defaults to 3 minutes and exposes it as `BUILD_TIMEOUT`. It does support Token Macro and we can add a build parameter that default to 3 or 30 or whatever, then over... [19:37:45] dies 'Post-merge build failed' mean I did something horrible? [19:38:00] ostriches: it was just a dumb mistake in horizon, shouldn't have affected anything outside the webui [19:39:41] Ah ok gotcha [19:45:48] 10Continuous-Integration-Config, 10Fundraising-Backlog: symfony-polyfill54 is breaking CI - https://phabricator.wikimedia.org/T143598#2573044 (10awight) [19:45:58] 10Continuous-Integration-Config, 10Fundraising-Backlog: symfony-polyfill54 is breaking CI - https://phabricator.wikimedia.org/T143598#2573056 (10awight) p:05Triage>03High [19:47:23] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Switch beta to use the proper wiki models for scoring (rather than "testwiki") - https://phabricator.wikimedia.org/T143567#2573058 (10Halfak) So, in order to do this, we'll need to re-write the API locations in the c... [20:01:33] 06Release-Engineering-Team, 10MobileFrontend: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupted. - https://phabricator.wikimedia.org/T143601#2573107 (10Jdlrobson) [20:11:45] (03PS1) 10Awight: Workaround broken php55lint [integration/config] - 10https://gerrit.wikimedia.org/r/306050 (https://phabricator.wikimedia.org/T143598) [20:22:22] hi yall, I'm going to have an extension update to deploy soon, probably during next weeks deploy train [20:22:54] i've never shepherded a mw change into prod before. Am reading on wikitech about deploy scheduling, but i'm still not exactly sure what to do. [20:23:09] If it's merged to master by Tuesday, it makes the normal train and you don't have to do anything special [20:23:16] do I just edit the schedule and add my change? [20:23:20] ah ok [20:23:21] great. [20:23:28] I think we don't pin OSM at least :) [20:23:29] Lemme check [20:23:33] so i just need to get reviewers to merge, sounds good [20:23:59] Yep, looks like a normal extension [20:24:21] So yeah, just get it reviewed and landed to master and it'll be caught up in the train automagically :) [20:24:42] And if you're worried about fallout: be around for Wednesday's deploy since that's the day it'll hit wikitechwiki [20:25:38] ok cool, so i merged the change on the extension repo, i think i need to get it into a submodule udpate somewhere, right? [20:25:43] or composer? [20:26:15] OH wait, found some docs [20:26:15] oh, merging the change on the extension gets it updated for prod? [20:26:34] y created and merged when you merge a commit to some extension's wmf/* branch, excep [20:26:34] ah [20:26:36] wmf branch... [20:26:37] ? [20:28:40] so according to schedule 1.28.0-wmf.16 is for next week [20:29:05] i thikn [20:29:45] 10Continuous-Integration-Infrastructure, 06Labs, 13Patch-For-Review, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2573190 (10hashar) Nice debugging! >>! In T143016#2559446, @thcipriani wrote: > Messages like this one: > > ``` > DEB... [20:33:13] ostriches: are these branches automatically created, or should I make a wmf/1.28.0-wmf.16 branch and push it to gerrit? [20:33:26] We create them automagically :) [20:33:32] Part of our tuesday train duties [20:34:01] (barring config otherwise, we branch from master when we create those) [20:35:46] 10Continuous-Integration-Infrastructure, 10MobileFrontend, 13Patch-For-Review: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupted. - https://phabricator.wikimedia.org/T143601#2573211 (10greg) [20:40:13] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MobileFrontend, 13Patch-For-Review: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupted. - https://phabricator.wikimedia.org/T143601#2573238 (10Jdlrob... [20:46:50] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07I18n: On Beta Cluster, MediaWiki namespace override is inconsistently applied - https://phabricator.wikimedia.org/T142863#2573272 (10greg) >>! In T142863#2558004, @greg wrote: > Is there a next step here? (even if someone isn't able to commit to... [20:48:52] 06Release-Engineering-Team: Preload TestingAcessWrapper in mwrepl - https://phabricator.wikimedia.org/T143607#2573275 (10Mattflaschen-WMF) [20:56:42] 10Beta-Cluster-Infrastructure, 15User-Luke081515: Granting sysop & import rights for Jogo.obb@beta-dewiki - https://phabricator.wikimedia.org/T143548#2573329 (10Luke081515) a:03Luke081515 [20:56:51] hm, wikibugs is a bit slow [20:59:34] 10Beta-Cluster-Infrastructure, 15User-Luke081515: Granting sysop & import rights for Jogo.obb@beta-dewiki - https://phabricator.wikimedia.org/T143548#2573369 (10Luke081515) 05Open>03Resolved Done. [21:09:15] oh ostriches ok [21:09:24] should i not have merged my change then? [21:09:28] on the extension? [21:09:34] i'd prefer if it went out next tuesday [21:09:55] i may be driving tomorrow during the train, and will be out at the end of the week [21:10:05] It'll go out Wednesday [21:10:07] Not tuesday [21:10:11] ah [21:10:13] (wikitechwiki is on tuesday's group) [21:10:13] ok, well, that's fine [21:10:17] but can we wait til next week? [21:10:21] should I revert my change? [21:19:40] ottomata: If you don't want it to go out, yeah that'd be best. [21:21:08] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MobileFrontend, 13Patch-For-Review, and 2 others: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupt... - https://phabricator.wikimedia.org/T143601#2573536 [21:22:34] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service, 15User-Ladsgroup: Switch beta to use the proper wiki models for scoring (rather than "testwiki") - https://phabricator.wikimedia.org/T143567#2573553 (10Ladsgroup) a:03Ladsgroup [21:23:05] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service, 15User-Ladsgroup: Switch beta to use the proper wiki models for scoring (rather than "testwiki") - https://phabricator.wikimedia.org/T143567#2573555 (10Ladsgroup) I'm already regretting this but I do it, it needs t... [21:27:20] oook [21:50:36] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2573684 (10Legoktm) https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/6750/console [21:54:01] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MobileFrontend, 13Patch-For-Review, and 2 others: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupt... - https://phabricator.wikimedia.org/T143601#2573107 [21:54:57] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T141551#2573730 (10Jdforrester-WMF) [21:55:36] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MobileFrontend, 13Patch-For-Review, and 2 others: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupt... - https://phabricator.wikimedia.org/T143601#2573107 [21:57:55] 10Continuous-Integration-Infrastructure, 06Labs, 13Patch-For-Review, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2573738 (10chasemp) yeah we puzzled over this for a good long while. https://graphite.wikimedia.org/render/?width=88... [22:00:43] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MobileFrontend, 13Patch-For-Review, and 2 others: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupt... - https://phabricator.wikimedia.org/T143601#2573742 [22:04:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MobileFrontend, 13Patch-For-Review, and 2 others: Jenkins complains on MobileFrontend commits with Could not read gem at /var/lib/gems/2.1.0/cache/rake-10.5.0.gem. It may be corrupt... - https://phabricator.wikimedia.org/T143601#2573765 [22:16:51] what's the deal with php55 failing with segfaults today? :/ e.g. https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/6754/console [22:18:21] i see, T142158 [22:18:33] (03CR) 10EBernhardson: "it's probably fine to not handle the special case i mentioned in previous commit, we can use annotations to suppress where it really shoul" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/301364 (owner: 10Lethexie) [22:18:40] (03PS2) 10EBernhardson: Report warnings when $dbr->query() is used instead of $dbr->select(). [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/301364 (owner: 10Lethexie) [22:22:02] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2571665 (10Krenair) Why is this wiki required in beta? [22:24:44] Project selenium-CentralAuth » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #123: 04FAILURE in 4 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/123/ [22:25:20] (03CR) 10EBernhardson: [C: 032] Report warnings when $dbr->query() is used instead of $dbr->select(). [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/301364 (owner: 10Lethexie) [22:31:41] (03Merged) 10jenkins-bot: Report warnings when $dbr->query() is used instead of $dbr->select(). [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/301364 (owner: 10Lethexie) [22:46:33] mobrovac, is the puppet disabling on -changeprop you? [22:50:26] if not, any idea who it is? [22:56:30] thcipriani: i've been looking into the problem of php5 crashing on integration, while just a workaround after some poking i'm 99% sure setting -dzend.enable_gc=0 will fix the problem. Not sure how best to test though? [22:57:13] the crash is always happening in gc_remove_from_buffer, there are a few patches related since 5.5.9 but not sure it's worth testing when we could turn gc off for this use case [22:57:38] (refcounted items will still be collected, this gc effects cycle collection afaik) [22:59:13] i'd run the tests directly, but for some reason i can't seem to deduce the appropriate mysql credentials even though the files that set everything up seem pretty explicit (perhaps PEBKAC ;P) [23:05:23] ebernhardson: hrm, that change would be a modification to a couple of things in here: https://github.com/wikimedia/integration-jenkins/tree/master/bin [23:05:53] for https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55 where it's failing a lot we'd need to change https://github.com/wikimedia/integration-jenkins/blob/master/bin/mw-run-phpunit-allexts.sh [23:07:04] hmm, ok i can handle that. Thanks! [23:08:12] thank you for looking into this! [23:10:00] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2525383 (10EBernhardson) dmesg on integration-slave-trusty-1001 reports we are consistently segfaulting at the... [23:10:17] [23:16:36] (03PS1) 10EBernhardson: Try disable garbage collection to prevent segfaults [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) [23:21:14] (03CR) 10Aaron Schulz: [C: 031] Try disable garbage collection to prevent segfaults [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) (owner: 10EBernhardson) [23:24:01] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 13Patch-For-Review, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2574011 (10EBernhardson) should note that disabling gc is just a workaround, ``` 16:19... [23:29:16] (03PS2) 10EBernhardson: Try disable garbage collection to prevent segfaults [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) [23:34:45] (03CR) 10Paladox: [C: 031] Try disable garbage collection to prevent segfaults [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) (owner: 10EBernhardson) [23:35:23] ebernhardson: is passing that flag to hhvm ok? [23:36:13] (03CR) 10Legoktm: "Is it okay to pass this parameter to HHVM?" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) (owner: 10EBernhardson) [23:36:36] legoktm: hmm, hhvm doesn't seem to complain about it, but also doesn't look to respect it [23:37:01] that's probably ideal for us? :P [23:37:44] (03CR) 10Legoktm: [C: 032] Try disable garbage collection to prevent segfaults [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) (owner: 10EBernhardson) [23:38:13] (03CR) 10EBernhardson: "hhvm accepts -d to do the same thing as php5 does. hhvm doesn't ever setup to read the zend.enable_gc flag though." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) (owner: 10EBernhardson) [23:38:57] * paladox watches movies (00:38am) [23:40:04] (03Merged) 10jenkins-bot: Try disable garbage collection to prevent segfaults [integration/jenkins] - 10https://gerrit.wikimedia.org/r/306072 (https://phabricator.wikimedia.org/T142158) (owner: 10EBernhardson) [23:40:35] !log updating slave_scripts on all slaves [23:40:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:41:04] [integration-saltmaster.integration.eqiad.wmflabs] out: integration-slave-trusty-1014.integration.eqiad.wmflabs: [23:41:04] [integration-saltmaster.integration.eqiad.wmflabs] out: Minion did not return. [No response] [23:41:28] ebernhar1son: should be deployed everywhere now [23:41:29] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:42:04] we only run that on +2, rather than a standard check so...i guess merge something ;) [23:42:45] https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/6764/console [23:49:59] well, at least one thing merged succesfully