[07:25:30] 10Deployment-Systems, 06Labs, 10wikitech.wikimedia.org: /etc/mediawiki/WikitechPrivateSettings.php not found on tin - https://phabricator.wikimedia.org/T139917#2446552 (10greg) [07:27:34] 10Deployment-Systems: l10nupdate syncs not working after rebuilding tin due to ssh-agent errors - https://phabricator.wikimedia.org/T125992#2446570 (10greg) 05Open>03Resolved a:03ori (resurrecting an old task...) I just checked `l10nupdate.log-20160711.gz` and there weren't any permission denied errors li... [09:18:52] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2446707 (10Paladox) It seems to be a gerrit bug since according to https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2... [09:33:11] hashar hi, im wondering if you can help me with gerrit please. I try to restart gerrit to load a css file but it wont restart it fails then puppet runs when ever it runs and gerrit starts but dosent load the css file or any changes i make. [09:33:26] Im not sure why it fails to restart, i carn't find anything in the error logs [09:33:45] zeljkof: !log would end up sending a message on https://tools.wmflabs.org/sal/releng :) [09:34:25] !log Enabled ZMQ Event Publisher on all Jobs in Jenkins [09:34:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:37:18] hashar chromium was updated in trusty to 51. [09:38:44] (Meaning we can now update the trusty slaves chromium) [09:39:05] 10Beta-Cluster-Infrastructure: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2446741 (10fgiunchedi) I've briefly looked at it, and it seems now the commons container is accessed in its non-sharded fashion? from `/var/log/swift/proxy-access.log`: ``` Jul 11... [09:51:36] hashar https://phabricator.wikimedia.org/T137525#2446707 [09:55:29] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling: Drop zeromq property from all jobs since it is now the default - https://phabricator.wikimedia.org/T139923#2446754 (10hashar) [10:04:27] 10Continuous-Integration-Infrastructure: Update chromium to 51 on trusty - https://phabricator.wikimedia.org/T139891#2446769 (10hashar) [10:04:30] 10Continuous-Integration-Infrastructure, 10Wikidata, 05MW-1.28-release-notes, 13Patch-For-Review, 07WorkType-Maintenance: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188#2446770 (10hashar) [10:06:48] 10Continuous-Integration-Infrastructure: Update chromium to 51 on trusty - https://phabricator.wikimedia.org/T139891#2446771 (10Paladox) Thankyou @hashar I coulden find that task. [10:06:52] zeljkof: and I am going to switch to Chromium 51 [10:07:30] :) [10:08:37] 10Continuous-Integration-Infrastructure, 10Wikidata, 05MW-1.28-release-notes, 13Patch-For-Review, 07WorkType-Maintenance: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188#2446773 (10hashar) Chromium `51.0.2704.79-0ubuntu0.14.04.1.1121` has reached trusty-up... [10:09:17] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate qunit jobs to Nodepool - https://phabricator.wikimedia.org/T136301#2446776 (10Paladox) [10:09:21] 10Continuous-Integration-Infrastructure, 10Wikidata, 05MW-1.28-release-notes, 13Patch-For-Review, 07WorkType-Maintenance: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188#2446775 (10Paladox) 05stalled>03Open [10:09:54] !log Unpinning Chromium v49 from the Trusty slaves and upgrading to v51 for T136188 [10:09:55] T136188: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188 [10:09:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:13:25] !log T136188: salt -v '*slave-trusty*' cmd.run 'rm /etc/apt/preferences.d/chromium-*' [10:13:26] T136188: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188 [10:13:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:15:58] !log T136188: on Trusty slaves, upgrading Chromium from v49 to v51: salt -v '*slave-trusty-*' cmd.run 'apt-get -y install chromium-browser chromium-chromedriver chromium-codecs-ffmpeg-extra' [10:15:59] T136188: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188 [10:16:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:28:20] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate qunit jobs to Nodepool - https://phabricator.wikimedia.org/T136301#2446803 (10hashar) [10:28:25] 10Continuous-Integration-Infrastructure, 10Wikidata, 05MW-1.28-release-notes, 13Patch-For-Review, and 2 others: qunit jobs have karma stall when chromium disconnect - https://phabricator.wikimedia.org/T136188#2446800 (10hashar) 05Open>03Resolved That was some random bad interaction between Karma / Quni... [10:28:51] heading out to lunch [11:48:48] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 15User-zeljkofilipin: Drop zeromq property from all jobs since it is now the default - https://phabricator.wikimedia.org/T139923#2446984 (10zeljkofilipin) a:03zeljkofilipin [12:16:17] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 15User-zeljkofilipin: Drop zeromq property from all jobs since it is now the default - https://phabricator.wikimedia.org/T139923#2447038 (10zeljkofilipin) p:05Triage>03Normal [13:06:43] (03PS1) 10Hashar: Migrate /cover/ under doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/298275 (https://phabricator.wikimedia.org/T139620) [13:09:32] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: puppet failed on deployment-logstash2.deployment-prep - https://phabricator.wikimedia.org/T139368#2447140 (10Gehel) [13:12:22] (03PS1) 10Hashar: Migrate coverage reports to doc.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/298276 (https://phabricator.wikimedia.org/T139620) [13:12:55] hashar hi [13:15:56] paladox: I am busy :D [13:16:03] Oh ok [13:16:41] thcipriani|afk: hello! Could use some review / feedback on moving the coverage reports from integration.wm.o to doc.wm.o ( https://phabricator.wikimedia.org/T139620 ) [13:16:51] thcipriani|afk: looks straightforward to me and the easiest path [13:16:52] Im wondering if there is anything to do in https://phabricator.wikimedia.org/T139900 task [13:22:04] noone can tell as everybody has to click that link to know what it's about. which costs time. [13:23:04] Oh sorry, it is about do we have to do anything to zuul / jenkins / integration to make it work with the new lead server (Gerrit new host) Which i think there aiming to do this week [13:24:17] If you don't know if there is anytthing to do, why did you create a task? [13:25:03] Well if gerrit is moved to a new host and we doint know if ci needs any changes then ci will suffer a ci outrage until we fix anything to make it work with the new gerrit host [13:25:28] Yes. And if you don't know if there is anything to do, why did you create a task? [13:26:13] Well so that we can be sure there are no breakages. [13:28:10] paladox, do I understand it correctly that you created a task about "updating zuul and jenkins config" but you have no idea if updating zuul and jenkins config is actually needed? [13:28:29] Yes, i want to find out if they need updating [13:28:44] paladox: Why haven't you found out first and THEN create a task? [13:28:57] Creating tasks about non-existing problems isn't welcome. [13:28:58] Not sure, it was the weekend [13:29:13] We don't track imaginary problems in Phabricator. [13:29:23] Ok [13:29:35] If you're not sure, your summary should at least mention "investigate whether..." or something. [13:29:48] But I really don't want to track non-existing problems. Which sound like existing problems. [13:30:11] Oh ok. [13:30:30] Don't make tasks sound as if they describe problems that you have faced, If you have not faced those problems. [13:31:05] 10Continuous-Integration-Infrastructure, 10Zuul, 07Jenkins: investigate whether zuul, jenkins and integration need updating to work with lead (New gerrit server) - https://phabricator.wikimedia.org/T139900#2447184 (10Paladox) [13:31:39] 10Continuous-Integration-Infrastructure, 10Zuul, 07Jenkins: investigate whether zuul, jenkins and integration need updating to work with lead (New gerrit server) - https://phabricator.wikimedia.org/T139900#2445665 (10Paladox) [13:31:41] Ok [14:11:28] (03PS1) 10Zfilipin: Drop ZeroMQ explicit configuration from all Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/298283 (https://phabricator.wikimedia.org/T139923) [14:17:11] zeljkof: lovely [14:18:00] hashar: tried to do it quickly, git grep says I have found all [14:24:06] zeljkof: looks all excellent . I am refreshing the jobs [14:24:30] !log Removing ZeroMQ config from the Jenkins jobs. It is now enabled globally. T139923 [14:24:31] T139923: Drop zeromq property from all jobs since it is now the default - https://phabricator.wikimedia.org/T139923 [14:24:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:24:44] (03CR) 10Hashar: [C: 032] "Excellent. I have refreshed the 55 impacted jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/298283 (https://phabricator.wikimedia.org/T139923) (owner: 10Zfilipin) [14:25:15] hashar: oh man, I am _slowly_ getting this CI thing :D [14:25:19] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 15User-zeljkofilipin: Drop zeromq property from all jobs since it is now the default - https://phabricator.wikimedia.org/T139923#2447340 (10hashar) 05Open>03Resolved Has been confirmed to work properly :) One less m... [14:25:26] zeljkof: as you said this morning. That is one less mistake to do :) [14:25:33] so yeah in short [14:25:47] Nodepool listen for Jenkins notification that builds have complete [14:26:01] and when a build has completed on a job that Nodepool manages it immediately schedule it to be deleted [14:26:11] eg removed from Jenkins as a slave, deleted in OpenStack [14:26:56] I am sure somebody would repeat my mistake, I am afraid sooner than later [14:26:56] dropping the zeromq parameter from JJB config is definitely a good suggestion! [14:27:07] I am sure I myself made the same mistake :( [14:27:16] so one less way of screwing up things! [14:28:00] (03Merged) 10jenkins-bot: Drop ZeroMQ explicit configuration from all Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/298283 (https://phabricator.wikimedia.org/T139923) (owner: 10Zfilipin) [14:28:17] yes, if we are serious about nodepool, making it more robust is important [14:34:15] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2447386 (10zeljkofilipin) A quick look at existing browsertests-Wikidata* jobs (12 of them): - browsertests-Wikidata-SmokeTests-linu... [15:22:01] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2447654 (10zeljkofilipin) A quick look at: - selenium-Wikibase - broken for 30 days - `no such file to load -- features/support/... [15:36:02] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2447704 (10zeljkofilipin) Can I suggest to #wikidata people (cc @Tobi_WMDE_SW) that we delete browsertests-Wikidata* jobs, fix seleni... [15:40:44] 07Browser-Tests, 10MediaWiki-General-or-Unknown, 07JavaScript, 07Ruby, 15User-zeljkofilipin: Port mediawiki/core selenium tests from ruby to javascript - https://phabricator.wikimedia.org/T139740#2447724 (10zeljkofilipin) a:05zeljkofilipin>03None [15:41:11] 10Browser-Tests-Infrastructure, 07JavaScript, 10Malu (Malu-Prototype), 15User-zeljkofilipin: Release malu 0.0.3 - https://phabricator.wikimedia.org/T139742#2447728 (10zeljkofilipin) a:05zeljkofilipin>03None [15:50:30] Project selenium-MobileFrontend ยป firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #76: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/76/ [16:01:50] Krenair: is the dirty hieradata/labs.yaml on deployment-puppetmaster yours? Leaving the git repo dirty breaks synchronization with the upstream ops/puppet.git [16:02:01] yes [16:02:21] can we turn that into a cherry-picked commit? [16:02:47] I am waiting for `git pull origin production` on my laptop to show some forms of life before uploading it as a proper commit to gerrit [16:02:58] some signs* [16:03:49] made a temp commit for it on -puppetmaster [16:04:05] awesome. thanks [16:04:26] I guess I didn't look to see if you were actively working on that or not :/ [16:04:43] I have a cherry-pick I'd like to apply for logstash [16:22:54] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2447904 (10Paladox) [16:25:35] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#1049136 (10bd808) What do we need to do to find out if there is consensus to turn this back off? I really can't imagine why we need to have patch sets like rOPUPb734a6a50c09daaa81663ce1a008b7636d9... [16:31:59] Someone just rang me from y'all's weekly meeting but I'm in a different weekly meeting already :) [16:32:19] andrewbogott: maybe a mistake, we're good :) [16:32:25] 'k [16:37:15] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2447967 (10mmodell) @bd808: I agree about spamming tasks. The primary motivation for importing those changes into phabricator is simply that gerrit had links to those changes in gitblit. Since we... [18:17:13] (03CR) 10Jforrester: "Yay." [integration/config] - 10https://gerrit.wikimedia.org/r/298276 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [18:28:28] hashar hi, with https://phabricator.wikimedia.org/T70271#2447660 i tested gerrit 2.12.3 and zuul and jenkins together and they work [18:54:53] 10Deployment-Systems, 06Labs, 10wikitech.wikimedia.org: /etc/mediawiki/WikitechPrivateSettings.php not found on tin - https://phabricator.wikimedia.org/T139917#2446552 (10Krenair) It's not supposed to exist there. I suppose we could make a dummy version of the file to put there... [19:01:06] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Documentation, 07Tracking: Repositories with Ruby code should be documented and appropriate Jenkins jobs should be running - https://phabricator.wikimedia.org/T1361#2449254 (10Danny_B) [19:03:33] 06Release-Engineering-Team, 15User-greg: Create FY1516Q4 RelEng Quarterly Review slide deck (due 7/11) - https://phabricator.wikimedia.org/T137728#2449283 (10greg) [19:10:58] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-ZeroBanner, 06Zero: [Blocking] Chromium/Firefox do not provide ability to set custom headers via Selenium and PhantomJS does not clear session (LocalStorage/Cookies/Cache) properly - https://phabricator.wikimedia.org/T96451#2449381 (10Danny_B) [19:37:27] paladox: hey :) [19:37:40] Hi [19:37:46] :) [19:37:53] Ive been testing zuul [19:37:58] paladox: though I am not sure which version of Zuul and combination of python module you ended with [19:38:06] Oh me either [19:38:08] + it is on Jessie not on Precise like prod :( [19:38:23] Oh, could you build one for jessie please [19:38:36] so i can test sinde you are an expert in dpkg. [19:39:30] I am nowhere near an expert [19:39:33] Oh [19:39:42] and the Jessie package ends up being different from the Precise one :( [19:39:45] different dependencies [19:39:53] Oh yep [19:40:05] which versions have you tried ? [19:40:15] Let me check [19:40:29] this one [19:40:30] 2.1.0-151-g30a433b-wmf1jessie1 [19:40:32] hashar ^^ [19:40:41] i git merged precise branch into jessie [19:40:47] and resolved the conflicts [19:43:02] hashar it dosent seem integration/config or integration/jenkins repo [19:43:05] need changing [19:43:09] works perfect for me [19:45:36] paladox: ostriches: I will probably sneakily upgrade Zuul in prod tomorrow [19:45:48] since Paladox test infra is running more or less the same code [19:45:53] hashar oh :) [19:46:11] hashar i belive we may have hit the gerrit 2.8 bug [19:46:11] for ssh [19:46:13] then I again I have no idea what is triggering the issue :( [19:46:27] ah yeah you linked to a Gerrit 2.9 bug [19:46:30] related to ssh right? [19:46:33] Yep [19:46:44] Seems to be related to stream-event which zuul uses [19:47:06] it was actually fixed in 2.9 but reverted for another reason then added back in 2.10.3 [19:48:14] hashar you can do recheck in http://gerrit-test.wmflabs.org/gerrit/#/q/status:open since none of the users have passwords [19:48:19] as it is a test [19:48:28] but you can create changes straight from browser [19:49:28] hashar would you be able to merge https://gerrit.wikimedia.org/r/#/c/293269/ please [19:49:30] :) [19:52:43] paladox: no [19:53:02] Oh [19:53:09] paladox: have to review that first and see whaqt it causes :]) [19:53:19] Oh ok [19:54:11] that commit is kept open until confirmed to be working properly [19:54:16] specially Zuul on Jessie [19:54:22] I meant [19:54:30] specially the zuul-server daemon on Jessie [19:54:32] Oh [19:54:46] Oh zuul seems to work good with gerrit 2.12 [19:54:48] jessie [19:55:28] hashar https://github.com/Microsoft/BashOnWindows/issues/637 [19:55:43] I got ubuntu working straight from windows 10. [19:55:52] :) [19:56:12] neat [19:56:22] Yep [19:56:34] I even can view my windows file sysntem from there [19:58:12] hashar should we update https://phabricator.wikimedia.org/T70271#2449105 those links (Gerrit 2.12) task [19:58:33] please [20:02:31] paladox: ostriches and i pretty much know all what is needed to be updated [20:02:36] and it is almost all in puppet [20:02:41] Oh [20:02:57] ;-] [20:02:57] I opened a task about investigating what needs updating [20:03:14] https://phabricator.wikimedia.org/T139900 [20:03:15] the jenkins-bot user for example does not need to be created [20:03:16] hashar ^^ [20:03:19] Oh [20:03:32] it is the ssh username Zuul uses to connect to Gerrit 27418 [20:03:36] oh [20:03:41] which ends up looking it up in the labs LDAP [20:03:46] so it is just going to work ! [20:03:51] oh :) [20:03:59] you also can use gpg keys [20:04:01] in gerrit [20:04:08] https://phabricator.wikimedia.org/T139900 that one can be abandoned yeah [20:04:10] and you can also sign the commits too :) [20:04:14] Ok [20:04:17] it is implied by the Gerrit upgrade task [20:04:36] 10Continuous-Integration-Infrastructure, 10Zuul, 07Jenkins: investigate whether zuul, jenkins and integration need updating to work with lead (New gerrit server) - https://phabricator.wikimedia.org/T139900#2449722 (10Paladox) 05Open>03declined https://phabricator.wikimedia.org/T139900 that one... [20:04:38] ostriches and I quickly spoke about it during the releng checkin afew hours ago [20:04:44] Oh [20:07:14] But im not sure when the update to gerrit will happen [20:07:16] hashar ^^ [20:36:26] Project beta-scap-eqiad build #110775: 04FAILURE in 0.26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/110775/ [20:37:19] 20:36:26 20:36:26 scap failed: LockFailedError Failed to lock /var/lock/scap: [Errno 11] Resource temporarily unavailable (duration: 00m 00s) [20:41:43] probably me [20:41:45] hmm, likely transient, file is no longer there. [20:41:53] I've been manually syncing things [20:42:02] still debugging swift thumbs [20:43:14] here's the thing [20:43:17] http://upload.beta.wmflabs.org/wikipedia/commons/thumb/c/c1/Narayam_has_a_strange_understanding_of_off.png/200px-Narayam_has_a_strange_understanding_of_off.png [20:43:32] Change the 200 to something else and you get File missing [20:44:09] But then change http://upload.beta.wmflabs.org/wikipedia/commons/thumb to http://commons.wikimedia.beta.wmflabs.org/w/thumb_handler.php and you get the image, go back and it's now available on upload.beta.wmflabs.org [20:44:38] wmf/rewrite.py must be doing something else I don't understand yet [20:44:43] (03PS1) 10Paladox: [XAnalytics] Update jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/298346 [20:44:58] hashar hi could you review https://gerrit.wikimedia.org/r/298346 please [20:52:45] paladox: sure :) [20:52:48] Yippee, build fixed! [20:52:48] Project beta-scap-eqiad build #110776: 09FIXED in 8 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/110776/ [20:52:59] hashar thanks :) [20:53:10] paladox: well the CI part yeah, the extension.json change ... really I have no clue how it works :) [20:53:17] well I have clue, but not enough to properly review +2 [20:53:32] Oh yep i mean ci [20:53:39] i linked there to show it worked [20:54:52] paladox: and mostly likely that extension is no more used [20:55:13] hashar yep probaly [20:55:49] I declare victory [20:56:46] wahoo [20:56:54] (what was the missing thing it did?) [20:57:18] Hurrah Krenair !!!!!!!!!!!!!!!!!!!! [20:58:19] I think I have both normal image downloads and thumbnailing working via swift in the beta cluster greg-g [20:58:35] that's awesome, Krenair [20:59:24] paladox: so yeah XAnalytics, I am not confortable with CR+2 the extension.json change :( [20:59:37] Ok, yep. [20:59:49] http://upload.beta.wmflabs.org/wikipedia/commons/thumb/d/d2/Empire_State_Building_Top.jpg/190px-Empire_State_Building_Top.jpg [20:59:52] change the 190 to whatever [20:59:54] it'll still work [21:00:12] w00t [21:01:43] still got a couple of minor things to clean up but it appears to be doing its main job [21:04:15] Hmm... https://gerrit.wikimedia.org/r/#/c/298054/ [21:04:22] Wonder if I should just merge that, and see what beta does [21:05:02] should effectively be a noop [21:10:14] Krenair: please make sure to announce it somewhere so we can reply and celebrate :] [21:10:24] PROBLEM - Host deployment-upload is DOWN: CRITICAL - Host Unreachable (10.68.16.189) [21:10:25] Krenair: thank you really! that has been a pain/tech-debt for years [21:10:40] and that unpuppetized deployment-upload instance as well [21:12:00] 06Release-Engineering-Team, 10Gerrit, 06Operations, 13Patch-For-Review: reinstall/upgrade gerrit server (ytterbium) from precise to jessie - https://phabricator.wikimedia.org/T125018#2450143 (10Dzahn) [21:12:08] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2450144 (10AlexMonk-WMF) ```krenair@deployment-upload:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 12.04.5 LTS Re... [21:12:30] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Packaging, 13Patch-For-Review: Create a Debian package for https://github.com/mozilla/geckodriver for at least Debian Jessie - https://phabricator.wikimedia.org/T137797#2450146 (10greg) [21:12:49] hashar, it's also a precise instance [21:12:55] yeah [21:13:05] and I think our last deployment-prep instance created while in pmtpa [21:13:23] triple word score! [21:13:54] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2450149 (10yuvipanda) \o/ <3 [21:14:46] 06Release-Engineering-Team, 10Gerrit, 06Operations, 13Patch-For-Review: replace gerrit server (ytterbium) with jessie server (lead) - https://phabricator.wikimedia.org/T125018#2450154 (10Dzahn) [21:22:07] grr [21:22:14] someone removed my commit from deployment-tin again [21:24:03] hashar https://gerrit.wikimedia.org/r/#/c/298377/ :) [21:25:16] Krenair: maybe that is the jenkins jobs ? [21:26:56] I am heading to bed anyway, have fun ! [21:45:02] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2450277 (10hashar) Congratulations @fgiunchedi and @krenair ! [21:47:01] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-File-management, 06Multimedia, 13Patch-For-Review: Images with extra parameters (e.g. low-quality JPEG) are not rendered in beta - https://phabricator.wikimedia.org/T71757#2450285 (10hashar) 05Open>03Resolved a:03ori http://upload.beta.wmflabs.... [21:50:41] 10Deployment-Systems, 06Release-Engineering-Team, 03releng-201617-q1, 07Epic: Merge to deployed branches instead of cutting a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#2450315 (10mmodell) p:05Normal>03High [21:51:01] 06Release-Engineering-Team (Deployment-Blockers), 05Release, 05WMF-deploy-2016-07-05_(1.28.0-wmf.9): MW-1.28.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T138555#2450316 (10mmodell) 05Open>03Resolved [21:55:57] 06Release-Engineering-Team, 15User-greg: Create FY1516Q4 RelEng Quarterly Review slide deck (due 7/11) - https://phabricator.wikimedia.org/T137728#2450331 (10greg) 05Open>03Resolved [21:56:49] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2450333 (10AlexMonk-WMF) My NFS import ended up working a bit like this: ```lang=bash subdirs=`echo "0 1 2 3 4 5 6 7 8 9 a b c d e f archive/0 archive/1 archiv... [21:58:09] 06Release-Engineering-Team, 07Tracking, 15User-greg: Redo some #RelEng -related project workboard columns (tracking) - https://phabricator.wikimedia.org/T138884#2450336 (10mmodell) [21:59:10] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2450341 (10AlexMonk-WMF) @fgiunchedi: Okay, so I think I need your help to finish this off completely. * Where are the temp URL and cirrus keys? * Can I put th... [22:06:12] hm [22:06:23] so viewing files works fine [22:06:36] there's a lock server issue with uploading [22:06:43] Could not acquire locks on server memc04. [22:06:43] Could not acquire locks on server memc05. [22:17:55] 06Release-Engineering-Team, 10Gerrit, 06Operations, 13Patch-For-Review: replace gerrit server (ytterbium) with jessie server (lead) - https://phabricator.wikimedia.org/T125018#2450423 (10Dzahn) on gallium (CI server) there are now rules to allow connections from lead [22:28:53] (03PS1) 10Legoktm: Save failure logs in php-compile-php55 too [integration/config] - 10https://gerrit.wikimedia.org/r/298390 [22:32:18] (03CR) 10Legoktm: [C: 032] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/298390 (owner: 10Legoktm) [22:33:12] (03Merged) 10jenkins-bot: Save failure logs in php-compile-php55 too [integration/config] - 10https://gerrit.wikimedia.org/r/298390 (owner: 10Legoktm) [23:32:43] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-File-management, 06Multimedia, 07Tracking: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#934455 (10AlexMonk-WMF) Once I'm done with T64835, I believe t... [23:35:09] 10Beta-Cluster-Infrastructure: beta cluster missing vips command needed to render tiffs and pngs, pnmtojpeg for DjVus - https://phabricator.wikimedia.org/T116816#2450798 (10AlexMonk-WMF) Yeah I imagine we'll get a proper imagescaler set up in T84950 which will fix this properly. Right now we use any MW appserver... [23:37:06] 10Beta-Cluster-Infrastructure, 06Labs: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#2450818 (10AlexMonk-WMF) a:03AlexMonk-WMF I'm far enough into {T64835} now that I think we can almost call this done. [23:51:24] 10Beta-Cluster-Infrastructure, 06Operations: /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config - https://phabricator.wikimedia.org/T129586#2450877 (10AlexMonk-WMF) a:03AlexMonk-WMF [23:59:49] 10Beta-Cluster-Infrastructure, 06Operations, 13Patch-For-Review: /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config - https://phabricator.wikimedia.org/T129586#2450916 (10AlexMonk-WMF) 05stalled>03Open