[00:40:18] 10scap, 13Patch-For-Review, 03Scap3 (Scap3-MediaWiki-MVP), 07WorkType-NewFunctionality: Basic scap{2,3} canary deployment process & checks - https://phabricator.wikimedia.org/T110068#2359855 (10GWicke) Example run of https://gerrit.wikimedia.org/r/#/c/292505/: python logstash_checker.py --host mw1167 --au... [01:06:27] 10scap, 13Patch-For-Review, 03Scap3 (Scap3-MediaWiki-MVP), 07WorkType-NewFunctionality: Basic scap{2,3} canary deployment process & checks - https://phabricator.wikimedia.org/T110068#2359895 (10thcipriani) One question in terms of integration with Scap: would it be possible to have logstash_checker.py acce... [01:34:38] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #648: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/648/ [01:34:45] (03PS3) 10Lethexie: Fix single space expected on single line comment. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/292313 (https://phabricator.wikimedia.org/T136791) [01:49:22] (03PS4) 10Legoktm: Fix single space expected on single line comment. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/292313 (https://phabricator.wikimedia.org/T136791) (owner: 10Lethexie) [01:49:33] (03CR) 10Legoktm: [C: 032] "Nice work :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/292313 (https://phabricator.wikimedia.org/T136791) (owner: 10Lethexie) [01:49:35] (03PS4) 10Lethexie: Add function commenting sniff. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/291489 [01:49:55] (03CR) 10Legoktm: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/275747 (https://phabricator.wikimedia.org/T130004) (owner: 10Lethexie) [01:51:38] (03Merged) 10jenkins-bot: Fix single space expected on single line comment. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/292313 (https://phabricator.wikimedia.org/T136791) (owner: 10Lethexie) [01:57:19] Project browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce build #70: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce/70/ [09:01:06] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#2360257 (10hashar) From discussion on https://github.com/jenkinsci/ircbot-plugin/commit/d18cc7b617155100... [09:02:56] !log Upgrading Jenkins IRC plugin 2.25..2.27 and instant messaging plugin 1.34..1.35 . The former should fix a deadlock on shutdowning Jenkins | T96183 [09:02:57] T96183: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183 [09:03:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:03:50] Project beta-code-update-eqiad build #107609: 15ABORTED in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/107609/ [09:20:07] Project beta-update-databases-eqiad build #9014: 04FAILURE in 7.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9014/ [09:36:34] Yippee, build fixed! [09:36:35] Project beta-update-databases-eqiad build #9015: 09FIXED in 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9015/ [10:31:23] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2360380 (10mobrovac) [10:33:20] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2360393 (10mobrovac) p:05Triage>03High [11:09:47] (03PS2) 10Hashar: Phase out perllint [integration/config] - 10https://gerrit.wikimedia.org/r/292926 [11:10:02] (03CR) 10Hashar: [C: 032] Phase out perllint [integration/config] - 10https://gerrit.wikimedia.org/r/292926 (owner: 10Hashar) [11:11:12] (03Merged) 10jenkins-bot: Phase out perllint [integration/config] - 10https://gerrit.wikimedia.org/r/292926 (owner: 10Hashar) [11:11:48] (03Abandoned) 10Hashar: Add params['PHP_BIN'] = 'php5' to npm-node-0.10 [integration/config] - 10https://gerrit.wikimedia.org/r/292626 (owner: 10Paladox) [11:14:38] (03CR) 10Hashar: [C: 04-1] "IIRC in production it still runs on Trusty and Node 0.10 so we probably want to keep it this way." [integration/config] - 10https://gerrit.wikimedia.org/r/291603 (owner: 10Paladox) [11:15:28] (03PS6) 10Hashar: Migrate mediawiki-phpunit-phpflavour-composer to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/290806 (https://phabricator.wikimedia.org/T135001) (owner: 10Paladox) [11:36:07] (03CR) 10Hashar: [C: 04-1] "The composer jobs on Nodepool ends up running phpunit with PHPUNIT_EXCLUDE_GROUP=ParserTests which filter out the parsertests." [integration/config] - 10https://gerrit.wikimedia.org/r/290806 (https://phabricator.wikimedia.org/T135001) (owner: 10Paladox) [11:38:52] (03PS10) 10Hashar: Migrate mwext-testextension-{phpflavor} to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/292509 (owner: 10Paladox) [11:47:17] (03CR) 10Hashar: [C: 04-1] "Few comments. We might have to update tests as well but I havent checked :)" (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/292509 (owner: 10Paladox) [12:02:46] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360632 (10Paladox) [12:03:17] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2360645 (10Paladox) [12:03:18] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360644 (10Paladox) [12:06:13] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360647 (10Paladox) [12:06:31] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360632 (10Paladox) [12:08:35] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360653 (10Paladox) [12:10:26] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2360655 (10Paladox) [12:10:28] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360632 (10Paladox) [12:10:42] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2360657 (10Paladox) [12:10:44] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#1819085 (10Paladox) [12:13:18] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, 07WorkType-NewFunctionality: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2360665 (10Paladox) [12:19:56] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Setup a codereview hour for non american people - https://phabricator.wikimedia.org/T136370#2360681 (10Nikerabbit) Any updates on the plans? It's been about one month since code review office hours started. [12:46:10] (03PS1) 10Hashar: mwext-mw-selenium to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/293096 (https://phabricator.wikimedia.org/T137112) [12:46:25] (03CR) 10Hashar: "Resent on https://gerrit.wikimedia.org/r/293096" [integration/config] - 10https://gerrit.wikimedia.org/r/292927 (https://phabricator.wikimedia.org/T137112) (owner: 10Hashar) [13:00:34] (03PS2) 10Hashar: mwext-mw-selenium to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/293096 (https://phabricator.wikimedia.org/T137112) [13:01:45] (03PS3) 10Hashar: mwext-mw-selenium to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/293096 (https://phabricator.wikimedia.org/T137112) [13:30:53] jzerebecki, hashar: CI meeting today? [13:31:06] my guess is that it is cancelled, but I could be wrong :) [13:32:11] no idea [13:32:12] zeljkof: ok I'll join [13:32:20] lets held a short one yeah [13:32:32] I will have to go around 5:20pm [13:32:33] I have a few questions related to browser tests :D [13:32:38] but I can stay for 20 minutes [13:32:56] jzerebecki: zeljkof if you two are available now, we can do it now [13:33:44] hashar, jzerebecki I am free now and until about 5:20 [13:34:43] * jzerebecki is setting up [13:35:47] I am in https://hangouts.google.com/hangouts/_/wikimedia.org/btest-ci-weekly :D [13:35:58] me too [13:36:36] zeljkof: join us :D [13:36:53] hashar, jzerebecki coming [13:36:55] in a minute [13:38:03] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T136040#2360791 (10thcipriani) 05Open>03Resolved [14:00:54] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2360822 (10JanZerebecki) [14:15:21] zeljkof: I think I will use a different JJB macro for the mwext-mw-selenium job [14:15:37] that is stripped out from most of the mess and be suitable for Jessie :D [14:16:13] lot of the crap was to get different jobs to share the same gem cache on a given instance [14:16:18] with nodepool that is no more needed [14:19:14] hashar: good idea [14:19:49] zeljkof: the mwext-mw-selenium still uses a hardcoded cucumber command [14:19:56] should have it migrated to rake probably [14:20:46] yes [14:20:56] rake all the things [14:23:13] I will copy paste / clean up the mess first [14:23:17] then attempt to switch to rake [14:24:07] zeljkof: https://phabricator.wikimedia.org/P3216 :D [14:25:31] :D [14:25:49] I was trying to refactor the existing macro [14:25:54] but starting fresh is easier [14:41:15] tip to do a bandwith testing: mvn clean package [14:49:35] hashar hi [14:49:39] lo [14:49:45] with mw-selenium we can make it an experimental test [14:49:50] nop [14:49:52] meaning make a nodepool test [14:50:00] I am just going to migrate all of them in one go [14:50:04] ok [14:50:07] :) [14:52:08] (03PS4) 10Hashar: mwext-mw-selenium to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/293096 (https://phabricator.wikimedia.org/T137112) [14:52:15] paladox: something messy such as ^^ [14:52:23] hashar ok. [14:52:24] :) [14:52:29] a bunch of what mw-selenium is doing is not needed anymore :D [14:52:33] hashar also can you review https://gerrit.wikimedia.org/r/290791 please. [14:52:38] migrates mw core jsduck test [14:53:05] hashar also https://gerrit.wikimedia.org/r/291501 [14:53:12] another migrate to nodepool [14:53:15] for codesniffer [14:53:41] hashar also https://gerrit.wikimedia.org/r/290703 please [14:55:17] Project beta-scap-eqiad build #105842: 15ABORTED in 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/105842/ [14:56:16] !log Restarting Jenkins to upgrade Rebuilder plugin with https://github.com/jenkinsci/rebuild-plugin/pull/34 (sort out parameters not being reinjected) [14:56:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:56:36] paladox: I have a reviewed a couple ones later [14:56:53] hashar ok thanks [15:03:00] hashar what would the alias be for php 5.6 on jessie [15:03:07] since we have php53 and php55 [15:03:17] but shoulden we add php56 for jessie. [15:03:41] please [15:04:20] (03PS11) 10Paladox: Migrate mwext-testextension-{phpflavor} to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/292509 [15:05:12] (03CR) 10jenkins-bot: [V: 04-1] Migrate mwext-testextension-{phpflavor} to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/292509 (owner: 10Paladox) [15:06:37] zeljkof: and that fails https://integration.wikimedia.org/ci/job/mwext-mw-selenium-jessie/7/console :D [15:06:57] avconv not found on your system [15:06:58] oh [15:07:02] f*9*********** [15:07:28] avconn [15:07:29] va [15:07:32] avconv? [15:09:07] (03CR) 10Hashar: [C: 04-2] "Tried on https://integration.wikimedia.org/ci/job/mwext-mw-selenium-jessie/7/console and that fails with:" [integration/config] - 10https://gerrit.wikimedia.org/r/293096 (https://phabricator.wikimedia.org/T137112) (owner: 10Hashar) [15:10:43] (03PS12) 10Paladox: Migrate mwext-testextension-{phpflavor} to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/292509 [15:10:56] hashar you said here https://gerrit.wikimedia.org/r/#/c/292509/10/jjb/mediawiki-extensions.yaml "archive-log" [15:11:07] but archive-log does not exist [15:11:10] as a macro or anything [15:11:22] it would be archive-log-dir [15:16:21] (03PS13) 10Paladox: Migrate mwext-testextension-{phpflavor} to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/292509 [15:17:01] 10Browser-Tests-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: migrate mwext-mw-selenium to Nodepool instances - https://phabricator.wikimedia.org/T137112#2361085 (10hashar) Eventually I had to start fresh regarding how we run bundler/cucumber Tried on https://integration.wikimedia.o... [15:20:30] hashar hi do you know how php is called in jessie [15:20:36] Do you do php or php5 [15:20:45] paladox: which php ? :D [15:20:48] since im updating [15:20:49] - builder: [15:20:49] name: assert-phpflavor [15:20:49] builders: [15:20:49] - shell: | [15:20:50] for Zend: /usr/bin/php5 [15:20:52] hashar 5.6 [15:20:54] for hhv: /usr/bin/hhvm [15:21:13] hashar oh so in jessie we can use php5 [15:21:33] not really [15:21:35] that is 5.6 [15:21:43] and Mediawiki claims to be 5.5 compatible [15:21:46] so it depends [15:21:53] if you dont really care, use whatever version [15:22:03] else should run on trusty [15:22:16] hashar oh. Im just trying to update the alias php in macro [15:22:21] so we can do php53 and php55 [15:22:38] and eventually add a test for php 5.6 to make sure it is compat with both 55 and 56 [15:22:48] in theory yeah [15:22:58] Ok thanks [15:23:08] paladox: then I dont really want 5.6 tests right now [15:23:26] that would add a lot of build on the infra, and there are not that many nodes available [15:23:31] hashar oh ok, im not adding php56 test im adding the alias for later. [15:23:32] maybe later when we have more capacity [15:23:41] Ok [15:23:58] hashar i will quickly upload to show you what i mean [15:24:48] (03PS1) 10Paladox: Add php56 alias [integration/config] - 10https://gerrit.wikimedia.org/r/293112 [15:24:53] hashar ^^ [15:25:09] meanwhile Jessie is making fun of me [15:25:50] hashar oh :) [15:26:10] hashar ive updated https://gerrit.wikimedia.org/r/#/c/292509/ [15:26:32] to address your comments :) [15:27:34] hashar is it mw-selenium failing for you [15:29:22] 10Browser-Tests-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: migrate mwext-mw-selenium to Nodepool instances - https://phabricator.wikimedia.org/T137112#2361112 (10hashar) Apparently with jessie-backports we can co install libav-tools and ffmpeg \o/ [15:30:52] (03PS4) 10Paladox: Migrate wikimedia/fundraising/dash node-0.10 test to node-4.3 test [integration/config] - 10https://gerrit.wikimedia.org/r/291603 [15:41:02] (03PS14) 10Paladox: Migrate mwext-testextension-{phpflavor} to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/292509 (https://phabricator.wikimedia.org/T137199) [15:41:49] heading out [15:46:48] 10Continuous-Integration-Infrastructure, 10Nodepool: Clean up apt:pin of python modules used for Nodepool - https://phabricator.wikimedia.org/T137217#2361186 (10hashar) [15:47:35] 10Continuous-Integration-Infrastructure, 10Nodepool: Clean up apt:pin of python modules used for Nodepool - https://phabricator.wikimedia.org/T137217#2361198 (10hashar) [15:51:18] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, 03Scap3 (Scap3-Adoption-Phase1): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2361210 (10Ottomata) a:03elukey [15:59:32] (03PS7) 10Paladox: Migrate mediawiki-phpunit-phpflavour-composer to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/290806 (https://phabricator.wikimedia.org/T135001) [16:00:53] (03CR) 10EBernhardson: "mostly looks good. This patch should not update files that havn't actually changed (differences in paths or runtime of the .expect files)." (036 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/275747 (https://phabricator.wikimedia.org/T130004) (owner: 10Lethexie) [16:01:39] Project selenium-CentralNotice » chrome,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #47: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/47/ [16:01:43] Project selenium-CentralNotice » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #47: 04FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/47/ [16:02:04] Project selenium-CentralNotice » chrome,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #47: 04FAILURE in 1 min 3 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/47/ [16:02:33] Project selenium-CentralNotice » firefox,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #47: 04FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/47/ [16:02:40] zeljkof ^^ [16:12:31] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Setup a codereview hour for non american people - https://phabricator.wikimedia.org/T136370#2361258 (10Dereckson) When I discussed the ideal timetable with Nikerabbit, in regard to some stability in summer/winter timezones, I woud recommend... [16:18:14] (03CR) 10EBernhardson: "recheck" (0324 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/291489 (owner: 10Lethexie) [16:20:25] Project beta-update-databases-eqiad build #9022: 04FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9022/ [16:50:40] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Setup a codereview hour for non american people - https://phabricator.wikimedia.org/T136370#2361381 (10mmodell) {E205} [16:50:42] (03CR) 10Paladox: "Ok done :)" [integration/config] - 10https://gerrit.wikimedia.org/r/290806 (https://phabricator.wikimedia.org/T135001) (owner: 10Paladox) [16:52:30] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Setup a codereview hour for non american people - https://phabricator.wikimedia.org/T136370#2361414 (10Paladox) @mmodell will 8-9pm continue too please. [16:53:57] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10Dzahn) [16:54:29] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361435 (10Dzahn) [16:56:18] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361445 (10greg) p:05Triage>03Normal [16:59:48] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361460 (10Dzahn) [16:59:51] 05Gitblit-Deprecate, 10releng-201516-q2: [keyresult] Deprecate gitblit in favor of Diffusion - https://phabricator.wikimedia.org/T111465#2361459 (10Dzahn) [17:00:44] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Setup a codereview hour for non american people - https://phabricator.wikimedia.org/T136370#2361464 (10Dereckson) >>! In T136370#2361258, @Dereckson wrote: > I woud recommend 13:00-14:00 UTC. You encoded 11:00-12:00 UTC, which would be 13:... [17:01:12] 05Gitblit-Deprecate, 10releng-201516-q2: [keyresult] Deprecate gitblit in favor of Diffusion - https://phabricator.wikimedia.org/T111465#1604788 (10Dzahn) @Paladox i made a subtask for this, let's see if we can come up with some rewrite rules [17:17:08] Yippee, build fixed! [17:17:08] Project selenium-CentralNotice » chrome,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #48: 09FIXED in 30 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/48/ [17:17:15] Yippee, build fixed! [17:17:15] Project selenium-CentralNotice » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #48: 09FIXED in 37 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/48/ [17:17:49] Yippee, build fixed! [17:17:49] Project selenium-CentralNotice » chrome,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #48: 09FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/48/ [17:18:09] Yippee, build fixed! [17:18:09] Project selenium-CentralNotice » firefox,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #48: 09FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/48/ [17:20:34] Yippee, build fixed! [17:20:34] Project beta-update-databases-eqiad build #9023: 09FIXED in 33 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9023/ [17:27:48] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361612 (10Paladox) @dzahn with https://phabricator.wikimedia.org/D250 it should make it easy to create redirect links. Since all we need t... [17:29:42] hey, is there anyone to land this? https://phabricator.wikimedia.org/D253 [17:29:57] it's accepted but I couldn't land it [17:31:09] hmm, you should be able to land IIRC [17:31:19] ^ twentyafterfour is that right? [17:39:10] thcipriani: no it was previously restricted to releng+bryan... I've updated the 'can push' policy on scap repo. Amir1: try again? [17:39:38] twentyafterfour: sure [17:41:00] twentyafterfour: couldn't do it [17:46:40] Amir1: what error are you seeing? [17:51:36] twentyafterfour: fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/MSCA/scap.git/': The requested URL returned error: 403 [17:51:37] Usage Exception: Push failed! Fix the error and run "arc land" again. [17:58:27] 10Continuous-Integration-Infrastructure, 10Nodepool: Clean up apt:pin of python modules used for Nodepool - https://phabricator.wikimedia.org/T137217#2361717 (10hashar) @Andrew python-openstackclient python-novaclient got imported to `jessie-wikimedia/backports` at a time when we did not use Debian jessie-back... [18:02:30] Amir1 are you pushing over https [18:02:34] or ssh [18:03:34] paladox: it seems I pushed over https [18:03:34] but I can't say for sure [18:04:04] Amir1 if you push over https since it looks like you are you have to create the password for it [18:04:07] in your settings [18:04:41] Amir1 you need to do it here https://phabricator.wikimedia.org/settings/panel/vcspassword/ please [18:04:46] hmm, thanks [18:04:58] Your welcome. [18:06:30] I just landed it [18:06:35] \o/ [18:06:44] nice! [18:07:10] Amir1: kudos. And thanks for the patches. Nice/needed improvements :) [18:08:05] thcipriani: Thanks, I would be more than happy to help, if you have easy tasks to do, tell me [18:10:25] Amir1: ack. I was going to suggest checking out the project workboard: https://phabricator.wikimedia.org/project/view/1449/ but it looks like I've got some triage to do :P [18:10:58] :D [18:19:48] Krinkle hi why did you disable workbords for https://phabricator.wikimedia.org/project/manage/135/ please [18:20:03] Since i found it was easy to look for tasks through work bords [18:20:37] since searching is not an option unless you know what you want to search for [18:28:35] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2360380 (10mmodell) there is no eventlogging userkey on deployment-eventlogging04 [18:35:25] paladox: Workboards make sense for managing multiple columns. Workboards do not make sense for generic projects that are used as tags. [18:35:42] They cause confusion with "Backlog" links and are hard to view in a tiny small column [18:36:21] Krinkle oh, but workbords make it easy to find tasks to work on since all we have at finding tasks is looking at the recent history but there are a ton of tasks in that tag. [18:36:26] Just click on "Open Tasks" to view the tasks [18:36:37] And searching for them works fine too [18:37:01] Oh ok thanks [18:37:20] You're welcome [18:53:51] (03PS5) 10Hashar: mwext-mw-selenium to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/293096 (https://phabricator.wikimedia.org/T137112) [18:53:53] (03PS1) 10Hashar: dib: libav-tools on Jessie for Selenium screenshots [integration/config] - 10https://gerrit.wikimedia.org/r/293146 (https://phabricator.wikimedia.org/T137112) [19:22:18] 06Release-Engineering-Team, 10MediaWiki-Special-pages, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284195 (10greg) This task is old (relatively, given it was marked as blocking 1.28.0-wmf1), untriaged, and open.... [19:24:50] 06Release-Engineering-Team, 10MediaWiki-Special-pages, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284195 (10Bawolff) Sounds like a job for PoolCounter to me. Special:RecentChangesLinked has always been slow in... [19:28:03] !log Nodepool has troubles spawning instances probably due to on going (?) labs maintenance [19:28:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:36:52] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool can barely spawn instances OpenStack - https://phabricator.wikimedia.org/T137241#2361986 (10hashar) [19:42:27] (03CR) 10Hashar: [C: 032] dib: libav-tools on Jessie for Selenium screenshots [integration/config] - 10https://gerrit.wikimedia.org/r/293146 (https://phabricator.wikimedia.org/T137112) (owner: 10Hashar) [19:43:15] (03Merged) 10jenkins-bot: dib: libav-tools on Jessie for Selenium screenshots [integration/config] - 10https://gerrit.wikimedia.org/r/293146 (https://phabricator.wikimedia.org/T137112) (owner: 10Hashar) [19:44:32] thcipriani: Taking it out of ops to reduce clutter. I started poking at repacking last night, but I think we'll need to find a way to do it async. [19:45:02] It shouldn't block a deployment, and for a sufficiently large repo (ie: 50mb or more) it would be stupidly slow to wait on [19:45:15] ah, yeah, that makes sense. [19:45:23] we could kick it off post-deployment [19:45:41] It only really needs to happen after initial clone. [19:45:53] reparent the process [19:46:03] But git.fetch() doesn't return whether it cloned or fetched so we can't know easily. [19:46:50] yeah, we could split that code up. Might be more obvious to do it that way. [19:47:49] Considering we only call git.fetch() in 2 places in deploy (both deploy.fetch()) it would be easy enough to move that logic up a level [19:47:57] hashar hi it seems https://integration.wikimedia.org/zuul/ is stuck [19:48:32] for mediawiki/extensions/CentralNotice [19:50:07] ostriches: can i bother you to make a fundraising/REL1_26 branch on gerrit? i tried to but bad perms. feel free to tell me to file a bug. [19:50:23] ostriches: yeah, I have no problem with a logic split there. It was only built that way because we were trying to abstract away the whole of fetch. If we could keep that, seems legit to kick off a repack when a clone happens. [19:51:05] paladox: nodepool lacks instances to run jobs on. ops are looking at it via #wikimedia-labs [19:51:25] Ok thanks [19:51:28] hashar ^^ [19:52:36] thcipriani: Although, as long as we spawn it as a detached process and don't rely on the return (really, there's not much way it can fail, and if it does, it would just result in the status quo) [19:52:42] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Setup a codereview hour for non american people - https://phabricator.wikimedia.org/T136370#2362041 (10mmodell) Sorry, my mistake. {E206} [19:52:56] clone + spawn a detached proc to repack + continue [19:53:55] yup, create a new process and disown it. As long as there's nothing weird about trying to clone from it before the repack. [19:54:23] like if we moved from fetch to promote before the repack completes. [19:55:58] paladox: if you look at https://grafana.wikimedia.org/dashboard/db/releng-zuul?panelId=18&fullscreen is an experimental graph that shows how long it takes for a build to start per Nodepool labels [19:56:17] hashar oh ok :) [19:56:18] paladox: in the top right change the time span to "Last 6 hours" [19:56:35] and it shows it went up to roughly 25 minutes [19:56:47] Yep [19:57:15] 06Release-Engineering-Team, 15User-greg: Review peer-review feedback and write up my review - https://phabricator.wikimedia.org/T136436#2362059 (10greg) 05Open>03Resolved [19:57:16] 06Release-Engineering-Team, 15User-greg: RelEng 2015/16 Annual Reviews - https://phabricator.wikimedia.org/T136434#2362061 (10greg) [19:57:30] oh [19:57:31] https://grafana.wikimedia.org/dashboard/db/releng-zuul?panelId=18&fullscreen&from=1465307829756&to=1465329129756 [19:57:37] thqat might be the proper link [19:59:40] Oh [20:00:30] hashar it shows ci-trusty having a huge increase [20:00:33] thcipriani: I'll work up a patch that we can test. [20:00:37] from 2 to 30 mins [20:02:16] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool can barely spawn instances OpenStack - https://phabricator.wikimedia.org/T137241#2362081 (10hashar) @Andrew knows about the issue > somehow rabbitmq for labvirt1010 is gummed up. It connects just fine, but no messages pass bac... [20:02:42] 10Browser-Tests-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: migrate mwext-mw-selenium to Nodepool instances - https://phabricator.wikimedia.org/T137112#2362084 (10hashar) Will need to refresh the Jessie image to have libav-tools provided but labs has some troubles right now T137241 [20:11:30] thcipriani: I'm working up a patch and testing now [20:12:40] ostriches: sweet, thanks. Other thought: we could just kick off a repack of the cache directory at the end of each successful deployment. [20:13:30] It only needs to happen after a fetch stage really, any other stages would be useless. [20:14:30] ah, because the hardlinks would all be to loose object files and the diskspace wouldn't be reclaimed until that checkout is removed? [20:15:17] Something like that. Mainly just that unless you're cloning or doing actual commits, the packs won't really come undone. [20:15:38] If you're just doing checkouts, nothing in the pack will change. [20:18:10] Hmmm, fresh vagrant, git rev-parse don't like meeeeeee [20:21:51] thcipriani: that is .git/objects which is hardlinked . If a commit get unreferenced in the reference repo and it is referenced in one of the child clones, the child becomes corrupt :( [20:22:23] same deal with git clone --reference [20:22:38] sure, but this isn't a reference clone, just a local clone. [20:22:40] that one doesnt hardlink .git/objects but adds a file that point it to the parent repo [20:22:45] with the same drawback [20:22:59] so git clone on same FS uses hardlinks which might be troublesome sometime [20:23:26] and git clone --reference leaves you with a basically empty .git/objects and some file that reference the copy [20:23:43] On Nodepool instance I have added a few huge/high activity repo under /srv/git cloned as bare repo [20:24:02] then the job can clone them (hardlinks) then fetch from Gerrit to grab the delta / prune branches etc [20:24:18] the mirrors on /srv/git are never touched, and the instance is gone at end of build anyway [20:24:22] so there is no room for corruption [20:25:15] yeah, so scap3 does a similar thing. Cached repo at /srv/deployment/[repo]-cache/cache [20:25:33] uses deployment host as remote [20:25:41] `git clone --reference --dissociate` is nice :) [20:25:43] the thing to watch at is commit becoming unreferenced in the repo cache [20:26:16] I think --dissociate was added in 2.3 [20:26:31] so then, kick off a deploy, updates cache repo, clones from cache to: /srv/deployment/[repo]-cache/[rev] which uses all hardlinks. Then symlink to final /srv/deployment/[repo] [20:27:13] new [rev] directories are created and cleaned up in a rolling way on subsequent deployments [20:27:24] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2362182 (10mobrovac) Since [eventbus uses the eventlogging user](https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/eventlogging/... [20:27:29] but the /srv/deployment/[repo]-cache is long-lived. [20:27:41] er, /srv/deployment/[repo]-cache/cache [20:29:07] and if you update the cache keeping loose objects for how long we keep [rev], that should be fine [20:29:23] git gc pruning them after 2 weeks by default iirc [20:29:36] then maybe it is a non issue :D [20:29:52] I always have been very paranoid about and never thoroughly tested the corruption scenario [20:30:11] Waiting 2 weeks on a repo that doesn't actually change on disk that much is kinda a waste. [20:30:14] but since it's a hardlink, and we clean the cache which removes the object, then wouldn't the clone just have the only extant copy of the object. [20:30:17] A checkout won't trigger it afaik. [20:30:31] We want to repack after clone, really. [20:30:39] (Which is what my patch will strive to do) [20:30:56] then for /srv/deployment/[repo]-cache/[rev] do you even need the git objects? [20:31:04] could a snapshot of the code (via git archive) be sufficient? [20:32:58] we don't need the git objects at that point. [20:33:15] hashar: No, because we might roll back. Deploying an old revision would be slower then. [20:33:33] well all targets still have rev-1 don't they? [20:33:56] I though Dan idea was to push code first under a rev number then get scap to just switch symlink [20:34:00] yup. [20:34:02] while keeping the old revs [20:34:05] that's still the case. [20:34:38] We have rev-1, but we could deploy a significantly older revision for some reason :p [20:34:51] I think git archive would just complicate things. [20:35:07] probably yeah [20:35:11] :D [20:35:12] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2362225 (10Ottomata) @mobrovac, just curious, how is this blocking changeprop deployment? Did you make changes to eventlogging/eventbus that you need to... [20:35:16] Honestly, .git/objects isn't going to waste that much space for most scenarios, we just want to shave off what stupid wasted space we can. [20:35:37] for most repos I dont think it is much of a trouble really [20:35:38] hence, repack all the things :p [20:35:49] mw/core and all its extension might though [20:35:56] Yeah. Plus a few others. [20:36:04] But most it will be negligible, you're right. [20:45:09] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T136042#2362275 (10thcipriani) [20:47:13] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362280 (10Paladox) [20:53:13] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362289 (10Paladox) [20:54:20] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2362291 (10mmodell) @mobrovac: all bets are off when `scap::target` has `manage_user => false`: ```lang=puppet scap::target { "eventlogging/${title}... [20:54:45] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2360380 (10Pchelolo) @Ottomata The problem is seems like the topics are not created automatically. We've created a transclude-continue topic in the event... [21:01:09] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362308 (10Paladox) [21:10:57] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362329 (10Paladox) [21:13:45] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, 13Patch-For-Review: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2362332 (10mobrovac) a:03Ottomata >>! In T137192#2362291, @mmodell wrote: > @mobrovac: all bets are off when `scap::target` has `... [21:14:07] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, 13Patch-For-Review: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2362334 (10mmodell) ca8a0c35275afb5a36 is now cherry-picked on beta. [21:15:01] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, 13Patch-For-Review: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2362348 (10mmodell) ```Info: Applying configuration version '1465333826' Notice: /Stage[main]/Eventlogging::Server/Ssh::Userkey[eve... [21:15:01] greg-g: Hi could we archive https://phabricator.wikimedia.org/project/manage/1523/ please. [21:15:11] Since were redirecting git.wikimedia.org links really soon. [21:16:06] or ostriches ^^ [21:16:37] or twentyafterfour [21:16:39] ^^ [21:21:26] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, 13Patch-For-Review: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2362352 (10mobrovac) Thank you, @mmodell! This did the trick. I was able to deploy eventbus in beta. Let's see what @Ottomata says... [21:32:24] yay for green: https://phabricator.wikimedia.org/diffusion/PHEX/ [21:32:44] individual commits are now building and reporting back test status [21:33:35] one quick question: Where is logs of the beta wikis (e.g. runJobs.log) [21:33:52] I know it was in a labs setup but I can't find it [21:35:40] fluorine [21:38:26] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10mmodell) D250 is now deployed. [21:43:56] paladox: Archived. [21:44:17] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362395 (10Paladox) [21:44:17] * twentyafterfour cheers [21:44:24] mini party [21:45:22] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10Paladox) [21:46:36] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362407 (10Paladox) [21:46:43] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2361418 (10Paladox) [21:46:55] ostriches thanks [21:49:00] 10Beta-Cluster-Infrastructure, 06Revision-Scoring-As-A-Service: ores-beta is down - https://phabricator.wikimedia.org/T135677#2362427 (10Ladsgroup) [21:49:03] 03Scap3, 06Revision-Scoring-As-A-Service: Scap3 checks should not be random - https://phabricator.wikimedia.org/T136253#2362426 (10Ladsgroup) 05Open>03Resolved [22:15:14] 10Continuous-Integration-Infrastructure, 05Gerrit-Migration, 03releng-201617-q1, 10Differential: Build glue from Differential to Nodepool - https://phabricator.wikimedia.org/T130950#2362534 (10mmodell) Build status now shows for individual commits on diffusion commit history lists. Example: [[ /r/p/phabric... [22:18:56] Reedy: thanks :) [22:22:05] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool can barely spawn instances OpenStack - https://phabricator.wikimedia.org/T137241#2362566 (10Andrew) 05Open>03Resolved a:03Andrew This turned out to not be a rabbitmq issue; rather, libvirtd was having some kind of crisis b... [22:37:36] (03PS1) 10Ladsgroup: Whitelist [lL]adsgroup [integration/config] - 10https://gerrit.wikimedia.org/r/293235 [22:41:40] (03CR) 10Ladsgroup: "^ As you can see :D" [integration/config] - 10https://gerrit.wikimedia.org/r/293235 (owner: 10Ladsgroup) [22:42:29] (03CR) 10Paladox: [C: 031] Whitelist [lL]adsgroup [integration/config] - 10https://gerrit.wikimedia.org/r/293235 (owner: 10Ladsgroup) [23:35:45] 05Gitblit-Deprecate, 10Diffusion: Diffusion blob viewer may not work as expected - https://phabricator.wikimedia.org/T129129#2362694 (10Paladox) I now fixed it, see https://www.mediawiki.org/w/index.php?title=Template:Git_file&oldid=2159131 please. [23:38:35] 05Gitblit-Deprecate, 10Diffusion: Diffusion blob viewer may not work as expected - https://phabricator.wikimedia.org/T129129#2362699 (10Paladox) @greg I'm not sure if I can close this as resolved could you review the link and if you think that fixes what is described here please could you close this as resolve... [23:51:35] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362721 (10Paladox) [23:56:20] PROBLEM - MD RAID on gallium is CRITICAL: CRITICAL: Active: 1, Working: 1, Failed: 1, Spare: 0