[00:36:25] 6RelEng-Admin, 15User-greg: Update Master Project List (MPL) by end of Q1 / before Q2 FY 2015-2016 - https://phabricator.wikimedia.org/T108629#1582592 (10greg) [00:39:43] 6RelEng-Admin, 10Gerrit-Migration, 15User-greg: Outline work (outcomes and outputs) of RelEng's Q2 Gerrit migration work - https://phabricator.wikimedia.org/T110623#1582601 (10greg) 3NEW [00:45:58] Project beta-scap-eqiad build #67538: FAILURE in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/67538/ [00:55:37] Yippee, build fixed! [00:55:37] Project beta-scap-eqiad build #67539: FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/67539/ [00:56:22] !log sudo keyholder arm on deployment-bastion fixed beta-scap-eqiad [00:56:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:58:53] 10Beta-Cluster, 10Pywikibot-OAuth, 7Pywikibot-tests: Set up Pywikibot account on beta sites to run user tests - https://phabricator.wikimedia.org/T100797#1582772 (10jayvdb) [02:59:16] 10Beta-Cluster, 10Pywikibot-OAuth, 7Pywikibot-tests: Set up Pywikibot account on beta sites to run user tests - https://phabricator.wikimedia.org/T100797#1320666 (10jayvdb) 5Open>3Resolved [04:24:13] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 7Pywikibot-network, 7Pywikibot-tests: Nonce already used regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1582862 (10jayvdb) [05:35:44] Yippee, build fixed! [05:35:44] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #524: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/524/ [05:36:27] legoktm: can you look at, https://phabricator.wikimedia.org/T110518 ? [05:36:43] * legoktm looks [05:40:57] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, 5ContentTranslation-Release6: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1582944 (10Legoktm) [05:41:23] kart_: I'm not sure, I cc'd jzerebecki who is a lot more familiar with Wikidata's CI setup [05:44:17] legoktm: thanks! [05:45:50] kart_: everything is fine on master though? [05:45:53] that's weird... [05:57:32] legoktm: yes. [06:46:42] 6RelEng-Admin, 10Gerrit-Migration, 15User-greg: Outline work (outcomes and outputs) of RelEng's Q2 Gerrit migration work - https://phabricator.wikimedia.org/T110623#1582987 (10Qgil) Do you think you will need some help from #Engineering-Community during October-December? Just checking for our own planning. [07:00:11] Yippee, build fixed! [07:00:11] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #552: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/552/ [08:07:09] zeljkof-meeting: hey sorry late again :( [08:07:20] been chatting with folks [08:07:28] hashar: no problem, I am in the hangout, join at will :) [08:36:25] (03CR) 10Filippo Giunchedi: [C: 04-1] Add service deploy via scap (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/224374 (owner: 10Thcipriani) [09:00:29] (03Abandoned) 10Florianschmidtwelzow: Add CodeMirror to auto-branched extensions [tools/release] - 10https://gerrit.wikimedia.org/r/230782 (https://phabricator.wikimedia.org/T95041) (owner: 10Florianschmidtwelzow) [09:01:07] zeljkof-meeting: sorry went out of power :( [09:01:14] anyway the idea is that nodepool add the instance to Jenkins master [09:01:15] ex https://integration.wikimedia.org/ci/computer/ci-dib-jessie-wikimedia-73/ [09:01:24] hashar: no problem [09:01:26] wich has a specific label, here 'ci-dib-jessie-wikimedia' [09:01:33] so the job have to be tied to that label [09:01:35] I have to go to another meeting [09:01:42] and passed a special parameter NODE_OFFLINE_WHEN_COMPLETE [09:01:57] when the job is compete, Jenkins will put the slave offline, preventing further jobs from running on it [09:02:09] and after sometime, nodepool will unpool the slave and delete the instance [09:02:10] I remember that offline setting [09:02:14] yeah [09:02:19] so from our 1/1 [09:02:30] I gotta fix up nodepool package so it works [09:02:58] and write down doc to generate images and upload them to openstack [09:06:32] sounds good [09:10:18] https://gerrit.wikimedia.org/r/234483 should let us login [09:11:30] ahh [09:11:41] and I found the issue with the python module dependencies [09:11:54] apparently missing jessie-backports [09:15:25] hashar: do you have a few minutes to join a meeting with Amir and me? [09:15:39] they are stuck on a CI problem, and I have no clue how to fix it :) [09:16:01] hashar: feel free to join https://plus.google.com/hangouts/_/wikimedia.org/aaharoni-zfilip?authuser=1 [09:23:10] 5Continuous-Integration-Isolation, 6operations: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1583324 (10hashar) 3NEW [09:27:21] hi hashar + zeljkof-meeting [09:27:32] https://phabricator.wikimedia.org/T110518 [09:28:09] aharoni: sorry, network problems, joining in a second [09:28:14] Yippee, build fixed! [09:28:14] zeljkof-meeting: ok [09:28:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #705: FIXED in 1 hr 18 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/705/ [09:31:10] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, 5ContentTranslation-Release6: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1583337 (10hashar) On th... [09:31:12] zeljkof-meeting: I have to cancel or meeting today, to much other stuff to do :-/ [09:31:27] CFisch_WMDE: just saw the mail, no problem, see you next week :) [09:31:38] jeep see zou :-) [09:32:09] but btw thanks for looking into the Wikidata issues ;-) [09:35:10] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, 5ContentTranslation-Release6: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1583346 (10hashar) The m... [09:35:22] hashar: https://gerrit.wikimedia.org/r/#/c/232837/ [09:35:48] CFisch_WMDE: still more to do there :) [09:37:14] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, 5ContentTranslation-Release6: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1583357 (10hashar) On th... [09:39:00] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, 5ContentTranslation-Release6: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1583359 (10hashar) That... [09:40:31] #wikidata [09:48:52] kart_: the CX patch issue you had yesterday is going to be fixed soonish hopefully [09:49:00] kart_: aharoni poked us a few minutes ago [09:49:10] kart_: was in a rush yesterday sorry :-/ [09:52:59] 5Continuous-Integration-Isolation, 6operations: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1583380 (10hashar) It is not showing up at http://apt.wikimedia.org/wikimedia/pool/backports/p/ The `python-gear` package is found... [10:18:30] 10Beta-Cluster, 3Reading-Web-Sprint-54-28-Days-Later: Get QuickSurveys enabled on beta cluster - https://phabricator.wikimedia.org/T110199#1583466 (10Jhernandez) [10:19:17] 10Beta-Cluster, 10QuickSurveys, 3Reading-Web-Next-Sprint-55: Get QuickSurveys enabled on beta cluster - https://phabricator.wikimedia.org/T110199#1583470 (10Jhernandez) p:5Triage>3High [10:19:36] 10Beta-Cluster, 10QuickSurveys, 3Reading-Web-Next-Sprint-55: Get QuickSurveys enabled on beta cluster - https://phabricator.wikimedia.org/T110199#1571710 (10Jhernandez) [10:19:45] 10Beta-Cluster, 10QuickSurveys, 3Reading-Web-Next-Sprint-55: Get QuickSurveys enabled on beta cluster - https://phabricator.wikimedia.org/T110199#1571710 (10Jhernandez) [10:20:49] 10Beta-Cluster, 10QuickSurveys, 3Reading-Web-Next-Sprint-55: Get QuickSurveys enabled on beta cluster - https://phabricator.wikimedia.org/T110199#1583477 (10Jhernandez) Moved to next sprint since this is over and it is unplanned work for sprint 54. Added project, blocked tasks and priority. [10:36:14] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, and 2 others: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1583540 (10Amire80) p:5Triage>3High a:... [10:38:00] 10Continuous-Integration-Infrastructure, 6Discovery, 6operations, 7Elasticsearch, 5Patch-For-Review: elasticsearch 1.6.0 fails to start after reboot - https://phabricator.wikimedia.org/T109497#1583546 (10hashar) Or we can remove ElasticSearch from the Jenkins CI slaves. I think I got it installed for th... [10:43:37] !log pooling back integration-slave-trusty-1016 Was once depooled for debugging purposes and repealed ( https://phabricator.wikimedia.org/T110054 ) but apparently Jenkins restart did not pool it back again :/ [10:43:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:43:56] 10Continuous-Integration-Infrastructure: integration-slave-trusty-1016 should be put back online - https://phabricator.wikimedia.org/T110054#1583582 (10hashar) I have put it back on again. [10:58:13] 5Continuous-Integration-Isolation, 6operations, 7Blocked-on-Operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1583651 (10hashar) The package has build dependencies which are not matched in Jessie (see table above). So when ba... [11:04:33] hashar: no worries. Thanks! [11:05:31] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, and 3 others: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1579765 (10JanZerebecki) [11:15:53] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1583708 (10Krinkle) >>! In T99096#1476167, @mmodell wrote: >>>! In T99096#1476090, @Krinkle wrote: >>... [11:26:16] hashar: Is mediawiki-core-code-coverage-2 still used? [11:26:18] I see it in Jenkins [11:26:35] it's causing https://integration.wikimedia.org/cover/mediawiki-core/ to no longer forward to /master/php because there is a php2 [11:26:40] and it still gets build every day [11:35:41] Krinkle: can't remember what it was for [11:35:44] will diff the xml [11:37:16] !log deleting https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-2 (same) [11:37:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:38:38] Krinkle: fixed! [11:39:08] !log gallium: rm -fR /srv/org/wikimedia/integration/cover/mediawiki-core/master/php2 . This way https://integration.wikimedia.org/cover/mediawiki-core/ redirects to the coverage report (thanks Krinkle) [11:39:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:39:17] Krinkle: do we now have a directory browser ? [11:39:30] hashar: Only since 6 months ago :P [11:39:34] But yes, I wrote one. [11:39:34] https://integration.wikimedia.org/cover/ !!!!!!!! [11:39:37] that is great [11:39:59] https://doc.wikimedia.org/mediawiki-core/ [11:40:24] amazing [11:40:55] one day we will have to find some common doc generator for php/js/css [11:41:06] doxygen is dated [11:41:20] hashar: and it auto redirects to the sub directory if there is only one. So for example, JS coverage runs in a browser of which the version can vary so you don't want to hard code https://integration.wikimedia.org/cover/visualeditor/Chromium%2044.0.2403%20(Ubuntu%200.0.0)/ [11:41:20] though it is faster than pure PHP doc generator I tested [11:41:28] but clicking Ve on https://integration.wikimedia.org/cover/ will go to https://integration.wikimedia.org/cover/visualeditor/ [11:41:29] yeah that is handy [11:41:30] and resolve [11:41:42] ohhh [11:41:54] hashar: There is not likely to be a common generator, personally I don't think there should be. [11:41:55] the code coverage is lovely [11:42:04] Documentation is much more then just parsing generic comments, it's understanding the langauge. [11:42:16] Doxygen is dumb and is probably a common generator, btu at least for JS we don't do that [11:42:28] JSDuck is smart and understands JS concepts. Its' better than any generic parser could be [11:42:29] yeah you demoed to me the JSDuck generators which let you have an interactive console to exercise the documented code [11:42:47] also with regards to how JS does class inheritance and Mixins [11:42:48] I thought about dishing doxygen for PHP doc [11:42:54] cause the interface is rather ugly [11:43:02] Yeah [11:43:06] and it's broken [11:43:16] but the pure PHP versions are rather slow for mwcore :-( [11:43:18] don't forget we have filter.php to make it work [11:43:22] yeah [11:43:25] annoying [11:43:27] mwdoc-filter.php [11:43:35] Very often comment syntax is broken [11:43:36] there was a patch to improve the PHP Doxygen parser but it never landed [11:43:44] our comment syntax is non-standard, most IDEs don't understand it [11:43:54] :-( [11:44:03] I think it is inherited from phpdoc [11:44:06] kind of [11:44:08] but not exactly [11:44:28] personally I don't use IDEs. they're a waste of time. [11:44:39] But for people who use them, it's annoying when the syntax isn't compatible. [11:55:18] Krinkle: I am using vim [11:55:33] but lack plugins to display the PHP comments :( [11:58:16] snack time! [12:41:09] Yippee, build fixed! [12:41:10] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #781: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/781/ [12:53:17] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 7Pywikibot-network, 7Pywikibot-tests: Nonce already used regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1583906 (10jayvdb) A bunch of failures at https://travis-ci.org/jayvdb/pywikibot-core/jobs/77671728#L72... [12:54:26] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #578: FAILURE in 25 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/578/ [13:00:09] sellenium user owner here? [13:11:24] (03PS1) 10Hashar: Remove legacy definition [integration/config] - 10https://gerrit.wikimedia.org/r/234519 (https://phabricator.wikimedia.org/T74001) [13:13:16] (03PS1) 10Hashar: Remove obsolete tools/fetch-mw-ext [integration/jenkins] - 10https://gerrit.wikimedia.org/r/234520 (https://phabricator.wikimedia.org/T74001) [13:15:04] 10Continuous-Integration-Infrastructure, 10Gitblit-Deprecate, 5Patch-For-Review, 7Technical-Debt: Remove dependency on git.wikimedia.org - https://phabricator.wikimedia.org/T74001#1583952 (10hashar) Patches above get rid of tools/fetch-mw-ext which used git.wikimedia.org to fetch extensions. Next step wil... [13:19:02] 10Continuous-Integration-Config, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikidata, and 3 others: ContentTranslation tests for mwext-testextension-zend tests fails in wmf19 and wmf20 - https://phabricator.wikimedia.org/T110518#1583961 (10JanZerebecki) 5Open>3Resolved [13:22:02] 10Browser-Tests: Fix easy problems reported by RuboCop in WikiLove - https://phabricator.wikimedia.org/T110672#1583964 (10zeljkofilipin) 3NEW a:3zeljkofilipin [13:22:29] 10Browser-Tests, 6Collaboration-Team-Backlog, 10WikiLove: Update WikiLove repository to mediawiki_selenium Ruby gem 1.x - https://phabricator.wikimedia.org/T99660#1583972 (10zeljkofilipin) [13:49:09] 5Continuous-Integration-Isolation, 6operations: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1584020 (10Andrew) I suspect the issue is that those packages were in the upstream backports repo. Earlier Jessie base images incl... [13:49:09] (03CR) 10JanZerebecki: [C: 032] Remove legacy definition [integration/config] - 10https://gerrit.wikimedia.org/r/234519 (https://phabricator.wikimedia.org/T74001) (owner: 10Hashar) [13:49:22] (03PS2) 10JanZerebecki: Remove legacy definition [integration/config] - 10https://gerrit.wikimedia.org/r/234519 (https://phabricator.wikimedia.org/T74001) (owner: 10Hashar) [13:49:28] (03CR) 10JanZerebecki: [C: 032] Remove legacy definition [integration/config] - 10https://gerrit.wikimedia.org/r/234519 (https://phabricator.wikimedia.org/T74001) (owner: 10Hashar) [13:49:59] (03CR) 10JanZerebecki: [C: 032] Remove obsolete tools/fetch-mw-ext [integration/jenkins] - 10https://gerrit.wikimedia.org/r/234520 (https://phabricator.wikimedia.org/T74001) (owner: 10Hashar) [13:51:24] (03Merged) 10jenkins-bot: Remove legacy definition [integration/config] - 10https://gerrit.wikimedia.org/r/234519 (https://phabricator.wikimedia.org/T74001) (owner: 10Hashar) [13:51:26] (03Merged) 10jenkins-bot: Remove obsolete tools/fetch-mw-ext [integration/jenkins] - 10https://gerrit.wikimedia.org/r/234520 (https://phabricator.wikimedia.org/T74001) (owner: 10Hashar) [13:53:15] 10Continuous-Integration-Infrastructure, 6Labs, 10Tool-Labs: Don't know what to put in setup.py/requirements.txt to satisfy both dpkg-buildpackage and tox-flake8 - https://phabricator.wikimedia.org/T110445#1584057 (10hashar) The `mysql-connector-python` module is [[ http://dev.mysql.com/downloads/connector/p... [13:53:44] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Don't know what to put in setup.py/requirements.txt to satisfy both dpkg-buildpackage and tox-flake8 - https://phabricator.wikimedia.org/T110445#1584058 (10hashar) [13:56:29] (03CR) 10Hashar: "Used git-deploy to push that to gallium. Labs slave will catch up via puppet." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/234520 (https://phabricator.wikimedia.org/T74001) (owner: 10Hashar) [14:00:26] 10Deployment-Systems, 6operations: Remove lanthanum.eqiad.wmnet from Trebuchet redis - https://phabricator.wikimedia.org/T110677#1584069 (10hashar) 3NEW [14:04:29] jzerebecki: thank you :) [14:05:04] you're welcome. thx for the patches. [14:10:31] 5Continuous-Integration-Isolation, 6operations: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1584116 (10hashar) I noticed our OpenStack puppet manifests used the 'jessie-backports' pinning and found out it had the python mod... [14:20:04] 10Browser-Tests, 5Patch-For-Review: Fix problems reported by RuboCop in WikiLove - https://phabricator.wikimedia.org/T110672#1584145 (10zeljkofilipin) [14:30:31] 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1584184 (10Andrew) ok, I grabbed those two packages from debian.org and added them to jessie-wikimedia backpor... [14:52:54] 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1584271 (10Andrew) Here's the complete list of packages I added to our backports repo: python-cinderclient_1.... [15:17:11] 6RelEng-Admin, 10Gerrit-Migration, 15User-greg: Outline work (outcomes and outputs) of RelEng's Q2 Gerrit migration work - https://phabricator.wikimedia.org/T110623#1584311 (10greg) @Qgil: Maybe? I'd love support, most likely. I don't have a firm idea yet (part of this task's work). It might be worth a quick... [15:23:53] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 7Pywikibot-network, 7Pywikibot-tests: Nonce already used regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1584316 (10greg) From my understanding, this is an issue in OAuth not Beta Cluster, yes? I'm removing t... [15:25:53] 10Staging: Create staging-ms-fe* / staging-ms-be* (swift frontend/backend) - https://phabricator.wikimedia.org/T91553#1584319 (10greg) [15:26:52] 10Continuous-Integration-Infrastructure, 10Gitblit-Deprecate, 5Patch-For-Review, 7Technical-Debt: Remove CI dependencies on git.wikimedia.org - https://phabricator.wikimedia.org/T74001#1584324 (10greg) [15:41:59] 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1584374 (10hashar) a:3Andrew hashar@labnodepool1001:~$ apt-cache policy python-novaclient python-novaclient:... [15:42:23] 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review: python-openstackclient and python-openstackclient no more available in jessie backports - https://phabricator.wikimedia.org/T110656#1584376 (10hashar) 5Open>3Resolved Looks all fine now: ``` hashar@labnodepool1001:~$ apt-cache policy pyth... [15:42:52] andrewbogott: looks I forgot to request a MySQL database for Nodepool :-} [15:43:08] I used one on localhost [16:03:48] 5Continuous-Integration-Isolation: MySQL database for Nodepool - https://phabricator.wikimedia.org/T110693#1584439 (10hashar) 3NEW [16:17:54] 6RelEng-Admin: Fill out matrix of skills/people - https://phabricator.wikimedia.org/T107906#1584474 (10greg) a:5zeljkofilipin>3None [16:45:46] 10Continuous-Integration-Infrastructure: integration-slave-trusty-1016 should be put back online - https://phabricator.wikimedia.org/T110054#1584559 (10greg) a:5Krinkle>3hashar [16:46:07] 6RelEng-Admin: Fill out matrix of skills/people - https://phabricator.wikimedia.org/T107906#1584563 (10greg) (just unassigning since it was a group task) [16:49:39] man, arc is pretty neat [16:50:02] I could, mostly, move all of my todo list stuff into it, and have a nice cli [16:50:21] greg@x230 ~ % arc tasks 0 3494 09:50:11 Fri 28.08.2015 [16:50:24] T109657 Setup meeting with $people to discuss code hosting exception policy High Open [16:50:27] T107905 Create KPIs for #releng-201516-Q2 Normal Open [16:50:30] T110396 Investigate https://scrutinizer-ci.com/pricing WMF logo use Normal Open [16:50:33] you don't get to see the colors/etc, but it's nice :) [16:50:33] 10Continuous-Integration-Infrastructure, 6Discovery, 6operations, 7Elasticsearch, 5Patch-For-Review: elasticsearch 1.6.0 fails to start after reboot - https://phabricator.wikimedia.org/T109497#1584579 (10dduvall) >>! In T109497#1583546, @hashar wrote: > Or we can remove ElasticSearch from the Jenkins CI... [16:51:23] (03PS1) 10Krinkle: Add jobs for CLDRPluralRuleParser [integration/config] - 10https://gerrit.wikimedia.org/r/234569 [16:51:43] (03CR) 10Krinkle: [C: 032] Add jobs for CLDRPluralRuleParser [integration/config] - 10https://gerrit.wikimedia.org/r/234569 (owner: 10Krinkle) [16:52:24] (03Merged) 10jenkins-bot: Add jobs for CLDRPluralRuleParser [integration/config] - 10https://gerrit.wikimedia.org/r/234569 (owner: 10Krinkle) [16:53:34] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/234569 [16:53:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:56:21] 6Release-Engineering, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem 1.x (tracking) - https://phabricator.wikimedia.org/T94083#1584589 (10dduvall) [16:56:21] 10Browser-Tests, 6Collaboration-Team-Backlog, 10WikiLove, 5Patch-For-Review: Update WikiLove repository to mediawiki_selenium Ruby gem 1.x - https://phabricator.wikimedia.org/T99660#1584588 (10dduvall) 5Open>3Resolved [17:18:12] 10Deployment-Systems, 10Scap3: Scap3 should be able to deploy/rollback service config as part of deploy - https://phabricator.wikimedia.org/T109512#1584693 (10thcipriani) p:5Triage>3Normal [17:18:15] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-Requests: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1584695 (10MarcoAurelio) A temporary hack would be to add .* to the titleblacklist and then nobody but users... [17:18:55] 10Continuous-Integration-Config, 10VisualEditor, 7Documentation, 5Patch-For-Review: VisualEditor documentation examples on doc.wikimedia.org not working - https://phabricator.wikimedia.org/T109170#1584700 (10Jdforrester-WMF) [17:24:31] 6RelEng-Admin, 10Gerrit-Migration: Outline work (outcomes and outputs) of RelEng's Q2 Gerrit migration work - https://phabricator.wikimedia.org/T110623#1584729 (10greg) [17:32:57] (03PS2) 10Jforrester: PronunciationRecording: Remove legacy-mwext-jslint job [integration/config] - 10https://gerrit.wikimedia.org/r/233186 [17:33:10] legoktm: What went wrong with https://gerrit.wikimedia.org/r/#/c/233186/ ? [17:38:45] there's a special job that runs only on gate [17:38:46] hmm [17:49:45] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-Requests: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1584851 (10Krenair) I think you'd need to create a `closed-labs.dblist` and add `eswiki` as the contents. [17:50:04] 6Release-Engineering, 10MediaWiki-Debug-Logger, 6Reading-Infrastructure-Team, 10Wikimedia-Logstash, and 2 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1584853 (10bd808) a:3bd808 [17:52:01] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-Requests: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1584862 (10Glaisher) Can't someone in the developer group lock the db through the special page? http://es.wik... [17:52:17] So deployment-videoscaler01 is still running jobs-loop.sh [17:52:26] which is a sure sign that it is not setup right [17:52:34] fun times [17:52:47] Maybe we should just build a new box? [17:53:25] 10Beta-Cluster, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 10Wikimedia-Video: Videoscalers (tmh*) not working in Beta Cluster - https://phabricator.wikimedia.org/T90332#1584865 (10greg) [17:53:33] 10Beta-Cluster, 7HHVM: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1584868 (10greg) 3NEW [17:53:53] * greg-g already started editing tasks based on -operations discussion :) [17:53:54] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1584876 (10mmodell) >>! In T99096#1583708, @Krinkle wrote: > > Using the `/w/static.php` approach we... [17:54:31] lol. there are no useful puppet roles at all applied to that box [17:54:40] heh [17:54:53] It's setup to get a copy of MW pushed to it but nothing else really [17:54:58] 10Beta-Cluster, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 6operations, and 2 others: Convert tmh100[12] to HHVM and trusty - https://phabricator.wikimedia.org/T104747#1584883 (10greg) >>! In T104747#1568015, @brion wrote: > Adding beta-cluster project for fixing/updating TMH video scaler job r... [17:55:07] 10Beta-Cluster, 7HHVM: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1584868 (10greg) [17:55:09] 10Beta-Cluster, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 6operations, and 2 others: Convert tmh100[12] to HHVM and trusty - https://phabricator.wikimedia.org/T104747#1584885 (10greg) [17:55:56] there ya go, the tasks are all there now and all you need to do is fix it ;) [17:56:06] * greg-g managers [17:56:16] So I think what we should do is: 1) nuke deployment-videoscaler01; 2) build a new host; 3) apply the right puppet magic; 4) PROFIT! [17:56:30] (03CR) 10Legoktm: [C: 04-1] "It's still referenced in zuul/layout.yaml, which needs to be removed:" [integration/config] - 10https://gerrit.wikimedia.org/r/233186 (owner: 10Jforrester) [17:56:40] James_F: ^ found it [17:56:52] I guess we should check to see if any config is pointing things at deployment-videoscaler01 too [17:57:22] \o/ [17:57:40] let me know if I can help with anything other than testing it :D [17:58:04] *nod* I'll get the basics going and then come looking for you [17:58:21] awesome [17:58:35] looks like my new monitors for my home office finally arrived, brb packages :D [17:59:35] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-Requests: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1584890 (10Krenair) Yep, locked. Looks like it's stored the lock at `/data/project/upload7/wikipedia/es/lock_... [17:59:42] legoktm: Aha, thanks. [18:01:00] !log Deleted deployment-urldownloader.deployment-prep.eqiad.wmflabs [18:01:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:01:07] err no [18:01:33] !log Nope, I deleted deployment-videoscaler01 [18:01:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:02:06] heh [18:02:23] (03PS3) 10Jforrester: PronunciationRecording: Remove legacy-mwext-jslint job [integration/config] - 10https://gerrit.wikimedia.org/r/233186 [18:03:21] !log Building deployment-tmh01.deployment-prep.eqiad.wmflabs to replace deployment-videoscaler01 [18:03:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:03:53] 10Beta-Cluster, 7HHVM: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1584897 (10bd808) a:3bd808 [18:04:04] 10Beta-Cluster, 7HHVM: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1584868 (10bd808) p:5Triage>3High [18:05:32] Project beta-scap-eqiad build #67638: FAILURE in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/67638/ [18:17:42] !log Cleaned up some puppet groups for deployment-prep that no longer exist in ops/puppet [18:17:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:24:03] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-Requests: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1584925 (10MarcoAurelio) First time I see Special:LockDB working :D I'll see if I can work the closed-labs.db... [18:33:21] greg-g: Don't suppose you saw my question a couple of days ago about whether wmf22 was the last 1.26 branch? [19:25:59] James_F: I did, and I figured we'd do what we did for 1.25, ie: cut at wmf24 which should be about almost 2 months before [19:26:02] ...release [19:26:16] greg-g: Okie-dokie. [19:26:27] * James_F goes to create the wmf23,24 tags then. [19:26:36] * greg-g nods [19:26:38] ty [19:28:53] greg-g: So 30 September will be 1.27wmf1 deployment? Fun. [19:31:47] is that bad? [19:38:18] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 7Pywikibot-network, 7Pywikibot-tests: Nonce already used regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1585295 (10Tgr) Unless there is a bug in OAuth or PWB that we are unaware of, this is a memcached avail... [19:55:38] greg-g: No, sorry, non-sarcastic-fun. [20:04:07] (03PS1) 10Krinkle: doc: Add link to CLDRPluralRuleParser [integration/docroot] - 10https://gerrit.wikimedia.org/r/234668 [20:04:21] James_F: :) [20:04:35] (03CR) 10Krinkle: [C: 032] doc: Add link to CLDRPluralRuleParser [integration/docroot] - 10https://gerrit.wikimedia.org/r/234668 (owner: 10Krinkle) [20:04:45] (03Merged) 10jenkins-bot: doc: Add link to CLDRPluralRuleParser [integration/docroot] - 10https://gerrit.wikimedia.org/r/234668 (owner: 10Krinkle) [20:07:11] !log deployment-puppetmaster has only one cherry-pick; looks like maybe dcausse dropped the prior stack when working on Icc95ac8 [20:07:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:09:34] bd808: I think it's because we had to reset the puppetmaster after the labvirt1007, it had a lot of empty objects [20:10:08] *reset the puppetmaster puppet repo after the migration from labvirt1007. [20:10:19] ah. did we lose anything important? I can dig in logs to see what used to be there [20:12:06] I don't know of anything offhand that was needed, but I didn't dig there too often. [20:12:40] "Move web::sites to web::prod_sites; begin unification in new class"; "scap: Add co-master configuration"; "Prevent access to hidden directories" [20:12:40] marxarelli: any issues with https://gerrit.wikimedia.org/r/234394 ? [20:12:43] e.g. switching to firefox [20:13:39] bd808, those all sound important [20:13:49] I can put all those back from gerrit; I know where they are [20:14:07] jdlrobson: doesn't look like it [20:14:13] 'mwext-mw-selenium SUCCESS in 15m 18s' [20:14:15] :) [20:14:39] marxarelli: can you merge? i'm getting false errors for some reasons on certain patches such as https://gerrit.wikimedia.org/r/#/c/233635/ [20:15:27] !log restored 3 cherry picks that were lost when rebuilding the ops/puppet git repo [20:15:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:16:24] thx as always marxarelli|brb [20:17:13] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/234599 to setup new tmh01 as scap target [20:17:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:30:39] 5Continuous-Integration-Isolation: Wrap diskimage-builder for easy image creations - https://phabricator.wikimedia.org/T110733#1585471 (10hashar) 3NEW [20:31:14] (03PS7) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [20:31:23] (03CR) 10Hashar: "Now linked to T110733" [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) (owner: 10Hashar) [20:39:40] 5Continuous-Integration-Isolation: Write a diskimage-builder element to run puppet - https://phabricator.wikimedia.org/T110735#1585509 (10hashar) 3NEW [20:42:33] 5Continuous-Integration-Isolation: Figure out how to inject facts in the diskimage-builder chroot - https://phabricator.wikimedia.org/T110737#1585532 (10hashar) 3NEW [20:43:33] bd808: ah yeah sorry :-/ [20:43:41] bd808: puppetmaster git repo ended up being corrupted [20:43:55] I got it back to the old normal I think [20:44:05] bd808: so I more or less had to dish out the old repo. I looked at the reflog and dangling commit to recover some [20:44:15] but most patches ended up lost -:=/ [20:44:25] should have poked the QA list about it probabl [20:45:26] Luckily we had a pretty short list of picks compared to the old days [20:45:49] I looked up the names in the update log and found them in gerrit [20:47:13] bd808: guess I have been too lazy :-/ [20:47:38] shame on you for not fixing all the things ;) [20:47:41] bd808: had a question for you, seems Grafana can uses ElasticSearch as a backend, I am wondering whether we could grab metrics from logstash :D [20:48:34] or I will ask fillippo [20:49:08] "metrics" from logstash? [20:49:48] I noticed Grafana defines both graphite and elasticsearch datasources [20:49:57] but the later might just be to save the dashboardds [20:49:59] grafana can hit the logstash elasticsearch cluster (it stores its dashboards there) [20:50:43] yeah I though we could use Grafana as a frontend to build logstash backed metrics [20:50:56] to unify kibana / grafana somehow [20:51:27] what I wanted is on a Grafana dashboard to show a graph representing fatals and exceptions as stored in logstash [20:51:53] I push metrics to graphite for that now [20:52:00] well, not fatals yet [20:52:25] hashar: https://grafana.wikimedia.org/#/dashboard/db/bd808test [20:52:39] https://github.com/grafana/grafana/issues/1034 Elasticsearch as timeseries datasource :D [20:52:55] ahh [20:53:05] so logstash has an output plugin to statsd/graphite isn't it ? [20:53:36] yeah. it has both. those metrics are passed through to statsd [20:54:07] * hashar books a fancy place to have dinner with bd808 [20:54:43] damn it is lovely [20:55:01] this bit of config does it -- https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/logstash.pp#L137-L143 [20:55:50] I need to add tracking for hhvm errors too but haven't gotten around to it yet [20:56:02] whenever I have some spare time I have to dig into logstash and its filters [20:56:44] I've built some semi-fancy config -- https://github.com/wikimedia/operations-puppet/tree/production/files/logstash [20:56:52] yeah was looking at those [20:56:57] but gotta RTFM first [20:57:16] are the fields the same for all raws of a given index ? [20:57:24] or the number of fields can vary ? [20:57:30] they can vary [20:57:41] it's just json blobs [20:57:54] there are some kind of messages for which I would like to extract some part and populate as fields [20:58:05] was worried it would grow the list of fields out of control [20:58:07] \O/ [20:58:35] grok patterns pick things out -- https://github.com/wikimedia/operations-puppet/blob/production/files/logstash/filter-mediawiki.conf#L51-L58 [20:59:18] so that one is space separated [20:59:33] %{QS:feature} would put the first word in the field 'feature' is it? [21:01:02] yeah [21:01:02] I think QS means "quoted string regex" [21:05:16] bd808: thank you ! [21:05:24] off for the week-end have fun [21:05:30] o/ [21:08:44] (03PS6) 10Niedzielski: Run tests in apps-android-wikipedia-gradlew [integration/config] - 10https://gerrit.wikimedia.org/r/230260 (https://phabricator.wikimedia.org/T62720) [21:20:43] (03PS7) 10Niedzielski: Run tests in apps-android-wikipedia-gradlew [integration/config] - 10https://gerrit.wikimedia.org/r/230260 (https://phabricator.wikimedia.org/T62720) [21:20:50] (03CR) 10Niedzielski: "@Hashar, thanks! I believe I've captured the differences appropriately. jenkins-jobs.py test passes but I'm not sure how to deploy. ready " (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/230260 (https://phabricator.wikimedia.org/T62720) (owner: 10Niedzielski) [22:18:47] 10Beta-Cluster, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth, 7Pywikibot-network, 7Pywikibot-tests: Nonce already used regularly occurring on beta cluster - https://phabricator.wikimedia.org/T109173#1585754 (10jayvdb) We have are testing oauth on multiple sites (see task description), and we have only... [22:18:54] brion: I think I may have gotten it working! I see avconv and ffmpeg2theora running from the jobrunner on deployment-tmh01.deployment-prep now. [22:19:20] http://en.wikipedia.beta.wmflabs.org/wiki/Special:TimedMediaHandler shows progress! [22:19:47] woot [22:20:04] The missing trick was setting up the jobrunner there correctly. That had never been done for the new jobrunner system on the old box either [22:20:12] "2 running transcodes" :DDD [22:20:15] thanks bd808 ! [22:20:17] heh [22:20:32] they say the fastest code is code that never runs [22:21:09] the ultimate perf boost I guess :) [22:21:43] * brion waits patiently while the videos run, will check back in a bit to make sure they succeeded :D [22:22:29] that's not a really beefy box. I set it to have 2 jobrunners. we may find that it can only handle 1 [22:27:15] 10Beta-Cluster, 7HHVM, 5Patch-For-Review: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1585791 (10bd808) * Deleted deployment-videoscaler01 instance * Created deployment-tmh01 instance using trusty base image * Applied beta::deployaccess and role::mediawiki... [22:32:13] yeah i've got my local ones set to 2 threads too, should be fine for beta usage since it won't get many local video uploads, just for testing occasionally [22:33:47] wtf "[buffer @ 0x109d4e0] Invalid pixel format string '-1'" [22:34:23] bd808: ah is that still on the old VM build, just with updated jobrunner config? [22:34:51] it's a brand new trusty vm [22:35:56] hmmmm [22:35:57] So it has all the hhvm hotness and whatever new debs have been pushed for trusty [22:36:14] ah! [22:36:20] it's still using avconv i bet [22:36:30] looks like it, yeah [22:36:31] we updated ffmpeg and left the old avconv [22:36:47] ok lemme see ..... where the beta mediawiki-config is ..... :D [22:37:48] bd808: does it use the main mediawiki-config repo like the production servers, or does it have a separate one? [22:38:11] main repo with "labs" variant files [22:38:21] https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings-labs.php#L153-L159 [22:38:25] spiff [22:38:28] ok lemme prep a patch [22:42:25] bd808: https://gerrit.wikimedia.org/r/#/c/234681/ [22:43:02] sweet. Let's deploy that and see what happens [22:43:08] :D [22:43:23] and if that doesn't explode, we're probably good to go with updating the live servers and then doing the same config update in prod [22:46:02] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #782: FAILURE in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/782/ [22:46:56] wow. sync-file is crawling along [22:47:10] I wonder wtf is up with that [22:56:41] (03PS1) 10BryanDavis: Force use of IPv4 addresses with ssh and rsync [tools/scap] - 10https://gerrit.wikimedia.org/r/234687 [22:59:16] (03CR) 10BryanDavis: "I finally got one of the horribly slow sync-file runs that twentyafterfour has complained of before (06:05 for wmf-config/CommonSettings-l" [tools/scap] - 10https://gerrit.wikimedia.org/r/234687 (owner: 10BryanDavis) [23:19:37] bd808: ok looks like we might need to update the puppet roles to install ffmpeg [23:19:39] "/bin/bash: /usr/bin/ffmpeg: No such file or directory" [23:21:40] brion: do you know the package name that is needed? I see a bunch of *-ffmpeg5* packges installed already [23:22:07] bd808: pretty sure 'ffmpeg' is the right package [23:22:18] the -ffmpeg5* ones will be backing libraries [23:22:22] but we need the CLI tool here :D [23:22:27] *nod* I see that in apt but not on the instance [23:22:37] makes sense if the puppet config is still just pulling avconv [23:22:45] ffmpeg2theora will be pulling in the pudated libs but not the ffmpeg cli [23:22:54] * bd808 goes rummaging around in the puppet repo [23:23:19] thx :D [23:23:43] bd808: Did I tell you yet this week that you're amazing? :-) [23:24:44] brion: so... https://github.com/wikimedia/operations-puppet/blob/production/modules/mediawiki/manifests/packages/multimedia.pp#L6-L7 [23:25:06] it's purposefully installing libav instead of ffmpeg? [23:25:22] does it really need both? [23:25:27] heh [23:25:59] bd808: should be safe to install ffmpeg only *but* ... [23:26:00] that line is like a year old [23:26:02] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1585997 (10Krenair) >>! In T109157#1584925, @MarcoAurelio wrote: > First time I see Spec... [23:26:16] ... if that's the same config that's used in production we want to make sure we don't *uninstall* libav-tools yet [23:26:23] since the mediawiki config still using avconv in prod [23:26:27] until we deploy that :D [23:26:48] *fun times* [23:26:58] oh! [23:27:14] but prod are not on trusty yet ..... [23:27:22] ok i *think* it'll be sfae to change that to just install ffmpeg [23:27:41] the tmh boxes shouldn't be trusty yet, no [23:27:43] bd808: yeah originally i think we were going to just keep using avconv but other things are switching (back) to ffmpeg these days [23:28:08] and it was easier to backport ffmpeg than avconv, so we're switching back to ffmpeg ahead of debian doing so :D [23:29:01] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1586003 (10Krenair) Actually I have no idea why that works in our setup. We set `$wgRead... [23:29:21] That change was introduced more than a year ago by Ori -- https://gerrit.wikimedia.org/r/#/c/147992/ [23:29:34] so I can see how things may have changed [23:29:46] brion: Does that mean we should decline https://phabricator.wikimedia.org/T103335 ? [23:30:14] Or, rather Resolve as 'we're not going this way'? [23:30:14] James_F: the naming is awkward on that one :) [23:30:19] brion: :-) [23:30:31] James_F: i think resolve as completed, 'we have investigated' :D [23:31:02] brion: Après vous. [23:31:34] Ok, so you'd like to try basically reverting https://gerrit.wikimedia.org/r/#/c/147841/ so everything uses ffmpeg? [23:31:52] Or do we still want/need libav too? [23:31:55] (03CR) 1020after4: "It happens almost every time for me, specifically with sync-dir, I still don't understand why that would be slower than a full sync?" [tools/scap] - 10https://gerrit.wikimedia.org/r/234687 (owner: 10BryanDavis) [23:32:16] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1586007 (10brion) 5Open>3Resolved a:3brion Ok, the details of setting up the updated ffmpeg... [23:32:33] (03CR) 1020after4: [C: 031] Force use of IPv4 addresses with ssh and rsync [tools/scap] - 10https://gerrit.wikimedia.org/r/234687 (owner: 10BryanDavis) [23:32:46] bd808: assuming that puppet config won't get run on the *old* tmh boxes I think it's safe to just switch in ffmpeg yes [23:32:54] oh wait it checks for 'trusty' so yes that's safe :D [23:32:57] bd808: lemme patch it up [23:33:51] I can manually install packages too if you want to A/B test or something [23:33:54] bd808: agh i don't hvae puppet repo checked out. lemme pull it down, be a couple mins [23:34:08] * bd808 goes to find a beer [23:34:15] I'll be back :) [23:34:33] :D [23:34:45] (03CR) 10Faidon Liambotis: [C: 04-1] "The suspicion that IPv6 is to blame for this remains unsubstantiated." [tools/scap] - 10https://gerrit.wikimedia.org/r/234687 (owner: 10BryanDavis) [23:38:39] bd808: ok https://gerrit.wikimedia.org/r/#/c/234699/ looks right [23:40:01] Cool. I'll cherry-pick it for beta cluster and we can try again [23:40:56] !log Cherry-picked https://gerrit.wikimedia.org/r/#/c/234699/ [23:41:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:43:54] brion: ok $ which ffmpeg -- /usr/bin/ffmpeg [23:44:07] and I manually purged the libav pacakges [23:44:09] woot [23:44:12] lemme retest the files [23:45:57] well it's still running after a minute, that's a good sign [23:46:03] usually it fails out much faster :D [23:46:32] I see lots of ffmpeg procs running [23:52:09] still running. :) [23:52:19] testing video transcoding requires patience :D