[04:31:38] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #540: FAILURE in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/540/ [05:36:33] Yippee, build fixed! [05:36:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #520: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/520/ [07:44:34] good morning [07:48:45] 6RelEng-Admin: Fill out matrix of skills/people - https://phabricator.wikimedia.org/T107906#1566374 (10hashar) a:5hashar>3zeljkofilipin Filled my bits, now is @zeljkofilipin turn. [07:54:10] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#1566414 (10hashar) https://integration.wikimedia.org/ci/job/pywikibot-tests-beta-clu... [08:31:24] 6RelEng-Admin: Fill out matrix of skills/people - https://phabricator.wikimedia.org/T107906#1566502 (10zeljkofilipin) 5Open>3Resolved Looks like everybody filled the table. [08:33:32] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1566515 (10Joe) So what we really need, in my understanding, is to introduce canary hosts in all our clusters, and also allow to identify... [08:38:15] (03CR) 10Filippo Giunchedi: Add service deploy via scap (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/224374 (owner: 10Thcipriani) [08:42:05] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1566536 (10hashar) Maybe we can give it a shot on #beta-cluster ? Though I am not sure whether the... [08:46:58] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #701: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/701/ [09:10:57] (03PS1) 10Hashar: Experimental tox jobs for operations/software/puppet-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/233362 [09:12:53] (03CR) 10Hashar: [C: 032] Experimental tox jobs for operations/software/puppet-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/233362 (owner: 10Hashar) [09:14:35] (03CR) 10jenkins-bot: [V: 04-1] Experimental tox jobs for operations/software/puppet-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/233362 (owner: 10Hashar) [09:17:11] !log recreating Jenkins job mwext-PronunciationRecording-jslint (still triggered by Zuul somehow) [09:17:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:17:27] (03CR) 10Hashar: [C: 032] Experimental tox jobs for operations/software/puppet-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/233362 (owner: 10Hashar) [09:19:00] (03Merged) 10jenkins-bot: Experimental tox jobs for operations/software/puppet-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/233362 (owner: 10Hashar) [09:25:04] (03PS1) 10Hashar: puppet-compiler jobs are now on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/233365 [09:25:17] (03CR) 10Hashar: [C: 032] puppet-compiler jobs are now on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/233365 (owner: 10Hashar) [09:26:51] (03Merged) 10jenkins-bot: puppet-compiler jobs are now on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/233365 (owner: 10Hashar) [09:30:29] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1566672 (10hashar) >>! In T107172#1512747, @jhobs wrote: > I was on vacation last week and am thus just getting to this now. I have a meeting with the Zero... [10:07:33] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1566782 (10zeljkofilipin) @faidon I have left a comment in [[ https://gerrit.wikimedia.org/r/#/c/226898/ | gerrit ]], but I am not sure if you will noti... [10:25:16] 6RelEng-Admin, 15User-greg: Get RelEng team members greater access - https://phabricator.wikimedia.org/T107926#1566824 (10hashar) [10:25:22] 10Continuous-Integration-Config, 7Puppet: Move RuboCop job from experimental pipeline to the usual pipelines for operations/puppet - https://phabricator.wikimedia.org/T110019#1566829 (10zeljkofilipin) 3NEW [10:26:19] 6RelEng-Admin, 15User-greg: Get RelEng team members greater access - https://phabricator.wikimedia.org/T107926#1507451 (10hashar) I have copy pasted the EtherPad content to P1916 [10:28:21] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit: Unreviewed commits merged in gerrit - https://phabricator.wikimedia.org/T103396#1566841 (10hashar) @awight abandoned the fault changes in Gerrit. Not ideal but at least they are no more left open :-) [10:30:27] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1566855 (10hashar) 5Open>3declined Apparently not needed anymore right now. Feel free to reopen when some more wi... [10:32:16] 10Beta-Cluster, 7Puppet: Puppet failures on deployment-mx: can't find puppet://private/dkim/wikimedia.org-wiki-mail.key - https://phabricator.wikimedia.org/T87848#1566859 (10hashar) [10:32:40] 10Beta-Cluster, 7Puppet: Puppet failures on deployment-mx: can't find puppet://private/dkim/wikimedia.org-wiki-mail.key - https://phabricator.wikimedia.org/T87848#1000610 (10hashar) Still occurring. I have refreshed the puppet error output since we are now using `secret()`. [10:34:30] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Wikimedia code repository browser in Phabricator - https://phabricator.wikimedia.org/T752#1566863 (10hashar) [10:34:32] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1566861 (10hashar) 5Resolved>3Open Reopening. A manually maintained list of call signs / project name at https://www.mediawiki.org/wiki/Module:Callsigns is... [10:53:17] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1566902 (10Nemo_bis) What do you propose instead? Your suggested syntax `{{phab file |callsign=EFLW |file=composer.json }}` is unwanted. [12:42:27] 10Continuous-Integration-Config, 7Puppet: Move RuboCop job from experimental pipeline to the usual pipelines for operations/puppet - https://phabricator.wikimedia.org/T110019#1567100 (10hashar) [12:42:29] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1567099 (10hashar) [12:43:12] 10Continuous-Integration-Config, 7Puppet: Move RuboCop job from experimental pipeline to the usual pipelines for operations/puppet - https://phabricator.wikimedia.org/T110019#1566829 (10hashar) I guess you want to write some tutorial / instructions for ops so they can run rubocop locally then announce the new... [12:43:59] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1353796 (10hashar) @zeljkofilipin yup any third parties code in operations/puppet should be ignored by rubocop. [12:47:07] 10Browser-Tests, 6Release-Engineering, 5Testing Initiative 2015: Improve browser testing page with templates : Emphasize testing documentation on mediawiki.org - https://phabricator.wikimedia.org/T108110#1567115 (10hashar) [12:47:08] 6Release-Engineering, 6Team-Practices, 5Testing Initiative 2015: Follow up workshop & brown bag ideas from Testing: Where does it hurt? - https://phabricator.wikimedia.org/T108122#1567114 (10hashar) [12:47:20] 10Browser-Tests, 6Release-Engineering, 5Testing Initiative 2015: Improve browser testing page with templates : Emphasize testing documentation on mediawiki.org - https://phabricator.wikimedia.org/T108110#1512449 (10hashar) That is a child task of {T108122} [12:48:44] 6Release-Engineering, 6Team-Practices: Organize "testing: where does it hurt?" workshop for the end of July - https://phabricator.wikimedia.org/T102713#1567122 (10hashar) Seems the meeting followup is tracked by {T108122} [12:49:08] 6Release-Engineering, 6Team-Practices, 5Testing Initiative 2015: Follow up workshop & brown bag ideas from Testing: Where does it hurt? - https://phabricator.wikimedia.org/T108122#1567124 (10hashar) [12:49:50] 6Release-Engineering, 5Testing Initiative 2015: Ideas for future meetings like the "Testing: where does it hurt?" initiative - https://phabricator.wikimedia.org/T108125#1567128 (10hashar) [12:49:50] 6Release-Engineering, 6Team-Practices, 5Testing Initiative 2015: Follow up workshop & brown bag ideas from Testing: Where does it hurt? - https://phabricator.wikimedia.org/T108122#1512611 (10hashar) [12:52:33] 6Release-Engineering, 5Testing Initiative 2015: Include links to unit testing (Emphasize testing documentation on mediawiki.org) - https://phabricator.wikimedia.org/T108105#1567137 (10hashar) [12:52:34] 6Release-Engineering, 6Team-Practices, 5Testing Initiative 2015: Follow up workshop & brown bag ideas from Testing: Where does it hurt? - https://phabricator.wikimedia.org/T108122#1567136 (10hashar) [12:53:21] 6Release-Engineering, 6Team-Practices, 5Testing Initiative 2015, 7Tracking: Follow up workshop & brown bag ideas from Testing: Where does it hurt? - https://phabricator.wikimedia.org/T108122#1512611 (10hashar) [12:53:31] 6Release-Engineering, 5Testing Initiative 2015: Guides for initializing a test suite: unit testing & browser testing - https://phabricator.wikimedia.org/T108107#1567142 (10hashar) [12:53:32] 6Release-Engineering, 6Team-Practices, 5Testing Initiative 2015, 7Tracking: Follow up workshop & brown bag ideas from Testing: Where does it hurt? - https://phabricator.wikimedia.org/T108122#1512611 (10hashar) [12:57:37] 6Release-Engineering, 10Gitblit-Deprecate: IPSet repository is not set up in Diffusion or Maniphest - https://phabricator.wikimedia.org/T108903#1567158 (10hashar) @demon @20after4 would you mind looking at it? [12:57:59] 6Release-Engineering, 10Gitblit-Deprecate, 10at-ease: at-ease repository is not set up in Diffusion - https://phabricator.wikimedia.org/T108902#1567160 (10hashar) @demon @20after4 would you mind looking at it? [13:02:24] 6Release-Engineering, 6Team-Practices, 5Testing Initiative 2015: Talk to Greg and Arthur about Onboarding's role in establishing good testing practices/habits - https://phabricator.wikimedia.org/T108121#1567165 (10hashar) #releng does provide the infrastructure to automatically run tests on patch proposal an... [13:03:56] 6Release-Engineering, 10Gitblit-Deprecate: Update mediawiki.org templates to link to Diffusion, not gitblit - https://phabricator.wikimedia.org/T108864#1567169 (10hashar) For what it is worth, I reopened T101358 on the basis that https://www.mediawiki.org/wiki/Module:Callsigns is manually maintained and is goi... [13:06:45] 6Release-Engineering, 10Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1567170 (10hashar) I have added the point to the #releng weekly meeting agenda for August 25th. [13:11:56] 10Deployment-Systems, 10Adminbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#1567185 (10hashar) Potentially we could point Wikitech to the new SAL system ( https://tools.wmflabs.org/sal/ ) as implemented via {T63544}. It is ju... [13:12:12] !sal [13:12:12] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:12:14] !help [13:12:22] !del sal [13:12:26] wm-bot: help [13:12:28] @del sal [13:12:42] !sal del [13:12:42] Successfully removed sal [13:12:46] !sal is https://tools.wmflabs.org/sal/releng [13:12:46] Key was added [13:12:48] !sal [13:12:48] https://tools.wmflabs.org/sal/releng [13:12:50] \O/ [13:15:09] 10Deployment-Systems, 10Adminbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#1567190 (10hashar) I have updated `wm-bot` `!sal` helpers in the ops and releng channels: > #wikimedia-operations: > !sal > htt... [13:17:13] 10Deployment-Systems, 10Adminbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#1567194 (10Krenair) Don't we want log entries to be backed up to wikitech-static and not just be kept in labs though? [13:28:44] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1567220 (10zeljkofilipin) @hashar: any idea on which folders contain third party code? [13:36:29] 10Continuous-Integration-Config: Reduce nosetests output in integration/config - https://phabricator.wikimedia.org/T107683#1567242 (10hashar) On failure I guess nosetests dumps all the Zuul logs at DEBUG level. We can set in `setup.cfg` `logging-level = WARNING`, and nose would no more report DEBUG/INFO levels.... [13:36:48] 10Continuous-Integration-Config, 7Puppet: Move RuboCop job from experimental pipeline to the usual pipelines for operations/puppet - https://phabricator.wikimedia.org/T110019#1567243 (10hashar) p:5Triage>3Normal [13:41:17] 10Continuous-Integration-Config, 6Performance-Team, 7JavaScript: Run PhantomJS across set of Wikimedia wiki pages to ensure sane JavaScript - https://phabricator.wikimedia.org/T71519#1567266 (10hashar) @krinkle mentioned that PhantomJS is based on a completely outdated Webkit version that no more reflect the... [13:42:33] 10Continuous-Integration-Config, 10pywikibot-core: Check the style of the commit message - https://phabricator.wikimedia.org/T109119#1567270 (10hashar) [13:42:35] 10Continuous-Integration-Config, 7Tracking: common job for all repositories that runs structural linting - https://phabricator.wikimedia.org/T101511#1567269 (10hashar) [13:47:42] 10Continuous-Integration-Config, 10pywikibot-core: Check the style of the commit message - https://phabricator.wikimedia.org/T109119#1567279 (10hashar) Made a blocker of tracking task {T101511}. [13:52:28] 10Continuous-Integration-Config, 3Wikipedia-Android-App: java-mwapi: Add verification tasks when new patch gets uploaded - https://phabricator.wikimedia.org/T108735#1529020 (10hashar) It seems `apps/android/java-mwapi` uses gradle just like `apps/android/wikipedia` for which we have Jenkins jobs. We would nee... [13:52:47] 10Continuous-Integration-Config, 10pywikibot-core: Check the style of the commit message - https://phabricator.wikimedia.org/T109119#1567290 (10hashar) p:5Triage>3Normal [13:53:03] 10Continuous-Integration-Config: Create jenkins job for Extension:Mailgun - https://phabricator.wikimedia.org/T108166#1567293 (10hashar) p:5Triage>3Normal [13:54:06] 10Continuous-Integration-Config: Allow running old tests for old branches that don't support npm - https://phabricator.wikimedia.org/T106790#1567304 (10hashar) For old branches, maybe we can bulk add a default package.json that just runs the json lint and banana i18n checker. Would be good enough. [13:54:15] 10Continuous-Integration-Config: Allow running old tests for old branches that don't support npm - https://phabricator.wikimedia.org/T106790#1567305 (10hashar) p:5Triage>3Normal [13:56:55] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Epic, 7Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#1567315 (10hashar) [13:58:44] 10Continuous-Integration-Config, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch - https://phabricator.wikimedia.org/T98374#1567321 (10hashar) @dduvall added support for that to the CI infrastructure (T103039). Reading has an epic about it {T104315} and they now h... [13:58:51] 10Continuous-Integration-Config, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch - https://phabricator.wikimedia.org/T98374#1567327 (10hashar) p:5Triage>3Normal [13:59:27] 10Continuous-Integration-Config, 10MediaWiki-Documentation, 7Documentation: The Doxygen version in CI parses README files as garbled C. - https://phabricator.wikimedia.org/T106116#1567328 (10hashar) p:5Triage>3Low [14:00:42] 10Continuous-Integration-Config: Write a test to ensure all jobs in Zuul are defined in JJB - https://phabricator.wikimedia.org/T103847#1567334 (10hashar) p:5Triage>3Low [14:01:05] 10Continuous-Integration-Config: Create generic "tox" job that runs multiple environments - https://phabricator.wikimedia.org/T103251#1567338 (10hashar) p:5Triage>3Normal [14:01:25] 10Continuous-Integration-Config: Reduce nosetests output in integration/config - https://phabricator.wikimedia.org/T107683#1567341 (10hashar) p:5Triage>3Low [14:02:50] 10Continuous-Integration-Config: some jjb inline bash snippets might miss set -eu - https://phabricator.wikimedia.org/T106384#1567343 (10hashar) The Jenkins internal default is `set -xe`, if you use a shebang in the shell script that overrides the default though e.g.: ``` - shell: | #!/bin/bash -u ``` You... [14:02:57] 10Continuous-Integration-Config: some jjb inline bash snippets might miss set -eu - https://phabricator.wikimedia.org/T106384#1567344 (10hashar) p:5Triage>3Low [14:03:12] 10Continuous-Integration-Config: mediawiki-core-npm jobs process git submodules - https://phabricator.wikimedia.org/T103014#1567345 (10hashar) p:5Triage>3Low [14:03:39] 10Continuous-Integration-Config, 10Wikibase-DataModel-Services, 10Wikidata, 10Wikidata-Page-Banner, 5Patch-For-Review: test WikidataPageBanner its integration with Wikibase - https://phabricator.wikimedia.org/T106772#1567348 (10hashar) p:5Triage>3High [14:04:30] 10Continuous-Integration-Config, 10Pywikibot-i18n, 7I18n: Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n - https://phabricator.wikimedia.org/T85335#1567351 (10hashar) p:5High>3Normal [14:04:46] 10Continuous-Integration-Config, 6Release-Engineering, 7HHVM: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#1567352 (10hashar) [14:06:19] 10Continuous-Integration-Config, 6Performance-Team, 7JavaScript: Run PhantomJS across set of Wikimedia wiki pages to ensure sane JavaScript - https://phabricator.wikimedia.org/T71519#1567357 (10hashar) See also {T101497} [14:07:58] (03PS1) 10Hashar: Move labs/tools/extdist tox tests to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/233402 (https://phabricator.wikimedia.org/T102894) [14:08:09] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1567370 (10jhobs) It was, but seeing as the Zero Engineering team is soon to no longer exist, I'm not exactly sure where the responsibility of this landed.... [14:08:12] (03CR) 10Hashar: [C: 032] Move labs/tools/extdist tox tests to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/233402 (https://phabricator.wikimedia.org/T102894) (owner: 10Hashar) [14:09:48] (03Merged) 10jenkins-bot: Move labs/tools/extdist tox tests to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/233402 (https://phabricator.wikimedia.org/T102894) (owner: 10Hashar) [14:10:29] 10Continuous-Integration-Config, 10Labs-project-extdist, 5Patch-For-Review: Move labs/tools/extdist tox tests to jessie - https://phabricator.wikimedia.org/T102894#1567395 (10hashar) 5Open>3Resolved a:3hashar [14:18:34] 10Continuous-Integration-Config, 6Performance-Team, 7JavaScript: Run PhantomJS across set of Wikimedia wiki pages to ensure sane JavaScript - https://phabricator.wikimedia.org/T71519#1567419 (10Krinkle) >>! In T71519#1567266, @hashar wrote: > @krinkle mentioned that PhantomJS is based on a completely outdate... [14:19:46] 10Continuous-Integration-Infrastructure, 10MonoBook, 10Vector: Set up phpunit structure tests for MediaWiki skin repositories - https://phabricator.wikimedia.org/T68926#1567422 (10hashar) >>! In T68926#1276232, @Paladox wrote: > Should this be closed since they run jobs now. There is only linting jobs, no a... [14:20:55] 10Continuous-Integration-Config, 5Patch-For-Review: Remove mediawiki/extensions/DataTypes from CI - https://phabricator.wikimedia.org/T108759#1567433 (10hashar) [14:25:34] (03PS1) 10Hashar: Experimental 'npm' job for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/233408 (https://phabricator.wikimedia.org/T108779) [14:25:54] (03CR) 10Hashar: [C: 032] Experimental 'npm' job for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/233408 (https://phabricator.wikimedia.org/T108779) (owner: 10Hashar) [14:26:52] (03PS2) 10Hashar: Experimental 'npm' job for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/233408 (https://phabricator.wikimedia.org/T108779) [14:27:08] (03CR) 10Hashar: [C: 032] Experimental 'npm' job for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/233408 (https://phabricator.wikimedia.org/T108779) (owner: 10Hashar) [14:28:26] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #238: FAILURE in 25 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/238/ [14:28:47] 10Beta-Cluster: deployment-puppetmaster no more reachable via SSH - https://phabricator.wikimedia.org/T110040#1567483 (10hashar) 3NEW a:3hashar [14:31:10] 6Release-Engineering, 10Gitblit-Deprecate, 10at-ease: at-ease repository is not set up in Diffusion - https://phabricator.wikimedia.org/T108902#1567500 (10demon) Any repo created after I mass-created them awhile back is likely to be missing. Individual tasks to import missing ones are...less than useful...as... [14:31:12] 10Beta-Cluster: deployment-puppetmaster no more reachable via SSH - https://phabricator.wikimedia.org/T110040#1567501 (10hashar) Apparently unreachable via salt. Hard rebooting instance via the Horizon dashboard. [14:33:18] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1567506 (10demon) Creating a new repo and moving it solves nothing. Either this should be killed outright, or the task declined. It's hurting //no one// whe... [14:34:37] 6Release-Engineering, 10Gitblit-Deprecate: IPSet repository is not set up in Diffusion or Maniphest - https://phabricator.wikimedia.org/T108903#1567510 (10demon) Any repo created after I mass-created them awhile back is likely to be missing. Individual tasks to import missing ones are...less than useful...as I... [14:35:31] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011 went offline - https://phabricator.wikimedia.org/T110041#1567514 (10hashar) 3NEW [14:37:28] 10Beta-Cluster: deployment-puppetmaster no more reachable via SSH - https://phabricator.wikimedia.org/T110040#1567528 (10hashar) 5Open>3Resolved Reboot solved the issue. Haven't investigated the root cause though. [14:38:38] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011 went offline - https://phabricator.wikimedia.org/T110041#1567534 (10hashar) Can't SSH to it. Hard rebooting the instance via the Horizon dashboard. [14:39:29] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011 went offline - https://phabricator.wikimedia.org/T110041#1567539 (10hashar) Seems there is an issue with ElasticSearch: ``` * Starting Elasticsearch Server [80G [74G[[31mfail[39;49m] [74G[[31mfail[39;49m] [74G[[31mfail[39;49m] [74G[[3... [14:41:22] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011 went offline - https://phabricator.wikimedia.org/T110041#1567544 (10hashar) Went through Salt to kill the startup script: ``` root@integration-saltmaster:~# salt 'integration-slave-precise-1011*' cmd.run 'ps -A|grep elast' integration-slave-... [14:42:30] (03Merged) 10jenkins-bot: Experimental 'npm' job for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/233408 (https://phabricator.wikimedia.org/T108779) (owner: 10Hashar) [14:48:34] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011 went offline - https://phabricator.wikimedia.org/T110041#1567605 (10hashar) Upgraded the packages and I removed a stalled lock /var/lib/puppet/state/agent_catalog_run.lock from Aug 17th 15:06. Running puppet deadlocks on starting elasticsea... [14:49:00] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011 went offline - https://phabricator.wikimedia.org/T110041#1567607 (10hashar) 5Open>3Resolved a:3hashar The machine is back. The elasticsearch issue is {T109497} [15:03:35] 10Continuous-Integration-Infrastructure, 6Discovery, 7Elasticsearch, 5Patch-For-Review: elasticsearch 1.6.0 fails to start after reboot - https://phabricator.wikimedia.org/T109497#1567716 (10hashar) https://gerrit.wikimedia.org/r/233413 teach puppet to create `/var/run/elasticsearch` before starting the se... [15:06:12] 10Continuous-Integration-Infrastructure, 6Discovery, 7Elasticsearch, 5Patch-For-Review: elasticsearch 1.6.0 fails to start after reboot - https://phabricator.wikimedia.org/T109497#1567739 (10hashar) a:3hashar [15:06:37] !log upgrading packages on precise slaves [15:06:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:11:33] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011 and integration-slave-precise-1014 went offline - https://phabricator.wikimedia.org/T110041#1567751 (10hashar) [15:12:00] !log integration-slave-precise-1011 and integration-slave-precise-1014 went offline due to elasticsearch not starting properly. https://phabricator.wikimedia.org/T109497 [15:12:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:14:59] !log apt-get upgrade on gallium [15:15:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:15:25] 6RelEng-Admin: Fill out matrix of skills/people - https://phabricator.wikimedia.org/T107906#1567785 (10greg) Thanks all! Fair warning, we'll do another one mid-month in September, then another one mid-month in October, to have more "stuff that kinda looks like data but really shouldn't be treated as such". :) [15:16:35] !log apt-get upgrade all Trusty slaves [15:16:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:19:47] !log hard rebooting integration-slave-trusty-1014 [15:19:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:23:57] !log hard rebooting integration-slave-trusty-1017 [15:23:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:29:51] weee [15:35:20] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1567851 (10jhobs) Unless I'm misunderstanding something, we still support the Firefox OS Wikipedia mobile app, so I see no reason why it should be killed. [15:35:53] 10Deployment-Systems, 10Adminbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#1567852 (10greg) Let's think through the failure modes we're trying to guard against: **Bad things** 1) Production wikis all returning blank pages/HH... [15:37:29] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1567853 (10greg) >>! In T107172#1567370, @jhobs wrote: > the Zero Engineering team is soon to no longer exist, I'm not exactly sure where the responsibility... [15:40:43] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567858 (10hashar) 3NEW [15:41:32] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1567866 (10jhobs) That's why I pinged someone who might be able to tell us who //will// be responsible moving forward. [15:41:56] Krinkle: hello :-) if you are around can you check whether the slave https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1016/ can be bring back ? [15:42:06] was depooled with the message: Debugging w/ Krinkle and Ori. [15:43:18] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1567868 (10greg) :) Let's just make sure that is answered. [15:49:53] 10Continuous-Integration-Infrastructure, 6Discovery, 7Elasticsearch, 5Patch-For-Review: elasticsearch 1.6.0 fails to start after reboot - https://phabricator.wikimedia.org/T109497#1567869 (10hashar) a:5hashar>3None Applied a puppet workaround to force create the directory [15:51:43] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1567872 (10demon) >>! In T107172#1567851, @jhobs wrote: > Unless I'm misunderstanding something, we still support the Firefox OS Wikipedia mobile app, so I... [15:52:08] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567875 (10hashar) [15:52:49] 10Deployment-Systems, 10Adminbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#1567877 (10bd808) >>! In T46791#1567194, @Krenair wrote: > Don't we want log entries to be backed up to wikitech-static and not just be kept in labs t... [15:53:47] !sa [15:53:48] !sal [15:53:48] https://tools.wmflabs.org/sal/releng [15:55:23] 10Continuous-Integration-Infrastructure: integration-slave-trusty-1016 should be put back online - https://phabricator.wikimedia.org/T110054#1567895 (10hashar) 3NEW [15:57:07] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567906 (10hashar) @andrew do you have any spare time to look at them please ? :-} [15:57:31] 10Continuous-Integration-Infrastructure: integration-slave-trusty-1016 should be put back online - https://phabricator.wikimedia.org/T110054#1567908 (10Krinkle) 5Open>3Resolved a:3Krinkle [16:00:24] Krinkle: :-] [16:16:29] 10Continuous-Integration-Infrastructure, 6Multimedia, 6operations, 5Patch-For-Review: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1567947 (10brion) @hashar adding notes on T104747 for testing [16:19:00] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567958 (10Andrew) This is probably because they are hosted on labvirt1007 which no longer has space... [16:25:47] 10Continuous-Integration-Config, 3Wikipedia-Android-App: java-mwapi: Add verification tasks when new patch gets uploaded - https://phabricator.wikimedia.org/T108735#1567997 (10Niedzielski) @Hashar, welcome back! @Hashar, @BearND, my vote is to keep the jobs and concerns separate. Additionally, the apps/androi... [16:26:27] thcipriani: I wonder if a patch to put a class-per-file for scap would be ok. main.py is getting...unwieldy [16:26:44] Or at least break up main.py, if not class-per-file [16:27:00] break it up as needed! [16:27:23] It has grown quite a bit since I started it [16:27:34] and not had any big refactorings [16:27:51] I think breaking it out seems like a good idea: particularly the new service stuff which is kinda completely different. Only relies on scap/cli rather than abstractsync in main [16:30:07] 10Beta-Cluster, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 6operations, and 2 others: Convert tmh100[12] to HHVM and trusty - https://phabricator.wikimedia.org/T104747#1568015 (10brion) Adding beta-cluster project for fixing/updating TMH video scaler job runner for beta cluster... [16:31:27] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1568029 (10jhobs) I'll do my best to get confirmation on who should be responsible moving forward this afternoon, after which I'll update this task and decl... [16:34:48] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1568045 (10hashar) At worth I will recreate tomorrow morning Europe time :-) Thanks Andrew! [16:53:03] 10Continuous-Integration-Config, 3Wikipedia-Android-App: java-mwapi: Add verification tasks when new patch gets uploaded - https://phabricator.wikimedia.org/T108735#1568134 (10hashar) Can you list out the gradle commands / end point for each of the run you want? Ie `gradle clean lint` ? [17:06:48] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1568183 (10greg) Thanks @jhobs. [17:07:45] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1568185 (10GWicke) @joe, we probably want *both* a canary deploy *and* a rolling deploy in general. With RB, we tend to deploy to one node... [17:13:00] 10Continuous-Integration-Config, 3Wikipedia-Android-App: java-mwapi: Add verification tasks when new patch gets uploaded - https://phabricator.wikimedia.org/T108735#1568203 (10Niedzielski) @hashar build, test, and Checkstyle would be covered by `./gradlew clean checkstyle build`. We don't have any lint or cove... [17:29:02] thcipriani: hey! [17:29:12] thcipriani: I see that you have like, 7 restbase instances in staging [17:29:25] thcipriani: can I delete 01, 06, or 10? [17:29:28] yeah, 10, we're using them for restbase deployment testing [17:29:33] ah [17:29:39] so I'll leave them be for now, I guess [17:30:05] I'll look for other things to kill :) [17:30:16] kk, if there are others outside of palladium and staging-test-tin we can probably ditch them [17:30:52] thcipriani: how about staging-mc3? [17:30:55] mc3 could, yeah [17:30:56] there's an mc1 and mc2 [17:31:08] shall I delete it? [17:31:13] sure, go for it [17:31:16] thanks [17:31:31] I think that's all that we have on 1007 that can die [17:32:21] 10Deployment-Systems, 10Scap3: Scap3 check to monitor logstash and detect changes in error frequency - https://phabricator.wikimedia.org/T110068#1568286 (10mmodell) 3NEW a:3mmodell [17:32:22] done [17:32:23] yeah [17:32:24] although, staging-test-tin may already be dead. YuviPanda can you confirm that instance still exists? It's been down a while. [17:32:31] I can see it [17:32:48] hmm, can't get to it via ssh or salt, not responding to ping [17:32:58] I can reboot it from nova once this blows over [17:33:11] kk, thanks [17:33:35] thcipriani: actually can do now [17:33:47] done [17:33:47] awesome, thanks! [17:34:17] 10Deployment-Systems, 10Scap3: Scap3 check to monitor logstash and detect changes in error frequency - https://phabricator.wikimedia.org/T110068#1568301 (10mmodell) [17:34:17] 10Deployment-Systems, 10Scap3: Scap3 should support post-deploy checks - https://phabricator.wikimedia.org/T109515#1550805 (10mmodell) [17:34:56] 10Deployment-Systems, 10Scap3: Scap3 check to monitor logstash and detect changes in error frequency - https://phabricator.wikimedia.org/T110068#1568286 (10mmodell) [17:34:57] 10Deployment-Systems, 10Scap3: Scap3 should support post-deploy checks - https://phabricator.wikimedia.org/T109515#1568303 (10mmodell) [17:57:24] greg-g: Is there a phab team for QA specifically? hashar took releng off T89353, I wasn't sure if there's a better backlog to get that on. [17:58:26] 6Release-Engineering, 6Labs: Cleanup quaity-assurance labs project - https://phabricator.wikimedia.org/T108087#1568427 (10yuvipanda) Ok, deleting! [18:01:12] csteipp: either is fine, we don't have a hard and fast rule on everything being in #releng [18:02:31] 6Release-Engineering, 6Labs: Cleanup quaity-assurance labs project - https://phabricator.wikimedia.org/T108087#1568458 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Deleted! [18:19:01] (03PS1) 10Krinkle: Add 'npm' job for performance/WebPageTest repo [integration/config] - 10https://gerrit.wikimedia.org/r/233480 [18:19:25] (03CR) 10Krinkle: [C: 032] Add 'npm' job for performance/WebPageTest repo [integration/config] - 10https://gerrit.wikimedia.org/r/233480 (owner: 10Krinkle) [18:21:08] (03Merged) 10jenkins-bot: Add 'npm' job for performance/WebPageTest repo [integration/config] - 10https://gerrit.wikimedia.org/r/233480 (owner: 10Krinkle) [18:23:25] !log Running on Beta Cluster: foreachwiki populateContentModel.php --ns=all --table=page | tee populateContentModel_page_table_all_namespaces_all_wikis_2015-08-24.log [18:23:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:24:20] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/233480 [18:24:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:34:06] !log Completed on Beta Cluster: foreachwiki populateContentModel.php --ns=all --table=page | tee populateContentModel_page_table_all_namespaces_all_wikis_2015-08-24.log [18:34:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:44:38] aww, puppet be broke on staging-tin [18:44:40] Error: Failed to apply catalog: Could not find dependent Exec[fetch_mediawiki] for Git::Clone[operations/mediawiki-config] at /etc/puppet/modules/scap/manifests/master.pp:23 [18:44:52] Live hack in progress? [18:47:37] Hmm, no. [18:52:46] thcipriani: Ok, tin working...better. Still failing on your keyservice tho [18:53:33] ostriches: are you looking at staging-test-tin or staging-tin? [18:53:46] I ought to delete staging-tin: that one has been busted [18:53:55] staging-test-tin should be working [18:53:58] Ah, dur. [18:54:00] Yeah, let's. [18:55:23] Ok, staging-test-tin puppet running ok (last success was 4225m before I logged in) [18:58:24] heh, looks like it's got the same problem: probably need to update puppet on palladium [19:00:28] Yeah, I'll work on that. [19:00:52] yeah, looks like it's probably something to do with the recent patch :( [19:02:35] Restbase conflicts on pull -r [19:03:17] a8fe0771fd5f7ca49dff8560077e5bf5a5a47719 is the conflict with production [19:03:39] Ok, I'm going to rebase the rest of it, then try to cherry-pick you on top [19:04:03] 6Release-Engineering, 10Gitblit-Deprecate: Update mediawiki.org templates to link to Diffusion, not gitblit - https://phabricator.wikimedia.org/T108864#1568848 (10Nemo_bis) The module is certainly ugly; the solution is T106130. The actual overhead I can't measure, but it's mostly about running P726 every now a... [19:04:54] That didn't work either. [19:05:09] I can take a look, see if I can work out of my mess [19:06:24] I'm going to try and upstream some of the more trivial ones [19:06:40] eg: adb6dd5, a29dd09 [19:07:52] ok, I got it [19:08:00] ostriches: ^ [19:08:01] a29dd098b9649cb8683bff0992b53509b949db9b can be tossed entirely, actually [19:08:03] Already upstreamed [19:08:17] Just with a one-line whitespace difference, which is why rebasing kept it [19:08:46] Rebasing to abandon it from staging-palladium [19:08:53] the main rebase conflict that I saw was the lvs_realserver thing that I wrapped. [19:09:35] Yeah [19:15:13] thcipriani: staging-tin nuked [19:15:16] $confusion-- [19:15:33] ostriches: heh, cool. Looks like there's still an issue on test-tin though :( [19:15:58] which is probably my doing [19:22:33] Now just suck on the Error: Failed to apply catalog: Could not find dependent Exec[fetch_mediawiki] for Git::Clone[operations/mediawiki-config] at /etc/puppet/modules/scap/manifests/master.pp:23 [19:22:48] Which, I have a diff locally that fixed it temporarily, but I'm curious why that's broken [19:51:26] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1568981 (10hashar) [20:21:24] 10Continuous-Integration-Infrastructure: Track and graph mean time to merge - https://phabricator.wikimedia.org/T70114#1569043 (10hashar) The `resident_time` metric reported by Zuul is the time delta between the change entering a pipeline and it leaving it (regardless of success). So that is related. We have y... [20:24:34] 10Continuous-Integration-Infrastructure: Track and graph mean time to merge - https://phabricator.wikimedia.org/T70114#1569052 (10hashar) I am pretty sure the mean/lower/upper aggregate are wrong in Graphite. They seem to always use average for consolidation :-( [20:39:52] Tests are failing for me on integration-slave-trusty-1016 with -- Catchable fatal error: Argument 4 to mysqli::real_connect() must be of type ?string, bool given in /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/includes/db/DatabaseMysqli.php on line 106 [20:40:10] that's happening during the setup phase for the tests [20:40:20] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/28413/consoleFull [20:46:02] the hhvm build there doesn't match integration-slave-trusty-1015 [20:47:36] !log Ran sudo apt-get install hhvm hhvm-dev on integration-slave-trusty-1016 to get hhvm 3.6.5+dfsg1-1+wm3 [20:47:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:51:34] ostriches: did you track down the staging-test-tin puppet error? I was getting ready to test a puppet change and noticed that error is back (or never went away). [20:51:40] !log Restarted puppetmaster on integration-puppetmaster [20:51:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:52:02] thcipriani: Lemme apply my 1-line hack that "works" [20:53:07] oh boy [20:53:18] https://phabricator.wikimedia.org/P1920 [20:54:01] New error! [20:54:13] Error: Could not get latest version: getaddrinfo: Name or service not known [20:54:13] Error: /Stage[main]/Mediawiki::Scap/Package[scap]/ensure: change from d203e80bde20c38fada1f00c1458375c872ec268 to latest failed: Could not get latest version: getaddrinfo: Name or service not known [20:54:41] Trebuchet not setup? [20:55:12] should be, I've deployed scap a couple times in that cluster [20:55:47] granted, this is monkey-patched with P1920 [20:55:51] Could be...not right [20:55:57] I was just trying to fix the broken dep. [20:55:59] "Could not get latest version" would be a failure calling to the deploy server http interface for the repo tags [20:57:00] "Name of service not known" [20:57:02] *or [20:57:06] Wonder what the name was [20:57:13] (wonder if killing staging-tin broke it) [20:57:23] Yep [20:57:33] 9: role::deployment::salt_masters::deployment_server: staging-tin.eqiad.wmflabs [20:57:33] 10: role::trebuchet::deployment_server: staging-tin.eqiad.wmflabs [20:57:40] in staging's common.yaml [20:57:52] I think that's overridden in wikitech, though [20:57:55] (or should be) [20:57:57] Lemme check [20:58:18] Looks like it.... [20:58:38] Ah, former is. Latter isn't. [21:01:57] No dice [21:09:10] that's weird. I can run sudo salt-call deploy.fetch/deploy.checkout scap/scap ... [21:28:41] what's the beta equivalent for terbium? deployment-bastion? [21:34:23] yeah [21:34:39] -bastion is overloaded in beta with too many other non-bastiony things. [21:34:40] So yeah [21:51:29] !log migrating OAuth to metawiki on beta - T108648 [21:51:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:17:04] 10Beta-Cluster, 6operations, 7Database: Possible to run writes (e.g. UPDATE) on slave - https://phabricator.wikimedia.org/T110115#1569333 (10Mattflaschen) 3NEW [22:17:13] 10Beta-Cluster, 6operations, 7Database: Possible to run writes (e.g. UPDATE) on slave - https://phabricator.wikimedia.org/T110115#1569342 (10Mattflaschen) [22:23:25] matt_flaschen: `sql` connects to master. [22:23:30] Not a slave. [22:23:42] ostriches, legoktm told me it was recently changed to be slave. [22:23:47] Since when? [22:23:56] [Ops] sql --write [22:24:08] ostriches: https://gerrit.wikimedia.org/r/#/c/223365/ aug 20th [22:25:08] LOL at fawiki [22:25:09] meh, ok [22:25:56] 10Beta-Cluster, 6operations, 7Database: Possible to run writes (e.g. UPDATE) on slave - https://phabricator.wikimedia.org/T110115#1569373 (10Krenair) All the `sql` command was changed to do was connect to a server returned by `wfGetLB()->getServerName(wfGetLB()->getReaderIndex())` by default instead of `wfGe... [22:28:01] matt_flaschen, what update did you run exactly? [22:28:18] I have not yet figured out where logs of queries are kept [22:28:40] 10Beta-Cluster, 6operations, 7Database: Possible to run writes (e.g. UPDATE) on slave - https://phabricator.wikimedia.org/T110115#1569386 (10Mattflaschen) >>! In T110115#1569373, @Krenair wrote: > Isn't that a production analytics host? What does it have to do with anything? There are also slaves of the wik... [22:29:11] Krenair, https://phabricator.wikimedia.org/T109982#1569297 [22:41:58] matt_flaschen, oh, so if you repeated it against the master it... might be okay? [22:42:57] Yeah, that's what Roan suggested. [22:43:18] It's idempotent, so I don't think it should break, but I could be wrong. [22:44:47] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1569454 (10dr0ptp4kt) Just noting I've seen the discussion here. I'll need to talk with some people and get back to everyone. [22:54:17] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review, 7Pywikibot-tests: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#1569483 (10jayvdb) [22:57:06] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #240: FAILURE in 5.4 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/240/ [23:11:29] 6Release-Engineering, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1569523 (10Krenair) a:3dr0ptp4kt [23:19:56] (03PS1) 10BryanDavis: Run mw-install-mysql.sh with statement tracing [integration/jenkins] - 10https://gerrit.wikimedia.org/r/233633 [23:21:12] legoktm: ^ [23:21:27] I'm trying to figure out wtf is causing https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/28462/console [23:21:57] I can't reproduce locally; installer seems to work fine [23:22:04] bd808: do you want me to deploy it too? [23:22:25] If you have time; or remind me where to read how to do it [23:22:43] it's git deploy on tin [23:22:50] and the labs slaves are automatically updated by puppet [23:22:59] (03CR) 10Legoktm: [C: 031] Run mw-install-mysql.sh with statement tracing [integration/jenkins] - 10https://gerrit.wikimedia.org/r/233633 (owner: 10BryanDavis) [23:23:04] k. I can do the git deploy [23:23:08] thx [23:24:22] (03CR) 10BryanDavis: [C: 032] Run mw-install-mysql.sh with statement tracing [integration/jenkins] - 10https://gerrit.wikimedia.org/r/233633 (owner: 10BryanDavis) [23:31:57] (03Merged) 10jenkins-bot: Run mw-install-mysql.sh with statement tracing [integration/jenkins] - 10https://gerrit.wikimedia.org/r/233633 (owner: 10BryanDavis) [23:33:17] legoktm: apparently nobody has bothered with the trebuchet push from tin for a very long time [23:33:49] 14 patches merged that aren't there [23:36:41] !log Updated tin:/srv/deployment/integration/slave-scripts to a2cdf4f (Run mw-install-mysql.sh with statement tracing) and synced via trebuchet [23:36:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:37:28] :| [23:37:39] bd808: there are very few jobs that still run on gallium [23:39:13] Looked like the last run was 2015-04-07