[01:54:57] 10MediaWiki-Releasing, 10Release-Engineering-Team: Consider releasing MediaWiki 1.28.3 - https://phabricator.wikimedia.org/T179780#3736542 (10Reedy) [02:54:23] 10MediaWiki-Releasing, 10Release-Engineering-Team: Consider releasing MediaWiki 1.28.3 - https://phabricator.wikimedia.org/T179780#3736549 (10Reedy) [04:31:29] Where do I find the contents of the wmfreleng/operations-puppet container? It seems to be not using the root level tox.ini in operations/puppet which means it is running flake8 on files that should be ignored. -- https://gerrit.wikimedia.org/r/#/c/389408/ [04:33:26] found it [06:56:56] (03PS1) 10Robert Vogel: Added dependency "ExtJSBase" to "BlueSpiceFoundation" [integration/config] - 10https://gerrit.wikimedia.org/r/389412 [06:58:07] (03CR) 10jerkins-bot: [V: 04-1] Added dependency "ExtJSBase" to "BlueSpiceFoundation" [integration/config] - 10https://gerrit.wikimedia.org/r/389412 (owner: 10Robert Vogel) [08:03:28] 10Release-Engineering-Team (Watching / External), 10Scap, 10Operations: Scap: Standardize git version - https://phabricator.wikimedia.org/T179353#3736639 (10MoritzMuehlenhoff) p:05Triage>03Normal [08:19:01] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3736673 (10Osnard) The error seems to be gone. Probably just some mismatch during the decoupling process. [08:24:04] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3736695 (10Osnard) @Paladox, I've got another issue: In BlueSpice Version 3 we wanted to split the ExtJS Application Framework from the BlueSpiceFoundation (BSF)... [09:00:00] (03CR) 10Dzahn: "sorry, i'm not the right person for "integration/composer", maybe ping Hashar about the status of this?" [integration/composer] - 10https://gerrit.wikimedia.org/r/340791 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [09:00:25] (03CR) 10Dzahn: "just added addshore to get this going but it's been 3 months so dunno" [integration/composer] - 10https://gerrit.wikimedia.org/r/340791 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [09:02:03] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:08:04] (03CR) 10Hashar: [C: 032] Archive the ThemeDesigner extension [integration/config] - 10https://gerrit.wikimedia.org/r/389292 (https://phabricator.wikimedia.org/T178754) (owner: 10Umherirrender) [10:09:54] (03Merged) 10jenkins-bot: Archive the ThemeDesigner extension [integration/config] - 10https://gerrit.wikimedia.org/r/389292 (https://phabricator.wikimedia.org/T178754) (owner: 10Umherirrender) [10:27:17] (03CR) 10Hashar: [C: 032] Add HAWelcome extension [integration/config] - 10https://gerrit.wikimedia.org/r/389330 (owner: 10Umherirrender) [10:29:06] (03Merged) 10jenkins-bot: Add HAWelcome extension [integration/config] - 10https://gerrit.wikimedia.org/r/389330 (owner: 10Umherirrender) [10:33:15] (03CR) 10Hashar: "Seems easy to make it voting:" [integration/config] - 10https://gerrit.wikimedia.org/r/389330 (owner: 10Umherirrender) [10:43:32] (03Draft1) 10MarcoAurelio: Archive the VectorBeta extension [integration/config] - 10https://gerrit.wikimedia.org/r/389444 (https://phabricator.wikimedia.org/T178932) [10:43:36] (03PS2) 10MarcoAurelio: Archive the VectorBeta extension [integration/config] - 10https://gerrit.wikimedia.org/r/389444 (https://phabricator.wikimedia.org/T178932) [10:46:47] (03CR) 10jerkins-bot: [V: 04-1] Archive the VectorBeta extension [integration/config] - 10https://gerrit.wikimedia.org/r/389444 (https://phabricator.wikimedia.org/T178932) (owner: 10MarcoAurelio) [10:47:02] (03CR) 10MarcoAurelio: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/389444 (https://phabricator.wikimedia.org/T178932) (owner: 10MarcoAurelio) [10:51:43] (03CR) 10Hashar: [C: 04-1] "Gotta clean it up from the Zuul parameters as well:" [integration/config] - 10https://gerrit.wikimedia.org/r/389444 (https://phabricator.wikimedia.org/T178932) (owner: 10MarcoAurelio) [10:56:16] jan_drewniak: good morning. I am grabbing a coffee etc :] [10:58:38] (03PS3) 10MarcoAurelio: Archive the VectorBeta extension [integration/config] - 10https://gerrit.wikimedia.org/r/389444 (https://phabricator.wikimedia.org/T178932) [10:59:05] (03CR) 10MarcoAurelio: "> Gotta clean it up from the Zuul parameters as well:" [integration/config] - 10https://gerrit.wikimedia.org/r/389444 (https://phabricator.wikimedia.org/T178932) (owner: 10MarcoAurelio) [11:01:28] hashar: yes, just got my coffee as well :) [11:02:32] jan_drewniak: I am around :) [11:27:03] (03Draft1) 10MarcoAurelio: Archive the BayesianFilter extension [integration/config] - 10https://gerrit.wikimedia.org/r/389463 (https://phabricator.wikimedia.org/T178936) [11:27:06] (03PS2) 10MarcoAurelio: Archive the BayesianFilter extension [integration/config] - 10https://gerrit.wikimedia.org/r/389463 (https://phabricator.wikimedia.org/T178936) [11:32:33] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:33:56] no_justification great news upstream are finally fixing the external id bug in 2.13 :). [11:34:05] we wont need a customised patch. [11:38:42] https://gerrit-review.googlesource.com/#/c/gerrit/+/138775/ [11:40:54] * paladox updates the look for plugin docs in gerrit :) [11:58:29] (03PS1) 10Jdrewniak: [WIP] JJB for building wikimedia portals [integration/config] - 10https://gerrit.wikimedia.org/r/389468 (https://phabricator.wikimedia.org/T179694) [12:07:32] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [12:22:40] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #578: 04FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/578/ [13:28:43] 10Release-Engineering-Team (Kanban): pws is unusuable on releng-secrets choking on a key - https://phabricator.wikimedia.org/T179824#3737242 (10hashar) [13:49:24] 10Release-Engineering-Team (Kanban): pws is unusuable on releng-secrets choking on a key - https://phabricator.wikimedia.org/T179824#3737296 (10hashar) Looks like `gpg --quick-set-expire 1DE8155880C174349E6511971F648D4855B8CF0E 1y` only updates the master key but skips sub keys: ``` $ gpg --list-key --verbose -... [14:28:03] (03CR) 10Hashar: [C: 032] Archive the BayesianFilter extension [integration/config] - 10https://gerrit.wikimedia.org/r/389463 (https://phabricator.wikimedia.org/T178936) (owner: 10MarcoAurelio) [14:29:14] (03Merged) 10jenkins-bot: Archive the BayesianFilter extension [integration/config] - 10https://gerrit.wikimedia.org/r/389463 (https://phabricator.wikimedia.org/T178936) (owner: 10MarcoAurelio) [14:33:47] Yippee, build fixed! [14:33:47] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #569: 09FIXED in 1 min 46 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/569/ [14:43:47] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<100.00%) [14:55:49] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3737483 (10phuedx) [15:08:08] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3737501 (10zeljkofilipin) [15:17:14] 10Continuous-Integration-Infrastructure (shipyard): When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747#3737509 (10hashar) Based on `docker ps --no-trunc`, there are containers still running for the images: | COUNT | IMAGE | COMMAND... [15:34:04] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [15:46:24] 10Release-Engineering-Team (Kanban): pws is unusuable on releng-secrets choking on a key - https://phabricator.wikimedia.org/T179824#3737563 (10thcipriani) Seems like he probably needs to extend the signing and encryption subkeys as you mention. FWIW, I've never used the --quick-set-expire command, I've only ev... [15:48:42] (03PS1) 10Umherirrender: Use voting jobs for HAWelcome [integration/config] - 10https://gerrit.wikimedia.org/r/389504 [15:53:41] (03PS3) 10Thcipriani: Add service-pipeline-test-only experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/388546 (https://phabricator.wikimedia.org/T177954) [15:55:44] (03CR) 10Thcipriani: [C: 032] Add service-pipeline-test-only experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/388546 (https://phabricator.wikimedia.org/T177954) (owner: 10Thcipriani) [15:56:53] (03Merged) 10jenkins-bot: Add service-pipeline-test-only experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/388546 (https://phabricator.wikimedia.org/T177954) (owner: 10Thcipriani) [15:59:48] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: pws is unusuable on releng-secrets choking on a key - https://phabricator.wikimedia.org/T179824#3737585 (10zeljkofilipin) a:03zeljkofilipin [16:00:03] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: pws is unusuable on releng-secrets choking on a key - https://phabricator.wikimedia.org/T179824#3737242 (10zeljkofilipin) p:05Triage>03Normal [16:02:35] !log Reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/388546/ and https://gerrit.wikimedia.org/r/#/c/389463/ [16:02:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:17:29] (03PS2) 10Paladox: Added dependency "ExtJSBase" to "BlueSpiceFoundation" [integration/config] - 10https://gerrit.wikimedia.org/r/389412 (owner: 10Robert Vogel) [16:18:52] (03CR) 10jerkins-bot: [V: 04-1] Added dependency "ExtJSBase" to "BlueSpiceFoundation" [integration/config] - 10https://gerrit.wikimedia.org/r/389412 (owner: 10Robert Vogel) [16:25:01] (03PS3) 10Paladox: Added dependency "ExtJSBase" to "BlueSpiceFoundation" [integration/config] - 10https://gerrit.wikimedia.org/r/389412 (owner: 10Robert Vogel) [16:26:30] (03CR) 10jerkins-bot: [V: 04-1] Added dependency "ExtJSBase" to "BlueSpiceFoundation" [integration/config] - 10https://gerrit.wikimedia.org/r/389412 (owner: 10Robert Vogel) [16:27:19] (03PS4) 10Paladox: Added dependency "ExtJSBase" to "BlueSpiceFoundation" [integration/config] - 10https://gerrit.wikimedia.org/r/389412 (owner: 10Robert Vogel) [16:30:06] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3737643 (10Paladox) @Osnard Oh sorry for late reply. This https://gerrit.wikimedia.org/r/#/c/389412/ is required to be merged for this https://gerrit.wikimedia.o... [16:48:09] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3737724 (10zeljkofilipin) [16:50:50] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<22.22%) [17:05:20] (03CR) 10EBernhardson: "I just booted a fresh debian stretch instance on labs, installed docker, built the tox-pyspark container and ran it. Didn't have any issue" [integration/config] - 10https://gerrit.wikimedia.org/r/387658 (owner: 10EBernhardson) [17:27:29] 10Beta-Cluster-Infrastructure, 10Multimedia: Reimage deployment-tmh01 with Debian Jessie - https://phabricator.wikimedia.org/T174477#3737836 (10MoritzMuehlenhoff) I doubt that system is used at all, there's deployment-videoscaler01, which I have been using to test stretch compat. I think we can simply remove d... [17:29:15] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 3 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3737841 (10zeljkofilipin) [17:31:55] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10Deployments, 10WorkType-NewFunctionality: Scap3 submodule space issues - https://phabricator.wikimedia.org/T137124#3737866 (10mmodell) Assuming that {T179013} is resolved then we should be able to bring {D826} and friends to prod... [17:32:50] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10scap2, 10Patch-For-Review: Eliminate symlinks in mediawiki-config (as much as possible) - https://phabricator.wikimedia.org/T126306#3737872 (10mmodell) a:05mmodell>03demon Chad's the one who's leading the way on this task so... [17:34:11] 10Release-Engineering-Team (Kanban), 10Phabricator: Allow users to restrict View Policy when creating Pastes - https://phabricator.wikimedia.org/T145749#3737893 (10mmodell) I guess I'll revert that change for now, pending some consensus on {T145832} [17:45:53] PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<40.00%) [17:46:57] (03CR) 10EBernhardson: "New update:" [integration/config] - 10https://gerrit.wikimedia.org/r/387658 (owner: 10EBernhardson) [17:47:10] (03CR) 10EBernhardson: [C: 031] Run search/mjolnir pyspark tests via tox [integration/config] - 10https://gerrit.wikimedia.org/r/387658 (owner: 10EBernhardson) [17:48:46] (03CR) 10EBernhardson: [C: 031] "I also found the appropriate way to configure jvm logs, those now go to /log/pyspark.jvm.log. By default mjolnir sets the log level to WA" [integration/config] - 10https://gerrit.wikimedia.org/r/387658 (owner: 10EBernhardson) [17:56:41] 10MediaWiki-Releasing, 10MW-1.29-release-notes, 10Patch-For-Review: Include release extensions/skins/vendor as submodules of core - https://phabricator.wikimedia.org/T137564#3738066 (10greg) [18:02:30] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2): Scap failing to rewrite submodule urls in beta - https://phabricator.wikimedia.org/T179013#3738075 (10mmodell) @awight: Can you confirm whether this is resolved now that rMSCA1c8be017 is on beta? [18:03:28] 10Release-Engineering-Team (Kanban), 10Phabricator: Allow users to restrict View Policy when creating Pastes - https://phabricator.wikimedia.org/T145749#3738077 (10mmodell) 05Open>03Resolved ok, paste is unlocked for all users. [18:20:49] 10Release-Engineering-Team (Kanban), 10Mathoid, 10Release Pipeline, 10Patch-For-Review: Add experimental blubber test build/run to mathoid jenkins test pipeline - https://phabricator.wikimedia.org/T177954#3738119 (10dduvall) a:05dduvall>03thcipriani [18:24:30] thcipriani: Broken record reporting for reporting for duty! I’m hitting more git glitches trying to scap things… Would you prefer this in a task? [18:29:35] (03CR) 10Jdlrobson: [C: 031] "We dont know how to test this but this looks sane." [integration/config] - 10https://gerrit.wikimedia.org/r/387653 (https://phabricator.wikimedia.org/T179425) (owner: 10Legoktm) [18:29:51] 10Continuous-Integration-Config, 10Page-Previews, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking), 10Unplanned-Sprint-Work: mediawiki/extensions/Popups runs qunit in normal "npm test" - https://phabricator.wikimedia.org/T179425#3738186 (10Jdlrobson) [18:38:06] 10Release-Engineering-Team (Kanban), 10Mathoid, 10Release Pipeline, 10Patch-For-Review: Add experimental blubber test build/run to mathoid jenkins test pipeline - https://phabricator.wikimedia.org/T177954#3738208 (10dduvall) a:05thcipriani>03dduvall [18:38:55] (03CR) 10Thcipriani: "@Giuseppe Lavagetto clarification from the SSD meeting last week: IIRC you said *not* to work on reviewing this yet because you were going" [integration/config] - 10https://gerrit.wikimedia.org/r/385996 (owner: 10Giuseppe Lavagetto) [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:12:15] 10Release-Engineering-Team, 10GitHub-Mirrors: Audit @wikimedia GitHub org access (2017) - https://phabricator.wikimedia.org/T179462#3725495 (10demon) - If you don't use your owner permissions, we should remove you. I'm inclined to JFDI here. - The staff group is useless (it's never complete, and gives a false... [19:22:14] (03PS6) 10EBernhardson: Run search/mjolnir pyspark tests via tox [integration/config] - 10https://gerrit.wikimedia.org/r/387658 [19:22:49] (03CR) 10EBernhardson: [C: 031] Run search/mjolnir pyspark tests via tox [integration/config] - 10https://gerrit.wikimedia.org/r/387658 (owner: 10EBernhardson) [19:28:51] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: expired gpg key preventing new entries to releng pws - https://phabricator.wikimedia.org/T179824#3738408 (10greg) [19:29:03] RECOVERY - Free space - all mounts on deployment-kafka01 is OK: OK: All targets OK [19:43:46] RECOVERY - Free space - all mounts on deployment-eventlog02 is OK: OK: All targets OK [20:13:37] 10Gitblit-Deprecate, 10Epic, 10MW-1.30-release-notes (WMF-deploy-2017-07-18_(1.30.0-wmf.10)), 10MW-1.31-release-notes (WMF-deploy-2017-11-07 (1.31.0-wmf.7)), 10Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3738535 (10Umherirrender) >>! In T... [20:35:56] !log deploy ores:93e8846 in beta cluster [20:35:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:36:45] https://www.irccloud.com/pastebin/aVkJJuka/ [20:37:03] I get this when deploying, is it a known issue? [20:37:57] awight: ^ [20:39:13] Amir1: I haven’t seen that before [20:39:23] is it "scap deploy" right? [20:39:39] I’ve been using the syntax “scap deploy -v ‘MSG HERE’" [20:39:55] looks like that became a required param and someone forgot to send us the memo [20:40:27] Amir1: btw real tin has the CELERY_4 branch checked out, so make sure you checkout master. [20:41:31] awight: sure [20:41:37] twentyafterfour: thcipriani ^ [20:42:27] I think a new patch must've landed for scap, looking [20:43:13] yeah https://phabricator.wikimedia.org/D866 [20:43:29] twentyafterfour: no_justification looks like D866 may have some issues ^ [20:43:30] D866: Use sh library to wrap git commands. - https://phabricator.wikimedia.org/D866 [20:48:53] thcipriani: so what should we do now? Can I deploy in prod? [20:49:07] yeah, prod is fine. [20:50:03] I'll revert for now if no one else is around to unblock beta deployments. [20:50:54] Thanks [20:51:04] I also think you need to move out of nosetests [20:51:39] yeah, I saw that email thread that went around, pytest is the new thing that's supposed to mostly be compatible, right? [20:52:16] yeah, I've done it before for several project [20:52:18] *projects [20:52:22] I can do it for you [20:52:29] will make a patch [20:53:13] that would be awesome! [20:54:40] you helped us a lot, I owe you way more [20:56:14] ok, so I landed the revert, once this job completes: https://integration.wikimedia.org/ci/job/phabricator-jessie-commits/ [20:56:24] it will trigger: https://integration.wikimedia.org/ci/job/phabricator-jessie-debs/ [20:57:16] and that should update the scap .deb on beta cluster. You can run: sudo apt-get update; sudo apt-get install scap on deployment-tin and any targets and you should get the latest deb that will let you deploy. Or just wait for puppet to run on all nodes (~1 hr) [20:59:50] Amir1: ok, I updated deployment-tin and deployment-sca03 [21:00:08] should be able to deploy/back to regular scap problems :) [21:07:48] PROBLEM - Puppet errors on deployment-eventlog02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:09:32] 10Release-Engineering-Team, 10Scap: scap deploy broken for Trusty targets - https://phabricator.wikimedia.org/T179869#3738783 (10Ottomata) [21:12:20] thcipriani: Thanks, I'm deploying prod atm [21:12:29] okie doke [21:21:50] 10Release-Engineering-Team, 10Scap: scap deploy broken for Trusty targets - https://phabricator.wikimedia.org/T179869#3738817 (10greg) [21:21:52] 10Release-Engineering-Team (Watching / External), 10Scap, 10Operations: Scap: Standardize git version - https://phabricator.wikimedia.org/T179353#3738820 (10greg) [21:28:25] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Document current Blubber types and functions - https://phabricator.wikimedia.org/T168000#3738841 (10dduvall) 05Open>03Resolved [21:37:42] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T174361#3738876 (10BBlack) [21:37:51] 10Release-Engineering-Team (Watching / External), 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3738870 (10BBlack) 05Open>03stalled p:05High>03Normal The timeout changes above... [21:38:50] RECOVERY - Free space - all mounts on deployment-sca03 is OK: OK: All targets OK [21:42:46] RECOVERY - Puppet errors on deployment-eventlog02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:29:29] !log killed stuck npm Docker containers on integration-slave-docker-1002 (due to T176747 ). Pooled the instance back, the slowness it experienced is probably not related to labvirt CPU usage ( T179378 ) [22:29:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:29:35] T176747: When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747 [22:29:35] T179378: some labvirt servers are at full CPU capacity - https://phabricator.wikimedia.org/T179378 [22:32:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Multimedia, 10Services (watching): mediawiki-config changes not deployed automatically to deployment-videoscaler01 - https://phabricator.wikimedia.org/T179688#3733002 (10greg) We talked briefly about this in our team meeting today; this is an ove... [22:35:38] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3739048 (10Paladox) @Hashar without pinning the other deps, zuul works for me at least on jessie. We wont need to generate zuul for trusty and precise now... [22:35:48] (03PS12) 10Paladox: Pin GitPython to >=0.3.3,<2.1.2 [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/388624 (https://phabricator.wikimedia.org/T158243) [22:36:08] I would love to know how to filter logstash using a host wildcard. Or generally, how to modify a “phrase” search term to use a regex. [22:37:16] awight: i think its standard regez [22:37:20] Regex* [22:37:36] Zppix: Do you have an example of how to do that in logstash? [22:37:48] That’s the part that’s bugging me [22:37:48] No [22:43:55] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Multimedia, 10Patch-For-Review, and 2 others: mediawiki-config changes not deployed automatically to deployment-videoscaler01 - https://phabricator.wikimedia.org/T179688#3739138 (10greg) p:05Triage>03Normal a:03greg [22:44:36] (03CR) 10Hashar: [C: 031] "> when the server reports a Content-Length header but the connection completes with a different amount of content than reported. Is it pos" [integration/config] - 10https://gerrit.wikimedia.org/r/387658 (owner: 10EBernhardson) [22:51:10] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Platform-Team, 10Wikidata: What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor? - https://phabricator.wikimedia.org/T179663#3739155 (10greg) Adding #mediawiki-platform-team and @b... [22:53:40] https://phabricator.wikimedia.org/D871 [22:53:52] thcipriani: ^ [22:54:14] cc Amir1 [22:54:26] cc awight [22:55:34] twentyafterfour: I rolled back, too, FYI. Didn't take a dive into what was happening, just wanted to unblock beta deployments. [22:55:47] thcipriani: oh [22:55:48] 10MediaWiki-Releasing, 10Release-Engineering-Team: Removal of EOL release branches broke SemanticMediaWiki CI - https://phabricator.wikimedia.org/T179754#3735024 (10greg) tl;dr: Sorry about the breakage! We did not mean to inflict that pain on you (hard for us to know as it's a CI outside of our CI). Please do... [23:26:32] Project beta-scap-eqiad build #180817: 04FAILURE in 2 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/180817/ [23:27:22] sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-tin.deployment-prep.eqiad.wmflabs::common/wikiversions*.{json,php} /srv/mediawiki on deployment-videoscaler01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. [23:27:47] muther [23:28:09] keyholder something or other for that host needs to be done? [23:28:39] no, manual host key verification [23:28:43] :-/ [23:28:46] should be fixed now [23:28:56] ty [23:28:59] Are you sure you want to continue connecting (yes/no)? yes [23:29:01] Warning: Permanently added 'deployment-videoscaler01.deployment-prep.eqiad.wmflabs,10.68.19.130' (ECDSA) to the list of known hosts. [23:29:06] right right [23:36:31] Project beta-scap-eqiad build #180818: 04STILL FAILING in 2 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/180818/ [23:39:00] twentyafterfour: ^ :( [23:46:34] Project beta-scap-eqiad build #180819: 04STILL FAILING in 2 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/180819/ [23:49:10] should be fixed now [23:49:13] !log ssh-keyscan deployment-videoscaler01.deployment-prep.eqiad.wmflabs >> /etc/ssh/ssh_known_hosts [23:49:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL