[00:15:07] PROBLEM - Host deployment-bastion is DOWN: CRITICAL - Host Unreachable (10.68.16.58) [01:51:38] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-GettingStarted, 6Operations: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#2037655 (10Mattflaschen) 5Open>3Resolved a:3Mattflaschen It hasn't happened recently that I know of. We can a... [02:29:49] 10Beta-Cluster-Infrastructure, 6Labs, 5Patch-For-Review: Soft mount remaining NFS mounts on deployment-prep - https://phabricator.wikimedia.org/T127224#2037726 (10yuvipanda) I've done this now for all things in deployment-prep that still need NFS. The mounts are soft, with a 10s timeout - if the NFS server t... [02:36:30] 10Beta-Cluster-Infrastructure, 6Labs, 5Patch-For-Review: Soft mount remaining NFS mounts on deployment-prep - https://phabricator.wikimedia.org/T127224#2037735 (10yuvipanda) 5Open>3Resolved a:3yuvipanda I just tested this by removing export for one instance from the NFS server, and it handled it perfec... [02:41:41] 3Scap3, 10scap, 6Phabricator, 5Patch-For-Review, 7WorkType-NewFunctionality: Deploy Phabricator with scap3 - https://phabricator.wikimedia.org/T114363#2037743 (10mmodell) [02:41:45] 3Scap3, 10scap, 6Phabricator, 7Blocked-on-Operations, 7WorkType-Maintenance: scap::target should use scap's debian package instead of trebuchet - https://phabricator.wikimedia.org/T127215#2037742 (10mmodell) 5Open>3Resolved [03:12:48] Project beta-scap-eqiad build #90445: 04FAILURE in 8 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90445/ [03:15:41] 6Release-Engineering-Team, 6Phabricator, 7WorkType-Maintenance: Phabricator errors when filing (creating) a new Maniphest task - https://phabricator.wikimedia.org/T127271#2037854 (10mmodell) [03:16:31] Yippee, build fixed! [03:16:32] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #981: 09FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/981/ [03:22:01] Yippee, build fixed! [03:22:01] Project beta-scap-eqiad build #90446: 09FIXED in 7 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90446/ [03:48:16] 10Continuous-Integration-Infrastructure, 7developer-notice: Switch MySQL storage to tmpfs - https://phabricator.wikimedia.org/T96230#2037935 (10Tgr) [03:48:18] 10Continuous-Integration-Infrastructure: mediawiki jobs fail intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2037933 (10Tgr) 5Resolved>3Open Still happening: https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/51176/consoleFull [03:50:38] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#2037936 (10greg) 5Open>3Resolved p:5Unbreak!>3Normal [06:47:19] PROBLEM - Puppet failure on integration-slave-trusty-1002 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [07:27:24] RECOVERY - Puppet failure on integration-slave-trusty-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [08:34:10] Yippee, build fixed! [08:34:11] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #881: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/881/ [08:46:24] (03CR) 10Legoktm: [C: 031] Remove ApiSandbox from branching configuration [tools/release] - 10https://gerrit.wikimedia.org/r/270903 (owner: 10Florianschmidtwelzow) [09:42:59] 10Continuous-Integration-Infrastructure: phpunit fails in mwext-testextension-php55 tests - https://phabricator.wikimedia.org/T127289#2038332 (10santhosh) [09:45:26] 10Continuous-Integration-Infrastructure: phpunit fails in mwext-testextension-php55 tests - https://phabricator.wikimedia.org/T127289#2038361 (10Paladox) @Krinkle can I add you to task please. [09:45:52] 10Continuous-Integration-Infrastructure: phpunit fails in mwext-testextension-php55 tests - https://phabricator.wikimedia.org/T127289#2038364 (10hashar) [09:45:54] 10Continuous-Integration-Infrastructure: mediawiki-extensions-hhvm jobs intermittently fail with "File not found: Filter.php" - https://phabricator.wikimedia.org/T127226#2038365 (10hashar) [09:46:31] 10Continuous-Integration-Infrastructure: mediawiki-extensions-hhvm jobs intermittently fail with "File not found: Filter.php" - https://phabricator.wikimedia.org/T127226#2036379 (10hashar) [09:47:39] 10Continuous-Integration-Infrastructure: mediawiki-extensions-hhvm jobs intermittently fail with "File not found: Filter.php" - https://phabricator.wikimedia.org/T127226#2038388 (10Paladox) Could this problem be fixed by upgrading phpunit to 4.8. [10:05:09] 10Continuous-Integration-Infrastructure: mediawiki-extensions-hhvm jobs intermittently fail with "File not found: Filter.php" - https://phabricator.wikimedia.org/T127226#2038427 (10hashar) I think both issues are related. The composer autoloader is wrong somehow and ends up missing files? [10:14:11] 10Continuous-Integration-Infrastructure: mediawiki-extensions-hhvm jobs intermittently fail with "File not found: Filter.php" - https://phabricator.wikimedia.org/T127226#2038447 (10Paladox) In phpunit 4.x it was updated since there seems to be an autoloader php file in phpunit 3.x but not in 4.x which is strange... [10:18:27] twentyafterfour: When i go and click edit task and add my self as a subscriber to https://phabricator.wikimedia.org/T91997 it brings up this error You must choose a space for this object. [10:27:17] known issue [10:28:52] paladox, I just successfully added myself as a subscriber however [10:29:04] did you set the space? [10:29:38] Krenair: Im not sure what the space is. I managed a workaround by clicking subscribe but editing the task is not possible for me since it brings up that error. [10:31:07] Krenair it seems to happend to me in all tasks. What is the space. [10:35:51] Krenair twentyafterfour https://secure.phabricator.com/T9999 [10:36:07] 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Technical-Debt, 7Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#2038510 (10hashar) [10:38:15] (03PS1) 10Paladox: [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 [10:38:19] (03PS1) 10Hashar: [SecurePoll] switch to jshint/jsonlint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271484 (https://phabricator.wikimedia.org/T63626) [10:38:33] (03Abandoned) 10Paladox: [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (owner: 10Paladox) [10:38:55] (03CR) 10Paladox: [C: 031] "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/271484 (https://phabricator.wikimedia.org/T63626) (owner: 10Hashar) [10:39:10] paladox: ah sorry dupe again :D [10:39:32] hashar: Ok. [10:39:38] (03Restored) 10Hashar: [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (owner: 10Paladox) [10:39:46] (03PS2) 10Hashar: [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:39:59] (03CR) 10Hashar: [C: 032] [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:40:08] (03Abandoned) 10Hashar: [SecurePoll] switch to jshint/jsonlint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271484 (https://phabricator.wikimedia.org/T63626) (owner: 10Hashar) [10:40:10] hashar: Thanks. [10:40:13] paladox: using your change instead :-))))) [10:40:17] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:40:26] hashar: :) thanks. [10:40:33] that is 3 Thanks !!!! :D [10:41:32] hashar: I will do more jslint changes. Im probaly going to convert the ones that are less use meaning not alot of changes excluding i18n updates. to jshint and jsonlint and ones that are popular or are being maintained to npm. [10:41:32] (03CR) 10jenkins-bot: [V: 04-1] [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:42:01] I will amend that one [10:42:32] hashar: We should setup a seperate task for npm to convert repos to it since jshint and jsonlint should be used in check: until there is some way of everyone using npm with causing problems by mistake. [10:42:37] hashar: Thanks. [10:42:39] (03PS3) 10Hashar: [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:42:53] paladox: well the jshint job should be removed eventually [10:43:00] it runs out of an outdated copy of jshint [10:43:38] hashar: Yes. Is it possible to run a custom npm version for non whitelisted users that it will only check for jshint and jsonlint in the packages.json. [10:44:19] hashar: It might be since in composer we check for phpunit in vendor/phpunit per krinkle patch what about doing node_modules/ not sure what the path is to the jshint npm package is. [10:44:23] !log Nodepool: JenkinsException: Could not parse JSON info for server[https://integration.wikimedia.org/ci/] [10:44:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:45:16] hashar: Could you also review https://gerrit.wikimedia.org/r/#/c/271154/ please since that patch for jslint was merged yesturday. [10:52:48] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038534 (10hashar) [10:54:02] (03CR) 10Hashar: [C: 032] [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:54:12] hashar: Thanks. [10:56:29] hashar: Do you know how to do in npm a way to detect jshint and jsonlint. Since we can setup a seperate job for npm for non whitelisted users that way jshint is always updated for non whitelisted users but they wont have the ability to run any other package without updating this job. [10:56:43] (03Merged) 10jenkins-bot: [SecurePoll] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:58:10] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038552 (10hashar) On gallium I do see the requests in /var/log/jenkins/access.log: ``` [18/Feb/2016:10:55:24 +0000] "GET /ci/api/json HTTP/1.1" 200 97014... [10:58:31] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/271483 (https://phabricator.wikimedia.org/T63626) (owner: 10Paladox) [10:58:49] paladox: the plan is to get rid of the whitelist thing [10:58:56] paladox: and just run 'npm test' for everyeone [10:59:11] hashar: Oh ok. Do you know when that will happen. [10:59:55] I have started migrating mediawiki/services/* repos slowly [11:00:01] gotta move some other *-npm jobs [11:00:04] and then the generic one [11:01:06] hashar: Oh so your starting to open npm to everyone. [11:01:13] Meaning non whitelisted users. [11:01:15] eventually yeah [11:01:24] hashar: :) [11:04:12] hashar: When editing this task https://phabricator.wikimedia.org/maniphest/task/edit/62619/ it keeps comming up with [11:04:13] You must choose a space for this object. [11:04:42] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038557 (10hashar) ``` $ curl --silent https://integration.wikimedia.org/ci/api/json|python -m json.tool Invalid control character at: line 1 column 96213... [11:08:56] andre__ Hi i seem to be having a problem on phabricator. When i go to https://phabricator.wikimedia.org/maniphest/task/edit/62619/ and click save i am getting this error [11:09:03] You must choose a space for this object. [11:09:04] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038563 (10hashar) There is something borked related to the Jenkins views: ``` lang=json {"name":"Labs","url":"https://integration.wikimedia.org/ci/view/La... [11:12:03] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038534 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVL0FFuU-0X0Il_jxsm1} [2016-02-18T11:12:00Z] Je... [11:24:10] paladox: Hi. Feel free to report Phabricator bugs in Phabricator. [11:24:22] In this case it's already reported, see https://phabricator.wikimedia.org/T127290 [11:24:38] andre__ Yes and thanks. [11:25:53] no "me too" needed on that task, but better and clear steps to reproduce are welcome. [11:27:08] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038656 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVL0IisChQaf1CQcCf4d} [2016-02-18T11:27:05Z] Je... [11:28:23] andre__ ok ive included steps to reproduce but i doint think the problem is happening to everyone. I think hashar managed to edit tasks. [11:34:03] thanks [11:34:19] yes, bugs often have conditions that not everybody can trigger. :D [11:34:52] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038676 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVL0KT8hW8txF7J0uUA0} [2016-02-18T11:34:49Z] Ha... [11:36:27] andre__ Ok it seems that all tasks created after phabricator was updated today can be edited but all tasks created before carn't. [11:57:50] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Newsletter: Add the Newsletter extension to the Beta Cluster - https://phabricator.wikimedia.org/T127297#2038740 (10Qgil) [12:09:26] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038829 (10hashar) The Jenkins service itself seems to be just fine. On gallium have been doing `curl --silent 'http://127.0.0.1:8080/ci/api/json'|python -... [12:17:18] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038873 (10hashar) We had some changes made to Nginx yesterday as part of {T107749} by @BBlack https://integration.wikimedia.org/ci/api/json started bein... [12:20:42] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Newsletter: Add the Newsletter extension to the Beta Cluster - https://phabricator.wikimedia.org/T127297#2038894 (10Qgil) [12:20:47] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Newsletter: Add the Newsletter extension to the Beta Cluster - https://phabricator.wikimedia.org/T127297#2038740 (10Qgil) [12:26:06] 10Continuous-Integration-Infrastructure: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038927 (10hashar) curl with headers from my machine: ``` $ curl --silent --verbose https://integration.wikimedia.org/ci/api/json |python -m json.tool>/de... [12:30:04] 10Continuous-Integration-Infrastructure, 6Operations, 10Traffic: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038934 (10hashar) I noticed that after some X time, the request pass just fine but subsequent ones all fail. I gave it a try... [12:30:14] 10Continuous-Integration-Infrastructure: mediawiki jobs fail intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2038939 (10JanZerebecki) During that job run there was no puppet run on that host. Also there is nothing in dmesg that shows mysql or an... [12:30:16] 10Continuous-Integration-Infrastructure, 6Operations, 10Traffic: ERROR nodepool.NodePool: Unable to check status of gallium.wikimedia.org - https://phabricator.wikimedia.org/T127294#2038943 (10hashar) [12:30:50] 10Continuous-Integration-Infrastructure, 6Operations, 10Traffic: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2038534 (10hashar) [12:31:49] (03PS1) 10Paladox: [Survey] Update extension unit tests to no voting and also update other tests on the repo [integration/config] - 10https://gerrit.wikimedia.org/r/271498 (https://phabricator.wikimedia.org/T63637) [12:39:14] 10Continuous-Integration-Infrastructure: run extension jobs with and without optional dependencies - https://phabricator.wikimedia.org/T114691#2038994 (10JanZerebecki) [12:55:08] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #761: 04FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/761/ [12:55:31] Project beta-scap-eqiad build #90501: 04FAILURE in 5 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90501/ [13:02:57] Yippee, build fixed! [13:02:58] Project beta-scap-eqiad build #90502: 09FIXED in 5 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90502/ [13:04:43] 5Gerrit-Migration, 10Phabricator-Upstream, 7WorkType-NewFunctionality: Workboard cards could show other projects tasks belong to - https://phabricator.wikimedia.org/T683#2039164 (10hashar) Our Phabricator instance has been updated via https://secure.phabricator.com/T4863 via commit https://secure.phabricator... [13:12:42] 3Scap3, 10scap, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation: Deploy CXServer with scap3 - https://phabricator.wikimedia.org/T120104#2039200 (10Amire80) [13:12:57] 3Scap3, 10scap, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation: Deploy CXServer with scap3 - https://phabricator.wikimedia.org/T120104#1844945 (10Amire80) [13:15:02] hashar: Like you did for phpcs by explaning which repos need converting, could you do the same to https://phabricator.wikimedia.org/T62619 please. I would but i carnt edit it because of a problem in phabricator since it was updated. [13:19:44] 10Beta-Cluster-Infrastructure, 10Citoid, 10ContentTranslation-cxserver, 10Graphoid, and 5 others: Migrate BetaCluster Node.JS services to Jessie and Node 4.2 - https://phabricator.wikimedia.org/T125003#2039236 (10Amire80) [13:25:59] RECOVERY - Puppet failure on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [13:50:23] 7Browser-Tests, 10Wikidata, 5Patch-For-Review: Sitelink browser test sometimes fails with firefox - https://phabricator.wikimedia.org/T126585#2039322 (10adrianheine) I checked again: This works locally. [14:13:22] 10Browser-Tests-Infrastructure: mediawiki_api::log_in does not work due to gzip issue - https://phabricator.wikimedia.org/T127309#2039382 (10adrianheine) [14:13:27] hashar: I have a few simple commits in core, if you have a few minutes to review them [14:13:42] https://gerrit.wikimedia.org/r/#/c/270929/ [14:13:51] https://gerrit.wikimedia.org/r/#/c/270924/ [14:14:43] working on this now, got distracted :| https://gerrit.wikimedia.org/r/#/c/270724/ [14:45:16] hashar: and https://gerrit.wikimedia.org/r/#/c/270724/ is ready now too [14:46:14] (03CR) 10Anomie: "I note this is identical to Ib4e194f6d54374600a881c70c279d43acac96a04." [tools/release] - 10https://gerrit.wikimedia.org/r/270903 (owner: 10Florianschmidtwelzow) [14:46:37] well, these two are ready too :) [14:46:43] https://gerrit.wikimedia.org/r/#/c/270905/2 [14:46:49] https://gerrit.wikimedia.org/r/#/c/270968/2 [14:48:03] (03PS4) 10Hashar: [AWS] Update Jenkins Tests [integration/config] - 10https://gerrit.wikimedia.org/r/266806 (owner: 10Paladox) [14:51:53] (03CR) 10Hashar: [C: 032] [AWS] Update Jenkins Tests [integration/config] - 10https://gerrit.wikimedia.org/r/266806 (owner: 10Paladox) [14:53:10] (03Merged) 10jenkins-bot: [AWS] Update Jenkins Tests [integration/config] - 10https://gerrit.wikimedia.org/r/266806 (owner: 10Paladox) [14:53:32] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/266806 (owner: 10Paladox) [14:56:34] (03CR) 10Hashar: [C: 031] "Ideally we would deprecate it in 1.7.0 and remove it with 1.8.0 But since we just migrate everything that can be removed I guess :-} Wil" [selenium] - 10https://gerrit.wikimedia.org/r/270905 (https://phabricator.wikimedia.org/T127042) (owner: 10Zfilipin) [15:01:56] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Newsletter: Add the Newsletter extension to the Beta Cluster - https://phabricator.wikimedia.org/T127297#2038740 (10greg) Getting a new extension on the Beta Cluster is part of the process of getting it into production (we don't just add random extensions p... [15:02:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Newsletter: Add the Newsletter extension to the Beta Cluster - https://phabricator.wikimedia.org/T127297#2039569 (10greg) [15:02:56] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Newsletter: Add the Newsletter extension to the Beta Cluster - https://phabricator.wikimedia.org/T127297#2038740 (10greg) [15:03:33] (03PS1) 10Paladox: [JsonData] Replace jslint with jsonlint and jshint [integration/config] - 10https://gerrit.wikimedia.org/r/271525 (https://phabricator.wikimedia.org/T63607) [15:06:30] (03PS3) 10Hashar: [SmiteSpam] Add jshint and jsonlint for non-whitelisted users [integration/config] - 10https://gerrit.wikimedia.org/r/264968 (owner: 10Paladox) [15:06:58] (03CR) 10Hashar: [C: 032] [SmiteSpam] Add jshint and jsonlint for non-whitelisted users [integration/config] - 10https://gerrit.wikimedia.org/r/264968 (owner: 10Paladox) [15:07:29] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/264968 (owner: 10Paladox) [15:07:59] (03Merged) 10jenkins-bot: [SmiteSpam] Add jshint and jsonlint for non-whitelisted users [integration/config] - 10https://gerrit.wikimedia.org/r/264968 (owner: 10Paladox) [15:08:45] (03Abandoned) 10Hashar: Update composer to dev-master [integration/composer] - 10https://gerrit.wikimedia.org/r/270548 (owner: 10Paladox) [15:11:22] hashar: if you are around, I’d like to make a hotfix to the glance policy and see if that makes nodepool happy. [15:11:41] andrewbogott: good morning 🍩 🍩 🍩 [15:11:48] (03PS1) 10Paladox: [LastModified] Replace jslint test with jsonlint and jshint tests [integration/config] - 10https://gerrit.wikimedia.org/r/271528 (https://phabricator.wikimedia.org/T63608) [15:12:08] (03PS2) 10Paladox: [LastModified] Replace jslint test with jsonlint and jshint tests [integration/config] - 10https://gerrit.wikimedia.org/r/271528 (https://phabricator.wikimedia.org/T63608) [15:12:21] andrewbogott: yeah I imagine something went revoked when some openstack system got changed earlier this week [15:12:21] can test as needed [15:12:21] * andrewbogott wishes he had a real donut [15:12:39] (03PS3) 10Paladox: [LastModified] Replace jslint test with jsonlint and jshint tests [integration/config] - 10https://gerrit.wikimedia.org/r/271528 (https://phabricator.wikimedia.org/T63608) [15:12:53] I would get a plain but you would not have the donut before tomorrow afternoon :/ [15:13:03] (not to mention the donut cost) [15:14:05] ok, I just changed the policy to (I think) allow image creation/deletion [15:14:07] any different? [15:14:15] 10Browser-Tests-Infrastructure, 5Release-Engineering-Epics, 7Epic, 7Tracking: Fix or delete failing browser tests Jenkins jobs - https://phabricator.wikimedia.org/T94150#2039599 (10greg) [15:15:26] andrewbogott: trying [15:16:52] ~$ openstack image delete ci-jessie-wikimedia-1455548346 [15:16:52] ERROR: openstack 403 Forbidden: Access was denied to this resource. (HTTP 403) [15:17:08] well… [15:17:09] ok [15:17:15] can you ‘image list’? [15:17:36] yeah [15:17:42] I see all of labs images [15:17:45] * andrewbogott turns on logging [15:17:53] and with --private the images form the project contintcloud [15:18:03] the account is 'nodepoolmanager' maybe it is not a project admin [15:18:13] weirdly the policy file doesn’t seem to distinguish between public and private images [15:18:27] at least it can still spawn / delete instances [15:18:50] ok, try the delete again? I want to see what the log says... [15:19:41] HTTPForbidden: 403 Forbidden: Access was denied to this resource. (HTTP 403) [15:19:53] DEBUG: urllib3.connectionpool "DELETE /v1/images/17c9f2ee-c66b-43d4-bd09-d51af98e0743 HTTP/1.1" 403 157 [15:20:10] to labcontrol1001 [15:23:15] hashar: ok, sorry, once more? [15:23:32] DEBUG: urllib3.connectionpool "DELETE /v1/images/17c9f2ee-c66b-43d4-bd09-d51af98e0743 HTTP/1.1" 403 157 [15:23:32] eek [15:23:34] no :( [15:23:58] I’m still just digging in the logs [15:24:51] maybe the glance policy is not properly applied [15:25:05] hashar: Jenkins has gone down. [15:25:12] Im getting [15:25:13] If you report this error to the Wikimedia System Administrators, please include the details below. [15:25:13] Request from 10.20.0.119 via cp3019 frontend ([10.20.0.119]:80), Varnish XID 213329669 [15:25:13] Forwarded for: 86.131.250.207, 10.20.0.119 [15:25:13] Error: 503, Service Unavailable at Thu, 18 Feb 2016 15:24:57 GMT [15:25:29] hashar works now. [15:26:01] hashar: try now, does this work at least? [15:26:13] (or, better yet, tell me where/how you’re running this command and I’ll try for myself) [15:26:22] paladox: yeah that happens from time to time. We also have a crazy issue going on and one of WMF ops is looking at it (some pages get corrupted randomly) [15:26:30] hashar: Could you merge https://gerrit.wikimedia.org/r/#/c/271529/ please. [15:26:31] andrewbogott: yeah that is smarter [15:26:33] hashar: Oh ok. [15:26:51] andrewbogott: so labnodepool1001.eqiad.wmnet then use the command 'become-nodepool' [15:26:56] which send to me your root key^H^H sudo you as nodepool [15:27:06] so you get the openstack credentials loaded from a .env file [15:27:22] then openstack CLI should do all actions as nodepoolmanager user against tenant contintcloud [15:27:22] hashar: easy! thanks [15:28:09] ok, so if I set the policy to ‘anyone can delete anything anytime’ it works fine [15:28:16] can’t say I love that as a long term solution though [15:28:30] (03PS1) 10Paladox: [SideBarMenu] Update test to use composer unit tests instead of non-voting ones [integration/config] - 10https://gerrit.wikimedia.org/r/271530 [15:28:56] although apparently that is the default glance security policy! What the hell, glance? [15:31:30] (03Abandoned) 10Florianschmidtwelzow: Remove ApiSandbox from branching configuration [tools/release] - 10https://gerrit.wikimedia.org/r/270903 (owner: 10Florianschmidtwelzow) [15:31:34] (03CR) 10Florianschmidtwelzow: [C: 031] Undeploy ApiSandbox [tools/release] - 10https://gerrit.wikimedia.org/r/263002 (owner: 10Anomie) [15:32:04] (03CR) 10Florianschmidtwelzow: [C: 032] Undeploy ApiSandbox [tools/release] - 10https://gerrit.wikimedia.org/r/263002 (owner: 10Anomie) [15:32:58] (03Merged) 10jenkins-bot: Undeploy ApiSandbox [tools/release] - 10https://gerrit.wikimedia.org/r/263002 (owner: 10Anomie) [15:35:22] andrewbogott: delete anything is scary :-} [15:36:14] hashar: can I safely delete any of these other images? [15:36:25] na they are used [15:36:34] nodepool image-list [15:36:38] list what nodepool knows [15:37:20] so… how do I test? [15:37:21] andrewbogott: ci-jessie-wikimedia-1455549539 can be deleted [15:37:21] it is marked as deleted [15:37:34] and you can try the ci-jessie-wikimedia_old_* [15:38:02] (03PS1) 10Paladox: [SideBarMenu] Replace jslint test with jshint and jsonlint tests [integration/config] - 10https://gerrit.wikimedia.org/r/271532 (https://phabricator.wikimedia.org/T63636) [15:38:07] ok [15:38:13] 5Gerrit-Migration, 10Phabricator-Upstream, 7WorkType-NewFunctionality: Workboard cards could show other projects tasks belong to - https://phabricator.wikimedia.org/T683#2039753 (10Aklapper) [15:38:19] I think I know how to fix this but I guess I have to create a custom role for this [15:38:23] in nodepool user homedir there is a qcow2 image /var/lib/nodepool/image-jessie-20160215T154059Z.qcow2 [15:38:26] so I don’t give every project admin the rights to delete images [15:38:40] and I push it using: [15:39:02] openstack image create --file /var/lib/nodepool/image-jessie-20160215T154059Z.qcow2 AndrewPutANameHere --disk-format qcow2 --property show=true [15:39:12] so that should let you fill in a bunch of images [15:39:21] just make sure to not name it ci-jessie-wikimedia ;-) [15:39:39] should take a few seconds only to push it to glance which is very amazing [15:40:04] adding andrew_test1 [15:40:15] openstack image create --file /var/lib/nodepool/image-jessie-20160215T154059Z.qcow2 --disk-format qcow2 --property show=true andrew_test1 [15:40:21] bah glance is not as fast as it used to be [15:40:42] you’re getting a 403 though, right? [15:40:44] that’s what I get... [15:40:51] WARNING: urllib3.connectionpool Connection pool is full, discarding connection: labcontrol1001.wikimedia.org [15:40:51] ERROR: openstack 403 Forbidden: Access was denied to this resource. (HTTP 403) [15:40:54] yeah can't create images anymore [15:41:14] hm, ‘upload image’ is a different right from ‘create image' [15:41:25] which is consistent with nodepool image-list .. the most recent one is from 72 hours ago [15:42:05] can you still upload images just fine with your account ? [15:49:22] hashar: https://gerrit.wikimedia.org/r/#/c/271536/1 should do it. see any other rights in there that you expect to need? [15:50:24] root : [ 'hashar' ] [15:50:32] looking [15:51:05] ah it is all mysterious [15:51:18] to me too, I guess I’ll merge that and we’ll stick with trial and error [15:51:37] wondering what are the *_image_location rules [15:51:49] but maybe that is the path were glance save the file [15:52:43] I am not sure how role:glanceadmin is set though [15:54:02] hashar: I assigned the role by hand on the cmdline [15:54:18] yeah, I was looking suspiciously at those image_location things too [15:54:23] I guess we’ll see if it works without [15:54:28] btw, is CI broken in the meantime? [15:54:49] I’m still waiting for my checkmark [15:56:05] yeah it is out of instances [15:56:24] so I just need to wait, right? [15:57:02] looking at nodepool logs [15:57:21] /var/log/nodepool/debug.log [15:57:50] yeah instances are building [15:57:55] it takes roughly a minute iirc [15:58:07] and there is a sleep(60) on deletion [15:58:21] so when the pool is exhausted it takes two minutes to recreate them [15:58:54] maybe we can raise the default pool a bit (it is 10 right now) [16:01:21] !log deleting instance integration-slave-precise-1003 think we have enough precise slaves [16:01:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:07:42] hashar: ok, everything is merged; is nodepool happy again? [16:18:37] andrewbogott: Assuming you're aware of the new "Create Access Task" form created recently, can it maybe be renamed or reordered to not be the top-most option? I imagine there'll be a fair number of wrongly filed tasks otherwise [16:19:40] Krinkle: I am… not aware [16:19:49] andre__: ^ [16:19:53] andrewbogott: sorry, wrong ping [16:25:27] andrewbogott: trying [16:25:43] andrewbogott: yeah managed to create an image [16:26:10] and can delete \O/ [16:26:28] and nodepool catches up [16:26:34] great — let me know if you run into other permission issues [16:26:43] I changed basically all of them a few days ago [16:27:07] hope your fix is not too wide [16:27:14] could nodepoolmanager delete the global images ? [16:27:33] hashar: I think it probably always could [16:27:49] there’s no way to distinguish as far as I can tell [16:27:56] thedj could i have help to fix the config issue in TimedMediaHandler [16:28:03] well, ok, actually — the ‘glanceadmin’ role is project-specific [16:28:12] so in theory that check only succeeds for project-local images [16:28:16] so maybe we’re safe [16:28:25] andrewbogott: would be worth digging and have the global images / project images to use different permissions [16:28:30] hashar: I’ll make a test case and try it in a bit. [16:28:32] we will see :) [16:29:38] Krinkle: try again? :) [16:29:54] andre__: Thanks! [16:30:27] Krinkle, thanks for mentioning! [16:34:50] paladox: i don't have much time available for coding and debugging, so i want to help you, but I have no idea when and how i would be able to manage that. [16:35:04] thedj: Oh ok. [16:35:13] andrewbogott: glance policies rules are cryptic to me http://docs.openstack.org/developer/glance/policies.html :/ [16:41:27] thedj: Could you quickly have a look at https://gerrit.wikimedia.org/r/#/c/271538/ please. Its todo with converting resouceloader to that hook. And also review https://gerrit.wikimedia.org/r/#/c/264586/ it is about updating videojs to the latest release. [16:44:57] andrewbogott: the images created have owner=contintcloud so maybe the glance policy should have something to allow the owner to delete them instead of glance admin [16:45:06] andrewbogott: at least nodepool is happy [16:48:18] 10Browser-Tests-Infrastructure, 6Multimedia, 10UploadWizard: Firefogg browser test - https://phabricator.wikimedia.org/T93174#2040100 (10matmarex) [16:55:43] (03CR) 10Hashar: [C: 032] [SideBarMenu] Replace jslint test with jshint and jsonlint tests [integration/config] - 10https://gerrit.wikimedia.org/r/271532 (https://phabricator.wikimedia.org/T63636) (owner: 10Paladox) [16:56:10] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/271532 (https://phabricator.wikimedia.org/T63636) (owner: 10Paladox) [16:56:12] 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Technical-Debt, 7Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#2040131 (10hashar) [16:57:22] (03Merged) 10jenkins-bot: [SideBarMenu] Replace jslint test with jshint and jsonlint tests [integration/config] - 10https://gerrit.wikimedia.org/r/271532 (https://phabricator.wikimedia.org/T63636) (owner: 10Paladox) [16:58:02] Project browsertests-Wikidata-SmokeTests-linux-firefox build #110: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox/110/ [17:00:54] (03PS1) 10Hashar: Remove non-voting testextension jobs from gate [integration/config] - 10https://gerrit.wikimedia.org/r/271555 (https://phabricator.wikimedia.org/T126682) [17:01:56] (03CR) 10Hashar: "That follows up https://gerrit.wikimedia.org/r/#/c/270651/" [integration/config] - 10https://gerrit.wikimedia.org/r/271555 (https://phabricator.wikimedia.org/T126682) (owner: 10Hashar) [17:09:42] (03CR) 10Paladox: [C: 031] Remove non-voting testextension jobs from gate [integration/config] - 10https://gerrit.wikimedia.org/r/271555 (https://phabricator.wikimedia.org/T126682) (owner: 10Hashar) [17:10:12] (03CR) 10Hashar: "For the context, I noticed that on SideBarMenu extension https://gerrit.wikimedia.org/r/271532" [integration/config] - 10https://gerrit.wikimedia.org/r/271555 (https://phabricator.wikimedia.org/T126682) (owner: 10Hashar) [17:23:48] hashar: I created a new image called ‘deletemehashar’ id 70a92cda-c359-443b-bdbc-60d7f2de864f [17:23:50] can you delete it? [17:24:03] checking [17:24:51] ERROR: openstack 403 Forbidden: Forbidden to delete image: 403 Forbidden: Access was denied to this resource. (HTTP 403) [17:25:24] awesome [17:25:33] so we’re in good shape then, the nodepool rights are project-specific [17:25:39] * andrewbogott deletes it [17:26:00] oh nice [17:26:06] we are safe [17:31:29] (03CR) 10Legoktm: [C: 032] Remove non-voting testextension jobs from gate [integration/config] - 10https://gerrit.wikimedia.org/r/271555 (https://phabricator.wikimedia.org/T126682) (owner: 10Hashar) [17:32:24] (03Merged) 10jenkins-bot: Remove non-voting testextension jobs from gate [integration/config] - 10https://gerrit.wikimedia.org/r/271555 (https://phabricator.wikimedia.org/T126682) (owner: 10Hashar) [17:36:02] !log deploying https://gerrit.wikimedia.org/r/271555 [17:36:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:37:25] reminder: In ~30 minutes all labs instances are going to start rebooting in a totally unpredictable sequence. [17:37:54] unpredictable is something i like [17:38:35] The order will, of course, be deterministic and computable. [17:38:42] (03PS3) 10Legoktm: Actually get rid of mwext-MobileFrontend-publish, for real this time [integration/config] - 10https://gerrit.wikimedia.org/r/271012 [17:39:03] somethingsomething penrose scheduling algorithm [17:39:08] we will have to rearm the keyholder on deployment-tin i think [17:41:21] hashar: we should probably add something that does that automatically [17:41:41] bd808: i think there is a task to have a service / upstart whatever to bootstrap keyholder [17:41:46] like a step in the scap jenkins job or something [17:41:53] we reboot only once in a while so not anywhere urgent [17:58:02] Project browsertests-Wikidata-WikidataTests-linux-firefox build #108: 15ABORTED in 4 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox/108/ [18:06:21] 10Continuous-Integration-Infrastructure, 6Operations, 10Traffic, 5Patch-For-Review: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2038534 (10BBlack) 5Open>3Resolved Fixed by disabling do_gzip on the misc... [18:12:33] (03CR) 10Legoktm: [C: 032] Actually get rid of mwext-MobileFrontend-publish, for real this time [integration/config] - 10https://gerrit.wikimedia.org/r/271012 (owner: 10Legoktm) [18:13:58] (03Merged) 10jenkins-bot: Actually get rid of mwext-MobileFrontend-publish, for real this time [integration/config] - 10https://gerrit.wikimedia.org/r/271012 (owner: 10Legoktm) [18:14:07] !log deploying https://gerrit.wikimedia.org/r/271012 [18:14:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:17:41] 10Continuous-Integration-Infrastructure, 6Operations, 10Traffic, 5Patch-For-Review: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2040636 (10hashar) Nodepool is all happy reaching out to Jenkins API since do_... [18:21:37] 10Continuous-Integration-Infrastructure, 6Operations, 10Traffic, 5Patch-For-Review: https://integration.wikimedia.org/ci/api/json is corrupted when required more than one time in a raw - https://phabricator.wikimedia.org/T127294#2040685 (10BBlack) For further followup, see also: https://phabricator.wikimed... [18:25:51] PROBLEM - Host deployment-elastic08 is DOWN: CRITICAL - Host Unreachable (10.68.17.188) [18:26:07] PROBLEM - Host deployment-memc03 is DOWN: CRITICAL - Host Unreachable (10.68.16.15) [18:31:07] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #601: 04FAILURE in 1 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/601/ [18:53:53] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #604: 04FAILURE in 53 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/604/ [19:12:54] (03PS1) 10Legoktm: Remove no-longer needed non-voting rules for testextension jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271581 (https://phabricator.wikimedia.org/T126682) [19:14:36] (03CR) 10Legoktm: [C: 032] "diff shows no-op" [integration/config] - 10https://gerrit.wikimedia.org/r/271581 (https://phabricator.wikimedia.org/T126682) (owner: 10Legoktm) [19:15:33] (03CR) 10jenkins-bot: [V: 04-1] Remove no-longer needed non-voting rules for testextension jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271581 (https://phabricator.wikimedia.org/T126682) (owner: 10Legoktm) [19:16:57] (03PS2) 10Legoktm: Remove no-longer needed non-voting rules for testextension jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271581 (https://phabricator.wikimedia.org/T126682) [19:16:59] (03PS1) 10Legoktm: Remove last vestiges of mwext-MobileFrontend-publish job [integration/config] - 10https://gerrit.wikimedia.org/r/271583 [19:19:07] (03CR) 10Legoktm: [C: 032] Remove no-longer needed non-voting rules for testextension jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271581 (https://phabricator.wikimedia.org/T126682) (owner: 10Legoktm) [19:19:27] (03CR) 10Legoktm: [C: 032] Remove last vestiges of mwext-MobileFrontend-publish job [integration/config] - 10https://gerrit.wikimedia.org/r/271583 (owner: 10Legoktm) [19:20:18] (03Merged) 10jenkins-bot: Remove last vestiges of mwext-MobileFrontend-publish job [integration/config] - 10https://gerrit.wikimedia.org/r/271583 (owner: 10Legoktm) [19:20:51] (03Merged) 10jenkins-bot: Remove no-longer needed non-voting rules for testextension jobs [integration/config] - 10https://gerrit.wikimedia.org/r/271581 (https://phabricator.wikimedia.org/T126682) (owner: 10Legoktm) [19:20:54] !log deploying https://gerrit.wikimedia.org/r/271583 and https://gerrit.wikimedia.org/r/271581, both no-ops [19:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:25:41] hey thcipriani, yt? [19:26:01] i'm talking with mobrovac and milimetric now, we are considering deploying aqs (restbase thing) with scap now today! [19:26:04] Project beta-scap-eqiad build #90541: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90541/ [19:26:05] for that I think we need to merge https://gerrit.wikimedia.org/r/#/c/269143/ [19:27:26] ottomata: yeah. That patch is in need of some love. A lot of the prerequisites have merged. [19:27:36] okcool, let's make it happen! [19:27:38] i can love it [19:29:26] Awesome! [19:30:31] ok, lookin at it... [19:30:36] ottomata: I just ran out to lunch. Can I connect with you when I get back? [19:30:42] sure [19:31:20] I think all that patch needs is a rebase. Then it's just a matter of the service::node adjustment. [19:32:01] 19:24:57 + rsync --recursive docs/js/ rsync://10.68.16.255/doc/mediawiki-core-jsduck-publish-6049 [19:32:01] 19:25:15 rsync: failed to connect to 10.68.16.255 (10.68.16.255): No route to host (113) [19:32:01] 19:25:15 rsync error: error in socket IO (code 10) at clientserver.c(128) [sender=3.1.0] [19:32:08] I assume those are labs reboots? [19:32:19] i'm looking at some of the comments, might need more than that [19:32:20] not sure [19:32:24] twentyafterfour: yt too? [19:33:04] yes we're rebooting all of labs [19:43:48] Yippee, build fixed! [19:43:48] Project beta-scap-eqiad build #90543: 09FIXED in 8 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90543/ [19:57:08] Project beta-scap-eqiad build #90544: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/90544/ [20:12:21] (03CR) 10Legoktm: [C: 04-2] "This won't make any difference at all. The generic jobs are faster and better because they can re-use workspaces instead of needing a diff" [integration/config] - 10https://gerrit.wikimedia.org/r/269768 (owner: 10Paladox) [20:21:18] (03PS1) 10Legoktm: PrefSwitch extension was archived [integration/config] - 10https://gerrit.wikimedia.org/r/271600 [20:39:23] bd808: ostriches: Hm.. it'd be nice if invoking sync-common with an argument (e.g. foo.php) will fail saying the user probably meant to use sync-file or sync-dir instead. [20:39:52] it would be nice if we renamed sync-common [20:39:56] Meh, sync-common is going away anyway [20:40:04] In favor of deploy-local :P [20:40:14] Oh, right. [20:40:25] all hail the new scap3 overloards [20:40:27] sync-common doesn't sync common the way sync-file syncs a file [20:40:35] I was afraid it was syncing everything [20:40:42] sync-common is sync-common-local [20:40:45] and sync-file is sync-file-all [20:40:48] it's really "pull-all" [20:41:07] so sync-common from tin doesn't change apaches? [20:41:13] nope [20:41:23] right, that makes sense. I use it on mw1017 from time to time [20:41:28] it changes the local /srv/mediawiki on tin [20:41:29] Wans't quite sure what it does on tin [20:41:33] but I gues it just ^ [20:41:35] yeah [20:42:03] bd808: btw, it seems scap-file has become slower since the tin-mira-and-back switch.Specifically the 'Started sync-masters' step seems to take forever [20:42:21] Maybe something changed over the last 2-3 weeks to cause that? [20:42:22] it's the cdb update on the remote side we think