[00:30:05] 3Phabricator: Blank Phabricator spam task descriptions - https://phabricator.wikimedia.org/T88888#1023983 (10Aklapper) [00:30:57] 3Phabricator: Blank Phabricator spam task descriptions - https://phabricator.wikimedia.org/T88888#1022459 (10Aklapper) > I don't really like that the Spam tag/project is used for both actual spam and spam-related tasks. Project description is clear. So whoever uses it for spam-related tasks shall stop that. :) [00:47:26] 3Phabricator: Make non-standard "opening" links normal links - https://phabricator.wikimedia.org/T88939#1024000 (10Aklapper) p:5Triage>3Volunteer? [02:34:39] Project beta-scap-eqiad build #40884: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/40884/ [02:51:08] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [02:54:48] Yippee, build fixed! [02:54:49] Project beta-scap-eqiad build #40886: FIXED in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/40886/ [03:05:09] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #437: FAILURE in 8,1 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/437/ [03:06:05] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #401: FAILURE in 4,2 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/401/ [03:07:05] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #303: FAILURE in 4,4 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/303/ [03:10:08] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #457: FAILURE in 7,4 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/457/ [03:11:09] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [03:18:06] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #111: FAILURE in 4,9 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/111/ [03:19:11] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #466: FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/466/ [03:28:10] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #393: FAILURE in 9,1 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/393/ [03:30:04] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #175: FAILURE in 3,7 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/175/ [03:31:08] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #472: FAILURE in 7,3 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/472/ [03:31:09] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [03:33:05] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #429: FAILURE in 4,7 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/429/ [03:37:06] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #337: FAILURE in 5,1 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/337/ [03:40:06] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #431: FAILURE in 5,6 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/431/ [03:41:06] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #397: FAILURE in 5,5 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/397/ [03:46:05] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #369: FAILURE in 4,7 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/369/ [03:47:06] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #110: FAILURE in 4,8 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/110/ [03:48:08] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #110: FAILURE in 7,4 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/110/ [03:50:07] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #457: FAILURE in 4,8 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/457/ [03:53:05] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #174: FAILURE in 4,4 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/174/ [03:54:03] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #328: FAILURE in 2,2 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/328/ [03:55:04] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #455: FAILURE in 2,7 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/455/ [03:57:03] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #456: FAILURE in 2,5 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/456/ [03:57:05] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #450: FAILURE in 3,1 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/450/ [03:58:04] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #400: FAILURE in 2,9 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/400/ [05:06:59] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1024128 (10Mattflaschen) [05:09:14] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1024132 (10Mattflaschen) This is mostly done. I'll be able to finish and test the server-side part, then port the gadget, once the bot is created. [05:13:45] 3Engineering-Community, Code-Review, Team-Practices: Organize a Gerrit Cleanup Day - https://phabricator.wikimedia.org/T88531#1024141 (10Quiddity) >>! In T88531#1017429, @Qgil wrote: > This is a query showing unassigned tasks with Patch-For-Review (let's assume that assigned tasks are in good hands, for the sake... [06:36:09] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [07:51:59] 3Engineering-Community, Code-Review, Team-Practices: Organize a Gerrit Cleanup Day - https://phabricator.wikimedia.org/T88531#1024186 (10Qgil) [07:53:22] 3Engineering-Community, Code-Review, Team-Practices: Organize a Gerrit Cleanup Day - https://phabricator.wikimedia.org/T88531#1014128 (10Qgil) @quiddity good catch. Fixed. Do you all agree with the goals of this first terrestrial day, as specified in the description? [08:54:37] Project beta-scap-eqiad build #40922: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/40922/ [09:07:17] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [09:13:17] !sal [09:13:17] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:14:21] 3Continuous-Integration: integration-puppetmaster does not respond to other instances - https://phabricator.wikimedia.org/T88960#1024242 (10hashar) 3NEW [09:14:52] Yippee, build fixed! [09:14:53] Project beta-scap-eqiad build #40924: FIXED in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/40924/ [09:20:04] !log starting puppet agent on integration-puppetmaster [09:20:06] Logged the message, Master [09:27:11] hashar: who we need to fix npm script? [09:27:13] :) [09:27:19] hashar: krinkle? [09:28:18] ok. afk for a while. Lets talk later. [09:28:33] kart_: we might have a better way to run the jobs :-] [09:28:40] will give it a try later on today [09:46:12] 3Continuous-Integration: integration-puppetmaster does not respond to other instances - https://phabricator.wikimedia.org/T88960#1024347 (10hashar) I have stopped the puppet master on integration-puppetmaster and started listening for connections using `netcat -l 8140`. From integration-slave1002 I tried to `te... [09:47:00] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: integration-puppetmaster does not respond to other instances - https://phabricator.wikimedia.org/T88960#1024349 (10hashar) Maybe ops have some idea? :-( [10:18:51] kart_: I am cleaning out the mess [10:21:07] (03CR) 10Hashar: [C: 032] "All jobs are being refreshed." [integration/config] - 10https://gerrit.wikimedia.org/r/189100 (owner: 10Hashar) [10:27:06] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1024429 (10hashar) [10:27:57] (03Merged) 10jenkins-bot: Purge node_modules explicitly [integration/config] - 10https://gerrit.wikimedia.org/r/189100 (owner: 10Hashar) [10:29:29] 3Deployment-Systems, operations: [Trebuchet] Salt times out on parsoid restarts - https://phabricator.wikimedia.org/T63882#651916 (10akosiaris) [10:29:30] 3Continuous-Integration: Bring integration-slave1001.eqiad.wmflabs back in the pool - https://phabricator.wikimedia.org/T88972#1024457 (10hashar) 3NEW [10:30:53] (03CR) 10Hashar: [C: 032] "I have moved the rm command to the jenkins jobs with https://gerrit.wikimedia.org/r/#/c/189100/" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189101 (owner: 10Hashar) [10:31:30] (03CR) 10jenkins-bot: [V: 04-1] npm-set-env.sh: stop purging node_modules there [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189101 (owner: 10Hashar) [10:34:02] (03PS1) 10Hashar: mw-api-siteinfo: module level import at top of file [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189452 [10:34:33] (03PS2) 10Hashar: npm-set-env.sh: stop purging node_modules there [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189101 [10:35:01] (03CR) 10Hashar: [C: 032] mw-api-siteinfo: module level import at top of file [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189452 (owner: 10Hashar) [10:36:03] (03Merged) 10jenkins-bot: mw-api-siteinfo: module level import at top of file [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189452 (owner: 10Hashar) [10:36:05] (03Merged) 10jenkins-bot: npm-set-env.sh: stop purging node_modules there [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189101 (owner: 10Hashar) [10:36:09] (03CR) 10Hashar: "Rebased on top of https://gerrit.wikimedia.org/r/#/c/189452/ which fix the flake8 issues." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189101 (owner: 10Hashar) [10:38:44] (03CR) 10Hashar: "Deployed on production hosts with git-deploy and on labs instance by running git pull in /srv/integration/slave-scripts" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/189101 (owner: 10Hashar) [10:45:10] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1024493 (10Qgil) >>! In T75854#1020525, @Aklapper wrote: >>>! In T75854#1019135, @Qgil wrote: >> and we are not gaining anything from having Elasticsearch. > > The default backend now supports stemming... [10:47:44] 3operations, Project-Creators, Phabricator, Triagers: Broaden the group of users that can create projects in Phabricator - https://phabricator.wikimedia.org/T706#1024495 (10Qgil) @awight, if you need to create projects, just follow https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects and req... [10:49:40] (03PS1) 10Hashar: Migrate cxserver back to regular npm job [integration/config] - 10https://gerrit.wikimedia.org/r/189454 [10:50:30] (03CR) 10Hashar: "That is overkill. I have moved cxserver back to the '{name}-npm' job template with https://gerrit.wikimedia.org/r/#/c/189454/" [integration/config] - 10https://gerrit.wikimedia.org/r/184609 (owner: 10KartikMistry) [10:54:41] 3Phabricator: Task creation by email is unclear - https://phabricator.wikimedia.org/T865#1024524 (10Qgil) @valhallasw, could you upload a screenshot of your test instance with the change, please? [10:58:12] 3Phabricator: Task creation by email is unclear - https://phabricator.wikimedia.org/T865#1024542 (10valhallasw) I don't have my external HD with the phab VM at hand, but the link rendered properly (I assume that's what you're wondering about) [10:58:49] (03CR) 10Hashar: [C: 032] Migrate cxserver back to regular npm job [integration/config] - 10https://gerrit.wikimedia.org/r/189454 (owner: 10Hashar) [11:02:58] 3Phabricator, MediaWiki-Core-Team, Security-Reviews: Install PHPExcel so I can export reports - https://phabricator.wikimedia.org/T152#1024566 (10Aklapper) [11:05:17] (03Merged) 10jenkins-bot: Migrate cxserver back to regular npm job [integration/config] - 10https://gerrit.wikimedia.org/r/189454 (owner: 10Hashar) [11:08:58] 3Phabricator: Enable select Fundraising people to modify policy for tasks - https://phabricator.wikimedia.org/T88762#1024581 (10Qgil) Note that "Fundraising-Backlog" was initially the Fundraising "people" project. There is also "Wikimedia-Fundraising", which is the backlog new bugs... Maybe these two could be or... [11:14:13] 3Zero, Phabricator: Migration of the Zero team to Phabricator - https://phabricator.wikimedia.org/T88978#1024601 (10Qgil) 3NEW [11:15:54] 3Phabricator: Process to request a private project - https://phabricator.wikimedia.org/T823#1024613 (10Qgil) There is an ongoing discussion about the possibility for the Fundraising-Tech team to get permissions to mark some tasks private. See {T88762}. [11:17:43] hashar: thanks! Should I disable jenkins on cxserver/deploy? [11:17:55] kart_: I am looking at it :° [11:18:02] will probably come with a new job template [11:18:07] that do something like: [11:18:19] NODE_PATH="./deploy" npm test [11:19:00] hashar: oh. nice! :) [11:20:00] (03Abandoned) 10KartikMistry: cxserver/deploy: Do not run cxserver-npm [integration/config] - 10https://gerrit.wikimedia.org/r/188755 (owner: 10KartikMistry) [11:20:48] kart_: on Friday I was with my kids at home. Was not ideal to think about the issue :-/ [11:21:11] turns out parsoid has those complicated job templates because there is at least 3 different test jobs [11:21:16] though cxserver just have "npm test" [11:25:22] right [11:26:49] kart_: so cxserver repo is fine now [11:26:58] kart_: the deploy one fails for some other reason though https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/9/console [11:29:08] looking.. [11:29:26] ah. we need node_modules updates which is that failure :/ [11:30:02] kart_: that build output is from https://gerrit.wikimedia.org/r/#/c/187253/ Update node_modules [11:34:28] checking again... [11:34:43] ah found it [11:34:55] when I cd src [11:34:59] that starts failing like that [11:35:36] despite having NODE_PATH set [11:35:55] should we check https://gerrit.wikimedia.org/r/#/c/186801/ first? [11:37:15] kart_: same deal https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/10/console [11:37:54] seems grunt.loadNpmTasks() does not use NODE_PATH :/ [11:39:06] 3Phabricator: Sending email to create tasks doesn't work for me - https://phabricator.wikimedia.org/T1077#1024659 (10Aklapper) @MarkAHershberger: Seeing the last two comments, is this still a problem? [11:39:20] :/ [11:40:11] ah [11:40:34] in the deploy repo using change https://gerrit.wikimedia.org/r/#/c/187253/ . The node_modules/.bin does not comes with grunt [11:41:41] might not be the issue though [11:52:36] hashar: how does it work with parsoid? :) [11:55:05] ok. no grunt there. [11:55:14] no clue :( [11:55:31] I have barely know npm / grunt :( [11:55:44] I barely know .. [11:57:15] hashar: it looks if we can add node_modules patch, following patches should work. [11:57:31] it only cries for 3 modules [11:57:32] well the node_modules patch does not work :/ [11:58:17] is it installing node_modules needed before test? [11:58:32] oh yes. [11:58:32] on the deploy repo no [11:58:33] :( [11:58:45] it assumes all modules are in /node_modules of the deploy repo [11:58:52] and just: NODE_PATH=./node_modules npm test [11:59:07] ensuring the deploy repo has all required modules [11:59:16] those 3 aren't. Right? [12:00:34] there is a node_modules/grunt-contrib-jshint/ [12:01:09] on my local checkout of deploy I do: [12:01:20] git-review -d 187253 [12:01:23] git submodule update --init [12:01:30] export NODE_PATH="`pwd`/node_modules" [12:01:32] cd src [12:01:34] npm test [12:01:39] and grunt can't find them :/ [12:02:23] >> Local Npm module "grunt-contrib-jshint" not found. Is it installed? [12:02:23] >> Local Npm module "grunt-contrib-watch" not found. Is it installed? [12:02:23] >> Local Npm module "grunt-jscs-checker" not found. Is it installed? [12:02:23] :( [12:10:37] kart_: parsoid does not use grunt [12:12:29] yes. realized that. [12:12:41] 11:55 kart_ : ok. no grunt there. [12:12:45] :) [12:19:17] hashar: what should we do as of now? Look like this seems known problem with npm/node [12:20:01] I am looking at grunt source code [12:20:18] grunt lib/grunt/task.js task.loadNpmTasks() function does not honor NODE_PATH :((( [12:20:32] it only look in the current directory path.resolve('node_modules'); [12:20:51] scary :/ [12:20:53] so maybe we should not cd /src [12:21:26] the Parsoid jobs do cd /src [12:21:34] and invoke npm test there [12:26:47] kart_: worth case we can add a symlink :D [12:27:06] that is proerpyl not going to work properly [12:27:19] much better stop cd src [12:27:44] hashar: how about adding new Gruntfile? [12:27:50] not good idea. [12:28:17] hashar: another question: why we should rerun tests at all? (which is already done in cxserver!) [12:28:45] to ensure the deploy repo has the proper node modules [12:30:26] let me try with the generic job [12:33:49] https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/12/console [12:36:58] hashar: I got something working :) [12:37:01] oh [12:37:18] well [12:37:24] I have updated the job meanwhile [12:37:31] made it to use npm install [12:37:35] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:37:36] which defeat the whole purpose [12:37:45] https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/13/console [12:37:56] hashar: ^^ [12:38:06] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1024711 (10Aklapper) Oh, I'm totally fine with temporarily changing the backend to the SQL-backed standard search. @Chad also proposes that in https://lists.wikimedia.org/pipermail/wikitech-l/2015-Februa... [12:38:10] yeah it ran npm install [12:38:14] sorry was messing with the job [12:38:18] ah [12:38:25] can you run above again? [12:38:26] https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/14/console :D [12:38:30] runs localy [12:38:45] blah [12:39:49] can you remove cd src/? [12:39:49] I will try without cd src [12:39:53] yeah [12:41:19] https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/15/console \O/ [12:41:24] that run no more cd [12:43:26] but then that fails for parsoid [12:43:27] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1024713 (10chasemp) Very sick today but in general I can do this, probably wed as it will create an unknown period of search outage. I wanted to give @springle a heads up since any fallout likely affect... [12:44:14] :/ [12:44:25] hashar: meanwhile, can you force merge https://gerrit.wikimedia.org/r/#/c/189203/ ? [12:52:38] (03PS1) 10Hashar: (WIP) Hack for npm oid jobs (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/189473 [12:52:50] kart_: so I have a ugly hack in progress :D [12:53:46] kart_: some jscs could use to be fixed apparently https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/18/console [12:54:38] ah no forget me [12:54:51] oh here: https://gerrit.wikimedia.org/r/#/c/186801/ [12:54:58] confusing dependencies :/ [12:55:25] I made the job to no more cd src [12:56:01] Is Parsoid happy with it? [12:56:40] * hashar yeah I added a hack for Parsoid https://gerrit.wikimedia.org/r/189473 [12:56:48] that is becoming very messy [12:57:42] hashar: we need force merge 186801. [12:58:08] I am trying to understand why https://gerrit.wikimedia.org/r/#/c/187253/ does not work [12:58:14] Fatal error: Unable to find Gruntfile. [12:58:24] though it works on my machine [12:58:29] because I added Gruntfile after it :) [12:58:47] in 189203 [12:59:13] one of them need to forge merge [12:59:18] ohhh [12:59:21] well [12:59:21] https://gerrit.wikimedia.org/r/#/c/189203/ [12:59:24] it is not ideal [12:59:31] cause that duplicates the logic from the source repo [12:59:55] should we symlink everything then? [13:01:39] In future, I should delete most of those tests and only make sure that node_modules are proper. [13:01:55] Everything else tested in cxserver. [13:09:24] AHHH [13:09:24] https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/22/console [13:09:50] :) [13:10:58] (03PS2) 10Hashar: (WIP) Hack for npm oid jobs (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/189473 [13:11:32] jenkins is happy finally https://gerrit.wikimedia.org/r/#/c/187253/ [13:11:35] sorry for all the mess [13:11:58] you will probably want to force merge https://gerrit.wikimedia.org/r/#/c/186801/ ( Update cxserver to a97063a ) [13:11:58] hashar: no worry and thanks for all help. [13:12:05] Yes [13:12:05] not even [13:12:08] it +2 [13:14:14] kart_: I am off for lunch [13:14:21] been starving for an hour + :D [13:14:23] hashar: enjoy! [13:14:24] :) [13:14:30] will get to revisit that mess later on [13:14:55] most probably we would want parsoid/deploy and cxserver/deploy to work with the simple job template [13:15:00] but that is a different mess [13:15:08] good evening! [13:27:55] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [13:37:58] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.006 second response time [13:43:32] 3Engineering-Community, Code-Review, Team-Practices: Organize a Gerrit Cleanup Day - https://phabricator.wikimedia.org/T88531#1024746 (10Qgil) [14:11:39] hashar: Q: how can I add patches submitted to cxserver and cxserver/deploy shown in #mediawiki-i18n IRC channel? [14:11:53] grrrit-wm ? [14:12:06] iirc under labs/tools/grrrit.git [14:12:22] ie gerrit [14:12:26] cool. [14:17:22] hashar: patch submitted. [14:17:38] kart_: I am not sure whom review / deploy them [14:19:07] need someone being a member of the tools labs project lolrrit-wm [14:20:39] hashar: Yuvi is the only one I know ;) [14:20:51] but he seems away. [14:21:00] i don't even know how to check group members [14:21:40] added dzahn. should be all good now! [14:21:46] :) [14:22:02] ah getent group tools.lolrrit-wm [14:32:54] hashar: added all persons? ;) :) [14:33:03] yeah [14:36:05] (03PS1) 10Hashar: Experimental operations-mw-config-composer job [integration/config] - 10https://gerrit.wikimedia.org/r/189482 (https://phabricator.wikimedia.org/T85947) [14:37:01] (03CR) 10Hashar: [C: 032] Experimental operations-mw-config-composer job [integration/config] - 10https://gerrit.wikimedia.org/r/189482 (https://phabricator.wikimedia.org/T85947) (owner: 10Hashar) [14:37:10] (03CR) 10jenkins-bot: [V: 04-1] Experimental operations-mw-config-composer job [integration/config] - 10https://gerrit.wikimedia.org/r/189482 (https://phabricator.wikimedia.org/T85947) (owner: 10Hashar) [14:43:41] (03Merged) 10jenkins-bot: Experimental operations-mw-config-composer job [integration/config] - 10https://gerrit.wikimedia.org/r/189482 (https://phabricator.wikimedia.org/T85947) (owner: 10Hashar) [14:49:43] hashar: ugh, I hate sge :P [14:49:52] restarting taskes aaaaages [14:50:39] hashar: ^ [14:51:46] valhallasw`cloud: our hero :-] [14:51:50] kart_: ^^^ [14:52:11] 3Deployment-Systems, operations, Scrum-of-Scrums: Update wikitech wiki with deployment train - https://phabricator.wikimedia.org/T70751#1024835 (10akosiaris) Let's move this forward again. Getting a server to host wikitech is easy. Actually hosting wikitech on a different server than virt1000... I have no idea i... [14:56:49] hashar: for? [14:57:12] oh. irc noise :) [14:57:43] hashar: valhallasw`cloud was very much helpful in deployment last time :) [14:57:57] so I +1 to your statement! [15:08:31] I try to be :-) although I should fabric-ify it a bit [15:11:27] hashar: anyway, added myself to gerrit-reviewer-bot for labs/tools/grrrit, so I should get notified in the future [15:13:54] valhallasw`cloud: awesome [15:14:56] Project beta-scap-eqiad build #40960: FAILURE in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/40960/ [15:35:03] (03CR) 10Hashar: "> > "I am not sure whether we need a qunit-mobile job as well."" [integration/config] - 10https://gerrit.wikimedia.org/r/188527 (https://phabricator.wikimedia.org/T86866) (owner: 10Hashar) [15:35:08] (03PS2) 10Hashar: Create qunit job for the Thanks extension [integration/config] - 10https://gerrit.wikimedia.org/r/188527 (https://phabricator.wikimedia.org/T86866) [15:36:37] (03PS3) 10Hashar: Create qunit job for the Thanks extension [integration/config] - 10https://gerrit.wikimedia.org/r/188527 (https://phabricator.wikimedia.org/T86866) [15:50:38] !sal [15:50:38] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:51:28] !log integration : allowed ssh from gallium 208.80.154.135/32 to the instances [15:51:30] Logged the message, Master [15:52:43] hashar: did Jenkins get sick overnight US time? all my builds went red, but I think it's back now [15:53:43] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1024943 (10coren) 5Open>3Resolved a:3coren The project security group did not (was changed not to?) include allowing ss... [15:54:13] 3Continuous-Integration: Bring integration-slave1001.eqiad.wmflabs back in the pool - https://phabricator.wikimedia.org/T88972#1024948 (10hashar) 5Open>3Resolved a:3hashar I have pooled the node back up. The OpenStack security rules were broken and missed a rule to allow ssh from gallium to the instances. [15:54:15] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1024951 (10hashar) [15:56:15] Yippee, build fixed! [15:56:15] Project beta-scap-eqiad build #40962: FIXED in 22 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/40962/ [15:59:00] Yippee, build fixed! [15:59:00] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #467: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/467/ [15:59:56] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1024977 (10hashar) 5Resolved>3Open The integration labs project was missing a security rule to allow ssh from gallium for... [16:06:39] (03CR) 10Hashar: [C: 032] "Jobs deployed :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/188527 (https://phabricator.wikimedia.org/T86866) (owner: 10Hashar) [16:06:47] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1025012 (10coren) The puppetmaster issue did appear related: adding an explcit rule to allow it fixed the immediate problem,... [16:07:22] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1025014 (10coren) 5Open>3Resolved [16:07:44] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1025015 (10hashar) The deployment-prep labs project also uses a local puppetmaster but it does not need any specific security... [16:10:22] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1025027 (10akosiaris) @hashar root@integration-slave1002:~# telnet 10.68.16.96 8140 Trying 10.68.16.96... Connected to 10.68... [16:11:56] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#1025039 (10JanZerebecki) [16:11:58] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1025041 (10coren) Yeah, things are working fine now with an explicit rule - but the necessity of //having// the explicit rule... [16:13:06] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1024242 (10hashar) Filled another ticket for investigation of the underlying issue: {T88995}. [16:15:05] (03Merged) 10jenkins-bot: Create qunit job for the Thanks extension [integration/config] - 10https://gerrit.wikimedia.org/r/188527 (https://phabricator.wikimedia.org/T86866) (owner: 10Hashar) [16:17:59] 3Continuous-Integration, Thanks: Thanks needs Jenkins qunit job - https://phabricator.wikimedia.org/T86866#1025064 (10hashar) 5Open>3Resolved a:3hashar Thanks now has both a qunit and a qunit-mobile job ( &useformat=mobile ) [16:18:01] 3Continuous-Integration, Mobile-Web, Thanks, §Collaboration-Team-Sprint-P-2015-02-11: Thanks is broken again (Mobile Thanks needs qunit tests) - https://phabricator.wikimedia.org/T86687#1025067 (10hashar) [16:23:44] (03CR) 10Hashar: [C: 032] "Lovely guidances :]" [selenium] - 10https://gerrit.wikimedia.org/r/188251 (owner: 10Dduvall) [16:24:23] 3Phabricator: Prevent private information being leaked via Herald notifications - https://phabricator.wikimedia.org/T493#1025120 (10scfc) What's stalling this? [16:25:26] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#1025124 (10JanZerebecki) Could someone take this up? It seems counter productive to work on workarounds when the we could deploy the real fix instead. [16:28:50] (03CR) 10Hashar: [C: 031] Step definitions refactored for the EAL [selenium] - 10https://gerrit.wikimedia.org/r/188102 (owner: 10Dduvall) [16:44:22] ^d: Can you grant Niharika +2 on wikimedia/iegreview? -- https://www.mediawiki.org/w/index.php?title=Gerrit/Project_ownership&oldid=1394988#.2B2_for_Niharika_Kohli_in_wikimedia.2Fiegreview [16:45:00] <^d> done [16:45:08] thanks [16:49:36] (03CR) 10Florianschmidtwelzow: "@Jon, @Antoine: Did you have time to talk about it last weeks? :)" [integration/config] - 10https://gerrit.wikimedia.org/r/181693 (https://phabricator.wikimedia.org/T74794) (owner: 10Jdlrobson) [17:07:30] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1025207 (10hashar) 3NEW [17:09:58] 3Continuous-Integration, Mobile-Web: Publish our JS Documentation - https://phabricator.wikimedia.org/T74794#1025231 (10hashar) We have too many different ways to generate documentation. Each time we end up having to create yet another job and Jon patch is an illustration ( https://gerrit.wikimedia.org/r/#/c/18... [17:11:53] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1025207 (10hashar) Alternatively, we can make the command to be a variable passed to the job. [17:17:56] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1025271 (10hashar) a:3hashar I have posted on wikitech-l https://lists.wikimedia.org/pipermail/wikitech-l/2015-February/080670.html I will use the convention 'make docs' unless wikitech-l discussio... [17:19:48] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1025276 (10hashar) p:5Triage>3Normal [17:20:12] 3Continuous-Integration, Mobile-Web: Publish our JS Documentation - https://phabricator.wikimedia.org/T74794#1025278 (10hashar) [17:20:13] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1025207 (10hashar) [17:21:25] 3Continuous-Integration, Mobile-Web: Publish our JS Documentation - https://phabricator.wikimedia.org/T74794#761166 (10hashar) That is merely to avoid pilling more tech debt on top of the existing debt. I have mailed [[ https://lists.wikimedia.org/pipermail/wikitech-l/2015-February/080670.html | wikitech-l ]] ab... [17:25:00] 3Phabricator, Wikimedia-Bugzilla: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#1025297 (10Dzahn) a:5JohnLewis>3Dzahn [17:25:11] 3Phabricator, Wikimedia-Bugzilla: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#939794 (10Dzahn) [17:27:22] I've written and merged "sleep 1" five times today. I feel dirty. [17:38:37] 3Phabricator: Enable select Fundraising people to modify policy for tasks - https://phabricator.wikimedia.org/T88762#1025327 (10atgo) @qgil I guess we could do that, but I'm not sure how that addresses this need? As @chasemp said, our confidentiality needs are somewhat fluid so a static group would miss the mark. [17:46:39] 3Phabricator: Enable select Fundraising people to modify policy for tasks - https://phabricator.wikimedia.org/T88762#1025349 (10Krenair) So you want to be able to define custom visibility policies with only user-based access on each object, not against just a restricted-join project? That sounds like a good way... [17:57:09] I'll do real virtual face-to-face introductions tomorrow, but everyone, meet Tyler ( thcipriani ). Tyler, meet everyone: chrismcmahon twentyafterfour ryasmeen|Away ^d (and a few others not online) [17:57:33] howdy all! [17:57:43] also, announcement email to follow (still to write ;) ) [17:57:58] hi thcipriani ! I understand you're in Colorado near Lyons, where I lived for a long time. [17:58:15] * greg-g goes to his next meeting :( [17:58:30] yeah, I'm in Longmont, so like 15 mins away. I love Lyons, though. [17:59:13] 3Phabricator: Decide on "Needs Volunteer" Priority field value in Phabricator - https://phabricator.wikimedia.org/T78617#1025359 (10Aklapper) [17:59:36] Once google drops an office in Boulder, I'm sure to be gentrified out to Berthoud, though. [18:00:27] thcipriani: ha, seriously, a big Google in Boulder? I know SketchUp has been there for years, but not heard of any other Google stuff coming in. [18:01:08] chrismcmahon: it's coming: http://www.dailycamera.com/news/boulder/ci_27072010/boulder-planning-board-challenges-google-campus-plan [18:01:37] Ann Arbor got a Google office, but it was only adsense related, and it was a bad choice all around, actually (they left recently) [18:02:37] yeah, I can't imagine that this is going to make Boulder a _more_ livable city. [18:02:53] 3Phabricator: Decide on "Needs Volunteer" Priority field value in Phabricator - https://phabricator.wikimedia.org/T78617#1025377 (10Dzahn) I would suggest do simply call the priorities: Lowest, Low, Normal, High and Highest. So replace "needs volunteer" with lowest and 'unbreak now' with highest. Other name... [18:03:39] * ^d waves [18:05:31] * thcipriani returns wave [18:06:33] thcipriani: Boulder always did make me a little crazy, in a lot of ways. [18:10:59] chrismcmahon: whatever drove you crazy, is now worse, undoubtedly. Boulder is mostly rich folks who look like they're about to go camping. [18:17:26] 3Phabricator: Decide on "Needs Volunteer" Priority field value in Phabricator - https://phabricator.wikimedia.org/T78617#1025431 (10matmarex) [18:19:02] 3Phabricator: Decide on "Needs Volunteer" Priority field value in Phabricator - https://phabricator.wikimedia.org/T78617#849449 (10matmarex) [18:20:42] hi ryasmeen I updated several VE tests but the build is still red because of https://phabricator.wikimedia.org/T89007 [18:22:10] (03CR) 10Dduvall: [C: 032] Step definitions refactored for the EAL [selenium] - 10https://gerrit.wikimedia.org/r/188102 (owner: 10Dduvall) [18:23:24] (03CR) 10jenkins-bot: [V: 04-1] Documentation on upgrading to 1.0.0 [selenium] - 10https://gerrit.wikimedia.org/r/188251 (owner: 10Dduvall) [18:23:27] chrismcmahon: thanks a lot, I am working on a patch too for cite and media test [18:23:43] ryasmeen: those should all pass in master now [18:23:55] I fixed them [18:24:04] ah ok [18:24:08] this morning [18:24:21] nice! [18:30:55] ryasmeen: I have a suggestion for you: would you take a look at the references.feature test? It is a real mess right now and it covers very little function. [18:33:29] chrismcmahon: sure, I will [18:36:11] Yippee, build fixed! [18:36:12] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #438: FIXED in 21 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/438/ [18:39:14] thcipriani: "rich folks who look like they're about to go camping" :) [18:39:37] Yippee, build fixed! [18:39:37] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #402: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/402/ [18:40:03] greg-g: o/ [18:40:18] thcipriani: got your email stuff now, right? [18:40:24] yessir [18:40:30] sweet [18:55:06] thcipriani: welcome [18:56:10] thanks! [19:00:06] 3Phabricator: Decide on "Needs Volunteer" Priority field value in Phabricator - https://phabricator.wikimedia.org/T78617#1025573 (10Qgil) >>! In T78617#1025377, @Dzahn wrote: > I would suggest do simply call the priorities: Lowest, Low, Normal, High and Highest. Looking at the description of the problem, this... [19:02:27] (03PS2) 10Dduvall: Step definitions refactored for the EAL [selenium] - 10https://gerrit.wikimedia.org/r/188102 [19:08:03] 3VisualEditor, Beta-Cluster: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1025597 (10Ryasmeen) This seems to get resolved with https://gerrit.wikimedia.org/r/188725. I am closing it for now,re-open if anyone experiences this issue. [19:08:29] 3VisualEditor, Beta-Cluster: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1025598 (10Ryasmeen) 5Open>3Resolved [19:09:47] 3Phabricator, Wikimedia-Bugzilla: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#1025603 (10Dzahn) p:5Volunteer?>3Low [19:11:47] Yippee, build fixed! [19:11:48] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #304: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/304/ [19:13:32] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1025622 (10Qgil) [19:17:45] Yippee, build fixed! [19:17:46] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #458: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/458/ [19:20:26] fix all the builds [19:22:40] Yippee, build fixed! [19:22:41] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #112: FIXED in 2 min 46 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/112/ [19:24:27] (03CR) 10Dduvall: Step definitions refactored for the EAL [selenium] - 10https://gerrit.wikimedia.org/r/188102 (owner: 10Dduvall) [19:24:35] (03CR) 10Dduvall: [C: 032] Step definitions refactored for the EAL [selenium] - 10https://gerrit.wikimedia.org/r/188102 (owner: 10Dduvall) [19:25:29] (03Merged) 10jenkins-bot: Step definitions refactored for the EAL [selenium] - 10https://gerrit.wikimedia.org/r/188102 (owner: 10Dduvall) [20:01:36] PROBLEM - Host deployment-elastic06 is DOWN: CRITICAL - Host Unreachable (10.68.17.79) [20:01:54] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: integration labs instance can not rsync/ssh to integration-publisher [10.68.16.255] instance - https://phabricator.wikimedia.org/T89026#1025736 (10hashar) 3NEW [20:02:10] PROBLEM - Host deployment-elastic07 is DOWN: CRITICAL - Host Unreachable (10.68.17.82) [20:02:35] 3operations, Continuous-Integration, Wikimedia-Labs-Infrastructure: labs security rules are flappy / invalid cause network communications issues - https://phabricator.wikimedia.org/T88960#1024242 (10hashar) The doc publishing jobs are failing as well and there is no workaround for it :( T89026 [20:03:32] PROBLEM - Host deployment-elastic08 is DOWN: CRITICAL - Host Unreachable (10.68.17.2) [20:04:11] (03CR) 10Hashar: "The jobs publishing documentation are broken since Friday due to an issue with labs security rules. Filled as T89026." [selenium] - 10https://gerrit.wikimedia.org/r/188102 (owner: 10Dduvall) [20:04:57] (03PS1) 10Legoktm: Run "composer update" instead of "composer install" [integration/config] - 10https://gerrit.wikimedia.org/r/189543 [20:09:20] (03CR) 10BryanDavis: [C: 031] "Using the update command rather than the install command will allow repos that are using a versioned composer.lock to still install additi" [integration/config] - 10https://gerrit.wikimedia.org/r/189543 (owner: 10Legoktm) [20:12:07] (03CR) 10Hashar: "I barely know the different between install and update. I guess if you get some composer gurus to endorse this patch we can get it merged " [integration/config] - 10https://gerrit.wikimedia.org/r/189543 (owner: 10Legoktm) [20:15:03] 3Engineering-Community, Code-Review: Organize a Gerrit Cleanup Day - https://phabricator.wikimedia.org/T88531#1025761 (10Awjrichards) [20:20:46] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: integration labs instance can not rsync/ssh to integration-publisher [10.68.16.255] instance - https://phabricator.wikimedia.org/T89026#1025802 (10hashar) [20:33:36] 3Engineering-Community, Code-Review: Organize a Gerrit Cleanup Day - https://phabricator.wikimedia.org/T88531#1025864 (10hashar) Slightly out topic but giving some context about how I do my reviews: For my review I use a [[ https://gerrit.wikimedia.org/r/#/q/status:open+reviewer:self+label:code-review%253D0%252... [20:35:06] 3MediaWiki-General-or-Unknown, Project-Creators, Phabricator, MediaWiki-Core-Team: Allow to search tasks about MediaWiki core and core only (create MediaWiki umbrella project?) - https://phabricator.wikimedia.org/T76942#1025866 (10matmarex) >>! In T76942#946690, @Nemo_bis wrote: >> rename MediaWiki-General-or-Un... [20:35:41] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: integration labs instance can not rsync/ssh to integration-publisher [10.68.16.255] instance - https://phabricator.wikimedia.org/T89026#1025875 (10hashar) a:3Andrew This has been fixed on spot by Andrew. See T88995 for details. [20:35:48] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: integration labs instance can not rsync/ssh to integration-publisher [10.68.16.255] instance - https://phabricator.wikimedia.org/T89026#1025877 (10hashar) 5Open>3Resolved [20:37:47] !log deleted /tmp on integration slaves 1006 1007 and 1008. Filled with npm temp directories [20:37:51] Logged the message, Master [20:38:36] !log reconnected jenkins slave agents 1006 1007 and 1008 [20:38:38] Logged the message, Master [20:39:15] Yippee, build fixed! [20:39:15] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #176: FIXED in 53 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/176/ [20:39:19] Yippee, build fixed! [20:39:20] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #394: FIXED in 58 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/394/ [20:41:22] RECOVERY - Host deployment-elastic08 is UP: PING OK - Packet loss = 0%, RTA = 0.82 ms [20:42:29] 3Phabricator: Get rid of "Export to Excel" - https://phabricator.wikimedia.org/T257#1025894 (10Dzahn) It now says "(disabled)" to clarify that. I merged the change above. [20:43:05] 3Phabricator: Get rid of "Export to Excel" - https://phabricator.wikimedia.org/T257#1025895 (10Dzahn) >>! In T257#3253, @flimport wrote: > **aklapper** wrote on `2014-06-18 00:35:57 (UTC)` > > Installing PHPExcel on the server is already covered by T243 ^ wrong ticket number, that's something else [20:43:40] (03CR) 10Hashar: "Labs fixed! I have rerun the documentation job from Jenkins web interface so the published doc should be up to date." [selenium] - 10https://gerrit.wikimedia.org/r/188102 (owner: 10Dduvall) [20:44:09] 3Phabricator: Get rid of "Export to Excel" - https://phabricator.wikimedia.org/T257#1025897 (10valhallasw) >>! In T257#1025895, @Dzahn wrote: >>>! In T257#3253, @flimport wrote: >> **aklapper** wrote on `2014-06-18 00:35:57 (UTC)` >> >> Installing PHPExcel on the server is already covered by T243 > > ^ wrong t... [20:44:29] 3Phabricator: Get rid of "Export to Excel" - https://phabricator.wikimedia.org/T257#1025898 (10valhallasw) 5Open>3Resolved a:3valhallasw [20:44:56] Yippee, build fixed! [20:44:57] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #430: FIXED in 2 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/430/ [20:49:37] 3Continuous-Integration, Wikimedia-Labs-Infrastructure: integration labs instance can not rsync/ssh to integration-publisher [10.68.16.255] instance - https://phabricator.wikimedia.org/T89026#1025926 (10Andrew) [20:51:44] thcipriani: how's the reading/first day stuff going? any questions? [20:52:27] greg-g: just setting up my contact info now, getting on mailing lists, etc. No questions yet. Thanks for checking in though. [20:52:27] Yippee, build fixed! [20:52:27] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #338: FIXED in 7 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/338/ [20:52:35] Yippee, build fixed! [20:52:35] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #473: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/473/ [20:53:31] Yippee, build fixed! [20:53:31] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #398: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/398/ [20:53:43] Yippee, build fixed! [20:53:44] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #432: FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/432/ [20:56:19] thcipriani: no problem, I know it's time consuming :) [20:56:46] it's good. Makes me feel productive on my first day. [20:59:14] ^d, we noticed something funny happening with opensearch on beta - https://phabricator.wikimedia.org/P274 [20:59:43] <^d> search is completely fucked up on beta [21:00:07] does the prefix search rely on the full search system? [21:00:22] this might be a special kind of fucked up [21:00:27] <^d> It uses the same search engine. [21:00:35] <^d> So when it can't fetch from the cache, boom [21:00:40] ok [21:00:45] <^d> I'm surprised you can even hit it right now, it has no indexes [21:00:53] I tested this from deployment-bastion and got useful results [21:01:07] but it appears that the actual app servers don't return useful results [21:01:15] Yippee, build fixed! [21:01:15] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #370: FIXED in 46 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/370/ [21:01:33] the search works but what the api returns is garbled [21:01:51] <^d> ES shouldn't be returning anything except 404s right now. [21:01:57] (for some value of "works") [21:02:04] Yippee, build fixed! [21:02:05] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #111: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/111/ [21:02:41] <^d> I was rebuilding all of ES on beta but I started hitting bugs. [21:02:47] <^d> So it's kind of very broken still [21:02:57] * ^d has been working on it all day [21:05:30] PROBLEM - Host deployment-elastic08 is DOWN: CRITICAL - Host Unreachable (10.68.17.2) [21:06:38] <^d> 08? /me stabs [21:11:15] 3Release-Engineering, Team-Practices: RelEng team offsite - https://phabricator.wikimedia.org/T89036#1026005 (10greg) 3NEW [21:18:14] (03CR) 10Legoktm: [C: 031] "I deployed the change and the tests pass: https://integration.wikimedia.org/ci/job/operations-mw-config-composer/3/console" [integration/config] - 10https://gerrit.wikimedia.org/r/189543 (owner: 10Legoktm) [21:18:20] 3Phabricator: Enable select Fundraising people to modify policy for tasks - https://phabricator.wikimedia.org/T88762#1026036 (10atgo) I'm happy to start with visibility based on a "people" project as a first pass if that makes people more comfortable. If we get into problems from there, we can revisit. [21:18:22] hashar: ^ [21:20:33] 3Release-Engineering, Team-Practices: RelEng team offsite - https://phabricator.wikimedia.org/T89036#1026039 (10greg) [21:20:39] legoktm: my hero :] [21:21:47] legoktm: I can +2. Jenkins is already updated right? [21:22:38] bd808: yup! [21:22:55] did you do all the effected jobs? [21:23:02] oh no. [21:23:04] RECOVERY - Host deployment-elastic07 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [21:23:06] just operations. [21:23:19] 3Release-Engineering, Team-Practices: RelEng team offsite - https://phabricator.wikimedia.org/T89036#1026048 (10hashar) [21:23:32] bd808: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Example is there a way I can deploy multiple jobs at once? [21:23:38] legoktm: you will want to refresh all impacted Jenkins jo [21:23:39] bs [21:23:45] and hope they are all still working :] [21:23:54] you can use wildcard [21:24:02] '*-composer' ? [21:24:05] jenkins-jobs update config/jjb *composer* [21:24:08] RECOVERY - Host deployment-elastic06 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [21:24:13] ok [21:24:35] * legoktm odes [21:24:38] does* [21:24:44] do I need to be !log'ing this anywhere? [21:25:06] legoktm: just comment in Gerrit [21:25:07] You can !log in #-releng if you feel fancy [21:25:23] or what bryan said :-) [21:25:30] Yippee, build fixed! [21:25:30] <^d> The only !log that matters is prod !log :) [21:25:31] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #111: FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/111/ [21:25:56] * bd808 just noticed that he is in #-releng at the moment [21:25:58] next hackathon I should come with CI badges to distributes to folks [21:26:10] CI barnstar! [21:26:16] And a beta barnstar [21:26:25] (03CR) 10Legoktm: "Deployed all the affected jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/189543 (owner: 10Legoktm) [21:26:26] RECOVERY - Host deployment-elastic08 is UP: PING OK - Packet loss = 0%, RTA = 1.27 ms [21:26:29] I am sure I can make Zuul to report a barnstar in Gerrit [21:26:31] bd808: ^ ready for +2 now [21:26:49] <^d> lol barnstars [21:27:00] oh there is only 3 jobs [21:27:06] (03CR) 10BryanDavis: [C: 032] "merging after legoktm did all the hard work" [integration/config] - 10https://gerrit.wikimedia.org/r/189543 (owner: 10Legoktm) [21:27:49] hashar: you should deputize legoktm by adding him to the integration gerrit group [21:29:06] (03CR) 10Hashar: [C: 032] "For your tirelessly work on Jenkins, CI, tests, Librarifizification, here is the official Continuous integration barnstar!" [integration/config] - 10https://gerrit.wikimedia.org/r/189543 (owner: 10Legoktm) [21:29:24] legoktm: here is your barnstar https://gerrit.wikimedia.org/r/#/c/189543/ [21:29:35] hashar: :> ty! [21:29:47] note [21:29:57] legoktm: zuul configuration changes have to be deployed manually [21:30:15] until one have time to waste to have Zuul changes handled via Trebuchet [21:31:02] !log Deputized legoktm to the Gerrit 'integration' group. Brings +2 on integration/* repos. [21:31:05] Logged the message, Master [21:31:25] Krinkle: ^^^ :-] [21:31:47] ok, I don't think I have access to do that.. [21:32:05] (that == deploy zuul updates) [21:32:08] just an operations/puppet patch away [21:32:09] need shell access on gallium.wikimedia.org which hosts the zuul server [21:32:23] then something like: sudo su - zuul [21:32:31] cd /etc/zuul/wikimedia && git pull [21:32:52] 3Engineering-Community, Code-Review: Organize a Gerrit Cleanup Day - https://phabricator.wikimedia.org/T88531#1026070 (10scfc) Changing the default Gerrit front page makes sense to me. To avoid confusion, I would rename the table caption though ("Incoming open reviews" or something similar). [21:33:16] (03PS1) 10Matanya: zuul: jenkins -1 on PS not passing lint in ops/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/189589 [21:33:18] (03Merged) 10jenkins-bot: Run "composer update" instead of "composer install" [integration/config] - 10https://gerrit.wikimedia.org/r/189543 (owner: 10Legoktm) [21:34:47] Project beta-scap-eqiad build #40998: FAILURE in 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/40998/ [21:37:20] (03CR) 10Hashar: "Hey Matanya, good to see you around :]" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (owner: 10Matanya) [21:37:25] stupid me [21:37:42] it is almost 11pm and I have forgot to take some time for me [21:38:13] hi hashar i saw your mail, hance prepare that patch :) [21:38:43] \O/ [21:39:11] and the reason the diff was before that link to phab was /me stupid, commit before setting the remote [21:39:12] matanya: a good share of errors are ignored though [21:39:17] and still have to clear out the warnings [21:39:23] but that is already a good step forward [21:39:54] I know, i was planning to continue with that work in the past, but good things in real life prevented that is recent months [21:40:34] *in recent [21:41:36] (03PS2) 10Matanya: zuul: jenkins -1 on PS not passing lint in ops/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/189589 [21:43:22] matanya: glad to know they were "good" things :D [21:43:31] some people continued on your work [21:43:39] I think most is in modules nowadays [21:43:44] puppet-lint is more or less used [21:44:01] namely daniel and andrew if i recall correcly [21:45:13] (03CR) 10Hashar: zuul: jenkins -1 on PS not passing lint in ops/puppet (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (owner: 10Matanya) [21:45:19] (03PS3) 10Hashar: zuul: jenkins -1 on PS not passing lint in ops/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (owner: 10Matanya) [21:45:34] matanya: I don't think phab: t1234 works [21:45:35] (03CR) 10Krinkle: [C: 04-1] "Please describe the change in the commit message subject. I can't tell whether the subject line describes a bug being addressed, an undesi" [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (owner: 10Matanya) [21:45:56] matanya: we all are using Bug: T####### [21:46:25] https://www.mediawiki.org/wiki/Git/Commit_message_guidelines :) [21:46:29] at least the yaml hack is fine :] [21:46:52] Krinkle: I am heading bed. Just to let you know that the labs security rules got broken on integration project but are all fine since like an hour or so ago [21:46:53] Krinkle: yeah, yeah, i'm rusty [21:46:54] puppet is back! [21:47:25] hashar: It was broken by andrew I suspect, but puppet should auto=fix that because the gallium ferm rule is in puppet integrationr ole [21:47:26] role [21:47:59] there are two levels of fire walling though [21:48:27] the one broken was Security Rules in openstackmanager. The rule that allow instances to communicate between each other got removed [21:49:02] (03PS4) 10Matanya: zuul: Make jenkins vote on patches not passing lint check [integration/config] - 10https://gerrit.wikimedia.org/r/189589 [21:49:56] Krinkle: hope it is better now [21:50:11] shouldn't push patches at midnight [21:50:39] (03CR) 10Hashar: "@Jon, we had a quick talk earlier today in #wikimedia-mobile. I have commented on the task T74794 and opened a proposal ( T88999 && wikite" [integration/config] - 10https://gerrit.wikimedia.org/r/181693 (https://phabricator.wikimedia.org/T74794) (owner: 10Jdlrobson) [21:51:14] great [21:51:22] I am off to bed finally [21:51:27] see you tomorrow! [21:51:28] night hashar [21:54:20] (03PS5) 10Krinkle: zuul: Make operations-puppet-puppetlint-lenient voting [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (https://phabricator.wikimedia.org/T87132) (owner: 10Matanya) [21:55:05] much better, thanks Krinkle [21:55:09] (03CR) 10Krinkle: [C: 031] "Updated commit message to capitalise Bug in the footer, and reflected the relevant job or repo in the subject ("lint" in general is alread" [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (https://phabricator.wikimedia.org/T87132) (owner: 10Matanya) [21:56:21] Dumb question of the day: How do I get a wikitech account with the same name as all my other accounts: TCipriani (WMF)? [21:57:14] thcipriani: sign up with it ? [21:57:17] <^d> (WMF) names are banned on wikitech :) [21:57:33] matanya: heh, yeah, gave that a shot :) [21:57:40] ask an admin to create it for you ? [21:57:50] <^d> Nobody has WMF names on wikitech :) [21:58:16] cool, that's all I needed to know, thanks ^d [21:59:35] better method: when you have an issue, poke ^d [22:01:40] ^d: Is there a Phabricator task for "search broken on Beta Cluster"? [22:01:49] btw ^d and thcipriani :Fabrice Florin (WMF) and LVilla (WMF) do exist [22:02:06] and Philippe (WMF) too [22:06:39] 3VisualEditor, Beta-Cluster: Search on the Beta Cluster is broken, and thus VE no longer shows suggestion list for Templates upon typing into Add template selector - https://phabricator.wikimedia.org/T89007#1026183 (10Jdforrester-WMF) [22:07:46] 3VisualEditor, Beta-Cluster: Search on the Beta Cluster is broken, and thus VE no longer shows suggestion list for Templates upon typing into Add template selector - https://phabricator.wikimedia.org/T89007#1025364 (10Jdforrester-WMF) Not sure where (if?) the wider Beta Cluster issue is being tracked. [22:11:24] 3VisualEditor, Beta-Cluster: Search on the Beta Cluster is broken, and thus VE no longer shows suggestion list for Templates upon typing into Add template selector - https://phabricator.wikimedia.org/T89007#1026205 (10Krenair) Apparently @Chad was working on fixing search on beta all day. I don't know why it was... [22:11:30] Yippee, build fixed! [22:11:30] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #458: FIXED in 28 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/458/ [22:13:07] Yippee, build fixed! [22:13:08] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #175: FIXED in 38 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/175/ [22:13:52] Yippee, build fixed! [22:13:52] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #329: FIXED in 44 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/329/ [22:15:14] Yippee, build fixed! [22:15:15] Project beta-scap-eqiad build #41000: FIXED in 21 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41000/ [22:20:08] Yippee, build fixed! [22:20:08] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #456: FIXED in 2 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/456/ [22:25:18] 3Phabricator: Get rid of "Export to Excel" - https://phabricator.wikimedia.org/T257#1026241 (10Dzahn) thanks, valhallasw. note to self: andre__ told me how to check for the new number of old tickets imported from fab labs. use prefix "fl" in the reference field in advanced search [22:28:27] 3operations, Release-Engineering, WMF-Design: Better WMF error pages - https://phabricator.wikimedia.org/T76560#1026252 (10bd808) @greg is going to find a #release-engineering helper for this project [22:32:58] 3Release-Engineering: Plan log responsibilities changes - https://phabricator.wikimedia.org/T89049#1026260 (10greg) 3NEW [22:33:15] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1026271 (10greg) [22:39:28] 3MediaWiki-Unit-tests, Continuous-Integration: mediawiki-extensions-zend builds are jammed half-way through phpunit execution - https://phabricator.wikimedia.org/T89050#1026297 (10Krinkle) 3NEW [22:40:01] !log Various mediawiki-extensions-zend builds are jammed half-way through phpunit execution (filed T89050) [22:40:06] Logged the message, Master [22:40:30] thcipriani: it just hit me your are the new rel eng, so welcome! [22:40:56] i'm a long term lurking nudge around here - FYI [22:41:15] matanya: thanks! Good to know you. [22:41:27] greg-g: don't ruin my rep here [22:43:06] Yippee, build fixed! [22:43:07] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #451: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/451/ [22:43:53] Yippee, build fixed! [22:43:53] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #457: FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/457/ [22:43:56] 3Flow, Continuous-Integration: Intermittent mediawiki-extensions-hhvm failures due to Flow unit test fetching live SpamBlacklist configuration - https://phabricator.wikimedia.org/T89052#1026316 (10Krinkle) 3NEW [22:44:18] Yippee, build fixed! [22:44:18] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #401: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/401/ [22:44:41] matanya: :P [22:44:54] thcipriani: matanya is good people :) [22:45:26] greg-g: I've filed T89052 and T89050, reporting the two common issues causing random builds to fail. This is a regression from last week. I'd say Flow should be tasked with fixing the test asap or be depooled from the test cluster for the time being as its slowing down other teams. [22:45:46] I'm out for the day, just got home from my flight and merely checking in for a minute here :) [22:46:16] The timeout is also an issue, no idea what's causing that. [22:46:33] Krinkle: /me nods [22:48:48] 3Continuous-Integration: Zuul cloner fails on extension jobs against a wmf branch - https://phabricator.wikimedia.org/T73133#1026346 (10Krinkle) Is this still an issue? [22:49:00] 3Flow, Continuous-Integration: Intermittent mediawiki-extensions-hhvm failures due to Flow unit test fetching live SpamBlacklist configuration - https://phabricator.wikimedia.org/T89052#1026349 (10greg) @DannyH can someone please fix this ASAP? It's breaking other people's builds. Fix is great, otherwise we'll h... [22:51:08] 3Flow, Continuous-Integration: Intermittent mediawiki-extensions-hhvm failures due to Flow unit test fetching live SpamBlacklist configuration - https://phabricator.wikimedia.org/T89052#1026361 (10Mattflaschen) a:3Mattflaschen [22:56:42] 3Continuous-Integration: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#1026395 (10Krinkle) https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/2597/console ``` 21:49:32 INFO:zuul.Cloner:Prepared mediawiki/extensions/EventLogging repo with bra... [23:00:52] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1026444 (10Krinkle) [23:16:22] PROBLEM - Host deployment-elastic08 is DOWN: CRITICAL - Host Unreachable (10.68.17.2) [23:18:42] PROBLEM - Host deployment-elastic06 is DOWN: CRITICAL - Host Unreachable (10.68.17.79) [23:19:06] PROBLEM - Host deployment-elastic07 is DOWN: CRITICAL - Host Unreachable (10.68.17.82) [23:29:31] 3Flow, Continuous-Integration: Intermittent mediawiki-extensions-hhvm failures due to Flow unit test fetching live SpamBlacklist configuration - https://phabricator.wikimedia.org/T89052#1026559 (10Mattflaschen) The underlying cause is that the SpamBlacklist extension defaults to using a page [from Meta](https://... [23:35:47] PROBLEM - Puppet staleness on deployment-salt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [23:36:24] RECOVERY - Host deployment-elastic08 is UP: PING OK - Packet loss = 0%, RTA = 1.64 ms [23:43:20] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1026634 (10greg) [23:43:26] 3operations, Release-Engineering, WMF-Design: Better WMF error pages - https://phabricator.wikimedia.org/T76560#1026635 (10Technical13) >>! In T76560#847830, @Nirzar wrote: > Here's [[ http://nirzar.github.io/prototypes/error-pages/error-template.html | final template ]] for the error page. > > - It's mobile o... [23:46:42] 3Flow, Continuous-Integration: Intermittent mediawiki-extensions-hhvm failures due to Flow unit test fetching live SpamBlacklist configuration - https://phabricator.wikimedia.org/T89052#1026640 (10EBernhardson) Ran flow's test suite through strace with the '-e trace=connect' parameters to record all outgoing con... [23:52:00] 3Release-Engineering, Team-Practices: RelEng team offsite - https://phabricator.wikimedia.org/T89036#1026652 (10greg)