[00:48:32] 10Continuous-Integration-Infrastructure: mwext-php70-phan-docker should've php-curl extension installed - https://phabricator.wikimedia.org/T183322#3850111 (10Florian) [00:48:46] legoktm: around? :) [00:49:20] hi FlorianSW [00:49:27] hi, how are you? :) [00:49:48] alright, you? :) [00:49:53] looking at the task now [00:49:59] I'm fine, too, thanks :) [00:50:15] one question regarding that: Are integration hosts managed by puppet contint-module? [00:50:23] If so, it should include php-curl :/ https://github.com/wikimedia/puppet/blob/e959321aa620b77403cc9379db2e86080323c6e8/modules/contint/manifests/packages/php.pp#L26 [00:51:04] the docker jobs are different [00:51:19] integration/config: dockerfiles dir [00:51:23] yeah, docker contains curl, too. [00:51:36] However, the job already exits in the slave, doesn't it? Not in the docker container [00:52:02] At least that's what I see (I firstly ran into "That docker container is missing curl", too :/) [00:53:17] dont' think so, it should all be in the container [00:53:40] 19:18:57 + exec docker run --rm --env-file /dev/fd/63 --volume /srv/jenkins-workspace/workspace/mwext-php70-phan-docker/src:/src --volume /srv/jenkins-workspace/workspace/mwext-php70-phan-docker/cache:/cache --volume /srv/git:/srv/git --entrypoint bash wmfreleng/ci-src-setup:v2017.11.21.12.57 /srv/setup-mwext.sh [00:54:50] oh damn it... I must have missed that :/ Then I can probably find the dockerfile myself, let me look :] [00:59:13] legoktm: so what I see is, that here https://github.com/wikimedia/integration-config/blob/master/dockerfiles/ci-src-setup/Dockerfile#L43 php7.0-curl should be added [00:59:46] yep [01:00:23] ok, I'll submit a change then, thanks for pointing me to that :) [01:02:53] FlorianSW: php-curl [01:02:59] Then you don't need to keep bumping the versions [01:03:28] mbstring and xml already uses the version, that's why I haven't used php-curl :) [01:04:45] someone should fix that :P [01:04:56] Things are still inconsistent as to what actually has packages [01:05:36] ok, then I'll check, that the ones used will work and change them, too :P [01:06:06] yeah the new docker thing makes it explict, but we're missing some packages that were inherited due to other things [01:09:04] FlorianSW: xml and mbstring have php- packages in debian [01:09:11] zip, too [01:09:35] and cli, too [01:09:51] docker is currently building it locally to test and then I submit my change :) [01:24:20] (03PS1) 10Florianschmidtwelzow: Add php-curl to ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/399318 (https://phabricator.wikimedia.org/T183322) [01:30:02] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [01:34:17] (03PS1) 10MarkAHershberger: Bump version req so phpcs:disable can be used [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 [01:34:33] PROBLEM - Free space - all mounts on deployment-mx is CRITICAL: CRITICAL: deployment-prep.deployment-mx.diskspace._var_log.byte_percentfree (<22.22%) [01:37:52] (03CR) 10jerkins-bot: [V: 04-1] Bump version req so phpcs:disable can be used [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [01:41:54] (03CR) 10Legoktm: [C: 04-1] "We're already at 3.2.1 which ~3.2 matches? We're aiming for a release at the end of this week, which will include the new upstream release" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [01:44:36] (03CR) 10MarkAHershberger: "~3.2 allows 3.2.2 as well, though. Which Looks like you need to use phpcs:disable. See https://gerrit.wikimedia.org/r/#/c/399315" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [01:49:26] (03CR) 10Reedy: "Eh? It's a Windows bugfix in 3.2.2" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [01:55:46] (03CR) 10MarkAHershberger: "Hmmm... thanks for pointing that out Reedy. Strange that when composer installed 3.2.1 locally it didn't work. And it looks like it isn'" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [01:57:39] (03CR) 10MarkAHershberger: "I thought this was a 3.2.2 v 3.2.1 issue b/c when I installed on my local machine those two versions behavior differed. Now, I'm confused." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [01:57:47] (03CR) 10Legoktm: [C: 04-1] "Ah, I hadn't checked me feeds to see that 3.2.2 had been released. Do you want to update this patch to bump to 3.2.2?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [02:03:38] 10Release-Engineering-Team (Next), 10VisualEditor, 10User-Ryasmeen, 10User-zeljkofilipin: LanguageScreenshotBot trying to edit a non-existent page without signing in - https://phabricator.wikimedia.org/T162454#3850174 (10Anomie) >>! In T162454#3849155, @Whatamidoing-WMF wrote: > As noted before, "using a d... [02:08:35] (03Abandoned) 10MarkAHershberger: Bump version req so phpcs:disable can be used [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/399321 (owner: 10MarkAHershberger) [02:34:21] (03PS1) 10Legoktm: Don't use composer-package images for normal composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/399322 [02:36:56] 02:36:33 + composer --ansi validate --no-check-publish [02:36:56] 02:36:33 Failed to initialize central HHBC repository: [02:36:56] 02:36:33 Failed to open /srv/jenkins-workspace/workspace/mwgate-composer-hhvm-docker/central.hhbc: 14 - unable to open database file [02:36:56] 02:36:33 Failed to open /nonexistent/.hhvm.hhbc: 14 - unable to open database file [02:38:06] (03CR) 10Legoktm: [C: 032] Don't use composer-package images for normal composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/399322 (owner: 10Legoktm) [02:39:43] (03Merged) 10jenkins-bot: Don't use composer-package images for normal composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/399322 (owner: 10Legoktm) [04:14:20] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:15:28] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:50:29] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:51:38] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [04:54:18] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:21:38] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [06:16:11] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<11.11%) [06:32:08] 10Beta-Cluster-Infrastructure, 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Unbreak replication in beta cluster - https://phabricator.wikimedia.org/T183252#3850436 (10jcrespo) p:05Triage>03Lowest No one is probably going to work on this any time soon, desp... [06:46:13] RECOVERY - Free space - all mounts on deployment-eventlog02 is OK: OK: All targets OK [06:52:24] 10Release-Engineering-Team (Next), 10Release Pipeline, 10User-Joe: Prove helm as a potential k8s deployment tool - https://phabricator.wikimedia.org/T173129#3850470 (10Joe) 05Open>03Resolved [07:10:02] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:35:51] (03CR) 10Hashar: "Sorry that was a terrible copy paste :(" [integration/config] - 10https://gerrit.wikimedia.org/r/399322 (owner: 10Legoktm) [08:59:10] (03PS1) 10Hashar: docker: migrate castor to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/399357 [09:01:33] (03CR) 10Hashar: [C: 032] docker: migrate castor to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/399357 (owner: 10Hashar) [09:03:16] (03Merged) 10jenkins-bot: docker: migrate castor to docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/399357 (owner: 10Hashar) [09:31:55] hashar: could you take a look at https://phabricator.wikimedia.org/T183324 ? [09:33:00] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:34:04] (03PS1) 10Hashar: docker: fix castor image build under python3.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/399364 [09:34:57] (03PS2) 10Hashar: docker: fix castor image build under python3.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/399364 [09:36:12] (03CR) 10Hashar: [C: 032] docker: fix castor image build under python3.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/399364 (owner: 10Hashar) [09:37:23] (03Merged) 10jenkins-bot: docker: fix castor image build under python3.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/399364 (owner: 10Hashar) [09:50:33] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations: Can't docker pull from docker-registry.discovery.wmnet - https://phabricator.wikimedia.org/T183342#3850688 (10hashar) [09:53:23] (03PS1) 10Hashar: docker: point Castor FROM to the public registry [integration/config] - 10https://gerrit.wikimedia.org/r/399366 (https://phabricator.wikimedia.org/T183342) [09:55:05] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: Can't docker pull from docker-registry.discovery.wmnet - https://phabricator.wikimedia.org/T183342#3850717 (10hashar) [09:55:24] (03CR) 10Hashar: [C: 032] docker: point Castor FROM to the public registry [integration/config] - 10https://gerrit.wikimedia.org/r/399366 (https://phabricator.wikimedia.org/T183342) (owner: 10Hashar) [09:56:34] (03Merged) 10jenkins-bot: docker: point Castor FROM to the public registry [integration/config] - 10https://gerrit.wikimedia.org/r/399366 (https://phabricator.wikimedia.org/T183342) (owner: 10Hashar) [10:02:54] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [10:03:30] (03PS1) 10Hashar: docker: add missing symlinks for castor [integration/config] - 10https://gerrit.wikimedia.org/r/399369 [10:17:31] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations: Can't docker pull from docker-registry.discovery.wmnet - https://phabricator.wikimedia.org/T183342#3850763 (10hashar) [10:24:20] 10Beta-Cluster-Infrastructure, 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Unbreak replication in beta cluster - https://phabricator.wikimedia.org/T183252#3850772 (10Addshore) @daniel I would mark this as blocking one of the MCR related tickets but i'm not re... [10:25:28] (03PS1) 10Hashar: Move composer-hhvm jobs back to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/399373 (https://phabricator.wikimedia.org/T183324) [10:26:40] 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324#3850200 (10Addshore) Has this been an issue since these images were introduced? If not, shouldn't we have an old version of the image to roll... [10:26:54] 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324#3850792 (10hashar) Sorry I completely failed even do a simple test :( Reverted back to Nodepool instances for now. /srv/jenkins-workspace/... [10:27:10] addshore: the container is broken most probably [10:27:13] (03CR) 10Hashar: [C: 032] Move composer-hhvm jobs back to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/399373 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [10:27:34] (03CR) 10Hashar: [C: 032] "The jobs were still present in Jenkins since I did not delete them." [integration/config] - 10https://gerrit.wikimedia.org/r/399373 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [10:28:15] addshore: I guess we need a hhvm configuration file. I totally forgot about that :] [10:28:19] 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324#3850794 (10Addshore) Did we end up passing all env vars through? or just a whitelist still? [10:28:24] haha :P [10:28:28] (03Merged) 10jenkins-bot: Move composer-hhvm jobs back to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/399373 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [10:36:16] 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324#3850805 (10hashar) 05Open>03Resolved a:03hashar I moved the jobs back to Nodepool. [10:37:57] addshore: I will dig again into it and this time add a few example-run.sh :] [10:39:04] :D [10:39:44] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10User-Ryasmeen, 10User-zeljkofilipin: LanguageScreenshotBot trying to edit a non-existent page without signing in - https://phabricator.wikimedia.org/T162454#3163641 (10zeljkofilipin) [10:39:49] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10User-Ryasmeen, 10User-zeljkofilipin: LanguageScreenshotBot trying to edit a non-existent page without signing in - https://phabricator.wikimedia.org/T162454#3163641 (10zeljkofilipin) a:03zeljkofilipin [10:48:00] addshore: easy https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/ci/hhvm.pp#L20-L36 :) [11:11:27] addshore: could you look at https://gerrit.wikimedia.org/r/#/c/398678/ ? easy codesniffer patch :) [11:12:35] 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Patch-For-Review, 10User-zeljkofilipin: Delete Echo Selenium tests - https://phabricator.wikimedia.org/T171848#3850873 (10zeljkofilipin) p:05Normal>03Low a:05Etonkovidova>03zeljkofilipin [11:27:34] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:37:00] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3850919 (10zeljkofilipin) [11:40:18] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3850923 (10zeljkofilipin) [11:44:26] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3850927 (10zeljkofilipin) [12:02:37] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [12:47:37] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [12:52:38] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [13:46:22] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3851246 (10zeljkofilipin) [13:52:37] 10Release-Engineering-Team (Kanban), 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin, 10Wikidata-Sprint-2017-12-20: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3851252 (10WMDE-leszek) I've tried to look into why there is so many test failures when targetin... [14:00:52] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3851302 (10zeljkofilipin) [14:23:33] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3851398 (10akosiaris) I 'll start this with a reiteration of some common... [14:50:04] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3851462 (10rafidaslam) [14:52:47] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#3851468 (10awight) Hi, thanks for your thoughts! >>! In T181071#3851398,... [14:59:51] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3851492 (10zeljkofilipin) [15:09:46] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3851504 (10zeljkofilipin) I am having trouble running CirrusSearch Selenium... [15:13:37] PROBLEM - Free space - all mounts on deployment-db03 is CRITICAL: CRITICAL: deployment-prep.deployment-db03.diskspace._mnt_sqldata.byte_percentfree (No valid datapoints found) deployment-prep.deployment-db03.diskspace._mnt_tmp.byte_percentfree (No valid datapoints found)deployment-prep.deployment-db03.diskspace.root.byte_percentfree (<40.00%) [15:15:01] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:28:37] RECOVERY - Free space - all mounts on deployment-db03 is OK: OK: deployment-prep.deployment-db03.diskspace._mnt_sqldata.byte_percentfree (No valid datapoints found) deployment-prep.deployment-db03.diskspace._mnt_tmp.byte_percentfree (No valid datapoints found) [15:33:41] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:10:59] (03PS1) 10Zfilipin: Add ansicolor, timeout and timestamps to selenium-{name}-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/399431 (https://phabricator.wikimedia.org/T175179) [16:22:00] (03CR) 10Zfilipin: [C: 032] Add ansicolor, timeout and timestamps to selenium-{name}-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/399431 (https://phabricator.wikimedia.org/T175179) (owner: 10Zfilipin) [16:23:11] (03CR) 10Zfilipin: [C: 032] "Updated jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/399431 (https://phabricator.wikimedia.org/T175179) (owner: 10Zfilipin) [16:24:30] (03Merged) 10jenkins-bot: Add ansicolor, timeout and timestamps to selenium-{name}-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/399431 (https://phabricator.wikimedia.org/T175179) (owner: 10Zfilipin) [17:06:42] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324#3851772 (10greg) [17:34:50] https://jenkins.io/2.0/ [17:34:51] o_0 [17:55:06] (03CR) 10Krinkle: (WIP) Cache node_modules between runs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/395610 (https://phabricator.wikimedia.org/T159591) (owner: 10Hashar) [17:57:17] Reedy: interesting read [17:57:53] (03PS4) 10Hashar: docker: polish hhvm container [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) [17:58:04] grblblbl [18:04:05] (03PS5) 10Hashar: docker: polish hhvm container [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) [18:06:12] (03PS6) 10Hashar: docker: polish hhvm container [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) [18:07:18] (03CR) 10Hashar: "I guess I will add the -docker based job to the experimental pipeline and then deploy this change. If the jobs/containers work fine, I gu" [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:01:08] 10Beta-Cluster-Infrastructure, 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Unbreak replication in beta cluster - https://phabricator.wikimedia.org/T183252#3852204 (10Catrope) 05Open>03Resolved a:03Catrope I exported the data on db03, imported it on db04... [19:03:31] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqdfw: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3852214 (10Dzahn) a:05Cmjohnson>03Dzahn [19:52:38] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [20:25:19] Yippee, build fixed! [20:25:19] Project selenium-Wikibase-chrome ยป chrome,beta,Linux,DebianJessie && contintLabsSlave build #49: 09FIXED in 38 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/49/ [20:30:13] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:34:29] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:35:49] Hi, I have problem with phabricator. I cleaned cache memory, tryed with incogto mode in google chrome, tryed in another browser and on pc and on phone but I no see correct things there [20:36:00] I asked on #wikimedia-operations but Zppix told me to I ask here [20:36:03] I have problem with "our" phabricator [20:36:03] See screenshot: http://prntscr.com/hq8b8y [20:36:03] I no see logo of phabricator, and pictuers there [20:36:03] Example 1: http://prntscr.com/hq8boz [20:36:03] Example 2: http://prntscr.com/hq8bza [20:36:04] To I open task about this problem? [20:36:35] See up my report there (I pasted it) [20:37:36] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [20:41:10] (03PS7) 10Hashar: docker: polish hhvm container [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) [20:42:13] Zoranzoki21: the browser console should have more details :) [20:42:51] Failed to load resource: the server responded with a status of 403 () [20:42:51] profile Failed to load resource: the server responded with a status of 403 () [20:42:51] alphanumeric_lato-dark_L.png-_48a3ba-0%2C0%2C0%2C0.png Failed to load resource: the server responded with a status of 403 () [20:42:51] profile-box.png Failed to load resource: the server responded with a status of 403 () [20:42:51] profile Failed to load resource: the server responded with a status of 403 () [20:42:52] profile Failed to load resource: the server responded with a status of 403 () [20:42:54] profile Failed to load resource: the server responded with a status of 403 () [20:42:56] profile Failed to load resource: the server responded with a status of 403 () [20:42:57] Zoranzoki21: in chrome Control + Shift + 1 [20:42:58] profile Failed to load resource: the server responded with a status of 403 () [20:43:00] alphanumeric_aleo-white_P.png-_296437-255%2C255%2C255%2C0.4.png Failed to load resource: the server responded with a status of 403 () [20:43:00] hehe [20:43:03] phab.wmfusercontent.org/file/data/jk46jzui2ugiwycznn2u/PHID-FILE-mypfkpsjr7g3rhenz2ob/pinboard-current_adyen_FR_desktop_1.png Failed to load resource: the server responded with a status of 403 () [20:43:06] phab.wmfusercontent.org/file/data/bfhgv35ojfman36fnj4n/PHID-FILE-vsom6jqokqvu3dclxcd3/pinboard-Screen_Shot_2017-11-20_at_11.11.44.png Failed to load resource: the server responded with a status of 403 () [20:43:09] phab.wmfusercontent.org/file/data/plwkcq7l7dz3g67jlsht/PHID-FILE-ilmrnddn42rzju6omhuy/pinboard-AB3DBA73-B771-4BD5-AB8E-7A2F687ACB28.png Failed to load resource: the server responded with a status of 403 () [20:43:11] works for me. [20:43:14] phab.wmfusercontent.org/file/data/psxqpqozjablmsa2d7ti/PHID-FILE-ohetwfju4isttbu2xyth/pinboard-IMG_5976.PNG Failed to load resource: the server responded with a status of 403 () [20:43:17] phab.wmfusercontent.org/file/data/7co5mkibsadubluomzd6/PHID-FILE-gisw6tq32axeyjyh3yhn/pinboard-488de7f-fe5f-4384-ac7d-a7c68345ce9a23_%281%29.png Failed to load resource: the server responded with a status of 403 () [20:43:20] phab.wmfusercontent.org/file/data/eu6iadvmigysezlieuga/PHID-FILE-q2e6txjeihlfg76633y7/pinboard-On_this_day_-_Outline_tool_flow_1.png Failed to load resource: the server responded with a status of 403 () [20:43:23] Sorry for this. I wanted to fast reply you [20:43:26] Do you use google chrome too? [20:43:34] 403 being "denied" [20:43:34] yes [20:43:44] Are you using wikipedia zero? [20:43:44] are you using a proxy? [20:44:27] Zoranzoki21: the network tab in the console might have more details [20:44:47] specially it would list all network requests, and clicking on one would give the headers sent back by phab.wmfusercontent.org [20:45:23] (which is served by our cache infrastructure (misc varnish) [20:45:29] http://prntscr.com/hq8hxq [20:46:28] Zoranzoki21: try the "network" tab [20:46:35] and you will get the details [20:46:58] (and also, you probably dont want to mine bitcoins in a browser using javascript :D ) [20:47:20] :D I no working it current [20:47:29] Only is opened tab [20:48:14] Wait, I will record you full opening phabricator [20:48:27] beware that your cookie would be included (iirc) [20:48:40] so if you make thatpublic, one can suddenly become you! :D [20:49:36] (03CR) 10Hashar: [C: 032] "# is deprecated in PHP ini files, so changed them to ;" [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [20:50:50] (03Merged) 10jenkins-bot: docker: polish hhvm container [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [20:51:54] !log Rebuilding hhvm Docker containers https://gerrit.wikimedia.org/r/399406 | T183324 [20:52:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:52:00] T183324: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324 [21:00:13] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:02:07] Finally, I recorded screen [21:02:58] I will give you link to see this [21:05:05] Only to I upload it and I will give you link [21:05:28] PROBLEM - Free space - all mounts on deployment-kafka03 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka03.diskspace.root.byte_percentfree (<44.44%) [21:06:44] https://youtu.be/U4Hw8kguxHw [21:31:20] Zoranzoki21: 20:44 <+ hashar> specially it would list all network requests, and clicking on one would give the [21:31:23] headers sent back by phab.wmfusercontent.org [21:33:33] Zoranzoki21: so at the end of the video there is a table with all the requests made [21:33:37] some 302 / 200 and 403 [21:33:58] Zoranzoki21: you would want to look at the details for one of the 403 [21:34:03] but really i have no idea what could be going on [21:35:54] Zoranzoki21: also, please answer all questions we give you. For instance, are you using a proxy? Wikipedia Zero? [21:36:43] sorry. I been in toilet [21:37:29] I no using proxy [21:37:37] And I no use wikipedia zero [21:38:21] I will try with another browser [21:39:01] and disable all relevant extensions like adblockers etc [21:39:53] No work [21:40:52] And in edge and in internet explorer no work [21:41:23] But... See this: http://prntscr.com/hq99qq [21:41:28] Are you using wikimedia zero or a proxy? [21:41:33] no [21:42:10] Hmm.. See this: http://prntscr.com/hq9a3u [21:53:42] I will come back for 2-3 minutes [21:53:46] I have to restart my pc [21:59:24] I am back [22:01:38] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324#3852735 (10hashar) 05Resolved>03Open p:05Unbreak!>03High The jobs are back to Nodepool so the... [22:01:52] 10Release-Engineering-Team (Kanban), 10User-greg: End of quarter grooming - https://phabricator.wikimedia.org/T183427#3852738 (10greg) p:05Triage>03Normal [22:04:49] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3852761 (10rafidaslam) [22:09:12] (03PS1) 10Hashar: docker: hhvm set HHVM_REPO_CENTRAL_PATH [integration/config] - 10https://gerrit.wikimedia.org/r/399481 (https://phabricator.wikimedia.org/T183324) [22:10:15] (03PS2) 10Hashar: docker: hhvm set HHVM_REPO_CENTRAL_PATH [integration/config] - 10https://gerrit.wikimedia.org/r/399481 (https://phabricator.wikimedia.org/T183324) [22:20:13] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:35:43] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:40:05] yeah first build passing with hhvm https://integration.wikimedia.org/ci/job/mwgate-composer-hhvm-docker/41/console (poke legoktm \o/ ) [22:40:38] (03PS1) 10Hashar: Do not pass HHVM_REPO_CENTRAL_PATH to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/399532 (https://phabricator.wikimedia.org/T183324) [22:42:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: HHVM docker images are failing due to hhbc repository - https://phabricator.wikimedia.org/T183324#3852816 (10hashar) I have rebuild the build mentioned originally in this task and it passed just fine:... [22:42:45] (03CR) 10Hashar: [C: 032] Do not pass HHVM_REPO_CENTRAL_PATH to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/399532 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [22:43:53] (03Merged) 10jenkins-bot: Do not pass HHVM_REPO_CENTRAL_PATH to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/399532 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [22:51:59] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Patch-For-Review, 10User-Joe: Unify production and CI docker image build process - https://phabricator.wikimedia.org/T177276#3852857 (10hashar) I have finally started the conversion of the CI images to docker-pkg and even sent a few patch... [22:52:13] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<11.11%) [22:54:53] hashar: woot :D [22:59:05] (03CR) 10Krinkle: docker: polish hhvm container (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/399406 (https://phabricator.wikimedia.org/T183324) (owner: 10Hashar) [22:59:25] legoktm: the whole thing is very dirty :( [22:59:42] legoktm: and thanks for a revert and some patch you did while I was sleeping. I should have been more careful [23:00:28] Krinkle: thank you for the typo check. Sorry I should reread myself again :( [23:01:22] so hopefully I will get the composer*hhvm jobs migrated to Docker [23:01:28] but I will test each and all of them first [23:15:30] PROBLEM - Free space - all mounts on deployment-kafka03 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka03.diskspace.root.byte_percentfree (<33.33%) [23:17:49] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3852890 (10Ryan10145) [23:18:33] About my report.. Error which I got now: http://prntscr.com/hqaejy [23:26:51] Zoranzoki21: ahh nice [23:27:36] Zoranzoki21: if you get access to Phabricator, you can fill a task against #traffic and attach that screenshot [23:27:48] and copy paste the message at the bottom [23:27:58] Zoranzoki21: hmm wait [23:28:10] maybe the blacklist is handled directly in Phabricator, for which twentyafterfour would know Iguess [23:28:16] or a task for #phabricator [23:28:31] really I am not sure how those blacklist are handled, but that looks like related to Zero mobile [23:29:07] the blacklist is just an apache deny rule managed in puppet [23:29:34] maybe that is the one that causes the failure [23:30:11] Zero is blacklisted so yeah... ^ [23:30:39] then I dont whether that is related or no ::] [23:30:50] anyway I gotta sleep. Good luck [23:30:54] https://github.com/wikimedia/puppet/blob/cfbd2b1000759b8b5234d6877fce6c8ce9e874b9/modules/phabricator/files/apache/phabbanlist.conf [23:30:57] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:31:56] ok.. thank you [23:32:42] doesnt seem to be matched by that [23:32:45] well :- [23:32:52] not much i can do though [23:32:55] good night! [23:36:08] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:24] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]