[00:01:24] Yippee, build fixed! [00:01:25] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #289: FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/289/ [00:10:27] Yippee, build fixed! [00:10:27] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #345: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/345/ [00:12:23] PROBLEM - Puppet staleness on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [00:18:55] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:08:05] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:11:05] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [01:33:04] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [01:36:06] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:36:16] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:42:40] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:45:26] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:49:02] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:52:05] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [01:53:37] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:54:20] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:17:06] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:19:20] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [02:21:15] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:23:37] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:30:25] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:32:38] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:34:04] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [03:04:12] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:11:19] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:15:29] Project beta-scap-eqiad build #33948: FAILURE in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33948/ [03:25:35] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #226: FAILURE in 19 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/226/ [03:26:45] 3Beta-Cluster: Puppet failures on deployment-bastion - https://phabricator.wikimedia.org/T75520#847239 (10mmodell) 5Open>3Resolved removed keyholder from the instance config because it's specified in the deployment role. [03:35:14] Yippee, build fixed! [03:35:14] Project beta-scap-eqiad build #33950: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33950/ [04:14:36] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [04:15:20] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [04:34:54] Project beta-scap-eqiad build #33956: FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33956/ [04:39:37] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:40:19] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [05:22:45] Yippee, build fixed! [05:22:45] Project beta-scap-eqiad build #33958: FIXED in 28 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33958/ [05:38:48] Yippee, build fixed! [05:38:49] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #321: FIXED in 3 min 34 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/321/ [07:40:11] Yippee, build fixed! [07:40:12] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #225: FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/225/ [08:46:52] Yippee, build fixed! [08:46:53] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #191: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/191/ [09:14:02] Yippee, build fixed! [09:14:02] 3Beta-Cluster: Puppet failures on deployment-bastion - https://phabricator.wikimedia.org/T75520#847414 (10hashar) 5Resolved>3Open Thanks @andrew @mmodell , I know start understanding the power of hiera for the beta cluster :-] deployment-bastion now fails with: ``` Error: /Stage[main]/Role::Deployment::Dep... [09:14:03] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #340: FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/340/ [09:20:53] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #346: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/346/ [09:27:49] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [09:34:18] Project beta-scap-eqiad build #33983: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33983/ [09:34:28] PROBLEM - Free space - all mounts on deployment-mediawiki02 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki02.diskspace._var.byte_percentfree.value (<22.22%) [09:35:53] PROBLEM - Free space - all mounts on deployment-mediawiki01 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki01.diskspace._var.byte_percentfree.value (<20.00%) [09:37:27] Yippee, build fixed! [09:37:28] Project beta-scap-eqiad build #33984: FIXED in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33984/ [09:41:29] !log deleted hhvm core files in /var/tmp/core from both mediawiki01 and mediawiki02 {{T1259}} and {{T71979}} [09:41:34] Logged the message, Master [09:44:58] 3Beta-Cluster: hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - https://phabricator.wikimedia.org/T71979#847463 (10hashar) I have noticed hhvm fills /var/tmp/core/ on beta cluster which is due to the kernel core directory being set to `/var/tmp/core`. From `modules/base/manifest... [09:47:48] (03CR) 10Hashar: [C: 032] "Lets do it! Will deploy on prod using git-deploy and via salt on labs instances." [integration/composer] - 10https://gerrit.wikimedia.org/r/178550 (owner: 10Legoktm) [09:48:08] (03CR) 10Hashar: [V: 032] Upgrading Composer to v1.0.0-alpha9 [integration/composer] - 10https://gerrit.wikimedia.org/r/178550 (owner: 10Legoktm) [09:55:04] (03CR) 10Hashar: [C: 032] "I have pushed the new jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/178714 (owner: 10AndyRussG) [09:58:53] (03Merged) 10jenkins-bot: Add browsers for CentralNotice [integration/config] - 10https://gerrit.wikimedia.org/r/178714 (owner: 10AndyRussG) [10:52:53] (03PS6) 10Hashar: Support MediaWiki PHP tests under HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/178862 [10:55:11] (03CR) 10Hashar: "I made the mediawiki extensions HHVM jobs to not trigger on REL1_23 and REL1_24: https://gerrit.wikimedia.org/r/#/c/178862/5..6/zuul/layou" [integration/config] - 10https://gerrit.wikimedia.org/r/178862 (owner: 10Hashar) [10:55:15] Project beta-scap-eqiad build #33992: FAILURE in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33992/ [10:55:33] (03PS7) 10Hashar: Support MediaWiki PHP tests under HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/178862 [10:59:59] 3Continuous-Integration: Investigate integration-jjb-config-diff job slowness - https://phabricator.wikimedia.org/T78532#847590 (10hashar) 3NEW [11:02:28] (03CR) 10Hashar: [C: 032] "Good enough for now. Will announce on wikitech-l for people to experiment the job then later in the week move the hhvm jobs from the exper" [integration/config] - 10https://gerrit.wikimedia.org/r/178862 (owner: 10Hashar) [11:07:27] (03Merged) 10jenkins-bot: Support MediaWiki PHP tests under HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/178862 (owner: 10Hashar) [11:15:56] Yippee, build fixed! [11:15:56] Project beta-scap-eqiad build #33994: FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33994/ [12:46:25] 3Beta-Cluster: hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - https://phabricator.wikimedia.org/T71979#847802 (10yuvipanda) You can set the core dump path from hiera (see Hiera:tools on wikitech) to be shared storage or somewhere else, but unfortunately not kill core dumps ent... [12:53:01] !log manually restarted diamond on all betalabs host, to see if that is why metrics aren’t being sent anymore [12:53:03] Logged the message, Master [13:03:32] 3Phabricator.org, Release-Engineering, Phabricator: Answer questions about ongoing maintenance of phabricator customizations/extensions - https://phabricator.wikimedia.org/T78464#847820 (10Aklapper) >>! In T78464#846549, @RobLa-WMF wrote: > We haven't yet clearly spelled out our maintenance strategy for Phabrica... [13:09:19] 3Phabricator.org, Release-Engineering, Phabricator: Answer questions about ongoing maintenance of phabricator customizations/extensions - https://phabricator.wikimedia.org/T78464#847839 (10Dzahn) > We don't have a dedicated phabricator team going just to be clear about it, this is _not T175 ? [13:10:46] 3Phabricator.org, Release-Engineering, Phabricator: Answer questions about ongoing maintenance of phabricator customizations/extensions - https://phabricator.wikimedia.org/T78464#847841 (10Dzahn) >>! In T78464#846063, @MZMcBride wrote: > key selling point to switching to Phabricator was that (unlike, say, Gerrit... [13:54:05] Project beta-scap-eqiad build #34009: FAILURE in 0.42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34009/ [14:16:49] Yippee, build fixed! [14:16:49] Project beta-scap-eqiad build #34011: FIXED in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34011/ [14:35:39] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:59:27] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#848079 (10zeljkofilipin) @Stan3: unfortunately, I think you will have to abandon all commits in mediawiki/selenium. @dduvall's branch it too far ahead. Please pick any other repository if you want to continue worki... [15:00:37] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:01:58] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/179460 (owner: 10Stan) [15:02:05] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/179464 (owner: 10Stan) [15:02:09] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/179463 (owner: 10Stan) [15:02:13] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/179462 (owner: 10Stan) [15:02:16] (03CR) 10Zfilipin: [C: 04-1] remove comma after last item of hash (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/179461 (owner: 10Stan) [15:02:21] (03CR) 10Zfilipin: "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/179461 (owner: 10Stan) [15:02:25] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176344 (owner: 10Stan) [15:02:30] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176342 (owner: 10Stan) [15:02:33] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176340 (owner: 10Stan) [15:02:37] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176282 (owner: 10Stan) [15:02:42] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176279 (owner: 10Stan) [15:02:45] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176277 (owner: 10Stan) [15:02:49] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176276 (owner: 10Stan) [15:02:53] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176275 (owner: 10Stan) [15:02:57] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176261 (owner: 10Stan) [15:03:00] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176260 (owner: 10Stan) [15:03:04] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176256 (owner: 10Stan) [15:03:09] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176255 (owner: 10Stan) [15:03:13] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176254 (owner: 10Stan) [15:03:18] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176253 (owner: 10Stan) [15:03:23] (03CR) 10Zfilipin: [C: 04-1] "This commit should be abandoned. For details see:" [selenium] - 10https://gerrit.wikimedia.org/r/176252 (owner: 10Stan) [15:06:21] (03CR) 10Zfilipin: [C: 031] Method for loading of default environment [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179375 (owner: 10Dduvall) [15:06:51] (03CR) 10Zfilipin: [C: 031] Minor cruft removal [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179376 (owner: 10Dduvall) [15:07:11] (03CR) 10Zfilipin: [C: 031] Cache authenticated API client by url and user [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179362 (owner: 10Dduvall) [15:08:29] (03CR) 10Zfilipin: [C: 031] Support for default environment configurations [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179048 (owner: 10Dduvall) [15:09:33] (03CR) 10Zfilipin: [C: 031] Remote browser factory fixes [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179049 (owner: 10Dduvall) [15:11:15] (03PS5) 10Zfilipin: Ignore additional RuboCop Metric/* rules [ruby/api] - 10https://gerrit.wikimedia.org/r/175870 (owner: 10Dduvall) [15:11:32] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#848089 (10Stan3) @zeljkofilipin: you've spotted that https://github.com/stan3/mediawiki-selenium/tree/T75898 is branched from env-abstraction-layer? [15:11:42] (03CR) 10Zfilipin: "Removed reference to a bug that does not exist." [ruby/api] - 10https://gerrit.wikimedia.org/r/175870 (owner: 10Dduvall) [15:13:00] (03CR) 10Zfilipin: [C: 032] Ignore additional RuboCop Metric/* rules [ruby/api] - 10https://gerrit.wikimedia.org/r/175870 (owner: 10Dduvall) [15:13:26] (03Merged) 10jenkins-bot: Ignore additional RuboCop Metric/* rules [ruby/api] - 10https://gerrit.wikimedia.org/r/175870 (owner: 10Dduvall) [15:16:02] 3Release-Engineering: Fix browsertests-VisualEditor-language-screenshot-windows_8.1-firefox Jenkins job - https://phabricator.wikimedia.org/T76133#848096 (10zeljkofilipin) a:5zeljkofilipin>3None [15:19:29] PROBLEM - Free space - all mounts on deployment-mediawiki02 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki02.diskspace._var.byte_percentfree.value (<44.44%) WARN: deployment-prep.deployment-mediawiki02.diskspace.root.byte_percentfree.value (<100.00%) [15:33:04] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#848114 (10zeljkofilipin) @stan3, sorry, did not notice that. What did you do? Why does Gerrit complain then if you are working on the same branch? [15:36:39] Project beta-scap-eqiad build #34019: FAILURE in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34019/ [15:37:14] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:51:41] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#848156 (10Stan3) @zeljkofilipin: I cherry-picked the changes onto the new branch so change ids are the same. Maybe gerrit doesn't support that? [15:54:59] Yippee, build fixed! [15:54:59] Project beta-scap-eqiad build #34021: FIXED in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34021/ [16:02:14] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:48] (03PS1) 10Hashar: zuul: drop multiple extensions experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/179942 [16:18:04] (03CR) 10Hashar: [C: 032] zuul: drop multiple extensions experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/179942 (owner: 10Hashar) [16:19:26] (03Merged) 10jenkins-bot: zuul: drop multiple extensions experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/179942 (owner: 10Hashar) [16:21:36] (03PS1) 10Hashar: zuul: suffix mw testextension jobs with '-zend' [integration/config] - 10https://gerrit.wikimedia.org/r/179946 [16:24:48] (03PS2) 10Hashar: zuul: suffix mw testextension jobs with '-zend' [integration/config] - 10https://gerrit.wikimedia.org/r/179946 [16:26:29] (03CR) 10Hashar: [C: 032] zuul: suffix mw testextension jobs with '-zend' [integration/config] - 10https://gerrit.wikimedia.org/r/179946 (owner: 10Hashar) [16:27:30] (03Merged) 10jenkins-bot: zuul: suffix mw testextension jobs with '-zend' [integration/config] - 10https://gerrit.wikimedia.org/r/179946 (owner: 10Hashar) [16:47:42] (03PS1) 10Hashar: Suffix mediawiki-phpunit with '-zend' + enable hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/179951 [16:49:41] Can anyone grant Josve05a-test adminship on betacommons for a sec? I need him to test something for me. [16:52:06] (03CR) 10Hashar: [C: 032] Suffix mediawiki-phpunit with '-zend' + enable hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/179951 (owner: 10Hashar) [16:52:56] (03Merged) 10jenkins-bot: Suffix mediawiki-phpunit with '-zend' + enable hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/179951 (owner: 10Hashar) [16:54:24] Yippee, build fixed! [16:54:25] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #88: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/88/ [16:57:18] marktraceur: {{done}} I think [16:57:25] Thanks :) [16:57:28] marktraceur: Do you not have global rights in beta? [16:57:36] I don't think so. [16:58:14] marktraceur: What's your beta username? I can give you real ultimate power!!! [16:58:23] bd808: mtraceur [16:59:40] bd808: you're just ahead of me, I was going to give marktraceur the keys to beta also [16:59:45] Heh [17:00:11] marktraceur: You are now developer + global_sysop + staff + steward in beta. [17:00:20] Sweet. [17:00:23] Sounds dangerous. [17:00:37] Randomly chosen because that's the set of rights somebody gave me [17:00:56] RECOVERY - Free space - all mounts on deployment-mediawiki01 is OK: OK: All targets OK [17:01:10] heh. I don't think we have all that many stewards [17:13:34] Project beta-scap-eqiad build #34030: FAILURE in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34030/ [17:22:42] 3Continuous-Integration, CirrusSearch, MediaWiki-extensions-TimedMediaHandler: hhvm on CI slaves can not find PEAR.php - https://phabricator.wikimedia.org/T78556#848283 (10hashar) 3NEW [17:24:07] 3Continuous-Integration, CirrusSearch, MediaWiki-extensions-TimedMediaHandler: hhvm on CI slaves can not find PEAR.php - https://phabricator.wikimedia.org/T78556#848283 (10hashar) [17:35:16] Yippee, build fixed! [17:35:16] Project beta-scap-eqiad build #34034: FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34034/ [18:57:34] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #1: SUCCESS in 1 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/1/ [18:58:08] 3MediaWiki-extensions-TimedMediaHandler, Continuous-Integration, CirrusSearch: hhvm on CI slaves can not find PEAR.php - https://phabricator.wikimedia.org/T78556#848566 (10hashar) Seems to be an issue in TimedMediaHandler which fails under hhvm https://integration.wikimedia.org/ci/job/mwext-TimedMediaHandler-tes... [18:59:52] (03CR) 10Awight: "@hasher: ping, this is the last piece we're waiting on, to do another iteration. Thanks in advance!" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/158554 (owner: 10Awight) [19:09:13] 3Beta-Cluster: No linking to mediawiki.org on Special:Version for beta wikis - https://phabricator.wikimedia.org/T78574#848583 (10Umherirrender) 3NEW [19:11:56] (03CR) 10Dduvall: [C: 032] "Thanks for the review! Self-merging this into the experimental branch." [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179048 (owner: 10Dduvall) [19:14:16] (03CR) 10Dduvall: [C: 032] "Thanks for the review! Self-merging this into the experimental branch." [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179049 (owner: 10Dduvall) [19:14:41] (03CR) 10Dduvall: [C: 032] "Self-merging this into the experimental branch." [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179362 (owner: 10Dduvall) [19:14:53] (03Merged) 10jenkins-bot: Remote browser factory fixes [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179049 (owner: 10Dduvall) [19:16:07] (03CR) 10Dduvall: [C: 032] "Thanks for the review! Self-merging this into the experimental branch." [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179375 (owner: 10Dduvall) [19:16:22] (03CR) 10Dduvall: [C: 032] "Thanks for the review! Self-merging this into the experimental branch." [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179376 (owner: 10Dduvall) [19:16:43] (03Merged) 10jenkins-bot: Minor cruft removal [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179376 (owner: 10Dduvall) [19:24:42] 3MediaWiki-extensions-TimedMediaHandler, Continuous-Integration, CirrusSearch: hhvm on CI slaves can not find PEAR.php - https://phabricator.wikimedia.org/T78556#848606 (10hashar) /etc/hhvm/php.ini needs to be amended: ``` lang=diff - hhvm.include_path = .:/usr/share/php + include_path = ".:/usr/share/php" ```... [19:29:55] 3MediaWiki-extensions-TimedMediaHandler, Continuous-Integration, CirrusSearch: hhvm on CI slaves can not find PEAR.php - https://phabricator.wikimedia.org/T78556#848616 (10hashar) a:3hashar [19:37:41] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:38:39] 3MediaWiki-extensions-TimedMediaHandler, Continuous-Integration: hhvm on CI slaves can not find PEAR.php - https://phabricator.wikimedia.org/T78556#848651 (10hashar) [19:39:04] 3MediaWiki-extensions-TimedMediaHandler, Continuous-Integration: hhvm on CI slaves can not find PEAR.php - https://phabricator.wikimedia.org/T78556#848283 (10hashar) Cherry picked on contint puppetmaster and it fixed the build of TimedMediaHandler under HHVM https://integration.wikimedia.org/ci/job/mwext-TimedMe... [20:18:49] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:24:20] 3Mobile-Web, Continuous-Integration: Some MobileFrontend qunit builds failed to archive artifacts (voting -2) - https://phabricator.wikimedia.org/T78590#848819 (10Florian) 3NEW [20:32:07] Dec 15 18:46:39 deployment-mediawiki02: #012Warning: Host lookup failed [-10001]: Unknown error -10001 [20:32:08] Dec 15 18:46:39 deployment-mediawiki02: #012Warning: Failed connecting to redis server at deployment-redis01.eqiad.wmflabs: [20:32:11] are those the dns errors? [20:44:35] 3Continuous-Integration: Jenkins: Implement mediawiki-phpunit-hhvm job for mediawiki-core repo (and voting) - https://phabricator.wikimedia.org/T75521#755659 (10hashar) [20:45:05] 3Continuous-Integration: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions - https://phabricator.wikimedia.org/T75521#848869 (10hashar) [20:45:18] 3Continuous-Integration: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#848871 (10hashar) a:3hashar [20:45:36] 3Continuous-Integration: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#755659 (10hashar) I have made it a tracking bug and widened scope to MediaWiki extensions as well. [20:48:48] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [20:51:16] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #322: FAILURE in 3 min 19 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/322/ [20:54:35] 3Mobile-Web, Continuous-Integration: Some MobileFrontend qunit builds failed to archive artifacts (voting -2) - https://phabricator.wikimedia.org/T78590#848930 (10hashar) a:3hashar [20:54:45] 3Mobile-Web, Continuous-Integration: Some MobileFrontend qunit builds failed to archive artifacts (voting -2) - https://phabricator.wikimedia.org/T78590#848819 (10hashar) p:5Triage>3Unbreak! [21:13:46] 3Mobile-Web, Continuous-Integration: Some MobileFrontend qunit builds failed to archive artifacts (voting -2) - https://phabricator.wikimedia.org/T78590#849024 (10hashar) MediaWiki is configured to write debug log under /log/ which is writable by both www-data (for qunit jobs) and jenkins-slave (for PHPUnit in c... [21:13:55] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #1: SUCCESS in 1 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/1/ [21:14:57] 3MediaWiki-General-or-Unknown, Mobile-Web, Continuous-Integration: MediaWiki QUnit test does not wait for all requests to complete, causing a race condition in Jenkins - https://phabricator.wikimedia.org/T78590#849025 (10hashar) a:5hashar>3None [21:47:37] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #1: SUCCESS in 1 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/1/ [21:55:24] Project beta-scap-eqiad build #34061: FAILURE in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34061/ [21:57:53] forgot to change my nick, I've been back a long time :-) [22:15:58] Yippee, build fixed! [22:15:59] Project beta-scap-eqiad build #34063: FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34063/ [22:22:04] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:47:04] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:49:58] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: integration-slave1001.eqiad.wmflabs can't start, mount.nfs yields failure in name resolution - https://phabricator.wikimedia.org/T76250#849427 (10yuvipanda) So I looked at this today - our documented method for getting an interactive console for this seems... [22:50:17] 3Wikimedia-Labs-Infrastructure, Continuous-Integration: integration-slave1001.eqiad.wmflabs can't start, mount.nfs yields failure in name resolution - https://phabricator.wikimedia.org/T76250#849429 (10Krinkle) [[ https://wikitech.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=138053#December_2 |... [22:56:08] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #226: FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/226/ [23:24:43] !log integration-slave1001 isn't coming back (T76250), building integration-slave1005 as its replacement. [23:24:47] Logged the message, Master [23:35:53] Project beta-scap-eqiad build #34069: FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34069/ [23:39:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #192: FAILURE in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/192/ [23:46:30] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%) [23:49:59] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:55:30] Yippee, build fixed! [23:55:30] Project beta-scap-eqiad build #34072: FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34072/