[00:53:35] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<20.00%) [01:03:35] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [01:25:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Evaluate different strategy for Docker CI instances - https://phabricator.wikimedia.org/T202160 (10dduvall) >>! In T202160#4581843, @dduvall wrote: > I'm hoping that a full days worth of builds is enough to comp... [01:34:37] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<20.00%) [01:44:36] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [02:15:39] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<20.00%) [02:25:35] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [02:36:36] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<10.00%) [03:37:39] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<10.00%) [03:42:36] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [05:13:36] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<10.00%) [05:23:35] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [05:36:59] (03PS1) 10Legoktm: seccheck for Echo and Flow [integration/config] - 10https://gerrit.wikimedia.org/r/460473 [05:37:12] (03CR) 10Legoktm: [C: 032] seccheck for Echo and Flow [integration/config] - 10https://gerrit.wikimedia.org/r/460473 (owner: 10Legoktm) [05:38:47] (03Merged) 10jenkins-bot: seccheck for Echo and Flow [integration/config] - 10https://gerrit.wikimedia.org/r/460473 (owner: 10Legoktm) [05:40:21] !log deployed https://gerrit.wikimedia.org/r/460473 [05:40:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:48:13] (03PS1) 10Legoktm: seccheck for CentralNotice [integration/config] - 10https://gerrit.wikimedia.org/r/460475 [05:48:26] (03CR) 10Legoktm: [C: 032] seccheck for CentralNotice [integration/config] - 10https://gerrit.wikimedia.org/r/460475 (owner: 10Legoktm) [05:49:51] (03Merged) 10jenkins-bot: seccheck for CentralNotice [integration/config] - 10https://gerrit.wikimedia.org/r/460475 (owner: 10Legoktm) [05:50:13] !log deployed https://gerrit.wikimedia.org/r/460475 [05:50:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:15:56] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [06:16:12] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [06:25:38] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [06:29:35] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<10.00%) [06:34:33] 10Project-Admins: Requesting personal project tag User-banyek - https://phabricator.wikimedia.org/T204297 (10Banyek) [06:39:35] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [07:28:47] 10Project-Admins: Create phabicator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10rosalieper) [08:03:49] o/ [08:04:08] so on my labs instance (project analytics) I am trying to deploy the analytics/refinery scap repo [08:04:23] but scap seems using deployment-tin by default [08:04:47] and I've added scap::deployment_server: deployment-deploy01.deployment-prep.eqiad.wmflabs to hiera [08:05:56] is there any known config that I am missing or do I probably have it set somewhere? [08:14:33] nuked /srv/deployment/analytics and solved the issue (on the host) [08:18:45] elukey: hello. I think the puppet sets the git remote once and does not have support to update the remote url [08:21:29] ack thanks :) [08:21:44] if you have time, any suggestion about the next failure, namely [08:21:45] fatal: unable to access 'http://deployment-deploy01.deployment-prep.eqiad.wmflabs/analytics/refinery/.git/': Failed to connect to webproxy.eqiad.wmnet port 8080: Connection timed out [08:22:09] I tried to telnet and indeed it doesn't work, but it is the first time that I see a webproxy error in labs [08:22:33] pbably an issue somewhere in the puppet provider modules/scap/lib/puppet/provider/scap_source/default.rb : [08:22:41] ah webproxy bah [08:23:06] I am not sure whether webproxy.eqiad.wmnet is reacheable from WMCS instances anymore [08:26:16] lovely :) [08:27:13] Alⅼɑh is doⅰᥒɡ [08:27:15] suᥒ is not ⅾoіᥒg Аllah is ԁοing [08:27:18] ⅿooᥒ іs nοt doiᥒg Alⅼɑһ iѕ doіᥒg [08:27:21] ѕtаrs аre not doinɡ Allah іs ⅾoⅰᥒɡ [08:27:24] рⅼaᥒеts аrᥱ not ԁоіᥒg Αⅼlaһ is dοⅰᥒg [08:27:26] Alⅼaһ is ԁoіng [08:27:26] ɡalaxieѕ are ᥒഠt dοiᥒɡ Alⅼɑh ⅰs dοіᥒg [08:27:29] ѕ∪n iѕ nⲟt dοing Αllaһ iѕ dഠinɡ [08:27:29] oceans are ᥒot doing Αⅼlaһ is dοiᥒɡ [08:27:31] ⅿഠon is ᥒot ԁoіᥒɡ Allаh is ԁoinɡ [08:27:34] stɑrs ɑre ᥒot dοiᥒɡ Ꭺllɑh is doing [08:27:37] рlanᥱts аrᥱ nⲟt ԁoіng Αⅼⅼɑh іѕ ԁⲟіng [08:27:39] ɡɑlaхiᥱѕ arᥱ not ԁоinɡ Aⅼlɑh is doinɡ [08:27:42] oⅽeans are not ⅾоiᥒɡ Ꭺlⅼah ⅰs doⅰᥒg [08:29:38] good bot :) [08:29:50] elukey: can you paste what is happening? [08:35:36] Αlⅼаh іѕ ԁoⅰng [08:35:39] ѕ∪n іs ᥒot doiᥒg Аllɑh іs doing [08:35:41] mⲟon iѕ ᥒഠt ԁoiᥒg Αllah іs dоіng [08:35:44] stɑrs are not ԁoⅰnɡ Aⅼlaһ is doⅰᥒɡ [08:35:47] planets ɑre not doing Аllaһ is ԁoіᥒg [08:35:49] hashar: sure https://phabricator.wikimedia.org/P7546 [08:35:49] gɑlаxiеs arе ᥒοt ԁοinɡ Аⅼlɑһ іs ԁοinɡ [08:35:52] oceɑns ɑre not dοiᥒg Ꭺllaһ iѕ ԁoinɡ [08:35:55] mountɑinѕ are not doіnɡ Allаh іѕ ⅾοіng [08:39:38] elukey: fun. I have no clue :\ [08:41:39] elukey: so that works for me [08:41:46] y$ /usr/bin/scap deploy-local --repo analytics/refinery -D log_json:False [08:41:51] 08:41:31 Fetch from: http://deployment-deploy01.deployment-prep.eqiad.wmflabs/analytics/refinery/.git [08:41:51] 08:41:32 Checkout rev: 407da92cede0de4ef6207d6139ed1cc8cc185516 [08:41:51] 08:41:32 Pulling large objects [using git-fat] [08:42:10] on what host? [08:42:21] deployment-deploy01 [08:42:27] with my user [08:43:50] elukey: maybe try with GIT_TRACE_CURL=1 ? [08:43:59] (not sure whether that debug env variable is passed by scap to git) [08:44:27] will try thanks! [09:16:06] 10Phabricator (Upstream), 10Upstream: Per-user projects for personal work in progress tracking - https://phabricator.wikimedia.org/T555 (10Peachey88) [09:16:08] 10Project-Admins: Requesting personal project tag User-banyek - https://phabricator.wikimedia.org/T204297 (10Peachey88) [09:16:14] 10Project-Admins: Requesting personal project tag User-banyek - https://phabricator.wikimedia.org/T204297 (10Peachey88) a:05Aklapper>03None [09:19:23] (03CR) 10Hashar: "> each of the three repos is failing for a different reason, and none of them are related to CI. I need to create this job for several mor" [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:36:16] (03CR) 10Hashar: "Seems df ignore the missing partition and show the rest, but it exit with exit code 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/460176 (owner: 10Thcipriani) [10:09:23] 10Project-Admins: Create phabicator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10Aklapper) Hi, is this hosted somewhere? :) I think the `tools` in the URL https://gerrit.wikimedia.org/r/admin/projects/labs/tools/Commons-twitter-bot implies that it should be a subproject of #Tools... [10:34:55] 10Continuous-Integration-Config, 10Gerrit: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [10:37:46] 10Continuous-Integration-Config, 10Gerrit: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [10:38:31] 10Continuous-Integration-Config, 10Gerrit: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [10:47:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [10:55:54] (03PS1) 10Hashar: Add labs/tools/cdnjs-index [integration/config] - 10https://gerrit.wikimedia.org/r/460503 (https://phabricator.wikimedia.org/T180318) [10:57:43] 10Continuous-Integration-Config, 10Gerrit, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [11:01:27] 10Continuous-Integration-Config, 10Gerrit, 10Tools, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [11:12:07] 10Continuous-Integration-Config, 10Gerrit, 10Tools, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [11:16:01] (03PS1) 10Hashar: Edit Project Config [wikimedia-cz] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/460509 [11:16:13] what? [11:16:16] Allɑһ iѕ ԁoiᥒg [11:16:19] sun iѕ ᥒഠt doⅰᥒg Aⅼlah is dⲟing [11:16:21] ⅿoon is ᥒot doiᥒg Aⅼⅼаһ іѕ dഠiᥒɡ [11:16:24] stɑrѕ are not ⅾoing Aⅼⅼаh iѕ dοіng [11:16:27] ⲣⅼaᥒetѕ are ᥒot doiᥒg Аⅼlɑһ is ⅾoіng [11:16:29] gaⅼɑхies ɑrᥱ not ԁഠiᥒg Aⅼlaһ іѕ doing [11:18:20] Allɑһ iѕ ԁoiᥒg [11:18:23] sun iѕ ᥒഠt doⅰᥒg Aⅼlah is dⲟing [11:18:25] ⅿoon is ᥒot doiᥒg Aⅼⅼаһ іѕ dഠiᥒɡ [11:18:27] errr [11:18:28] stɑrѕ are not ⅾoing Aⅼⅼаh iѕ dοіng [11:18:28] hashar ^^ [11:18:31] ⲣⅼaᥒetѕ are ᥒot doiᥒg Аⅼlɑһ is ⅾoіng [11:18:34] gaⅼɑхies ɑrᥱ not ԁഠiᥒg Aⅼlaһ іѕ doing [11:21:34] Project beta-scap-eqiad build #221922: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221922/ [11:22:06] Project beta-update-databases-eqiad build #28312: 04FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28312/ [11:22:25] Project beta-scap-eqiad build #221923: 04STILL FAILING in 4.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221923/ [11:25:24] bah [11:30:46] 10Beta-Cluster-Infrastructure: Beta cluster update.php fails for en_rtlwiki: sudo: no tty present and no askpass program specified - https://phabricator.wikimedia.org/T204340 (10hashar) [11:31:13] !log Manually running https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ | T204340 [11:31:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:31:19] T204340: Beta cluster update.php fails for en_rtlwiki: sudo: no tty present and no askpass program specified - https://phabricator.wikimedia.org/T204340 [11:33:42] Yippee, build fixed! [11:33:43] Project beta-update-databases-eqiad build #28313: 09FIXED in 2 min 49 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ [11:36:29] Project beta-scap-eqiad build #221924: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221924/ [11:37:45] 10Beta-Cluster-Infrastructure: Beta cluster update.php fails for en_rtlwiki: sudo: no tty present and no askpass program specified - https://phabricator.wikimedia.org/T204340 (10hashar) 05Open>03Resolved a:03hashar It is working now [11:38:54] '/tmp/scap_l10n_773633672/l10n_cache-mai.cdb': No space left on device [11:38:56] bah [11:49:29] !log Cleaned /tmp on deployment-deploy01 [11:49:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:50:40] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: mwselenium-quibble-docker jobs broken on WikibaseLexeme - https://phabricator.wikimedia.org/T204214 (10Addshore) Is this fixed? :D [11:51:19] Yippee, build fixed! [11:51:19] Project beta-scap-eqiad build #221925: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221925/ [11:55:19] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-Recent-changes, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), and 2 others: Recent Changes shows Internal Error on de.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T203759 (10Ladsgroup) 05Open>03Resolved It's... [12:14:44] (03PS1) 10Hashar: Bump mwselenium-quibble-docker to Chromium 60 [integration/config] - 10https://gerrit.wikimedia.org/r/460512 (https://phabricator.wikimedia.org/T204214) [12:16:38] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: mwselenium-quibble-docker jobs broken on WikibaseLexeme - https://phabricator.wikimedia.org/T204214 (10hashar) WikibaseLexeme has bee... [12:20:40] (03PS1) 10Hashar: docker: remove Firefox from quibble-stretch-bundle [integration/config] - 10https://gerrit.wikimedia.org/r/460513 (https://phabricator.wikimedia.org/T204214) [12:22:21] (03PS1) 10Hashar: Update mediawiki_selenium job to drop Firefox [integration/config] - 10https://gerrit.wikimedia.org/r/460514 (https://phabricator.wikimedia.org/T204214) [12:23:00] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: mwselenium-quibble-docker jobs broken on WikibaseLexeme - https://phabricator.wikimedia.org/T204214 (10hashar) And the remove Firefox... [12:23:06] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: mwselenium-quibble-docker jobs broken on WikibaseLexeme - https://phabricator.wikimedia.org/T204214 (10hashar) a:03hashar [12:31:15] (03CR) 10Zfilipin: "> Patch Set 7:" [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [12:35:08] (03PS1) 10Zfilipin: Create selenium-daily-beta-AdvancedSearch Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) [12:37:36] (03PS1) 10Zfilipin: Create selenium-daily-beta-ORES Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) [13:14:35] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10Niedzielski) This happened again for readers-web-backlog but the fix su... [13:21:53] (03PS1) 10Zfilipin: Create selenium-daily-beta-Wikibase Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460526 (https://phabricator.wikimedia.org/T188742) [13:22:19] Αlⅼaһ іs doiᥒg [13:23:25] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<10.00%) [13:23:49] (03PS1) 10Zfilipin: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) [13:24:59] (03Abandoned) 10Zfilipin: WIP selenium-daily-beta jobs [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:27:32] Project beta-scap-eqiad build #221932: 04FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221932/ [13:28:05] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [13:29:47] (03CR) 10Zhuyifei1999: [C: 031] Add labs/tools/cdnjs-index [integration/config] - 10https://gerrit.wikimedia.org/r/460503 (https://phabricator.wikimedia.org/T180318) (owner: 10Hashar) [13:29:56] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [13:34:32] Project beta-scap-eqiad build #221933: 04STILL FAILING in 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221933/ [13:40:41] (03PS1) 10Hashar: Ignore ./quibble/test.py when generating coverage [integration/quibble] - 10https://gerrit.wikimedia.org/r/460533 [13:40:51] (03CR) 10Hashar: [C: 032] Ignore ./quibble/test.py when generating coverage [integration/quibble] - 10https://gerrit.wikimedia.org/r/460533 (owner: 10Hashar) [13:41:36] (03Merged) 10jenkins-bot: Ignore ./quibble/test.py when generating coverage [integration/quibble] - 10https://gerrit.wikimedia.org/r/460533 (owner: 10Hashar) [13:42:24] (03CR) 10jenkins-bot: Ignore ./quibble/test.py when generating coverage [integration/quibble] - 10https://gerrit.wikimedia.org/r/460533 (owner: 10Hashar) [13:44:34] Project beta-scap-eqiad build #221934: 04STILL FAILING in 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221934/ [13:48:11] !log Tagged Quibble 0.0.25 01663f5d9bcc9e77473732a9fd230271bbd43db4 [13:48:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:54:00] (03PS1) 10Hashar: docker: quibble 0.0.25 [integration/config] - 10https://gerrit.wikimedia.org/r/460536 [13:54:30] Project beta-scap-eqiad build #221935: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221935/ [13:55:02] 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28) [13:55:10] !log deployment-deploy01 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28) [13:55:11] :( [13:55:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:57:33] !log deployment-deploy01: mv /srv/mediawiki/php-master/cache/l10n/*.php /srv/mediawiki/php-master/cache/l10n/backup-php/ [13:57:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:06:26] Project beta-scap-eqiad build #221936: 04STILL FAILING in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221936/ [14:14:34] so [14:14:57] deployment-deploy01 has a giant /srv/mediawiki/.git for some reason [14:15:13] I am just gonna nuke it [14:16:30] Project beta-scap-eqiad build #221937: 04STILL FAILING in 2 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221937/ [14:16:48] !log deployment-deploy01: sudo rm -fR /srv/mediawiki && mkdir /srv/mediawiki && chown mwdeploy:mwdeploy /srv/mediawiki && scap pull [14:16:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:18:22] and there is some huge /srv/deployment/analytics/refinery-cache [14:18:29] scap related :\ [14:20:06] Project beta-update-databases-eqiad build #28316: 04FAILURE in 5.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28316/ [14:27:51] !log deployment-deploy01 /srv/mediawiki synced [14:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:29:38] Yippee, build fixed! [14:29:39] Project beta-update-databases-eqiad build #28317: 09FIXED in 2 min 34 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28317/ [14:36:18] PROBLEM - Puppet errors on deployment-deploy01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:38:11] Yippee, build fixed! [14:38:12] Project beta-scap-eqiad build #221938: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221938/ [14:52:09] 10Phabricator, 10Operations, 10Traffic: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10ema) [14:52:22] 10Phabricator, 10Operations, 10Traffic: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10ema) p:05Triage>03Normal [14:53:24] hashar: o/ - I added a proxy to the deployment-prep project to be able to contact deployment-deploy01:80 [14:53:42] that seems to work fine, but I have not find a way to make it use by scap [14:54:02] probably the easiest for me is to recreate deployment-deploy01 in my project :( [14:56:08] 10Phabricator, 10Operations, 10Traffic: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10Dzahn) I hit edit on https://phabricator.wikimedia.org/phame/blog/edit/11/ and from there clicked "Custom Policy" -> "Advanced"->"Custom Policy" next to "Editable by"... [15:06:32] PROBLEM - Puppet errors on deployment-deploy02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:22:16] 10MediaWiki-Codesniffer: Add sniff for order of public and static in class property declaration - https://phabricator.wikimedia.org/T204357 (10Umherirrender) [15:26:54] Αⅼlɑһ іs dоiᥒg [15:26:54] suᥒ iѕ not dഠinɡ Alⅼɑh is ԁoіng [15:53:17] (03PS3) 10Thcipriani: Refactor maintenance to timeout after 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/460174 (https://phabricator.wikimedia.org/T204077) [15:54:08] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [15:59:04] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [15:59:47] (03PS1) 10Zfilipin: Create selenium-daily-beta-ElectronPdfService Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460552 (https://phabricator.wikimedia.org/T188742) [16:01:04] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:07:30] (03PS1) 10Zfilipin: Create selenium-daily-beta-Newsletter Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) [16:07:40] (03CR) 10Thcipriani: [C: 032] "This has been deployed and seems to be working as expected (https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/264" [integration/config] - 10https://gerrit.wikimedia.org/r/460174 (https://phabricator.wikimedia.org/T204077) (owner: 10Thcipriani) [16:09:48] (03PS2) 10Thcipriani: Add /var/lib/docker partition to maintenance check [integration/config] - 10https://gerrit.wikimedia.org/r/460176 [16:09:51] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:11:42] (03Merged) 10jenkins-bot: Refactor maintenance to timeout after 5 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/460174 (https://phabricator.wikimedia.org/T204077) (owner: 10Thcipriani) [16:21:15] (03CR) 10Thcipriani: [C: 032] "> Seems df ignore the missing partition and show the rest, but it" [integration/config] - 10https://gerrit.wikimedia.org/r/460176 (owner: 10Thcipriani) [16:22:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: maintenance-disconnect-full-disks gets stuck - https://phabricator.wikimedia.org/T204077 (10thcipriani) 05Open>03Resolved Deployed this Thursday and actually did have a build timeout yesterday and nobody had... [16:22:28] (03CR) 10Bstorm: [C: 031] Add labs/tools/cdnjs-index [integration/config] - 10https://gerrit.wikimedia.org/r/460503 (https://phabricator.wikimedia.org/T180318) (owner: 10Hashar) [16:23:26] (03Merged) 10jenkins-bot: Add /var/lib/docker partition to maintenance check [integration/config] - 10https://gerrit.wikimedia.org/r/460176 (owner: 10Thcipriani) [16:25:54] (03PS1) 10Zfilipin: Create selenium-daily-beta-TwoColConflict Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) [16:28:45] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:38:15] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:38:42] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:39:19] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [16:48:13] 10Phabricator, 10Operations, 10Traffic: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10greg) {F25864089} Ya'll should be able to do that now (ftr: I didn't make any additions, that's what it looked like after mutante's edits). The "custom policy" thing... [16:49:13] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:51:25] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [16:58:03] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [17:11:31] (03CR) 10Hashar: [C: 032] Add labs/tools/cdnjs-index [integration/config] - 10https://gerrit.wikimedia.org/r/460503 (https://phabricator.wikimedia.org/T180318) (owner: 10Hashar) [17:13:01] (03Merged) 10jenkins-bot: Add labs/tools/cdnjs-index [integration/config] - 10https://gerrit.wikimedia.org/r/460503 (https://phabricator.wikimedia.org/T180318) (owner: 10Hashar) [17:14:36] 10Continuous-Integration-Config, 10Gerrit, 10Tools, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [17:15:30] Αlⅼаh ⅰs doіnɡ [17:15:30] ѕun iѕ not doіᥒɡ Αllaһ iѕ dοinɡ [17:15:30] mooᥒ іѕ nഠt ⅾoinɡ Allah is dоⅰnɡ [17:20:12] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:35:15] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:36:19] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:37:33] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:38:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:39:21] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:40:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:41:12] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:44:22] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:45:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:47:22] !log adding instance-type-* labels to m4executor nodes in jenkins [17:47:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:47:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:48:18] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [17:50:43] Allɑh ⅰs doіnɡ [17:52:01] Aⅼⅼɑh іs doinɡ [17:52:01] sᥙᥒ is ᥒot doing Allah is doіᥒg [17:52:01] ⅿⲟon iѕ ᥒot dഠinɡ Allaһ iѕ doinɡ [18:06:05] Allaһ iѕ ԁοing [18:06:38] (03CR) 10Zfilipin: Create selenium-daily-beta-AdvancedSearch Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [18:08:54] (03CR) 10Zfilipin: Create selenium-daily-beta-ElectronPdfService Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460552 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [18:11:13] (03CR) 10Zfilipin: Create selenium-daily-beta-Newsletter Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [18:12:25] (03CR) 10Zfilipin: Create selenium-daily-beta-TwoColConflict Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [18:14:23] (03CR) 10Zfilipin: Create selenium-daily-beta-ORES Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [18:16:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Request increased quota for integration Cloud VPS project - https://phabricator.wikimedia.org/T204373 (10dduvall) [18:16:56] (03PS1) 10MarcoAurelio: Archive the CommunityTwitter extension [integration/config] - 10https://gerrit.wikimedia.org/r/460578 (https://phabricator.wikimedia.org/T204372) [18:17:37] (03PS2) 10MarcoAurelio: Archive the CommunityTwitter extension [integration/config] - 10https://gerrit.wikimedia.org/r/460578 (https://phabricator.wikimedia.org/T204372) [18:22:22] (03CR) 10Zfilipin: Create selenium-daily-beta-Wikibase Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460526 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [18:28:58] (03PS1) 10Thcipriani: Remove stopped containers after build [integration/config] - 10https://gerrit.wikimedia.org/r/460581 [18:31:41] (03PS2) 10Thcipriani: Remove stopped containers after build [integration/config] - 10https://gerrit.wikimedia.org/r/460581 [18:46:58] (03PS3) 10Thcipriani: Remove containers after build [integration/config] - 10https://gerrit.wikimedia.org/r/460581 [18:49:05] (03CR) 10Dduvall: [C: 031] "Yes!" [integration/config] - 10https://gerrit.wikimedia.org/r/460581 (owner: 10Thcipriani) [18:53:40] Ꭺlⅼaһ ⅰs doⅰnɡ [18:53:54] (03PS1) 10Dduvall: Remove containers after run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460586 [18:54:05] (03CR) 10jerkins-bot: [V: 04-1] Remove containers after run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460586 (owner: 10Dduvall) [18:58:17] (03CR) 10Thcipriani: [C: 032] "Deployed, working https://integration.wikimedia.org/ci/job/service-pipeline/51" [integration/config] - 10https://gerrit.wikimedia.org/r/460581 (owner: 10Thcipriani) [19:00:05] (03Merged) 10jenkins-bot: Remove containers after build [integration/config] - 10https://gerrit.wikimedia.org/r/460581 (owner: 10Thcipriani) [19:01:08] (03PS1) 10Dduvall: ci: Load dependencies as a separate gradle task [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460588 [19:07:45] (03CR) 10Thcipriani: [C: 032] ci: Load dependencies as a separate gradle task [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460588 (owner: 10Dduvall) [19:08:49] (03Merged) 10jenkins-bot: ci: Load dependencies as a separate gradle task [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460588 (owner: 10Dduvall) [19:09:56] (03PS2) 10Thcipriani: Remove containers after run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460586 (owner: 10Dduvall) [19:13:06] (03CR) 10Thcipriani: [C: 032] Remove containers after run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460586 (owner: 10Dduvall) [19:13:31] (03Merged) 10jenkins-bot: Remove containers after run [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/460586 (owner: 10Dduvall) [19:36:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Quibble: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 (10thcipriani) [19:43:56] Allаh іѕ ԁoiᥒɡ [20:01:13] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747 (10hashar) [[ https://issues.jenkins-ci.org/browse/JENKINS-17116 | J... [20:21:12] Alⅼaһ ⅰѕ dഠinɡ [20:21:12] s∪n ⅰѕ not ⅾoiᥒg Aⅼⅼaһ іѕ dоing [21:03:07] Aⅼⅼah іs dоiᥒɡ [21:19:16] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: Update Debian Package for Scap to 3.8.6-1 - https://phabricator.wikimedia.org/T204383 (10thcipriani) p:05Triage>03High [21:21:41] Project beta-scap-eqiad build #221965: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221965/ [21:21:54] Project beta-update-databases-eqiad build #28324: 04FAILURE in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28324/ [21:25:46] what, a sudo error from wmf-beta-update-databases.py ? ^ [21:26:10] https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28324/console [21:36:32] Yippee, build fixed! [21:36:33] Project beta-scap-eqiad build #221966: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221966/ [21:38:08] Allɑһ is doing [21:58:13] greg-g: still around? [22:22:18] Yippee, build fixed! [22:22:19] Project beta-update-databases-eqiad build #28325: 09FIXED in 2 min 17 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28325/ [23:08:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Request increased quota for integration Cloud VPS project - https://phabricator.wikimedia.org/T204373 (10dduvall) [23:08:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Evaluate different strategy for Docker CI instances - https://phabricator.wikimedia.org/T202160 (10dduvall)