[03:58:38] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #426: 04FAILURE in 2 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/426/ [04:09:33] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [04:16:55] Yippee, build fixed! [04:16:56] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #426: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/426/ [06:32:08] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #395: 04FAILURE in 1 hr 52 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/395/ [07:04:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:47:35] 10Release-Engineering-Team (Kanban), 10Phabricator: Unhandled Exception ("InvalidArgumentException") error when clicking on search link in "Activity Feed" panel - https://phabricator.wikimedia.org/T168108#3358091 (10mmodell) p:05Triage>03Normal a:03mmodell [08:16:58] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: Unhandled Exception ("InvalidArgumentException") error when clicking on search link in "Activity Feed" panel - https://phabricator.wikimedia.org/T168108#3358111 (10mmodell) upstreamed: https://secure.phabricator.com/T12851 [09:47:18] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:22:19] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:31:55] 10Continuous-Integration-Scaling, 10Repository-Admins: New git repository: - https://phabricator.wikimedia.org/T168195#3358197 (10Waxmiguel) [12:20:46] 10Continuous-Integration-Scaling, 10Repository-Admins: New git repository: - https://phabricator.wikimedia.org/T168195#3358280 (10Luke081515) 05Open>03Invalid [15:15:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [15:29:34] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [15:47:10] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #459: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/459/ [15:55:41] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #459: 04FAILURE in 33 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/459/ [16:01:38] Yippee, build fixed! [16:01:38] Project selenium-CentralNotice » firefox,beta,Linux,BrowserTests build #430: 09FIXED in 37 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/430/ [16:50:54] 10Release-Engineering-Team (Kanban), 10Phabricator: Remove "Release Details" field from simple task creation form or explain what it is / means - https://phabricator.wikimedia.org/T168186#3358471 (10mmodell) 05Open>03Resolved a:03mmodell Sorry about that @aklapper, I meant to hide it. [16:58:59] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3358476 (10Paladox) I've got the full log here ``` [2017-06-18 16:42:32,415] [HTTP-77] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /gerrit/changes... [17:15:47] 10Release-Engineering-Team (Kanban), 10Phabricator: "Security" field is invisible, but still exists, and the value can't be changed, causing Herald problems - https://phabricator.wikimedia.org/T153742#3358480 (10mmodell) 05Open>03Resolved a:03mmodell [17:23:30] 10Release-Engineering-Team (Kanban), 10Phabricator: "Security" field is invisible, but still exists, and the value can't be changed, causing Herald problems - https://phabricator.wikimedia.org/T153742#3358489 (10mmodell) @aklapper: you should be able to edit the policy on tasks if you weren't previously. [19:11:30] 10Beta-Cluster-Infrastructure, 10Wikidata: Wikidata on beta is getting too many edits - https://phabricator.wikimedia.org/T168101#3358595 (10Reedy) [19:11:32] 10Beta-Cluster-Infrastructure, 10Wikidata: Prune items and properties from beta wikidatawiki - https://phabricator.wikimedia.org/T168106#3358592 (10Reedy) 05Open>03Resolved a:03Reedy I've killed a load more now. Content pages 9,649 Pages 14,718 [19:20:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3358602 (10Reedy) So after nuking hundreds of thousands of items and properties... ``` ...index t... [19:22:34] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3358603 (10Reedy) a:03aude I'm guessing it's pretty much related to https://github.com/wikimedia... [19:25:18] !log A lot of items on beta wikidatawiki deleted T168036 T167981 [19:25:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:25:23] T168036: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036 [19:25:24] T167981: beta-update-databases-eqiad times out on wikidatawiki (maintenance update) - https://phabricator.wikimedia.org/T167981 [19:26:53] !log Re-enabled beta-update-databases-eqiad as wikidatawiki takes < 10 minutes T168036 T167981 [19:26:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:36:10] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3358616 (10Paladox) @demon could you pull in the 2.13 branch instead please (since the fix was merged ) also we need this other fix https://gerrit-review.googlesource.com/... [19:55:26] Project beta-scap-eqiad build #160323: 04FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160323/ [19:59:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap: beta-scap-eqiad failed and linked to production logstash - https://phabricator.wikimedia.org/T168211#3358637 (10Reedy) [20:03:26] Yippee, build fixed! [20:03:27] Project beta-scap-eqiad build #160324: 09FIXED in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160324/ [21:35:23] Project beta-scap-eqiad build #160334: 04FAILURE in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160334/ [21:45:45] Yippee, build fixed! [21:45:46] Project beta-scap-eqiad build #160335: 09FIXED in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160335/ [22:57:37] 10Continuous-Integration-Config, 10Wikimedia-Site-requests, 10Patch-For-Review: Enforce our PHP coding standards on operations/mediawiki-config via composer? - https://phabricator.wikimedia.org/T162835#3358732 (10Reedy)