[00:07:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL [03:44:52] Project beta-code-update-eqiad build #131772: 04FAILURE in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131772/ [03:54:56] Project beta-code-update-eqiad build #131773: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131773/ [04:04:56] Project beta-code-update-eqiad build #131774: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131774/ [04:14:54] Project beta-code-update-eqiad build #131775: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131775/ [04:24:55] Project beta-code-update-eqiad build #131776: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131776/ [04:34:51] Project beta-code-update-eqiad build #131777: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131777/ [04:44:51] Project beta-code-update-eqiad build #131778: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131778/ [04:54:51] Project beta-code-update-eqiad build #131779: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131779/ [05:04:53] Project beta-code-update-eqiad build #131780: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131780/ [05:14:50] Project beta-code-update-eqiad build #131781: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131781/ [05:24:52] Project beta-code-update-eqiad build #131782: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131782/ [05:34:52] Project beta-code-update-eqiad build #131783: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131783/ [05:44:50] Project beta-code-update-eqiad build #131784: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131784/ [05:54:56] Project beta-code-update-eqiad build #131785: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131785/ [06:04:53] Project beta-code-update-eqiad build #131786: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131786/ [06:14:54] Project beta-code-update-eqiad build #131787: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131787/ [06:24:48] Project beta-code-update-eqiad build #131788: 04STILL FAILING in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131788/ [06:34:51] Project beta-code-update-eqiad build #131789: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131789/ [06:44:51] Project beta-code-update-eqiad build #131790: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131790/ [06:54:49] Project beta-code-update-eqiad build #131791: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131791/ [07:04:54] Project beta-code-update-eqiad build #131792: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131792/ [07:14:53] Project beta-code-update-eqiad build #131793: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131793/ [07:24:50] Project beta-code-update-eqiad build #131794: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131794/ [07:34:55] Project beta-code-update-eqiad build #131795: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131795/ [07:44:50] Project beta-code-update-eqiad build #131796: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131796/ [07:54:53] Project beta-code-update-eqiad build #131797: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131797/ [08:04:56] Project beta-code-update-eqiad build #131798: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131798/ [08:08:46] 03:43:00 INFO:main:Starting updating tasks... [08:08:46] 03:43:00 INFO:mwcore:cwd: /srv/mediawiki-staging/php-master [08:08:46] 03:43:00 INFO:mwcore:running: git pull [08:08:46] 03:43:03 remote: internal server error [08:08:46] 03:43:03 fatal: protocol error: bad pack header [08:14:55] Project beta-code-update-eqiad build #131799: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131799/ [08:24:56] Project beta-code-update-eqiad build #131800: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131800/ [08:34:53] Project beta-code-update-eqiad build #131801: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131801/ [08:44:55] Project beta-code-update-eqiad build #131802: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131802/ [08:54:53] Project beta-code-update-eqiad build #131803: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131803/ [09:04:54] Project beta-code-update-eqiad build #131804: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131804/ [09:14:53] Project beta-code-update-eqiad build #131805: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131805/ [09:24:52] Project beta-code-update-eqiad build #131806: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131806/ [09:34:53] Project beta-code-update-eqiad build #131807: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131807/ [09:44:51] Project beta-code-update-eqiad build #131808: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131808/ [09:54:56] Project beta-code-update-eqiad build #131809: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131809/ [10:04:54] Project beta-code-update-eqiad build #131810: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131810/ [10:14:58] Project beta-code-update-eqiad build #131811: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131811/ [10:24:50] Project beta-code-update-eqiad build #131812: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131812/ [10:34:52] Project beta-code-update-eqiad build #131813: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131813/ [10:44:53] Project beta-code-update-eqiad build #131814: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131814/ [10:54:51] Project beta-code-update-eqiad build #131815: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131815/ [10:57:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: Build step 'Execute a set of scripts' marked build as failure - https://phabricator.wikimedia.org/T151670#2824272 (10TTO) Possibly UBN? Is it blocking merge of all core patches? [10:59:47] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: Build step 'Execute a set of scripts' marked build as failure - https://phabricator.wikimedia.org/T151670#2824274 (10Ladsgroup) p:05Triage>03Unbreak! [11:00:05] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: Build step 'Execute a set of scripts' marked build as failure - https://phabricator.wikimedia.org/T151670#2824227 (10Ladsgroup) It's blocking everything: https://gerrit.wikimedia.org/r/#/c/323649/ [11:00:41] FYI: https://phabricator.wikimedia.org/T151670 [11:04:54] Project beta-code-update-eqiad build #131816: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131816/ [11:07:36] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: Build step 'Execute a set of scripts' marked build as failure - https://phabricator.wikimedia.org/T151670#2824280 (10Peachey88) [11:14:53] Project beta-code-update-eqiad build #131817: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131817/ [11:24:53] Project beta-code-update-eqiad build #131818: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131818/ [11:34:49] Project beta-code-update-eqiad build #131819: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131819/ [11:44:50] Project beta-code-update-eqiad build #131820: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131820/ [11:54:50] Project beta-code-update-eqiad build #131821: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131821/ [12:04:54] Project beta-code-update-eqiad build #131822: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131822/ [12:14:54] Project beta-code-update-eqiad build #131823: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131823/ [12:22:50] Yippee, build fixed! [12:22:51] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #222: 09FIXED in 50 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/222/ [12:24:53] Project beta-code-update-eqiad build #131824: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131824/ [12:34:52] Project beta-code-update-eqiad build #131825: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131825/ [12:44:53] Project beta-code-update-eqiad build #131826: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131826/ [12:45:07] 12:33:00 INFO:mwcore:cwd: /srv/mediawiki-staging/php-master [12:45:08] 12:33:00 INFO:mwcore:running: git pull [12:45:08] 12:33:02 remote: internal server error [12:45:46] WFM [12:46:57] root@deployment-tin:/srv/mediawiki-staging/php-master# sudo -u jenkins-deploy git pull [12:46:57] remote: internal server error [12:46:57] fatal: protocol error: bad pack header [12:50:51] 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10Reedy) [12:52:17] 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10Reedy) Seems fine locally [12:54:50] Project beta-code-update-eqiad build #131827: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131827/ [12:54:53] 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824346 (10Reedy) [12:58:38] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: Build step 'Execute a set of scripts' marked build as failure - https://phabricator.wikimedia.org/T151670#2824348 (10TTO) [12:58:41] 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824351 (10TTO) [12:58:53] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: Build step 'Execute a set of scripts' marked build as failure - https://phabricator.wikimedia.org/T151670#2824227 (10TTO) Merged upwards since Reedy seems to have spotted the underlying issue [12:59:16] 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10TTO) p:05Triage>03Unbreak! copying UBN from other task [13:04:55] Project beta-code-update-eqiad build #131828: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131828/ [13:05:37] !log marked deployment-tin as offline due to T151670 [13:05:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:24:17] 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824376 (10Reedy) Clean clone of all of core over HTTPS is fine too [13:30:24] 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824381 (10Paladox) It is affecting my local pc too. [13:33:16] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824383 (10Paladox) Adding #operations as this seems like this will require gerrit's server to be accessed to fix this. [13:46:18] Yippee, build fixed! [13:46:19] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #225: 09FIXED in 2 min 18 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/225/ [13:52:09] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824399 (10Paladox) I found this https://bugs.chromium.org/p/gerrit/issues/detail?id=1954 and seems to be the error as we are having. the error should be in the logs. /var/l... [13:59:12] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824401 (10Paladox) Re cloning worked for me. [14:14:11] !log moved old /srv/mediawiki-staging/php-master to /tmp/php-master, recloned MW Core, copied in LocalSettings, skins, vendor and extensions. T151676. scap sync-dir running [14:14:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:14:44] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824410 (10Paladox) [14:14:46] 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2824409 (10Paladox) [14:15:49] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10Paladox) There seems to be a fix for gc in jgit 4.3.1 that looks like something that will fix our problem. Note I am not 100% correct but hopefully it will. https:... [14:17:43] 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2824412 (10Paladox) p:05Low>03Normal Raising priority to normal as we will be updating very soon. [14:28:41] Project beta-update-databases-eqiad build #13082: 04FAILURE in 0.45 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13082/ [14:28:47] shush [14:29:13] Project beta-code-update-eqiad build #131829: 04STILL FAILING in 32 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131829/ [14:33:36] Project beta-code-update-eqiad build #131830: 04STILL FAILING in 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131830/ [14:36:14] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:37:51] 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2824442 (10Paladox) p:05Normal>03High Raising to high pri as this T151676 should make this a priority to fix. I doint know 100% that updating to 2.13 will fix it, but jgit 4.3.1 looks promising. @demon could... [14:46:41] Project beta-code-update-eqiad build #131831: 04STILL FAILING in 3 min 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131831/ [14:48:49] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824459 (10Reedy) Quick workaround is: ``` cp .git/config .git/config.backup git remote remove origin mv .git/config.backup .git/config git fetch ``` Thanks to @Volans on P4510 [14:50:46] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10ArielGlenn) guess I should add the info from the logs here: /var/lib/gerrit2/review_site/logs/error_log, the first relevant thing seems to be [2016-11-26 00:44:... [15:00:14] Project mediawiki-core-code-coverage build #2412: 04FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2412/ [15:05:10] Yippee, build fixed! [15:05:11] Project beta-code-update-eqiad build #131832: 09FIXED in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131832/ [15:15:08] 10Gerrit, 13Patch-For-Review: Update gerrit to 2.13.3 - https://phabricator.wikimedia.org/T146350#2824494 (10Paladox) The logs provided by @ArielGlenn seems to confirm that the fix in jgit 4.3.1 will fix it for us as a602a8b2400730a8d9a0d20b79bfb55ec0de4d86 is an abandoned commit so if I am correct a602a8b2400... [15:15:28] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [15:21:25] Yippee, build fixed! [15:21:26] Project beta-update-databases-eqiad build #13083: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13083/ [15:25:16] Project mediawiki-core-code-coverage build #2413: 04STILL FAILING in 9.9 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2413/ [15:31:13] Project mediawiki-core-code-coverage build #2414: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2414/ [15:35:39] Project mediawiki-core-code-coverage build #2415: 04STILL FAILING in 13 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2415/ [15:44:25] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:50:59] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #241: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/241/ [15:51:02] !log deleted /srv/jenkins-workspace/workspace/mediawiki-core-code-coverage/src on integration-slave-trusty-1006 to force a reclone [15:51:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:08:36] Yippee, build fixed! [16:08:36] Project mediawiki-core-code-coverage build #2416: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2416/ [16:15:15] !log killed /srv/jenkins-workspace/workspace/mediawiki-core-*/src and /srv/jenkins-workspace/workspace/mwext-*/src from integration slaves to get rid of borked MW dirs [16:15:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:20:22] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824506 (10Reedy) p:05Unbreak!>03High [16:20:44] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10Reedy) Lowered slightly; beta and CI have been unbroken, still some gerrit related stuff to be done [16:41:19] paladox: no more recheck on that change of mine please, Reedy is trying to fix the merger [16:41:35] mafk i am helping him [16:41:50] I am speaking to him in a different channel [16:41:54] In that case pretend I've not said a word :) [16:42:00] Running recheck seconds after I have... isn't helping thouhg ;) [16:42:04] (and sorry) [16:42:30] Ok, Reedy sorry, i thought that the tool would have cleaned something up after you rechecked [16:42:32] but sorry [16:42:46] renaming and letting it reclone seems to have fixed it [16:42:55] it's not erroring out in the same way [16:43:00] it's doing something [16:43:10] Reedy it's testing [16:43:11] yay [16:43:24] mediawiki-extensions-hhvm failed [16:43:33] Just delete all the repo's and allow them to reclone them selfs when some one tests. [16:43:51] and it's failing now on zuul [16:43:56] I didn't delete mediawiki-extensions [16:44:16] Wait for this to finish, then I'll delete mediawiki-extensions-* [16:44:21] Ok [16:44:26] Reedy mysql may have gone down [16:44:27] 16:42:21 + /srv/deployment/integration/slave-scripts/bin/mw-teardown-mysql.sh [16:44:28] 16:42:21 ERROR 1269 (HY000) at line 1: Can't revoke all privileges for one or more of the requested users [16:44:31] according to ^^ [16:44:39] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/83002/console [16:44:41] is showing the same error [16:45:14] We need to find a way to find where the corrupt commit is comming from [16:45:19] and where it started [16:45:42] it started in gerrit [16:45:52] Yep i mean repo [16:46:23] as here https://phabricator.wikimedia.org/T151676#2824462 it says it is comming from mediawiki/extensions/Lingo [16:46:40] "Cannot check trivial rebase of new patch " [16:49:01] deleting some more folders [16:49:17] :) [16:51:42] Reedy it's doing something [16:51:47] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/83003/console [16:52:19] Its working [16:54:20] paladox: Of course, it may be worth not merging any changes to core until it's been looked at [16:54:34] Ok [16:54:35] Yeh [16:54:43] I advise against merging any thing [16:54:49] Until ops give the clear and chad [16:56:34] It seems it's working [16:56:44] at least on zuul no failures so far [16:56:49] Yep [16:56:58] it was git gc that caused all this [16:57:25] jgit gc [16:57:32] Yep that ^^ [17:03:35] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824559 (10Paladox) @Reedy had to delete the local zuul clone for ci to unbreak it's self. Clones will take longer for all mw extensions then usual. [17:15:32] 06Release-Engineering-Team, 03releng-201617-q2, 05MW-1.28-release: Release MediaWiki 1.28 - https://phabricator.wikimedia.org/T148087#2824561 (10Nemo_bis) [17:17:36] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations, 13Patch-For-Review: gerrit jgit gc caused mediawiki/core repo problems - https://phabricator.wikimedia.org/T151676#2824565 (10Reedy) [19:06:48] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations, 13Patch-For-Review: gerrit jgit gc caused mediawiki/core repo problems - https://phabricator.wikimedia.org/T151676#2824660 (10demon) a:03demon [20:52:20] 10Beta-Cluster-Infrastructure, 10Gerrit, 06Operations, 13Patch-For-Review: gerrit jgit gc caused mediawiki/core repo problems - https://phabricator.wikimedia.org/T151676#2824703 (10demon) Ran `git fsck --full --strict` on all repos. Most were fine. A lot of these are probably fine too, but here's the resul... [21:14:51] Project beta-code-update-eqiad build #131868: 04FAILURE in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131868/ [21:24:50] Project beta-code-update-eqiad build #131869: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131869/ [21:34:49] Project beta-code-update-eqiad build #131870: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131870/ [21:44:46] Project beta-code-update-eqiad build #131871: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131871/ [21:54:51] Project beta-code-update-eqiad build #131872: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131872/ [22:00:08] Reedy ^^ [22:00:08] error: Unable to append to .git/logs/refs/remotes/origin/master: Permission denied [22:00:11] ostriches ^^ [22:00:40] Wonder why that suddenly started [22:00:50] I'll look in a bit [22:01:25] ok [22:01:27] thanks [22:01:42] I got 99 problems but beta ain't 1 [22:02:19] LOL [22:04:31] What was merged? [22:04:44] Localisation updates? [22:04:50] Project beta-code-update-eqiad build #131873: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131873/ [22:05:16] Reedy not sure [22:05:45] Reedy looks like some merges in mw core [22:06:05] Yeah [22:06:58] oh [22:14:49] Project beta-code-update-eqiad build #131874: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131874/ [22:24:48] Project beta-code-update-eqiad build #131875: 04STILL FAILING in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131875/ [22:26:15] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824817 (10demon) I also want `git fsck --name-objects` [22:27:52] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824819 (10Paladox) We could just build git our selfs by creating a git repo in gerrit and then adding the debian/ folder to build a deb for Jessie... [22:34:45] Project beta-code-update-eqiad build #131876: 04STILL FAILING in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131876/ [22:39:05] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824822 (10demon) >>! In T140927#2824819, @Paladox wrote: > We could just build git our selfs by creating a git repo in gerrit and then adding the d... [22:39:59] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824823 (10Paladox) >>! In T140927#2824822, @demon wrote: >>>! In T140927#2824819, @Paladox wrote: >> We could just build git our selfs by creating... [22:44:46] Project beta-code-update-eqiad build #131877: 04STILL FAILING in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131877/ [22:54:45] Project beta-code-update-eqiad build #131878: 04STILL FAILING in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131878/ [23:04:45] Project beta-code-update-eqiad build #131879: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131879/ [23:07:52] did some chmods, lets see if that fixes it [23:08:13] thanks. [23:14:48] Yippee, build fixed! [23:14:48] Project beta-code-update-eqiad build #131880: 09FIXED in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/131880/ [23:15:20] Project beta-scap-eqiad build #130589: 04FAILURE in 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/130589/ [23:16:04] ostriches ^^ permision denied for the second test [23:16:07] first one fixed. [23:18:32] Ah, second easily fixed. [23:19:30] I thinks. [23:19:48] Ok [23:19:50] thanks [23:25:20] Project beta-scap-eqiad build #130590: 04STILL FAILING in 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/130590/ [23:31:39] ostriches ^^ [23:35:31] Project beta-scap-eqiad build #130591: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/130591/ [23:36:13] lol [23:36:26] 23:35:31 cp: cannot create regular file ‘/srv/mediawiki-staging/php-master/cache/l10n/l10n_cache-ab.cdb’: Permission denied [23:37:34] I dunno who's supposed to own it [23:37:53] I chown -R'd the whole dir to fix the first bug [23:38:00] But I guess cache/l10n/ shouldn't be that [23:38:09] I'll fix it [23:38:15] chmod -R 0777 lol [23:38:20] THAT'LL FIX IT [23:40:50] Reedy: I'm tempted to hose all those *.cdb files [23:40:56] So it just creates them fresh [23:40:59] Yeah [23:41:03] jenkins will just remake them [23:41:05] WFM [23:41:16] They were l10nupdate at some point earlier today [23:41:30] Ah, mwdeploy [23:41:35] Should own it [23:41:40] Not jenkinsbot-deploy or w/e [23:41:46] jenkins-deploy [23:42:15] Let's wait and see on this [23:45:22] Project beta-scap-eqiad build #130592: 04STILL FAILING in 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/130592/ [23:46:13] Fucks sake jenkins [23:47:41] jenkins needs to take a long walk off a short pier [23:48:18] sudo rm -f *.cdb [23:48:45] Watch permissions on the dir itself not work [23:55:19] Project beta-scap-eqiad build #130593: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/130593/