[01:56:53] Project browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce build #49: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce/49/ [04:03:13] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,contintLabsSlave && UbuntuTrusty build #13: 04FAILURE in 7 min 12 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/13/ [04:05:56] Yippee, build fixed! [04:05:57] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #13: 09FIXED in 9 min 56 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/13/ [06:32:28] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:04:55] !log rebooting deployment-tin (denying access yet again) [07:05:46] euh? [09:39:20] Yippee, build fixed! [09:39:20] Project beta-code-update-eqiad build #104578: 09FIXED in 2 min 26 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/104578/ [09:44:10] Project beta-scap-eqiad build #102702: 04FAILURE in 4 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102702/ [09:47:59] Project beta-scap-eqiad build #102703: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102703/ [09:55:31] Project beta-scap-eqiad build #102704: 04STILL FAILING in 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102704/ [10:05:29] Project beta-scap-eqiad build #102705: 04STILL FAILING in 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102705/ [10:15:34] Project beta-scap-eqiad build #102706: 04STILL FAILING in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102706/ [10:25:39] Project beta-scap-eqiad build #102707: 04STILL FAILING in 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102707/ [10:35:29] Project beta-scap-eqiad build #102708: 04STILL FAILING in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102708/ [10:45:30] Project beta-scap-eqiad build #102709: 04STILL FAILING in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102709/ [10:55:34] Project beta-scap-eqiad build #102710: 04STILL FAILING in 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102710/ [11:13:32] Yippee, build fixed! [11:13:32] Project beta-scap-eqiad build #102711: 09FIXED in 8 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102711/ [12:04:01] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests: Allow RelEng nova log access - https://phabricator.wikimedia.org/T133992#2251644 (10Joe) I think the people that need to access nova logs should be the same that are admins for contint. So I think we... [12:33:57] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [13:37:27] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2297074 (10zeljkofilipin) @Jdlrobson: sure, for which repositories and to which channel? [14:15:33] Project beta-scap-eqiad build #102730: 04FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102730/ [14:25:49] Yippee, build fixed! [14:25:50] Project beta-scap-eqiad build #102731: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102731/ [14:35:53] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests: Allow RelEng nova log access - https://phabricator.wikimedia.org/T133992#2297222 (10chasemp) >>! In T133992#2296862, @Joe wrote: > I think the people that need to access nova logs should be the same th... [14:37:20] greg-g: Any other feedback on https://phabricator.wikimedia.org/T134963 ? [14:38:01] (ie Compact Language Link Beta -> Non-beta feature) [14:39:42] kart_: I don't have any that I know of [14:40:06] OK! [14:40:16] I'll close the thing then. [14:40:44] greg-g: can you give +1 on the task, so we've written proof :) [14:42:25] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests: Allow RelEng nova log access - https://phabricator.wikimedia.org/T133992#2297270 (10thcipriani) >>! In T133992#2296862, @Joe wrote: > I think the people that need to access nova logs should be the same... [14:50:30] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2297306 (10mmodell) [14:50:31] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2297305 (10mmodell) 05Open>03declined [14:51:08] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2209132 (10mmodell) a:05mmodell>03None [14:57:03] 10MediaWiki-Codesniffer: Improving an static analysis tools for MediaWiki - Weekly reports - https://phabricator.wikimedia.org/T134225#2297332 (10Lethexie) [15:17:47] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2297432 (10mmodell) I'm not fighting for this any more. It's not worth it if nobody else sees th... [15:50:39] thanks greg-g [16:31:50] 05Gerrit-Migration, 10Diffusion, 10GitHub-Mirrors, 06Repository-Admins: Have Phabricator take over replication to Github - https://phabricator.wikimedia.org/T115624#2297791 (10Paladox) @demon I asked @mmodell to turn replication on https://phabricator.wikimedia.org/diffusion/SMTL/ and it works https://gith... [16:40:10] ostriches Hi, i tested with the metrolook repo See https://github.com/wikimedia/mediawiki-skins-Metrolook please [16:40:31] To prove it was diffusion mirroring i looked at this ref https://github.com/wikimedia/mediawiki-skins-Metrolook/commit/68fbfced9fbaa26a9f38916251dc271ec860ec47 [16:40:43] and it is an open change which mirroring that from gerrit is not supported. [16:40:56] Would we be able to try it again on mediawiki-core [16:41:18] Without doing https://git.wikimedia.org/commitdiff/mediawiki%2Fcore.git/b55feadc6694a2dfe182d75b3bbf9ce53858c3c9 please [16:41:27] twentyafterfour ^^ [16:44:52] Turned it back on for MW [16:45:17] ostriches: Thanks. [17:05:57] ostriches: Would you know how long it takes for diffusion to start mirroring. [17:06:08] Nope [17:06:08] Since https://github.com/wikimedia/mediawiki/commit/0b764e687066652e318fa218c5d0f20e8431bd69 should be shown. [17:06:39] Oh ok [17:25:54] Project beta-scap-eqiad build #102754: 04FAILURE in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102754/ [17:35:23] ostriches: Would you be able to enable mirroring on diffusion for these two repos please. [17:35:23] https://phabricator.wikimedia.org/diffusion/ECLV/ [17:35:29] https://phabricator.wikimedia.org/diffusion/EDPG/ [17:35:49] Project beta-scap-eqiad build #102755: 04STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102755/ [17:36:57] lovely [17:37:42] or twentyafterfour ^^ [17:37:43] please [17:38:13] Im testing to see weather we can enable replication on other repos from diffusion [17:42:51] oh good, can't login to tin :( [17:45:37] thcipriani: really? [17:45:44] Project beta-scap-eqiad build #102756: 04STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102756/ [17:45:57] grr... I armed keyholder not sure why that is still failing [17:45:59] twentyafterfour: I just got back in [17:46:28] Too many authentication failures for mwdeploy [17:46:30] hmmm [17:48:17] we might need to do: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh-add -D and re-add them [17:48:19] ? [17:48:46] thcipriani: doubtful [17:48:55] I'll try that [17:49:52] probably need to run puppet on every beta cluster node [17:50:11] well, now I'm getting the keyholder error: failure to sign key [17:50:55] working [17:51:01] but puppet is broken on mediawiki [17:51:07] 01 [17:51:13] :( [17:51:30] I think it'll probably be fixed now [17:51:31] Could not find class base::debdeploy for deployment-mediawiki01.deployment-prep.eqiad.wmflabs on node deployment-mediawiki01.deployment-prep.eqiad.wmflabs [17:51:45] well, beta-scap-eqiad will be fixed. [17:54:49] Project beta-scap-eqiad build #102757: 04STILL FAILING in 0.31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102757/ [17:55:19] nope [17:55:48] looks like the lock file is still in place [17:55:56] someone else was scapping? Something. [17:56:13] OSError: [Errno 1] Operation not permitted: '/var/lock/scap' [17:57:21] oh [17:57:34] it'll work now [18:05:47] Yippee, build fixed! [18:05:47] Project beta-scap-eqiad build #102758: 09FIXED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/102758/ [18:08:31] yay [18:19:10] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2298270 (10thcipriani) a:03thcipriani >>! In T132747#2294799, @Ladsgroup wrote: > I had to redefine the ssh key because keyholder couldn't connect to the sca-01 wit... [18:47:07] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298408 (10thcipriani) [18:55:14] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298455 (10thcipriani) As of today we have 13 patches cherry picked to beta of various ages by various authors: ``` thcipriani@deployment-puppetmaster:/var/lib/git/opera... [18:57:27] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298466 (10mmodell) One simple solution which seems fairly straightforward would be to switch to using a branch for beta. Admittedly that may only achieve some of the goa... [19:15:21] RECOVERY - Puppet run on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:16] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298536 (10bd808) ``` $ git show --pretty 5ddacd7 commit 5ddacd7deaca60ce51290454c3344ffb36e70c40 Author: Bryan Davis Date: Wed Oct 7 15:45:48 201... [19:38:33] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298733 (10thcipriani) >>! In T135427#2298536, @bd808 wrote: > This was needed on deployment-bastion when @Krenair was trying to get l10nupdate setup there and the /var p... [19:49:49] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298827 (10Krenair) `* 4849f99 2015-10-20 (7 months ago) root` adds self-signed certificate, which was needed to keep https working (ish) in beta until {T50501} is done I... [19:53:22] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298892 (10Krenair) `* b0cd9dd 2016-02-11 (3 months ago) Timo Tijhof` - this got merged in https://gerrit.wikimedia.org/r/#/c/270009/ and is now just adding a single line... [19:55:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests: Allow RelEng nova log access - https://phabricator.wikimedia.org/T133992#2298923 (10Dzahn) The contint-admins group already has this: 159 'ALL = NOPASSWD: /bin/journalctl*', But tha... [19:57:51] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2298925 (10Krenair) It now looks like this: ```root@deployment-puppetmaster:/var/lib/git/operations/puppet# git log --pretty=format:"%h %ad (%ar) %an - %s" --graph --date... [20:13:56] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests: Allow RelEng nova log access - https://phabricator.wikimedia.org/T133992#2298968 (10Dzahn) [labnodepool1001:~] $ id thcipriani uid=11634(thcipriani) gid=500(wikidev) groups=500(wikidev),719(contint-adm... [20:24:13] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2298975 (10Jdlrobson) >>! In T134492#2297074, @zeljkofilipin wrote: > @Jdlrobson: sure, for which repositories and to which channel? #wikipedia-mobile would be gr... [20:31:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests: Allow RelEng nova log access - https://phabricator.wikimedia.org/T133992#2298978 (10thcipriani) When we were troubleshooting nodepool problems, we could tail logs from `labnodepool1001`. The permission... [20:55:37] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427#2299039 (10thcipriani) >>! In T135427#2298943, @Krenair wrote: > We should allow patches to be put on the puppetmaster to test them temporarily (say... for each version u...