[00:00:15] No [00:00:29] It's trying to (think next sequence ids in postgres) [00:00:37] But then (rightly) fails when it sees a name conflict [00:00:48] This is what we saw a few months ago, for which we had a ton of fixes [00:01:24] I wonder if it regressed in 3.13.4 -> 3.13.8 [00:02:20] would it make sense to try both upper and lower case, just in case "localUsernameToLowerCase = true" stopped working? [00:02:43] Yeah also worth checking [00:03:30] tried logging in with a lowercase name, same error [00:07:43] Our account_external_ids entries are identical, accounts index was refreshed too. Hmm. [00:08:24] https://phabricator.wikimedia.org/P5553 [00:08:57] last log line was me trying with a wrong password, as a sanity check [00:11:38] Weird, it's not letting me login at all with my normal password now. [00:12:32] well, that's progress of a sort [00:12:54] As in: we've managed a state change? :p [00:13:31] at least we're not staring at that paste trying to spot a nonexistent difference [00:13:36] [2017-06-07 00:12:36,195] [HTTP-99] INFO com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'Chad' failed to sign in: Cannot assign external ID "gerrit:chad" to account 4938; external ID already in use. [00:13:46] Yeah [00:15:09] I can roll back to the prior release. [00:15:15] (just upgraded a few hours ago) [00:16:46] can still login just fine, also lowercase and thenback to uppercase [00:20:05] TimStarling: Try now [00:20:30] works [00:20:42] Ok, so there was a regression somewhere, rolling back worked. [00:20:58] thanks [00:21:03] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:21:10] btw it was 2.13.4 not 3.13.4 [00:21:11] <3 gerrit [00:21:58] TimStarling: Thanks for letting me know. Hadn't shown up in testing or for anyone else yet. Gonna have to run this down again. [00:22:15] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:22:15] * RainbowSprinkles sighs, reopens task [00:23:47] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3321830 (10demon) 05Resolved>03Open Ugh. This came back with 2.13.8. @tstarling was unable to log in. After logging out,... [00:25:14] RainbowSprinkles i think i know why [00:25:24] remeber luca had a change that was not merged [00:25:30] but we cherry picked it [00:25:49] it is still not merged in stable-2.13 but we didnt re cherry pick it. [00:26:10] Oh really? I could've sworn it had [00:26:14] Yep [00:26:26] Because therwise we would have a git hash [00:26:35] 10Gerrit, 10Release-Engineering-Team (Kanban): Update gerrit to 2.13.8 - https://phabricator.wikimedia.org/T158946#3321834 (10demon) 05Resolved>03Open Rolled back because T152640. [00:27:37] RainbowSprinkles https://gerrit-review.googlesource.com/#/c/92830/ [00:28:45] Wait, so that never made it to master either? [00:29:23] I doint know if the problem still exists on master. [00:29:39] though maybe the change should be merged upstream now. [00:30:44] Oh yeah, they fixed it another way in master. That's why I was confused. [00:30:52] Could've sworn that had made it into 2.13.x [00:31:09] They have it all stored in the repo now. [00:31:58] You know, we never had these problems back in 2.8.x when they used a database for everything ;-) [00:32:11] :) [00:32:20] They deprecated the db. [00:32:34] I found it hard to purswade them to support mariadb. [00:33:14] * RainbowSprinkles shrugs [00:33:42] But i wonder if they actually fixed our problem in 2.14. [00:34:21] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:41:13] Ok, I went ahead and prepped a new 2.13.8 release (2.13.8-1-g7c438d37a2 to be specific) for the debian package. I won't be around tomorrow to deploy it, but can do thursday when I'm back. In the meantime, we're safely running back on the 2.13.4 build from before [00:41:33] thanks :) [00:41:40] merges the deb change and nods :) [00:41:55] I am thinking i will restore luca [00:42:07] luca's change if that turns out that it will fix it. [00:42:10] mutante thanks :) [00:42:24] the restore button shows for me :) [01:01:04] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:18] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [01:09:21] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:32:28] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:12:27] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:24:57] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [02:27:51] 10Continuous-Integration-Infrastructure, 10Operations: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3321960 (10greg) >>! In T166888#3321592, @faidon wrote: > So it doesn't really like sound like the primary use case is jobs like the operations/puppet linting, at least... [03:05:00] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [03:41:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#2956401 (10Dzahn) yep, checked this: jenkins docs say: "Both the nocanon option to ProxyPass, and AllowEncodedSlas... [04:04:41] I'm exploding all extensions [04:04:45] sorry :/ [04:05:44] Amir1: Need a hand? [04:05:58] Reedy: for +2ing, yeah :D [04:06:02] ? [04:06:21] https://gerrit.wikimedia.org/r/#/q/owner:%22Ladsgroup+%253CLadsgroup%2540gmail.com%253E%22 [04:06:36] I'd love to get push access to do it all in on go [04:06:37] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #415: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/415/ [04:07:44] Dunno what the rules on just pushing are [04:23:40] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia projects - https://phabricator.wikimedia.org/T165540#3322022 (10Ladsgroup) Sorry for lots jenkins things here. I used this since I don't have push rights: ``` USER=Ladsgroup OKAY="No" f... [04:27:12] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia projects - https://phabricator.wikimedia.org/T165540#3322027 (10MZMcBride) Adding this file to every Git repository seems like a completely ridiculous idea. [06:31:07] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #384: 04FAILURE in 1 hr 51 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/384/ [06:55:58] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:30:58] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:44] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540#3322293 (10Nemo_bis) [07:42:21] hashar: hey, to do this: https://phabricator.wikimedia.org/T165540#3322022 Can I have push access for two hours on all mediawiki extensions? [07:42:31] https://gerrit.wikimedia.org/r/#/q/owner:%22Ladsgroup+%253CLadsgroup%2540gmail.com%253E%22 [07:42:50] This will be a huge mess if going through gerrit (500-ish extensions) [07:46:39] Amir1: yeah we definitely wanna push it [07:46:56] then really adding a dummy file like https://gerrit.wikimedia.org/r/#/c/357548/1/CODE_OF_CONDUCT.md [07:47:05] is really lame when it can be done directly in the README.md file [07:48:25] Amir1: and make sure to skip mediawiki/extensions/Wikidata :-} [07:48:51] Sure :) [07:49:01] tell me when I have the rights so I start [07:50:22] !lo gerrit: granted push right to Amir "Ladsgroup" to push the CODE_OF_CONDUCT.md file to all extensions and skins - T165540 [07:50:22] T165540: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540 [07:52:41] hashar: [07:52:44] https://www.irccloud.com/pastebin/UPffV2Fz/ [08:01:42] Amir1: grmblblb [08:02:03] :D [08:02:12] let me clean that mess [08:02:42] Amir1: try again ? [08:03:00] I have added you to a different Gerrit group which should give you push access [08:03:03] on it [08:03:29] hashar: yup, was successful [08:04:15] hashar: now, it's time to +2 gerrit ones [08:04:16] :D [08:04:42] Amir1: dont please [08:04:52] Amir1: if the commits are in Gerrit, you can just push them [08:05:05] hmm [08:05:08] okay [08:05:10] when you push the commit, Gerrit will notice a change exist and automatically close it [08:05:26] then in the Gerrit web interface one would see something like: "Amir successfully pushed to master" [08:05:32] or something like that [08:05:41] for repositories that do not have a change in Gerrit, just push [08:13:02] 10Deployment-Systems, 10Scap, 10Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3322358 (10fgiunchedi) 05Open>03Resolved [08:13:16] okay [08:13:41] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540#3269385 (10Nemo_bis) I think the GitHub guide is clearly aimed at smaller projects which don't know better and have few reposito... [08:13:57] change id is different in every run because it's random (see the bash script) so I'm not sure if it takes it as the same [08:14:02] hashar: do you know? [08:17:39] OH [08:17:47] ideally the Change-Id would have been the same for all commits [08:18:16] with something like: git add CODE_OF_CONDUCT.md && git commit -m 'Adding code of conduct' -m 'Change-Id: ' [08:19:00] alternatively, you can just download them and push locally [08:19:39] there is only a dozen of them [08:23:53] hashar: yeah, okay [08:24:13] 10Release-Engineering-Team, 10Language-Team, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3322387 (10Nemo_bis) [08:24:13] that was suggested by Gergo, I didn't know it's better to stay the same [08:24:43] Amir1: or you can abandon the one currently pending [08:24:50] and regenerate commits with the same Change-Id [08:24:54] but really it does not matter that much [08:25:25] hashar: okay, that'd be probably the best option [08:25:45] btw. It's still working and mediawiki/extensions/CategoryTagSorter failed for 403, that's the only one I saw [08:28:48] !log upgrading kibana to v5.4.1 on deployment-logstash2 [08:28:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:35:48] !log rolling back to kibana 5.3.2, incompatible elasticsearch version [08:35:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:43:32] !log upgrading kibana to v5.3.3 on deployment-logstash2 [08:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:55:46] Amir1: for CategoryTagSorter maybe try again or eventually send a patch for review in Gerrit and +2 it ? [08:56:24] Some intermittently fail, CategoryTagSorter is one of them [08:56:33] gehel: good morning. Erik has installed a Kibana for me that points to relforge1001.eqiad.wmnet . Currently Kibana 5.1.2 so I guess I should upgrade it to 5.3.2 as well ? [08:58:18] I'm actually upgrading to 5.3.3 atm [08:59:10] Relforge is a bit of a special case. Let me get back to you after coffee... [09:00:07] !log adding deployment-zookeeper02.eqiad.wmflabs to Hiera:deployment-prep [09:00:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:03:32] gehel: it is just a quick experiment to figure out how Jenkins insert data in elasticsearch. Nothing to worry about :} [09:08:47] relforge is still running elasticsearch 5.1.2, which is most probably not compatible with kibana 5.3.x [09:09:04] (I have not actually checked, but that's the usual strategy) [09:09:42] we were waiting for upgrades to some of he plugins we are testing to upgrade relforge [09:11:27] hashar: so you should stay on kibana 5.1.x at the moment. I'll ping you when I upgrade relforge to 5.3 next week (and you will probably see your kibana breaking at that point) [09:12:02] gehel: sounds good. And I guess when I see it broken I will just have to apt-get upgrade kibana right? [09:12:26] yep, that should be the case [09:32:56] I need to relocate [09:33:10] it's there until "I" [09:33:45] !log restart kafka brokers to pick up the new zookeeper settings [09:33:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:35:41] ok all good, next round is later this afternoon to remove the old trusty host [09:35:47] (zookeeper01) [09:50:38] is really lame when it can be done directly in the README.md file Amir1: let me know when you are done so I can clean up the permission in Gerrit [09:52:04] p858snake: yeah then people will eventually refactor it :-} [09:56:37] It's in "L" now [09:57:18] :-} [10:08:13] 10Release-Engineering-Team (Watching / External), 10Operations, 10Goal, 10Kubernetes, and 3 others: Prepare and maintain base container images - https://phabricator.wikimedia.org/T162042#3322681 (10Joe) [10:22:27] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:29:00] (03PS1) 10Jonas Kress (WMDE): Add selenium tests to gate and submit [integration/config] - 10https://gerrit.wikimedia.org/r/357582 [10:59:32] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:31:22] 10Continuous-Integration-Infrastructure, 10Operations: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3322963 (10faidon) Great, thanks :) I'm looking at the output of a Jenkins job and it looks like it takes about a minute to execute, so I guess we have two semi-related... [12:02:25] 10Beta-Cluster-Infrastructure, 10Cognate, 10MediaWiki-extensions-InterwikiSorting, 10Patch-For-Review, 10User-Addshore: Create beta hewiktionary for testing InterwikiSorting & Cognate - https://phabricator.wikimedia.org/T158628#3323110 (10Tobi_WMDE_SW) [12:03:44] 10Continuous-Integration-Config, 10Revision-Slider, 10TCB-Team, 10MW-1.29-release (WMF-deploy-2017-01-03_(1.29.0-wmf.7)), and 2 others: Apparent random failing of RevisionSlider qunit tests - https://phabricator.wikimedia.org/T153121#3323151 (10Tobi_WMDE_SW) [12:03:48] 10Release-Engineering-Team (Kanban), 10Scap (Scap3-MediaWiki-MVP), 10Electron-PDFs: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#3323154 (10Tobi_WMDE_SW) [12:04:16] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10Operations, 10TCB-Team, 10Wikidata: Add Andrew and Aleksey to ldap/wmde group - https://phabricator.wikimedia.org/T152088#3323163 (10Tobi_WMDE_SW) [13:00:25] releng people, I'm done with the changes [13:01:26] These extensions didn't let me push into them: "VisualEditor" "DonationInterface" "Memento" "CategoryTagSorter" [13:04:03] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540#3323428 (10Ladsgroup) I pushed in all active extensions and skins. These ones didn't let me push them and probably we should do... [13:04:48] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:15:41] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3323440 (10Paladox) This is a blocker for gerrit 2.14 as well. This is apparently also happening for gerrit 2.14 per luca co... [13:37:43] 10Continuous-Integration-Infrastructure, 10Operations: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3323538 (10MoritzMuehlenhoff) [13:37:45] 10Continuous-Integration-Infrastructure, 10Operations: Collate jessie-wikimedia/backports into jessie-wikimedia/main - https://phabricator.wikimedia.org/T167292#3323526 (10MoritzMuehlenhoff) [13:39:07] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3323543 (10Paladox) new bug filled upstream https://bugs.chromium.org/p/gerrit/issues/detail?id=6443 [13:39:32] 10Continuous-Integration-Infrastructure, 10Operations: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3310890 (10MoritzMuehlenhoff) [13:51:32] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540#3323623 (10MZMcBride) >>! In T165540#3272932, @greg wrote: > I'd propose that we should JFDI across all repos in Gerrit. How is... [13:57:13] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3323670 (10Paladox) Actually this doesn't block 2.14. In 2.14 you can reindex which fixes it for them. Our problem is differ... [13:59:59] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540#3323678 (10MZMcBride) @Tgr: I'm struggling to see how you filing this task, reviewing wi... [14:03:29] PROBLEM - Free space - all mounts on deployment-phab01 is CRITICAL: CRITICAL: deployment-prep.deployment-phab01.diskspace.root.byte_percentfree (<100.00%) [14:21:57] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:33:26] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:33:43] Yippee, build fixed! [14:33:44] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #416: 09FIXED in 1 min 41 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/416/ [14:41:51] so, how do i debug https://phabricator.wikimedia.org/T167216 in beta? can i SSH into something and put some printf() statements somewhere? (or do i need to commit all by debugging to Gerrit?) [14:55:55] 10Release-Engineering-Team (Kanban), 10MediaWiki-Authentication-and-authorization, 10Reading-Infrastructure-Team-Backlog, 10Epic, and 4 others: Release AuthManager with MediaWiki 1.27 - https://phabricator.wikimedia.org/T135498#3324018 (10Fjalapeno) [14:56:02] 10Release-Engineering-Team (Kanban), 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-Campaigns, 10MediaWiki-extensions-General, and 3 others: Update Campaigns to use AuthManager - https://phabricator.wikimedia.org/T135043#3324021 (10Fjalapeno) [14:56:15] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10Reading-Infrastructure-Team-Backlog, 10Wikimedia-log-errors: Could not find local user data for Riley Huntley@jawiki - https://phabricator.wikimedia.org/T129214#3324027 (10Fjalapeno) [15:01:58] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [15:05:11] MatmaRex: i've seen something like that before. [15:05:38] MatmaRex: it happens if someone accidently overrides the array, instead of appending to it [15:05:54] hmm. [15:11:04] thedj: seems like all skins that use ResourceModuleSkinStyles do it via skin.json, which hopefully doesn't clobber the values… [15:11:44] hmmmmmmmmmmm. [15:13:28] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:13:29] but extensions can also add stuff. [15:13:35] yes [15:13:48] thedj: i think i've found it. you're the best [15:14:09] awesomeness [15:17:40] thedj: it's freaking mobilefrontend. https://phabricator.wikimedia.org/T167216#3324218 [15:17:41] thank you [15:23:39] MatmaRex: If a module defines a skinStyles['default'] the skin may want to extend that instead of replacing them. This can be done using the + prefix. [15:23:44] $wgResourceModuleSkinStyles['foo'] = array( '+bar' => 'skins/Foo/bar.css', [15:23:48] ); [15:24:17] https://www.mediawiki.org/wiki/Manual:$wgResourceModuleSkinStyles#Documentation [15:24:45] but that those modules are being touched for theses skins is indeed suspect to begin with. [15:25:14] yeah. i'm figuring out what this is meant to do, and i'll fix it [15:25:36] in general extensions should never be using ResourceModuleSkinStyles, but MF is weird because it's also a skin [15:26:44] yeah, it basically means: "This extensions doesn't support my skin, so i'm patching it here instead". [15:27:03] this module [15:45:20] 10Beta-Cluster-Infrastructure, 10TemplateStyles, 10Wikimedia-Extension-setup, 10Patch-For-Review: Deploy TemplateStyles to the beta-cluster - https://phabricator.wikimedia.org/T133414#3324394 (10Jdforrester-WMF) 05stalled>03Resolved a:03Anomie [15:46:18] Yippee, build fixed! [15:46:18] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #447: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/447/ [15:55:59] Yippee, build fixed! [15:56:00] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #447: 09FIXED in 33 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/447/ [16:01:33] Yippee, build fixed! [16:01:33] Project selenium-CentralNotice » chrome,beta,Linux,BrowserTests build #419: 09FIXED in 31 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/419/ [16:01:42] Yippee, build fixed! [16:01:42] Project selenium-CentralNotice » firefox,beta,Linux,BrowserTests build #419: 09FIXED in 41 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/419/ [16:01:58] Yippee, build fixed! [16:01:58] Project selenium-CentralNotice » chrome,beta,OS X 10.9,BrowserTests build #419: 09FIXED in 57 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/419/ [16:28:51] 10Gerrit: mirror search/MjoLniR repository to github - https://phabricator.wikimedia.org/T167315#3324566 (10EBernhardson) [16:35:13] Project beta-scap-eqiad build #158722: 04FAILURE in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/158722/ [16:45:06] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3324672 (10Jdlrobson) [16:45:38] Yippee, build fixed! [16:45:39] Project beta-scap-eqiad build #158723: 09FIXED in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/158723/ [17:02:44] how often the deployment-prep puppet master syncs with the prod one? [17:05:45] PROBLEM - Puppet errors on swift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:19] PROBLEM - Puppet errors on swift-storage-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:49] elukey: every 10 minutes I think [17:08:09] that's the default in role::puppetmaster::standalone anyway [17:08:38] i know puppet runs every 30 mins is that not the case for deployment-prep? [17:22:34] weird still not in sync with my latest change merged [17:24:30] ah there is a /var/log/git-sync-upstream.log [17:24:42] Rebase failed! See error messages above. [17:24:46] ah! there you go [17:25:42] so it is messed up by the cherry picks [17:28:44] might be due to the change for jobrunners deployed with scap3 [17:29:41] should be fixed now [17:30:21] !log manually fixed rebase issue for operations/puppet on puppetmaster02 (empty commit due to the change for scap3 and jobrunners) [17:30:24] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:30:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:31:53] yeppa [17:33:10] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:33:30] 10Deployment-Systems, 10Release-Engineering-Team (Kanban): Bot to handle recurring tasks to manage wikitech:Deployments - https://phabricator.wikimedia.org/T114488#3324981 (10mmodell) p:05Normal>03High [17:42:00] elukey: ah, crap. Yeah, the jobrunners thing merged and I forgot to remove the cherry-pick :( [17:42:29] thcipriani: not a problem! Now I know where to check for issues :) [17:42:49] thanks :) [17:45:23] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:49:32] !log forced /usr/local/bin/git-sync-upstream manually on puppetmaster02 [17:49:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:54:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3325070 (10mmodell) [18:13:08] RECOVERY - Puppet errors on integration-slave-docker-1000 is OK: OK: Less than 1.00% above the threshold [0.0] [18:27:35] 10Release-Engineering-Team (Kanban), 10Developer-Relations, 10Team-Practices: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#3325231 (10Tgr) Apparently there is a `#wikimedia-codereview` and a `#mediawiki-codereview` channel, and the latter is invite-only, which confuses people.... [18:34:48] 10Beta-Cluster-Infrastructure: Getting error HTTP 500 while loading VE in beta cluster - https://phabricator.wikimedia.org/T167341#3325243 (10Ryasmeen) [18:39:43] RainbowSprinkles hi, can you try logging into https://gerrit.git.wmflabs.org/r/#/q/status:open to see if the problem happens there for you please? [18:40:05] (it's running gerrit 2.13.8) [19:03:49] 10Deployment-Systems, 10Release-Engineering-Team (Kanban): Bot to handle recurring tasks to manage wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488#3326890 (10mmodell) [19:18:33] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3328061 (10Krinkle) [19:31:57] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:32:10] Diffusion is being redesnged, if anyone has feedback leave it on https://secure.phabricator.com/T12804 :) [19:32:15] twentyafterfour ^^ :) [19:43:34] 10Beta-Cluster-Infrastructure, 10VisualEditor: Getting error HTTP 500 while loading VE in beta cluster - https://phabricator.wikimedia.org/T167341#3330028 (10Deskana) [19:46:40] 10Beta-Cluster-Infrastructure, 10VisualEditor: Getting error HTTP 500 while loading VE in beta cluster - https://phabricator.wikimedia.org/T167341#3330068 (10Deskana) p:05Triage>03Unbreak! VisualEditor being this badly broken on our test infrastructure warrants urgent investigation. [20:13:56] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Fix Blubber variant expansion for boolean/int config properties - https://phabricator.wikimedia.org/T166353#3330214 (10dduvall) [20:33:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3330313 (10mmodell) [20:34:26] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:39:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3330332 (10mmodell) [20:42:02] Yippee, build fixed! [20:42:02] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #418: 09FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/418/ [20:43:50] TimStarling hi, i am wondering could i ask you to try to log into https://gerrit.git.wmflabs.org/r/ please the way you try yesturday please. I am trying to investigate why T152640 is still broken [20:43:50] T152640: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640 [20:44:24] It is running 2.13.8. [20:53:27] Project beta-scap-eqiad build #158750: 04FAILURE in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/158750/ [20:58:07] Yippee, build fixed! [20:58:08] Project beta-scap-eqiad build #158751: 09FIXED in 3 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/158751/ [21:14:28] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:17:59] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3308953 (10mmodell) All unblocked, will resume the train shortly... [21:35:17] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3330523 (10mmodell) @bblack reported in IRC: > yeah pending from icinga too (but not 3/3 to alert yet): > PYBAL CRITICAL - api-https_443 - Could not de... [21:37:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: Popups aren't working in 1.30.0-wmf.4 - https://phabricator.wikimedia.org/T167358#3330531 (10mmodell) [21:37:38] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3330543 (10mmodell) [21:38:52] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3308953 (10mmodell) [21:45:17] 10Beta-Cluster-Infrastructure, 10TemplateStyles, 10Wikimedia-Extension-setup, 10Patch-For-Review: Deploy TemplateStyles to the beta-cluster - https://phabricator.wikimedia.org/T133414#2231195 (10Tgr) Test page: https://en.wikipedia.beta.wmflabs.org/wiki/TemplateStylesTest (not working ATM due to T167349) [21:46:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3330599 (10mmodell) [23:11:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T166829#3330730 (10Jdlrobson) [23:46:38] (03PS1) 10Thcipriani: Docker for operations-puppet-tests [integration/config] - 10https://gerrit.wikimedia.org/r/357741 (https://phabricator.wikimedia.org/T166888)