[00:01:41] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [00:22:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:02:42] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [04:17:21] Yippee, build fixed! [04:17:22] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #471: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/471/ [06:03:23] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:13:37] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:43:24] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:53:38] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:58:38] (03PS1) 10Bekicot: Fix Rubocop Offenses Fix rubocop offenses generated by rubocop. Ommit 1 offense on `mediawiki_api.gemspec:14` beacause it would reduce the readability [ruby/api] - 10https://gerrit.wikimedia.org/r/368608 [07:06:06] (03PS1) 10Bekicot: Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 [07:08:41] (03Abandoned) 10Bekicot: Fix Rubocop Offenses Fix rubocop offenses generated by rubocop. Ommit 1 offense on `mediawiki_api.gemspec:14` beacause it would reduce the readability [ruby/api] - 10https://gerrit.wikimedia.org/r/368608 (owner: 10Bekicot) [09:13:24] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Reading-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3477969 (10hashar) Add a parameter to the job: `DISPLAY=':94'` [09:31:26] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3485329 (10hashar) [09:32:06] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3485332 (10hashar) [09:34:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3485342 (10hashar) [09:34:36] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3145359 (10hashar) I have created sub task T172094 to attach Gerrit changes to it. That is to avoid spamming this task. [09:35:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3485369 (10hashar) [09:49:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3485453 (10hashar) [10:07:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3485501 (10hashar) [10:15:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:17:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3485532 (10hashar) [10:21:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3485549 (10hashar) [10:52:52] (03CR) 10Zfilipin: "recheck" [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [10:55:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:59:07] (03CR) 10Zfilipin: "Why is this needed?" (032 comments) [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [11:15:59] (03CR) 10jerkins-bot: [V: 04-1] Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [11:28:47] 10Browser-Tests-Infrastructure, 10MinervaNeue, 10Reading-Web-Backlog: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3485743 (10Jhernandez) [11:55:23] (03CR) 10Hashar: "You seem to be using a local rubocop version which does not match the version we have defined in the /Gemfile." [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [11:59:06] (03CR) 10Bekicot: "> You seem to be using a local rubocop version which does not match" [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [12:40:15] hashar: hi phabricator notifications spammer :D [12:40:32] Sagan: :D [12:40:34] Notifications before viewing T172094: 196. After: 14 [12:40:35] T172094: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094 [12:40:45] sorry it is a bit messy to mass update extensions [12:41:11] not a real problem, I was just a bit shocked as I opened my notifications :D [12:41:15] Sagan: https://integration.wikimedia.org/zuul/ is quite busy as well [12:41:51] wow [12:42:31] 8 instances building, 8 at deletion [12:44:40] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:50:08] (03PS2) 10Bekicot: [WIP] Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 [13:17:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3486079 (10hashar) [13:19:42] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [13:24:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3486096 (10hashar) [13:25:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3485762 (10hashar) [13:49:30] hmm i wonder is apple managing to get login details by caching phabricator.wikimedia.org spotted by someone else [13:49:35] https://lookup-api.apple.com/phabricator.wikimedia.org/ [13:53:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3486212 (10hashar) [13:53:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: npm 3 requires peer dependencies to be explicitly defined in package.json - https://phabricator.wikimedia.org/T172094#3486209 (10hashar) 05Open... [14:23:51] (03CR) 10Zfilipin: "recheck" [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [15:42:46] (03CR) 10Jforrester: [C: 031] "Shall we get this deployed?" [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) (owner: 10Hashar) [15:44:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3486641 (10hashar) I think I got rid of the peer dependencies found in the repositories master branches. Other branches would... [15:45:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3486659 (10hashar) @Jdforrester-WMF I guess I will do the npm upgrade tomorrow during european morning then copy paste your ann... [15:49:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3486678 (10Jdforrester-WMF) Sounds good. [16:04:02] 10Deployment-Systems, 10Release-Engineering-Team (Next), 10Scap (Scap3-Adoption-Phase1), 10MediaWiki-JobRunner, 10Operations: Figure out how to disable starting of jobrunner/jobchron in the non-active DC - https://phabricator.wikimedia.org/T167104#3486758 (10thcipriani) I made a couple of patches that at... [16:15:15] RainbowSprinkles hi, upstream work fast they already have the patch for removing gwtui from gerrit https://gerrit-review.googlesource.com/#/c/gerrit/+/116790/ [16:15:27] i can see alot of merge conflicts in the future with that change [16:16:33] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:29:31] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Upstream, 10WorkType-NewFunctionality: JJB should support YAML axis - https://phabricator.wikimedia.org/T128462#3486849 (10hashar) #upstream patch has been merged :-} So I guess I just have to update our fork. [16:32:11] (03PS3) 10Bekicot: Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 [16:33:26] (03CR) 10Hashar: "recheck" [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [16:34:27] (03CR) 10John Vandenberg: [C: 031] Fix Rubocop Offenses [ruby/api] - 10https://gerrit.wikimedia.org/r/368609 (owner: 10Bekicot) [16:56:31] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:04] 10Gerrit, 10Developer-Wishlist (2017): Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131#3486908 (10Paladox) This will have to be written for polygerrit's ui since gwtui has begun to be removed upstream. We can still make it for gwtui but it will need to support polygerrit ui... [17:01:22] 10Gerrit: Provide description of linked Phabricator objects in tooltips - https://phabricator.wikimedia.org/T64053#3486909 (10Paladox) This will have to be written for polygerrit's ui since gwtui has begun to be removed upstream. We can still make it for gwtui but it will need to support polygerrit ui too. [17:02:46] 10Gerrit: Add an icon to a patchset in a changeset view when the patchset has attached comments - https://phabricator.wikimedia.org/T52600#574644 (10Paladox) This feedback will need to be re reviewed for polygerrit. gwtui has started to be removed upstream thus unlikely they will accept any new ui changes. Poly... [17:10:11] 10Gerrit, 10Developer-Wishlist (2017), 10Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#3486929 (10Paladox) This https://www.mediawiki.org/wiki/Gerrit/Tagging sounds like the hashtag feature in gerrit. That will require us to migrate to notedb. Which will probably not happen... [17:11:22] 10Gerrit: Gerrit code review view jumps/scrolls up and down when commenting - https://phabricator.wikimedia.org/T159919#3486933 (10Paladox) gwtui is being removed upstream now https://gerrit-review.googlesource.com/#/c/gerrit/+/116790/ [18:01:51] 10Deployment-Systems, 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase1), 10scap2, and 3 others: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) - https://phabricator.wikimedia.org/T129148#3487115 (10mmodell) Do we have a mechanism to automatically select the right environment?... [18:04:59] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487123 (10mmodell) @dzahn: anything I can do to help get this one moving? I tried to log in to phab1001 so that I could verify that puppet has... [18:18:37] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487147 (10Dzahn) @mmodell It doesn't have the puppet role for phab on it because we had to remove it. The role just isn't ready for being used... [18:20:26] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487154 (10mmodell) @dzahn: Ok, I can fix that. Thanks! I think there is a lot of room for improvement in the way we handle IP addresses. [18:28:39] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487196 (10mmodell) So what should we do instead of having host-specific IPs in `hieradata/role/[datacenter]/phabricator_server.yaml`? Should... [18:29:37] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3487198 (10Jrbranaa) @zeljkofilipin sounds good. @Rfarrand I don't suspect that you'll need much from me on this one :-) I've got another series of TTs... [18:42:02] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487219 (10Paladox) @mmodell maybe a host specific hiera level [18:42:14] 10Release-Engineering-Team (Backlog), 10Operations, 10Phabricator: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129#3487220 (10mmodell) The latest change of plans is to set up `phab1001.eqiad.wmnet` before `phab2001.codfw.wmnet` as we can probably switch dir... [18:44:06] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487228 (10mmodell) [18:44:08] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Traffic: Verify that the codfw lvs is configured correctly for Phabricator - https://phabricator.wikimedia.org/T168699#3487227 (10mmodell) [18:46:40] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#3487236 (10mmodell) [18:46:46] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3487232 (10mmodell) 05Open>03stalled Currently blocked on #traffic. Attempting to make some headw... [18:51:02] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487265 (10Dzahn) role/[datacenter]/ seems actually correct and better than host names. [18:53:35] twentyafterfour: yea, i think we should use hieradata/hosts/iridium.yaml and hieradata/hosts/phab1001.yaml [18:53:42] until iridium is gone and replaced [18:53:51] then we can just use role/[datacenter]/ again [18:54:23] or we could use role/[datacenter] AND hosts/iridium.yaml to override that again for just iridium [18:54:38] so later we'd just have to remove that iridium.yaml again [18:56:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3487299 (10mmodell) [18:59:54] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487325 (10mmodell) @dzahn: Do we have an IP assigned for `git-ssh` on phab1001? [19:13:07] mutante: either way works for me [19:15:25] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:15:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:20:14] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487412 (10Dzahn) @mmodell Here's the thing. There is the git-ssh IP for eqiad 208.80.154.250 and git-ssh for codfw 208.80.153.250. This IP... [19:21:42] twentyafterfour: ^ it's probably easiest if we can just switch iridium to phab1001 in one step.. where we also move the existing secondary IPs [19:21:53] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: replace sdb and then setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3487427 (10mmodell) Yeah I think scheduled downtime to switch the IP is reasonable. I'll make a patch and we can do it this week if you're up f... [19:21:56] or we have to add new IPs but come up with some "random" name [19:22:00] mutante: agreed [19:22:10] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #38: 04FAILURE in 33 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/38/ [19:22:10] ok [19:22:10] lets do it with a scheduled maintenance [19:22:19] sounds good, ok [19:22:34] either the existing wednesday night timeslot or we can make another one if you have a preferred time to do that [19:23:44] dzahn: I'll submit a patch that will allow us to run puppet on phab1001 without the IP change, so that we can get everything else ready and then just switch the IP with one small patch [19:24:25] twentyafterfour: ok, both sound good to me. existing timeslot and that kind of puppet change would be great [19:25:27] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3487438 (10mmodell) [19:25:38] :) [19:25:42] mutante: thank you, I'll add you as reviewer on the patches [19:25:54] sure, yes, add me [19:28:06] twentyafterfour: just thought "i wonder if we need to ask DBA for updated DB grants" [19:28:41] mutante: good question. Probably! [19:28:42] we should test manually with mysql console if it can talk to db [19:28:49] yeah [19:28:59] mutante: I can't log in to phab1001 so I can't test anything yet [19:29:19] twentyafterfour: actually... let me just fix that part [19:29:27] i can add the admin group on the host directly for now [19:29:38] since it's tied to the role [19:30:01] ok [19:30:16] twentyafterfour doint we need to rsync /srv/repos to phab1001? [19:30:19] paladox: about rsyncing files... [19:30:30] we should use this: [19:30:44] rsync::quickdatacopy {} [19:30:54] it's easy with that puppet abstraction now [19:30:59] yep [19:31:04] you just specifcy source and dest host and path [19:31:13] and you can set automatic sync to true or false [19:31:15] like we did for gerrit [19:31:20] so with or without a cron that constantly syncs it [19:31:36] also, see how i did for releases: [19:31:52] https://gerrit.wikimedia.org/r/#/c/368333/ [19:31:59] here you have one active server, set in Hiera [19:32:15] ah [19:32:25] and then you have rsync based on $active_server and $passive_server [19:32:35] so you just change Hiera.. and boom.. rsync direction changes [19:32:39] incl. ferm rules and stuff [19:32:43] nice [19:33:39] I still want to eventually get phab's git clustering set up since that takes care of active-active replication [19:34:14] when you request a git repo that's out of date, phab just proxies the connection to the server with the newer copy of that repo [19:35:06] it also allows git repos to be spread across several servers with each server hosting only a subset of all repos [19:50:25] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:50:39] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [21:06:47] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129#3487635 (10mmodell) a:03mmodell [21:06:50] (03CR) 10Hashar: [C: 032] "Yes definitely. Thank you everyone for the reviews :]" [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) (owner: 10Hashar) [21:07:58] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129#3487640 (10mmodell) [21:08:02] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3487641 (10mmodell) [21:08:26] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129#2839436 (10mmodell) [21:08:30] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#2559489 (10mmodell) [21:08:49] (03PS2) 10Hashar: Run npm on mediawiki/core on stylelint/css changes [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) [21:09:21] (03CR) 10Hashar: [C: 032] Run npm on mediawiki/core on stylelint/css changes [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) (owner: 10Hashar) [21:10:14] (03Merged) 10jenkins-bot: Run npm on mediawiki/core on stylelint/css changes [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) (owner: 10Hashar) [21:11:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Jenkins, 10Patch-For-Review, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core when only CSS files are touched - https://phabricator.wikimedia.org/T171866#3487648 (10hashar) 05Open>03Resolved a:03hashar Sh... [21:13:47] Yippee, build fixed! [21:13:47] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #39: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/39/ [21:16:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikimediaMaintenance, 10User-MarcoAurelio: Allow dumpInterwiki.php to be run on Cloud VPS/Toolforge - https://phabricator.wikimedia.org/T172040#3487685 (10MarcoAurelio) @bd808 And could then be possible to run that on deployment-prep, and how could one ge... [21:16:42] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:22:37] Yippee, build fixed! [21:22:37] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #39: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/39/ [21:30:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Analytics-Kanban, 10Analytics-Wikistats: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3487815 (10hashar) I am most probably going to upgrade npm to version 3 this Tuesday. The blocker was to expli... [21:30:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3487820 (10hashar) p:05Triage>03Normal [21:34:06] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:46] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:48] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:51:42] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [22:04:25] 10Browser-Tests-Infrastructure, 10MinervaNeue, 10Reading-Web-Backlog: MinervaNeue browser test are flaking (waiting for {:class=>"mw-notification", :tag_name=>"div"} to become present ) - https://phabricator.wikimedia.org/T170890#3487873 (10Jdlrobson) Summarising what I see: In [[ https://integration.wikime... [22:55:29] twentyafterfour: you have the shell access now (and the other phab admins too) [22:55:39] now going to merge the rsync change by paladox [22:55:41] lgtm [22:55:46] thanks :) [22:56:24] paladox: just link to ticket please [22:56:31] ok [22:56:36] which ticket though [22:56:39] theres so many [22:56:40] heh [22:56:57] ehmm.. https://phabricator.wikimedia.org/T163938 [22:57:04] https://phabricator.wikimedia.org/T163938 [22:57:16] ok doing it now [22:57:16] :) [22:57:17] yea, your change is needed to solve that, ack [22:57:48] ok done mutante :) [23:30:34] twentyafterfour: what creates the dir /srv/repos itself? [23:31:30] is about to rsync the contents but i wonder if we should puppetize that [23:31:39] since it's not there yet [23:32:12] but wasnt sure if that's package/puppet/deploy/manual/X [23:34:22] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:35:14] mutante found it [23:35:15] Notice: /Stage[main]/Phabricator/File[/srv/repos]/ensure: created [23:35:25] ah :) thanks! [23:35:31] your welcome :) [23:35:39] also, let's remove that old rsync code i made in the past [23:35:54] the new class you used should replace it [23:36:10] and then that "active_server" setting should be in ONE place, and one only [23:36:13] not multiple ones [23:36:35] where did i put it if not common.yaml, i guess by role [23:37:57] yeh [23:38:30] ok, merging yours.. then we make another one [23:39:06] Ok [23:39:07] thanks [23:39:18] mkdir /srv/repos [23:40:04] no, actually not doing that [23:40:21] we are just watching what puppet does on iridium now [23:40:38] mutante: https://gerrit.wikimedia.org/r/#/c/368956/ [23:41:21] ok, it's adding the rsyncd and config [23:41:22] and fail [23:41:30] DNS query for 'phab1001.eqiad.wmnet' failed: NXDOMAIN [23:41:33] :) [23:41:34] oh [23:41:41] did i do the wrong address? [23:41:47] no, you didn't [23:41:55] the thing is that phab1001 doesn't have an AAAA record [23:42:00] but ferm needs it [23:42:03] or firewall fails [23:42:11] oh [23:42:22] so we add a aaaa record for phab1001 :) [23:42:27] Error in /etc/ferm/conf.d/10_srv-repos line 4: [23:42:28] phab1001.eqiad.wmnet [23:42:28] ) [23:42:28] , AAAA [23:42:33] yea, we need to add it [23:42:38] ok [23:42:44] i'll do that [23:43:30] fortunately this kind of thing just breaks puppet nowadays [23:43:38] in the past it broke iptables and the whole service [23:43:48] ok thanks [23:44:00] common issue. we need to always add AAAA from the start [23:44:12] but it's not really part of standard procurement [23:44:13] yet [23:44:53] it always happens once you put roles on it that have ferm rules [23:45:54] yep [23:45:58] upmf.. getting more complicated [23:46:02] yep [23:46:09] phab1001 doesnt actually get the right v6 IP yet [23:46:13] as that is also from the puppet role [23:46:19] which we dont apply currently [23:46:44] checking site.pp [23:47:14] yea, it was missing "interface::add_ip6_mapped { 'main': }" adding [23:47:21] that's on node level, not role [23:47:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:50:32] oh [23:50:40] :) [23:53:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:55:16] mutante i can create the aaa record change in the dns repo for you :) [23:56:24] oh i see [23:56:28] needs ipv6