[00:00:10] would have shown up in an hour [00:00:56] now parked next to Highway 1.. the only problem is the networks is spotty [00:01:24] mutante: did you see my pm? we need to rsync the repos, apparently the rsync didn't happen [00:02:48] twentyafterfour: yes, give me one minute to try the sync [00:06:03] sync is now running, logged in SAL [00:06:42] no, it's stuck, debugging [00:10:32] ok, fixed [00:10:43] now it's really syncing. it was a firewall problem and the fix was restarting ferm [00:10:49] the rule was there but not applied [00:10:57] like in puppet config but not iptables -L [00:13:16] twentyafterfour: it will take a few minutes but not hours or anything like that [00:13:25] wathcing it. making good progress [00:13:26] ok cool [00:16:24] working from https://www.google.com/maps/place/Swanton+Berry+Farm/@37.0303684,-122.2534024,13z/data=!4m5!3m4!1s0x808e5affa8ca38cf:0xc929c3b316511b94!8m2!3d37.0303684!4d-122.2183835?hl=en kind of off the grid. but LTE :p [00:16:49] because i was going to drive further and was wrong about the time [00:17:20] 11G of 31G [00:17:21] mutante: we can do it later if you want [00:17:33] it's cool for now :) [00:18:11] just adds to interesting places to work from. they have great food [00:18:17] :) [01:24:18] twentyafterfour: Is there a task for having beta cluster not use 'eqiad' as its dc name? I think it would improve stability if we do that and reduce confusion as well. E.g. instead of having prod/eqiad and labs/eqiad (:realm/:dc) we could have labs/labs or labs/labs-eqiad so that at least the same key isn't used twice and its' forced to be set differently instead of inheriting "prod"'s commonsettings.php default. [01:24:24] especially in LabsServices.php [01:24:42] It seems etc/wikimedia-cluster already says "labs" but we map it back to eqiad in MWReal.php [01:24:56] but that predates Services.php, so perhaps worth a shot [01:25:21] Krinkle: I don't know of anything like that but it sounds like a good idea [01:25:27] make a task? [01:25:37] twentyafterfour: What is the tag for wmf-config stuff that isn't a site request? [01:26:05] https://phabricator.wikimedia.org/project/profile/359/ ? [01:26:09] mediawiki-configuration [01:26:22] hmm no that's not right [01:26:40] That's core's config system itself [01:27:09] yeah I don't think there is a tag for this... though I guess it falls under deployment-systems roughly [01:27:13] k [01:35:27] 10Deployment-Systems: Use a cluster name in beta that doesn't clash with production - https://phabricator.wikimedia.org/T172353#3495906 (10Krinkle) [01:35:37] 10Deployment-Systems, 10Release-Engineering-Team: Use a cluster name in beta that doesn't clash with production - https://phabricator.wikimedia.org/T172353#3495919 (10Krinkle) [01:35:53] twentyafterfour: ^ [01:36:03] I'm curious what we did during the switch over. [01:38:42] ? [01:38:59] Krinkle: which switchover? [01:39:09] twentyafterfour: the one to codfw! [01:39:17] oh. I don't know honestly [01:39:39] Setting wmfDatacenter=codfw would break labs since it aint got no codfw key in LabsServices.php [01:39:59] probably just left it as eqiad [01:40:14] https://github.com/wikimedia/operations-mediawiki-config/commit/9f591ba802edad5f1ef460afea5f50dee281a8ce [01:40:25] twentyafterfour: Yeah, but we can't because $wmfDatacenter is only defined in one place [01:40:34] The above fixed it, but it has changed again since [01:40:38] and will likely break the next switch over again the same way [01:41:10] wmfMasterDatacenter = 'eqiad' in CommonSettings, then includes LabsServices/ProdServices, then $wmfMasterServices = $wmfAllServices[$wmfMasterDatacenter]; [01:42:43] hmm, then I really have no idea [01:48:46] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3495944 (10mmodell) [01:50:38] twentyafterfour: As it is, it would break beta with an E_NOTICE for accessing undefined 'codfw' from $wmfAllServices. This was hot-fixed after the last switch over (by hardcoding the assignmente differently for LabsServices.php) but since then reverted again. The reason it worked on beta before the switch over, and the reason it works today, is because prod and labs both use a cluster named 'eqiad'. Hence this task :) [01:51:37] Krinkle: makes sense. I support this change :) [01:52:02] to the extent that I just gave it a thumbs-up token in phabricator, heh [01:52:16] and I'll be glad to help if there is something I can help with [01:52:20] * Krinkle is looking through CommonSettings-labs.php [01:52:28] I didn't know there was a "testwiki" on beta [01:52:37] Although.. testwiki in beta is at test.wikiMedia.*, not test.wikiPedia.org. Wonder why that is? [01:52:43] https://test.wikimedia.beta.wmflabs.org/wiki/Main_Page [01:53:14] Krinkle: We're nothing if not inconsistent [01:53:24] Also, RCStream still exists in beta [01:53:27] * Krinkle pokes analytics [01:54:07] Reedy: https://gerrit.wikimedia.org/r/369818 [01:56:52] 10Beta-Cluster-Infrastructure, 10Analytics, 10Wikimedia-Stream: Decom RCStream in Beta Cluster - https://phabricator.wikimedia.org/T172356#3495960 (10Krinkle) [01:57:00] 10Beta-Cluster-Infrastructure, 10Analytics, 10Wikimedia-Stream: Decom RCStream in Beta Cluster - https://phabricator.wikimedia.org/T172356#3495976 (10Krinkle) Also, is there EventStreams in Beta Cluster? [01:57:24] Krinkle: deployment-stream.eqiad.wmflabs [02:12:20] Reedy: Hm.. it's doing both? [02:12:33] I guess because of EventBus being configured the same without needing an override? [02:13:11] Honestly not sure [02:23:33] 10Deployment-Systems, 10MediaWiki-extensions-CentralAuth: $wgLocalVirtualHosts should include login.wikimedia.org, wikidata.org and others? - https://phabricator.wikimedia.org/T172357#3495988 (10Krinkle) [02:28:47] (03CR) 10Krinkle: Assert MediaWiki does not generate error logs (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/369676 (https://phabricator.wikimedia.org/T50002) (owner: 10Hashar) [03:41:23] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3496059 (10mmodell) I'm making this comment from `phab1001.eqiad.wmnet` :) [04:01:17] 10Deployment-Systems, 10Release-Engineering-Team (Watching / External), 10Operations, 10Performance-Team, 10HHVM: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#3496076 (10Krinkle) [04:07:51] Yippee, build fixed! [04:07:52] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #474: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/474/ [04:10:19] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3490687 (10Antoine2711) >>! In T172217#3490701, @Reedy wrote: > What is the conference guide? >... [04:17:36] Yippee, build fixed! [04:17:37] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #474: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/474/ [04:23:42] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3496112 (10Dzahn) Hi @Antoine2711 Are the files already in a public git repository by any chance... [04:48:34] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3496134 (10Antoine2711) Hi @Dzahn, For me, it's really a simple HTTP need. The ideal of the PHP... [04:50:03] !log update cherry-pick for https://gerrit.wikimedia.org/r/#/c/299825/8 on deployment-puppetmaster02.deployment-prep.eqiad.wmflabs [04:50:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:58:38] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3496137 (10Dzahn) Hi Antoine2711, the easiest way to upload the files would probably be if you w... [05:11:40] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3496143 (10Antoine2711) Hi @Dzahn, >>! In T172217#3496137, @Dzahn wrote: > Hi Antoine2711, the... [05:16:25] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3496148 (10Dzahn) @Antoine2711 Sounds all good. I will also get back to you tomorrow, we'll get... [05:24:30] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3496157 (10Mostafa2018k) admin ipad air 2 [07:15:34] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:44:54] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, 10Scoring-platform-team: Simplify git-fat support for pulling from both production and labs - https://phabricator.wikimedia.org/T171758#3496219 (10fgiunchedi) CC'ing #operations here too for wider distribution [07:46:50] (03PS2) 10Giuseppe Lavagetto: Move some functionality from run.sh to the Dockerfile [integration/config] - 10https://gerrit.wikimedia.org/r/369677 [07:46:52] (03PS2) 10Giuseppe Lavagetto: Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 [07:46:54] (03PS2) 10Giuseppe Lavagetto: Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 [07:55:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:49:12] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:53:34] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-General-or-Unknown, 10Tracking: Let ApiDocumentationTest structure test pass on all repos - https://phabricator.wikimedia.org/T154838#3496359 (10Umherirrender) [08:53:35] 10Continuous-Integration-Config, 10FanBoxes, 10Social-Tools, 10Easy, and 2 others: Extension FanBoxes failing tests due to missing apihelp messages - https://phabricator.wikimedia.org/T155023#3496356 (10Umherirrender) 05Open>03Resolved p:05Triage>03Normal a:05SamanthaNguyen>03Umherirrender [08:59:12] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:17:06] 10Continuous-Integration-Infrastructure, 10Composer, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: Build: Handle extensions autoloading entry point from composer.json - https://phabricator.wikimedia.org/T168738#3496376 (10Umherirrender) >>! In T168738#3493673, @Kghbln... [09:18:40] (03PS1) 10Hashar: Make FanBoxes tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/369858 (https://phabricator.wikimedia.org/T155023) [09:19:27] (03CR) 10Hashar: [C: 032] "They passed on https://gerrit.wikimedia.org/r/#/c/368555/" [integration/config] - 10https://gerrit.wikimedia.org/r/369858 (https://phabricator.wikimedia.org/T155023) (owner: 10Hashar) [09:20:20] (03Merged) 10jenkins-bot: Make FanBoxes tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/369858 (https://phabricator.wikimedia.org/T155023) (owner: 10Hashar) [09:39:27] 10Beta-Cluster-Infrastructure, 10Analytics, 10Wikimedia-Stream: Decom RCStream in Beta Cluster - https://phabricator.wikimedia.org/T172356#3496434 (10hashar) It runs on `deployment-stream.deployment-prep.eqiad.wmflabs` `10.68.17.106` created by @ori when he did the RCStream project. The instance has a float... [10:12:02] !log gerrit: marked wikimedia/communications/WP-Victor read-only and [ARCHIVED] - T107430 [10:12:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:12:06] T107430: Decom bits.wikimedia.org hostname - https://phabricator.wikimedia.org/T107430 [10:17:35] (03PS1) 10Hashar: qa: ignore some gerrit repositories [integration/config] - 10https://gerrit.wikimedia.org/r/369867 [10:19:15] (03PS1) 10Hashar: Add npm to wikimedia-ui-base.git [integration/config] - 10https://gerrit.wikimedia.org/r/369868 [10:23:54] (03PS2) 10Hashar: Add npm to wikimedia-ui-base.git [integration/config] - 10https://gerrit.wikimedia.org/r/369868 [10:23:55] (03PS2) 10Hashar: qa: ignore some gerrit repositories [integration/config] - 10https://gerrit.wikimedia.org/r/369867 [10:26:39] (03CR) 10Hashar: [C: 032] Add npm to wikimedia-ui-base.git [integration/config] - 10https://gerrit.wikimedia.org/r/369868 (owner: 10Hashar) [10:26:43] (03CR) 10Hashar: [C: 032] qa: ignore some gerrit repositories [integration/config] - 10https://gerrit.wikimedia.org/r/369867 (owner: 10Hashar) [10:27:32] (03Merged) 10jenkins-bot: Add npm to wikimedia-ui-base.git [integration/config] - 10https://gerrit.wikimedia.org/r/369868 (owner: 10Hashar) [10:28:09] (03Merged) 10jenkins-bot: qa: ignore some gerrit repositories [integration/config] - 10https://gerrit.wikimedia.org/r/369867 (owner: 10Hashar) [10:32:01] (03PS1) 10Hashar: operations/puppet rake now invokes tox [integration/config] - 10https://gerrit.wikimedia.org/r/369870 (https://phabricator.wikimedia.org/T166888) [10:36:01] (03CR) 10Giuseppe Lavagetto: [C: 031] operations/puppet rake now invokes tox (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/369870 (https://phabricator.wikimedia.org/T166888) (owner: 10Hashar) [10:46:44] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Containers, 10Kubernetes, 10Services (designing), 10User-mobrovac: RFC: Container path conventions - https://phabricator.wikimedia.org/T169998#3415931 (10Joe) I think the proposal is pretty sound - with a couple of suggestions to keep things m... [10:57:27] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10VisualEditor: add VE tests for all "Insert" items - https://phabricator.wikimedia.org/T65708#3496656 (10Liuxinyu970226) [10:57:30] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Flow: Flow reply_moderation browser test has erroneous selector for "3rd comment on the topic" - https://phabricator.wikimedia.org/T85201#3496658 (10Liuxinyu970226) [11:06:33] PROBLEM - Puppet errors on integration-slave-docker-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:21:13] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:22:07] twentyafterfour it seems https://phabricator-new.wikimedia.org/ redirects mutiple times causing the browser to cut it [11:22:13] accessing https://phabricator-new.wikimedia.org/dashboard/view/1/ works though :) [11:23:45] oh i found this [11:23:46] https://phabricator-new.wikimedia.org/source/extensions/ [11:23:55] Command failed with error #128! [11:24:27] it seems all the repos are doing it [11:29:31] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3496703 (10Paladox) I found https://phabricator-new.wikimedia.org/ continuously redirecting which then the browser short circuits it. But http... [12:02:34] !log Added integration-slave-docker-1004 to the pool of jenkins slaves - T150502 [12:02:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:02:35] T150502: Set up experimental Docker CI slave - https://phabricator.wikimedia.org/T150502 [12:11:31] RECOVERY - Puppet errors on integration-slave-docker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [12:25:21] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:45:22] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:47:40] (03PS2) 10Hashar: operations/puppet rake now invokes tox [integration/config] - 10https://gerrit.wikimedia.org/r/369870 (https://phabricator.wikimedia.org/T166888) [12:49:01] 10Continuous-Integration-Infrastructure, 10Composer, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: Build: Handle extensions autoloading entry point from composer.json - https://phabricator.wikimedia.org/T168738#3496850 (10Kghbln) > The autoloading could break when the... [13:01:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3496881 (10hashar) Eventually I was looking at the rabbitMQ reported stat... [13:02:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Create a basic RSpec unit test for operations/puppet - https://phabricator.wikimedia.org/T78342#3496885 (10hashar) 05Open>03Resolved Bulk of the rspec integration has been done a while... [13:45:06] (03CR) 10Hashar: [C: 032] operations/puppet rake now invokes tox [integration/config] - 10https://gerrit.wikimedia.org/r/369870 (https://phabricator.wikimedia.org/T166888) (owner: 10Hashar) [13:46:49] (03Merged) 10jenkins-bot: operations/puppet rake now invokes tox [integration/config] - 10https://gerrit.wikimedia.org/r/369870 (https://phabricator.wikimedia.org/T166888) (owner: 10Hashar) [13:49:48] (03PS1) 10Hashar: Job for puppet no more generate stuff to /log [integration/config] - 10https://gerrit.wikimedia.org/r/369911 [13:50:25] (03CR) 10Hashar: [C: 032] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/369911 (owner: 10Hashar) [13:52:53] (03Merged) 10jenkins-bot: Job for puppet no more generate stuff to /log [integration/config] - 10https://gerrit.wikimedia.org/r/369911 (owner: 10Hashar) [14:05:32] (03PS3) 10Giuseppe Lavagetto: Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 [14:11:29] 10Continuous-Integration-Config, 10MediaWiki-extensions-Contest, 10Easy, 10I18n: Extension Contest failing tests due to missing apihelp messages - https://phabricator.wikimedia.org/T155020#3497134 (10Umherirrender) p:05Triage>03Normal a:03Umherirrender [14:16:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Analytics-Kanban, 10Analytics-Wikistats: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3497168 (10fdans) @hashar thank you so much! I've changed a couple of wrong calls to semantic that I added some... [15:04:11] Reedy: which is the ssh for deployment-prep? [15:04:20] What do you mean? [15:04:29] You can go via bastion.wmflabs.org if you wanted [15:04:42] https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep <-- that [15:05:45] Go via a bastion [15:05:59] login to labs first I supose then [15:06:17] You can setup your ssh config so you can go "directly" [15:06:24] or just shh -A bastion.wmflabs.org first [15:06:28] *ssh [15:08:13] Reedy: permission denied (publickey) [15:08:29] maurelio@tools-bastion-03:~$ ssh -A bastion.wmflabs.org [15:08:47] You don't need to go tools-bastion to bastion [15:08:56] you'll need to key forward to hop to another host [15:09:06] so you need the -A on your original ssh command from your local box [15:15:52] maurelio@tools-bastion-03:~$ ssh deployment-tin.deployment-prep.eqiad.wmflabs [15:16:00] greg-g: ^ ? [15:16:03] that correct? [15:17:20] Does it work? [15:17:36] doesn't fail either but I don't receive any message [15:17:37] You can just use `ssh deployment-tin` from there [15:18:05] ssh: connect to host deployment-tin.deployment-prep.eqiad.wmflabs port 22: Connection timed out [15:18:07] doint you have to have your ssh key in your user home if you doint proxy it through? [15:19:09] * TabbyCat is waiting for another timeout [15:20:22] that's what ssh -A [15:21:17] TabbyCat: [16:06:24] or just shh -A bastion.wmflabs.org first [15:21:43] Reedy: I guess username@bastion.. ? [15:21:54] or just ssh -A bastion.wmflabs.org? [15:22:02] if your local username isn't the same as your labs shell username [15:23:09] Reedy: okay I'm now at bastion01 [15:23:18] ssh deployment-tin [15:23:51] Permission denied (publickey). [15:23:58] Did you use ssh- A? [15:24:20] maurelio@bastion-01:~$ ssh -A deployment-tin [15:24:22] Permission denied (publickey). [15:24:30] I mean the first time [15:24:44] $ ssh -A maurelio@bastion.wmflabs.org ? [15:24:46] yes I did [15:25:05] but ssh to deployment-tin with and w/o -A fails [15:25:13] what's your shell username? [15:25:20] maurelio [15:25:35] you don't have a home dir on deployment-tin [15:25:40] (03PS1) 10Umherirrender: [Contest] Make unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/369935 [15:25:59] Let's try re-adding you [15:26:12] Reedy: I got one automatically at bastion01 [15:26:19] Completely different [15:26:24] yep [15:32:27] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Wikimedia-Stream, 10Patch-For-Review: Decom RCStream in Beta Cluster - https://phabricator.wikimedia.org/T172356#3497562 (10elukey) [15:33:20] Nfi what it's doing [15:33:28] I'll just update the IW map so it's done [15:33:36] Reedy: okay [15:33:37] (03CR) 10jerkins-bot: [V: 04-1] [Contest] Make unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/369935 (owner: 10Umherirrender) [15:35:17] (03PS3) 10Thcipriani: Move some functionality from run.sh to the Dockerfile [integration/config] - 10https://gerrit.wikimedia.org/r/369677 (owner: 10Giuseppe Lavagetto) [15:35:19] (03PS3) 10Thcipriani: Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 (owner: 10Giuseppe Lavagetto) [15:35:21] (03PS4) 10Thcipriani: Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [15:35:23] (03PS1) 10Thcipriani: Docker: bump image for operations-puppet to 0.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/369938 [15:39:59] (03CR) 10Thcipriani: [C: 032] Move some functionality from run.sh to the Dockerfile [integration/config] - 10https://gerrit.wikimedia.org/r/369677 (owner: 10Giuseppe Lavagetto) [15:40:04] (03CR) 10Thcipriani: [C: 032] Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 (owner: 10Giuseppe Lavagetto) [15:40:05] TabbyCat: Can you update the updated date on https://meta.wikimedia.org/wiki/Interwiki_map please? [15:40:08] (03CR) 10Thcipriani: [C: 032] Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [15:40:14] Reedy: sure [15:41:11] {{done}} [15:41:31] (03Merged) 10jenkins-bot: Move some functionality from run.sh to the Dockerfile [integration/config] - 10https://gerrit.wikimedia.org/r/369677 (owner: 10Giuseppe Lavagetto) [15:42:06] (03CR) 10jerkins-bot: [V: 04-1] Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [15:43:42] (03PS2) 10Umherirrender: [Contest] Make unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/369935 [15:43:47] (03PS3) 10Umherirrender: [Contest] Make unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/369935 [15:48:38] Cheers [15:51:14] (03CR) 10Thcipriani: [C: 032] Docker: bump image for operations-puppet to 0.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/369938 (owner: 10Thcipriani) [15:51:21] (03CR) 10jerkins-bot: [V: 04-1] Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 (owner: 10Giuseppe Lavagetto) [15:51:23] (03CR) 10jerkins-bot: [V: 04-1] Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [15:51:26] (03CR) 10jerkins-bot: [V: 04-1] Docker: bump image for operations-puppet to 0.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/369938 (owner: 10Thcipriani) [15:57:43] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-MarcoAurelio, 10User-greg: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3497699 (10MarcoAurelio) Thank you very much @greg. However once I `ssh -A maurelio@bastion.wmflabs.org`,... [15:58:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-MarcoAurelio, 10User-greg: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3489008 (10Reedy) It seems a home directory hasn't been created by puppet [16:06:35] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Scap: keyholder Too many authentication failures - https://phabricator.wikimedia.org/T172333#3497717 (10demon) [16:06:38] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3497716 (10demon) [16:09:17] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-MarcoAurelio, 10User-greg: Requesting access to deployment-prep for @MarcoAurelio - https://phabricator.wikimedia.org/T172182#3497721 (10bd808) >>! In T172182#3497702, @Reedy wrote: > It seems a home directory hasn't been created by... [16:09:46] hi bd808 ;) [16:10:27] hey TabbyCat. I'm tailing auth.log on deployment-tin if you want to try again [16:10:56] bd808: if you could guide me step to step I'd appreciate, the docs needs some KISS updates :) [16:11:14] I'll ssh -A maurelio@bastion.wmflabs.org for starters if that's okay [16:11:33] bd808: I'm guessing a lack of a home dir with a ssh public key in is a big factor [16:11:44] ssh keys are read from ldap [16:12:08] TabbyCat: agent forwarding isn't ideal, but yeah that should work for now [16:12:38] Reedy: the way that ssh works in cloud vps is under-documented and 'interesting' :) [16:12:50] bd808: on bastion01 now [16:13:08] what now? [16:13:33] ssh deployment-tin.deployment-prep.eqiad.wmflabs [16:14:11] maurelio@bastion-01:~$ ssh deployment-tin.deployment-prep.eqiad.wmflabs [16:14:13] Permission denied (publickey). [16:14:14] maurelio@bastion-01:~$ [16:15:33] I didn't see anything that even looked like a failed connection attempt :/ [16:15:48] I think we need a trace from the client too. [16:16:08] ssh -vvv deployment-tin.deployment-prep.eqiad.wmflabs -- and then paste the log output [16:16:12] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-General-or-Unknown, 10Tracking: Let ApiDocumentationTest structure test pass on all repos - https://phabricator.wikimedia.org/T154838#3497732 (10Umherirrender) [16:17:37] bd808: mind if I PM you a link with the contents? It's very big [16:18:06] you can just paste it at https://phabricator.wikimedia.org/paste/edit/form/14/ and then link here [16:18:18] it won't have any 'secrets' in it [16:19:10] https://phabricator.wikimedia.org/P5849 [16:20:21] fwiw the form 14 is just avalaible to wmf-nda people :) [16:20:28] so I cannot use it [16:20:50] TabbyCat https://phabricator.wikimedia.org/paste/ [16:20:56] yepo [16:21:00] *yes [16:21:07] ah. https://phabricator.wikimedia.org/paste/edit/form/26/ I guess [16:21:28] bd808: https://phabricator.wikimedia.org/P5849 [16:21:36] TabbyCat: your session starting from bastion01 does not have an ssh key [16:21:57] so how I could even login there? [16:22:08] you can probably confirm with `ssh-add -l` [16:22:20] it asks for the password of .ssh/id_rsa on my local machine before connecting [16:22:27] to tools-bastion01 [16:22:40] there is a key on your laptop, but it is not not being forwarded into the bastion. probably means you are not running an ssh agent [16:22:58] https://wikitech.wikimedia.org/wiki/Help:Access [16:23:24] maurelio@bastion-01:~$ ssh-add -l [16:23:26] Could not open a connection to your authentication agent. [16:23:27] maurelio@bastion-01:~$ [16:24:01] Unfortunately I don't have time right now to walk you through how to ssh via a jump host [16:24:11] maybe someone else here does though? [16:24:14] it's strange 'cause on other instances of Labs and Gerrit I don't face any issues [16:24:29] those are both directly accessible [16:24:37] TabbyCat why not do something like Host gerrit-test [16:24:38] ProxyCommand ssh -a -W %h:%p paladox@primary.bastion.wmflabs.org [16:24:39] UseRoaming no [16:24:40] User paladox [16:24:41] woops [16:24:42] that was in a paste [16:24:43] https://phabricator.wikimedia.org/P5850 [16:24:54] in ~/.ssh/config [16:24:57] on your computer [16:25:23] assuming that you mean "Toolforge" when you say "other instances of Labs" [16:25:25] SSH suckssssssssssss for gitttttttttttttt [16:25:44] * RainbowSprinkles puts on the "use HTTPS its way easier" record [16:26:12] TabbyCat: this method is ideal -- https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 [16:26:13] RainbowSprinkles you carn't http into a host :) [16:27:01] or if using PuTTY -- https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP [16:29:12] * TabbyCat doesn't have a /config file in the .ssh folder @ laptop [16:29:25] You create it :) [16:29:30] vi ~/.ssh/config [16:29:51] on windows i belive it's in c:\\Users//.ssh [16:30:51] TabbyCat ^^ :) [16:31:19] I'm archiving a rfa, hold on :) [16:31:30] I'll create the file in a minute [16:31:47] okay I'm back [16:34:27] well I've got the config file now [16:34:32] will repeat previous steps [16:35:33] TabbyCat if you copied the config file change the host name ie gerrit-test to the instance you want to connect [16:35:39] and change paladox with your username :) [16:35:42] then do [16:35:46] ssh [16:35:50] ie ssh gerrit-test [16:36:03] paladox: I did copied the one bd808 provided to me [16:36:07] Help:Access [16:36:08] ok [16:37:34] nope, doesn't work [16:38:12] TabbyCat try https://phabricator.wikimedia.org/P5850 [16:44:15] paladox: packet_write_wait: Connection to UNKNOWN port 65535: Broken pipe [16:50:28] paladox: fixed with ssh-add [16:50:32] I'm now in [16:50:33] ok [17:09:58] (03CR) 10Thcipriani: Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 (owner: 10Giuseppe Lavagetto) [17:10:03] (03PS4) 10Thcipriani: Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 (owner: 10Giuseppe Lavagetto) [17:10:11] (03CR) 10Thcipriani: [C: 032] Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 (owner: 10Giuseppe Lavagetto) [17:15:17] (03Merged) 10jenkins-bot: Update bundle if gemfile is changed [integration/config] - 10https://gerrit.wikimedia.org/r/369678 (owner: 10Giuseppe Lavagetto) [17:16:41] (03CR) 10Thcipriani: Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [17:16:43] (03PS5) 10Thcipriani: Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [17:16:45] (03CR) 10Thcipriani: [C: 032] Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [17:18:43] duh, now I have to learn how to do things there and not to break anything :) [17:19:37] TabbyCat: glad you got in :) [17:20:08] greg-g: so, say that I want to run the dumpInterwiki.php script, where should I do that? [17:21:42] (03Merged) 10jenkins-bot: Switch to using Rakefile.ci, remove tox and parallel execution [integration/config] - 10https://gerrit.wikimedia.org/r/369679 (owner: 10Giuseppe Lavagetto) [17:24:39] TabbyCat: deployment-tin I suppose, pretty sure we don't have a "work machine" in beta cluster [17:24:55] eg terbium in production (for long running maint scripts etc) [17:26:13] (03CR) 10Thcipriani: Docker: bump image for operations-puppet to 0.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/369938 (owner: 10Thcipriani) [17:26:18] (03PS2) 10Thcipriani: Docker: bump image for operations-puppet to 0.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/369938 [17:26:30] (03CR) 10Thcipriani: [C: 032] Docker: bump image for operations-puppet to 0.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/369938 (owner: 10Thcipriani) [17:29:11] 10Continuous-Integration-Config, 10MediaWiki-extensions-Survey, 10Easy, 10I18n, 10Patch-For-Review: Extension Survey failing tests due to missing apihelp messages - https://phabricator.wikimedia.org/T155031#3497951 (10Umherirrender) p:05Triage>03Normal a:03Umherirrender [17:31:30] (03Merged) 10jenkins-bot: Docker: bump image for operations-puppet to 0.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/369938 (owner: 10Thcipriani) [17:37:00] (03PS1) 10Umherirrender: [Survey] Make unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/369968 [17:39:28] bd808: btw, I'm downloading the mw-vagrant iso now and it's super slow (other things aren't on my connection right now), like 80-100kBs [17:39:59] that vm has been a pain :/ [17:40:14] 8hours remaining :( [17:40:56] sigh now rejecting me again aargh [17:41:18] Reedy: about how long does it take for a subdomain creation (to an existing wiki page) to be approved/merged? [17:42:00] harej: What do you mean? [17:42:25] Reedy: https://phabricator.wikimedia.org/T172417 [17:42:39] (I don't expect it to be done by Wikimania, but it took me several days to get that task submitted, so.) [17:43:29] It's easy enough [17:43:46] But I guess it requires ops seeing it to sign it off [17:44:02] And is this something that can be SWATted or does it have to wait for the deployment train? [17:44:31] Neither [17:44:37] It could go out in puppet swat if you find an opsen that is comfortable doing apache changes [17:45:23] greg-g: I'm pulling it faster than that onto a vm [17:45:24] Will give you a link when it's there [17:45:34] Reedy: thanks :/ [17:45:41] guess i could do that to my vps too [17:45:50] I guess what I am wondering is how much of the wait is approval and how much of it is waiting for the proper deploy window [17:46:02] harej: How long is a piece of string? [17:46:10] Unfortunately [17:46:23] yeah, 10megs/sec on my vps, I'll just do that Reedy thanks [17:46:44] (maybe it's my crappy connection?!) [17:47:06] it could be, or just peering to EQIAD. or.. or... [17:47:20] * greg-g nods [17:53:58] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3498026 (10Dzahn) @Antoine2711 Please go to [[ https://wikitech.wikimedia.org | wikitech wiki ]... [17:53:59] Reedy: thank you :D [17:55:20] harej: FWIW, it should be easy to get ops to sign off on a text subdomain [17:55:44] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3498043 (10Dzahn) @Antoine2711 Please go to [[ https://wikitech.wikimedia.org | wikitech wiki ]... [18:07:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-Other, 10Easy, 10I18n: Extension WikiObjectModel failing tests due to missing apihelp messages - https://phabricator.wikimedia.org/T155034#3498132 (10Umherirrender) p:05Triage>03Normal a:03Umherirrender [18:13:15] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3498151 (10Antoine2711) Hi @Dzahn, my WikiTech account is the same as my Wiki account: Antoine27... [18:13:21] grrr, it's my connection for some reason :/ rsync'ing from my vps isn't going any faster. I'll download to my NAS so I can let it run for the 8-10 hours needed [18:14:18] greg-g: download manager with multiple streams? [18:16:07] greg-g: heh, you are lucky. I currently get 6 mbits per sec here [18:16:24] * paladox gets over 70 here [18:17:22] ok, this is weird [18:17:59] rsync'ing from my vps: getting about 400 now, wget'ing on my NAS from labs^WCloud is now 700 (kB/s) [18:18:05] at the same time :) [18:18:24] lol [18:18:26] 1 meg on NAS now. Ok, this'll work [18:18:46] I wonder if my NAS was doing something stupid and using my bandwidth [18:58:28] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:33] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3498327 (10mmodell) [19:24:58] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3498484 (10mmodell) [19:28:46] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikiLexicalData-or-OmegaWiki, 10Easy, 10I18n: Extension OmegaWiki failing tests due to missing apihelp messages - https://phabricator.wikimedia.org/T155044#3498557 (10Umherirrender) a:03Umherirrender [19:29:07] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3498559 (10mmodell) [19:52:38] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3498691 (10mmodell) [20:06:34] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3498769 (10Dzahn) fixed simplelap puppet role to support stretch so we can put Apache/PHP on Deb... [20:18:09] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3498820 (10Krinkle) [20:24:10] 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimania-Hackathon-2017-Organization: Wikimania needs hosting on a server for onsite conference guide - https://phabricator.wikimedia.org/T172217#3498846 (10Dzahn) @Antoine2711 I created a new VM (instance) in the existing project called "w... [20:45:41] twentyafterfour: Not sure I understand why wmf.12 went to (almost) all wikis given the 5 open blockers? [20:48:16] At least 3 of the bugs are Wikidata... [21:34:27] Krinkle: yeah it's almost all wikidata, the rest are probably not really blockers (they are infrequent) [22:13:48] twentyafterfour hi, when we do the last rsync for phabricator we will need to stop phd to prevent it corrupting the rsync :) [23:01:58] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3499254 (10Krinkle) [23:13:30] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Performance-Team: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3499316 (10Krinkle) [23:20:34] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3499331 (10Dzahn) steps for migration: - stop phd on iridium - rsync /srv/repos - merge https://gerrit.wikimedia.org/r/369834 - merge https:/... [23:21:48] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Performance-Team, 10Regression: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3499336 (10Krinkle) [23:33:58] twentyafterfour: paladox: https://phabricator.wikimedia.org/T163938#3499331 see you in half an hour? [23:34:06] mutante thanks :) [23:34:12] tried to script the whole thing more or less [23:34:19] the IP switch i mean [23:35:55] yep [23:35:56] thanks [23:36:46] mutante: thanks for your help here, as always [23:37:39] :) [23:39:59] mutante: ok