[00:03:04] are you planning to rename the channel? [00:58:06] Krinkle: I merged the patch but didn't do anything after that. I don't know if that script requires running by hand or not. [00:59:01] 06Labs, 10DBA, 13Patch-For-Review, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#2836353 (10Andrew) I merged the puppet change, but maybe this needs to be run by hand -- I've never done it. [01:29:06] 06Labs, 10Labs-Infrastructure: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3152853 (10Dzahn) a:03Dzahn [01:36:15] 06Labs, 10Labs-Infrastructure: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3152889 (10Dzahn) we won't be able to move this to LE since it's in .wmnet and LE isn't allowed to sign non-public certs. [1] [2] " http-01 and tls-sni-01 need the domain to be publicly access... [01:40:34] 06Labs, 10Labs-Infrastructure: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3152895 (10Dzahn) p:05Triage>03High wow, only 2 days left? ouch, raising priority. openssl x509 -enddate -noout -in labvirt-star.eqiad.wmnet.crt notAfter=Apr 5 19:36:12 2017 GMT this... [11:15:41] 06Labs, 10DBA: Prepare and check storage layer for khw.wikipedia - https://phabricator.wikimedia.org/T160870#3153421 (10Rachitrali) 05stalled>03Open Dear brothers and Sisters, I am affiliated with Khowar Wikipedia incubator project as test admin since 2008 and regularly contributing with wikimedia foundati... [11:43:35] 06Labs, 10DBA: Prepare and check storage layer for khw.wikipedia - https://phabricator.wikimedia.org/T160870#3153454 (10Urbanecm) 05Open>03declined Hence the parent task is closed, this doesn't make sense anymore. [12:00:13] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3153506 (10fantasticfears) [12:05:44] I was going to look at prvisioning instances on labs using ansible, does anyone know what the openstack api location is that I would need to point to? [13:41:03] 06Labs, 10Labs-Infrastructure, 07artificial-intelligence: Provide large disk space to wikibrain for memory-mapped file - https://phabricator.wikimedia.org/T161554#3153726 (10Halfak) @Shilad can probably answer better than I can on most points. It seems that, right now, at least 200GB is essential. I expe... [13:46:00] 06Labs, 10Labs-Infrastructure, 07artificial-intelligence: Provide large disk space to wikibrain for memory-mapped file - https://phabricator.wikimedia.org/T161554#3153745 (10Halfak) FYI: https://en.wikipedia.org/w/index.php?diff=773800647 > Hey The Transhumanist! I think WikiBrain is a great project. I real... [14:04:37] 06Labs, 10DBA, 06Operations: eqiad: (2) hardware access request for labsdb1004 & 5 refresh - https://phabricator.wikimedia.org/T161754#3153870 (10chasemp) [14:15:18] addshore: The keystone api is on labcontrol1001 port 5000 [14:15:52] Access to query calls is open to Labs VMs but in general you won't be able to create/delete things from a VM [14:21:00] ahh okay, so no provisioning via it? :( [14:36:47] 06Labs, 06Operations, 13Patch-For-Review: Instance creation fails before first puppet run around 1% of the time - https://phabricator.wikimedia.org/T160908#3153926 (10Andrew) I shutdown a failed instance and mounted the drive. ``` andrew@labvirt1008:/tmp/mnt/var/lib/dhcp$ ls dhclient.eth0.leases dhclient.... [14:43:17] addshore: only if your service is running on a production host. [14:43:24] What is it that you want/need to do? [14:48:34] andrewbogott: nope, I was just going to see if I could ansibilise setting up various testing environments on labs, including the spin up of instances (so avoiding the UI) [14:49:14] addshore: that's a reasonable thing to want to do, but I'm leery of automated instance creation since our cluster is easily overwhelmed [14:49:32] understandable :) [14:49:54] basically, I'm playing around with ansbible and seeing how it can help me in day to day stuff [14:52:59] #addshoreislazy [14:59:03] 06Labs, 06Operations, 13Patch-For-Review: Instance creation fails before first puppet run around 1% of the time - https://phabricator.wikimedia.org/T160908#3153973 (10Andrew) Until I can get install1001 to STOP responding to labs dhcp requests, I'm going to assume that that's the problem. [15:12:45] Reedy: yus ;) [15:16:22] 06Labs, 10DBA, 06Operations: eqiad: (2) hardware access request for labsdb1004 & 5 refresh - https://phabricator.wikimedia.org/T161754#3154025 (10chasemp) 05Open>03stalled [15:16:33] 06Labs, 10DBA, 06Operations: eqiad: (2) hardware access request for labsdb1006 & 7 refresh - https://phabricator.wikimedia.org/T161755#3154026 (10chasemp) 05Open>03stalled [15:29:16] 06Labs, 06Operations, 13Patch-For-Review: Instance creation fails before first puppet run around 1% of the time - https://phabricator.wikimedia.org/T160908#3154082 (10Andrew) Ok, I no longer thing that install1001 is involved. Instead, it's something to do with an IP getting assigned to two instances at onc... [15:47:12] andrewbogott or chasemp hi, could we make linux 4.9 the default kernal in labs please? I found it has performance improvements. Unless it is just the rebooting i did. I've updated 2 instances without any problems. [15:57:50] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Bgoncalves was created, changed by Bgoncalves link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Bgoncalves edit summary: Created page with "{{Tools Access Request |Justification=Research on language use. |Completed=false |User Name=Bgoncalves }}" [16:00:32] paladox: I hope to roll out a new image sometime soon, it'll have the latest kernel &c. [16:00:43] ok thanks [16:17:35] no patches for puppet swat [16:23:16] jynus was that meant for -operations? Or -labs? [16:23:30] yes, wrong channel [16:23:38] ok [17:04:10] 06Labs, 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3154301 (10hashar) [17:04:31] 06Labs, 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3154301 (10hashar) [17:06:18] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#3154317 (10hashar) [17:06:21] 06Labs, 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3154316 (10hashar) [17:08:43] 06Labs, 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: For contintcloud either add RAM or Swap to the instances - https://phabricator.wikimedia.org/T162166#3154319 (10hashar) [18:12:04] 06Labs, 10Labs-Infrastructure, 10labs-sprint-118, 10Monitoring, and 2 others: monitor expiration of labvirt-star SSL cert - https://phabricator.wikimedia.org/T116332#3154475 (10Dzahn) after the merge above now Icinga checks turned CRIT as they should have. due to a bug they stayed just WARN before for long... [18:39:39] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kobitate94 was created, changed by Kobitate94 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Kobitate94 edit summary: Created page with "{{Tools Access Request |Justification=Creating a Twitter bot similar to twitter.com/congressedits for publishing edits made by people on campus at Georgia Southern University...." [19:17:35] 06Labs, 10Labs-Infrastructure: Experiment with Linux KSM (dedupe memory shared by instances) on labs infra - https://phabricator.wikimedia.org/T146037#3154840 (10hashar) [19:19:14] 06Labs, 10Labs-Infrastructure: Experiment with Linux KSM (dedupe memory shared by instances) on labs infra - https://phabricator.wikimedia.org/T146037#2648431 (10hashar) [21:26:39] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3155253 (10Dzahn) tested on labvirt1014 first, then deployed on all. TLDR: [puppetmaster1001:/srv/private/modules/secret/secrets/ssl] $ - revoke old cert: sudo openssl... [21:27:13] 06Labs, 10Labs-Infrastructure, 06Operations: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3155258 (10Dzahn) [21:27:24] 06Labs, 10Labs-Infrastructure, 06Operations: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3152009 (10Dzahn) 05Open>03Resolved [21:29:01] 06Labs, 10Labs-Infrastructure, 06Operations: labvirt-star.eqiad.wmnet.crt expiring soon - https://phabricator.wikimedia.org/T162085#3152009 (10Dzahn) @labvirt1014:~# openssl x509 -in /etc/ssl/localcerts/labvirt-star.eqiad.wmnet.crt -text -noout | grep After Not After : Mar 25 21:00:52 2019 GMT [22:15:25] 10MediaWiki-extensions-OpenStackManager, 10Librarization, 10MediaWiki-extensions-Translate: Bring in spyc for OpenStackManager and Translate via composer - https://phabricator.wikimedia.org/T75945#3155408 (10Reedy) We have 0.6.2... https://packagist.org/packages/mustangostang/spyc [22:18:49] 10MediaWiki-extensions-OpenStackManager, 10Librarization, 10MediaWiki-extensions-Translate: Bring in spyc for OpenStackManager and Translate via composer - https://phabricator.wikimedia.org/T75945#3155412 (10Reedy) 05stalled>03Open [23:30:31] Hey gang, I'm having difficulties SSH'ing in for the first time, wanted to make sure everything was good on your end. [23:38:34] with what username? [23:38:40] what error message do you get? [23:41:50] gry: drewmutt https://pastebin.com/TzmLxjG9 [23:43:01] At wikitech, you already added your ssh key in the settings, yes? [23:43:19] I believe so, lemme 2x check. [23:43:31] I see it on toolsadmin [23:44:30] And I also see it in my wikitech prefs. [23:45:24] How long ago did you sign up and how long ago did you add the ssh key in? [23:46:03] SSH key about 3 weeks ago, access granted about 3 days ago. [23:46:18] I can just spin up another key if you think that would help. [23:47:17] $ id drewmutt [23:47:17] uid=16693(drewmutt) gid=500(wikidev) groups=50062(project-bastion),50380(project-tools),500(wikidev) [23:47:26] the access should be okay [23:48:13] probably key issue afaict [23:48:36] Well, if you're sure it good on your side, I can just hammer on it on my side. [23:50:44] Btw, if I put up a new key, is there a waiting period for that? [23:51:53] it's updated via ldap. I'm not 100% familiar with its internal workings [23:52:40] drewmutt: it should be pretty close to an instant change [23:53:03] Awesome guys, I'll give it a shot, thanks! [23:53:46] drewmutt: what host were you trying to ssh into? I could try to dig up the auth.log data [23:54:11] drewmutt@login.tools.wmflabs.org [23:55:30] "Failed publickey for drewmutt from xxx.xxx.xxx.xxx port 55565 ssh2: RSA SHA256:WLsfOslnP1UFAL7YVHjO2ohZlCL1GaQV5TIINDtX8R0" [23:56:23] it looks like the key you are presenting doesn't match LDAP [23:56:56] Ah.. okie, well, I'll update it then and try again.