[00:36:30] 10Labs-project-Wikistats: wikistats: add new wikipedias: kbp, khw, dty - https://phabricator.wikimedia.org/T160947#3135956 (10Dzahn) [00:43:19] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fbgonzaga was created, changed by Fbgonzaga link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Fbgonzaga edit summary: Created page with "{{Tools Access Request |Justification=We have developed research on mathematical content. For information retrieval purposes, we would like to measure the update rate of mathe..." [03:27:33] (03CR) 10BryanDavis: [C: 032] Ping sending nick when reporting !log errors [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/343925 (owner: 10BryanDavis) [03:28:17] (03Merged) 10jenkins-bot: Ping sending nick when reporting !log errors [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/343925 (owner: 10BryanDavis) [03:30:43] !log foo bar [03:30:43] bd808: Unknown project "foo" [03:31:21] !log stashbot Updated to 98b4407 Ping sending nick when reporting !log errors [03:31:21] bd808: Unknown project "stashbot" [03:31:22] bd808: Did you mean to say "tools.stashbot" instead? [03:31:30] !log tools.stashbot Updated to 98b4407 Ping sending nick when reporting !log errors [03:31:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [03:33:44] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Have Edit Counter use same architecture and front-end as the other pieces that have been re-written - https://phabricator.wikimedia.org/T160481#3136121 (10Samwilson) The edit counter has a table of "SUL editcounter" totals for the top 10 for the given user. Th... [03:53:59] (03PS1) 10MtDu: Wikibugs realname should use HTTPS over HTTP [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/345087 (https://phabricator.wikimedia.org/T161421) [04:00:29] 10Wikibugs, 07Easy, 13Patch-For-Review: Wikibugs realname should use HTTPS - https://phabricator.wikimedia.org/T161421#3136126 (10MtDu) a:03MtDu Should be fixed now. [06:38:35] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:17:41] whois madhuvishy [07:18:33] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:32] madhuvishy: Thanks for assisting with the repliaca issue. I've managed to connect to the databases from one of the shell users. I have one more thing I need to solve, i.e. the issue of not being able to connect via shell with the cff user. One of the people there said to create a private task for that. However I'm not sure how to do that. Can you assist? The issue is this: https://phabricator.wikimedia.org/T151296#2823885 [07:23:21] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3136241 (10MnemonicFlow) @madhuvishy Thanks! Replica file works for user mnemonicflow! However, for shell user cff, I couldn't test as I have the problem that I cannot connect through ssh to the se... [09:56:15] 06Labs, 10Tool-Labs, 10DBA: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3136514 (10jcrespo) 14% now maybe enough until decommission? [10:01:49] 06Labs, 10Tool-Labs, 10DBA: u3532__ (=marcmiquel) table using 64G on labsdb1001 and 108 GB on labsdb1003 - https://phabricator.wikimedia.org/T133322#3136515 (10jcrespo) @chasemp @madhuvishy @bd808 The user @marcmiquel didn't answer after 2 weeks notice, maybe we should consider dropping his 100GB database co... [10:06:43] 06Labs, 10Tool-Labs, 10DBA: u3532__ (=marcmiquel) table using 64G on labsdb1001 and 108 GB on labsdb1003 - https://phabricator.wikimedia.org/T133322#3136519 (10marcmiquel) You can delete the databases. I am sorry for not replying, I did not see the e-mail. Thanks. [10:22:42] 06Labs, 10Tool-Labs, 10DBA: u3532__ (=marcmiquel) table using 64G on labsdb1001 and 108 GB on labsdb1003 - https://phabricator.wikimedia.org/T133322#3136543 (10jcrespo) 05Open>03Resolved Done. [10:22:44] 06Labs, 10Tool-Labs, 10DBA: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3136545 (10jcrespo) [12:57:02] (03PS1) 10Gehel: elasticsearch - add dummy certificates for testing [labs/private] - 10https://gerrit.wikimedia.org/r/345136 [12:58:55] (03CR) 10Gehel: [C: 032] elasticsearch - add dummy certificates for testing [labs/private] - 10https://gerrit.wikimedia.org/r/345136 (owner: 10Gehel) [12:59:00] (03CR) 10Gehel: [V: 032 C: 032] elasticsearch - add dummy certificates for testing [labs/private] - 10https://gerrit.wikimedia.org/r/345136 (owner: 10Gehel) [13:14:22] (03PS1) 10Gehel: elasticsearch - moved dummy certificates to the correct location [labs/private] - 10https://gerrit.wikimedia.org/r/345137 [13:14:33] (03CR) 10Gehel: [V: 032 C: 032] elasticsearch - moved dummy certificates to the correct location [labs/private] - 10https://gerrit.wikimedia.org/r/345137 (owner: 10Gehel) [14:36:59] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: labvirt1001 and 1002 cannot launch new VMs - https://phabricator.wikimedia.org/T159721#3136986 (10Andrew) 05Open>03Resolved I wasn't able to reproduce any problems with 1002, so I've put it back in the scheduling pool. 1001 has it's own bug, T159835, wh... [14:42:32] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:56:04] bd808: FYI, that reportcard 301 is no longer a 404, we will updating it within the week [14:56:14] bd808: thanks for the help again [15:07:22] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3137099 (10bd808) p:05High>03Normal >>! In T151296#3136241, @MnemonicFlow wrote: > However, for shell user cff, I couldn't test as I have the problem that I cannot connect through ssh to the ser... [15:08:49] nuria: glad to help. also glad I remembered that Yuvi and I had built a thing to do this :) [15:22:31] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:30] 06Labs, 13Patch-For-Review: Issues in enabling NFS for new projects (was 'adding project to nfs-mounts.yaml does not create directories') - https://phabricator.wikimedia.org/T158883#3137305 (10chasemp) [15:48:28] !log ores staged ores-wmflabs-deploy:18beebf [15:48:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [15:48:34] beebf! [15:57:46] !log ores deployed ores-wmflabs-deploy:18beebf [15:57:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [15:58:13] !log ores deployed ores-wmflabs-deploy:18beebf (T160638) [15:58:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [15:58:15] T160638: Deploy ORES late march - https://phabricator.wikimedia.org/T160638 [16:02:34] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3137379 (10Cmjohnson) To clarify labstore1001 and 2 arrays are going to rack B5 labstore1002 and all arrays are going to B6 Each h... [16:07:03] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3137404 (10madhuvishy) @Cmjohnson - One edit > labstore1002 and all arrays are going to B6 I assume that's really all other* arra... [16:08:18] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3137405 (10Cmjohnson) @madhuvishy that is correct ... labstore1001 and 2 arrays in B5 and labstore1002 and 2 arrays in B6 [16:33:27] Is there a way to connect a console direct to a labs instance? I currently have two instances that after reboot timeout when I try to ssh to them. It looks like I'm struggling with puppet certificates and somehow this is preventing them from starting everything up. They should be using my own puppetmaster (and were before reboot) but the boot up log (on [16:33:27] horizon) suggests after reboot they are now trying to connect to labs-puppetmaster-eqiad.wikimedia.org again. [16:36:12] tarrow: there isn't a user-facing console but I can have a look. What project and instances? [16:36:31] (of course puppet breakage shouldn't interfere with logins since those settings all persist after reboot) [16:38:16] andrewbogott: project: wikifactmine hostnames (instance): elasticsearch-20(69ea20c5-95ea-40fb-8493-9e9bc49ab07a), elasticsearch-21(e61de77a-5069-4866-ba9c-172770acba33) [16:38:32] thanks! [16:38:56] I can ping them but I can ssh in [16:39:42] puppet is fine on elasticsearch-20, using puppetmaster-01.wikifactmine.eqiad.wmflabs [16:40:08] tarrow: can you try to log in to elasticsearch-20 now? I'm watching the auth log [16:40:53] ok, I'm trying now [16:41:26] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#3137645 (10madhuvishy) [16:41:28] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3137642 (10madhuvishy) 05Open>03Resolved a:03madhuvishy Thanks @bd808. Since the original scope of this task is working db credentials for user MnemonicFlow, and that has been done and confirm... [16:41:34] tarrow: what os are you using there? [16:41:37] locally I mean [16:41:41] ubuntu [16:41:46] but I just managed to login [16:41:50] yeah, looks fine to me [16:41:55] so clearly I was doing something stupid [16:42:12] Thanks; sorry for wasting your time :( [16:42:31] well, it might've just taken a long time for them to reboot [16:42:34] is -21 working for you too? [16:42:59] yep; I guess that might be it [16:43:03] great [16:43:08] thanks again [16:44:28] andrewbogott: actually I think it was becasue I was trying to login from the tools bastion. Apparently this worked earlier but now the instances seem to be firewalled off [16:44:56] from the tools bastion, or via? [16:45:09] I wouldn't expect 'from' to work unless your keys are stored on the tools bastion, which seems bad [16:45:15] Oh, unless you use key forwarding I guess [16:45:23] I used forwarding and from [16:45:41] Last login: Mon Mar 27 14:55:43 2017 from tools-bastion-03.tools.eqiad.wmflabs [16:46:06] ah, ok [16:46:12] Dunno what changed there [16:46:59] but if I try that again (from tools-bastion-03) now it seems to be firewalled off. I just timeout on port 22 [16:48:05] I'd expect that to be something set in your specific project's security groups [16:48:49] the instances are using default. I guess perhaps for some unusual reason they aren't applied until reboot? [16:50:15] it shouldn't matter, that firewall doesn't live on the VMs. [16:50:25] But of course the VMs might have their own firewalls, if someone set them up. [16:51:12] 06Labs, 10Tool-Labs, 10DBA: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3137674 (10jcrespo) 05Open>03Resolved a:03jcrespo For now. [16:53:52] interesting, I've spawned a new test instance in the same security group (default) and I'm trying to connect from the tools bastion. It seems to be working but my ssh key hasn't yet been copied [16:55:37] yep; I just successfully logged on from tools-bastion-03 [16:56:02] hm, and after you reboot it? [16:56:09] just trying now [16:59:09] 10Tool-Labs-tools-Xtools: "not valid wiki" error when I check Commons' contributions in X!s Tools - https://phabricator.wikimedia.org/T161427#3137703 (10Matthewrbowker) p:05Triage>03Low a:03Matthewrbowker Thank you for the very specific report. I will provide a fix as part of the rewrite. [16:59:20] I don't know if it's on purpose, but you are running a firewall on -20 and -21. So most likely whatever's happening is happening there. [16:59:56] After reboot I can still log into my test instance from the tools bastion. I still can't log into the other two [17:00:08] hm.. [17:00:25] I guess it is coming from the elasticsearch puppet role [17:01:36] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3137733 (10MnemonicFlow) @bd808 Can I please know the temporary email address used? To prove that I am really the owner of the account, can you please email it to me at the email address provided i... [17:15:10] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jericho25 was created, changed by Jericho25 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Jericho25 edit summary: Created page with "{{Tools Access Request |Justification=Translation_Transition#jgaj I would like to publish a new project callled #Transition_Translation. This is a process wherein even a..." [17:22:22] 06Labs, 10Labs-Infrastructure, 10DNS, 06Discovery, and 3 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256#3137827 (10MaxSem) The only valid use for labs is WMF projects, and those don't support JS in IE under 9 (9 iS soon... [17:39:56] 06Labs, 10Tool-Labs: Cannot access replica databases - access denied - https://phabricator.wikimedia.org/T151296#3137858 (10bd808) @MnemonicFlow this is probably WP:BEANS, but anyone with shell access to a Labs host can read the LDAP directory and the email addresses for any account are right there in the dire... [18:12:12] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3138004 (10Cmjohnson) labstore1001 and 2 arrays are in B5 connected to ge-5/0/4 labstore1002 and 2 arrays are in B7 connected to ge... [18:38:12] 06Labs, 06DC-Ops, 06Operations: Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3) - https://phabricator.wikimedia.org/T158913#3138091 (10chasemp) labs-support vlan please :) [18:38:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rompert was created, changed by Rompert link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Rompert edit summary: Created page with "{{Tools Access Request |Justification=I want to contribute by making tools to fight spam by investigating ip-numbers and things like that. |Completed=false |User Name=Rompert }}" [19:12:34] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3138188 (10hashar) [19:14:38] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3138189 (10Andrew) [19:55:05] 06Labs, 10Labs-Infrastructure, 10DNS, 06Discovery, and 3 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256#3138272 (10grin) >>! In T161256#3137827, @MaxSem wrote: > The only valid use for labs is WMF projects, This is no... [20:17:57] 06Labs, 10Labs-Infrastructure, 10DNS, 06Discovery, and 3 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256#3126688 (10Peachey88) >>! In T161256#3138272, @grin wrote: > I would expect some background check from you before a... [21:04:13] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#488200 (10Multichill) The problem was solved. Re-opening an old task to get some new work done is probably not the best approach. See also... [21:08:01] 06Labs, 10Tool-Labs: toolserver-home-archive is using 52G on Tools - https://phabricator.wikimedia.org/T136202#3138418 (10Nemo_bis) Examples of big files inside: ``` tools.toolserver-home-archive@tools-bastion-03:~$ tar tvaf archive-2014-06-05.tar.xz | grep -E " [0-9]{8,} 20" -rw-r--r-- ant/users 114052... [21:08:21] 06Labs, 10Tool-Labs: toolserver-home-archive is using 52G on Tools - https://phabricator.wikimedia.org/T136202#3138419 (10Nemo_bis) p:05High>03Low [21:22:21] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:57:22] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fbgonzaga was modified, changed by Fbgonzaga link https://wikitech.wikimedia.org/w/index.php?diff=1753880 edit summary: [22:12:22] 10Labs-project-Wikistats: wikistats: add new wikipedias: kbp, khw, dty - https://phabricator.wikimedia.org/T160947#3138625 (10Dzahn) [22:23:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fbgonzaga was modified, changed by Fbgonzaga link https://wikitech.wikimedia.org/w/index.php?diff=1753893 edit summary: [22:25:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fbgonzaga was modified, changed by Fbgonzaga link https://wikitech.wikimedia.org/w/index.php?diff=1753894 edit summary: [23:00:08] Hi, I'm new here, and I've already submitted a Tools Access Request. But I saw that since march 10, no new requests were accepted. Is this information right? [23:00:16] I saw this here: [23:01:51] https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&q=[[Category%3ATools+Access+Requests]]&p=format%3Dbroadtable%2Flink%3Dall%2Fheaders%3Dshow%2Fmainlabel%3DRequest%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3F%3DRequest%23%0A%3FIs+Completed%0A%3FModification+date%0A&sort=Is+Completed%2CModification+date [23:16:10] 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3138818 (10bd808) [23:22:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fbgonzaga was modified, changed by Fbgonzaga link https://wikitech.wikimedia.org/w/index.php?diff=1753924 edit summary: [23:34:31] fbgonzaga hi, i belive one of the labs members do that. andrew*bogott chas*emp mad*huvishy (remove *) [23:37:26] 06Labs, 10Tool-Labs, 10wikitech.wikimedia.org, 15User-bd808: Replace SMW data on project instances with a tool using the OpenStack APIs - https://phabricator.wikimedia.org/T161662#3138905 (10bd808) [23:49:27] 06Labs, 10Tool-Labs, 10wikitech.wikimedia.org, 15User-bd808: Replace SMW data on project instances with a tool using the OpenStack APIs - https://phabricator.wikimedia.org/T161662#3138931 (10bd808) Removed admin, member, and instance list from `[[Template:Nova Project]]`: https://wikitech.wikimedia.org/w/i... [23:53:25] delete all the things