[00:56:23] 10Tool-Labs-tools-Other, 07Epic: [Epic] Toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#3039596 (10Liuxinyu970226) [00:56:36] 10Tool-Labs-tools-Other, 07Epic: [Epic] Toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#605796 (10Liuxinyu970226) per T60865#2490902 [01:32:19] /ns logout [02:04:21] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [06:52:24] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:32:24] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:43:05] ssh -i ~/.ssh/id_rsa_labs tr8d@login.tools.wmflabs.org --> Connection closed by 208.80.155.163 [07:43:10] any ideas ^ ? [11:33:12] 06Labs, 10wikitech.wikimedia.org: Decide which group(s) will be able to manage 'shell' on wikitech - https://phabricator.wikimedia.org/T158516#3040254 (10MarcoAurelio) Bureaucrat looks safe enough to me as well. I'll patch it. [11:40:34] 06Labs, 10wikitech.wikimedia.org: Allow 'contentadmin' being managed by wikitech bureaucrats - https://phabricator.wikimedia.org/T158554#3040298 (10MarcoAurelio) [12:16:32] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Remove shellmanagers group on wikitech - https://phabricator.wikimedia.org/T158482#3040343 (10MarcoAurelio) Patches submitted to remove the shellmanagers group and to grant the ability to add/remove the shell usergroup to Wikitech bureaucrats. [12:30:10] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Allow 'contentadmin' being managed by wikitech bureaucrats - https://phabricator.wikimedia.org/T158554#3040383 (10MarcoAurelio) p:05Triage>03Normal [12:31:07] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Allow 'contentadmin' being managed by wikitech bureaucrats - https://phabricator.wikimedia.org/T158554#3040398 (10MarcoAurelio) p:05Normal>03Low [12:31:15] 06Labs, 10wikitech.wikimedia.org: Remove 'shellmanagers' rights from all users before removing the group on Wikitech - https://phabricator.wikimedia.org/T158515#3040401 (10MarcoAurelio) p:05Normal>03Low [12:31:17] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Decide which group(s) will be able to manage 'shell' on wikitech - https://phabricator.wikimedia.org/T158516#3040400 (10MarcoAurelio) p:05Normal>03Low [12:31:19] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org, 13Patch-For-Review: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3040403 (10MarcoAurelio) p:05Normal>03Low [12:31:22] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Remove shellmanagers group on wikitech - https://phabricator.wikimedia.org/T158482#3040402 (10MarcoAurelio) p:05Normal>03Low [13:11:41] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Decide which group(s) will be able to manage 'shell' on wikitech - https://phabricator.wikimedia.org/T158516#3040529 (10scfc) I don't have a very strong opinion on this, but for me `bureaucrat` in the MediaWiki context roughly means "can give all users al... [13:56:09] 06Labs, 10Tool-Labs: Problems accessing replica.my.cnf in my Tool Labs repository - https://phabricator.wikimedia.org/T158568#3040680 (10AlvaroMolina) [13:58:59] 06Labs, 10Tool-Labs: Problems accessing replica.my.cnf in my Tool Labs repository - https://phabricator.wikimedia.org/T158568#3040716 (10AlvaroMolina) [14:29:36] Is there any particular problem with problem with logging in to an execution node on tools using qlogin to inspect something you're running on that node? Is there anything special I need to do to clean up after myself? [14:47:18] PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:54:15] PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:17:00] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3040978 (10Andrew) Emailed dschwen, aude, rillke, max on 2017-02-20. [15:18:40] (03PS2) 10Lokal Profil: Change DB engine to InnoDB [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/324396 (https://phabricator.wikimedia.org/T138517) [15:18:59] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3040979 (10Andrew) Emailed DannyB 2017-02-20 [15:20:03] (03CR) 10Lokal Profil: "Updated the commit message due to T138517#3040791" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/324396 (https://phabricator.wikimedia.org/T138517) (owner: 10Lokal Profil) [15:22:21] RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:54] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3040986 (10dschwen) I personally am not using this instance. It might be in use live for the OSM gadget on the german Wikipedia (does https://tiles.wmflabs.org/ point to that machine?) If nobody els... [15:27:16] 06Labs, 10Tool-Labs: Problems accessing replica.my.cnf in my Tool Labs repository - https://phabricator.wikimedia.org/T158568#3040680 (10scfc) The file `/data/project/alvaro/replica.my.cnf` is readable by the tool itself: ``` scfc@tools-bastion-03:~$ ls -l ~tools.alvaro/replica.my.cnf -r-------- 1 tools.alva... [15:29:13] RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:10] 06Labs, 10Tool-Labs: Problems accessing replica.my.cnf in my Tool Labs repository - https://phabricator.wikimedia.org/T158568#3041013 (10AlvaroMolina) >>! In T158568#3040987, @scfc wrote: > The file `/data/project/alvaro/replica.my.cnf` is readable by the tool itself: > > ``` > scfc@tools-bastion-03:~$ ls -l... [15:34:35] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041014 (10dschwen) Ok, there is an apache2 with modtile, a renderd, and a (small 34MB)_ postgres 9.1 database running on there. modtile and renderd are installed from `http://ppa.launchpad.net/kakr... [15:46:21] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041027 (10Andrew) I think step one is figuring out whether or not it's used by the rest of the maps project, or is fully vestigial? [15:47:09] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3041031 (10Andrew) I regard the fate of the CI instances as in @hashar's hands for now. The releng team will discuss this issue tomorrow and report back here with a plan. [15:53:42] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041034 (10dschwen) Hmmm, Yeah. I did `/etc/init.d/renderd stop` and `apache2ctl stop` and the OSM widget tile service on de.WP still functioned just as good/bad (lots of 404 at high zooms either way). [15:54:35] (03CR) 10Jean-Frédéric: [C: 031] "Still happy about that :) Do we need to do anything in particular before doing this? I suppose we need to drop all tables first − includin" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/324396 (https://phabricator.wikimedia.org/T138517) (owner: 10Lokal Profil) [15:56:35] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041035 (10dschwen) Looking closer at the apache2 config it looks liek this is a debug/experimental server ``` ServerName tile.mytileserver.org ServerAlias a.tile.mytileserver.org b.tile.my... [16:00:47] 06Labs, 10Tool-Labs: replica.my.cnf and .kube/config is not readable by the group members - https://phabricator.wikimedia.org/T158581#3041043 (10scfc) [16:03:31] 06Labs, 10Tool-Labs: replica.my.cnf and .kube/config is not readable by the group members - https://phabricator.wikimedia.org/T158581#3041062 (10scfc) [16:04:18] 06Labs, 10Tool-Labs: Problems accessing replica.my.cnf in my Tool Labs repository - https://phabricator.wikimedia.org/T158568#3041063 (10scfc) 05Open>03Resolved a:03scfc [16:24:30] (03CR) 10Lokal Profil: "Don't we drop monuments_all as part of the harvesting run? If so this should be deployable as is." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/324396 (https://phabricator.wikimedia.org/T138517) (owner: 10Lokal Profil) [16:24:49] 06Labs, 10Tool-Labs: replica.my.cnf and .kube/config is not readable by the group members - https://phabricator.wikimedia.org/T158581#3041082 (10zhuyifei1999) > only readable by the tool account itself ... but why should it be readable by the group? The credentials are supposed to be used by only the tool acc... [16:38:53] can someone please help me with a new Tool Labs login? It's immediately closing the connection, like I don't have a shell set or something [16:39:07] I type: $ ssh -i ~/.ssh/labs tr8d@login.tools.wmflabs.org [16:39:20] the response is: Connection closed by 208.80.155.163 [16:39:56] if I do something else like misspell the account name it says: Permission denied (publickey,hostbased). [16:40:11] so it's apparently logging me in but not running a shell? [16:41:36] tr8dr: have you been added to tool labs? [16:41:57] I think so [16:42:31] I did the first 6 steps at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Quick_start [16:42:38] do I need to do something else? [16:43:59] you can see Tim Landscheidt's welcome at https://wikitech.wikimedia.org/wiki/User_talk:Tr8dr [16:44:14] hmm ldap says you're part of tool labs, in https://tools.wmflabs.org/contact/search/tr8d [16:44:38] but: $ id tr8d [16:44:38] uid=16480(tr8d) gid=500(wikidev) groups=500(wikidev) [16:44:57] no project-tools, weird [16:46:21] when I was signing up it let me pick Tr8dr in OAuth and LDAP but on the shell account it said it was in use so I used Tr8d [16:47:26] maybe I should try adding the public key to shell account name tr8dr? [16:47:43] oh wait, you're using tr8d not tr8dr? [16:48:22] that shell user is groupless [16:48:53] I wanted tr8dr but it told me it was in use [16:49:05] clearly it alloated the uid immediately previously [16:50:03] according to https://tools.wmflabs.org/contact/search/tr8d the shell account named "tr8d" is owned by User:Tr8d, "tr8dr" is owned by User:Tr8dr [16:50:28] yeah I saw that [16:51:00] "Error saving OAuth credentials. [req id: 6d261b8aeda043c885b1e3981c6f0a1f]" [16:51:25] that https://wikitech.wikimedia.org/wiki/User_talk:Tr8dr is the message that the shell account "tr8dr" is okay to login to tool labs, but it says nothing about "tr8d" (which is owned by User:Tr8d) [16:51:49] "Error saving OAuth credentials." <= where's that? [16:52:06] "User account "Tr8d" is not registered." [16:52:17] when I tried to connect the LDAP account [16:52:46] okay, it's working now [16:52:54] which site are you connecting to? [16:53:23] I can log in to the tool labs shell now. I should ask that all the tr8d things get deleted [16:54:19] tr8dr: which site / page shows the message about connecting? "Error saving OAuth credentials."? [16:54:44] that was when I tried to connect the LDAP for Tr8rd to https://toolsadmin.wikimedia.org/profile/settings/accounts [16:54:55] bd808: ^ [16:56:07] bd808: fyi I think the ldap entry for "tr8d" is somehow broken. https://tools.wmflabs.org/contact/ points to invalid user at wikitech. [16:56:15] I think so too [16:56:50] it should work (see tgr's entry) [16:57:07] but I am okay for now, as all I need is the shell and I can get that now [16:57:14] thank you very much for your help [16:57:16] tr8dr: k [16:57:19] np [17:22:46] (03PS1) 10Jcrespo: Add labsdb root pass fake string to make puppet compiler work [labs/private] - 10https://gerrit.wikimedia.org/r/338800 (https://phabricator.wikimedia.org/T104900) [17:24:44] (03CR) 10Marostegui: [C: 031] Add labsdb root pass fake string to make puppet compiler work [labs/private] - 10https://gerrit.wikimedia.org/r/338800 (https://phabricator.wikimedia.org/T104900) (owner: 10Jcrespo) [17:25:26] (03CR) 10Jcrespo: [V: 032 C: 032] Add labsdb root pass fake string to make puppet compiler work [labs/private] - 10https://gerrit.wikimedia.org/r/338800 (https://phabricator.wikimedia.org/T104900) (owner: 10Jcrespo) [17:39:44] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#3041287 (10Andrew) [17:48:32] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Decide which group(s) will be able to manage 'shell' on wikitech - https://phabricator.wikimedia.org/T158516#3041302 (10MarcoAurelio) I don't have any issues with bureaucrats granting sysop as well. If this is desired, it only takes a couple of minutes fo... [17:57:37] (03PS4) 10Paladox: Add phabricator-upstream to #wikimedia-dev and #wikimedia-devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 [18:02:40] (03CR) 10Jean-Frédéric: [C: 031] "> Don't we drop monuments_all as part of the harvesting run?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/324396 (https://phabricator.wikimedia.org/T138517) (owner: 10Lokal Profil) [18:22:13] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Decide which group(s) will be able to manage 'shell' on wikitech - https://phabricator.wikimedia.org/T158516#3041384 (10bd808) I broadly agree with @scfc's analysis. I don't have an particular reason why `bureaucrat` should not be able to manage `sysop` m... [18:27:01] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Decide which group(s) will be able to manage 'shell' on wikitech - https://phabricator.wikimedia.org/T158516#3041393 (10MarcoAurelio) >>! In T158516#3041384, @bd808 wrote: > I think doing that would be good idea as well as making some local message overri... [18:28:42] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041396 (10Andrew) So... shut it down, and delete on 3/30 if no one complains? [18:41:17] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3013957 (10jcrespo) Is this related related or maybe dependent on these? T157359 (postgres osm servers?). [19:17:04] 06Labs, 10wikitech.wikimedia.org: Remove 'accountcreators' right on wikitech - https://phabricator.wikimedia.org/T158413#3041547 (10MarcoAurelio) @Reedy can you do some DB magic here? Thanks! [19:18:22] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:22:49] 06Labs, 10WikimediaMessages, 10wikitech.wikimedia.org, 13Patch-For-Review: Group messages for Wikitech - https://phabricator.wikimedia.org/T158417#3041556 (10Reedy) [19:22:51] 06Labs, 10wikitech.wikimedia.org: Remove 'accountcreators' right on wikitech - https://phabricator.wikimedia.org/T158413#3041553 (10Reedy) 05Open>03Resolved a:03Reedy ```lang=sql mysql:wikiadmin@silver [labswiki]> select * from user_groups where ug_group = 'accountcreators'; +---------+-----------------+... [19:35:15] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041560 (10dschwen) Yes, shut down now, delete later. [20:33:12] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:54] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041632 (10Andrew) @jcrespo, as I understand it this labs project doesn't interact with production Maps systems at all. It seems to be a fully volunteer (or at least wmde) run setup. [20:37:57] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3041639 (10Andrew) [20:39:00] 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2961338 (10Andrew) [20:39:03] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3041640 (10Andrew) 05Open>03Resolved @dschwen, fantastic. Thank you for investigating! [20:40:18] 06Labs, 10Continuous-Integration-Infrastructure, 06Operations, 10netops: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3041644 (10Paladox) [20:41:09] 06Labs, 10Continuous-Integration-Infrastructure, 06Operations, 10netops: git clone over EQIAD (wmflabs) CODFW timeout due to low bandwidth (~250 KiB/s) - https://phabricator.wikimedia.org/T158601#3041581 (10Paladox) Is this happening to any other repos? Should we set this as normal or high priority? [20:46:20] !log tools.admin did 'webservice restart', seems to have helped [20:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [20:48:04] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.038 second response time [21:28:22] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:42:28] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Remove shellmanagers group on wikitech - https://phabricator.wikimedia.org/T158482#3041694 (10Dereckson) p:05Low>03Normal Now we have a patch for that, we can prioritize it a little bit more. [21:42:53] I just learned about grid, and I can't figure out why my job won't run [22:46:13] 06Labs, 06Discovery, 06Maps: Update, replace or delete maps-tiles1 - https://phabricator.wikimedia.org/T157708#3013957 (10aude) not aware of this being used though not entirely sure. I think it was experimental at one point before we had production maps support. not sure if anything on German Wikipedia uses...