[01:36:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/FDMS was created, changed by FDMS link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/FDMS edit summary: Created page with "{{Tools Access Request |Justification=move http://t.fdms.eu/relgen.php to toollabs for https://commons.wikimedia.org/wiki/Commons:Email_templates as explained in https://commo..." [03:01:21] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2183516 (10jayvdb) Maybe an alternative is to block specific tools being accessed by zero, or kill the connection if the zero connection uploads more than a preset limit t... [03:18:26] 10Tool-Labs-tools-Other, 06WMF-Legal: Another request to review privacy policy and rules - https://phabricator.wikimedia.org/T104784#1426962 (10jayvdb) It appears as if the games logged in mode (oauth) is not yet enabled. I was able to 'play' to find one person so it is working, but maybe only using one image... [04:00:20] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2216742 (10DFoy) @BBlack @jayvdb My expectation (and current understanding) is that the IP ranges we provide to for zero whitelisting are only for the production servers,... [04:37:02] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2216775 (10BBlack) @DFoy - I think all of your assumptions are correct above. This really isn't a Zero-related problem. It's just that the abusive traffic happens to com... [04:40:46] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2216782 (10jayvdb) @DFoy , thanks for explaining that the 'zero' customer is being charged for using wmflabs. The "problem" is that these tools download data from other s... [05:04:03] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2216845 (10BBlack) I still think we're barking up the wrong tree here trying to identify them by their Zero rating. The Zero rating is irrelevant. They just happen to be... [05:10:25] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2216851 (10zhuyifei1999) 05Open>03Invalid I guess this is impossible then, given that the traffic to labs are not zero. >>! In T131934#2216782, @jayvdb wrote: > The p... [05:15:47] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2216856 (10zhuyifei1999) >>! In T131934#2216845, @BBlack wrote: > Can we not ban accounts? Are they creating a constant stream of new junk accounts? Please see https://co... [05:16:22] 06Labs, 10Tool-Labs, 06Operations, 10Traffic, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2216857 (10Nemo_bis) By "this" I assume you mean the list above? I'd like more comments on the methods proposed in the description. [05:42:33] 10Tool-Labs-tools-Other, 06Community-Tech, 07Category, 07Community-Wishlist-Survey: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2216877 (10kaldari) [06:45:07] webservice2 is broken [06:45:49] webservice2 uwsgi-python start [06:45:55] shows this error [06:46:03] Traceback (most recent call last): File "/usr/local/bin/webservice2", line 301, in main() File "/usr/local/bin/webservice2", line 249, in main server, release = get_service_defaults() File "/usr/local/bin/webservice2", line 182, in get_service_defaults if 'version' in manifest and manifest['version'] >= 2: TypeError: argument of type 'NoneType' is not iterable [06:46:59] http://kl1p.com/rNkZ [06:54:14] CKoerner_WMF: who can solve this error? now tools are down [07:32:20] 06Labs, 06Operations, 10ops-codfw: labtestneutron2001.codfw.wmnet does not appear to be reachable - https://phabricator.wikimedia.org/T132302#2216978 (10MoritzMuehlenhoff) I'd say let's either reimage it or drop it from site.pp until reimaged. [07:56:07] reza1615: can you file a bug in phabricator in the "tool-labs" project? [08:06:33] RECOVERY - Puppet run on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [08:16:30] 06Labs, 10Tool-Labs, 06Operations, 10Traffic, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2073537 (10valhallasw) > * make a list of tools.wmflabs.org URLs and test them all for unsecure resources with a simple URL fetching script; > * some sm... [08:31:33] 06Labs, 10Tool-Labs, 06Operations, 10Traffic, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2217117 (10Magnus) As a side note, I set up a VM for my PetScan tool: http://petscan.wmflabs.org/ This does not require http, but works for either, as... [08:36:11] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 06Operations: On deployment-prep, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#2217130 (10hashar) p:05High>03Low [08:36:26] 06Labs, 10Salt: Setup multimaster salt for large projects using salt-syndic - https://phabricator.wikimedia.org/T78466#2217131 (10hashar) [08:44:45] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2217140 (10hashar) [09:03:35] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2217197 (10hashar) [11:01:24] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2217482 (10jayvdb) >>! In T131934#2216851, @zhuyifei1999 wrote: > I guess this is impossible then, given that the traffic to labs are not zero. > >>>! In T131934#2216782,... [11:32:47] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2217591 (10zhuyifei1999) 05Invalid>03Open >>! In T131934#2217482, @jayvdb wrote: > During the tool's OAuth process, the user must log into MediaWiki.org, and while the... [11:38:00] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2217612 (10zhuyifei1999) On a side note, my tool is not the only one being abused. [[https://commons.wikimedia.org/w/index.php?title=Special:RecentChanges&tagfilter=OAuth+... [11:41:12] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2217626 (10jayvdb) [12:14:39] You've probably already seen this, but a friendly reminder just to be on the safe side: We won't be able to edit the wikis for about half an hour or so today because of some testing. [12:14:43] This starts in about two hours. https://meta.wikimedia.org/wiki/Tech/Server_switch_2016 [14:40:49] (03CR) 10Jean-Frédéric: [C: 032] Standardise SQL formatting [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281924 (owner: 10Lokal Profil) [14:42:10] !log authmanager updated core + extensions code; disabled centralauth [14:46:58] looks like morebots fell asleep [14:48:26] morebots, you there? [14:57:41] 10Tool-Labs-tools-Other, 06WMF-Legal: Another request to review privacy policy and rules - https://phabricator.wikimedia.org/T104784#2218077 (10Ricordisamoa) >>! In T104784#2216193, @ZhouZ wrote: > Hi @Ricordisamoa, is this game still in development? If so, I would be happy to discuss any potential legal issue... [14:59:31] (03Merged) 10jenkins-bot: Standardise SQL formatting [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281924 (owner: 10Lokal Profil) [15:10:12] labs-morebots: back? [15:10:13] I am a logbot running on tools-exec-1201. [15:10:13] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [15:10:14] To log a message, type !log . [15:10:34] !log authmanager tgr says: updated core + extensions code; disabled centralauth [15:10:38] thanks andrew! [15:10:53] hm... [15:12:38] !log fakeprojectname this should fail [15:12:39] fakeprojectname is not a valid project. [15:14:01] !log testlabs test log message [15:14:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL, dummy [15:14:15] tgr: looks like it's personal :( [15:14:32] !log authmanager tgr says: updated core and extensions code and disabled centralauth [15:14:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Authmanager/SAL, dummy [15:14:44] tgr: punctuation maybe? [15:16:07] andrewbogott: it borks if there is no header on the page [15:16:17] adding a random amount of ='s seems to help most of the time [15:16:40] valhallasw`cloud: why did it succeed on my third attempt? Did you/somone add a header? [15:16:46] hm, no [15:16:50] that /is/ weird [15:17:04] in any case, it just stays silent on exceptions, which is a bit silly [15:17:11] yeah [15:17:32] labs-morebots, you probably need a ground-up rewrite. [15:17:33] I am a logbot running on tools-exec-1201. [15:17:33] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [15:17:33] To log a message, type !log . [15:17:38] oh, yeah, on your second attempt you typed ' !log' instead of '!log' [15:18:13] hah, really? [15:18:31] Ok, so maybe the behavior is something like "The first log message for a new project is discarded and after that all is well" [15:42:51] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:44] Hi, I can't ssh in to a labs instance. Is this related to the server switch? [16:01:20] andrewbogott: ^ [16:01:45] Niharika: the server switch should not really affect labs. What instance and project? [16:01:52] And are you able to access other instances? [16:02:00] andrewbogott: commtech-1 on Commtech project. [16:02:20] andrewbogott: I don't have any other instances but I can ssh in to tool labs just fine. [16:02:30] This is the second time this instance has gone into limbo. [16:02:53] andrewbogott: Where can I find the link to restart the instance? I was expecting it to be here: https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech but it's not. [16:03:05] Niharika: 'manage instances' [16:03:06] or in horizon [16:03:53] andrewbogott: I did get to https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech from Manage instances. There's no reboot link anywhere. [16:04:07] the reboot link is ON the manage instances page [16:04:11] not the instance info page [16:04:29] if you are not projectadmin then you can't reboot things though [16:04:48] hm, you are though [16:04:57] andrewbogott: I am an admin but I still don't see the reboot link on the Manage Instances page. It just lists the project. [16:05:38] probably you need to log out and in again — there are some caching issues on wikitech. Or use horizon, which does not have this bug. [16:06:04] andrewbogott: Okay. I'll obtain the Totp token. [16:06:04] the log for that box is doing a lot of 'INFO: task init:1 blocked for more than 120 seconds.' [16:06:10] which means maybe a kernel lockup or smething [16:06:39] sorry, I don't remember much about that instance so I don't know the origins of the issue [16:06:51] It was fine yesterday and I don't think anybody attempted a login since then... [16:06:57] Okay. Thanks andrewbogott. [16:07:31] If this is happening a lot you might be better off rebuilding it [17:00:40] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:06:15] 06Labs: Put a firewall on labtestcontrol2001 - https://phabricator.wikimedia.org/T132598#2218729 (10Andrew) a:05Andrew>03MoritzMuehlenhoff [17:20:54] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [17:32:51] 10Tool-Labs-tools-Other, 06WMF-Legal: Another request to review privacy policy and rules - https://phabricator.wikimedia.org/T104784#2218995 (10ZhouZ) > In T104784#2216193, @ZhouZ wrote: > Hi @Ricordisamoa, is this game still in development? If so, I would be happy to discuss any potential legal issues with th... [17:35:45] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:28] my access request (https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/FDMS) has been open for 16 hours, can someone take a look at it (or tell me if there is something wrong with it)? [17:42:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/FDMS was modified, changed by Yuvipanda link https://wikitech.wikimedia.org/w/index.php?diff=447390 edit summary: [17:42:31] FDMS: done! thank you for your patience [17:42:49] that was fast – thanks! [17:43:05] only other one open is https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/LegoFan4000 [17:43:07] afaict [17:43:28] * YuviPanda eyes legoktm suspiciously [17:59:19] 06Labs, 07Tracking: Support instance manipulation, proxies, dns with Horizon (Quarterly goal tracking bug) - https://phabricator.wikimedia.org/T124181#2219119 (10Andrew) [17:59:21] 06Labs, 13Patch-For-Review: Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2219117 (10Andrew) 05Open>03Resolved a:03Andrew [17:59:28] 06Labs, 07Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#2219124 (10Andrew) [17:59:30] 06Labs, 07Tracking: Support instance manipulation, proxies, dns with Horizon (Quarterly goal tracking bug) - https://phabricator.wikimedia.org/T124181#2219122 (10Andrew) 05Open>03Resolved a:03Andrew [18:01:54] YuviPanda: not meeeeeee [18:04:51] YuviPanda: Hi, is that normal not being able to use "screen΅ on the toolabs? [18:05:00] chasemp: ^ [18:05:15] oh, wait. Kelson try 'script /dev/null' first then try? [18:05:22] chasemp: false alarm, ignore. [18:05:28] ok :) [18:05:45] YuviPanda: better :) [18:05:47] YuviPanda: thx [18:05:50] Kelson: yw [18:09:09] Kelson: although you probably shouldn't use screen as tool [18:09:20] valhallasw`cloud: alternative? [18:09:21] (submit jobs to the grid instead) [18:09:23] YuviPanda: can you live with https://gerrit.wikimedia.org/r/#/c/284103/ if it doesn't break puppet immediately? [18:09:47] valhallasw`cloud: hmm, not really the same usability [18:10:16] valhallasw`cloud: any concrete problem with using screen on the labs? [18:10:24] valhallasw`cloud: I don't want to break something [18:10:33] Kelson: you're not allowed to run long-running jobs on the bastions. [18:10:40] Kelson: but when we restart bastions for whatever reason, your bot will die and not come back. It also affects other peoples uses of the bastion. [18:11:14] YuviPanda: ok, got it, this is a test, not something I'm going to do on a regular base [18:11:23] i cannot login into horizon :( [18:11:28] was working about half an hour ago [18:11:30] Kelson: right, so it's ok to use it for testing but not for long periods of time. Usually you do it for a bit and then submit to grid. [18:11:55] I'll [18:12:11] Kelson: :) cool! [18:12:54] yurik: let me check... [18:12:57] what does it tell you when you try? [18:13:11] andrewbogott, no errors, simply goes back to the login card [18:13:23] andrewbogott, sorry - Unauthorized. Please try logging in again. [18:15:24] andrewbogott: left a comment on patch [18:15:56] yurik: works for me… try again and I'll see what's in the logs? [18:17:42] andrewbogott, just did [18:17:47] same err [18:18:12] Speaking of, did you figure out the case problem? [18:18:23] yurik: once more please? [18:18:49] done [18:19:07] yurik: what url are you hitting 'reload' on? [18:19:21] https://horizon.wikimedia.org/auth/login/?next=/identity/maps-team/detail/ [18:19:34] can you try just horizon.wikimedia.org? [18:20:02] andrewbogott, that worked :) [18:20:50] yurik: this is a bug that keeps popping up. Horizon invites you to follow a link and then when you follow it's all "DENIED! LOGGED OUT! DENIED" and then invites you to log in again just to punish you more [18:20:59] andrewbogott, but when i go to dns / projects [18:21:08] it kicks me out when i click on maps-team [18:21:09] yurik: can you open a bug with the specific process that got you there please? [18:21:14] sure [18:21:28] andrewbogott, does it mean i have no way to configure DNS at this point? [18:21:39] yurik: are you projectadmin in that project? [18:21:40] andrewbogott: That smells like an improperly terminated session. [18:21:44] i should be [18:22:06] Coren: I still don't know what's up with the case mismatch of your name, except that you probably just have a cursed ldap record. I guess a one-off fix might be best. [18:22:53] It may be because my account dates from the svn elder days - dunno how many people using Horizon for whom that it true. [18:24:22] can we not query to find other users with the same issue? [18:24:52] 06Labs: Unable to login to DNS horizon - https://phabricator.wikimedia.org/T133082#2219275 (10Yurik) [18:24:54] andrewbogott, ^ [18:25:06] yurik: thanks [18:25:13] yurik: although, 'login'? [18:25:25] Coren: Krenair, so the issue is having sn != uid right? [18:25:26] ?? [18:25:58] no andrewbogott [18:25:59] or is it having a cn start with lowercase? [18:26:37] yurik: I don't understand… what does identify/projects have to do with dns? [18:26:40] I have differing sn and uid [18:27:33] yurik: if you want to change your active project, use the project pulldown at the top, or else click the 'set active project' button on the right of the project list [18:27:35] Coren, what was your uid again? [18:27:39] andrewbogott: I think the cn-with-lowercase is the issue [18:27:40] 'marc' [18:27:51] uid=marc sn=coren cn=coren [18:28:05] right, it's coren instead of Coren [18:28:10] Krenair, Coren, think I can just change cn in ldap without causing a million repercussions elsewhere? [18:28:19] no [18:28:41] 06Labs: Horizon: Project links on the project panel often drop user into logged-out hell - https://phabricator.wikimedia.org/T133082#2219336 (10Andrew) [18:29:17] Krenair: what else will it break? I would think that services that don't do case-insensitive compare may already be broken for him... [18:29:19] andrewbogott: cn-renaming is the whole painful procedure in https://wikitech.wikimedia.org/wiki/Renaming_users [18:29:21] stuff like gerrit probably [18:29:21] * andrewbogott thinkgs about this a bit more [18:29:38] but wikitech apparently doesn't care about the capitalization? [18:29:41] andrewbogott, i was trying to change DNS, so i figured i will go into that. I just noticed that there is unexpanded DNS entry [18:29:51] valhallasw`cloud, mediawiki uppercases the first letter [18:29:56] but turns out DNS and Identity are two top level menu items [18:30:02] Krenair: yes, so I would expect it to be not able to find the record [18:30:14] it was a misclick on my part. Still, the question remains - what was it that i was trying to do :) [18:30:25] but wikitech seems to do a case-insensitive lookup [18:30:31] Coren, what username do you use to log in to wikitech? [18:30:35] (I can also login with 'Merlijn van deen' instead of 'Merlijn van Deen') [18:30:40] (lowercase or uppercase version?) [18:30:45] ah [18:31:02] * valhallasw`cloud tries gerrit [18:31:21] gerrit also doesn't seem to care [18:31:38] so maybe horizon should just also do a case-insensitive lookup? [18:32:39] gerrit> select full_name from accounts where full_name = 'Coren'; [18:32:39] full_name [18:32:39] --------- [18:32:40] coren [18:32:40] (1 row; 2 ms) [18:32:40] indeed [18:33:17] (notice uppercase in query, lowercase in result) [18:50:38] ok, so… Coren, do you have the time and attention to do some testing if I mess with your cn? [18:51:04] 10Tool-Labs-tools-Other, 10DBA: Killed very long transaction that was blocking the replica on labsdb1001 - https://phabricator.wikimedia.org/T133086#2219430 (10Volans) [18:58:08] andrewbogott: Attention is spotty, but sure. [19:00:01] And yes, none of Gerrit, Phab nor Wikitech have issues with case. [19:00:25] Ah, nevermind phab since it auths against SUL not LDAP [19:00:25] Coren: ok, done — want to see what broke and what unbroke? [19:00:33] * Coren tries stuff. [19:00:59] Aw, poo. My phone is drained. No 2fa. I'll test once it wakes. [19:02:37] ok [19:05:00] 06Labs, 06Operations: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#2219505 (10RobH) [19:23:58] 06Labs, 10Tool-Labs: Web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2219634 (10Nettrom) [19:33:12] (03PS1) 10Yurik: added #wikimedia-interactive [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/284270 [19:34:12] could someone +2 and refresh the bot pls? ^ [19:34:15] thx :) [19:41:27] 06Labs, 10Tool-Labs: signpostlab and telegrambot webservices flapping (registering/deregistering) - https://phabricator.wikimedia.org/T133092#2219725 (10valhallasw) [19:45:11] 06Labs, 10Tool-Labs: Web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2219634 (10valhallasw) Thanks for creating this ticket, Morten. The 20 minute period is really weird to me -- we've seen issues with a race condition where, when a webservice restarts, the new webservice f... [19:46:08] Kelson: are you currently running things as tools.enwp10? Please don't run I/O heavy commands on tools-bastion. [19:54:37] 06Labs, 10Tool-Labs: signpostlab and telegrambot webservices flapping (registering/deregistering) - https://phabricator.wikimedia.org/T133092#2219832 (10valhallasw) signpostlab: uwsgi.log: ``` *** Operational MODE: preforking *** mounting /data/project/signpostlab/www/python/src/app.py on /signpostlab Traceba... [19:55:28] (03CR) 10Merlijn van Deen: [C: 032] added #wikimedia-interactive [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/284270 (owner: 10Yurik) [19:55:55] YuviPanda: time to try the new docs! [19:56:02] valhallasw`cloud: \o/ [19:57:31] YuviPanda: should I not git pull? [19:57:46] valhallasw`cloud: yeah, you should if you want latest changes. [19:58:38] yes, seems to be doing stuff now [20:00:32] You cannot push a "root" repository. Please rename your repository to / (ex: valhallasw/docker-registry.tools.wmflabs.org) [20:01:13] (03PS1) 10Yurik: added jsonconfig & maps to interactive [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/284280 [20:01:25] * valhallasw`cloud groans [20:01:57] forgot to add maps & json config :( ^ [20:02:17] (03CR) 10Merlijn van Deen: [C: 04-1] "maps should probably also still go to #wikimedia-dev ?" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/284280 (owner: 10Yurik) [20:02:59] Merlijn van Deen, what's your IRC ? [20:03:06] * valhallasw`cloud . [20:03:14] valhallasw`cloud: Did I document it to be docker-registry.tools.wmflabs.org/grrrit as tag? [20:03:15] valhallasw`cloud, ah [20:03:16] yes :) [20:03:42] valhallasw`cloud, does adding it to maps/.* removes it from all others? [20:03:56] yurik: it removes it from the default channel = wikimedia-dev [20:04:09] so how should it be added? [20:04:10] YuviPanda: let me try... [20:04:20] add maps/.* to wikimedia-dev as well :-) [20:04:25] hmm, ok [20:04:45] valhallasw`cloud: removing it from wikimedia-dev seems appropriate tho. I don't know if maps/ should go to -dev. but then again, I'm no longer there due to immense botspam, so I probably shouldn't talk :P [20:04:55] hehe [20:05:04] yurik is probably the best judge on this :-) [20:05:23] assuming he's in -dev. My vague rule is if it isn't PHP mw extension it shouldn't be in -dev [20:05:35] for example, the android / iOS apps show up in -dev. that makes 0 sense [20:05:36] i'm in everything :) [20:05:48] anything maps/ is basically services [20:05:51] YuviPanda: ok, let's do it as-is then :-) [20:05:52] nodejs [20:06:03] valhallasw`cloud: :D [20:06:08] (03CR) 10Merlijn van Deen: [C: 032] "After discussing on IRC, this is probably OK." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/284280 (owner: 10Yurik) [20:06:12] Head https://docker-registry.tools.wmflabs.org/v2/grrrit/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4: no basic auth credentials [20:06:13] baaaah [20:06:26] thx :) [20:06:27] * valhallasw`cloud sudo su yuvipanda and tries again [20:06:36] valhallasw`cloud: sudo su to root and try? [20:06:38] * valhallasw`cloud fails again [20:06:42] valhallasw`cloud: echo $HOME? [20:06:44] yeah that works [20:06:54] valhallasw`cloud: yeah, should be root. root has credentials put in place by puppet. [20:07:58] * valhallasw`cloud prods jenkins [20:08:27] " labs-morebots, you probably need a ground-up rewrite." -- stashbot only needs to be taught how to write to the on-wiki SAL and the twitter feed at this point :) [20:08:43] and that can mostly be done by copy-n-paste from morebots [20:09:10] (03CR) 10Merlijn van Deen: [V: 032] "blah." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/284280 (owner: 10Yurik) [20:09:27] * andrewbogott makes a 'be my guest!' hand gesture [20:09:46] it's the "what do I work on next" problem [20:10:13] I need to get stashbot into k8s and a bit more stable too [20:10:43] it's had uptime problems on the grid lately [20:10:47] what's the advantage of k8s? [20:11:03] Luke081515: mostly that there is no NFS dependency [20:11:08] https://lists.wikimedia.org/pipermail/labs-announce/2015-September/000071.html [20:11:27] ah [20:11:29] thx [20:12:19] Or I guess more correctly that it would be possible to avoid an NFS dependency [20:12:20] is there a documentation, what I have to do, if I want to use it? [20:12:23] YuviPanda: sudo su is always a source of issues; use sudo -i or sudo -s instead. [20:12:36] * valhallasw`cloud sudo -H'ed [20:12:52] Luke081515: not yet, no. We are working on migration procedures. [20:12:55] in this case I specifically want to be root and expunge all of my own personality from the shell [20:12:56] so many ways to rome [20:13:00] ok [20:13:02] that would be sudo -i [20:13:10] ah ok! [20:13:12] valhallasw`cloud: -H is a sort of odd half-way session setup. [20:13:21] (-i = login shell, -H = 'just set $HOME') [20:13:39] also -H comes before -i so it was the first thing I found [20:13:47] * Coren chuckles. [20:13:50] yurik: should work now! [20:13:55] `sudo su` should tell you something like "Dude! we use sudo for a reason" [20:14:13] YuviPanda: ok, docs updated! [20:14:20] valhallasw`cloud, thanks! [20:14:50] valhallasw`cloud: \o/ awesome :) [20:15:36] YuviPanda: although I have no clue how to check whether the right docker image has been loaded [20:15:50] but that'll probably be available in some web interface at some point [20:15:58] kubectl is teh horror [20:16:57] sudo runas root su - [20:17:00] valhallasw`cloud: the way we use kubctl is a bit not ideal. we specify 'latest' when in reality we should specify a image version directly. then kubectl describe will show you [20:17:24] valhallasw`cloud: yeah, https://github.com/kubernetes/dashboard. We can probably setup an admin-only version of that if you think that'll be useful [20:17:24] yeah, describe shows me a container id and an image id [20:17:35] I suppose the latter is the docker image [20:17:41] valhallasw`cloud: right. so we can use that to verify, and using 'latest' is specifically an antipattern. [20:17:45] valhallasw`cloud: yup [20:17:56] it's not what I pushed, though [20:18:04] I pushed sha256:571ef67d42552e5e7f522dbf08fd5d8614e202191fd679d9cf30078b588ff5ea [20:18:10] and it reports docker://0d142ebc223555a6fadaebab067a108ebaa76c7a7e54e418b94db3c49c961c14 [20:18:17] fun fun fun. [20:18:32] YuviPanda: I think a web interface would be super useful, yes [20:18:32] hmm, I see [20:18:57] valhallasw`cloud: we probably can't open it up to everyone, but am pretty sure we can setup an admin only one. Want to file a bug? [20:19:33] "It is likely that Dashboard is already installed on your cluster." [20:19:51] valhallasw`cloud: if you're on Google Container Engine or somesuch. not true for us. [20:20:13] k [20:20:33] YuviPanda: shouldn't we have a kubernetes or tools-kubernetes? project? [20:20:49] valhallasw`cloud: like, labs project? [20:20:56] phab project [20:20:57] in phab [20:21:01] valhallasw`cloud: aah, yes we should. [20:21:03] k8s tag? [20:21:06] yeah, k8s [20:21:29] I'm fine with k8s if we are never going to call it kubernetes again [20:21:34] but we are [20:21:49] I don't personally mind either :) [20:21:51] because kay-eight-es is not really pronounceable :-D [20:21:52] yeah we can make both work [20:22:07] 06Labs, 10Tool-Labs: Set up (admin-only for now) kubernetes dashboard - https://phabricator.wikimedia.org/T133098#2220003 (10valhallasw) [20:22:11] should it be a tag or a project? [20:22:29] * bd808 has phabe create rights and will JFDI [20:23:10] I'd kind of lean towards a tag since k8s is headed to prod too [20:23:18] bd808: Beware waking the sleeping Andre! :-) [20:23:45] I'd rather have a project tools-kubernetes and production-kubernetes as they will probably be reasonably distinct [20:23:59] and a 'tools-and-prod k8s' workboard doesn't really make sense [20:24:01] reasonable [20:24:36] 06Labs, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2220032 (10valhallasw) 05Open>03Resolved a:03valhallasw It's still limited to people with access to the k8s cluster, but the steps are now (reasonably) well-documented at https://wikitech.wikime... [20:25:15] * valhallasw`cloud is off again [20:25:19] * valhallasw`cloud bows and waves [20:28:07] valhallasw`cloud: BTW, ReleaseTaggerBot looks like it's died? [20:28:25] * valhallasw`cloud reappears [20:28:27] Oh, wait, no, it came back. Never mind. [20:28:39] No no no, I was right first time. [20:28:41] valhallasw`cloud: https://phabricator.wikimedia.org/p/ReleaseTaggerBot/ [20:28:53] valhallasw`cloud: No actions since last Wednesday. Was confusing dates in my head. [20:28:59] I remember doing something with RTB at some point [20:29:05] * valhallasw`cloud checks SAL [20:29:21] YuviPanda, valhallasw`cloud: https://phabricator.wikimedia.org/project/view/1957/ [20:29:31] bd808: \o/ thanks :D [20:29:42] It now uses a fancy new logging system which isn't available unless I ssh in and unfortunately I broke my ssh set-up for Labs a while ago. :-( [20:30:00] Ah, right, I changed the logging during the hackathon [20:30:06] Yup. [20:30:11] But it worked for a bit. [20:30:15] I think this is unrelated. [20:30:26] (Also, sorry for grabbing you as you were leaving. :-)) [20:32:23] so RTB thinks it added stuff later than the 16th [20:34:10] James_F: it seems it's re-processing the same emails over and over again [20:34:23] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Set up (admin-only for now) kubernetes dashboard - https://phabricator.wikimedia.org/T133098#2220072 (10bd808) [20:34:48] It does that if it errors in the run and doesn't delete them (which is a feature), right? Is it erroring or just failing to delete? [20:35:05] Yes, but I don't see any exceptions [20:35:20] ah, I do [20:35:21] Exception: No PHID found for slug #mw1.27.0-wmf.21! [20:36:15] Umm. [20:36:27] https://phabricator.wikimedia.org/project/edit/1890/ is missing that hashtag [20:36:36] Aha. [20:36:37] Fixed. [20:36:54] so, yes, it's a feature ;-D [20:37:03] Well, except. [20:37:18] but maybe it should have another feature where it actually reports it to someone when it errors out [20:38:08] The code for "this task is merged into master which will be a future branch" looks for 'mw1.27.0-wmf21' and the code for "this task is merged into a specific branch" looks for 'mw1.27.0-wmf.21'. [20:38:14] But eh. [20:40:04] my brain is currently incapable of parsing forrestbot.py [20:40:05] 06Labs, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2220110 (10Krenair) 05Resolved>03Open a:05valhallasw>03None >>! In T132828#2220032, @valhallasw wrote: > It's still limited to people with access to the k8s cluster then the task is not actua... [20:42:20] valhallasw`cloud: No-one's brain is capable of that. Don't worry. :-) [20:44:44] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2220142 (10bd808) [20:45:00] valhallasw`cloud: Anyway, thank you, you're awesome, I'm a fool. :-) [20:45:10] James_F: you're welcome :-) [20:50:00] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2220162 (10yuvipanda) I've edited the page to mention that only tools roots have access to it for now. This will change once we make *direct* k8s access more widely available, bu... [20:56:44] (03PS95) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [21:03:33] (03CR) 10Ricordisamoa: "PS95 removes DraggableElement.prototype.getValidity, the only getValidity() call comes from the new EditableDraggableElement.prototype.get" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [21:11:10] (03PS96) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [21:12:05] 06Labs, 10Horizon: Horizon dashboard for managing instance puppet config - https://phabricator.wikimedia.org/T91990#2220363 (10Andrew) [21:12:55] 06Labs, 10Horizon: Horizon dashboard for managing instance puppet config - https://phabricator.wikimedia.org/T91990#2220377 (10Andrew) [21:19:39] (03CR) 10Ricordisamoa: "PS96 adds documentation to some private DraggableElement methods" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [21:21:42] (03PS1) 10Luke081515: Implement hook "kick" [labs/tools/Luke081515IRCBot] - 10https://gerrit.wikimedia.org/r/284355 [21:22:00] (03CR) 10jenkins-bot: [V: 04-1] Implement hook "kick" [labs/tools/Luke081515IRCBot] - 10https://gerrit.wikimedia.org/r/284355 (owner: 10Luke081515) [21:23:56] (03PS2) 10Luke081515: Implement hook "kick" [labs/tools/Luke081515IRCBot] - 10https://gerrit.wikimedia.org/r/284355 [21:26:50] 06Labs, 10Horizon: Horizon dashboard for managing instance puppet config - https://phabricator.wikimedia.org/T91990#1100588 (10bd808) I think generally speaking there are two related but different use cases for Puppet management of Labs instances: # "I want to make a Labs instance that does well defined task X... [21:32:56] (03CR) 10Luke081515: [C: 031 V: 031] "Works." [labs/tools/Luke081515IRCBot] - 10https://gerrit.wikimedia.org/r/284355 (owner: 10Luke081515) [21:38:06] (03CR) 10Luke081515: [C: 032] Implement hook "kick" [labs/tools/Luke081515IRCBot] - 10https://gerrit.wikimedia.org/r/284355 (owner: 10Luke081515) [21:41:14] 06Labs, 10Monitoring, 06Operations, 10wikitech.wikimedia.org: Bacula recovery of sql files from silver/wikitech fails - https://phabricator.wikimedia.org/T131195#2220548 (10akosiaris) So restoring on different clients in not really possible without some mambo jumbo first. Namely http://www.bacula.org/5.2.x... [21:41:30] (03Merged) 10jenkins-bot: Implement hook "kick" [labs/tools/Luke081515IRCBot] - 10https://gerrit.wikimedia.org/r/284355 (owner: 10Luke081515) [21:47:43] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2220571 (10BBlack) I get that the current abusers happen to be on a Zero network, but since their traffic into these tools isn't actually Zero-rated, that's merely a coinc... [22:03:57] James_F: Now ReleaseTaggerBot is active again, he spam my notifications :) [22:04:10] a botflag for phabricator maybe a good idea [22:04:36] Luke081515: You're welcome. ;-) [22:04:50] Luke081515: I just don't read my Phab e-mail. Makes things easier. :-) [22:04:56] hi [22:05:22] James_F: I'm using the notifications, and mark > 100 via one click as read [22:05:27] yannf: hi [22:05:43] yannf: I saw your access request ;). Problems? [22:05:54] I am confused [22:06:15] yannf: I guess you just want shell to labs? [22:06:17] https://phabricator.wikimedia.org/T133113 this is not needed [22:06:20] yes [22:06:21] Luke081515: Gosh, the notifications work for you? [22:06:42] James_F: Wait a moment, then I can tell you how to activate them, or are they already active? ^^ [22:06:54] https://wikitech.wikimedia.org/wiki/Requesting_shell_access <- confused [22:06:54] yannf: For general shell access create an account here: https://wikitech.wikimedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page [22:07:14] Luke081515: I get random numbers of notifications sometimes changing but they don't reflect anything. [22:07:16] yannf: > This is not for access to Git/Gerrit or Wikimedia Labs but you will need to already have a Labs account to continue. [22:07:20] then you got access to labs. If you need access to tool labs, ask me, after the acocunt is created ;) [22:07:23]