[00:26:03] 06Labs, 10Tool-Labs, 13Patch-For-Review: Investigate Tool Labs webservice outage on 2016-05-25 - https://phabricator.wikimedia.org/T136162#2325315 (10scfc) I don't think this deserves major action, and T136168 can't prevent it. If software needs to be updated on multiple hosts, regardless of any deployment... [00:28:38] 06Labs, 10Tool-Labs: Switch toollabs-webservice to be deployed with an actual deployment mechanism - https://phabricator.wikimedia.org/T136168#2325488 (10scfc) >>! In T136162#2328765, @scfc wrote: > I don't think this deserves major action, and T136168 can't prevent it. If software needs to be updated on mult... [00:34:26] 06Labs, 10Tool-Labs: Unmount unneeded NFS mounts from tool labs hosts - https://phabricator.wikimedia.org/T136222#2327313 (10scfc) I think implementing T120165/T136080 might bring this forward a long way (IIUIC). If the variables that enable the NFS mounts were in Hiera, and those Hiera variables could be rol... [00:40:42] 06Labs, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#1847021 (10Dzahn) https://wikitech.wikimedia.org/wiki/Puppet_Hiera#Role-based_lookup It's that "the new parser function/keyword, called role" is something that we (Joe) made ourself and do... [01:04:55] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [04:49:15] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328954 (10bd808) [05:01:09] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing PaaS solutions - https://phabricator.wikimedia.org/T136265#2328978 (10bd808) [05:01:31] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328954 (10bd808) Step one is to follow the lead of {T106475} and develop evaluation criteria. What features are must haves? What are... [05:02:06] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328995 (10bd808) [05:04:26] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328997 (10bd808) Step two involves evaluation, so we need to come up with a list of possible candidates. At the moment I'm... [05:04:29] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing PaaS solutions - https://phabricator.wikimedia.org/T136265#2328978 (10yuvipanda) I would also suggest spending some time playing with Heroku / Google App Engine to get a sense of the PaaS developer... [05:08:32] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing PaaS solutions - https://phabricator.wikimedia.org/T136265#2329001 (10bd808) {T128158} includes some discussion of pain points and accidental complexity in the current workflows that may be useful... [05:43:46] 10Quarry: feature for quarry tool: show quarry time used - https://phabricator.wikimedia.org/T136266#2329010 (10Yamaha5) [05:58:43] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [06:36:11] 10PAWS: I can not write some special characters in PAWS - https://phabricator.wikimedia.org/T136118#2329048 (10Framawiki) a:05Framawiki>03None Ok [07:17:20] 06Labs, 10Tool-Labs: wikidata-analysis is using 153G on Tools - https://phabricator.wikimedia.org/T136211#2329095 (10Lucie) I'd suggest to backup the .pngs of the maps (with the date they were created) somewhere but assume the rest could be deleted. The images should all be in the public_html folders. Want on... [08:08:47] RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [08:34:52] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:45:18] 06Labs, 10Tool-Labs, 07Tracking: Contact tool maintainters using large amounts of disk space (tracking) - https://phabricator.wikimedia.org/T136212#2329276 (10Addshore) [08:45:21] 06Labs, 10Tool-Labs: wikidata-analysis is using 153G on Tools - https://phabricator.wikimedia.org/T136211#2329274 (10Addshore) 05Open>03Resolved This should now be all cleaned up! Most of the data was from backups of old working code (which also then included some dump files in the directories) [09:15:45] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329337 (10jcrespo) I have converted labswiki to `CHARSET=utf8 COLLATE=utf8_bin`. However, I am skeptical about this working because: * The whole database uses latin1 charset/collation, which means this only... [09:22:37] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329343 (10jcrespo) ``` jynus: I still get "An error occurred authenticating. Please try again later ``` ``` MariaDB WIKITECH localhost labswiki > SELECT * FROM user WHERE user_name = 'Tgr'; Empty set... [09:32:15] one probably stupid question. Is the puppetmaster of labs (not beta) up-to-date with origin/production or people cherry-picked commits into it [09:32:30] nvm [09:32:40] I was looking into to wrong instance [09:32:42] :D [09:40:37] 06Labs, 10Tool-Labs: wikidata-exports is using 256G in Tools - https://phabricator.wikimedia.org/T136194#2326402 (10Addshore) Yeh I think sending these to the Internet Archive would make sense. The code that I have written to use the JSON dumps based on the Wikidata Toolkit actually uses archive.org as a sourc... [10:02:20] !log ores deploying 4dc6050 into web and worker nodes, updating to revscoring 1.2.6 [10:02:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [10:05:14] !log ores rebooting ores-web-04 [10:05:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [10:16:54] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329456 (10Tgr) My shell name is `tgr` but my username should be `Gergő Tisza` (assuming this is the MediaWiki user table). [10:20:31] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329458 (10jcrespo) oh, I can see it now- I assumed Tgr or tgr was the user_name and Gergő Tisza the real name- I can see now that it is "corrupt": ``` 392 | GergÅ‘ Tisza | GergÅ‘ Tisza ``` Let me... [10:23:52] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329465 (10jcrespo) Maybe you can try one more time? ``` MariaDB WIKITECH localhost labswiki > SELECT * FROM user WHERE user_id=392; +---------+--------------+----------------+---------------+---------------... [10:30:10] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329475 (10Krenair) Can we identify all other now-broken users? [10:34:56] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329478 (10jcrespo) > Can we identify all other now-broken users? Difficult, in an unreliable way- maybe running a script that detects typical utf8->latin1 character errors. I have not yet 1''% clear that t... [10:38:30] 06Labs, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#2329486 (10hashar) [10:38:51] 06Labs, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#1847021 (10hashar) [10:39:44] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#1847021 (10hashar) [10:44:53] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet: Implement role based hiera lookups for labs - https://phabricator.wikimedia.org/T120165#2329534 (10Joe) The role keyword is used in production to refer to large groups of hosts; we DEFINITELY don't want to have role lookups in labs for the same... [10:45:22] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329536 (10jcrespo) I have created a list of users here with a very simplistic list of non-ASCII characters: {P3180} As I am not sure if listing users from the database is private data, I have for now put i... [10:51:56] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329540 (10Tgr) I can log in now. Thanks @jcrespo! > Can we identify all other now-broken users? Just need to do something like `UPDATE user SET user_name = CONVERT(CONVERT(CONVERT(user_name USING latin1) U... [10:56:42] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329549 (10jcrespo) See my above list, I used `^[a-zA-Z0-9 \-\.\=]*$` and that probably had a low false positive rate, given the wiki doesn't have a lot of users. [11:00:05] (03PS1) 10Gergő Tisza: Do not report Media Viewer patches to #wikimedia-multimedia [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290914 [11:12:38] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329575 (10jcrespo) This is the result of applying Tgr's convertion back and forth to fix the latinized utf8 characters. I would really want 1 or more reviewers, as it could have a really bad impact for users... [11:19:11] 10Quarry: show time of execution in quarry - https://phabricator.wikimedia.org/T136266#2329589 (10Ladsgroup) [11:25:57] !log librarybase updated to 1.27 abd broke it [11:26:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Librarybase/SAL, Master [11:36:53] !log librarybase fixed botched config and finished updating to 1.27 [11:36:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Librarybase/SAL, Master [12:01:39] 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329625 (10Tgr) The extended latin ones are correct. The Japanese/Chinese/Arabian/Hindi/Russian ones look sane (I can't read those alphabets, but they don't have obvious garbage characters). The ones that lo... [12:03:48] 06Labs, 10DBA, 10Horizon: Tgr unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329639 (10Nemo_bis) [12:15:52] (03CR) 10Merlijn van Deen: [C: 032] Do not report Media Viewer patches to #wikimedia-multimedia [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290914 (owner: 10Gergő Tisza) [12:22:33] 06Labs, 10DBA, 10Horizon: Tgr unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329673 (10jcrespo) As I mentioned, MySQL's utf8 is not true UTF-8, maybe it is safer to use real utf8 for characters beyond the BMP? Or binary directy, as we do with the main wikis. [12:28:46] 06Labs, 10DBA, 10Horizon: Tgr unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329691 (10jcrespo) As 3 people have already said this looks sane "eg better than what we have now", I will do this now, and maybe we can reopen if some issues arise soon. Given the limited scope of labswiki... [12:33:10] 06Labs, 10DBA, 10Horizon: Tgr unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2329703 (10jcrespo) ``` UPDATE user SET user_name = CONVERT(CONVERT(CONVERT(user_name USING latin1) USING binary) USING utf8) WHERE user_id != 392 and user_name NOT RLIKE '^[a-zA-Z0-9 \-\.\=]*$' ORDER BY user... [12:43:41] jynus: I'm running a bot to message all potentially affected users on their talk page [12:44:07] I was going to send an email to wikitech [12:44:53] but if they could not logging before, I am not sure if the talk ping would be that effective [12:45:15] does wikitech send emails by default? [12:45:21] I think so, but I'm not 100% sure [12:46:06] let me check some ongoing issues on production, I will go back to wikitech later [12:48:34] (03CR) 10Merlijn van Deen: [V: 032] "BOO!" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290914 (owner: 10Gergő Tisza) [12:54:03] !log tools.lolrrit-wm deployed https://gerrit.wikimedia.org/r/290914 [12:54:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [13:06:46] 06Labs, 10DBA, 10Horizon: Tgr unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2173147 (10valhallasw) I have added a message to the talk page of all affected users (except the obvious vandal ones). [13:32:52] (03CR) 10Bartosz Dziewoński: "Thanks!" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290914 (owner: 10Gergő Tisza) [13:34:59] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2329954 (10Krenair) > Future services: > Nova api Does it need to be in the labs support network then? > What do we call this thing? "Striker" is... [13:54:29] 06Labs, 10DBA, 10Horizon: Tgr unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2330085 (10jcrespo) Thank you, @valhallasw. Please report if some user contacts you back saying that they can now log it / they still cannot log in. [14:10:34] Hi, how can I connect from one labs instance to another, from an non LDAP user acc? e.g. I want to allow a user, which is used by a script (exists only at this instance, not LDAP) to connect to another instance [14:12:52] Luke081515_: this is tricky atm, I believe it requires a entry for access.conf (.d style) allow the user as native rules say only project members and that is stored in ldap. is this a project w/ a self hosted master? you could do that, or you could put up a changeset w/ a role for this and we could apply it via normal puppet labs master [14:13:06] or you could hack things up, depending on if this is a long lived thing or some kind of short term PoC [14:13:40] I think it is more a long lived thing [14:14:03] what is the project / purpose / is there puppetization? [14:14:24] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing PaaS solutions - https://phabricator.wikimedia.org/T136265#2328978 (10scfc) No criteria, but Fedora ships OpenShift Origin with its current (24) beta. I haven't tested it, but my understanding is... [14:14:48] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-elapsed-time/en]] needs PLURAL - https://phabricator.wikimedia.org/T136300#2330166 (10Nemo_bis) a:03MusikAnimal [14:14:54] puppetization no. I currently have one instance at this project for execution of code, and want to use one of the other instances for this too, because it is actually used by a proxy, but has a lot of free ressources I want to use [14:17:44] chasemp: what do I have to add at access.conf? It's my project, I have root at that instances [14:17:57] yes but it will be overriden by puppet [14:18:05] and not running puppet for a long lived thing is basically impossible [14:18:15] you want this setup like we have for mwdeploy in beta? [14:18:20] pull down ops/puppet and look at modules/security/manifests/access/config.pp [14:18:33] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-num-pages/en]] needs PLURAL - https://phabricator.wikimedia.org/T136299#2330184 (10Nemo_bis) a:03MusikAnimal [14:18:46] my advice regardless of ^ is to puppetize this, at least the basic setup if you want it last long term [14:18:58] Krenair: what is the advantage of this method? [14:18:58] hm [14:19:01] because most of our model is built around it like this case where puppet is canonical [14:20:02] if you are doing something complex enough to have services triggering events across hosts [14:20:15] it's going to be a very rainy day if these instances corrupt and you lose a lot of adhoc work and reasoning [14:20:25] and I mean that in teh best way, I've had it happen and it sucks man [14:20:52] or blow that one away and spin up the largest instance your quota allows :) idk [14:25:26] 06Labs, 13Patch-For-Review: Unpuppetized database things on labservices* - https://phabricator.wikimedia.org/T136065#2321843 (10Krenair) Aren't we going to install the mysql package using puppet? [14:34:17] 06Labs, 13Patch-For-Review: Unpuppetized database things on labservices* - https://phabricator.wikimedia.org/T136065#2330219 (10Andrew) @krenair, that's what package { 'mysql-client' is for... unless you meant something else? [14:36:27] 06Labs, 13Patch-For-Review: Unpuppetized database things on labservices* - https://phabricator.wikimedia.org/T136065#2330222 (10Krenair) Right... Somehow I missed that, sorry. [14:37:12] 06Labs, 06Operations, 06Project-Admins: Archive old Incident-* projects - https://phabricator.wikimedia.org/T134624#2330224 (10Danny_B) [14:41:39] 06Labs, 06Operations, 06Project-Admins: Archive old Incident-* projects - https://phabricator.wikimedia.org/T134624#2330268 (10Danny_B) [14:44:45] 06Labs, 06Operations, 06Project-Admins: Archive old Incident-* projects - https://phabricator.wikimedia.org/T134624#2330287 (10Danny_B) [14:59:04] 06Labs, 10Labs-Infrastructure, 06Operations: rcstream not working for wikitech wiki - https://phabricator.wikimedia.org/T136245#2330352 (10Krenair) I think we might need to change `@resolve(wikitech.wikimedia.org)` to `@resolve(wikitech.wikimedia.org, AAAA)` [15:03:54] (03PS2) 10Jforrester: Don't duplicate things from #wikimedia-collaboration into -dev [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/230945 [15:03:57] (03PS1) 10Jforrester: Fix comment about two duplicated repos [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290947 [15:04:00] (03PS1) 10Jforrester: Add Graph and Kartographer to the main Editing channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290948 [15:04:22] (03CR) 10Jforrester: "Ping. This is still hugely wanted by the team." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/230945 (owner: 10Jforrester) [15:04:34] 10Quarry: puppet disabled on quarry-main-01 - https://phabricator.wikimedia.org/T136315#2330367 (10valhallasw) [15:06:49] (03CR) 10Merlijn van Deen: [C: 032] Add Graph and Kartographer to the main Editing channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290948 (owner: 10Jforrester) [15:07:12] (03CR) 10Merlijn van Deen: [C: 032] Fix comment about two duplicated repos [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290947 (owner: 10Jforrester) [15:08:55] !log toolsbeta toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host [15:09:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL, Master [15:12:00] (03Merged) 10jenkins-bot: Fix comment about two duplicated repos [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290947 (owner: 10Jforrester) [15:12:03] (03Merged) 10jenkins-bot: Add Graph and Kartographer to the main Editing channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/290948 (owner: 10Jforrester) [15:12:43] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328954 (10scfc) I would like to propose deploying a testing instance not only for the top solution, but (at least) the top... [15:15:10] 06Labs, 10Tool-Labs: Fwd: Alert: puppet failed on toolsbeta-webgrid-lighttpd-1201.toolsbeta.eqiad.wmflabs - https://phabricator.wikimedia.org/T136317#2330421 (10Krenair) [15:15:46] Krenair: bah, phabricator ignored my !project tag, it seems [15:15:48] thanks for the fix [15:16:21] so much for an easy forward option... oh well. [15:17:23] 06Labs, 10Tool-Labs: Fwd: Alert: puppet failed on toolsbeta hosts - https://phabricator.wikimedia.org/T136317#2330426 (10valhallasw) [15:23:03] 06Labs, 10Tool-Labs: Fwd: Alert: puppet failed on toolsbeta hosts - https://phabricator.wikimedia.org/T136317#2330441 (10valhallasw) * toolsbeta-mail: no clear cause, puppet seemed to be hanging. Host had load 0.8-1.0 without clear cause, rebooted the host. * toolsbeta-bastion: load 118 (?!). Rebooted. * tools... [15:32:18] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328954 (10tom29739) >>! In T136264#2328997, @bd808 wrote: > Step two involves evaluation, so we need to come up with a lis... [15:40:13] 06Labs, 10DBA, 07Tracking: Labs users missing grants on replicas (tracking) - https://phabricator.wikimedia.org/T136319#2330501 (10valhallasw) [15:40:35] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2330516 (10valhallasw) [15:40:37] 06Labs, 10DBA, 07Tracking: Labs users missing grants on replicas (tracking) - https://phabricator.wikimedia.org/T136319#2330514 (10valhallasw) [15:40:39] 06Labs, 10Tool-Labs: Cannot create database with s52590 - https://phabricator.wikimedia.org/T136247#2330515 (10valhallasw) [15:40:41] 06Labs, 10Tool-Labs: labsdb accounts being created without grants to create personal databases - https://phabricator.wikimedia.org/T130595#2330517 (10valhallasw) [15:41:14] 06Labs, 10Tool-Labs, 10DBA: Cannot create database with s52590 - https://phabricator.wikimedia.org/T136247#2328079 (10valhallasw) [15:48:09] valhallasw`cloud: any idea how I can see what tool uas mysql user s52953? [15:48:25] and also valhallasw`cloud is it likely http://librarybase.wmflabs.org/w/api.php is caused by all the db related tickets I see above? [15:48:26] addshore: https://tools.wmflabs.org/contact [15:48:31] fill in 52953 [15:49:04] aweosme, librarybase-db [15:49:04] addshore: no, that tool should use librarybasewiki_p, most likely [15:49:35] well, valhallasw`cloud the database did exist, and now Im getting access issue, just going to double check the details and things [15:49:38] afaik I didnt break anything ;) [15:49:58] addshore: I don't think that database should ever have been accessible [15:50:10] oh wait, yeh librarybasewiki_p [15:51:17] ahh wait, valhallasw`cloud it doesn't need a _p as its not a public db ;) [15:52:16] why would you have access to a private database? [15:52:37] if it's a user database, it would start with s52953__ [15:53:15] ahh yes s52953__librarybase [15:57:50] awesome, have it all figured out! [16:08:54] 10PAWS: I can not write some special characters in PAWS - https://phabricator.wikimedia.org/T136118#2330714 (10Framawiki) @yuvipanda Mmm do you think that the problem is to Jupyther's app, not Paws ? Same as the copy issue ? [16:24:17] 06Labs, 10Tool-Labs, 10DBA: Cannot create database with s52590 - https://phabricator.wikimedia.org/T136247#2330778 (10jcrespo) [16:24:19] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2330780 (10jcrespo) [16:24:34] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2316117 (10jcrespo) p:05Low>03Normal [16:25:22] (03Abandoned) 10Luke081515: [WIP] Implement command "akick" [labs/tools/Luke081515IRCBot] - 10https://gerrit.wikimedia.org/r/284448 (owner: 10Luke081515) [16:26:35] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2330800 (10jcrespo) @tom29739 Try now. (The task should continue being open for other missing users) [16:27:06] tom29739 ^ [16:27:35] Trying now.. [16:27:57] Works. [16:28:24] for some reason some users did not get the right provileges, I have to make a list of all users affected and fix it [16:29:20] jynus, is it fixed for new users? [16:29:38] valhallasw`cloud, mentions probably yes, but I am not 100% sure [16:29:46] tom29739, when did you create that account? [16:29:55] the tools one [16:30:00] not the user one [16:30:04] Not sure, I didn't create it. [16:30:14] Matthew_ did. [16:30:15] I mean, when did you register that tool? [16:30:34] ok, then, when did he gave you access? [16:30:35] Around July last year I should think. [16:30:45] jynus: I'm not sure either -- if those three commands are sent to the sql server in one go, is there any reasonable way the account creation works and but the grants do not? [16:30:45] That;s when I think it was created. [16:30:57] mmm, yeah, probably then it is fixed for new users [16:31:03] I got given access just a few days ago. [16:31:05] and this was a one-time problem [16:31:28] the other users was also a 1-year old account [16:31:51] Yes? [16:32:07] I don't have the problem with my 3-1 month old tools. [16:32:22] Matthew_, jynus was asking when the captcha-dev tool got created. [16:32:32] Oh goodness I don't remember. [16:32:42] jynus, can you fix it on 'captcha' too? [16:33:47] valhallasw`cloud, I didn't want to assume you or yuvi were wrong, but I know by experience that combining 2 languages' escape sequences (python + SQL) + templates is not preciselly obvious [16:34:04] :-) [16:34:41] jynus: mmm, yeah, that could potentially cause issues [16:35:04] and \_\_ is not preciselly obvious [16:35:20] (which was what caused issue in the first place) [16:35:29] Matthew_, 'Jul 9 2015 replica.my.cnf [16:35:30] ' July the 9th by the looks of it. [16:35:44] Okay [16:36:08] however, everything points you are right, because later accounts have it [16:36:21] assuming they are created sequentially [16:37:03] jynus, can you fix it for s52589 too? [16:37:08] (tool.captcha) [16:37:24] It doesn't work for that either. [16:42:07] tom29739, please add a comment on the ticket above with the number [16:43:33] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2330882 (10tom29739) @jcrespo, s52589 is also missing grants, 'Access denied for user 's52589'@'%' to database 's52589__laravelapi''. [16:43:41] jynus, done ^ [16:44:53] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2316117 (10jcrespo) a:03jcrespo [16:45:39] Why did it stop giving the grants in the first place? [16:47:09] 10Tool-Labs-tools-Other: polluted data on toollabs - https://phabricator.wikimedia.org/T136329#2330910 (10Bearcat) [16:48:49] 'At some point near the end of Jun 17, 2015 the filesystem backing the NFS storage used by LABS suffered a catastrophic failure, preventing most of Labs from working. Because efforts to recover the filesystem did not succeed, the decision was made to restore from a June 8th backup to a fresh set of volumes.' That ties in with the creation date of the 9th June. [16:50:42] 10Tool-Labs-tools-Other: dplbot untagged uncats: polluted data on toollabs - https://phabricator.wikimedia.org/T136329#2330941 (10valhallasw) [16:54:56] 06Labs, 10Labs-Infrastructure: Provide a web service displaying the true replag of labsdb instances - https://phabricator.wikimedia.org/T52010#2330959 (10valhallasw) 05Open>03Resolved https://tools.wmflabs.org/replag/ [16:55:43] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2330962 (10jcrespo) Running: ``` GRANT ALL PRIVILEGES ON `s52587\_\_%`.* TO 's52587'@'%'; GRANT ALL PRIVILEGES ON `s52589\_\_%`.* TO 's52589'@'%'; GRANT ALL PRIVILEGES ON `s52592\_\_%`.* TO 's5259... [16:56:18] tom29739, try again [16:57:16] Now works. [16:57:23] it wouldn't be surprising given that the process creating the grants is probably a tool [16:58:27] 10Tool-Labs-tools-Other: dplbot untagged uncats: polluted data on toollabs - https://phabricator.wikimedia.org/T136329#2330823 (10valhallasw) Thank you for filing a bug with such an extensive description! I'm not familiar with the dplbot tool, so I'm not sure what could cause these kinds of issues. When it comes... [16:58:55] jynus: it's a cronjob or a daemon on the labstore host [16:59:28] although maybe it wasn't yet at that point in time... I'm not 100% sure [17:01:05] If it was a tool, then it would be affected by NFS. [17:01:35] Which makes me think it was (or something else that depends/is affected by NFS) [17:02:09] 06Labs, 10DBA, 07Tracking: Labs users missing grants on replicas (tracking) - https://phabricator.wikimedia.org/T136319#2330981 (10jcrespo) [17:02:11] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2330979 (10jcrespo) 05Open>03Resolved @Yuvipanda, the creation date of these users, according to some users match last years' NFS failure. It is strange that because of that some grants worked... [17:03:26] I am doing a sanity check to see that there are no empty or %user users [17:04:35] tom29739: labstore == the nfs host, so when labstore went down, this process was definitely affected [17:04:46] tom29739: that doesn't explain the race condition, however [17:05:29] I will do some cleanup to disable accounts that have not been used for over a year [17:06:33] that might give us a lot of 'hey, I tried to continue with a project that had been lying there for a while and the database credentials don't work anymore' bugs [17:07:10] I will do something like that, not necesarily like that, or witout coordination, etc. [17:07:16] There's more attack vectors if there's accounts sitting doing nothing. [17:07:41] maybe in a way that the users could reenable it somehow [17:08:10] Maybe tie into that console.wmflabs.org thingy that bd808 is making. [17:08:29] Have an 'Enable/disable db user' button. [17:09:04] tom29739: write a ticket up. I can't promise anything but it sounds like something worth looking into [17:09:28] 10Quarry: puppet disabled on quarry-main-01 - https://phabricator.wikimedia.org/T136315#2330998 (10yuvipanda) Yes, when it restarted it came up trying to use python3 instead of 2, and I disabled it and hand hacked to fix it. Must fix puppet... [17:11:52] bd808, do you want it added/blocked to a certain ticket? [17:13:27] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2331033 (10yuvipanda) The script was also completely rewritten after that crash, so I think if it was due to that it's probably fixed by now. [17:16:41] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328954 (10yuvipanda) >>! In T136264#2330394, @scfc wrote: > I would like to propose deploying a testing instance not only... [17:17:04] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2331069 (10yuvipanda) >>! In T136264#2330485, @tom29739 wrote: >>>! In T136264#2328997, @bd808 wrote: > There's [[ https://... [17:18:32] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2331071 (10tom29739) >>! In T136264#2331069, @yuvipanda wrote: >>>! In T136264#2330485, @tom29739 wrote: >>>>! In T136264#2... [17:21:10] 06Labs, 10Tool-Labs, 10DBA: Add database options to console.wmflabs.org - https://phabricator.wikimedia.org/T136335#2331080 (10tom29739) [17:21:16] bd808, ^ [17:23:28] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2331102 (10yuvipanda) http://www.kelproject.com/ is a super new one. [17:24:33] 06Labs, 10Tool-Labs, 10DBA: Allow self-serve database credential and permissions management for Tool Labs projects - https://phabricator.wikimedia.org/T136335#2331104 (10bd808) [17:25:49] 06Labs, 10Tool-Labs, 10DBA: Allow self-serve database credential and permissions management for Tool Labs projects - https://phabricator.wikimedia.org/T136335#2331080 (10bd808) [17:25:51] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2331105 (10bd808) [17:36:18] Does anyone know how to put a sitenotice up on Wikitech? I am interested in putting a link to https://meta.wikimedia.org/wiki/Labs_TOU_Consultation_Round_1_(2016) to get more community input [17:37:27] chasemp, YuviPanda, andrewbogott: ^ do we have any guidelines about use of [[MediaWiki:Sitenotice]] on wikitech? [17:38:22] nope, I don't think it's been done before [17:38:38] just do it? [17:39:06] ^ [17:44:10] I use it sometimes, but usually only for notices about wikitech itself [17:44:29] bd808: andrewbogott shall we put it on the MOTD? [17:45:05] Both sound good to me [17:45:51] zzhou_: can you figure out what you want it to say? I can do the edit to make it live easily enough [17:46:02] Yeah, I agree, probably both [17:46:32] bd808: I can probably reuse that text for MOTD :D [17:46:43] ok let me think of a text [17:47:13] I put the rally boring "Labs TOU consultation: https://meta.wikimedia.org/wiki/Labs_TOU_Consultation_Round_1_%282016%29" in the /topic here a while ago [17:47:26] *really [17:49:11] You put 'TOU consultation', I added the 'Labs' bit. [17:50:13] Tried to make a little less boring. [17:50:52] Maybe for the sitenotice something like this with a ilnk to the TOU: “Labs developers: Join the community consultation on the Labs Terms of Use.”? [17:51:35] Make it stand out a little more, that sounds really formal. [17:51:41] (and boring) [17:52:15] It's likely to get more exposure on wikitech. [17:57:08] How about this “Labs developers: Join us to contribute your thoughts and ideas about the Labs Terms of Use!” Sorry I am not great at this stuff [17:57:24] bwiw cool w/ site notice and motd inclusion [17:57:26] thanks bd808 [17:58:10] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2331277 (10bd808) [17:58:54] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Evalute Kubernetes based workflow replacement options for SGE - https://phabricator.wikimedia.org/T136264#2328954 (10bd808) >>! In T136264#2331065, @yuvipanda wrote: >>>! In T136264#2330394, @scfc wrote: >> I would like to propos... [18:11:19] 10Tool-Labs-tools-Other: dplbot untagged uncats: polluted data on toollabs - https://phabricator.wikimedia.org/T136329#2331324 (10Bearcat) Just for the record, the current replag isn't the issue at all -- these are all ongoing problems that I've encountered even when replag was at zero. Replica drift may be the... [18:12:21] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [18:23:54] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#1906120 (10chasemp) So we can do this with limits.conf ```* - fsize 10000``` This would set a hard limit of 10M for files for example. Behavior is: * Writing a file that would be bi... [18:24:15] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2331363 (10chasemp) p:05Triage>03High [18:28:10] 06Labs, 10Tool-Labs: toolserver-home-archive is using 52G on Tools - https://phabricator.wikimedia.org/T136202#2331376 (10chasemp) p:05Triage>03High [18:28:29] 06Labs, 10Tool-Labs: toolserver-home-archive is using 52G on Tools - https://phabricator.wikimedia.org/T136202#2326568 (10chasemp) @Nemo_bis ping! Any objections to removing this old archive? [18:30:18] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 06Community-Tech-Tool-Labs: Develop evaluation criteria for comparing PaaS solutions - https://phabricator.wikimedia.org/T136265#2331399 (10bd808) [18:32:14] 06Labs, 10Tool-Labs: Fix webservice start check from tools webservice - https://phabricator.wikimedia.org/T136347#2331418 (10yuvipanda) [18:39:44] 06Labs, 10Tool-Labs: wikidata-exports is using 256G in Tools - https://phabricator.wikimedia.org/T136194#2331468 (10chasemp) >>! In T136194#2329393, @Addshore wrote: > Yeh I think sending these to the Internet Archive would make sense. > The code that I have written to use the JSON dumps based on the Wikidata... [18:41:46] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2331472 (10chasemp) On second thought as a blanket limit maybe 50G makes sense and allows the liberal use of scratch share, but either way the current 'unlimited' is not working out. [18:42:53] 06Labs, 10Tool-Labs, 07Tracking: Contact tool maintainters using large amounts of disk space (tracking) - https://phabricator.wikimedia.org/T136212#2331475 (10chasemp) [18:42:55] 06Labs, 10Tool-Labs: oar is using 207G on Tools - https://phabricator.wikimedia.org/T136201#2331473 (10chasemp) 05Open>03stalled >>! In T136201#2328244, @A930913 wrote: > This tool comes from the grant at https://meta.wikimedia.org/wiki/Grants:IEG/Open_Access_Reader applied for by Ed Saperia. > He is sayin... [18:43:17] 06Labs, 10Tool-Labs: wikidata-analysis is using 153G on Tools - https://phabricator.wikimedia.org/T136211#2331476 (10chasemp) Thank you @addshore [18:43:27] 06Labs, 10Tool-Labs: Fix webservice start check from tools webservice - https://phabricator.wikimedia.org/T136347#2331477 (10yuvipanda) This seems to be because the webservice is stuck in 'dt' state: ``` 6216398 0.32846 lighttpd-t tools.toolsc dt 05/15/2016 02:03:43 webgrid-lighttpd@tools-webgrid 1... [18:44:58] 06Labs, 10Tool-Labs: liangent-php is using 348G on Tools - https://phabricator.wikimedia.org/T136208#2331482 (10chasemp) thank you @liangent for responding. [18:47:41] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2331498 (10bd808) [18:50:12] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2331498 (10chasemp) If we are going to do this kind of shuffle I'm a fan of an authentication/authorization service model here, but it's all c... [18:50:22] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2331516 (10bd808) Input from #security-team on the general idea and implementation gotchas would be appreciated. [18:50:39] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286402 (owner: 10Jean-Frédéric) [18:51:04] chasemp: I'm pretty sure I agree that using a service would be nicer [18:51:27] selecting, configuring, deploying a service ... that's harder IMO [18:53:08] (distracted but) yeah..I know it. I'm not saying we can do it but I want to at least state we should consider it. I /think/ we would find out we didn't realize how much we were missing it. etc [18:53:22] but I appreciate you approaching the problem :) [19:03:39] 06Labs, 10Tool-Labs: Fix webservice start check from tools webservice - https://phabricator.wikimedia.org/T136347#2331549 (10yuvipanda) that's just one of the issues. The fundamental problem seems to be that: http://tools.wmflabs.org/toolschecker/ is returning 200 while it says it returns 503! [19:12:19] 06Labs, 10Tool-Labs: toolserver-home-archive is using 52G on Tools - https://phabricator.wikimedia.org/T136202#2331566 (10Nemo_bis) What do you mean by offline storage, why would that be a benefit and what makes you think this is not used? [19:15:37] 06Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2331578 (10Nemo_bis) So we went from considering deletion too heavy-handed to being fine with unrecoverable errors above a certain threshold? What is wrong with my middle ground? >>! In T122508#1938971, @Nemo_bis... [19:21:58] 06Labs, 10Tool-Labs: toolserver-home-archive is using 52G on Tools - https://phabricator.wikimedia.org/T136202#2331595 (10chasemp) Storage somewhere that isn't actively using up NFS share space, or really anywhere anyone wants to make their own copy I imagine. NFS space is costly, we back it up, we replicate... [19:23:03] (03PS1) 10Yuvipanda: Set 503 error code for 503 errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291019 (https://phabricator.wikimedia.org/T136347) [19:23:26] bd808: valhallasw`cloud ^ can you review a small change to the tools home stuff? [19:24:46] YuviPanda: that caused problems before [19:25:21] YuviPanda: https://gerrit.wikimedia.org/r/#/c/278984/ [19:26:04] bd808: I see. that seems untrue in my testing, let me test again [19:27:24] I get a 503 status for https://tools.wmflabs.org/toolschecker/ [19:27:28] bd808: so I've cherry picked my patch and http://tools.wmflabs.org/toolschecker/ continues to work [19:27:41] bd808: right, because my patch is cherry picked ther enow [19:27:43] *there [19:28:14] huh. I'm sure we did that prior patch because nginx was puking on the 503 status code [19:28:35] bd808: I uncherry picked it and it's back to 200 now [19:29:03] yah, your right [19:29:09] *shrug* [19:29:21] bd808: possibly an nginx upgrade fixed a bug somewhere maybe? [19:29:21] We certainly do want the 503 status code [19:29:24] or something along those lines [19:30:05] (03CR) 10BryanDavis: [C: 031] "This is basically reverting Ibf5cfc8db73ebb70698ba06bf7275675eada5452 but that looks like it works now. *shrug*" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291019 (https://phabricator.wikimedia.org/T136347) (owner: 10Yuvipanda) [19:30:19] Random question: Can Symfony run on tool labs? [19:30:29] Matthew_: sure. that's just php [19:30:33] (03PS2) 10Yuvipanda: Set 503 error code for 503 errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291019 (https://phabricator.wikimedia.org/T136347) [19:30:39] Okay. [19:30:44] bd808: I've amended the message. can you +2 too? [19:31:01] I'm somewhat sure that check //worked// at some point w/ Ibf5cfc8db73ebb70698ba06bf7275675eada5452 in place but...*shrug* I don't have a real timeline [19:31:22] (03CR) 10BryanDavis: [C: 032] Set 503 error code for 503 errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291019 (https://phabricator.wikimedia.org/T136347) (owner: 10Yuvipanda) [19:31:37] * bd808 is proud of YuviPanda for wanting actually +2 review ;) [19:32:09] bd808: I think that's one of the lessons of the webservice outage. I think the fact that --register-proxy was a breaking change was something I missed, and would've probably been caught in CR [19:32:35] bd808: there's still the problem of not enough people to CR, but that's alleviated a little bit with you around :D [19:32:55] one more pair of eyes anyway :) [19:33:14] YuviPanda: meeting [19:33:20] chasemp: bah, yes. [19:37:39] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2331693 (10bd808) >>! In T136350#2331516, @bd808 wrote: > Input from #security-team on the general idea and implementation gotchas would be ap... [19:47:04] zzhou_: I posted a sitenotice -- https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Sitenotice&oldid=576691 [19:49:40] thanks bd808! Do you mind adding “potential” to the statement: “Give your feedback on potential Labs Terms of Use changes [19:50:08] sure, easy enough [19:50:13] great [19:51:12] {{done}} [19:53:00] (03Merged) 10jenkins-bot: Set 503 error code for 503 errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291019 (https://phabricator.wikimedia.org/T136347) (owner: 10Yuvipanda) [19:53:19] YuviPanda: ^ wow that took forever to merge [19:57:10] !log tools.admin Updated to "Set 503 error code for 503 errors" (8361146) [19:57:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL, Master [20:00:16] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2331739 (10csteipp) If you want to use ldap to store the secret, then mediawiki's Ex:OATHAuth needs to be ldap aware (or have hooks to let ano... [20:10:00] (03CR) 10Tim Landscheidt: "I believe the original issue (T128898) was actually fixed by 6c3c1eaa1b4b29c15174b87a53e6b9123f2a1892, and b3b4d1843de969b780797049d0bfa81" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291019 (https://phabricator.wikimedia.org/T136347) (owner: 10Yuvipanda) [20:12:09] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2331812 (10bd808) >>! In T136350#2331739, @csteipp wrote: > If you want to use ldap to store the secret, then mediawiki's Ex:OATHAuth needs to... [20:13:19] bd808: do you think you'll be generally interested in offering CR for the changes I'm going to start making to the toollabs-webservice code to add k8s support to it? [20:13:38] yes [20:13:56] I'd love to know how it all works and code review is a great way to learn [20:14:22] bd808: awesome! I'll also involve you in all the dockerfile stuff? [20:14:40] sure [20:14:59] bd808: cool. I'll give you merge rights on all those things too and add you as reviewer. [20:17:59] bd808: my talk at kubecon is probably the most complete documentation of how the gridengine replacement would work, so I should write it down somewhere too. [20:18:04] hmm I probably did actually [20:18:06] nvm [20:18:09] it was the etherpad I was sharing everywher [20:18:11] e [20:20:35] I need to start making a wiki page about striker [20:35:57] (03CR) 10BryanDavis: "> Doesn't need http_response_code() be called before anything is output?" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291019 (https://phabricator.wikimedia.org/T136347) (owner: 10Yuvipanda) [20:40:29] 06Labs, 10Tool-Labs, 10DBA: Allow self-serve database credential and permissions management for Tool Labs projects - https://phabricator.wikimedia.org/T136335#2331080 (10valhallasw) - Reset database password - recreate replica.my.cnf - Recreate grants [20:43:12] (03PS1) 10BryanDavis: www: cleanup minor index.php issues [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291072 [20:44:17] (03CR) 10Yuvipanda: www: cleanup minor index.php issues (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291072 (owner: 10BryanDavis) [20:47:07] (03CR) 10BryanDavis: www: cleanup minor index.php issues (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291072 (owner: 10BryanDavis) [20:48:14] (03CR) 10Yuvipanda: [C: 031] www: cleanup minor index.php issues (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291072 (owner: 10BryanDavis) [20:48:44] bd808: so one convention I used to have for this is someone looks at the code, then someone tests and merges [20:49:21] I'm trying to figure out how to establish that as a pattern [20:51:01] YuviPanda: by telling people that's what you want to see happen :) [20:51:11] bd808: right. [20:51:25] so you want 2 reviewers for everything? [20:52:01] bd808: there are 4 roles: CR, Testing, Merging, and Author, and otehr than Author != CR I think the rest aren't mutually exclusive [20:52:15] and ideally CR is same as Merge and Author is same as Test [20:53:09] I kind of like the diffusion pattern where the author actually does the merge but only after getting "approved" in CR [20:53:10] bd808: this is slightly different from rest of wikimedia perhaps because not everyone can test due to access privilages. [20:53:15] bd808: yeah, I like that too [20:53:28] but it doesn't really matter to me in any direction [20:53:31] bd808: so maybe one way to do it is [20:54:07] bd808: author tests and +2s and merges it themselves - shepherds the change to the end, since they've the most information about it. But someone CR+1s it before that [20:54:29] bd808: this jives well with the puppet model, since if you make a change ultimately you're responsible for rolling it out [20:55:08] *nod* I think the responsibility of +2 mw-core is one of the things that makes CR there so slow [20:55:24] right [20:55:30] you have to find someone who feels they completely understand your change and the implications [20:55:50] you aslo feel as a reviewer like this may just be some crap off the top of someone's head [20:55:50] and that's difficult and it kinda devolves into some form of old boy's club where trust plays a big factor for big changes [20:56:54] the power of trust is true for any FLOSS community I've participated in though [20:57:03] that's human nature if nothing else [20:57:22] bd808: totally, but it's good to have that explicitly mentioned rather than implicitly happen [20:57:29] sets different expectations and wha tnot [20:57:31] *what not [20:57:43] Reviewing takes ages and sometimes doesn't happen at all. [20:58:05] {{cn}} and overly broad tom29739 :) [20:58:16] it all depends on the project and team [20:58:23] korma.wmflabs.org for numbers [20:58:34] mw-core is a dumping ground though [20:58:39] It's one of the many reasons I don't like code review (and especially gerrit) [20:58:49] we get drive by patches of random quality with no tests all the time [20:59:20] but I won't defend that the MediaWiki community has healthy code review norms [20:59:25] because we don't [20:59:48] i dont think gerrit is specifically making code review slower than another tool, humans still have to read and feel like they understand all the code as bd808 said [21:01:10] bd808: https://etherpad.wikimedia.org/p/review-workflow-yuvi [21:01:11] MW code is really hard to review too [21:02:28] andrewbogott: I could rewrite it to make use of the 'roles' structure, which might be clearer [21:03:23] bah [21:03:26] bd808: ^ not andrewbogott [21:03:32] but I tihnk you're all in a meeting so I'll stop bugging [21:03:42] YuviPanda: :) I knew you meant me. In an interview now [21:03:54] ok [21:14:48] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2331498 (10Parent5446) I've lost track of exactly what features AuthManager supports, but does it allow storing of arbitrary user authenticati... [22:02:13] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2332183 (10csteipp) > The only thing that makes me sad about this is that it would mean that wikitech remains an LDAPAuth wiki indefinitely bl... [22:14:08] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2332237 (10dpatrick) >>! In T136350#2332183, @csteipp wrote: > No argument that we do need to consolidate authentication-- linotp would be one... [22:17:15] 06Labs, 10Tool-Labs: wikidata-exports is using 256G in Tools - https://phabricator.wikimedia.org/T136194#2332261 (10ArielGlenn) I would not rsync every hour, that seems like overkill. I could try to schedule the cron job to run shortly after the entity dump completes, maybe put them together in a little bash s... [22:29:11] Will c1.labsdb give me access to all projects database dumps? Or will I have to connect to all of the shards individually? [22:30:04] I'm not pulling data from all projects but I want to be able to pull data from any project based on user's choice. [22:36:21] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2332343 (10bd808) >>! In T136350#2332237, @dpatrick wrote: >>>! In T136350#2332183, @csteipp wrote: >> No argument that we do need to consolid... [22:37:55] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2332349 (10bd808) >>! In T136350#2332046, @Parent5446 wrote: > I've lost track of exactly what features AuthManager supports, but does it allo... [22:48:40] 06Labs, 10Tool-Labs, 10DBA: Allow self-serve database credential and permissions management for Tool Labs projects - https://phabricator.wikimedia.org/T136335#2332388 (10tom29739) [22:49:40] YuviPanda: Do you know anything about using sshfs with Tool Labs? [22:52:29] 06Labs, 10Tool-Labs, 10DBA: Allow self-serve database credential and permissions management for Tool Labs projects - https://phabricator.wikimedia.org/T136335#2332392 (10tom29739) [23:06:54] 10Tool-Labs-tools-Other: dplbot untagged uncats: polluted data on toollabs - https://phabricator.wikimedia.org/T136329#2332413 (10Bearcat) Replag is down to zero at the moment, so I'm going to take the opportunity to add a bit more context. I haven't gotten all the way through the entire batch yet today, but her... [23:37:45] my attempts at logging in to horizon are hanging after submitting the login form... any known issues? [23:37:56] oh geez spoke too soon. I guess it's just running REALLY slow