[00:46:20] (03PS1) 10Greg Grossmeier: Make RelEng projects sub-project proof [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/293663 (https://phabricator.wikimedia.org/T137494) [00:51:05] I just setup a new server instance using Horizon. Now i'd like to to add role::labs::mediawiki_vagrant. How do I do that in Horizon? [00:51:17] you can't yet [00:51:30] you have to go to wikitech to manage puppet config [00:51:47] it's being worked on though! [00:52:11] Okay, where do I do that in wikitech? [00:52:40] Do these instructions still apply? https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs [00:53:42] Yes those instructions should be right. The puppet config screen is linked from https://wikitech.wikimedia.org/wiki/Special:NovaInstance [00:54:12] That's what I thought. [00:54:16] Some info at https://wikitech.wikimedia.org/wiki/Help:Instances#Managing_Instances [00:54:46] I don't have configure or delete links. [00:55:20] hmmm... but you can see the instance in the table? [00:55:36] Yep, I see the instance table for the project (security-tools). [00:56:24] mind if I join the project to see if I can see the links? [00:56:29] Sure, go for it. [00:56:49] i can see the instance names on https://wikitech.wikimedia.org/wiki/Nova_Resource:Security-tools [00:56:54] without being logged in [00:57:45] dapatrick: can you see https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=f4d46a5c-f8cd-405d-9cbf-82710a9564e2&project=security-tools®ion=eqiad [00:58:03] "The specified resource does not exist." [00:58:30] weird. Sounds like your wikitech session and the nova session are out of sync [00:58:42] I'd try logging out of wikitech and logging back in next [00:58:45] try logging out and logging back in ..heh [00:58:52] this has been known to happen [00:58:55] it sounds weird but it once fixed it for me too [00:58:55] It's been like this through several authentications. [00:58:59] But I'll try again. [00:59:42] No dice. I still don't see the links. [01:01:06] problem exceeds my nova debugging knowledge :/ [01:01:25] bd808: can you remove and re-add dapatrick? [01:01:26] I think you are going to need andrewbogott or maybe Krenair [01:01:28] to the project? [01:01:33] yeah I can do that [01:01:59] I can't help right now, sorry [01:02:13] I have a few minutes, let me read the backscroll... [01:03:16] !log security-tools Removed and re-added Dpatrick as project member and admin [01:03:20] i think this is different [01:03:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Security-tools/SAL, Master [01:03:28] i can confirm it with my user too [01:03:32] and its only in this project [01:03:35] not in other projects [01:03:42] there i still see the configure links [01:04:09] dapatrick: were you recently added to the project or have you been projectadmin for a while? [01:04:16] the "Actions" column is just empty [01:04:17] mutante: I don't see you in the admin list there [01:04:26] oh, only just added, huh? [01:04:42] I've been an admin like a year. [01:04:52] andrewbogott: I just cycled his membership on the off chance that would clear some cache [01:04:56] Oh, I see, I misread the log entry [01:05:09] bd808: you are right o course [01:05:27] i see them now , after being admin [01:05:32] mutante: not of course :) I'm totally wrong a lot [01:05:34] also just added [01:06:43] interesting is that when i follow one of the links i get [01:06:46] Your account is not in the project security-tools. [01:06:49] but it is [01:07:33] Okay, in the meantime while this is being figured out, can someone add the vagrant role? [01:07:46] yup I can do that [01:07:51] I need it on Two-factor.security-tools.eqiad.wmflabs [01:07:55] yea odd combo, before i was admin in it i did not get to see the links, but now that i see them it says i am not in the project [01:08:10] dapatrick: {{done}} [01:08:23] Thanks. [01:10:11] dapatrick: I don't know why this is happening. You get the same results for every project that you're projectadmin in, or just that one? [01:11:27] bastion appears the same as well. [01:11:28] No links. [01:11:51] So does testlabs [01:15:24] And you have 2fa enabled I presume? [01:16:47] andrewbogott Yes, I have 2fa enabled. [01:17:02] andrewbogott But, I also disabled and checked for the links, just to rule it out. They still did not appear. [01:17:32] Hm, I wouldn't expect that page to display at all without 2fa [01:20:24] well, short of ripping the guts out of wikitech this is a hard thing to debug. I'm going to hope that it fixes itself when the cache turns over, and in the meantime you should ping me if you need puppet changes. [01:21:04] andrewbogott Okay, thanks. [01:21:23] andrewbogott It's been like this for a while, so I'm not sure that it will fix itself. [01:21:30] huh [01:21:52] andrewbogott I figured you all were making some changes or something, so I didn't worry about it, until today when I need to add a role to a server. [01:24:04] dapatrick: can you try logging out and in one more time, and tell me if anything changed? [01:25:23] andrewbogott Look the same. No configure or delete links. [01:25:28] ok [01:33:32] Hmm. So, I've setup a proxy, but I'm getting a 504 timeout. [01:35:15] I followed the instructions at https://wikitech.wikimedia.org/wiki/Help:Proxy, but I think I can't do the last part because of problems with my account. [01:35:23] Also, I can't login to horizon now. [01:35:54] Okay, i'm logged in to horizon. [01:37:19] dapatrick: after adding the proxy you also need a security group to allow port 80 rom 0.0.0.0 [01:37:36] (or 10.0.0.0/8) [01:37:42] I've already associated the instance with a group that allows that. [01:37:48] let me try if i can click that [01:38:51] no, i cant [01:38:56] i dont see existing security groups listed [01:39:05] and Failed to create security group. [01:40:00] i still see the conigure instance links but it claims i am not a project member [01:41:23] Does I need to update iptables rules on the instace? [01:41:24] *instance [01:41:24] logged out and back in, still shown as admin but no security groups in the project [01:42:04] normally the security group membership would do that [01:42:24] That's what I figured. It didn't. [01:42:51] only domain and bootps are in ACCEPT in INPUT [01:43:17] I'm expecting to see 8080, 80, and 443. [01:44:10] hrmm, yea, i dont even see the group [01:44:19] andrewbogott: can you add the group maybe? [01:44:32] since you could also add classes to instnaces [01:44:51] I'm doing this in horizon, correct? [01:45:19] yes, in horizon [01:45:51] Okay, in horizon, I've added a newly created security group, "vagrantwebserver", that allows 8080 from 10.0.0.0/8. [01:46:08] That should allow the proxy to reach 8080 on the instance, correct? [01:46:24] yeah, but as far as I know the proxy only uses 80 [01:46:33] ssl is terminated at the proxy and relayed over 80 [01:46:42] did the security groups part switch to horizon within the last couple days? [01:46:48] it works in both places [01:46:51] ah! [01:47:17] i suppose it would if it thought i was in the project [01:47:45] andrewbogott You're saying SSL is terminated at the proxy and related over port 80, only? [01:47:54] correct [01:48:42] What is the purpose of creating a proxy in horizon and specifying the backend port at 8080 and backend instance to be the instance I'm targeting? [01:49:23] If what you're saying is correct, then that should have not effect whatsoever, and therefore, https://wikitech.wikimedia.org/wiki/Help:Proxy would be incorrect. [01:49:26] oh [01:49:27] Or am I misunderstanding what you're saying. [01:49:43] um… you're right, if you specified 8080 then it will use 8080 [01:49:51] I've never heard of anyone doing anything other than plain old 80 though [01:50:05] * andrewbogott looks at the gui [01:50:24] I'm doing this: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs [01:51:08] Nevermind, it's working now. [01:51:15] I'm not sure why, though. [01:55:27] I think I'm good for the time being. Thanks for your help everyone! [01:56:49] great, sorry about all the pitfalls [02:21:25] andrewbogott: mw-vagrant on a Labs instance exposes the webserver on 8080 for "reasons". Mostly do to LXC and how the container port is exposed to the Labs VM. [04:59:50] !log commtech Trying vm.dirty_background_ratio = 5; vm.dirty_ratio = 10 on commtech-1 [04:59:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL, Master [06:29:26] 10Quarry: Queries running for more than 4 hours and not killed - https://phabricator.wikimedia.org/T137517#2370649 (10Dvorapa) [06:30:04] 10Quarry: err - https://phabricator.wikimedia.org/T137518#2370652 (10Dvorapa) [06:30:19] 10Quarry: err - https://phabricator.wikimedia.org/T137518#2370664 (10Dvorapa) 05Open>03Invalid [06:32:03] 10Quarry: Queries running for more than 4 hours and not killed - https://phabricator.wikimedia.org/T137517#2370670 (10Dvorapa) [06:46:08] 06Labs, 10Tool-Labs: toolserver-home-archive is using 52G on Tools - https://phabricator.wikimedia.org/T136202#2370684 (10Nemo_bis) Thanks, Dispenser. I'd welcome some repackaged version of the archive, there's surely some big portion of non-code. At the time I only managed to remove some 10 GB worth of templa... [07:32:58] !log ores-staging deploying 4efc5b7 into staging [07:33:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores-staging/SAL, Master [07:34:44] 10Quarry: Queries running for more than 4 hours and not killed - https://phabricator.wikimedia.org/T137517#2370631 (10Krenair) It doesn't appear to be running anything at the moment: ```root@quarry-main-01:~# mysql -h enwiki.labsdb -u u2029 -p enwiki_p -e "show processlist" Enter password: +----------+-------+-... [07:36:09] !log ores deploying 4efc5b7 into prod [07:36:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [07:40:06] "Cannot allocate memory" are you kidding me [07:45:19] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2370748 (10jcrespo) > The Toolserver previously used MySQL views for this An escalation of privileges vulnerability would expose user private data. It also exposes the th... [07:52:25] 06Labs, 10Labs-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2370750 (10hashar) Zuul (since 2.1.0-95) now measures the time for a build to actually start on a Node. That represents how... [08:24:29] SMalyshev: can wdq-varnish and undeltest in the wikidata-query labs project be deleted? [08:58:39] 06Labs, 10Labs-Infrastructure, 06Operations: investigate slapd memory leak - https://phabricator.wikimedia.org/T130593#2370970 (10MoritzMuehlenhoff) I have tested the 2.4.41 packages in vagrant with a syncrepl setup and seems fine. Update will happen next week, not really something for a Friday... [09:04:08] (03PS1) 10Lokal Profil: Granting l10n-bot the necessary rights to migrate to using local i18n files. [labs/tools/heritage] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/293691 (https://phabricator.wikimedia.org/T137015) [09:04:28] (03CR) 10jenkins-bot: [V: 04-1] Granting l10n-bot the necessary rights to migrate to using local i18n files. [labs/tools/heritage] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/293691 (https://phabricator.wikimedia.org/T137015) (owner: 10Lokal Profil) [09:08:15] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2371001 (10Gehel) some reverse engineering has already been done by @jcrespo, documented on [[ https://wikitech.wikimedia.org/wiki/MariaDB/Sanitarium_and_Labsdbs... [09:14:18] (03CR) 10Jean-Frédéric: [C: 031] Granting l10n-bot the necessary rights to migrate to using local i18n files. [labs/tools/heritage] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/293691 (https://phabricator.wikimedia.org/T137015) (owner: 10Lokal Profil) [09:35:34] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2371014 (10jcrespo) It is scheduled. It is difficult to give an estimation, but it can be done after enwiki is finished, so 3-6 months? Labs hosts, by its own nature cannot and will probably not be 100% ever... [09:52:02] (03PS1) 10Lokal Profil: Make two cosmetic fixes [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293697 [10:13:31] (03CR) 10Lokal Profil: [V: 04-1] "I get:" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 (owner: 10Jean-Frédéric) [10:16:43] Hi, I added new SSH key (comment is martin@raspberrypi) to my Gerrit account yesterday or later. But I still cannot connect. My username is "Urbanecm". Do anybody know why this happened? I thought that this was something with puppet but puppet run should be done now... Can anybody help me? Or should I ask on another channel? Thanks for your reply. [10:18:13] Verbose output from my SSH agent is published on https://cs.wikipedia.org/wiki/Wikipedista:Martin_Urbanec/Gerrit_error . [10:19:00] The command was ssh gerrit, in my ~/.ssh/config I have [10:19:02] Host gerrit [10:19:04] Port 29418 [10:19:06] HostName gerrit.wikimedia.org [10:19:08] User Urbanecm [10:19:09] IdentityFile ~/.ssh/id_rsa [10:28:10] (03CR) 10Jean-Frédéric: [C: 032] Ensure all monuments_config entries are used [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292533 (https://phabricator.wikimedia.org/T136704) (owner: 10Lokal Profil) [10:28:59] (03Merged) 10jenkins-bot: Ensure all monuments_config entries are used [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/292533 (https://phabricator.wikimedia.org/T136704) (owner: 10Lokal Profil) [10:30:17] !log tools.heritage Deployed latest from Git: d25eda5 (T136704) [10:30:18] T136704: Identify entries in monument_config which are not used in fill_table_monuments_all (or fill_table_wlpa_all) - https://phabricator.wikimedia.org/T136704 [10:30:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [10:45:29] 10Quarry: Queries running for more than 4 hours and not killed - https://phabricator.wikimedia.org/T137517#2371169 (10Dvorapa) Therefore it looks like there is an error with status of the query. Please see screenshot: {F4149692} [10:51:33] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2371192 (10Blahma) Thank you. I did not realize there were also SQL dumps, not only XML. Would it perhaps be possible to have the latest dump readily available on an SQL server? That could be an alternative fo... [10:54:24] (03CR) 10Lokal Profil: [C: 031] "Looks good. I added one comment but that could be addressed in a later patch aimed at making the toolbox functional again (hence the +1, b" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [11:17:40] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2371239 (10jcrespo) @Blahma what do you want me to do? I can load those files, but those would be out of sync as soon as they are imported, and impossible to get updated. You can load those tables to the same... [11:23:38] (03PS3) 10Jean-Frédéric: Do not import Intuition from toolbox PHP files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 [11:28:35] (03CR) 10Jean-Frédéric: "> I added one comment but that could be addressed in a later patch aimed at making the toolbox functional again (hence the +1, but I'm ok " (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [11:53:33] (03CR) 10Lokal Profil: [C: 031] "Answered your comment. Looks good to me but I'll leave the decision to you to say if you consider that to be a blocker." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [11:56:03] (03PS1) 10Lokal Profil: Import messages from https://github.com/Krinkle/intuition [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) [12:05:34] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2371339 (10Blahma) @jcrespo Thanks for staying on the constructive line. FYI, the output in question is https://cs.wikipedia.org/wiki/Wikipedie:%C3%9Adr%C5%BEba/Nekategorizovan%C3%A9_%C4%8Dl%C3%A1nky_s_ohledem... [12:06:05] 10Tool-Labs-tools-Other, 10Phabricator, 15User-bd808: Stashbot shouldn't subscribe itself to tasks - https://phabricator.wikimedia.org/T135790#2371343 (10Aklapper) >>! In T135790#2369420, @Paladox wrote: > @bd808 this is known upstream. @Paladox: Any link handy? Thanks in advance! [12:06:25] (03CR) 10Lokal Profil: "If the assumption in my comment was wrong then I'll hapilly merge this." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (owner: 10Jean-Frédéric) [12:08:27] (03PS3) 10Jean-Frédéric: Add testing infrastructure with npm/grunt [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (https://phabricator.wikimedia.org/T137544) [12:12:12] 10Tool-Labs-tools-Other, 10Phabricator, 15User-bd808: Stashbot shouldn't subscribe itself to tasks - https://phabricator.wikimedia.org/T135790#2371369 (10Paladox) @Aklapper https://secure.phabricator.com/T11035 [12:12:33] (03CR) 10Jean-Frédéric: "That actually rings a bell ; but OTOH I can run npm install just fine... Maybe the `private: true` makes it optional?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293633 (https://phabricator.wikimedia.org/T137544) (owner: 10Jean-Frédéric) [12:46:10] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2371458 (10jcrespo) This is very easy to fix- tell your users to mark those that are incorrect, and exclude them from your query- that is very easy to do and doesn't require waiting. [12:46:21] (03PS2) 10Jean-Frédéric: Add CSS linting via stylelint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 [12:47:25] (03CR) 10Jean-Frédéric: "> "Running "stylelint:src" (stylelint) task" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 (owner: 10Jean-Frédéric) [12:53:17] (03CR) 10Jean-Frédéric: "It’s fine like this I think, this can be in a future patch." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [12:53:24] (03CR) 10Jean-Frédéric: [C: 032] Do not import Intuition from toolbox PHP files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [12:55:02] (03Merged) 10jenkins-bot: Do not import Intuition from toolbox PHP files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293651 (owner: 10Jean-Frédéric) [12:56:07] (03CR) 10Jean-Frédéric: [C: 031] "One small question otherwise good to me." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293713 (https://phabricator.wikimedia.org/T136566) (owner: 10Lokal Profil) [13:16:10] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2371555 (10MZMcBride) >>! In T126946#2371458, @jcrespo wrote: > This is very easy to fix- tell your users to mark those that are incorrect, and exclude them from your query- that is very easy to do and doesn't... [13:18:56] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2371577 (10jcrespo) > The latest "solution" to the constant stream of data integrity issues on Wikimedia Labs database replicas is to further inconvenience volunteers? No, the latest solution is the reimport... [14:06:56] (03CR) 10Jean-Frédéric: "Does this need to be rebased? I’m not sure I understand how it works to be honest ^__^'" [labs/tools/heritage] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/293691 (https://phabricator.wikimedia.org/T137015) (owner: 10Lokal Profil) [14:09:05] !log tools.heritage Deployed latest from Git: 74d9086 [14:09:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [15:46:09] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2371978 (10bd808) My attempt to intercept and correct HTTP traffic from MerlBot's scripts to the Wikimedia servers has not been successful. Error... [18:27:44] topic change because TOU consultation round is closed [20:17:50] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: mediawiki-extensions-qunit failing "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2372819 (10Legoktm) p:05Triage>03High I'm seeing this pretty often now... [20:24:41] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2372823 (10bd808) @valhallasw dug up some information that setting an `http_proxy` environment variable may influence the HttpCore library. I hav... [20:25:55] !log ores deleted 99-redis.yaml and restarted celery-ores-worker on all worker nodes [20:25:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [20:31:11] Hey yuvipanda. Around? I'm looking at http://graphite.wmflabs.org/ and it's down. No one is sure if it is supposed to be up and we've been relying on it. I figured you'd have thoughts [20:34:14] hey halfak. IN a meeting - the browser is aving problems [20:34:18] it is still collecting data and the data should be visible in grafana [20:34:24] I'll take a look once meeting is over [20:34:32] OK thanks dude [22:19:13] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2373089 (10bd808) >>! In T121279#2372823, @bd808 wrote: > @valhallasw dug up some information that setting an `http_proxy` environment variable m... [22:58:55] halfak so I think graphite.wmflabs.org is totally working - but only over https [23:12:35] Is the database corruption known? https://en.wikipedia.org/wiki/User_talk:Dispenser#DAB_Challenge [23:13:24] select * from pagelinks where pl_namespace=0 and pl_title="Where_Was_I"; is wrong from http://enwp.org/Special:WhatLinksHere/Where_Was_I [23:18:27] Ok, reporting as dup of T134203 [23:18:27] T134203: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203 [23:34:48] (03PS1) 10Jean-Frédéric: Enable PHP CodeSniffer with MediaWiki preset [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293889 (https://phabricator.wikimedia.org/T134764) [23:45:07] 06Labs, 10Horizon, 13Patch-For-Review: Switch dynamicproxy to point back to IP rather than domain names - https://phabricator.wikimedia.org/T133554#2373236 (10AlexMonk-WMF) I just need to make it carry out the same change to the data in redis, @YuviPanda?