[01:00:49] RECOVERY - Host tools-exec-1433 is UP: PING OK - Packet loss = 0%, RTA = 1.56 ms [01:19:04] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:24:44] Hey guys, I'm running a SQL statement which is timing out. It's optimized as well as it can be, any ideas of anything else (perhaps a memcache or something) I can run it against? [01:27:32] nope [01:27:50] memcached works in a completely different way [01:28:17] Ah, I'm most familiar with ElasticSearch. [01:28:30] I understand it's apples / orange tho. [01:28:55] Depending on what your queries actually are, you might want ot split them into smaller ones, or request smaller numbers of records, and do pagination [01:29:12] If you pastebin the query, someone might be able to advise how to sort it [01:32:58] It's super simple, I assume it's in the text comparison, but not sure how else to pull it off.. https://pastebin.com/AKtD0Zzj [01:33:42] yeah. that's always going to be a table scan. [01:33:56] I wonder what indices we have on revision? [01:34:08] Not on comments, which makes sense. [01:34:41] I tried sub querying it, get all edits from user then string compare. But same. [01:34:48] and a double wildcard like that can't be indexed in mysql [01:35:07] I tried a = but same same. [01:35:08] * bd808 is called away to dinner [01:35:38] rev_user probably isn't indexed by itself [01:36:53] https://www.mediawiki.org/wiki/Manual:Revision_table#Indices [01:37:15] looks like it should be the first in user_timestamp though [01:38:10] Getting just a top 50 of all revisions by user is pretty quick. [01:39:34] yeah. so the trick will be in how the query optimizer decides to deal with that nasty like clause [01:39:59] So, why is it still bad when I say = "...." [01:46:01] drewmutt: it probably just takes some time, the data is on spinning rust and the user has some 12k revisions, so it has to seek to 12k places on disk to find each revision [01:46:52] ebernhardson: Ah, okay, so I threw out a grid job for it and after about 30min, it timed out :/ [01:52:55] as for other ideas ... it might be possible to break up the query into batches of revision ids. i wouldn't know what to use though as the bucket sizes. i think the revision id, as primary key, is stored in the index allowing it to decide before doing a bunch of seeks if it should visit a revision, but i'm not 100% sure on that it might be some sort of mariadb internal row identifier [01:54:03] explain claims there isn't a second step using `and rev_id between 0 and 100000000` at least. *shrug* [01:56:38] Ahh.. I could do the bucket thing. What's confounding is that there's some XTools that seem to do this kinda thing pretty quickly, well, I don't know that they're doing string compares though. [01:58:51] problem might be deciding bucket sizes ... another option: `select rev_id from revision where rev_user = NNNN`, and then make a query for groups of rev_id's ? might be a pain from quarry (guessing you are using quarry?) but you could reasonably automate it in https://paws.wmflabs.org/paws/hub/home [01:59:05] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:29] but at least then you could issue reasonably sized queries that check 100 revisions at a time or some such [02:00:07] ebernhardson: Ah, good idea. I was using Quarry, now using mysql in SSH. [02:01:33] paws doesn't seem to like me today though, getting in a redirect loop. oh well. Anything you can do in paws you could do from a shell via sh [02:01:37] sh [02:01:39] * ebernhardson fails typing .. [02:01:41] ssh [02:01:43] anyways dinner calls, good luck! [02:02:18] ebernhardson: Thanks, enjoy! [02:02:58] !log tools.best-image Installed flask in $HOME/www/python/venv to fix webservice restart loop [02:03:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.best-image/SAL [02:08:16] 06Labs, 10Labs-Infrastructure, 05MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#3182249 (10Andrew) [02:08:33] 06Labs, 10Labs-Infrastructure, 05MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#2773808 (10Andrew) removed todo about service groups because it turns out to not be needed. [02:11:02] 06Labs, 10Labs-Infrastructure: Create top-level domain for a project on project creation - https://phabricator.wikimedia.org/T162977#3182262 (10Andrew) [02:11:17] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3182267 (10Andrew) [02:11:19] 06Labs, 10Labs-Infrastructure, 05MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)), 13Patch-For-Review: Support project creation without OpenStackManager - https://phabricator.wikimedia.org/T150091#3182263 (10Andrew) 05Open>03Resolved I moved the item about domain creation into T162977; the rest... [02:15:23] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: perfectbot runs ListSpeedyDeletions job once per minute - https://phabricator.wikimedia.org/T162978#3182275 (10bd808) [02:19:25] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: request tool runs secWatch job once per minute - https://phabricator.wikimedia.org/T162979#3182288 (10bd808) [02:26:20] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3182308 (10Paladox) How would we import the ssh keys as this is also currently handled by the extension? [03:52:29] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3135041 (10bd808) >>! In T161553#3182308, @Paladox wrote: > @andrew How would we import the ssh keys as this is also currently handled by the extension?... [03:54:59] PROBLEM - Host tools-exec-1433 is DOWN: CRITICAL - Host Unreachable (10.68.22.87) [04:02:44] 06Labs, 10wikitech.wikimedia.org: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3182348 (10bd808) [04:02:46] 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3182349 (10bd808) [04:04:16] 06Labs, 10Striker, 10Tool-Labs, 15User-bd808: Implement Tool Labs membership application and processing in Striker - https://phabricator.wikimedia.org/T162508#3182350 (10bd808) a:03bd808 [04:15:38] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3182355 (10bd808) >>! In T161553#3141039, @Jdforrester-WMF wrote: > I thought there were some quite horrible issues with the shell/account naming stuff?... [04:18:26] 06Labs, 10wikitech.wikimedia.org: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3182370 (10bd808) [04:29:32] 06Labs, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#3182373 (10bd808) [05:48:47] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3182385 (10Dzahn) @bd808 Thank you for the detailed explanation! (PS. Maybe if i can get T113792 solved on the side of this, i'd be happy, hehe :)) [06:13:31] PROBLEM - Puppet run on tools-exec-1430 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:35:38] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:48:30] RECOVERY - Puppet run on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [07:15:42] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [08:09:30] PROBLEM - Puppet run on tools-exec-1430 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:36:39] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: perfectbot runs ListSpeedyDeletions job once per minute - https://phabricator.wikimedia.org/T162978#3182471 (10Fluff) Yeah, should probably be redesigned - old tasks I've inherited from other users. I'll scale down to 5 minutes for now. [08:49:30] RECOVERY - Puppet run on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:58] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: request tool runs secWatch job once per minute - https://phabricator.wikimedia.org/T162979#3182567 (10FNDE) Hi @bd808, thank you for that note. Please have a look at the logfile: ``` 06-04-2017 | 14:46:03 | <--> Start (164301664) 06-04-2017 | 14:46:03 | <--> P... [10:43:27] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: request tool runs secWatch job once per minute - https://phabricator.wikimedia.org/T162979#3182288 (10Betacommand) why not just keep a continuous thread going with a sleep(60) that way you minimize the number of jobs and don't loose functionality. [10:55:32] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3182596 (10Paladox) >>! In T161553#3182344, @bd808 wrote: >>>! In T161553#3182308, @Paladox wrote: >> @andrew How would we import the ssh keys as this i... [11:43:54] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3182686 (10fantasticfears) That helps, thanks :) I would like to ask help for the mail of the infrastructure. As of Discourse, it will need SMTP server for outgoing emails (with verp support) and a... [13:12:34] 10Labs-project-other: Is the 'webplatform' labs project defunct? - https://phabricator.wikimedia.org/T162951#3181213 (10Renoirb) You can close this project. WebPlatform.org is phased out. I guess if I want to contribute as myself, I can do it in another project, right? [13:43:20] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3182889 (10Andrew) [14:25:41] 10Labs-project-other: Is the 'webplatform' labs project defunct? - https://phabricator.wikimedia.org/T162951#3182975 (10Andrew) 05Open>03Resolved @Renoirb, yep! You can always join an existing project or request a new one as a subtask of T76375. [14:38:38] hi - how can I validate the DB acccess rights for a user ? (I want to make sure one tool have access rights to update a DB of a different tool) [14:47:55] 06Labs, 10Tool-Labs, 06Community-Tech: Plagiabot API frequently returns an error - https://phabricator.wikimedia.org/T163001#3183006 (10Ragesoss) [14:53:36] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179794 (10Marostegui) In addition to what Jaime said looks like the server has been crashing lately: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fop... [15:03:27] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179794 (10eranroz) I may have fixed it: ``` repair table copyright_diffs; +-------------------------------------+--------+----------+-------------------------------------------------------... [15:12:00] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3183067 (10eranroz) >>! In T162932#3181900, @jcrespo wrote: > Cannot reproduce: > > > ``` > root@labsdb1001[s51306__copyright_p]> SELECT * FROM copyright_diffs ORDER BY diff_timestamp DES... [15:13:25] PROBLEM - Puppet run on tools-exec-1431 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:32:10] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: request tool runs secWatch job once per minute - https://phabricator.wikimedia.org/T162979#3183089 (10FNDE) @Betacommand: well, a continuous would be okay, instead of a recurring job in the grid? Thanks! [15:32:57] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3183091 (10Marostegui) Yes, you can use `alter table s51306__copyright_p.copyright_diffs ENGINE=InnoDB;` to migrate it to InnoDB if you like. [15:48:25] RECOVERY - Puppet run on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:28] !log tools.alaabot Killed job 3912666 cron-tools - seemed to be stuck in some endless wait loop on /usr/bin/yes causing IOWait to spike on tools-exec-1415 [17:27:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.alaabot/SAL [17:28:27] madhuvishy: question does k8s stuff not show up on the status page on tools.wmflabs.org? [17:30:00] Zppix: nope - I think that just shows grid status [17:33:09] Hi all, is it possible to force HTTPS for my toollabs tool in .lighttpd.conf? [17:36:02] Urbanecm: i dont believe so... [17:36:18] Zppix: And do you know any other method how can I do it? [17:36:39] have access to the root domain of tools.wmflabs.org i believe [17:36:54] I mean some available way for me as the tool owner. [17:37:04] why do you need https? [17:37:54] !log tools.alaabot Also killed 3912152 - was stuck in Ds on tools-exec-1423 - tools.alaabot 29083 2.0 0.0 5376 1400 ? Ds 14:59:03 00:03:14 yes [17:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.alaabot/SAL [17:39:00] Why wouldn't https://redmine.lighttpd.net/projects/1/wiki/HowToRedirectHttpToHttps work? [17:39:15] Because HTTPS traffic and HTTP traffic are counted as two different origins by Chrome which make my tool Wikinity throw No 'Access-Control-Allow-Origin' header is present on the requested resource error when visiting through http [17:39:30] You may try it, http://tools.wmflabs.org/wikinity and https://tools.wmflabs.org/wikinity [17:39:31] Reedy: https is usually handled by the root (or the / url not a subdir of a domain) [17:39:45] What [17:39:58] You can have non https root and https subdomains [17:40:15] Reedy: when I insert the code to the .lighttpd.conf the browser complains about too many redirects. [17:40:38] It redirect to https but then redirect to https again and again and again... [17:41:26] Reedy: oh nevermind im thinking of ssl [17:41:28] Is that because ssl termination is done somewhere else infront of lighttpd? [17:41:50] For most purposes, there's no difference between ssl and https [17:41:55] Zppix: I can send Access-Control-Allow-Origin instead but having the traffic encrypted is better I think [17:42:33] probably yes [17:42:44] Zppix: yes what? Yes it is better or yes I can? [17:42:56] Urbanecm: Might be worth filing a task for labs admins... [17:43:02] Can't be the only tool that wants to force https [17:43:26] !log wikilabels staged wikilabels-wmflabs-deploy:01bf4bc [17:43:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [17:43:46] What should be the task about? About "allowing tool's maintainers to force HTTPS"? [17:44:06] 06Labs, 06Operations, 10hardware-requests: Codfw: (2) hardware access request for labtest [region 2] - https://phabricator.wikimedia.org/T161766#3142263 (10RobH) [17:44:22] !log wikilabels deployed wikilabels-wmflabs-deploy:01bf4bc [17:44:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [17:44:31] Yeah, something like that [17:45:07] 06Labs, 06Operations, 10hardware-requests: eqiad: (1) hardware access request for labnodepool1002 - https://phabricator.wikimedia.org/T161753#3141989 (10RobH) [17:45:42] There could be scope to just do it for all tools [17:45:50] But that might break some stuff that requires HTTP for some silly reason [17:46:35] Reedy: If a tool include HTTP-only webpage is loaded using HTTPS the tool get broken. [17:46:53] (or not HTTP only but including it own page over HTTP) [17:48:48] yeah [17:48:55] and they should get a slap for doing that ;) [17:48:57] Urbanecm: i mean i dont if you can but both are better than nothing i would assume [17:49:12] If you include HTTP resources on a HTTPS page, stuff breaks [17:49:16] That's what he's meaning [17:49:30] But at least, being able to have a tool say "yes, make me all https" should be a thing [17:49:46] Yeah, opt-in feature. [17:50:09] wouldnt this be up to security? [17:53:14] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183510 (10Urbanecm) [17:53:17] T163019 ^^ [17:53:17] T163019: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019 [17:57:29] 06Labs, 06Operations, 10hardware-requests: eqiad: (1) hardware access request for labnodepool1002 - https://phabricator.wikimedia.org/T161753#3141989 (10RobH) [17:57:43] 06Labs, 06Operations, 10hardware-requests: eqiad: (1) hardware access request for labnodepool1002 - https://phabricator.wikimedia.org/T161753#3141989 (10RobH) [18:01:18] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183510 (10Freddy2001) Same for Labs Projects please [18:13:08] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183577 (10Reedy) >>! In T163019#3183560, @Freddy2001 wrote: > Same for Labs Projects please Can't they just do them themselves? [18:14:11] No it's not upto security [18:16:10] Urbanecm: Might be worth mentioning in a comment what you tried in your lighttpd conf to force https that just resulted in a redirect loop [18:16:29] Reedy: I'll add it there. [18:18:37] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183608 (10Urbanecm) I've tried to add the following code to my .lighttpd.conf according to https://redmine.lighttpd.net/projects/1/wiki/HowToRedirectHttpToHt... [18:31:26] So, question (And I'm sure I've missed the answer in here so feel free to point me to the task) S1 replag is over 10 hours. Is that a known issue? If so, is there an estimate fix timeframe? S3 is also lagging, by about 20 minutes. [18:35:28] What makes you say that? [18:35:51] Reedy: guess https://tools.wmflabs.org/replag/ [18:35:59] Oh, we're in labs [18:36:03] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183632 (10Freddy2001) >>! In T163019#3183577, @Reedy wrote: >>>! In T163019#3183560, @Freddy2001 wrote: >> Same for Labs Projects please > > Can't they just... [18:36:11] I thought we were talking about production for a minute [18:36:40] Matthew_: FWIW, it's a WMF holiday today [18:37:07] Oh, well that might complicate things. [18:37:43] But yes, I actually noticed it because #wikipedia-en-abuse-log is registering only replag errors. [18:42:04] Let me see if I can raise anyone [18:42:45] If not, it's not the end of the world :) [18:48:57] 06Labs, 10Tool-Labs, 06Community-Tech: Plagiabot API frequently returns an error - https://phabricator.wikimedia.org/T163001#3183646 (10jcrespo) @Anomie - Can I ask you to stop BrokenRedirectDeleter queries? It is correlating to swapping and eventual server crash https://grafana.wikimedia.org/dashboard/file/... [18:52:23] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183648 (10Urbanecm) Seems this Is the same issue for toollabs. [18:52:34] 06Labs, 10Tool-Labs, 06Community-Tech: Plagiabot API frequently returns an error - https://phabricator.wikimedia.org/T163001#3183649 (10jcrespo) If that doesn't help (main user of the server), other heavy hitters are merl tool and some random quarry tools with multiple revision queries (but those doesn't see... [18:56:47] Matthew_: As always, it's not a bad idea to file a task if there's not one there [18:57:40] I will. If another one exists, it can get closed as uplicate. [19:00:38] 06Labs, 10DBA: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3183654 (10Matthewrbowker) [19:00:49] Reedy: ^ [19:01:20] Well, it's improved by 7 hours in half an hour... ;) [19:01:40] Yeah, but S3 doubled xD [19:01:58] I hope it improves to zero and the task can be closed. That would make me happy :) [19:02:24] 06Labs, 10DBA: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3183654 (10jcrespo) It is getting better now. [19:03:12] 06Labs, 10DBA: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3183670 (10Matthewrbowker) Shard 1 is, it has improved by about 7 hours recently. Shard 3 has doubled though, it was at 21 minutes a half hour ago. [19:10:30] 06Labs, 10DBA: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3183678 (10jcrespo) Yep, 2 problems in one. s1 is labsdb1001 crashing regularly in the last 2 days. s3 was the filtering server getting stuck on s3 only due to a table corruption on a TokuDB index. Both solved now. [19:11:02] 06Labs, 10DBA: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3183679 (10jcrespo) I will merge this into a general ticket for the labsdb1001 issues, but lag should be going down now everywhere. [19:12:08] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3183680 (10jcrespo) [19:12:29] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3183006 (10jcrespo) [19:12:30] 06Labs, 10DBA: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3183685 (10jcrespo) [19:15:06] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3183686 (10jcrespo) I have imposed a long running query time of 1500 seconds. I know, not good, but it is either a few queries are affected, or the server keeps crash... [19:15:36] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183687 (10Freddy2001) Might be because Tool Labs uses the infrastructure of Labs. [19:22:28] 06Labs, 10DBA: S1 replag at 3 hours - https://phabricator.wikimedia.org/T163023#3183690 (10Matthewrbowker) @jcrespo Thank you so much for the quick look. [19:25:05] Urbanecm: your tool that you mention in t163019 what is it written in? it looks cool to me [19:32:06] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3183718 (10jcrespo) > It's been down about half the time BTW, the downtime is unacceptable, but based on actual stats, the server only takes minutes to restart and y... [19:46:22] Matthew_: Looks like it's all gone now [19:48:24] Reedy: could you lookinto something abusefilters wont let a user with admin and 'crat user groups edit abusefilters [19:48:47] the user is vito-genovese in #wikimedia-tech [19:50:55] Reedy: sweet! Thank you. [19:58:47] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183743 (10Framawiki) Look at it if it can help you: https://mozilla.github.io/server-side-tls/ssl-config-generator/ [20:25:28] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestneutron refresh - https://phabricator.wikimedia.org/T154706#2921133 (10RobH) [20:25:39] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestnet2003 [region 2] - https://phabricator.wikimedia.org/T161764#3142232 (10RobH) [20:29:58] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: request tool runs secWatch job once per minute - https://phabricator.wikimedia.org/T162979#3183828 (10bd808) The basic point is not the amount of CPU or wall clock time that a job uses to do its useful work. It's more that the act of asking the grid to schedule,... [20:32:10] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183510 (10valhallasw) >>! In T163019#3183608, @Urbanecm wrote: > I've tried to add the following code to my .lighttpd.conf according to https://redmine.light... [20:44:29] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3183886 (10bd808) >>! In T161553#3182596, @Paladox wrote: > But that sounds like tool related and not labs. [[https://en.wikipedia.org/wiki/A_rose_by_a... [20:46:28] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3183888 (10Paladox) Oh nope the host name doesn't bother me, just was confusing on weather that was for tools only. but thanks for explaining. [20:52:16] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183892 (10Urbanecm) Do you know how can I add the redirect rule to lighttpd config? Or do it need to be set somewhere else? [21:09:24] PROBLEM - Puppet run on tools-exec-1431 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:35:23] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3183923 (10Anomie) Ok, I paused it for the moment. I hope I don't have to keep it paused for too long. It might restart on its own if Tool Labs decides to restart the... [21:44:26] RECOVERY - Puppet run on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [22:39:51] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestvirt2003 [region 2] - https://phabricator.wikimedia.org/T161765#3184042 (10RobH) [22:40:03] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestvirt2003 [region 2] - https://phabricator.wikimedia.org/T161765#3142249 (10RobH) [22:45:16] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labcontrol1003/1004 - https://phabricator.wikimedia.org/T158207#3029754 (10RobH) [22:45:28] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestvirt2003 [region 2] - https://phabricator.wikimedia.org/T161765#3184051 (10RobH) [23:50:34] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3184099 (10jcrespo) Some of it could be the limiter I imposed (maybe it can be done in batches of less time?). Please force the use of InnoDB for recreation- InnoDB t...