[00:40:30] PROBLEM - Puppet run on tools-exec-1430 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:11:54] 06Labs, 10Labs-Infrastructure, 10Datasets-Archiving, 10Datasets-General-or-Unknown, 10Wikidata: [Bug] Wikidata JSON dumps gets deleted after every new Wikidata dump - https://phabricator.wikimedia.org/T107226#3184140 (10Hydriz) 05Open>03Resolved a:03Hydriz This has been resolved quite a while ago,... [01:20:30] RECOVERY - Puppet run on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [01:41:30] PROBLEM - Puppet run on tools-exec-1430 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:04:38] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3184166 (10Anomie) I thought InnoDB was the default. I'll have it explicitly specify that. I'd be happy to change it to use less memory, but I have no idea what abou... [02:28:59] 06Labs, 10Tool-Labs, 06Community-Tech: labsdb1001 crashing regularly in the last 2 days due to OOM - https://phabricator.wikimedia.org/T163001#3184184 (10Anomie) And to reiterate, according to my logs my bot was failing at the `SELECT COUNT(*) FROM $schema.BrokenRedirectDeleter;` line for almost all of the t... [03:21:29] RECOVERY - Puppet run on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [05:57:53] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3184264 (10eranroz) OK, I changed the engine InnoDB. Hopefully this should make it more stable to prevent further cases [07:01:52] !log tools.linkwatcher Killed job 25739 since the tool seemed to spawning too many perl processes that are stuck in sleep https://phabricator.wikimedia.org/P5278 [07:01:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.linkwatcher/SAL [09:21:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Snd96 was created, changed by Snd96 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Snd96 edit summary: Created page with "{{Tools Access Request |Justification=To create open source instructional software for Wikiversity, Wiktionary, Wikipedia, and Moodle proposals. |Completed=false |User Name=Sn..." [09:30:48] RECOVERY - Host tools-exec-1433 is UP: PING OK - Packet loss = 0%, RTA = 1.85 ms [10:00:46] PROBLEM - Host tools-exec-1433 is DOWN: CRITICAL - Host Unreachable (10.68.22.87) [13:13:52] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3184466 (10Andrew) Petrb has confirmed that service groups in the 'bots' project are no longer needed. [15:12:32] PROBLEM - Puppet run on tools-exec-1430 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:20:05] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:47:32] RECOVERY - Puppet run on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:05] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:37] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3184653 (10Marostegui) >>! In T162932#3184264, @eranroz wrote: > OK, I changed the engine InnoDB. Hopefully this should make it more stable to prevent further cases Just to be clear, proba... [17:50:30] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3184660 (10Freddy2001) >>! In T163019#3183892, @Urbanecm wrote: > Do you know how can I add the redirect rule to lighttpd config? Or do it need to be set some... [17:59:37] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3184675 (10Urbanecm) >>! In T163019#3183892, @Urbanecm wrote: > Do you know how can I add the redirect rule to lighttpd config? Or do it need to be set somewh... [18:03:01] PROBLEM - High iowait on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: tools.tools-webgrid-generic-1402.cpu.total.iowait (>10.00%) [18:13:03] RECOVERY - High iowait on tools-webgrid-generic-1402 is OK: OK: All targets OK [18:34:06] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3184681 (10Urbanecm) >>! In T163019#3184660, @Freddy2001 wrote: >>>! In T163019#3183892, @Urbanecm wrote: >> Do you know how can I add the redirect rule to li... [18:39:12] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3184682 (10Freddy2001) Where do I get this information in the header? [19:40:53] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3184699 (10Urbanecm) This is a HTTP header. For example in PHP you may do the following ``` if (getallheaders()['X-Forwarded-Proto'] == "http") { header("Lo... [19:42:04] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3184700 (10Urbanecm) Note that the code must be at the beginning of your PHP code before any other output. Otherwise the location thing would just appear on t... [23:13:57] 06Labs, 10Tool-Labs, 05Security, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3183510 (10Bawolff) Slightly off topic to this bug, but why do we even allow non secure http on tool labs? Https-only (+hsts) is quickly becoming the norm for...