[06:53:04] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.016 second response time
[07:23:07] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.027 second response time
[07:26:05] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: erwin's tools doesn't show a skin - https://phabricator.wikimedia.org/T146569#2665137 (10Akoopal)
[08:09:32] <shinken-wm>	 PROBLEM - Puppet staleness on tools-checker-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[10:45:23] <wikibugs>	 10Tool-Labs-tools-stewardbots: Automatically start the irc bots - https://phabricator.wikimedia.org/T144461#2665305 (10MarcoAurelio) A couple of days ago I received an email from bigbrother saying that it restarted a job. Maybe that means that this is now working as expected?
[10:59:57] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Shell script for extracting the last full log [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312207 (owner: 10Lokal Profil)
[11:00:39] <grrrit-wm>	 (03Merged) 10jenkins-bot: Shell script for extracting the last full log [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312207 (owner: 10Lokal Profil)
[12:16:26] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: erwin's tools doesn't show a skin - https://phabricator.wikimedia.org/T146569#2665393 (10Akoopal) 05Open>03Resolved a:03Akoopal With the help of Krenair, pointed to alternatives via en-wiki. This is not guaranteed to keep working, but for now it works again.
[12:44:55] <shinken-wm>	 RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms
[12:51:24] <hidaz>	 hi
[12:51:40] <hidaz>	 hi
[13:48:19] <shinken-wm>	 PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22)
[13:55:24] <wikibugs>	 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review, 07Technical-Debt, 05WMF-deploy-2016-10-04_(1.28.0-wmf.21): OpenStackManager should not call WikiPage::getText() - https://phabricator.wikimedia.org/T145753#2665438 (10enigmaeth) 05Open>03Resolved a:03enigmaeth
[15:23:37] <halfak>	 !log ores restarted precached and uwsgi-ores on ores-web-03
[15:24:30] <halfak>	 ...
[15:25:24] <Krenair>	 fixing
[15:26:21] <Krenair>	 halfak, try now
[15:26:29] <halfak>	 !log ores restarted precached and uwsgi-ores on ores-web-03
[15:26:32] <halfak>	 Thanks Krenair 
[15:26:33] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master
[15:27:06] <Krenair>	 !log tools restarted labs-logbot under tools.morebots
[15:27:10] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[15:59:41] <Amir1>	 !log ores c2ba205 going to staging
[15:59:45] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master
[16:02:34] <Amir1>	 everything looks fine, deploying to ores.wmflabs.org
[16:03:12] <Amir1>	 !log ores deploying c2ba205 (T146581)
[16:03:15] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master
[16:03:20] <stashbot>	 T146581: celery log level is INFO causing disruption on ORES service  - https://phabricator.wikimedia.org/T146581
[16:49:54] <shinken-wm>	 PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[17:24:51] <shinken-wm>	 RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:59:55] <wikibugs>	 10Labs-Kubernetes, 06Wikisource, 03Community-Tech-Sprint: Make Google OCR API on Tool Labs work under Kubernetes - https://phabricator.wikimedia.org/T146311#2665661 (10Bodhisattwa)
[18:20:57] <shinken-wm>	 RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms
[18:23:42] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: Erwin's tools not working for nl.wikipedia.org - https://phabricator.wikimedia.org/T146590#2665682 (10Akoopal)
[18:24:21] <valhallasw`cloud>	 akoopal: 
[18:24:42] <valhallasw`cloud>	 could you add some more details? e.g. link, input, etc?
[18:25:34] <akoopal>	 :-)
[18:26:40] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: Erwin's tools not working for nl.wikipedia.org - https://phabricator.wikimedia.org/T146590#2665695 (10Akoopal) For an example, the link I use to see changes related to the 'Molen' category:  https://tools.wmflabs.org/erwin85/relatedchanges.php?lang=nl&family=wikipedia&category=Mo...
[18:26:43] <akoopal>	 in the ticket
[18:36:10] <shinken-wm>	 RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms
[18:38:36] <shinken-wm>	 PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170)
[19:01:08] <shinken-wm>	 PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218)
[19:37:20] <shinken-wm>	 PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[21:30:39] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: Erwin's tools not working for nl.wikipedia.org - https://phabricator.wikimedia.org/T146590#2665794 (10Akoopal) 05Open>03Resolved a:03Akoopal With the help of valhallas, solved. The query to get the database was not specific enough due to which it queried the database for nd...
[21:49:59] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: For related changes make category changes switchable. - https://phabricator.wikimedia.org/T146594#2665823 (10Akoopal)