[00:12:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Naraht was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1199452 edit summary: [00:27:11] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:00] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.048 second response time [00:41:04] 06Labs, 10wikitech.wikimedia.org: Can't login into a newly created nova instance through ssh - https://phabricator.wikimedia.org/T113451#2891676 (10Krenair) a:03devunt assigning to author for information [00:43:07] 06Labs, 10wikitech.wikimedia.org: Can't login into a newly created nova instance through ssh - https://phabricator.wikimedia.org/T113451#2891679 (10Krenair) 05Open>03Resolved a:05devunt>03None actually, no, from reviewing the log it looks like this was a temporary issue (unable to connect to LDAP) and... [00:45:04] 06Labs, 10Labs-Infrastructure, 10Monitoring, 06Operations: Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656#2891683 (10AlexMonk-WMF) Basic check added in T42022. My question above should be pretty easy though. [00:45:48] 06Labs, 10Labs-Sprint-109, 10labs-sprint-116, 10labs-sprint-117, and 3 others: Monitor nova services - https://phabricator.wikimedia.org/T90784#2891686 (10AlexMonk-WMF) Basic API HTTP code check added in T42022 [00:58:55] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Demo Unicorn was modified, changed by Demo Unicorn link https://wikitech.wikimedia.org/w/index.php?diff=1199892 edit summary: [01:00:53] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Demo Unicorn was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1199921 edit summary: [01:29:42] 06Labs, 06Operations: openstackclient/keystoneclient on silver broken - https://phabricator.wikimedia.org/T153807#2891728 (10AlexMonk-WMF) [01:35:07] (03CR) 10BryanDavis: "I tested the LDAP/openssh side of handling the new key types by manually adding an ed25519 key to my `dunicorn` alt account using ldapvi a" [labs/striker] - 10https://gerrit.wikimedia.org/r/328403 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [01:36:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:11:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:21:13] (03CR) 10Legoktm: [C: 04-1] "I archived the project, so this should probably be abandoned now." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307441 (owner: 10Paladox) [02:22:29] (03CR) 10Legoktm: [C: 04-1] "@Hashar: As it's a gerrit plugin, I assume releng maintains it. So probably -releng is the rign place for this?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307439 (owner: 10Paladox) [02:37:43] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:45:42] "Ensure mysql credential creation for tools users is running" is CRIT [02:45:47] on labstore1004 [02:46:10] no idea if that's important [02:47:48] it's succeeding on labstore1005 it seems [02:48:24] "unit create-dbusers is inactive" [02:48:28] 1004 has been broken for 9 hours [02:50:55] don't really know what might cause that sort of thing, would have to ask ops to debug [02:52:05] 06Labs, 10Tool-Labs: Make switching Redis server simpler - https://phabricator.wikimedia.org/T153810#2891805 (10scfc) [02:53:34] 06Labs, 10Tool-Labs: Make switching Redis server simpler - https://phabricator.wikimedia.org/T153810#2891823 (10scfc) [02:53:57] Krenair: i just saw that it has puppet disabled with reason "yuvi testing". so i will just consider it part of testing [02:54:11] ah [02:54:19] it's possible he was working then got knocked offline [02:54:23] yea [02:54:41] this service is not something that affects existing tools [02:54:46] just new ones being made [02:54:57] ok [02:55:18] so it can probably be left a little while for one of the labs ops to get online and look [02:55:46] good [03:02:07] 10Tool-Labs-tools-Other: SVG Translate does not accept HTTPS URLs - https://phabricator.wikimedia.org/T125743#2891843 (10Glrx) My gut tells me this is a duplicate of T138780 which has been resolved. The filename would enter correctly, but when the PHP script searched the file, it would throw an Intuition error... [03:16:27] mutante, Krenair: Yuvi has been working on a replacement script for managing credentials. I would guess he it running it on 1004 to make sure things are working as expected. [03:28:38] 06Labs, 10Labs-Infrastructure, 06Community-Tech-Tool-Labs: invisible-unicorn (dynamicproxy) should provide an easy way to see where a host routes without knowing the project - https://phabricator.wikimedia.org/T115752#2891877 (10bd808) Thanks to recent work by @Andrew and @Krenair's keystone knowledge there... [03:37:43] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:39:08] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Move tools-db and tools-redis into DNS - https://phabricator.wikimedia.org/T139190#2891884 (10scfc) [03:39:11] 06Labs, 10Tool-Labs: Make switching Redis server simpler - https://phabricator.wikimedia.org/T153810#2891883 (10scfc) [04:03:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:38:43] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:58:11] 06Labs, 10Tool-Labs: Make role::aptly::server work with Apache - https://phabricator.wikimedia.org/T153814#2891947 (10scfc) [05:09:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:34:24] 06Labs, 10wikitech.wikimedia.org: SRF preference messages broken - https://phabricator.wikimedia.org/T128027#2061756 (10MZMcBride) From : {F5115288} This is silly. [05:40:36] How do I add a user to a project? [05:40:53] I've been looking on https://wikitech.wikimedia.org/ for like 10 minutes and still can't find it. [05:41:04] I'm a member of the "dbreps" project and want to add someone else. [05:42:44] Oh, at https://tools.wmflabs.org/?list of course. [05:45:34] 06Labs, 10Tool-Labs: apache::static_site should provide a way to work without SSL or with automatically created self-signed certificates - https://phabricator.wikimedia.org/T153818#2892058 (10Aklapper) Hi @scfc. Please associate at least one [[ https://phabricator.wikimedia.org/project/query/G9vp6zKs.IfE/#R |... [05:46:27] Service groups, hrmph. [05:58:17] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2892062 (10yuvipanda) This has been deployed now, and all tools have a... [06:01:38] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2892079 (10yuvipanda) I still need to figure out best way of making th... [06:05:23] 06Labs, 10Labs-Infrastructure, 06Community-Tech-Tool-Labs: invisible-unicorn (dynamicproxy) should provide an easy way to see where a host routes without knowing the project - https://phabricator.wikimedia.org/T115752#2892083 (10scfc) I don't think it's necessary to access Keystone for a list of projects. Y... [06:11:53] 06Labs, 10Analytics, 10Pageviews-API, 10wikitech.wikimedia.org: wikitech.wikimedia.org missing from pageviews API - https://phabricator.wikimedia.org/T153821#2892095 (10Krinkle) [06:26:10] 10Tool-Labs-tools-Database-Queries, 06Project-Admins: Archive Tool-Labs-tools-Database-Queries project - https://phabricator.wikimedia.org/T107699#2892139 (10Aklapper) @Bugreporter: Any comments / input to my last comment, please? If not I'm going to decline this task. [06:39:59] 06Labs, 10Tool-Labs, 10DBA, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#2892193 (10Krinkle) [07:58:51] 10Tool-Labs-tools-Other: SVG Translate does not accept HTTPS URLs - https://phabricator.wikimedia.org/T125743#2892311 (10Jdx) Full HTTPS URLs still don't work – "error-notsvg Hit your browser's back button to retry." is displayed. Wiki links work at first glance, but I doubt it works at all – when I translated B... [08:56:01] (03CR) 10Gergő Tisza: "Uh, sorry, did not realize you were waiting for me! Yeah, looks fine to me." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/320363 (https://phabricator.wikimedia.org/T137882) (owner: 10Jean-Frédéric) [08:56:08] (03CR) 10Gergő Tisza: [C: 031] Set-up Raven client for error reporting via Sentry [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/320363 (https://phabricator.wikimedia.org/T137882) (owner: 10Jean-Frédéric) [09:01:27] (03CR) 10Jean-Frédéric: [C: 032] Set-up Raven client for error reporting via Sentry [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/320363 (https://phabricator.wikimedia.org/T137882) (owner: 10Jean-Frédéric) [09:01:58] (03CR) 10Jean-Frédéric: [C: 032] "> Uh, sorry, did not realize you were waiting for me! Yeah, looks fine to me." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/320363 (https://phabricator.wikimedia.org/T137882) (owner: 10Jean-Frédéric) [09:02:26] (03Merged) 10jenkins-bot: Set-up Raven client for error reporting via Sentry [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/320363 (https://phabricator.wikimedia.org/T137882) (owner: 10Jean-Frédéric) [09:34:08] !log tools.heritage Deploy latest from Git master: 26a5049 (T137882) [09:34:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [09:34:11] T137882: Set-up Sentry for Heritage API - https://phabricator.wikimedia.org/T137882 [09:53:33] 06Labs, 10Tool-Labs, 10DBA: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2892615 (10Marostegui) @yuvipanda do you think we can kill these two users? (after holidays period) [10:27:49] 06Labs, 10Analytics, 10Pageviews-API, 10wikitech.wikimedia.org: wikitech.wikimedia.org missing from pageviews API - https://phabricator.wikimedia.org/T153821#2892095 (10Krenair) {T102178} [10:56:25] 10Tool-Labs-tools-Other: s51412__data reverted to last year state - https://phabricator.wikimedia.org/T150759#2892909 (10jcrespo) @APPER Everything ok now? [11:01:07] (03PS3) 10Lokal Profil: Disallow adding hidden categories [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/328378 (https://phabricator.wikimedia.org/T153746) [12:07:02] 06Labs, 10Labs-Infrastructure, 07Upstream: vmbuilder fails for Trusty images at /etc/sudoers - https://phabricator.wikimedia.org/T153853#2893098 (10scfc) [12:14:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:21:08] 06Labs, 10Tool-Labs: apache::static_site should provide a way to work without SSL or with automatically created self-signed certificates - https://phabricator.wikimedia.org/T153818#2893128 (10scfc) Hmmm. Nothing really fits, but (one of) the end consumer will be #Tool-Labs, so it's not that wrong. [12:24:44] 06Labs, 10Tool-Labs, 10DBA: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2893147 (10yuvipanda) @Marostegui yup! [12:25:29] 06Labs, 10Tool-Labs, 10DBA: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2893148 (10Marostegui) Great - thanks! I will get rid of them after freeze [12:34:26] 06Labs, 10Tool-Labs, 10DBA: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2893169 (10Marostegui) This is what will be done ``` root@neodymium:~# host labstore1001 labstore1001.eqiad.wmnet has address 10.64.37.6 root@neodymium:~# host labstore1002 labstore100... [12:38:37] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Celia sumbane was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1203807 edit summary: [12:40:52] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Brynda1231 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1203845 edit summary: [13:05:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:15:41] did i do this right? https://phabricator.wikimedia.org/T153711 [15:15:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:16:12] i couldn't find the template i used last time, not sure if that makes a difference [15:19:15] urandom: it's here https://phabricator.wikimedia.org/T140904 :) [15:19:38] aha, so it makes it a child of this one [15:20:06] not really "makes", but yeah [15:20:37] we need to surface this better and convert a tracking task ta project but yes :) [15:21:11] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2893583 (10Eevans) [15:21:13] 06Labs: Request increased quota for services-test labs project - https://phabricator.wikimedia.org/T153711#2893582 (10Eevans) [15:21:23] chasemp: is that enough? or should i create a new issue? [15:22:41] urandom: makes sense I think, it may be delayed due to holidays and getting feedback from us [15:22:44] but we'll sort it out [15:22:47] k [16:06:29] 06Labs, 06Operations, 13Patch-For-Review: openstackclient/keystoneclient on silver broken - https://phabricator.wikimedia.org/T153807#2893685 (10Andrew) 05Open>03Resolved a:03Andrew Attached patch plus a bit of fussing with packages (python-oslo-serialization vs python-oslo.serialization) and this is f... [16:06:40] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:39:35] 06Labs, 10Labs-Infrastructure, 10DBA, 07Epic, 07Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2893764 (10chasemp) [16:39:38] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provide at least 2 separate service endpoints: one for slow, long running queries; and another for quick, web requests - https://phabricator.wikimedia.org/T147051#2893761 (10chasemp) 05Open>03Resolved a:03chasemp The service endpoints establis... [16:40:30] 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2893769 (10chasemp) [16:40:34] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production - https://phabricator.wikimedia.org/T147052#2893765 (10chasemp) 05Open>03Resolved a:03chasemp enwiki_p is now fu... [16:43:56] 06Labs, 10Labs-Infrastructure, 10DBA, 07Epic, 07Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2893778 (10chasemp) [16:43:59] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement a frontend failover solution for labsdb replicas - https://phabricator.wikimedia.org/T141097#2893774 (10chasemp) 05Open>03Resolved We have settled on DNS based failover with haproxy for the time being between the two dbproxy hosts (101... [16:53:40] Yvette: I'm hoping to rework all of the tool membership management stuff "soon". First I have to finish the account creation workflow and ship it (its close!) [16:54:19] If you are curious, the new workflow is on the demo server at https://striker.wmflabs.org/ [17:09:39] 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2893829 (10jcrespo) @scfc This is technically solved on the new servers, enwiki is fine there, but we most likely will not fix them on the current servers. When run on enwiki production master I get:... [17:10:22] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision with data the new labsdb servers and provide replica service with at least 1 shard from a sanitized copy from production - https://phabricator.wikimedia.org/T147052#2893833 (10jcrespo) [17:10:26] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2893831 (10jcrespo) 05Open>03Resolved a:03jcrespo [17:17:14] 06Labs, 10DBA: Missing data on labs replica database - https://phabricator.wikimedia.org/T133715#2893851 (10jcrespo) 05Open>03Resolved The revision is available on the new labsdb servers (which will fix now and in the future any drift with production). This is difficult to solve on the current (old) server... [17:17:17] 06Labs, 10DBA, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#2893853 (10jcrespo) [17:20:19] jynus: hi! [17:20:34] I see you just closed that replication issue on Phabricator. [17:21:55] 06Labs, 10Tool-Labs, 10DBA: Tool Labs queries die - https://phabricator.wikimedia.org/T127266#2893866 (10jcrespo) 05Open>03Resolved a:03jcrespo Dispenser, we just finished setting up 3 new labsdb servers with 2 separate entry points- one for fast webrequests, and another for analytics-like queries. On... [17:22:14] Will existing connection to the replica db on labs just switch over to the new servers at some point, or will a configuration change be required? [17:24:56] ragesoss, we have yet to think the proper way- which will require feedback [17:25:00] for now it will opt in [17:25:08] eventually it will, as you said [17:25:29] okay. and announcement about how to opt in will be very soon, I take it? [17:25:29] the new replicas do not yet contain all data [17:25:34] yes [17:25:36] cool. [17:25:39] we are in christmas season [17:25:47] I'll switch my tools over once the announcement comes. [17:25:49] so we may delay until kanuary [17:25:50] thanks much! [17:26:00] but I wanted to notify tose who reported issues [17:26:07] that those are solved [17:26:12] thank to you for reporting [17:26:31] :-) [17:30:59] 06Labs, 10DBA, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#2893895 (10jcrespo) [17:31:03] 06Labs, 10DBA: Data missing from June 11/12 on s3.labsdb - https://phabricator.wikimedia.org/T115517#2893892 (10jcrespo) 05Open>03Resolved a:03jcrespo These are the results on the new replica service servers: ``` root@localhost[srwiki_p]> select max(rev_timestamp) as max_rev, min(rev_timestamp) as min_r... [17:35:03] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2893932 (10jcrespo) This is almost done (the table is available on the current labsdb servers), but to close it we need to: * Puppetize thinking there may be other tables like th... [17:35:38] 06Labs, 10Labs-Infrastructure, 10DBA, 07Epic, 07Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#2893936 (10jcrespo) [17:42:30] paladox: fyi, a .pyc file is a compiled python file, they're generated dynamically as .py files are loaded. [17:42:44] oh, thanks. [17:46:28] 06Labs, 10Labs-Infrastructure, 10DBA: Explore 'Analyze' statement as substitute for Explain - https://phabricator.wikimedia.org/T141095#2893989 (10jcrespo) 10.1 Analyze requires also SELECT grants for the underlying tables, so it cannot be used. This doesn't take away from the fact that there is already a wo... [17:46:45] (and updated dynamically) [18:11:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:03:02] 06Labs, 15User-Urbanecm: Data disappeared from labs replica in cswiki_p.page_props - https://phabricator.wikimedia.org/T153888#2894171 (10Urbanecm) [19:03:13] 06Labs, 10Tool-Labs, 15User-Urbanecm: Data disappeared from labs replica in cswiki_p.page_props - https://phabricator.wikimedia.org/T153888#2894184 (10Urbanecm) [19:07:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:13:40] 06Labs, 10Labs-Infrastructure, 10DBA: Explore 'Analyze' statement as substitute for Explain - https://phabricator.wikimedia.org/T141095#2894390 (10yuvipanda) @jcrespo what do you think of a http service that is hosted on hardware somewhere and maintained by the labs team, that lets you send it a query and gi... [22:12:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:22:38] 10Labs-project-other, 10Wikimedia-Mailing-lists: Wikimedia-l <-> Discourse synchronization (PRELIMINARY) - https://phabricator.wikimedia.org/T126458#2894810 (10Dzahn) [22:31:22] 10Labs-project-Wikistats: all kinds of mixed issues with miraheze table (was: allthetropes is not updating on wikistats) - https://phabricator.wikimedia.org/T146712#2894824 (10Dzahn) opened https://phabricator.miraheze.org/T1176 downstream to provide a dblist including the custom domain names, otherwise it's not... [22:38:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:50:40] 06Labs, 10Tool-Labs, 15User-Urbanecm: Data disappeared from labs replica in cswiki_p.page_props - https://phabricator.wikimedia.org/T153888#2894858 (10Blahma) This broke my tool that generates a map of geographical articles on cswiki with no page image. On production, page images are still shown, so the info... [22:51:22] 06Labs, 10Tool-Labs, 10PageImages, 15User-Urbanecm: Data disappeared from labs replica in cswiki_p.page_props - https://phabricator.wikimedia.org/T153888#2894860 (10Blahma) [23:27:38] 10Labs-project-Wikistats, 13Patch-For-Review: wikistats - make table columns sortable in display.php - https://phabricator.wikimedia.org/T153762#2894941 (10Dzahn) tables like http://wikistats.wmflabs.org/display.php?t=wp are now sortable and get the datatables search field etc see gerrit change for details [23:27:45] 10Labs-project-Wikistats, 13Patch-For-Review: wikistats - make table columns sortable in display.php - https://phabricator.wikimedia.org/T153762#2894942 (10Dzahn) 05Open>03Resolved a:03Dzahn [23:27:54] 10Labs-project-Wikistats: wikistats - make table columns sortable in display.php - https://phabricator.wikimedia.org/T153762#2890243 (10Dzahn) [23:43:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0]