[00:42:02] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:58:50] 06Labs, 10Tool-Labs: Flannel etcd nodes CRIT on checker.tools.wmflabs.org - https://phabricator.wikimedia.org/T149072#2741012 (10Dzahn) [04:01:29] 06Labs, 10Tool-Labs: Flannel etcd nodes CRIT on checker.tools.wmflabs.org - https://phabricator.wikimedia.org/T149072#2741028 (10Dzahn) p:05Triage>03Low [04:19:22] !log tools reboot tools-flannel-etcd-01 for https://phabricator.wikimedia.org/T149072#2741012 [04:19:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [04:20:17] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2741036 (10yuvipanda) https://phabricator.wikimedia.org/T149072#2741012 is probably this too. Given that it looks like we don't have the resources to actually investigate this and this... [04:22:21] RECOVERY - Host tools-flannel-etcd-01 is UP: PING OK - Packet loss = 0%, RTA = 1.32 ms [04:26:19] RECOVERY - SSH on tools-flannel-etcd-01 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [04:38:21] RECOVERY - Puppet staleness on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [04:39:08] !log tools.stashbot Restarting to test new file organization [04:39:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [04:57:37] !log tools.stashbot That took a bit longer than I was hoping. [04:57:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [05:10:54] 06Labs, 10Tool-Labs: Flannel etcd nodes CRIT on checker.tools.wmflabs.org - https://phabricator.wikimedia.org/T149072#2741079 (10Dzahn) 05Open>03Resolved a:03Dzahn Status Information: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 1.896 second response time [05:11:07] 06Labs, 10Tool-Labs: Flannel etcd nodes CRIT on checker.tools.wmflabs.org - https://phabricator.wikimedia.org/T149072#2741082 (10Dzahn) a:05Dzahn>03yuvipanda [07:22:04] (03CR) 10Lokal Profil: [C: 032] Harvest Wikidata item for es-ct (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317538 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:23:10] (03Merged) 10jenkins-bot: Harvest Wikidata item for es-ct (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317538 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:36:32] (03PS4) 10Lokal Profil: Enable YAML linting via tox using yamllint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [07:38:43] (03CR) 10Lokal Profil: [C: 032] Harvest Wikidata item for de-he (de) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317580 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:39:11] (03CR) 10Lokal Profil: [C: 032] Harvest Wikidata item for es (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317579 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:41:12] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement proxysql both for labs and for later production usage - https://phabricator.wikimedia.org/T148500#2724798 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['dbproxy1011.eqiad.wmne... [07:43:37] (03CR) 10Lokal Profil: [C: 032] Harvest Wikidata item for ad (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317578 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:43:40] (03CR) 10Lokal Profil: [C: 032] Harvest Wikidata item for fr (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317577 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:44:26] (03CR) 10Lokal Profil: [C: 032] Harvest Wikidata item for es-vc (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317576 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:45:11] (03Merged) 10jenkins-bot: Harvest Wikidata item for es-vc (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317576 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:45:15] (03Merged) 10jenkins-bot: Harvest Wikidata item for fr (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317577 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:46:52] (03Merged) 10jenkins-bot: Harvest Wikidata item for ad (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317578 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:48:14] (03Merged) 10jenkins-bot: Harvest Wikidata item for es (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317579 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [07:48:49] (03CR) 10Lokal Profil: [C: 031] Enable YAML linting via tox using yamllint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [07:48:57] (03Merged) 10jenkins-bot: Harvest Wikidata item for de-he (de) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317580 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [08:05:09] 06Labs, 10DBA: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#2741130 (10jcrespo) a:05jcrespo>03None [08:09:52] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Implement proxysql both for labs and for later production usage - https://phabricator.wikimedia.org/T148500#2741133 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dbproxy1011.eqiad.wmnet'] ``` and were **ALL** successful. [09:36:40] 10Tool-Labs-tools-Pageviews, 06Community-Tech, 05Goal: Integrate Pageviews Analysis with Page Pile - https://phabricator.wikimedia.org/T134920#2741266 (10Magnus) Addendum: "limit" parameter suggested by @MusikAnimal has now been implemented. [11:40:36] (03PS1) 10Addshore: Report InterwikiSorting extension to wmde tech channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/317808 [11:43:13] (03PS1) 10Addshore: Report Cognate to wmde tech chan [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/317809 [11:43:15] (03PS1) 10Addshore: Report WMDE team boards to wmde tech chan [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/317810 [11:53:53] (03CR) 10Tobias Gritschacher: [C: 031] Report Cognate to wmde tech chan [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/317809 (owner: 10Addshore) [11:54:15] (03CR) 10Tobias Gritschacher: [C: 031] Report WMDE team boards to wmde tech chan [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/317810 (owner: 10Addshore) [11:55:13] (03CR) 10Tobias Gritschacher: [C: 032] Report InterwikiSorting extension to wmde tech channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/317808 (owner: 10Addshore) [11:55:40] (03Merged) 10jenkins-bot: Report InterwikiSorting extension to wmde tech channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/317808 (owner: 10Addshore) [12:01:03] PROBLEM - Puppet staleness on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [12:46:49] 10Tool-Labs-tools-Pageviews, 03Community-Tech-Sprint: Data since 16 October not showing for some users - https://phabricator.wikimedia.org/T149058#2740438 (10He7d3r) This is the graph I see on Firefox 49 for [[https://tools.wmflabs.org/pageviews/?project=pt.wikipedia.org&platform=all-access&agent=user&range=la... [14:15:45] Hi im wondering has anyone brought this idea up with hosting an irc bouncer on wmflabs for wikimedia and voulenteers? [14:17:18] that would be a tou violation [14:20:08] Oh what's a tou? [14:27:30] https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#What_uses_of_Labs_do_we_not_like.3F [14:27:37] PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:37:21] RECOVERY - Host tools-exec-cyberbot is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [14:44:21] PROBLEM - Puppet staleness on tools-exec-cyberbot is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [14:45:35] 06Labs: Request increased quota for services-test labs project - https://phabricator.wikimedia.org/T148869#2735296 (10GWicke) FYI, most of our other work is in the "services" project. IIRC -testbed was a duplicate created around the same time. [14:46:13] PROBLEM - Host tools-exec-cyberbot is DOWN: CRITICAL - Host Unreachable (10.68.16.39) [14:57:59] 10Tool-Labs-tools-Zppixbot, 05Goal: Complete config - https://phabricator.wikimedia.org/T148944#2741834 (10Zppix) Completed keep alive (stop labs from dropping instance using linux's virtual terminal (tmax) [14:58:07] paladox: you are not the first to think about hosting irc bouncers in Labs. It was actually something I asked about very soon after starting here. -- https://www.mediawiki.org/wiki/User:BDavis_(WMF)/Projects/WMF_BCN [14:59:01] The biggest problem with the idea is that it would require holding secrets (users' freenode passwords) in Labs which is very much against the TOU. [14:59:41] it would also be a bit tricky to manage the potential for abuse leading to blocks of the whole bouncer [14:59:52] ^? [15:00:55] quick question i gtg asap but if labs restarts will my bot need auto restarted (im using virtualenv within tmax console) [15:01:02] correction i mean manually [15:02:04] jobs on the grid should get restarted but anything being run manually will not [15:02:16] how do i move it to grid\ [15:02:28] bd808 but you doint have to have freenode passwords on labs [15:02:35] I doint store mine [15:02:39] RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [15:02:49] on my host [15:02:55] Zppix: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid [15:02:58] pl [15:02:59] ok [15:03:07] thanks gtg o [15:03:10] o/* [15:04:16] paladox: that kind of defeats the purpose of a bouncer doesn't it? If you have to interactively authenticate then you will be disconnected until you do that. [15:04:43] Oh not really since you will be logged in just that your client will authenticate [15:04:51] beyond the secret storage, using Labs as a relay to external networks is also against the published TOU [15:04:56] oh [15:05:26] its generally not a solution that we are interested in supporting with donor dolalrs [15:05:31] ok [15:05:32] *dollars [15:05:54] even if you didn't *store* it in labs [15:06:02] exposing it to labs is still a bad idea [15:06:14] oh [15:06:29] paladox: I just didn't want you to think you were crazy for bringing it up. It does seem like a nice thing to offer on the surface. [15:06:37] yep [16:29:07] 10Tool-Labs-tools-Pageviews, 03Community-Tech-Sprint: Data since 16 October not showing for some users - https://phabricator.wikimedia.org/T149058#2742036 (10MusikAnimal) 05Open>03Resolved Fixed with [[ https://github.com/MusikAnimal/pageviews/commit/7523316baef1d7e1ab36c2fe9ae94f577e92c855 | 640d226 ]] T... [16:44:05] !log tools.heritage Deploy latest from Git master: c943e89, e3ba148, 063f9e2 (T148772 & T148773), 1db9f42, fe119e1, de2e590, 82147b8, f6ff350 & 63a1819 (T140795) [16:44:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [17:49:14] my bot hates the grid [17:50:26] he probably feels trapped in such net :P [17:50:58] hell connect then when i run a command hes like screw this coffee break [17:51:26] but on bastion its fine xD [17:51:28] lol [17:51:31] it has too much AI [17:51:36] NOTICE: I am about to reboot labvirt1001 and, with it, tools-bastion-02. [17:51:50] of course [17:52:00] THATS wwhat im on :/ [17:53:43] ... [17:58:20] andrewbogott: can you do a 'wall' on bastion-02 too? [17:58:53] yuvipanda: you mean tools-bastion-02? I send a shutdown notice there, it will turn itself off before we reboot labvirt1001 [17:59:03] yeah [17:59:04] ok [17:59:16] 10Tool-Labs-tools-Zppixbot, 05Goal: Complete config - https://phabricator.wikimedia.org/T148944#2742293 (10Zppix) Tool Grids and sopel (bot software) do not agree with each other... However on bastion it works without errors Current Setup' Tmux running VirtualEnv Running the sopel (installed via virtualenv'... [18:06:50] PROBLEM - Host tools-bastion-02 is DOWN: CRITICAL - Host Unreachable (10.68.16.44) [18:13:55] PROBLEM - Host tools-exec-1416 is DOWN: CRITICAL - Host Unreachable (10.68.23.14) [18:14:01] PROBLEM - Host tools-exec-1417 is DOWN: CRITICAL - Host Unreachable (10.68.23.172) [18:14:15] PROBLEM - Host tools-webgrid-lighttpd-1410 is DOWN: CRITICAL - Host Unreachable (10.68.18.44) [18:14:25] PROBLEM - Host tools-exec-1408 is DOWN: CRITICAL - Host Unreachable (10.68.18.14) [18:15:13] PROBLEM - Host tools-webgrid-generic-1404 is DOWN: CRITICAL - Host Unreachable (10.68.18.53) [18:15:30] PROBLEM - Host tools-webgrid-lighttpd-1409 is DOWN: CRITICAL - Host Unreachable (10.68.18.43) [18:15:44] PROBLEM - Host tools-puppetmaster-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.61) [18:16:02] PROBLEM - Host tools-exec-1209 is DOWN: CRITICAL - Host Unreachable (10.68.17.129) [18:16:04] PROBLEM - Host tools-exec-1206 is DOWN: CRITICAL - Host Unreachable (10.68.17.105) [18:16:10] PROBLEM - Host tools-exec-1218 is DOWN: CRITICAL - Host Unreachable (10.68.18.19) [18:16:12] ^ planned downtime etc [18:16:31] I guess that's why morebots is gone too [18:16:39] should be rescheduled/reappear? [18:16:46] PROBLEM - Host tools-exec-1202 is DOWN: CRITICAL - Host Unreachable (10.68.16.57) [18:16:47] by itself I mean [18:17:00] PROBLEM - Host tools-webgrid-lighttpd-1411 is DOWN: CRITICAL - Host Unreachable (10.68.17.51) [18:17:02] PROBLEM - Host tools-exec-1213 is DOWN: CRITICAL - Host Unreachable (10.68.17.252) [18:17:14] PROBLEM - Host tools-exec-1217 is DOWN: CRITICAL - Host Unreachable (10.68.18.20) [18:17:16] PROBLEM - Host tools-exec-1201 is DOWN: CRITICAL - Host Unreachable (10.68.17.49) [18:18:32] PROBLEM - Host tools-exec-1204 is DOWN: CRITICAL - Host Unreachable (10.68.17.88) [18:35:34] RECOVERY - Host tools-exec-1204 is UP: PING OK - Packet loss = 0%, RTA = 1.47 ms [18:35:48] RECOVERY - Host tools-bastion-02 is UP: PING OK - Packet loss = 0%, RTA = 2.33 ms [18:36:04] RECOVERY - Host tools-exec-1209 is UP: PING OK - Packet loss = 0%, RTA = 1.56 ms [18:36:06] RECOVERY - Host tools-exec-1213 is UP: PING OK - Packet loss = 0%, RTA = 1.16 ms [18:36:10] RECOVERY - Host tools-exec-1218 is UP: PING OK - Packet loss = 0%, RTA = 1.31 ms [18:36:10] godog, it runs on the tools grid [18:36:14] so hopefully it'll be restarted? [18:36:35] Krenair: that was my assumption too, doesn't seem to be the cause though [18:36:41] case even [18:36:46] RECOVERY - Host tools-exec-1416 is UP: PING OK - Packet loss = 0%, RTA = 4.06 ms [18:36:48] RECOVERY - Host tools-exec-1202 is UP: PING OK - Packet loss = 0%, RTA = 3.39 ms [18:36:50] RECOVERY - Host tools-webgrid-lighttpd-1409 is UP: PING OK - Packet loss = 0%, RTA = 2.89 ms [18:36:58] RECOVERY - Host tools-exec-1417 is UP: PING OK - Packet loss = 0%, RTA = 2.40 ms [18:37:01] huh [18:37:02] RECOVERY - Host tools-exec-1206 is UP: PING OK - Packet loss = 0%, RTA = 4.28 ms [18:37:10] RECOVERY - Host tools-exec-1408 is UP: PING OK - Packet loss = 0%, RTA = 2.37 ms [18:37:11] Your job 34741 ("production-logbot") has been submitted [18:37:17] * morebots (tools.more@instance-tools-exec-1410.tools.wmflabs.org) has joined [18:37:17] RECOVERY - Host tools-exec-1201 is UP: PING OK - Packet loss = 0%, RTA = 2.72 ms [18:37:23] RECOVERY - Host tools-puppetmaster-01 is UP: PING OK - Packet loss = 0%, RTA = 2.35 ms [18:37:26] it wasn't on qstat [18:37:27] RECOVERY - Host tools-webgrid-lighttpd-1411 is UP: PING OK - Packet loss = 0%, RTA = 2.78 ms [18:37:27] RECOVERY - Host tools-webgrid-lighttpd-1410 is UP: PING OK - Packet loss = 0%, RTA = 1.35 ms [18:37:31] thanks Krenair! [18:37:37] ah, so manual restart heh [18:37:39] RECOVERY - Host tools-webgrid-generic-1404 is UP: PING OK - Packet loss = 0%, RTA = 2.02 ms [18:38:45] RECOVERY - Host tools-exec-1217 is UP: PING OK - Packet loss = 0%, RTA = 1.10 ms [18:43:04] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [18:47:48] !log repool all depooled instances [18:47:48] repool is not a valid project. [18:48:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [18:48:55] !log tools repool all depooled instances [18:48:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:50:37] mmhh I don't see stashbot either, it is running though according to bin/k8s.sh status [18:54:04] bd808: ^ perhaps [18:56:41] the k8s version is just a test [18:56:47] not the main bot [18:58:32] labs-morebots: there? [18:58:33] I am a logbot running on tools-exec-1410. [18:58:33] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [18:58:33] To log a message, type !log . [18:58:57] bd808: ah ok, I got confused because ./stashbot.sh isn't there anymore [18:59:22] yeah. caught me mid-reorg. docs aren't updated yet [18:59:40] hoping to move to the k8s version of the job [19:00:15] thanks for checking on it godog :) [19:01:01] bd808: np! thanks for working on it, I use t.w.o/sal all the time [19:01:29] ah was there a task for the stashbot/morebots feature merge you mentioned? [19:02:32] godog: bd808 hopefully we can make it not require NFS at all, and I can have a couple of k8s nodes that don't need NFS rn these :D [19:04:06] godog: kind of -- https://github.com/bd808/tools-stashbot/issues/7 -- nothing explictly in phab though and I haven't talked with others besides yuvipanda to see if it is generally seen as a good idea [19:05:21] yuvipanda: awesome! next step will be to run it on production k8s "and we're done" [19:05:29] godog: :D [19:05:36] bd808: thanks! yeah gh is good enough [19:23:07] 06Labs, 10wikitech.wikimedia.org: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#2742515 (10bd808) [19:37:14] 06Labs, 10wikitech.wikimedia.org: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#2742553 (10bd808) The account I'm trying to make is "Bd808-test-20161025"/"bd_20161025". I'm not seeing anything obvious in logstash explaining wh... [19:42:04] 06Labs, 10wikitech.wikimedia.org: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#2742561 (10bd808) I finally got an account created with "Bd20161025"/"bd161025". I think maybe my original shell name was too long. If so this is... [19:42:55] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Account creation failing with "The authentication plugin denied the account creation." - https://phabricator.wikimedia.org/T149109#2742562 (10bd808) 05Open>03Resolved a:03bd808 [20:32:26] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms [20:38:35] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [20:40:53] hey doodlees [20:41:01] i lost my phone, but its coming back to me eventually [20:41:03] in the meantime [20:41:05] i got a new phone [20:41:10] how do I deal witih 2 factor auth? [20:41:14] i was using google authenticator ap [20:41:15] app [20:41:25] i need to re-login to wikitech [20:43:31] ottomata: https://wikitech.wikimedia.org/wiki/Password_reset [20:45:25] yeehaw, thanks yuvipanda [20:45:32] ottomata: yw [20:45:53] well, you do have your 2fa reset codes, don't you? [20:46:01] oh, i probably did. [21:38:21] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.77 ms [21:43:20] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [21:43:34] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 2.21 ms [21:46:50] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:52:06] for the database commonswiki_p, what is the address? [21:52:41] the address? [21:52:58] the hostnames of the database servers? [21:53:01] Right. [21:53:15] Just as for user databases it's tools-db, for commonswiki_p it's ____. [21:53:38] commonswiki.labsdb [21:53:41] thank you [21:53:47] which is actually labsdb1001.eqiad.wmnet [21:54:08] if I am doing this on Tool Labs though, do I need to spell that out or would commonswiki.labsdb suffice? [21:54:09] but don't rely on those labsdb* hostnames [21:54:24] commonswiki.labsdb is best [21:54:29] Thank you [22:01:41] 06Labs, 10Horizon: Add a 'remember me' feature to Horizon - https://phabricator.wikimedia.org/T149036#2743177 (10Andrew) a:03Andrew [22:06:06] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [22:26:53] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]