[01:13:53] 10Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1120423 (10MZMcBride) >>! In T60196#1109452, @coren wrote: > The one thing I am missing to implement this at this time is either > (a) the whitelist of properties to collate or > (b) the method... [07:06:56] YuviPanda: Are you there ? [12:10:26] 6Labs: db servers for designate and labs pdns - https://phabricator.wikimedia.org/T92694#1121194 (10Springle) After poking around I'm hoping the answers are: "small", "small", "no, internal only behind open stack", and "maybe we'll see". So I've created the following: pdns DB Server: m1-master.eqiad.wmnet pdns... [13:09:48] 10Tool-Labs, 10Wikidata: Make wbs_propertypairs table on wikidata available in tool labs - https://phabricator.wikimedia.org/T92825#1121255 (10aude) 3NEW [13:13:13] 10Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1121266 (10coren) >>! In T60196#1120423, @MZMcBride wrote: > Can we start with "gadget-*" and then decide on the others in a separate future task? Sounds like a reasonable first pass. I'll ha... [13:21:39] 6Labs, 10Wikidata: Make wbs_propertypairs table on wikidata available in labs - https://phabricator.wikimedia.org/T92825#1121280 (10coren) p:5Triage>3Normal a:3coren [13:40:22] Coren: any more progress on https://phabricator.wikimedia.org/T92351 [13:40:22] ? [13:40:25] is set to UBN [13:41:02] Ah, I saw Krinkle move it to the 'done' column and I thought he had already closed it. [13:41:29] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1121309 (10yuvipanda) a:3coren [13:42:01] * Coren edits the ticket to track the underlying cause. [13:42:41] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1121312 (10coren) p:5Unbreak!>3Normal The issue has been worked around on the CI side,... [13:43:33] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1121316 (10coren) [13:44:17] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1121318 (10yuvipanda) Interestingly, dig saucelabs.com on tools-trusty works fine. [13:44:36] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1121319 (10coren) @scfc The SOA records are there; though it's not immediately clear that they... [13:46:04] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1121324 (10coren) @yuvipanda: That's not so much "interesting" as "expected". Dig ignores the... [13:47:40] 6Labs, 6operations, 5Patch-For-Review: Puppetize labstore1003 - https://phabricator.wikimedia.org/T91573#1121327 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Is done. [13:48:27] 6Labs, 6operations: Make labs salt use instance names than ids - https://phabricator.wikimedia.org/T1154#1121330 (10yuvipanda) [13:54:46] 6Labs: db servers for designate and labs pdns - https://phabricator.wikimedia.org/T92694#1121355 (10Andrew) Sorry for the slow response! > Any idea what sort of load will this generate? Not much. Worst case is that pdns will hit the db every time a ttl expires. The other events will be very infrequent (e.g.,... [14:50:26] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 10OOjs, and 2 others: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1121449 (10scfc) So what does `dig notexist.eqiad.wmflabs` return on the server where `dnsmasq`... [15:17:41] (03PS1) 10Andrew Bogott: Added dummy passwords for designate, pdns [labs/private] - 10https://gerrit.wikimedia.org/r/197054 [15:27:01] 10Tool-Labs, 5Patch-For-Review: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1121564 (10coren) The view has been created (on all but size 1 wikis) but the timestamps currently return NULL because none of the user_touched values are on the replicas a... [15:29:44] 10Tool-Labs, 5Patch-For-Review: Database replicas: replicate user.user_touched - https://phabricator.wikimedia.org/T92841#1121568 (10coren) 3NEW [15:32:45] 6Labs, 10Wikidata, 5Patch-For-Review: Make wbs_propertypairs table on wikidata available in labs - https://phabricator.wikimedia.org/T92825#1121582 (10coren) 5Open>3Resolved Done, and applied. [15:34:17] 6Labs, 10Wikidata: Make wbs_propertypairs table on wikidata available in labs - https://phabricator.wikimedia.org/T92825#1121589 (10coren) [15:45:20] (03CR) 10Andrew Bogott: [C: 032 V: 032] Added dummy passwords for designate, pdns [labs/private] - 10https://gerrit.wikimedia.org/r/197054 (owner: 10Andrew Bogott) [16:01:02] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1121687 (10scfc) First stab: ``` #!/usr/bin/perl -w use strict; use warnings; use Data::Dumper; use DBI; my @servers = ('10.64.37.4', '10.64.37.5', '10.64.37.9', '10.64.4.11'); my %users = ('s12345' =>... [16:02:25] 10Tool-Labs: Remove unneeded tools - https://phabricator.wikimedia.org/T91740#1121693 (10scfc) 5Open>3Resolved Verified with the script in T91231#1121687 that none of those tools has databases on any of the servers. [16:02:50] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1121705 (10yuvipanda) Let's not use perl please. I'd rather not have any more perl in our ops/puppet repo than we already do (And I think @mark agrees). [16:04:52] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1121710 (10chasemp) >>! In T91231#1121705, @yuvipanda wrote: > Let's not use perl please. I'd rather not have any more perl in our ops/puppet repo than we already do (And I think @mark agrees). If I could... [16:11:43] 6Labs, 10Tool-Labs, 10Wikimedia-Hackathon-2015: Organize Wikimedia Labs activities at the Wikimedia Hackathon 2015 - https://phabricator.wikimedia.org/T92274#1121726 (10silke) Hi all! I'm less involved with Tool Labs now that good old Toolserver is gone. And I'll probably not make it to Lyon. I see there a... [16:22:30] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1121753 (10scfc) >>! In T91231#1121705, @yuvipanda wrote: > Let's not use perl please. I'd rather not have any more perl in our ops/puppet repo than we already do (And I think @mark agrees). That's not me... [16:23:10] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1121757 (10yuvipanda) :) cool! I'll see if I can port this to python and put it in the package at some point - think this will be useful in the future as well. [16:23:48] 10Wikimedia-Labs-General: centralauth_p is missing tables - https://phabricator.wikimedia.org/T68533#1121760 (10coren) @Legoktm can you explain why you need those tables? It might be simpler to give you the actual data you need rather than open up the underlying tables. [16:28:13] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1121776 (10scfc) Eh, the translation isn't that complicated so that I couldn't do that myself :-). But that only makes sense //after// the proper way to solve the problem has been determined. [16:32:10] 10Wikimedia-Labs-General: centralauth_p is missing tables - https://phabricator.wikimedia.org/T68533#1121786 (10Legoktm) Mainly so I can test out centralauth queries before running them in production. [16:36:07] 6Labs, 10Wikidata: Make wbs_propertypairs table on wikidata available in labs - https://phabricator.wikimedia.org/T92825#1121801 (10aude) thanks @coren :) [16:38:02] 6Labs: Setup a REST service that helps do NFS exports for OpenStack instances properly - https://phabricator.wikimedia.org/T92638#1121824 (10scfc) As we depend on LDAP for user logins, I don't share the fear for NFS mounts which happen even less frequently (essentially only on instance boot-up). If the polling... [16:49:20] 10Tool-Labs, 5Patch-For-Review: Add wiki title case sensitivity flag (is_sensitive) to meta_p.wiki to support jbo.wp and wiktionary tools - https://phabricator.wikimedia.org/T69476#1121878 (10coren) 5Open>3Resolved ```> select is_sensitive, count(*) from meta_p.wiki group by is_sensitive; +--------------+-... [16:49:21] 10Tool-Labs, 7Tracking: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#1121880 (10coren) [16:52:09] 10Tool-Labs: Add wiki title case sensitivity flag (is_sensitive) to meta_p.wiki to support jbo.wp and wiktionary tools - https://phabricator.wikimedia.org/T69476#1121891 (10coren) p:5Triage>3Low [17:05:22] Fun fact: labs storage currently holds a bit over 117 million files. [17:15:22] (03CR) 10Legoktm: [C: 032] Add more projects for devtools and mobile [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196551 (owner: 10Yuvipanda) [17:15:41] (03Merged) 10jenkins-bot: Add more projects for devtools and mobile [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/196551 (owner: 10Yuvipanda) [17:16:50] !log tools.wikibugs Updated channels.yaml to: 23240bd0dc5aebcc2a94b6f1ac268e2e3ad41114 Add more projects for devtools and mobile [17:16:54] Logged the message, Master [18:12:07] 6Labs, 7Design: Fix horizon logo - https://phabricator.wikimedia.org/T91780#1122243 (10Dzahn) [18:19:22] 6Labs, 7Design: Fix horizon logo - https://phabricator.wikimedia.org/T91780#1122302 (10Andrew) [18:46:14] !log deployment-prep upgraded Elasticsearch on deployment-logstash1 [18:46:17] Logged the message, Master [18:56:41] Was Phab-01 reset? [19:02:26] last week yes [19:03:38] Ok. Thought I lost my password :) [19:47:06] andrewbogott: I think the deployment-prep project has reached its quota, can you bump? [19:47:23] ottomata: I can, if there’s nothing that can be deleted :/ [19:47:33] i don't know if there is, who should I ask? [19:47:38] i just need oneMoreNode [19:48:07] maybe bd808? [19:48:49] I raised it to 50 [19:48:51] danke! [19:49:14] it works, much obliged. [19:50:06] * bd808 is amazed how much power people think he must have :) [19:50:22] 6Labs: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#1122769 (10Andrew) [19:50:25] 6Labs, 6operations: setup / deploy holmium as designate server - https://phabricator.wikimedia.org/T92507#1122767 (10Andrew) 5Open>3Resolved Working fine -- thanks! [20:51:38] bd808: Are you not all-powerful? My dreams are shattered! [22:08:07] Coren: I apparently messed up a VM to the point that I cannot SSH into it [22:08:17] is there a way to force a puppet run remotely? [22:11:07] tgr: No, but if your VM is still healthy enough to run cron it will get it on the next run. [22:12:42] tgr: That said, the hardcoded root key might still work. What instance is this? [22:12:58] multimedia-jsperf [22:13:25] eh, I'm being stupid [22:13:26] Ah, no, your ssh daemon is dead. Nothing I can do there. [22:13:37] it has a self-hosted puppetmaster [22:13:43] so no auto runs I suppose [22:14:03] is there a way to get an admin console via the virtualization software? [22:14:08] Well, runs yes but that won't help you as it won't pick the global manifests up unless you added an automatic pull. [22:14:32] I don't think puppet itself is messed up [22:14:46] tgr: Yes, but it's rather pointless as there is no password authentication possible. [22:15:29] direct disk access? :) [22:15:38] I really just want to recover a patch from it [22:16:21] tgr: Direct disk access is a nightmare at best; and would be both delicate and involved. I take it you didn't do that work off /data/project or /home? [22:16:59] no, it's a puppet patch so it's in /var/lib/git or something like that [22:17:29] whereever the self-hostem puppetmaster keeps its files [22:18:52] Yeah, that's a local disk. Our system really isn't designed to make that storage accessible. :-( [22:19:01] (From without the instance) [22:19:11] I take it a simple reboot won't suffice to bring sshd back? [22:19:23] it did not [22:19:34] Hm. [22:19:38] Actually, it did. [22:19:45] You just weren't patient enough. :-) [22:19:54] I'm not sure what happened, I stopped sshd manually for some debugging but did not touch it's puppet config [22:20:29] stopping sshd manually should not be enough to kick you out - though you have to remember to start it again before you close your last connection. :-) [22:20:34] At any rate, it's back now. [22:20:47] oh wow [22:20:50] thanks [22:21:52] looks like asking you is enough to scare the box into submission (I did restart it before) [22:35:12] It's the sysadmin effect. It's a Well Known Fact{{cn}} that pointing out an issue to a sysadmin makes it evaporate - at least until said sysadmin is no longer paying attention. :-) [22:36:04] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:38:34] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:46:51] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [23:00:58] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [23:03:40] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [23:12:00] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:43:07] 10Tool-Labs, 5Patch-For-Review: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1123825 (10MZMcBride) It was `user_properties_anonym` on the Toolserver and it's `user_properties_anon` now. Probably doesn't really matter, just noting. Thank you for