[00:27:21] Coren: I left it unresolved because it seems you have a different solution in mind for labs in general, e.g. it isn't resolved in general. But for CI purposes the resolv.conf change fixes the immediate bug. [00:27:32] We could create a separate task for the greater issue if you like [00:47:20] 10Tool-Labs, 5Patch-For-Review: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1123999 (10Ricordisamoa) >>! In T60196#1123825, @MZMcBride wrote: > Thank you for ! > > Looking at 10Tool-Labs, 5Patch-For-Review: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1124091 (10MZMcBride) >>! In T60196#1123999, @Ricordisamoa wrote: > AFAICS, and as noted in this task's description, the user_properties table already contains "preferences... [02:38:00] quiddity: Gotta run, but: https://en.wikipedia.org/wiki/User:Resident_Mario/sandbox [09:33:42] 10MediaWiki-extensions-OpenStackManager, 7Easy, 7I18n, 5Patch-For-Review: OpenStackManager: Missing message "<novaresources>" on Special:SpecialPages - https://phabricator.wikimedia.org/T69852#1124481 (10ImPacific) a:3ImPacific [10:16:15] YuviPanda|zzz: ping. [10:30:05] 10Tool-Labs, 5Patch-For-Review: Database replicas: replicate user.user_touched - https://phabricator.wikimedia.org/T92841#1124679 (10Springle) a:3Springle [10:46:52] hey Vivek! [10:47:02] sorrry, I’ve been sick as well :( [10:47:45] Vivek: are you still around? [11:35:44] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1124858 (10Springle) IDK how unix users map to db users, but user databases are all prefixed by the db-user name (which is enforced by that third grant statement you're parsing) so I expect your SHOW DATAB... [12:42:20] YuviPanda: Yes [12:43:44] I'll go grab a cup of coffee and be back. [12:44:23] YuviPanda: Will you be okay? Nothing serious I hope! [12:47:31] YuviPanda: Take rest, no hurries. [12:47:53] I'll be here the whole day, so we can talk. [12:56:54] YuviPanda: can you just message me. [13:00:13] 10Tool-Labs: Create a utility that dumps all databases of a user - https://phabricator.wikimedia.org/T91231#1125047 (10coren) >>! In T91231#1124858, @Springle wrote: > IDK how unix users map to db users [...] FWIW, the mapping is straightforward: non-service-group users are mapped to uX and service group users... [14:37:53] 6Labs, 10Wikimedia-Labs-wikitech-interface, 5Patch-For-Review: Use a Puppet ENC to define which classes are included in which nodes (in Labs) - https://phabricator.wikimedia.org/T85279#1125270 (10scfc) Looks cool; needs documentation, though, for the code flow in `modules/puppetmaster/files/ldap-yaml-enc.py`... [14:41:24] 10Tool-Labs: Default umask should be set to 0002 - https://phabricator.wikimedia.org/T48468#1125280 (10coren) 5Open>3declined No remaining issues, and users are better off setting umask explicitly if they need something other than the default. [14:55:17] (03PS3) 10coren: Tool Labs: Add .jsubrc support to jsub/jstart [labs/toollabs] - 10https://gerrit.wikimedia.org/r/156608 (https://bugzilla.wikimedia.org/54054) [14:56:32] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1125310 (10Thgoiter) 3NEW [15:01:49] 10Tool-Labs: Normal jobs sometimes run on tools-webgrid-tomcat.eqiad.wmflabs - https://phabricator.wikimedia.org/T64942#1125320 (10coren) Sorry it took so long to figure this out, but the error was too obvious and staring us in the face and it took a new fresh look to notice it: Your job script does not specify... [15:09:07] 6Labs, 5Patch-For-Review: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#1125363 (10Andrew) Update: designate is up and running on holmium. Integration with nova via designate-sink seems to be working. Bad news is that floating IPs don't seem to be suppor... [15:15:27] 10Tool-Labs: Move list of proxies to hiera - https://phabricator.wikimedia.org/T91954#1125396 (10scfc) Rethinking this, while we don't encourage custom start-up scripts, if we were to (continue to) hardcode the proxy servers in the scripts like I did for `$cron_host`, maintaining those custom scripts would be ve... [15:16:53] 10Tool-Labs: Move list of proxies to hiera - https://phabricator.wikimedia.org/T91954#1125404 (10scfc) Not to be misunderstood: `/etc/portgrabber.conf` will be populated by Puppet with the data in hiera. [15:18:47] Coren: yeah, am ok - just very very tired, unsure why. [15:19:04] Vivek: we keep missing each other, sadly :( let’s talk here? [15:20:04] YuviPanda: Please to consult a doctor. Catching $minorissue early is infinitely superior to waiting until it becomes $majorproblem. [15:20:26] Coren: I am mostly sure what the problem is - I was out in the sun for 3 hours yesterday…. [15:20:48] Heat stroke? [15:22:46] something like that [15:22:49] dehydration, probably [15:22:51] 10Tool-Labs, 7Tracking: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#1125431 (10coren) [15:22:52] 10Wikimedia-Labs-Other, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1125432 (10coren) [15:22:57] 10Tool-Labs: Provide namespace IDs and names in the databases similar to toolserver.namespace - https://phabricator.wikimedia.org/T50625#1125429 (10coren) 5Open>3Resolved The database exists, and is enduser-provided. Nothing for ops to do here. [15:22:58] 10Tool-Labs-tools-Other: Migrate http://toolserver.org/~dispenser/* to Tool Labs - https://phabricator.wikimedia.org/T68868#1125433 (10coren) [15:23:06] YuviPanda: You need to treat this very seriously. [15:31:57] 10Wikimedia-Labs-General, 6Release-Engineering: Increase quota for deployment-prep (beta) project - https://phabricator.wikimedia.org/T73886#1125449 (10greg) Currently.... #staging is: Cores: 29/30 RAM: 59392/102400 Floating IPs: 0/0 Instances: 12/15 Security Groups: 13/20 and deployment-... [15:36:27] Coren: yeah. re-hydrating myself and sleeping through the noon :) [15:45:47] YuviPanda: Don't forget to not just drink water. Need the salts back too. [15:54:35] 10Wikimedia-Labs-General, 6Release-Engineering: Increase quota for deployment-prep (beta) project - https://phabricator.wikimedia.org/T73886#1125542 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Set staging's quota to match deployment-prep's. [16:08:09] 10Tool-Labs: Default umask should be set to 0002 - https://phabricator.wikimedia.org/T48468#1125583 (10scfc) Sorry for ignoring your previous question. Our setup is designed that users are members of the tool's group so that they have write access to the tool's files and directories. Setting `umask` to `0022`... [17:22:29] Coren: Did you ever get a chance to look at T90658, like you said you would on the 6th? [17:22:40] Lemme check. [17:23:18] anomie: No, it fell through my cracks. Lemme do it now before I tackle something else. [17:23:54] anomie: Your price to pay is a quick sanity check review of https://gerrit.wikimedia.org/r/#/c/197325/1 if you speak some perl. :-) [17:24:08] I think you know I speak perl ;) Looking [17:31:38] 6Labs, 5Patch-For-Review: Create an admin user for the pdns db - https://phabricator.wikimedia.org/T92984#1125807 (10Andrew) 3NEW [17:32:45] 10Tool-Labs, 5Patch-For-Review: Comment lines in crontab are modified if their first characters are not '#' - https://phabricator.wikimedia.org/T75256#1125813 (10Anomie) [17:33:22] 6Labs, 5Patch-For-Review: Create an admin user for the pdns db - https://phabricator.wikimedia.org/T92984#1125815 (10Joe) a:3Joe [17:45:38] 6Labs, 5Patch-For-Review: Create an admin user for the pdns db - https://phabricator.wikimedia.org/T92984#1125950 (10Joe) grants for DDL (alter, drop, create) and standard usage grants given to the dedicated user. [17:55:01] 6Labs, 6operations: Wikitech registration for prior SVN user - https://phabricator.wikimedia.org/T90658#1125984 (10coren) Done. @Dragons_flight: please # reset your password via https://wikitech.wikimedia.org/wiki/Special:PasswordReset # log into https://gerrit.wikimedia.org/ # Add your ssh key there And you... [17:55:10] 6Labs, 6operations: Wikitech registration for prior SVN user - https://phabricator.wikimedia.org/T90658#1125985 (10coren) 5Open>3Resolved a:3coren [18:07:00] 10Tool-Labs: Comment lines in crontab are modified if their first characters are not '#' - https://phabricator.wikimedia.org/T75256#1126002 (10coren) [18:07:57] 10Tool-Labs: Comment lines in crontab are modified if their first characters are not '#' - https://phabricator.wikimedia.org/T75256#1126003 (10coren) 5Open>3Resolved Crontab is now much more lenient about whitespace around comments (and setting environment variables) and should no longer meddle in things tha... [18:45:31] (03PS1) 10Dzahn: add Brandon Black to labs roots [labs/private] - 10https://gerrit.wikimedia.org/r/197385 [18:47:35] YuviPanda: ok [18:47:47] hey Vivek :) [18:47:50] YuviPanda: I am here now. [18:48:02] (03PS2) 10Dzahn: add Brandon Black to labs roots [labs/private] - 10https://gerrit.wikimedia.org/r/197385 [18:48:37] But working on some network in our DC Lab which is causing some OSDS on our Ceph cluster to flap. [18:49:16] Yesterday, i went home at 5.30 Am slept till around 11 AM and came to office at 2 PM and I am still here :) [18:49:35] And I have not idea, how long I have to stay today :) [18:49:46] YuviPanda: we can talk np :) [18:49:58] Vivek: :) been there, know the feeling :) [18:50:40] Vivek: so if you want to just get started with our infrastructure / test how to submit patches etc (gerrit, testing, etc) https://phabricator.wikimedia.org/T75904 is a good small start [18:50:52] YuviPanda: ok. [18:50:58] (03PS3) 10Dzahn: add Brandon Black to labs roots [labs/private] - 10https://gerrit.wikimedia.org/r/197385 [18:53:08] (03PS4) 10Dzahn: add Brandon Black to labs roots [labs/private] - 10https://gerrit.wikimedia.org/r/197385 [18:54:24] YuviPanda: Can I register my username there ? [18:54:35] Vivek: in phabricator? sure! :) [18:54:42] Ok. [18:57:12] Registered my username 'vivekvc'. [18:58:38] YuviPanda: I think I need access to the puppet master. [18:58:52] Vivek: right, so you can test this on a self-hosted puppetmaster. [18:59:11] Ok. [18:59:16] * YuviPanda finds docs [18:59:26] Vivek: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [18:59:33] Vivek: have you used our labs environment before? [19:00:34] YuviPanda: No. [19:01:05] Vivek: ah, right. so https://wikitech.wikimedia.org/wiki/Main_Page is the main page for that. [19:01:09] https://wikitech.wikimedia.org/wiki/Help:FAQ has help... [19:01:23] YuviPanda: Ok. [19:03:58] Vivek: there’s also the mailing list you should probably join https://lists.wikimedia.org/mailman/listinfo/labs-l [19:06:23] Ok. [19:06:39] Will join. [19:25:24] YuviPanda: Do I have to use my own VPS to self host ? [19:28:57] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0] [19:30:58] PROBLEM - Puppet failure on tools-webproxy-01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [19:31:23] Vivek: nope. you can sign up on wikitech and get a VPS... [19:32:16] PROBLEM - Puppet failure on tools-webgrid-06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:53:40] 6Labs, 10hardware-requests, 6operations: eqiad: (5) labs virt nodes - https://phabricator.wikimedia.org/T89752#1126241 (10RobH) [19:53:51] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:56:06] RECOVERY - Puppet failure on tools-webproxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:18] RECOVERY - Puppet failure on tools-webgrid-06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:36] YuviPanda: Cool, my ceph cluster is healthy now, so I can go home and get some rest. [19:57:45] :) [19:57:50] Vivek: +1 :) [20:00:27] YuviPanda: I am leaving for the day, I have the logs, I'll come back tommorow and get started fresh :) [20:00:32] YuviPanda: nite. [20:00:35] Vivek: night! [20:23:58] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1126408 (10Aklapper) > Isn't there any possibility to make this tool run more stable? That might be a question for Para in https://wikitech.wikimedia.org/wiki/User_talk:Para#Moving_kmlexport_to_trusty.3F (as I don't know if P... [20:25:28] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1126418 (10yuvipanda) Have there been more outages since the move to trusty? [20:27:42] Is the latest wd dump available on the labs? [20:32:31] a930913: The latest complete dump should be, yes. [20:32:45] Coren: Where? [20:33:16] I see 20150207 in /public/dumps/public/wikidatawiki/ [20:34:40] Coren: Hmm, that date doesn't exist in http://dumps.wikimedia.org/other/wikidata/ Am I looking at the wrong thing? [20:38:57] a930913: There may be different dumps kept on different infrastructures, perhaps? [20:39:38] Ah, there is also the issue of "completed" vs "started". [20:39:43] Coren: Possibly. It's still over a month old though. [20:39:48] But yeah, the discrepancy is odd. [20:40:04] a930913: Coren those are the json wikidata dumps, they aren’t in the rsync [20:40:09] someone should file a bug and get those added [20:40:20] Ah! [20:40:36] Good point - I hadn't noticed. You're right - those should be added. [20:40:53] a930913: What I pointed you to are the xml dumps. [20:41:01] Coren: Ah, didn't check :p [20:41:15] * a930913 self trouts. [21:10:01] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1126562 (10Thgoiter) When I wrote the bug report, the tool wasn't running. This comment was written after the move to trusy (March 15): https://en.wikipedia.org/w/index.php?title=Template_talk:GeoGroup&diff=651530301&oldid=65... [21:10:50] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1126566 (10yuvipanda) @Coren can we increase the VMEM limit for kmlexport to something higher? [21:19:17] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1126587 (10Teslaton) Yes, the tool was returning a 500 resposne for several hours today as well. Isn't it possible to setup some quick-and-dirty "intermediate solution" - e.g. to cron some periodic check&autorestart script us... [21:25:03] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1126616 (10yuvipanda) I've increased its memory limit to 7G (from 4G). Let's see if that helps. [23:12:41] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1126847 (10Teslaton) Right now it seems to be down again.