[00:02:08] <wikibugs>	 6Labs, 10Tool-Labs-tools-Other, 6Developer-Relations: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#1729232 (10He7d3r)
[00:24:13] <wikibugs>	 10Wikibugs, 10Differential, 5Gerrit-Migration: Broadcast Differential activity to IRC - https://phabricator.wikimedia.org/T116330#1747187 (10mmodell)
[02:06:39] <samwilson>	 hello, how might I go about resetting my password for gerrit? I can't log in
[03:20:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[03:27:48] <Cyberpower678>	 Coren, are you still awake?
[03:28:02] <Coren>	 Sorta.  Emergency?
[03:28:14] <Cyberpower678>	 Not quite, but I can make it quick.
[03:28:18] <Cyberpower678>	 PM?
[03:28:21] <Coren>	 Shoot\
[04:25:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0]
[04:27:28] <yurik>	 yuvipanda hi, do you know if osm db server was updated to the latest version in labs?  
[04:27:49] <yurik>	 i remember there was an task for it, can't find it though
[04:45:06] <wikibugs>	 6Labs, 6Discovery, 10Maps: Update wiki page with OSM Postgres access info - https://phabricator.wikimedia.org/T116355#1747429 (10Yurik) 3NEW
[05:07:41] <yurik>	 yuvipanda|maybe, ^
[06:49:48] <YuviPanda>	 yurik I've no idea. akosiaris would know
[07:23:51] <akosiaris>	 ?
[08:37:48] <wikibugs>	 6Labs, 7Article-Recommendation: Investigate possible instances with 32G of RAM to test article-reccomendations - https://phabricator.wikimedia.org/T116321#1747587 (10yuvipanda) Will exist at least for 2-3 months
[08:46:05] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: jwt_secret for cxserver [labs/private] - 10https://gerrit.wikimedia.org/r/248296 
[09:14:38] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] jwt_secret for cxserver [labs/private] - 10https://gerrit.wikimedia.org/r/248296 (owner: 10Alexandros Kosiaris)
[09:53:56] <shinken-wm>	 PROBLEM - Puppet failure on tools-web-static-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[10:23:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-web-static-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[13:37:04] <wikibugs>	 6Labs, 6Discovery, 7Elasticsearch, 3labs-sprint-116, 3labs-sprint-117: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1747927 (10Halfak) Just got directed here from @yuvipanda.  I was hoping to be able to use Elastic search to index/query past answers in [t...
[13:53:16] <wikibugs>	 6Labs, 10Incident-20150617-LabsNFSOutage: Start regular snapshots of the labs filesystems and cross-server backups - https://phabricator.wikimedia.org/T103691#1747952 (10coren) 5Open>3Resolved This was done during T106464
[14:00:18] <Luke081515>	 Is here an admin of wikitech?
[14:01:53] <Luke081515>	 I need one
[14:01:57] <Luke081515>	 Coren: Are you here?
[14:02:13] <Coren>	 Luke081515: Yes.  What's up?
[14:02:20] <Luke081515>	 Vandalism at wikitech
[14:02:27] <Coren>	 Linky?
[14:02:30] <Luke081515>	 https://wikitech.wikimedia.org/w/index.php?title=User_talk:Tim_Landscheidt&redirect=no <-- this page was moved 3 or 4 times
[14:02:38] <Luke081515>	 to this page: https://wikitech.wikimedia.org/w/index.php?title=Wikitech_talk:Tim4t3t5w34t3w5t_Landt4rtrtertscheidt&redirect=no
[14:02:42] <Luke081515>	 can you move it back?
[14:03:29] <Luke081515>	 The vandal moved Tims useroage too, but copied it to the old position with c&p
[14:06:26] <Luke081515>	 Coren: Please move this (https://wikitech.wikimedia.org/wiki/Wikitech_talk:Tim4t3t5w34t3w5t_Landt4rtrtertscheidt) back to user talk:Tim Landscheidt
[14:06:29] <Luke081515>	 thanks
[14:08:02] <Coren>	 Luke081515: Yeah, I'm cleaning up history and getting rid of the crap redirects.  Don't worry - I've been undoing vandalism for many years.  :-)
[14:08:15] <Luke081515>	 Coren: Thanks for quick reaction
[14:08:17] <Luke081515>	 ;)
[14:09:14] <Coren>	 Hm.  He also switched user and user talk; which I hadn't noticed.
[14:09:30] <Coren>	 s/switched/mixed around/
[14:10:01] <Luke081515>	 yeah, I think this vandale tried to make much damage which this page as he can
[14:11:41] <Coren>	 I think I found all the crap in all the namespaces.
[14:12:17] <Luke081515>	 Coren: Maybe delete this copy of tims page, too? https://wikitech.wikimedia.org/wiki/User:Qazxswedcvfrtgbnhyujmkiolpbeatsr
[14:12:47] <Luke081515>	 yeah, you find all other pages
[14:12:50] <Luke081515>	 *found
[14:13:03] <Luke081515>	 thanks
[14:13:13] <Coren>	 Ah, missed that one.  Deletationned.
[14:13:53] <Luke081515>	 Coren: maybe better to protect tims pages against moving?
[14:15:06] <Coren>	 If it happens again, sure.  Otherwise, Tim can pick his own protection level.
[14:15:19] <Luke081515>	 hm, ok
[14:15:57] <Krenair>	 Luke081515, in future feel free to poke me about wikitech admin stuff
[14:16:04] <Luke081515>	 ok :)
[14:16:47] <Luke081515>	 I found that, because I search for something at the recent changes
[14:34:53] <wikibugs>	 6Labs: Provide a simple way to backup arbitrary files from instances - https://phabricator.wikimedia.org/T104206#1748007 (10akosiaris) >>! In T104206#1725377, @yuvipanda wrote: > So ideally the workflow I'd like people to have: >  > # Specify filesets in hiera (so can do it per host or per-project)  Agreed. Wher...
[14:37:48] <ottomata>	 heya, is there a general statsd instance in labs?
[14:38:55] <ottomata>	 labmon1001.eqiad.wmnet:8125?
[14:40:03] <ottomata>	 yes, thank you me for answering my question. :)
[15:17:43] <JohnFLewis>	 andrewbogott: hi hi, just a poke that https://phabricator.wikimedia.org/T97480 could really do with being fixed with the labs lists :)
[15:19:28] <Krenair>	 I find these trivial requests that sit waiting for months really frustrating.
[15:19:49] <JohnFLewis>	 Krenair: I know how you feel :)
[15:20:32] <Krenair>	 can't you do something with the master password? :p
[15:20:53] <JohnFLewis>	 It's a shame I don't have it but I could if it did :)
[15:21:39] <Krenair>	 JohnFLewis, um, don't you have access to the server it's on?
[15:22:13] <JohnFLewis>	 nope, the password is encrypted on fermium. it's only raw format is on iron and James' email
[15:22:32] <Krenair>	 I'm kind of surprised to hear that mailman encrypts passwords
[15:23:24] <JohnFLewis>	 well, it only encrypts the master and list creator / list passwords [admin,mod,poster]
[15:24:09] <Krenair>	 And that you need things like passwords to make changes when you have full access to the machine as the same user as mailman itself :p
[15:25:19] <JohnFLewis>	 I could do it via shell but it seriously easier via the interface and so. I'll probably end up doing it via shell out of annoyance later knowing me :)
[15:58:43] <andrewbogott>	 JohnFLewis: I don’t have any idea how fulfill that request, or even what the request means.
[15:59:49] <andrewbogott>	 might be that… valhalla is maintaining the lists?  I don’t know.  YuviPanda maybe?
[16:00:33] <JohnFLewis>	 andrewbogott: to my knowledge it's you, Yuvi and Coren, that's what mailman says anyway :)
[16:01:25] <JohnFLewis>	 andrewbogott: all it is is editing; https://lists.wikimedia.org/mailman/edithtml/labs-l/listinfo.html and going down to where it links to meta.wikimedia.org and adding in a link to labs-announce mailing list and then viceversa for labs-announce to labs-l
[16:02:10] <JohnFLewis>	 andrewbogott: or I could do it server side if you'd prefer (since I can't do it via lists.wm.o)
[16:04:13] <andrewbogott>	 everytime this topic comes up i am morally required to reiterate that I don’t know why we have two different lists
[16:05:37] <andrewbogott>	 JohnFLewis: https://lists.wikimedia.org/mailman/listinfo/labs-l <- how’s that?
[16:05:41] <JohnFLewis>	 andrewbogott: Yuvi requested it because people only want important announcements about labs without all the chatter and so that labs-l has
[16:05:43] * JohnFLewis looks
[16:06:06] <JohnFLewis>	 andrewbogott: yeah, that's fine :)
[16:09:57] <wikibugs>	 6Labs, 10Wikimedia-Mailing-lists: expand labs listinfo pages and link them to eachother - https://phabricator.wikimedia.org/T97480#1748276 (10Andrew) 5Open>3Resolved a:3Andrew
[16:10:26] <JohnFLewis>	 :)
[16:21:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]
[16:44:05] <wikibugs>	 6Labs, 10Tool-Labs: Delete "toolserver" tool - https://phabricator.wikimedia.org/T116389#1748337 (10Nemo_bis) 3NEW
[16:44:28] <wikibugs>	 6Labs, 10Tool-Labs: Delete "toolserver" tool - https://phabricator.wikimedia.org/T116389#1748345 (10Nemo_bis)
[16:56:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:18:31] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6operations, 3labs-sprint-117: Allocate labs subnet in dallas - https://phabricator.wikimedia.org/T115491#1748532 (10RobH)
[17:43:07] <YuviPanda>	 JohnFLewis: I also have no idea how to fulfill that request :) Also if andrewbogott isn't labs-l admin he should be...
[17:43:25] <andrewbogott>	 YuviPanda: I did it, after John told me how :)
[17:43:32] <YuviPanda>	 wheee!
[17:53:26] <Luke081515>	 zeljkof: Are you here? I got some question concerning browser-tests
[18:00:30] <wikibugs>	 6Labs, 6Discovery, 6operations, 7Elasticsearch: Mount /dev/md1 on nobelium to /var/lib/elasticsearch - https://phabricator.wikimedia.org/T114856#1748806 (10yuvipanda)
[18:01:05] <andrewbogott>	 YuviPanda: I can make a special high-memory flavor that’s local to a specific project.  That project should be art-recs?
[18:01:14] <andrewbogott>	 And, what specs does the flavor need, other than lots of ram?
[18:01:15] <YuviPanda>	 andrewbogott: yes
[18:01:50] <YuviPanda>	 andrewbogott: hmm, 8cores maybe? disk can be tiny (40G? 60G?)
[18:01:54] <andrewbogott>	 ok
[18:07:49] <wikibugs>	 6Labs, 7Article-Recommendation: Investigate possible instances with 32G of RAM to test article-reccomendations - https://phabricator.wikimedia.org/T116321#1748840 (10Andrew) 5Open>3Resolved a:3Andrew ok, the art-recs project now has a special flavor named 'm1.highmem.'  Please let me know when this is wr...
[18:11:29] <zeljkof>	 Luke081515|AFK: sorry, sick all week, please send mail or cc me in a phabricator task
[18:11:53] <YuviPanda>	 andrewbogott: \o/ thanks
[18:12:37] <andrewbogott>	 YuviPanda: I suspect that you can also make custom flavors within a project on horizon.  I haven’t tried it though
[18:12:52] <YuviPanda>	 andrewbogott: woo, that's intersting...
[18:12:59] <YuviPanda>	 andrewbogott: I suspect we'll want to not do that much?
[18:13:22] <andrewbogott>	 I’m not sure.  It seems pretty safe inasmuch as quotas still apply
[18:14:20] <YuviPanda>	 oh I see
[18:14:49] <YuviPanda>	 that would indeed be nice since the current ratios aren't best fit for a bunch of projects
[18:15:01] <YuviPanda>	 I'll look at it after I finish with the nobelium stuff
[18:27:22] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ssmith was created, changed by Ssmith link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Ssmith edit summary: Created page with "{{Tools Access Request |Justification=I'm building a project for the design research team (a wordpress site) that I need to have hosted here. |Completed=false |User Name=Ssmit..."
[18:42:44] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6operations, 3labs-sprint-117: Allocate subnet for labs test cluster instances - https://phabricator.wikimedia.org/T115492#1748999 (10RobH)
[18:43:40] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1749002 (10RobH) Note I cannot create the onsite allocation of these sysetms until the network is setup for labs in codfw; due to the fact I'm not sure...
[19:44:29] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ssmith was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=196551 edit summary: 
[21:17:07] <Luke081515>	 Hello, I got a problem with puppet, my instance can not read the information form the puppetmaster. Can someone help me?
[21:24:04] <Luke081515>	 this is the error message: https://phabricator.wikimedia.org/F2760884
[21:26:05] <Krenair>	 Luke081515, need to run it as root
[21:26:05] <Krenair>	 use sudo
[21:26:39] <Luke081515>	 Krenair: Thanks
[21:26:53] <Krenair>	 I don't know why it gives that useless error
[21:27:00] <Krenair>	 has confused me before too
[22:15:28] <wikibugs>	 6Labs, 6Phabricator: phabricator puppet at labs broken - https://phabricator.wikimedia.org/T116442#1749654 (10Luke081515)
[22:27:49] <wikibugs>	 6Labs, 6Phabricator: phabricator puppet at labs broken - https://phabricator.wikimedia.org/T116442#1749706 (10Krenair) Doesn't make much sense... scope.lookupvar('network::constants::all_networks') fails?