[00:25:33] <grrrit-wm>	 (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/251339 (owner: 10Jean-Frédéric)
[01:09:58] <grrrit-wm>	 (03PS2) 10Tim Landscheidt: Import sql from operations/puppet [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268563 
[01:10:00] <grrrit-wm>	 (03PS1) 10Tim Landscheidt: Add man page for sql [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268602 
[01:11:42] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Import sql from operations/puppet [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268563 (owner: 10Tim Landscheidt)
[01:12:54] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add man page for sql [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268602 (owner: 10Tim Landscheidt)
[01:15:33] <grrrit-wm>	 (03PS3) 10Tim Landscheidt: Import sql from operations/puppet [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268563 
[01:15:35] <grrrit-wm>	 (03PS2) 10Tim Landscheidt: Add man page for sql [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268602 
[01:17:09] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Import sql from operations/puppet [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268563 (owner: 10Tim Landscheidt)
[01:18:28] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add man page for sql [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268602 (owner: 10Tim Landscheidt)
[01:20:51] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Add unit test for CH1903Converter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/251339 (owner: 10Jean-Frédéric)
[01:22:46] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add unit test for CH1903Converter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/251339 (owner: 10Jean-Frédéric)
[01:26:43] <grrrit-wm>	 (03CR) 10Tim Landscheidt: "The Jenkins log looks like it is not related to the change of this patch, so I'll recheck it in a day or two." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268602 (owner: 10Tim Landscheidt)
[01:26:46] <grrrit-wm>	 (03CR) 10Tim Landscheidt: "The Jenkins log looks like it is not related to the change of this patch, so I'll recheck it in a day or two." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268563 (owner: 10Tim Landscheidt)
[06:49:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-elastic-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[09:14:10] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:18:11] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 778857 bytes in 3.232 second response time
[11:04:43] <putnik>	 How do you know who has access to the https://analytics.wmflabs.org/demo/pageview-api/ tool? Or where is git repo, if it is?
[12:51:07] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Avm99963 was created, changed by Avm99963 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Avm99963 edit summary: Created page with "{{Tools Access Request |Justification=I plan to use this project to host web tools that can be useful for the Spanish Wikipedia and its community. To start off, I'm going to d..."
[13:53:26] <grrrit-wm>	 (03CR) 10Ricordisamoa: "recheck" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 (owner: 10ArthurPSmith)
[14:11:31] <grrrit-wm>	 (03CR) 10Ricordisamoa: [C: 032] "Yes, SPARQL is definitely faster. Merging for the impatient :)" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 (owner: 10ArthurPSmith)
[14:12:10] <grrrit-wm>	 (03Merged) 10jenkins-bot: Added a Wikidata-based "chart of the nuclides" under /nuclides [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 (owner: 10ArthurPSmith)
[14:21:29] <jynus>	 cc:labs/tools !log reimporting nlwiktionary revision into labs (expect some temporary lag on labs-s3)
[14:34:22] <grrrit-wm>	 (03PS1) 10Ricordisamoa: Use url_for() instead of bare relative URLs in templates [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/268672 
[14:37:42] <grrrit-wm>	 (03CR) 10Ricordisamoa: [C: 032] Use url_for() instead of bare relative URLs in templates [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/268672 (owner: 10Ricordisamoa)
[14:38:09] <grrrit-wm>	 (03Merged) 10jenkins-bot: Use url_for() instead of bare relative URLs in templates [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/268672 (owner: 10Ricordisamoa)
[15:12:14] <Ricordisamoa>	 hi
[15:12:29] <Ricordisamoa>	 i've messed up permissions again
[15:12:37] <Ricordisamoa>	 please delete /data/project/ptable/www/python recursively
[15:16:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-worker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[15:18:46] <Ricordisamoa>	 thanks in advance
[15:23:46] <moritzm>	 headsup, the server maintenance mentioned on wikitech is now starting
[15:24:00] <moritzm>	 you can also follow the steps via https://wikitech.wikimedia.org/wiki/Server_Admin_Log, 
[15:25:35] <Ricordisamoa>	 :(
[15:32:50] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GratusBot was created, changed by GratusBot link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/GratusBot edit summary: Created page with "{{Tools Access Request |Justification=Hello, I wan't use the Tools project for use https://github.com/kaldari/hotarticles in french wikipedia (after bot's request). |Completed..."
[15:35:23] <andrewbogott>	 I’m rebooting the wikitech host, so wikitech will be down for a few minutes
[15:41:00] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[15:43:14] <shinken-wm>	 PROBLEM - Puppet failure on tools-cron-02 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0]
[15:45:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[15:45:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-elastic-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[15:45:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]
[15:47:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0]
[15:47:16] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[15:47:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[15:48:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0]
[15:48:16] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[15:48:17] <shinken-wm>	 PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[15:48:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-k8s-master-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[15:48:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]
[15:49:32] <andrewbogott>	 wikitech is back up, but I’m resetting user sessions so probably best to not log in for a few minutes.
[15:53:08] <gehel>	 !log Deployment-prep restarting deployment-elastic07
[15:53:08] <labs-morebots>	 Deployment-prep is not a valid project.
[15:57:35] <gehel>	 !log deployment-prep restarting deployment-elastic07
[15:57:39] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master
[16:10:39] <andrewbogott>	 wikitech should now be completely back to normal.  Everyone will need to log in afresh.
[16:21:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:22:25] <grrrit-wm>	 (03CR) 10Tim Landscheidt: "recheck" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268563 (owner: 10Tim Landscheidt)
[16:22:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:23:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-cron-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:26:05] <shinken-wm>	 RECOVERY - Puppet failure on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:27:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:28:21] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:28:21] <shinken-wm>	 RECOVERY - Puppet failure on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:28:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:32:59] <greg-g>	 where's wikibugs?
[16:36:16] <grrrit-wm>	 (03CR) 10Tim Landscheidt: "Filed T125999 for the Jenkins failure; basically, it tries to install an obsolete version of a package that mirrors.wikimedia.org no longe" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268563 (owner: 10Tim Landscheidt)
[16:44:10] <Krenair>	 greg-g, it's here now
[16:44:22] <Krenair>	 and should join other channels as it becomes necessary to send messages to them, I think
[16:45:02] <greg-g>	 Krenair: ty
[16:49:31] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GratusBot was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=288898 edit summary: 
[16:49:59] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Avm99963 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=288901 edit summary: 
[16:52:28] <wikibugs>	 6Labs, 10Tool-Labs, 10DBA: labs copy of nlwiktionary db corrupted - https://phabricator.wikimedia.org/T125927#2002554 (10jcrespo) 5Open>3Resolved Fixed, the 14 pending are *not* a labs db issue:  ``` $ mysql -h s3-master nlwiktionary -e "SELECT count(rev_id) FROM revision LEFT JOIN page ON rev_page=page_...
[16:54:14] <wikibugs>	 6Labs, 10Tool-Labs, 10DBA: labs copy of nlwiktionary db corrupted - https://phabricator.wikimedia.org/T125927#2002559 (10jcrespo) Also: ``` $ mysql -h labsdb1003 nlwiktionary -e "SELECT count(*) from page where page_namespace=0 and page_title='keelholte'" +----------+ | count(*) | +----------+ |        1 | +...
[17:00:16] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: Wikitech login token form field should use mw-ui-input so it's obvious it is an input field - https://phabricator.wikimedia.org/T125945#2002573 (10Florian) a:3Florian Oha, you're right. It's the login page "for small screens". Let me see if...
[17:00:26] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GratusBot was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=288928 edit summary: 
[17:00:30] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10MobileFrontend, 10wikitech.wikimedia.org: Wikitech login token form field should use mw-ui-input so it's obvious it is an input field - https://phabricator.wikimedia.org/T125945#2002575 (10Florian)
[17:00:39] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GratusBot was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=288929 edit summary: 
[17:03:54] <Luke081515>	 anomie: Currently here?
[17:04:06] <anomie>	 Luke081515: Yes, but busy at the moment
[17:04:46] <Luke081515>	 anomie: Ok, then feel free to ping me, if you are less busy
[17:12:32] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10MobileFrontend, 10wikitech.wikimedia.org, 5Patch-For-Review: Wikitech login token form field should use mw-ui-input so it's obvious it is an input field - https://phabricator.wikimedia.org/T125945#2002617 (10Florian) Result of the above change: {F3313568}
[17:14:32] <wikibugs>	 6Labs, 10Tool-Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org, and 2 others: Create a wiki documentation page for each tool - https://phabricator.wikimedia.org/T122865#2002620 (10bd808) Consensus poll now running until 2016-02-19. See <https://lists.wikimedia.org/pipermail/labs-l/2016-February/00...
[17:18:08] <wikibugs>	 6Labs, 10Tool-Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org, and 2 others: Create a wiki documentation page for each tool - https://phabricator.wikimedia.org/T122865#2002624 (10bd808)
[17:33:06] <wikibugs>	 6Labs, 10Beta-Cluster-Infrastructure: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#2002681 (10hashar) I have cleaned a few old logs files and obsolete ones from `/data/project`.
[17:51:48] <anomie>	 Luke081515: I'm unbusy now, but I have a meeting in about 8 minutes.
[17:52:07] <Luke081515>	 ok, I only got one question
[17:52:36] <Luke081515>	 extension:maths requires action=purge&mathpurge=true to purge via web
[17:52:50] <Luke081515>	 anomie: Do you know if there is a way to do this too via API?
[17:56:34] <Cyberpower678>	 anomie, when will .13 be deployed, and will that require me to restart my bots again?
[17:56:34] <anomie>	 Luke081515: You can probably just add "mathpurge=true" to the api.php query and ignore the unrecognized parameter warning. Ideally the extension would provide a better way to do it than just grabbing a random query string parameter though.
[17:56:50] <Luke081515>	 anomie: Ok, thanks :)
[17:57:34] <anomie>	 Cyberpower678: https://www.mediawiki.org/wiki/MediaWiki_1.27/Roadmap, and hopefully not
[17:57:59] <Cyberpower678>	 anomie, page doesn't exist
[17:58:28] <Cyberpower678>	 Well so far every deploy destroyed my, my bot's sessions.
[17:58:41] <anomie>	 Cyberpower678: Don't include the comma
[17:59:39] <Cyberpower678>	 ah
[18:32:12] <wikibugs>	 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: Create a Tool namespace on wikitech for documentation of Tool Labs projects - https://phabricator.wikimedia.org/T123429#2003018 (10Purodha) If there is no way to automatically fill in some useful content, and unless there is a way for tool owners to opt out, I op...
[18:49:05] <multichill>	 !log nlwikibots Archivering uitgezet. Er werden pagina's zoals https://nl.wikipedia.org/wiki/Wikipedia:Verzoekpagina_voor_moderatoren/Terugplaatsen/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01 aangemaakt
[18:49:06] <labs-morebots>	 nlwikibots is not a valid project.
[18:49:26] <multichill>	 !log tools.nlwikibots Archivering uitgezet. Er werden pagina's zoals https://nl.wikipedia.org/wiki/Wikipedia:Verzoekpagina_voor_moderatoren/Terugplaatsen/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01/Archief/2016_01 aangemaakt
[18:49:29] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.nlwikibots/SAL, Master
[18:50:07] <multichill>	 valhallasw`cloud: ^
[18:53:53] <valhallasw`cloud>	 multichill: template op verkeerde plaats op de pagina
[18:54:32] <valhallasw`cloud>	 Waardoor template mee verplaatst wordt, etc
[19:09:49] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2003299 (10demon) >>! In T122993#2001037, @bd808 wrote: > Maybe there is a way we can (ab)use [[https://www.mediawiki.org/wiki/Help:CirrusSearch#Boost-templates|boost-template...
[19:18:29] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:34:30] <wikibugs>	 6Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 7Mobile: Install MobileFrontend on wikitech - https://phabricator.wikimedia.org/T87633#2003377 (10Florian)
[19:34:32] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10MobileFrontend, 10wikitech.wikimedia.org, 5Patch-For-Review: Wikitech login token form field should use mw-ui-input so it's obvious it is an input field - https://phabricator.wikimedia.org/T125945#2003376 (10Florian) 5Open>3Resolved
[19:45:21] <wikibugs>	 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: Create a Tool namespace on wikitech for documentation of Tool Labs projects - https://phabricator.wikimedia.org/T123429#2003428 (10bd808) >>! In T123429#2003018, @Purodha wrote: > If there is no way to automatically fill in some useful content, and unless there i...
[19:54:55] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2003482 (10Sadads) Hey all, discovered that @MusikAnimal has been working on the demo API to make it more functional: https://tools.wmflabs.org/musikanimal/pagev...
[20:03:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:09:45] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2003522 (10MusikAnimal) Indeed. I thought you all were close to finishing the native interface? T125917  My fork of marcelrf's app just adds history push states...
[20:35:23] <wikibugs>	 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2003617 (10hashar) The jenkins-debian-glue scripts invokes `cowbuilder --update` which refresh the image (time UTC):  ``` 16:23:21 I: upgr...
[20:39:48] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Reading-Web, 6operations, and 3 others: https://wikitech.m.wikimedia.org/ serves wikimedia.org portal - https://phabricator.wikimedia.org/T120527#2003637 (10Dzahn) >>! In T120527#1870974, @Krenair wrote: > That domain would need to be changed into a CNAME for silver for this...
[20:44:44] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:59:29] <grrrit-wm>	 (03PS1) 10Awight: Send Education-* to wikimedia-ed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/268794 
[21:09:21] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2003773 (10Nuria) >Indeed. I thought you all were close to finishing the native interface?  FYI that analytics team is not working on this near term.    >My fork...
[21:20:01] <twentyafterfour>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: invalid byte sequence in US-ASCII at /etc/puppet/modules/scap/manifests/init.pp:1 on node deploy.phabricator.eqiad.wmflabs
[21:20:26] <twentyafterfour>	 anyone know how to make a labs instance puppet agent recognize utf8?  The locale is set correctly in the OS
[21:20:47] <twentyafterfour>	 all the help I find online points to fixing the puppetmaster - but I assume labs puppetmaster isn't the culprit?
[21:22:56] <chasemp>	 it would be odd if it was different from prod as it's basically teh same setup
[21:23:12] <chasemp>	 afaik
[21:25:06] <valhallasw`cloud>	 twentyafterfour: is this in cron?
[21:25:16] <valhallasw`cloud>	 twentyafterfour: cron uses the C locale by default
[21:26:23] <twentyafterfour>	 valhallasw`cloud: no running it from the cli
[21:26:50] <valhallasw`cloud>	 hm, wait, it's also an error on the *server* rather than on the client
[21:26:51] <twentyafterfour>	 relatedly: I got a similar error in production recently but the error was about invalid UTF8
[21:27:04] <valhallasw`cloud>	 so maybe the labs puppetmaster has a slightly different configuration
[21:27:04] <valhallasw`cloud>	 ?
[21:27:20] <twentyafterfour>	 valhallasw`cloud: I don't think I have access to labs puppetmaster to answer that question
[21:28:24] <twentyafterfour>	 it seems as if labs puppetmaster doesn't support utf8... only a very few of our puppet files are saved as utf8
[21:28:38] <twentyafterfour>	 so the problem wouldn't normally surface for most people I'd guess
[21:28:56] <valhallasw`cloud>	 it seems it's complaining about the UTF-8 BOM, but I don't see anything odd in the file
[21:28:59] * valhallasw`cloud git pulls
[21:29:22] <twentyafterfour>	 valhallasw`cloud: if it's set to ASCII then even the BOM would cause it to abort, right?
[21:29:53] <twentyafterfour>	 I doubt the file needs to be in UTF8 but saving it as ascii is only a workaround, fixing the server would be the right fix, IMO
[21:30:33] <valhallasw`cloud>	 valhallasw@maeglin:manifests$ hexdump -C init.pp| head
[21:30:33] <valhallasw`cloud>	 00000000  23 20 3d 3d 20 43 6c 61  73 73 20 73 63 61 70 0a  |# == Class scap.|
[21:30:45] <valhallasw`cloud>	 there's nothing obviously weird there :/
[21:32:18] <twentyafterfour>	 so then it is an incorrectly configured puppetmaster for labs?
[21:33:57] <valhallasw`cloud>	 I'm not sure -- there's no non-ascii characters there (first digit of each hex byte is < 8)
[21:34:58] <valhallasw`cloud>	 maybe it's something else altogether -- facter returning something non-ascii?
[21:39:26] <chasemp>	 twentyafterfour: run w/ --debug?
[21:43:09] <twentyafterfour>	 chasemp: what should I look for?
[21:44:10] <chasemp>	 :) not sure, did you see valhallasw`cloud changeset tho?
[21:44:15] <chasemp>	 -ops
[21:47:09] <twentyafterfour>	 I do now
[22:01:28] <chasemp>	 !log tools throttle some vm nfs write speeds
[22:01:33] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[22:27:12] <wikibugs>	 6Labs, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: Permissions error on creating Hiera pages on wikitech - https://phabricator.wikimedia.org/T121602#2004069 (10madhuvishy) @Krenair - I still don't see that I can edit - https://wikitech.wikimedia.org/wiki/Hiera:Dashiki/host/dashiki-staging-01....
[22:35:17] <wikibugs>	 6Labs, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: Permissions error on creating Hiera pages on wikitech - https://phabricator.wikimedia.org/T121602#2004098 (10Krenair) 5Resolved>3Open Oops, we need to enable subpages in that namespace.
[22:36:52] <wikibugs>	 6Labs: Return NFS to its standard ports - https://phabricator.wikimedia.org/T103052#2004112 (10chasemp) 5Open>3Resolved a:3chasemp assuming the standard 2049 is the target seems resolved
[22:37:07] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2004115 (10MusikAnimal) @Nuria I can add in code to skip push/pop states if it's not supported. That's easy to do, but I don't plan on putting much more effort i...
[22:42:47] <wikibugs>	 6Labs, 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: Permissions error on creating Hiera pages on wikitech - https://phabricator.wikimedia.org/T121602#2004150 (10Krenair) 5Open>3Resolved
[23:05:17] <wikibugs>	 6Labs: Change the way manage-nfs-volumes is monitored - https://phabricator.wikimedia.org/T91806#2004231 (10chasemp) I think this is obsolete now.  will confirm.
[23:06:09] <wikibugs>	 6Labs, 10Tool-Labs: labs NFS slowness / high load - https://phabricator.wikimedia.org/T122743#2004233 (10chasemp) 5Open>3Resolved a:3chasemp I am resolving not because it's sunshine and roses but as this fall under the guise of other ongoing work.  Let me know if there is anything specific.
[23:09:09] <wikibugs>	 6Labs, 6operations: evaluate possibility for nscd use with useldap - https://phabricator.wikimedia.org/T124991#2004248 (10chasemp)
[23:09:11] <wikibugs>	 6Labs, 10Tool-Labs: user `marcmiquel` gzipping 41G file on NFS - https://phabricator.wikimedia.org/T124877#2004249 (10chasemp)
[23:09:13] <wikibugs>	 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: templatetiger runs `sort` on huge NFS files - https://phabricator.wikimedia.org/T124822#2004250 (10chasemp)
[23:09:15] <wikibugs>	 6Labs, 10Tool-Labs: labs NFS slowness / high load - https://phabricator.wikimedia.org/T122743#2004251 (10chasemp)
[23:09:19] <wikibugs>	 6Labs, 10Tool-Labs: Prevent overly-large log files - https://phabricator.wikimedia.org/T122508#2004252 (10chasemp)
[23:09:23] <wikibugs>	 6Labs, 10Labs-Sprint-105: NFS exports broken for new projects which want NFS - https://phabricator.wikimedia.org/T104881#2004255 (10chasemp)
[23:09:25] <wikibugs>	 6Labs, 6operations: One instance hammering on NFS should not make it unavailable to everyone else - https://phabricator.wikimedia.org/T95766#2004257 (10chasemp)
[23:09:27] <wikibugs>	 6Labs, 10Labs-Sprint-100, 10Tool-Labs: Clean up huge logs on toollabs - https://phabricator.wikimedia.org/T98652#2004256 (10chasemp)
[23:09:29] <wikibugs>	 6Labs, 10Labs-Infrastructure: New instances don't always get their /data and /home project mounted from NFS - https://phabricator.wikimedia.org/T91822#2004258 (10chasemp)
[23:09:31] <wikibugs>	 6Labs: Change the way manage-nfs-volumes is monitored - https://phabricator.wikimedia.org/T91806#2004259 (10chasemp)
[23:09:35] <wikibugs>	 6Labs, 5Patch-For-Review: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#2004260 (10chasemp)
[23:16:23] <wikibugs>	 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2004308 (10scfc) AFAIUI `debian-glue` sits on top of `package_builder`, so adding @akosiaris.  On closer look, I noted another error more...
[23:28:05] <harej>	 Once I set up a web proxy on a labs instance, how long should it take to take effect?
[23:28:51] <Krenair>	 instantly, I think?
[23:28:53] <Krenair>	 are you having issues?
[23:29:13] <harej>	 http://wpx.wmflabs.org gets an error 504 from nginx, which is odd because my instance uses apache
[23:31:09] <Krenair>	 what is your labs project name?
[23:31:12] <harej>	 wpx
[23:31:30] <harej>	 i have it set up to point wpx.wmflabs.org to http://wpx-prod-01.wpx.eqiad.wmflabs:80
[23:32:43] <Krenair>	       "backends": [
[23:32:43] <Krenair>	         "http://wpx-prod-01.wpx.eqiad.wmflabs:80"
[23:32:44] <Krenair>	       ], 
[23:32:44] <Krenair>	       "domain": "wpx.wmflabs.org"
[23:32:57] <Krenair>	 ok
[23:33:47] <Krenair>	 harej, you sure it's listening on that port?
[23:34:16] <harej>	 well i do lynx http://127.0.0.1:80 in the terminal and it works fine
[23:34:50] <bd808>	 do you have port 80 open on your instance? Via https://wikitech.wikimedia.org/wiki/Special:NovaSecurityGroup
[23:34:59] <Krenair>	 I think you may have to play with security groups
[23:35:00] <Krenair>	 yeah
[23:35:06] <Krenair>	 I don't see port 80 open
[23:35:09] <bd808>	 by default that isn't open
[23:35:11] <harej>	 ohhhh.
[23:35:16] <harej>	 that'll do it
[23:35:23] <bd808>	 secret Labs tricks
[23:35:33] * bd808 wonders where that is documented
[23:35:43] <Krenair>	 bd808, clearly you are new here :p
[23:36:07] <Krenair>	 :D
[23:36:13] <bd808>	 I'm actually just trying to make it better :P
[23:36:15] <harej>	 which protocol, CIDR ranges do I want?
[23:36:24] <bd808>	 tcp 0.0.0.0/0
[23:36:47] <bd808>	 that just opens it up to all of Labs internally
[23:37:02] <harej>	 works! thank you
[23:37:29] <bd808>	 harej: where on wikitech would you have hoped to find a note about that?
[23:38:18] <bd808>	 I did find this -- https://wikitech.wikimedia.org/wiki/Help:LAMP_issues#502_Bad_Gateway_nginx.2F1.1.19
[23:38:32] <harej>	 Is there a page about babby's first labs instance? if so it would go there
[23:38:38] <harej>	 (even though, er, this is the second one i've created)
[23:38:48] <bd808>	 :) that would be a good page to create actually
[23:39:42] <bd808>	 "Creating an instance for smarties"
[23:40:00] <harej>	 "Beginner traps"
[23:40:31] <bd808>	 I'm not under the illusion that people actually read docs before they hit a problem but some percentage actually do after
[23:46:54] <wikibugs>	 6Labs: Create a "Beginners guide to creating a Labs instance" wiki page - https://phabricator.wikimedia.org/T126094#2004452 (10bd808) 3NEW
[23:47:27] <mobrovac>	 bd808: murphy's law says: "when it doesn't work, read the manual"
[23:47:48] <bd808>	 "read the second hit in the google search"
[23:47:58] <mobrovac>	 albeit, there's another one of his laws that simply states "it doesn
[23:48:01] <mobrovac>	 it doesn't work
[23:48:31] <wikibugs>	 6Labs: Create a "Beginners guide to creating a Labs instance" wiki page - https://phabricator.wikimedia.org/T126094#2004462 (10bd808)
[23:48:33] <wikibugs>	 6Labs, 6Developer-Relations, 10wikitech.wikimedia.org, 7Epic: [EPIC] Make wikitech more friendly for the multiple audiences it supports - https://phabricator.wikimedia.org/T123425#2004461 (10bd808)
[23:53:08] <wikibugs>	 6Labs, 6operations: One instance hammering on NFS should not make it unavailable to everyone else - https://phabricator.wikimedia.org/T95766#2004476 (10chasemp) on vm usage patterns for write  https://phabricator.wikimedia.org/T126083#2004459  I am looking at shaping traffic using tc to some extent (though pre...