[00:59:07] Do you have logging turned on? [00:59:32] just undo your last commit [01:41:02] CoolCanuck, it's not about my last commit [01:41:22] oh [01:41:27] CoolCanuck, somebody did a big quantity of commits, its not a alone project [01:41:43] And he is sleeping right now [01:41:52] gotcha [01:41:58] ;) [03:35:22] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:56:05] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 383 bytes in 2.002 second response time [04:01:03] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3669 bytes in 0.031 second response time [04:34:20] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:09:21] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [06:25:45] (03CR) 10Ricordisamoa: "PS66 ref T109584" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [06:47:02] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:52:57] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:54:29] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:58:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:58:57] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:59:45] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:59:49] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:00:07] PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:01:42] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:04:40] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:07:04] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:09:28] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:13:49] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:14:05] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:14:35] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:32:09] 06Labs, 10Labs-Infrastructure, 10DBA: labsdb1001 crashed yesterday at 21:48:07 - https://phabricator.wikimedia.org/T135971#2317328 (10jcrespo) [07:47:10] 06Labs, 10DBA: Lots of rows are missing from enwiki_p.`revision` - https://phabricator.wikimedia.org/T115207#2317355 (10jcrespo) a:03jcrespo [07:56:40] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2317370 (10jcrespo) These also have been imported: ``` 610990450 templatelinks 119323634 externallinks 103124031 categorylinks 92999329 user_properties 80917852 imagelinks ``` Revision table is ongoing now, b... [08:17:04] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 383 bytes in 2.005 second response time [08:22:03] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3669 bytes in 0.031 second response time [10:54:09] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2317668 (10hashar) I have checked after the week-end and deployment-cache-upload04 shows the FD leak. Via `lsof -X -n|grep deleted`: * Lot of... [11:05:20] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2317675 (10Joe) @hashar the reason you see all those deleted "varnishd" lines is that varnish has been updated on disk but not restarted, which... [11:10:05] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2317678 (10Joe) So the problem - that we have in production too (!!!) is that the logrotate receipt calls ``` invoke-rc.d varnishlog reload ```... [11:11:31] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: Varnishlog doesn't properly rotates logs, varnish.log is empty since forever (was: deployment-cache-upload04 (m1.medium) / is almost full) - https://phabricator.wikimedia.org/T135700#2317679 (10Joe) p:05Low>03High [11:15:03] (03CR) 10Jean-Frédéric: [C: 032] Re-add wikitext in statistics id [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/289427 (https://phabricator.wikimedia.org/T55688) (owner: 10Lokal Profil) [11:15:52] (03Merged) 10jenkins-bot: Re-add wikitext in statistics id [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/289427 (https://phabricator.wikimedia.org/T55688) (owner: 10Lokal Profil) [11:22:59] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:34:08] !log tools.wikibugs temporarily offline for mass edit by Danny_B [11:34:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [11:50:06] 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#2318058 (10valhallasw) fdsa [11:56:47] (03PS1) 10Youni Verciti: Rev 0.7 Etape 1 from Aide to Transwiki with nsLib [labs/tools/fr-wikiversity-ns] - 10https://gerrit.wikimedia.org/r/290203 [12:03:03] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [12:17:39] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2318111 (10Danny_B) [12:38:14] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: Varnishlog doesn't properly rotates logs, varnish.log is empty since forever (was: deployment-cache-upload04 (m1.medium) / is almost full) - https://phabricator.wikimedia.org/T135700#2318200 (10Joe) A third option is we just stop varnishlog as... [12:38:26] 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2318201 (10Maathavan) Maathavanbot என்ற கணக்கே வேலை செய்யவில்லை. [12:50:40] !log tools.heritage Deployed latest from Git, 50915bf (T55688) [12:50:42] T55688: Statistics module uses country field instead of lang field to link to Wikipedia - https://phabricator.wikimedia.org/T55688 [12:50:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [13:03:46] 10Wikibugs: Create function to temporarily mute reports - https://phabricator.wikimedia.org/T75900#784787 (10Danny_B) Mass changes are typically crossproject, so plain general `@mute` / `@speak` commands would be enough. Muting on one channel would cause wikibugs to notice on every channels it is that it has be... [13:12:18] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic, 13Patch-For-Review: Varnishlog doesn't properly rotates logs, varnish.log is empty since forever (was: deployment-cache-upload04 (m1.medium) / is almost full) - https://phabricator.wikimedia.org/T135700#2318265 (10Joe) 05Open>03Resolved [13:28:06] !log tools 'apt-get install hhvm -y --force-yes' across trusty hosts to handle hhvm downgrade [13:28:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [13:49:04] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [13:53:56] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [14:02:59] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [14:06:18] 06Labs: raise quota limit for project video - https://phabricator.wikimedia.org/T135560#2318421 (10Andrew) 05Open>03Resolved You should have enough headroom to create an m1.small now. Please reopen if you run into trouble. [14:09:38] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [14:16:40] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [14:20:59] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet is hanging on labvirt1003 - https://phabricator.wikimedia.org/T135850#2318462 (10Andrew) p:05Triage>03High [14:21:46] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2318463 (10Andrew) 05Open>03Resolved This seems better! [14:36:12] PROBLEM - SSH on tools-webgrid-lighttpd-1408 is CRITICAL: Server answer [16:10:54] !log tools.stashbot Bot died due to https://github.com/bd808/tools-stashbot/issues/9 [16:10:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [16:18:25] 10Tool-Labs-tools-Other, 06Wikisource: OCR scripts need updating at tools labs by updating the "tesseract-ben" package - https://phabricator.wikimedia.org/T117711#2318951 (10Bodhisattwa) [16:19:29] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-elapsed-time/en]] i18n issue - https://phabricator.wikimedia.org/T136013#2318958 (10Aklapper) [16:19:38] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-elapsed-time/en]] i18n issue - https://phabricator.wikimedia.org/T136013#2318931 (10Aklapper) [16:31:19] !log tools.stashbot Taking bot offline while I try to figure out what is wrong with tool labs ES clsuter [16:31:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [16:33:35] !log tools Rebooting tools-elastic-02.tools.eqiad.wmflabs [16:33:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:41:48] 06Labs, 10Labs-Infrastructure, 10DBA: Missing grants on tools.labsdb - https://phabricator.wikimedia.org/T135947#2316117 (10valhallasw) The relevant SQL is here: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/labstore/files/create-dbusers;774873e507ca990399f49f370957f33635a0803f$1... [16:44:32] !log tools.stashbot Bot back online [16:44:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [16:46:25] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2319090 (10kaldari) I'll take a look. [17:03:06] NOTICE: I'm going to suspend all instances on labvirt1003 for a few minutes. Affected instances: https://phabricator.wikimedia.org/P3159 [17:12:24] way to go, nova, apparently 'nova suspend' is code for 'stop instance and drop into an error state' [17:15:31] PROBLEM - Host tools-proxy-01 is DOWN: CRITICAL - Host Unreachable (10.68.21.49) [17:15:53] PROBLEM - Host tools-services-02 is DOWN: CRITICAL - Host Unreachable (10.68.18.36) [17:16:12] PROBLEM - Host tools-checker-02 is DOWN: CRITICAL - Host Unreachable (10.68.16.17) [17:30:16] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:08] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:22] are we expecting a longer downtime? [17:49:54] RECOVERY - Host tools-proxy-01 is UP: PING OK - Packet loss = 0%, RTA = 0.97 ms [17:50:50] RECOVERY - Host tools-services-02 is UP: PING OK - Packet loss = 0%, RTA = 0.88 ms [17:51:12] RECOVERY - Host tools-checker-02 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms [18:05:46] RECOVERY - Puppet staleness on tools-services-02 is OK: OK: Less than 1.00% above the threshold [3600.0] [18:17:54] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet is hanging on labvirt1003 - https://phabricator.wikimedia.org/T135850#2320581 (10Andrew) 05Open>03Resolved Reboot seems to've fixed whatever this was. [18:24:03] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2320614 (10RobH) >>! In T131184#2299085, @EBernhardson wrote: > I'm thinking it will be simpler to give them a service cluster name, makes thi... [18:26:33] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2320621 (10EBernhardson) Service cluster documented [18:40:40] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:43:17] PROBLEM - Puppet run on tools-exec-1210 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:44:51] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:45:09] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:45:23] PROBLEM - Puppet run on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:50:10] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2320711 (10kaldari) It looks like it's still in use, but it has data going back to 2009. Since the reports are output to Wikipages, I don't think there's any reason... [18:55:30] RECOVERY - Puppet staleness on tools-grid-master is OK: OK: Less than 1.00% above the threshold [3600.0] [18:55:40] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2320736 (10kaldari) I deleted all the data older than 2010 as a test. If there are no issues with the next report generation, I'll delete more. To connect to the da... [19:10:18] https://wikitech.wikimedia.org/wiki/Special:UserLogin/signup looks terrible [19:10:23] it's defaced [19:11:12] 06Labs: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#2320760 (10bd808) While chatting with @ZhouZ I had an idea for an opt-out system: * Tool X needs the user to interact with a 3rd-party service directly * Tool... [19:11:58] Krinkle: I don't see anything odd? [19:11:58] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by 197.156.95.154 link https://www.mediawiki.org/w/index.php?diff=2146087 edit summary: [+1] adunya.akamit.jirti [19:12:10] valhallasw`cloud: Title is missing, everything just floats in air. It's a mess. [19:12:18] The message about terms at the top is also missing styles [19:12:34] It uses {{notice}} but its mediawiki common css requirement is not fulfilled due to security policy on that page [19:12:37] * Krinkle removes the template [19:12:49] It has page title "- Wikitech" [19:12:51] which is odd [19:13:16] https://wikitech.wikimedia.org/w/index.php?title=MediaWiki%3ACreateaccount&type=revision&diff=64942&oldid=64940 [19:15:27] RECOVERY - Puppet run on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [19:15:28] Also the " is made by people like you" is shown on the bottom of the page [19:15:32] Instead of along the side [19:21:11] Why is it at the bottom? [19:21:18] 06Labs, 10wikitech.wikimedia.org, 07Regression: Wikitech sign-up page is defaced - https://phabricator.wikimedia.org/T136032#2320821 (10Krinkle) [19:21:22] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Legoktm link https://www.mediawiki.org/w/index.php?diff=2146090 edit summary: [-1] Reverted edits by [[Special:Contributions/197.156.95.154|197.156.95.154]] ([[User talk:197.156.95.154|talk]]) to last revision by [[User:BDavis (WMF)|BDavis (WMF)]] [19:21:56] Krinkle: 'defaced' suggests it's full of swastikas [19:22:04] someone forgot to close a tag in a message somewhere? [19:22:12] (also what valhallasw`cloud said -.-) [19:27:12] 06Labs, 10wikitech.wikimedia.org, 07Regression: Wikitech sign-up page is defaced - https://phabricator.wikimedia.org/T136032#2320857 (10Krinkle) Fixed empty title by reverting [this edit](https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Createaccount&diff=64942&oldid=64940) from 2013. Looks like th... [19:28:23] 06Labs, 10wikitech.wikimedia.org, 07Regression: Wikitech sign-up page has bad styling following AuthManager rollout - https://phabricator.wikimedia.org/T136032#2320862 (10bd808) [19:36:46] !log tools switched tools-checker to tools-checker-03 [19:36:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [19:52:14] 06Labs, 10wikitech.wikimedia.org, 07Regression: Wikitech sign-up page has bad styling following AuthManager rollout - https://phabricator.wikimedia.org/T136032#2320939 (10Anomie) Since `$wgDisableAuthManager = true` is set on Wikitech like everywhere else, there shouldn't be any effects from AuthManager goin... [19:56:20] 06Labs, 10wikitech.wikimedia.org, 07Regression: Wikitech sign-up page has bad styling following AuthManager rollout - https://phabricator.wikimedia.org/T136032#2320954 (10Anomie) I also see that https://web.archive.org/web/20160420225342/https://wikitech.wikimedia.org/wiki/Special:UserLogin/signup looks the... [19:59:16] 06Labs, 10wikitech.wikimedia.org, 07Regression: Wikitech sign-up page has bad styling - https://phabricator.wikimedia.org/T136032#2320956 (10Krinkle) [20:26:00] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:38:19] PROBLEM - Host ToolLabs is DOWN: check_ping: Invalid hostname/address - tools.wmflabs.org [20:38:59] zhmm [20:39:01] andrewbogott: ^ [20:39:03] more DNS? [20:39:21] YuviPanda: probably, I should have it fixed in a moment [20:39:26] kk [20:41:04] Is there anything happening now, ores just went down [20:41:21] Amir1: DNS issues [20:41:29] YuviPanda: thanks :) [20:42:08] I'm worrying that I will never understand what hiera does [20:42:09] RECOVERY - Host ToolLabs is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [20:42:13] but, ok, should be fixed now [20:43:10] hiera input strings, output Tears_and_Anguish [20:43:25] Are labs instances not affected by eqiad.yaml? [20:43:40] Because… https://gerrit.wikimedia.org/r/#/c/290314/ [20:43:48] totally did not do what I expected [20:44:23] they aren't [20:44:29] andrewbogott: they are not [20:44:34] labs instances are on a totally different hierarchy [20:44:35] modules/puppetmaster/files/labs.hiera.yaml [20:44:40] doesn't take into account site [20:44:43] well… ok then [20:44:50] let's try this again then [21:30:15] My instance wpx-prod-01 is shut off. I tried rebooting it in https://wikitech.wikimedia.org/wiki/Special:NovaInstance and it failed to reboot. Anyone know what's going on? [21:31:56] andrewbogott: ^ [21:32:06] shall I reboot it for harej? [21:32:20] (If someone at the WMF was upset at me for some reason I would've gotten an email about it right?) [21:32:23] also my battery is gonna die soon [21:32:26] harej: yes [21:32:27] YuviPanda: sure, although I don't know that you can do anything he can't do [21:32:37] !log ores-staging manually rebooting sabya-precached.ores-staging.eqiad.wmflabs [21:32:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores-staging/SAL, Master [21:32:43] andrewbogott: I was gonna do it on nova cmdline [21:32:53] yeah, in theory that does the same thing that horizon does [21:32:54] but, do try [21:32:57] ah i see [21:32:59] ok [21:33:08] andrewbogott: in that case should I just defer to you? my laptop's gonna die in 12mins [21:33:14] I assumed it would just be a simple nova reboot [21:33:15] sure [21:34:08] (I can't get over the fact that we've hacked MediaWiki to have it manage VMs.) [21:34:36] (It's so brilliantly insane, like turning a cat into a drone.) [21:35:57] harej: try now? [21:36:23] Anyways, once my server is back up I'll need to create an account for someone. I haven't had to do this in a while, but it's basically the same thing as the normal Debian command, isn't it? How does it hook into the LDAP universal login? [21:36:55] Server's back up. [21:36:57] What went wrong? [21:37:29] harej: I don't know. It was in 'shutoff' state, and I started it. [21:37:37] Brilliant. [21:37:44] harej: for creating an account... [21:37:55] you should just have the user create a wikitech account and then add them to your project [21:38:16] ...Oh, right. [21:44:47] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&instanceid=52221813-455f-4eb8-b6cd-860916c83f2f&project=ores-staging®ion=eqiad [21:44:58] " failed to bind to LDAP server ldap://ldap-labs.eqiad.wikimedia.org:389: Invalid credentials" [21:45:01] is it normal [21:45:04] ? [21:45:23] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:50:47] Amir1: it's hard to say if it's normal or not… what are you trying to do? [21:50:55] (I can't easily view that link, though) [21:51:17] andrewbogott: I acutally just wanted to connect [21:51:34] but I deleted that instance, lots of bad configurations there already [21:51:45] ok, sounds like it's moot for now then :) [21:51:59] yeah [21:52:02] thanks :) [22:08:36] 06Labs: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#2321327 (10tom29739) Why not use an OAuth system for something like this? (OAuth 2), preferably). 😃 [22:10:19] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [22:14:00] 06Labs: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#2321339 (10tom29739) Another thing, if you used cookies, then it would an all or nothing approach because all cookies can be read across the tools domain. [22:33:50] !log ores stopping precaching in ores-web-03 manually. Testing something [22:33:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [22:38:52] !log ores precaching brought back online [22:38:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [22:43:11] labs-recursor1.wm.org is reported as DOWN [22:46:03] mutante: new service andrewbogott is working on I think. [22:46:51] YuviPanda: alright! [23:06:45] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Update tag and racktables for holmium: rename to labservices1002. - https://phabricator.wikimedia.org/T119533#2321490 (10Andrew) a:05Andrew>03None [23:18:01] hi folks, could I (BenKurtovic) be added to bastion? seems I'm not part of it for some reason... [23:20:42] no [23:21:29] thanks lego knew I could count on you [23:22:51] bd808 can probably help you. Though I thought all users were automatically added to bastion once getting shell access... [23:23:02] me too, that's why I'm confused [23:23:19] * bd808 is looking at ldap... [23:30:20] Earwig: I added you back to the bastion project. I have no idea how you lost that memebership [23:30:54] thanks, it works now. I don't think I was ever a part of it [23:31:05] very strange [23:32:16] bd808: that change happened maybe a year-ish ago [23:32:28] so people who had accoutns before that don't get it [23:33:04] YuviPanda: so they can't ssh into projects? [23:33:15] I could always get into tool labs [23:33:23] but that has a public IP and doesn't go through bastion, so... [23:33:32] bd808: tools didn't require bastion access [23:34:12] He's in the wpx project too, but maybe that grant is older than the change [23:35:02] bd808: I think that was just done today [23:36:01] I thought there was some magic that checked and automatically gave you bastion access when you were added to a project? [23:36:06] anyhow should be fixed now [23:38:06] oh right [23:38:09] that should've worked then [23:38:13] not sure why not [23:47:17] o/ I'm having puppet failures on snuggle-en.eqiad.wmflabs. It looks like puppet just hangs on "Applying configuration version '1464042873'" [23:47:20] Any suggestions? [23:47:51] halfak: is it killable? does ctrl-c work? [23:47:58] halfak: and what's output of 'ls /public/dumps [23:48:00] ' [23:48:06] ^C is hanging [23:48:45] 'ls /public/dumps' hangs and ^C doesn't work lol [23:48:56] It's still on Ubuntu 12.04 [23:49:01] COuld that be the problem? [23:49:09] nah, it's just NFS fuckery [23:49:15] kk. Reboot? [23:49:28] halfak: easiest, yeah [23:49:39] 06Labs: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#2321566 (10Tgr) Such a system would probably reduce usability of the tool for users who use incognito mode and break it for those who do not accept cookies at... [23:49:52] Do I still reboot through wikitech or horizon? [23:50:15] halfak: both work atm, horizon sucks lessish [23:50:22] Horizon it is!