[00:00:59] Is the instance that manages web proxies down and needs to be rebooted? [00:01:57] I'm getting a 502 Bad Gateway error on all our labs-hosted dashboards :\ cc yuvipanda [00:03:14] bearloga: give a moment, something happend w/ dns momentarily [00:03:33] chasemp: okie dokie, thanks! [00:05:24] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 797666 bytes in 7.717 second response time [00:05:56] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:12:45] 6Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2082625 (10chasemp) seems to have just happened again: similar symptoms and recovery with restarts but it did not coincide with the assumed time of day. [00:19:38] !log labs making wikitech dumps available for T54170 [00:19:39] T54170: Provide dumps of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T54170 [00:19:39] labs is not a valid project. [00:19:54] !log testlabs making wikitech dumps available for T54170 [00:19:55] T54170: Provide dumps of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T54170 [00:19:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL, Master [00:21:27] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:35] 6Labs, 10Dumps-Generation, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Provide dumps of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T54170#2082702 (10Dzahn) @MZMcBride -> https://wikitech.wikimedia.org/dumps/ [00:34:56] (03Abandoned) 10MarcoAurelio: Updating HTML for main ~stewardbots page. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273772 (owner: 10MarcoAurelio) [00:35:00] 6Labs, 10Dumps-Generation, 6Operations, 10wikitech.wikimedia.org: Provide dumps of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T54170#2082704 (10Dzahn) [00:37:28] (03PS1) 10MarcoAurelio: Updating HTML main page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274610 [00:37:57] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, 10wikitech.wikimedia.org: since dumps are public on wikitech, do we still want them on dumps.wm.org ? - https://phabricator.wikimedia.org/T128680#2082710 (10Dzahn) [00:38:12] anybody knows what this means: Error: /Stage[main]/Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/client/ssl/certs/ca.pem [00:38:14] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, 10wikitech.wikimedia.org: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2082710 (10Dzahn) [00:38:20] (03CR) 10MarcoAurelio: [C: 032 V: 032] Updating HTML main page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274610 (owner: 10MarcoAurelio) [00:38:23] started happening on my labs puppetmaster today [00:38:33] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, 10wikitech.wikimedia.org: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2082710 (10Dzahn) a:5Dzahn>3None [00:38:59] 6Labs, 10Dumps-Generation, 6Operations, 10wikitech.wikimedia.org: Provide dumps of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T54170#562758 (10Dzahn) 5Open>3Resolved [00:39:10] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [00:39:24] RECOVERY - Puppet failure on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [00:39:59] yuvipanda: which puppet master is controlling labstestweb servers [00:42:55] yuvipanda: any idea about this --> Error: /Stage[main]/Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/client/ssl/certs/ca.pem [00:42:55] ? [00:43:23] mutante: no idea, andrewbogott would know [00:43:30] SMalyshev: can you file a bug? at a conference [00:43:40] yuvipanda: sure [00:44:35] ok, andrew asked me for a fix :) [00:45:42] actually, ignore me. got it [00:45:51] i never asked :p [00:50:29] 6Labs, 6Project-Admins, 15User-Luke081515: Create a project for horizon - https://phabricator.wikimedia.org/T127439#2043777 (10Negative24) Possibly a duplicate of T104910? [00:51:58] (03PS1) 10MarcoAurelio: Modify readme.md file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274615 [00:52:41] (03CR) 10MarcoAurelio: [C: 032 V: 032] Modify readme.md file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274615 (owner: 10MarcoAurelio) [00:54:44] SMalyshev: if that’s on a box using self-hosted puppet, probably just restart the puppetmaster [00:54:51] (but, I’m at dinner :) ) [00:55:58] 6Labs, 10Labs-Infrastructure, 6Operations: labservices1001 ran out of disk space - https://phabricator.wikimedia.org/T126572#2082775 (10Dzahn) 5Resolved>3Open 16:58 < icinga-wm> PROBLEM - Disk space on labservices1001 is CRITICAL: DISK CRITICAL - free space: / 350 MB (3% inode=76%) [00:56:40] 6Labs, 10Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2082778 (10Negative24) p:5Triage>3Normal [00:58:35] 6Labs, 10Labs-Infrastructure, 6Operations: labservices1001 ran out of disk space - https://phabricator.wikimedia.org/T126572#2082779 (10Dzahn) 5Open>3Resolved 17:02 < icinga-wm> RECOVERY - Disk space on labservices1001 is OK: DISK OK ? [01:02:13] andrewbogott: tried that, doesn't help :( [01:04:01] I think it is confused between if it's client or server, but I can't see why [01:06:32] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: must be a simple hostname. The project-specific domain will be automatically appended. at /etc/puppet/manifests/role/puppet.pp:20 on node wdqs-puppetmaster.wikidata-query.eqiad.wmflabs [01:06:38] that's a new one... [01:08:47] ok, found the problem - in Hiera, I'm not supposed to add domain now... [01:13:17] Error: Could not request certificate: Server hostname 'wdqs-puppetmaster.eqiad.wmflabs' did not match server certificate; expected wdqs-puppetmaster.wikidata-query.eqiad.wmflabs [01:13:20] ugh [01:54:35] 6Labs, 6Project-Admins, 15User-Luke081515: Create a project for horizon - https://phabricator.wikimedia.org/T127439#2082899 (10Krenair) a:5Luke081515>3Andrew @Andrew, thoughts? [04:59:51] 10Tool-Labs-tools-Other: Zoomviewer disfunctionally laggy for very large images - https://phabricator.wikimedia.org/T128580#2083249 (10dschwen) Fae, I'm working on something. The logic that refreshes the image cache is borked. I'm reimplementing this completely. Sorry it is taking so long. [05:07:04] 6Labs, 10Labs-Infrastructure, 6Operations: Estimate hardware requirements for relevance lab elasticsearch servers - https://phabricator.wikimedia.org/T128433#2083262 (10EBernhardson) I've been running some more relevance lab tests against nobelium, this time with a more reperesentative query set. Specificall... [05:21:06] 6Labs, 6Operations: revise/fix labstore replicate backup jobs - https://phabricator.wikimedia.org/T127567#2083269 (10Dzahn) just my 2 cents from the merged task. that output line "Last run result for unit replicate-tools was exit-code " really looked as if there was just a typo where it should be 'was $exit-co... [06:42:26] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:24] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 798092 bytes in 6.425 second response time [07:17:12] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, 10wikitech.wikimedia.org: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2083348 (10ArielGlenn) p:5Lowest>3Low a:3ArielGlenn [08:34:51] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [10:44:29] 6Labs, 10Tool-Labs, 6Operations: Add other Tools administrators to the Icinga notification group - https://phabricator.wikimedia.org/T128715#2083683 (10scfc) [10:45:07] 6Labs, 10Tool-Labs, 6Operations: Make icinga-wm report Tools homepage check at #wikimedia-labs, too - https://phabricator.wikimedia.org/T128716#2083696 (10scfc) [12:14:25] 10Tool-Labs-tools-stewardbots, 10Continuous-Integration-Config: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2083992 (10MarcoAurelio) Since there's also HTML pages, tests on those would also be helpful if possible. [12:14:43] (03CR) 10Siebrand: [C: 04-1] "There is no support for processing of the PLURAL magic word in the Python application Crosswatch, as far as I know. That would have to be " [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265647 (https://phabricator.wikimedia.org/T114876) (owner: 10MtDu) [12:53:50] 6Labs, 10Tool-Labs, 10Monitoring, 6Operations: Make icinga-wm report Tools homepage check at #wikimedia-labs, too - https://phabricator.wikimedia.org/T128716#2084074 (10Peachey88) [12:54:13] 6Labs, 10Tool-Labs, 10Monitoring, 6Operations: Add other Tools administrators to the Icinga notification group - https://phabricator.wikimedia.org/T128715#2084075 (10Peachey88) [14:12:58] 6Labs, 10Tool-Labs: Cluebot writes massive logs that are making labstore run out of space and surge in load making toollabs unavailable - https://phabricator.wikimedia.org/T127222#2084229 (10Aklapper) No news here for two weeks. Can this task get an update and/or an assignee, please? [14:58:50] !log tools.stewardbots Started overhaul of file system from toolserver [15:09:59] (03PS1) 10MarcoAurelio: HTML update and fixed links [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274722 [15:12:03] hey yall [15:12:09] (03CR) 10MarcoAurelio: [C: 032 V: 032] HTML update and fixed links [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274722 (owner: 10MarcoAurelio) [15:12:26] we've tried to set up a self-hosted puppetmaster a few times and we keep failing [15:12:28] here's the error: [15:12:33] https://www.irccloud.com/pastebin/Qg7EF7Tz/ [15:13:11] we're following the directions on wikitech, and making sure to let a full puppet agent -tv complete successfully before going forward [15:13:15] the instance is 14.04 [15:14:50] (03PS1) 10MarcoAurelio: Revert "HTML update and fixed links" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274725 [15:15:28] !log tools.stewardbots maurelio@tools-bastion05 Sync. repo [15:15:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [15:15:54] 6Labs: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2084359 (10Andrew) [15:15:57] (03CR) 10MarcoAurelio: [C: 032 V: 032] Revert "HTML update and fixed links" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274725 (owner: 10MarcoAurelio) [15:16:07] just tried with a debian jessie instance, that didn't work either, error there: [15:16:10] https://www.irccloud.com/pastebin/aHWrWRM6/ [15:18:07] andrewbogott: any idea what could be causing ^ [15:18:26] I promise I'm not setting up yet another self-hosted puppet master to make your life miserable, I'm just trying to test a puppet change [15:18:38] 6Labs: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2084376 (10Andrew) Proposed steps: [] Get mysql running on holmium and labservices1001 [] Create replicated pdns dbs on holmium and labservices, each replicating from the pdns db on m5-master [] (quickly) Turn of holmium... [15:19:44] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2084388 (10Andrew) [15:19:53] milimetric: will look shortly [15:21:04] milimetric: so it sounds like the puppetmaster service isn’t coming up [15:21:11] did you try a ‘service puppetmaster restart’ to see what’s going wrong? [15:21:58] yes, but once that puppet run fails, puppet won't run cleanly any more, even if I remove self-hosted from config [15:22:38] I think service puppetmaster restart just said that wasn't a service, but I will try it again [15:22:41] sure, puppet configs don’t undo themselves when you remove them… whatever it did when you applied it stays did [15:24:19] I mean, in this case I did: 1. create instance 2. ssh into it and run puppet 3. check "self-hosted" role 4. run puppet - fail 5. uncheck "self-hosted" role 6. run puppet - still fail [15:24:34] hm [15:24:35] in step 6 it says it has trouble finding the catalog [15:24:35] that’s new [15:24:41] I’ll see if I can reproduce it. [15:24:46] I guess that makes sense 'cause it's probably thinking it's self-hosted [15:24:56] Sure, I’m just saying that after step 4 failed, there’s no going back. Step 5 doesn’t do anything. [15:25:05] ok, that makes sense [15:26:02] so your end goal is to have this on jessie or trusty? [15:26:09] doesn't matter [15:26:13] whichever's easier [15:26:28] btw, I tried to do it again and now on step 2. above it goes: [15:26:32] Error: Could not request certificate: getaddrinfo: Name or service not known [15:26:32] Exiting; failed to retrieve certificate and waitforcert is disabled [15:26:48] * andrewbogott tries it [15:33:36] milimetric: here’s what I did: [15:34:17] 1) Build new instance 2) confirm clean puppet run 3) tick box role::puppet::self box on wikitech 4) puppet run 5) service puppetmaster restart 6) puppet run again [15:34:22] not obvious, but works [15:34:35] I’ll open a bug to sort out why 5 and 6 are needed [15:34:41] huh, ok, thought i tried that [15:34:49] thanks! [15:36:08] 6Labs: role::puppet::self requires a puppetmaster restart during apply - https://phabricator.wikimedia.org/T128740#2084429 (10Andrew) [15:37:12] (03PS1) 10MarcoAurelio: [WIP] Folders for each tool [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274734 [15:37:51] !log tools.stewardbots reverted last patches [15:37:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [15:38:19] 6Labs: role::puppet::self requires a puppetmaster restart during apply - https://phabricator.wikimedia.org/T128740#2084446 (10Milimetric) Just in case this helps with debugging, we tried this on Friday Feb. 26 and it was working. But when we tried it on Monday Feb. 29, it was not working. [15:44:35] (03PS2) 10MarcoAurelio: Folders for each tool and link updates [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274734 [15:46:10] (03CR) 10MarcoAurelio: [C: 032 V: 032] Folders for each tool and link updates [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274734 (owner: 10MarcoAurelio) [15:47:58] andrewbogott: I tried it a couple of times and after your 5) service puppetmaster restart, I still can't get a clean "puppet agent -tv", it errors with the same failure of * Starting puppet master ... fail! [15:48:13] milimetric: jessie? [15:48:22] is that in a project with hiera stuff set? [15:48:25] no, 14.04, should I have tried jessie? [15:48:38] there's some hiera stuff in analytics, I think, yea [15:48:39] jessie is what I tried, although I’d expect it to be the same [15:48:54] ok, should I try in a simpler project and with jessie? [15:49:39] if that will still get you what you need, yes :) [15:51:25] hashar: I tidied up the ‘projects’ tab in horizon and enabled a bunch of instance functions. I need to pare that list back down but am interested in your thoughts in the meantime [15:52:14] !log tools.stewardbots MarcoAurelio: merged and submitted {{gerrit|274734}} [15:52:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [15:55:56] (03PS1) 10MarcoAurelio: Modify elections.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274737 [15:56:43] (03CR) 10MarcoAurelio: [C: 032 V: 032] Modify elections.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/274737 (owner: 10MarcoAurelio) [15:57:46] !log tools.stewardbots Sync. {{gerrit|274737}} [15:57:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [16:01:54] andrewbogott: so, all jessie instances I've tried with just give me this: [16:01:58] https://www.irccloud.com/pastebin/MNToHgHU/ [16:02:19] (that's on the very first puppet run before checking the self-hosted) [16:02:34] milimetric: re-using an instance name maybe? [16:02:41] no, I was careful not to do that [16:02:52] I started with puppet-test and I'm at dan-puppet-test-1 [16:04:17] what project are you in? I’ll try again [16:06:14] milimetric: ^ [16:06:40] andrewbogott: I tried in "analytics" and "reportcard" [16:07:03] "analytics" is the more complicated one with hiera [16:07:14] "reportcard" is simpler, I only tried jessie there as you were saying that might wor [16:07:16] *work [16:07:21] Can someone restart nagf? [16:11:25] 10Tool-Labs-tools-stewardbots: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2084516 (10MarcoAurelio) [16:11:51] 10Tool-Labs-tools-stewardbots: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2084529 (10MarcoAurelio) p:5Triage>3Low [16:18:06] milimetric: you can use reportcard-puppet-testing for now. And if you continue to have trouble creating new instances in the future please file a bug :) [16:18:11] 10Tool-Labs-tools-stewardbots: hat-web-tools import for stewardbots - https://phabricator.wikimedia.org/T128743#2084536 (10MarcoAurelio) [16:18:13] 10Tool-Labs-tools-stewardbots: hat-web-tools import for stewardbots - https://phabricator.wikimedia.org/T128743#2084548 (10MarcoAurelio) p:5Triage>3Normal [16:18:38] andrewbogott: k, what's the best tag for bugs like this? [16:19:00] I don’t know, ‘labs’ I guess, and cc: me [16:19:37] Why is wikitech using the default mediawiki-logo at the moment? [16:21:16] hashar: Can you restart nagf? you are shown as maintainer [16:21:30] Luke081515: no clue how it works sorry :(( and in a meeting [16:21:36] ok [16:21:40] 10Tool-Labs-tools-stewardbots: General update of HTML and CSS for stewardbot tools and portals - https://phabricator.wikimedia.org/T128745#2084562 (10MarcoAurelio) [16:21:47] Luke081515: it’s not the mediawiki logo, it’s the wikitech logo :) [16:21:48] 10Tool-Labs-tools-stewardbots: General update of HTML and CSS for stewardbot tools and portals - https://phabricator.wikimedia.org/T128745#2084574 (10MarcoAurelio) p:5Triage>3Normal [16:22:23] Luke081515: previously it was the ‘labs’ logo which is increasingly inappropriate. So… https://gerrit.wikimedia.org/r/#/c/274457/ [16:22:26] andrewbogott: do we have more head room for some beefy video servers ? [16:22:46] the current servers are under high demand [16:22:47] matanya: I don’t think so. I need to do a capacity audit (like, today) but things have been very crowded [16:22:52] andrewbogott: Do you think this is the labs logo? https://phabricator.wikimedia.org/F3514928 :P [16:22:53] Luke081515: I restarted it try now [16:23:07] chasemp: WFM, thank you [16:23:38] andrewbogott: any advise going forward ? [16:24:02] matanya: what is lacking, disk, cpu, mem etc? [16:24:03] matanya: I really don’t have enough information to have an opinion right now [16:25:34] chasemp: all three, i will grab some monitoring info to have a better picture [16:27:52] 10Tool-Labs-tools-stewardbots, 10Continuous-Integration-Config: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2084581 (10MarcoAurelio) Oh, and also PHP and CSS stuff. [16:31:43] andrewbogott: Did you take a look at my screenshot? I think this is not the labs/wikitech logo, but is this onlay at my machine visible? [16:32:24] Luke081515: did you look at https://gerrit.wikimedia.org/r/#/c/274457/ ? [16:32:33] https://paws.wmflabs.org/ is down! [16:32:48] oh, ok [16:33:37] matanya: something like this may be a candidate for reshuffling of resources or something I'm not sure but if you could outline on a task and ping us directly in the style of "here is what I'm trying to do here and can't and possibly why" we can try to help [16:33:40] andrewbogott: But maybe you can take a look at T127439 later [16:33:40] T127439: Create a project for horizon - https://phabricator.wikimedia.org/T127439 [16:34:30] chasemp: will do. [16:36:00] 6Labs, 10wikitech.wikimedia.org: 404 poweredby_mediawiki_88x31.png when viewing wikitech - https://phabricator.wikimedia.org/T128747#2084612 (10JanZerebecki) [16:43:56] This: http://prntscr.com/aaqfdm is showing when I try to look at phab links [16:44:43] tom29739: report to #wikimedia-devtools if you can, it has been known to happen while some work goes on [17:06:28] chasemp: Nagf is down again [17:06:59] 6Labs, 10wikitech.wikimedia.org: 404 poweredby_mediawiki_88x31.png when viewing wikitech - https://phabricator.wikimedia.org/T128747#2084612 (10Krenair) I think this was caused by some change in {T99096}, see my patch over there [17:07:10] bd808: Do you know if there is a possibilty to create a wiki for a CA cluster with an own name? I don't know what I have to put into the new puppet role [17:07:45] Luke081515: can you rephrase? I'm not understanding the question [17:07:59] 6Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: 404 poweredby_mediawiki_88x31.png when viewing wikitech - https://phabricator.wikimedia.org/T128747#2084727 (10Krenair) a:3Krenair [17:08:24] bd808: I want to add a wiki to my CentralAuth Cluster, so normaly I would add a role like 'testwiki', but I want to define a URL for that new wikis, is that possible? [17:09:03] Luke081515: I'm not sure what the deal is, I can restart but who runs nagf idk [17:09:15] thanks. Maybe bd808 nows more? [17:09:20] about nagf? [17:09:26] It went down again [17:09:31] Krinkle's toy [17:09:44] but it's just a web server I thought? [17:09:48] it is [17:09:54] but it seems to not stay up for long [17:10:03] hmm... OOM problems? [17:10:25] It's a plain php tool [17:10:31] static [17:10:40] However, it runs in Yuvi's Kubernetes experiment [17:10:45] so that's likely the issue [17:11:06] ah in that case we should invole yuvi as I still don't have much insight there [17:11:11] involve [17:11:19] it's dark magic [17:11:57] chasemp: the only k8s stuff I've seen documented is -- https://wikitech.wikimedia.org/wiki/Grrrit-wm [17:13:25] Luke081515: for your other question... it may be possible to get a url that's not like $wikidb-clustername.wmflabs.org. Let me look at the puppet code [17:14:53] bd808: I don't habe a problem with $wikidb-clustername.wmflabs.org, nut I want to specify the wikidb name [17:15:37] It seems the nagf k8s pod is not running [17:15:38] Node: tools-worker-1003.tools.eqiad.wmflabs/ [17:15:38] Start Time: Wed, 24 Feb 2016 15:15:30 +0000 [17:15:39] Labels: deployment=a687c60ee61e111f79b6c74816b2c4ea,name=nagf [17:15:39] Status: Pending [17:15:39] Reason: GeneralError [17:15:40] Message: Query container info failed with error (API error (500): Unknown device 966fb112d1e7c82f3c012c94d2aae03332be62bb5dfa026074e67836b27f0b7c [17:16:08] I've deleted it. Let's see if the controller restarts it [17:16:10] Luke081515: look at puppet/modules/mediawiki/manifests/wiki.pp for all the options you can pass to mediawiki::wiki [17:16:16] db_name is one [17:16:49] but... if it doesn't end in "wiki" I'm not sure if it will work or not with some of the other magic in the Puppet config [17:16:53] I've never tried [17:17:38] bd808: Do you know, in which path I have to put my custom puppet role, to see it with 'vagrant roles list'? [17:17:52] Nagf is up again [17:18:21] Luke081515: that only looks in puppet/modules/roles/manifests [17:18:31] ok, thanks [17:20:20] Luke081515: but you don't need `roles list` to show a class to add it to your cluster. You can manually add any class via an array named "classes" in puppet/hieradata/local.yaml [17:20:53] e.g. "classes: [ '::local::foo' ]" [17:21:09] 6Labs, 6Project-Admins, 15User-Luke081515: Create a project for horizon - https://phabricator.wikimedia.org/T127439#2084768 (10Andrew) Inasmuch as openstackmanager has a project tag, we should have one for Horizon as well. BUT, I think that horizon.wikimedia.org is a dumb name for a website, so can this wai... [17:21:41] puppet/modules/local exists but is excluded in .gitignore to make local dev easier [17:22:25] ok, I will add it do my notes [17:23:40] I'm trying to wrote a docu for people how to setup a CA cluster, if I'm sure that my way is the rights way, so I'm testing a bit at the moment ;) [17:26:12] * yuvi|panda waves at chasemp / Krinkle [17:26:31] yuvi|panda: I deleted the pod and it came back properly [17:26:35] cool [17:26:40] yuvi|panda: but I don't want to do that nex time :P [17:26:46] It knew it was broken [17:26:49] not sure what kept it from fixing it [17:26:50] yeah [17:27:19] Krinkle: what did you run to get that info? [17:27:22] It seems it detected an issue, deleted the pod, created a new one but then that one got stuck in 'pending; [17:27:27] yuvipanda: describe pod [17:27:33] $ kubectl describe pods nagf-u3f63 [17:27:35] Krinkle: ok [17:27:37] (which is deleted now) [17:27:39] yeah [17:27:44] not suer what that error means [17:27:59] and then it never re-tries to get the status or somehow recover [17:29:43] Krinkle: so the rc as it is now doesn't actually have a healthcheck defined [17:29:47] so we should define a health check [17:29:49] for it [17:29:56] yuvipanda: OK. [17:30:00] I suppose it should just http request for /nagf/ [17:30:04] and that should be 200 [17:30:07] yuvipanda: I doubt that would[ve helped though, since this was a controller-level issue or something [17:30:11] Yeah [17:30:43] yuvipanda: Is there something on there from yaml we can put back onto github? [17:30:46] (maybe in the docker branch) [17:33:31] Krinkle: yeah [17:33:34] Krinkle: let me paste it too [17:35:31] am writing out the health check now [17:37:03] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2084918 (10kaldari) [17:37:53] yuvipanda: thx! [17:38:02] yuvipanda: should I file a bug report about the thing not working? [17:39:14] yeah! [17:40:06] * yuvipanda is in a meeting [17:43:13] Krinkle: I'm going to restart nagf [17:44:51] Krinkle: restarted, and seems ok [17:45:07] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2085023 (10bd808) It looks almost like [[Help:Tool Labs]] and it's subpages are being excluded from indexing. This search: :( [17:50:07] bd808: I guess I need help again: I setup a secomnd cluster for testing. Loginwiki is in theory available, and visible aat Special:SiteMatrix, and I create a web proxy to the instance with login-.wmflabs.org and port 8080, but the server is still not reachable [17:51:07] Luke081515: did you open up the project firewall for the new instance? [17:51:21] (add it to the right security group) [17:51:34] It is added to default as the other cluster, and the other cluster works [17:52:28] ok. how about the puppet/hieradata/local.yaml changes that are needed to set mediawiki::multiwiki::base_domain ? [17:53:24] Luke081515: if you look at settings.d/wikis/loginwiki/wgConf.php you can see what the wiki thinks its cannonical url is [17:53:34] I can take a look, but the last time, I enabled role centralauth, I don't have to do this [17:53:37] ok, I will take a look [17:54:14] if you don't set mediawiki::multiwiki::base_domain to something custom it defaults to the Labs VM's name [17:54:32] and we certainly spent a bunch of time on that a week or two ago :) [17:54:42] the wgCanonicalServer URL is the same for whoch I added the web proxy, but not reachable :-/ [17:57:11] What happens on the host computer when you query -- curl -LH 'Host: login-authmanager.wmflabs.org' 127.0.0.1:8080 -- replace "authmanager" with your expected name [17:58:22] the command shows mainpage code in the global sitelanguage [17:58:31] so settings etc and the page works [17:59:19] and it's not only loginwiki, the centralauthtestwiki, which is set up by the centralauth role is affected by the same problem too [18:00:22] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2084918 (10EBernhardson) We can use [[ https://wikitech.wikimedia.org/wiki/Help:Tool_Labs?action=cirrusdump | action=cirrusdump ]] to see if a particular page is being i... [18:01:11] ebernhardson: can I convince you to not use the word 'labs' in relevancy labs? [18:01:18] ebernhardson: there's... http://wikitech.wikimedia.org/wiki/Labs_labs_labs [18:01:24] it'll have to become Labs_labs_labs_labs [18:02:20] yuvipanda: there is also https://www.mediawiki.org/wiki/Wikimedia_Research/Design_Research/Prototyping_Labs [18:02:35] I tried to get them to change the name of that before it started, but lost the argument [18:04:03] bd808: I have never heard of it [18:04:07] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2085104 (10EBernhardson) An alternate look at what is indexed can be seen by running this from a machine in the prod cluster: curl 'search.svc.eqiad.wmnet:9200/labswi... [18:06:59] bd808: abbey is standing next to me so I'll try convince her again [18:07:07] 6Labs, 10Labs-Infrastructure, 6Operations, 13Patch-For-Review: labservices1001 ran out of disk space - https://phabricator.wikimedia.org/T126572#2085112 (10Andrew) 5Resolved>3Open The logs were still gigantic due to a setting I overlooked. Attached patch should help a lot. [18:07:09] yuvipanda: heh. good luck [18:07:40] yuvipanda: :P [18:07:58] ebernhardson: not joking at all... [18:08:03] yuvipanda: naming things is hard ... magicRelvanceFigureOuter [18:08:16] ebernhardson: yes, but avoiding mistakes for the 4th time shouldn't be hard :) [18:08:38] wikipedia, mediawiki, wikimedia, ... [18:08:56] yuvipanda: i'll poke trey to come up with some alternate names. How hard is renaming gerrit repo's? [18:09:30] ebernhardson: easy enough [18:09:37] ebernhardson: create new one, and import the old one :D [18:09:41] no URL redirect tho [18:09:45] lol [18:09:49] i suppose that works though [18:09:53] yeah [18:10:35] ebernhardson: and thanks :D [18:13:49] I'm trying to resetup the second cluster [18:15:45] Krinkle: https://github.com/yuvipanda/nagf/commit/6f1b62a855a2c979aa1f2b195d65e7f155889e54 added k8s file [18:28:22] 6Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: 404 poweredby_mediawiki_88x31.png when viewing wikitech - https://phabricator.wikimedia.org/T128747#2085248 (10Krenair) After that patch we now get 400 Invalid request [18:32:47] 6Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: 404 poweredby_mediawiki_88x31.png when viewing wikitech - https://phabricator.wikimedia.org/T128747#2085314 (10Krenair) This involves w/static.php in operations/mediawiki-config [18:42:59] i'm still getting "connection refused" error when trying to connect to commonswiki_p db from my php script (other dbases are connectable though) [18:43:48] but i can connect from cli [18:44:01] any idea, what's wrong? [18:44:16] Danny_B: are you sure you're connecting to the correct host this time? [18:45:11] valhallasw`cloud: i assume that connecting from cli uses the same dns resolution as connecting from php, doesn't it? [18:45:29] I think so, but I don't know if php does any weird caching or so [18:45:48] mysql --defaults-file=~/replica.my.cnf -h commonswiki.labsdb -D commonswiki_p works perfectly [18:46:04] right, and how are you connecting in php? [18:46:15] $linkCommons = mysql_connect( 'commonswiki.labsdb', $dbUser, $dbPassword ); [18:46:31] returns error 111 - connection refused [18:46:35] although [18:46:58] eg. $linkWikiquote = mysql_connect( 'cswikiquote.labsdb', $dbUser, $dbPassword ); in the same script connects perfectly [18:47:33] that does suggest that somehow it resolves commonswiki.labsdb differently [18:47:37] on which host are you running these? [18:47:52] exec host with stale /etc/hosts? [18:48:14] valhallasw`cloud: how can i find it? [18:48:17] (on which host are you running mysql, on which host is php running? tools/otherwise?) [18:48:50] my instance on labs (but i don't remember which machine it is) [18:49:07] ok, and you're running php under apache? [18:49:13] have you tried restarting apache? [18:49:30] yes / no, will try [18:49:41] bd808: Something changed at the vagrant roles? puppet agent --test --verboe throws: [18:49:45] Error: Could not request certificate: getaddrinfo: Name or service not known [18:49:48] Exiting; failed to retrieve certificate and waitforcert is disabled [18:50:32] Luke081515: are you trying to run puppet manually in a vagrant VM? [18:51:01] no, I setup a new instance and want to isntall vagrant now [18:51:13] point 4 at https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs [18:51:16] valhallasw`cloud: didn't help [18:51:46] Luke081515: ah. sounds like the initial provision either isn't done or failed. [18:52:01] that's not a mw-vagrant or vagrant in labs problem [18:53:03] Danny_B: I'm not sure then. what does gethostbyname('commonswiki.labsdb'); return in php? [18:53:06] ok. I will delete the instance, and recreate it in an hour or so [18:57:44] valhallasw`cloud: 10.64.4.11 [19:02:41] Danny_B: that's the right IP. [19:03:11] hm, let me see. cswikiquote is on c3 [19:03:34] Danny_B: does cswiki.labsdb work from php? [19:03:42] that's also on c1 / .11 [19:08:53] valhallasw`cloud: 10.64.37.4Can't connect to MySQL server on 'cswiki.labsdb' (111) [19:09:09] Danny_B: what tool is this? [19:11:19] chasemp: one of my tools. the source hasn't been changed since the last time used when it worked correctly [19:11:30] yes but what is the name of it? [19:11:56] chasemp: it's not on toollabs, if you're asking about that [19:12:04] ah ok [19:12:30] well mainly I wanted to know where it's running which I can find on tools [19:12:31] HIiiii [19:12:48] can anyone tell what's up with deployment-analytlics102 in deployment-prep project? [19:12:52] i created it a few hours ago [19:12:56] but, it doesn't have an instance ID [19:12:57] and [19:13:02] but i can get console output from it [19:13:19] puppet has run on it according to that [19:17:15] attemptin ga reboot... [19:17:41] I'm looking but yeah weird [19:17:55] may have to ask andrew it's possible it was allocated somewhere it couldn't spin up due to disk or ? [19:18:04] if it doesnt' come back open a task I guess [19:18:06] well, it started up though [19:18:09] puppet ran [19:18:13] its just that wikitech doesn't think it exists i guess [19:18:17] i can get console logs from it [19:18:25] i can't log into it either [19:18:40] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=c446da1b-d7a7-4d79-b8fd-3673ed481000&project=deployment-prep®ion=eqiad [19:19:08] ottomata: you broke the internet dude well done [19:19:15] you googled 'google' classic mistake [19:19:19] but really idk [19:19:32] hehe [19:19:49] pssh dunno, just going to delete it and create a new one [19:24:16] Danny_B: .4 is the server that's offline [19:26:34] complete newb here, forgive my intrusion... looking for some hand-holding to create a wikipedia bot [19:26:47] just an overview, really [19:26:54] any experienced bot makers out there? [19:27:20] telotism: https://en.wikipedia.org/wiki/Wikipedia:Creating_a_bot [19:27:43] then depending on which method you choose (awb, pywikibot, ...) the exact way of doing things differs [19:27:53] @valhallasw`cloud read that [19:28:05] have you built one? [19:29:11] valhallasw`cloud: so what can i do with it? [19:29:19] telotism: yes [19:29:22] Danny_B: I don't know. [19:29:51] Danny_B: somehow your VM is getting the wrong IP somewhere, but I'm not sure how you could figure out why [19:30:08] Danny_B: possibly by strace-ing apache [19:33:18] but commonswiki resolves to .11 and doesn't work too [19:34:34] it suggests that somehow mysql /doesn't/ get .11 as IP [19:37:09] valhallasw`cloud: better? [19:37:16] telotism: yep [19:37:19] sweet [19:39:46] telotism: so when it comes to importing data on a wiki, each wiki has their own policy. Generally, the policy part is harder than the actual bot work [19:40:41] telotism: for census data, I would actually suggest importing it to wikidata, which is the central database for that kind of data, and in some future moment, many wikipedia pages will get their data from there (or, at least, that's what we all hope) [19:40:50] telotism: https://www.wikidata.org/wiki/Wikidata:Main_Page [19:41:33] valhallasw`cloud: agreed. should be in wikidata, but we'd also like to put it into pages (e.g., places like states, counties, zip codes, etc.) [19:42:01] valhallasw`cloud: we might even do something fancy like a map or visualization [19:43:34]