[00:01:45] (03Merged) 10jenkins-bot: Keep alive connection to second MySQL database using Ping [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/277925 (https://phabricator.wikimedia.org/T117045) (owner: 10Jean-Frédéric) [00:05:53] random question, would it be possible to setup a round robin dns for a set of labs instances? I know we can't make LVS work due to differences in networking, but then i could at least get them behind a shared hostname [00:37:22] setting some cname's looks plausible...i just need to figure out the right way [00:42:52] 6Labs, 10MediaWiki-extensions-OpenStackManager: OS-EXT-SRV-ATTR:instance_name not set for some instances - https://phabricator.wikimedia.org/T123162#2128230 (10Krinkle) [00:43:39] * ebernhardson somehow gets the feeling inventing a tld is not the right answer ;) [00:44:55] 6Labs, 10MediaWiki-extensions-OpenStackManager: OS-EXT-SRV-ATTR:instance_name not set for some instances - https://phabricator.wikimedia.org/T123162#2128231 (10Krenair) Not clear to me whether OSM's expectation that the attribute exists is a reasonable one or not. @Andrew? [00:45:53] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2128235 (10bd808) >>! In T128158#2114100, @scfc wrote: > My $ 0.02: > > 1. Tool authors sign up for #Tool-Labs once, and they create one tool every x, with the median user maint... [01:40:29] ebernhardson: I believe wdq has something set up like that, so wdq.wmflabs.org actually goes to one of two vms [01:51:26] wonder what they are using, dig comes back with a single ip so its not dns round robin'd. I'll poke them. thanks! [01:52:14] ahh, they are using an nginx round robin [02:17:44] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2128313 (10scfc) >>! In T128158#2128235, @bd808 wrote: >> […] >> 1. Tool authors sign up for #Tool-Labs once, and they create one tool every x, with the median user maintaining y... [03:25:25] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2128397 (10bd808) Thanks for your replies @scfc. I think we are closer to agreement than either of us probably thought initially. >>! In T128158#2128313, @scfc wrote: > I don't... [06:37:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:51:55] PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:26:54] RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [08:34:52] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [09:57:05] 6Labs, 10Tool-Labs: Install "hub" on Tool Labs - https://phabricator.wikimedia.org/T130149#2128784 (10Aklapper) [09:57:47] 6Labs, 10Tool-Labs: Install "hub" on Tool Labs - https://phabricator.wikimedia.org/T130149#2128787 (10Aklapper) Hi @tom29739, thanks for taking the time to report this! Is there a Debian package available? What does "makes it much easier to work with Github" mean and who is "we"? [10:07:48] 6Labs, 10Tool-Labs, 7Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#2128808 (10valhallasw) [10:07:50] 6Labs, 10Tool-Labs: Install "hub" on Tool Labs - https://phabricator.wikimedia.org/T130149#2128806 (10valhallasw) 5Open>3declined There is no debian/ubuntu package available as far as I can see. It's also super easy to install in user space (either build it manually or download a statically linked binary),... [10:22:02] you may get inconsistent results on some s2 wikis while reimport is in progress. In that case, try again in a few minutes [10:56:18] 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2128927 (10jcrespo) 5Open>3Resolved I think the core of the cause was a corruption of the relay log, started due to a permission mismatch that stopped a forced stop of... [11:18:12] I am trying to connect to the replica databases through PHP in my tools account, where can I find the server details for connection? [11:20:08] TheDaveRoss: have a look at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database [11:22:23] I have read through that, and am able to connect via the mySQL workbench, but all of the server strings I have tried to run from my tool's directory have encountered network errors [11:22:43] I thought there might be something different which had to be done on a non-SSH connection [11:29:55] This is all I know. Maybe someone else can help... Sorry. [12:14:21] 6Labs, 6DC-Ops, 6Operations, 10ops-eqiad: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2129169 (10Aklapper) [12:15:56] 6Labs, 10Labs-Infrastructure, 10Labs-Sprint-102, 6DC-Ops, and 2 others: Locate and assign some MD1200 shelves for proper testing of labstore1002 - https://phabricator.wikimedia.org/T101741#2129249 (10Aklapper) [12:16:01] 6Labs, 10Labs-Infrastructure, 6DC-Ops, 10Incident-Labs-NFS-20151216, and 2 others: labstore1002 issues while trying to reboot - https://phabricator.wikimedia.org/T98183#2129251 (10Aklapper) [13:45:38] 6Labs, 10Beta-Cluster-Infrastructure, 7Tracking: Beta Cluster <-> WMF Labs policy compliance (tracking) - https://phabricator.wikimedia.org/T114615#2129523 (10greg) [14:33:17] 6Labs, 6Operations: revise/fix labstore replicate backup jobs - https://phabricator.wikimedia.org/T127567#2129752 (10chasemp) re: > Multi-week historical copies as space allows I'm open to what makes sense here but I didn't explain the purpose clearly above I think. This is not primarily intended as any kin... [14:48:50] (03CR) 10ArthurPSmith: [C: 031] "Thanks, no problem with adding my name!" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277479 (owner: 10Ricordisamoa) [15:09:58] (03CR) 10Ricordisamoa: [C: 032] Credit ArthurPSmith in toolinfo.json [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277479 (owner: 10Ricordisamoa) [15:20:17] TheDaveRoss: surprisingly there is no clear tutorial page on that on wikitech. A pretty simple example is at https://tools.wmflabs.org/replag/?source [15:20:31] (03Merged) 10jenkins-bot: Credit ArthurPSmith in toolinfo.json [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/277479 (owner: 10Ricordisamoa) [15:24:35] hello, I have created a new proxy on labs that despite my apache config seems to be trying to access "/". See: https://browser-reports-test.wmflabs.org/, on top of dashiki-staging-01.dashiki.eqiad.wmflabs [15:24:39] any ideas? [15:27:20] nuria: I have a meeting in a few minutes but if you ping me later in the day I can look. Offhand, though — how would the proxy know to hit a different url? [15:27:58] andrewbogott: let me look at our apache config again and will ping you if needed, thank you [15:29:49] ah, you mean / in your filesystem, not the root url :) [15:32:24] andrewbogott: rightt [16:16:28] 10Tool-Labs-tools-stewardbots, 6Stewards-and-global-tools: Unified and centralized CSS and JS for all tools in the project - https://phabricator.wikimedia.org/T130030#2130069 (10MarcoAurelio) I think @Glaisher took care or is taking care of this right now. [16:47:34] 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2105526 (10dungodung) Could the similar issue at T115517 be solved in a similar fashion? [16:51:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [17:13:58] 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2130281 (10jcrespo) Thank you for pointing it, I was not aware of that issue. I will do the same, solving the most important consistency errors, then solve things complete... [17:36:21] 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2130360 (10Superyetkin) It looks like the logging table is still missing some records. Could you please take a look at [[ https://tr.wikipedia.org/wiki/Fort | this ]]? [17:40:42] 6Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2130397 (10jcrespo) 5Resolved>3Open I will. I may have reinserted some records, but not deleted them, because they can be recreated later. I will see if it is easily s... [18:03:13] Warning: wikitech is going to down for a few minutes for a system reboot [18:07:35] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:08:19] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:10:20] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 323 bytes in 1.396 second response time [18:10:23] https://tools.wmflabs.org/wikidata-todo/creator_from_wikidata.php 502 [18:11:28] PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:13:10] PROBLEM - Puppet run on tools-cron-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:13:42] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:13:43] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:14:41] wikitech is unhappy but should be back shortly... [18:15:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:15:23] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 805070 bytes in 4.746 second response time [18:15:25] PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:16:23] PROBLEM - Puppet run on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:17:05] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:17:05] PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [18:17:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:17:27] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:17:27] PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:17:28] PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:17:49] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:17:57] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [18:18:16] ok, wikitech back, sorry for the delay [18:18:47] PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:18:59] PROBLEM - Puppet run on tools-k8s-bastion-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:19:53] andrewbogott: any reason tools.wmflabs.org is avaible under http ? [18:20:37] PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:20:42] PROBLEM - Puppet run on tools-exec-cyberbot is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:20:54] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:21:36] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:22:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:27:54] matanya: as far as I know everything in labs that's behind a proxy supports http and https both [18:28:01] unless you noticed a change [18:28:13] One could argue that we should switch off http but it hasn't really been discussed [18:28:19] my question is why it is not https only [18:31:13] 6Labs, 10Tool-Labs: Should we change labs and tools proxies to https-only? - https://phabricator.wikimedia.org/T130236#2130573 (10Andrew) [18:31:17] matanya: ^ [18:31:52] 6Labs, 10Tool-Labs: Should we change labs and tools proxies to https-only? - https://phabricator.wikimedia.org/T130236#2130573 (10Matanya) Yes, definitely. [18:32:17] andrewbogott: that is for the proxy, what about the home page: tools.wmflabs.org ? [18:32:45] it's behind the proxy [18:32:57] as far as I know that page is just another tool [18:42:43] 10Tool-Labs-tools-wikiloves: Criar pagina de configuração da ferramenta no commons - https://phabricator.wikimedia.org/T130240#2130647 (10Danilo) [18:43:01] !log rcm killed rcm-wiki instance, unused [18:43:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master [18:43:21] andrewbogott: Medium instance, clened [18:43:24] *cleaned [18:43:35] * Luke081515 plans to kill a large instance in the near future [18:43:38] Luke081515: thanks! [18:44:52] 10Tool-Labs-tools-wikiloves: Desenvolver aplicativo básico em Flask para a ferramenta - https://phabricator.wikimedia.org/T129712#2130677 (10Danilo) 5Open>3Resolved p:5Triage>3Normal a:3Danilo [18:47:28] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [18:48:44] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:48:45] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [18:52:01] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [18:52:57] 6Labs, 10Tool-Labs: Should we change labs and tools proxies to https-only? - https://phabricator.wikimedia.org/T130236#2130573 (10tom29739) Some of the tools don't support http, or don't need https, so the developers haven't added support for it in their tools. Other than that setback, I don't think there is a... [18:56:58] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:26] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:27] RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:27] RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:56] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [18:58:58] RECOVERY - Puppet run on tools-k8s-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:00:43] RECOVERY - Puppet run on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [19:00:43] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:29] RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:44] 10Tool-Labs-tools-wikiloves: Transferir ferramenta que lista imagens do projeto images para o wikiloves - https://phabricator.wikimedia.org/T130245#2130757 (10Danilo) [19:03:47] RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [19:06:32] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [19:19:42] If I add a directory to my path, or a tools path, will it get removed when puppet runs, etc? [19:22:33] tom29739: in general puppet only maanges things it knows have been declared and afaik that is not one of those things, I'm also not sure how you intend to do it, .profile? [19:24:39] 10Tool-Labs-tools-wikiloves: Criar ferramenta de mapa - https://phabricator.wikimedia.org/T130248#2130836 (10Danilo) [19:25:58] chasemp, I would have thought so. I'm trying to install a program in my userspace. [20:07:04] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2131086 (10scfc) I just signed up for Heroku, but haven't tested it any further. Now I find the idea that you //must// put your application in an SCM and automate the deployment... [20:59:54] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [21:37:19] 6Labs, 6WMF-Legal: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2132217 (10ZhouZ) Thanks to everyone for bringing this issue to our attention. There has also been some discussions on labs-l about changing and clarifying the Labs Ter... [21:41:57] 6Labs, 6WMF-Legal: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2132231 (10bd808) I'd be all for just disallowing, but yeah my word of mouth understanding is that 3rd party server interactions should require consent. It would be nice... [21:46:05] 6Labs, 6WMF-Legal: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2132250 (10ZhouZ) Thanks Bryan. Will do. We'll keep this task updated as we look into updating the TOU. [22:25:52] 10PAWS, 6Research-and-Data-Backlog: Create a mailing list for PAWS - https://phabricator.wikimedia.org/T129297#2132611 (10ggellerman) [22:26:27] 6Labs, 10Tool-Labs, 10labs-sprint-117, 6Design-Research-Backlog, and 5 others: Organize a (annual?) toollabs survey - https://phabricator.wikimedia.org/T95155#2132625 (10ggellerman) [23:16:44] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2132893 (10scfc) I don't have data for "denied" requests (that could be estimated by all requests in https://wikitech.wikimedia.org/wiki/Category:Tools_Access_Requests that have... [23:22:17] RECOVERY - Puppet run on tools-cron-02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:48:00] RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [23:58:33] i have changed a role class that is included on deployment server role [23:59:02] i'm fixing it in prod and if i broke something beta i apologize and will look there next [23:59:16] it's just about names of classes, not the content