[00:01:46] hmmm there certainly is something funny happening there [00:05:21] physikerwelt1: I can't recreate on another labs hosted mw-vagrant vm. it looks like its something weird on that particular host [00:05:52] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Create Labs project and Puppet magic to do redirects for things that have moved to Tool Labs - https://phabricator.wikimedia.org/T134508#2267756 (10Dzahn) http://hatjitsu.wmflabs.org to https://tools.wmflabs.org/hatjitsu/. <-- that works now.. resolved? [00:06:49] 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2268861 (10bd808) [00:06:51] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Create Labs project and Puppet magic to do redirects for things that have moved to Tool Labs - https://phabricator.wikimedia.org/T134508#2268859 (10bd808) 05Open>03Resolved Yes, resolved. Thanks for the Puppet merge assist @Dzahn [00:07:07] 06Labs, 06WMF-Legal, 07Epic, 07Privacy: [EPIC] Protect end-user privacy by restricting non-consentual third-party browser interactions - https://phabricator.wikimedia.org/T133919#2268865 (10bd808) [00:07:09] 06Labs, 06Team-Practices, 07Privacy, 15User-bd808: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2268866 (10bd808) 05Open>03Resolved a:03bd808 [00:07:23] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Create Labs project and Puppet magic to do redirects for things that have moved to Tool Labs - https://phabricator.wikimedia.org/T134508#2268867 (10bd808) [00:07:23] I change the hostname to the ip since this is the only way to access the instance from outside [00:07:26] 06Labs, 06Team-Practices, 07Privacy, 15User-bd808: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2268868 (10bd808) [00:07:30] 06Labs, 10Labs-Infrastructure, 15User-bd808: Static IP for yandex-proxy01.yandex-proxy.eqiad.wmflabs - https://phabricator.wikimedia.org/T132982#2268869 (10bd808) [00:07:33] 06Labs, 15User-bd808: Create project Yandex-proxy - https://phabricator.wikimedia.org/T132950#2268870 (10bd808) [00:07:49] 06Labs, 10Tool-Labs, 10labs-sprint-117, 06Community-Tech-Tool-Labs, and 6 others: Organize a (annual?) toollabs survey - https://phabricator.wikimedia.org/T95155#2268872 (10bd808) [00:08:11] legoktm: ack! why are column changes pinging on irc? [00:08:58] physikerwelt1: which instance? the vagrant vm or the labs host? [00:10:42] it's the mathosphere instance [00:11:00] and within this instance it's the vagrant vm [00:11:50] I'm still not following which your changed the hostname on and why actually [00:12:08] but I also don't think that should effect outbound curl requests [00:12:53] its acting like there is some PROXY env settings or something [00:13:25] I changed role::mediawiki::hostname to the vm to the IP to get around T134539 [00:13:26] T134539: Set up an domain name for the instance - https://phabricator.wikimedia.org/T134539 [00:16:07] physikerwelt1: rather than a domain you should setup a "web proxy" like "math-beta" and point it to the port on the Labs instance where your service is. For a normal mw-vagrant wiki in Labs that would be port 8080 [00:17:15] then you set role::mediawiki::hostname to the web proxy name (e.g. math-beta.wmflabs.org) [00:17:35] bd808: that's what we tried in the first place. But we could not get reasonable performance indicators since the web proxy caused problems [00:17:45] that's why we got the IP [00:17:54] caused problems? [00:18:03] 503 [00:18:18] oh you have a bare ip for the instance? [00:18:26] Hi [00:18:35] hi WolfStar [00:18:41] bd808: yes [00:18:57] How Do You Create A Page? [00:19:20] physikerwelt1: I don't know what magic it takes to setup a new domain for that. You probably need help from andrewbogott [00:20:43] Does Anyone Know? [00:20:50] WolfStar: assuming you are asking how to create a page on a MediaWiki wiki, you generally just visit the page that you want to create and then start editing [00:21:29] bd808: I still don't understand why the outbound connection cause problems and if that related to the hostname at all [00:21:50] I kind of doubt that it is related to the hostname [00:22:05] What Do You Mean? [00:22:06] it might be related to the public ip I guess [00:22:54] WolfStar: https://www.mediawiki.org/wiki/Help:Starting_a_new_page [00:23:02] Do You Mean The Name I Am Using? [00:23:22] WolfStar: oh, no. I was talking to physikerwelt1 about something else [00:23:25] bd808: do you know if vagrant configures some type of proxy [00:24:03] physikerwelt1: it sets up port forwards based on the config but no proxy that I can think of [00:24:40] Oh, Okay [00:24:48] physikerwelt1: it will say bind port 8080 on the host to be a port forward to port 80 inside the VM by default [00:25:38] physikerwelt1: what did you do to get port 80 on the host to forward to the mw-vagrant VM? [00:26:18] 06Labs, 10Math, 10MathSearch, 10Mathoid: Set up an domain name for the instance - https://phabricator.wikimedia.org/T134539#2268911 (10Krenair) an admin of the math project should do this [00:26:45] I set up iptables to forward traffic to port 8080 [00:27:09] let me switch that off [00:28:26] bd808: oh it seems that that was the problem [00:29:31] 06Labs, 10Math, 10MathSearch, 10Mathoid: Set up an domain name for the instance - https://phabricator.wikimedia.org/T134539#2268829 (10bd808) >>! In T134539#2268911, @Krenair wrote: > an admin of the math project should do this I think there is some sort of additional magic needed from a user with higher... [00:30:17] physikerwelt1: maybe a better way to forward port 80 traffic into the vm would be setup nginx as a reverse proxy [00:31:40] bd808: you mean this http://nginx.org/en/ or some labs specific version [00:31:52] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2268947 (10Krenair) [00:32:53] interestingly in deployment-prep I can't create domains [00:32:59] But there should exist a math.wmflabs.org domain [00:34:28] physikerwelt1: yeah the normal nginx package that you can install via apt-get. [00:34:46] but I'm still a bit baffled that you couldn't use the normal Labs proxy [00:35:17] I run lots and lots of mw-vagrant instances behind it and have never had problems [00:35:39] other than occasionally misconfiguring the destination for the proxy [00:36:59] bd808: I think it was related to timeouts, setting up nginx seems to be fairly easy (almost as simple as apache) [00:37:02] I'm not quite sure why this project doesn't have it's own domain [00:37:20] I might be able to make on [00:37:21] one* [00:37:49] I'm sure I've seen a script for doing it somewhere [00:39:23] physikerwelt1: you could certainly use apache too. nginx is just a bit more light weight than apache for a simple reverse proxy (or just about anything else) [00:41:24] bd808: yes I agree apache is quite heavy for that [00:41:35] ImportError: No module named v2 [00:41:35] ugh [00:44:26] Krenair: math.wmflabs.org used be an instance [00:44:40] python-designateclient version on labtestweb2001 is 1.5.0-1~cloud0... but on silver, 1.1.1-0ubuntu1~cloud0 [00:44:46] maybe that causes problems [00:49:19] 06Labs, 10Labs-Infrastructure, 06Operations: python-designateclient package version does not match between labtestweb2001 and silver - https://phabricator.wikimedia.org/T134543#2268964 (10Krenair) [00:50:43] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2268978 (10Krenair) I noticed that I can't create domains under deployment-prep, but math.wmflabs.org. should still be a domain that exists... [02:11:34] PROBLEM - Puppet staleness on tools-prometheus-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [02:59:11] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T133968#2269087 (10lfschenone) [03:20:28] Anybody else noticing labs login is a bit laggy right now? [03:41:06] sshing to bastion.wmflabs.org is slow enterprisey? [03:41:17] it was, until a few minutes ago [03:41:21] now it's fast, as usual [03:41:39] when it was slow, though, there was like a minute between one of my keypresses and it showing up on the terminal [03:44:02] 06Labs, 10Tool-Labs: Web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2219634 (10scfc) I just looked at `error.log` for which the last relevant line was: ``` 2016-05-06 03:16:39: (server.c.1558) server stopped by UID = 0 PID = 11824 ``` Then I looked up the corresponding... [03:45:51] based on "ldapsearch -x dc:dn:=208.80.155.153", you should be able to use bastion2-eqiad.wmflabs.org or bastion2.wmflabs.org or bastion-test.wmflabs.org to access bastion-02 instead [03:45:56] but none of them resolve :/ [03:46:24] additionally bastion1.wmflabs.org for the normal one [03:51:00] It'll probably just work as `ssh username@208.80.155.153` [03:51:51] I do have the rights necessary to fix up the domains but I'll probably just file a ticket later about those going missing [05:20:04] RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [06:36:54] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2269240 (10jcrespo) At 6:30 today: ``` MariaDB m5-master (none) > select * FROM information_schema.table_statistics WHERE table_schema='pdns'; +--------------+-----------------+----------... [08:34:53] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:56:00] (03PS1) 10Addshore: Send WMDE-Design to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/287182 [08:57:53] (03CR) 10WMDE-leszek: [C: 031] Send WMDE-Design to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/287182 (owner: 10Addshore) [08:59:45] (03CR) 10Merlijn van Deen: [C: 032] Send WMDE-Design to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/287182 (owner: 10Addshore) [08:59:54] thanks valhallasw`cloud ! :D [09:00:20] (03Merged) 10jenkins-bot: Send WMDE-Design to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/287182 (owner: 10Addshore) [09:01:03] hm, I can't add people to the wikibugs2 group [09:09:17] =o [09:12:18] !log tools.wikibugs Updated channels.yaml to: 904ce69d7f515ea9b47f621686f612a04ae94cc2 Send WMDE-Design to #wikimedia-de-tech [09:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [09:39:54] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:03:52] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [11:32:10] 06Labs, 10Labs-Sprint-115, 10Tool-Labs, 10labs-sprint-116, and 3 others: Attribute cache issue with NFS on Trusty - https://phabricator.wikimedia.org/T106170#2269592 (10JeanFred) (Not sure how I changed that a few days ago) [11:42:48] (03PS1) 10Lokal Profil: Add test for checkLat/checkLon and to test checkers/converters are triggered. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287198 [11:55:03] (03CR) 10Lokal Profil: "I'm unsure if test_trigger_problematic_check should really be included since it must either be dropped (or changed into a copy of test_tri" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287198 (owner: 10Lokal Profil) [12:02:58] (03CR) 10Lokal Profil: "@Krinkle: I agree about the migration, this is now https://phabricator.wikimedia.org/T134565" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286796 (owner: 10Lokal Profil) [12:14:12] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:49] !log tools.heritage Deployed latest from Git: db46042, c765e76, b5a731a, 7c27207, d4de720 (T134236), e7823ab & c83003b (T132647), 615ab28 [12:15:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [12:16:04] T132647: Prettify URLs in API/FormatHTML only works on Wikipedia links - https://phabricator.wikimedia.org/T132647 [12:16:05] T134236: Drop use of preg_replace(): with the /e modifier - https://phabricator.wikimedia.org/T134236 [12:44:13] (03PS1) 10Lokal Profil: Remove trailing separator from arguments [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287202 [13:19:41] 10Quarry: Quarry queries editable by several people - https://phabricator.wikimedia.org/T134570#2269818 (10Base) [13:29:35] 10Quarry: Limit number of concurrent queries one user can run at a time - https://phabricator.wikimedia.org/T104316#1413282 (10Base) Would be nice if it is multilevelled limitation with basing on perhaps users rights on wikis or something like this (perhaps also some additional whitelist for users who are not ac... [13:30:49] (03PS1) 10Lokal Profil: Correcting field matchings for two fr.wiki templates [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287209 [13:37:17] 10Quarry: Quarry shows old username after user was renamed - https://phabricator.wikimedia.org/T73064#771640 (10Base) I do not know hot it is build so do not know if that might occur (well hopefully it is based on IDs and not on usernames behind the scenes so it might not), but I hope that those renames if you m... [13:44:57] 10Labs-Other-Projects, 10Quarry, 06Discovery, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/ (like Quarry) - https://phabricator.wikimedia.org/T104762#1426314 (10Base) A stupid question by a person with bad SQL (progressed from almost zero mostly thanks to quarry) and almo... [13:46:26] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2269921 (10Andrew) There are two things happening here: 1) Creation of domains is currently an Ops-only activity. That's not a bug in our... [13:49:45] 06Labs, 10Labs-Infrastructure, 06Operations: python-designateclient package version does not match between labtestweb2001 and silver - https://phabricator.wikimedia.org/T134543#2269922 (10Andrew) 05Open>03Invalid Horizon is designed to be backwards-compatible with different OpenStack API versions, and we... [13:52:21] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2269940 (10Andrew) @jcrespo -- I want to make sure I understand your last comment. Those stats are cumulative since the db was created, right? So you're establishing a baseline and then w... [13:54:11] (03PS1) 10Lokal Profil: Add project to db-fields accessible by api [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287214 (https://phabricator.wikimedia.org/T134567) [13:54:25] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2269959 (10chasemp) A few notes on 286897 max-cache-entries 1M to 3M Currently on labservices1001: cache-bytes 89051465 cache-entries 817477 cache-hits 713865 cache-misses 4724778 significant miss ra... [13:55:16] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2269960 (10Physikerwelt) I wonder who is using http://math.wmflabs.org ? Is it used for anything? [13:58:10] andrewbogott: good morning [14:03:29] 'morning! [14:03:35] Sorry for my long-winded and confusing response [14:04:11] andrewbogott: I'm very happy about your support [14:04:58] If you don't mind my breaking math.wmflabs.org for a bit, I can try to get things organized better. I think I've done it before, just can't quite remember the steps [14:05:29] What is math.wmflabs.org ? [14:05:49] it might belong to another labs project [14:06:50] nope, yours [14:07:01] if you look at the 'Web Proxies' tab in horizon... [14:07:11] you will see that your project has very very many proxies set up [14:07:25] it says Displaying 10 items [14:07:29] including that one [14:07:48] ah ok sorry now I realize that [14:08:08] ok I deleted that one [14:08:55] really? That's easy then :) [14:09:49] bah, and there's a bug in my tool here, hang on... [14:09:56] there are some others, I can not delete [14:10:10] mws.beta.wmflabs.org for instance [14:13:02] andrewbogott: can you delete all that of the kind *.(testme|beta).wmflabs.org ? [14:15:21] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2270021 (10jcrespo) > Those stats are cumulative since the db was created, right? So you're establishing a baseline and then will compare those stats at a later date to make sure there's b... [14:15:31] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2270022 (10chasemp) >>! In T124680#2266224, @scfc wrote: > @chasemp: Thanks. Does that mean that some Labs instance(s) are effectively causing the outages? Could you identify those with your debugging?... [14:26:36] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2270062 (10Krenair) krenair@novaproxy-01:~$ host math.wmflabs.org Host math.wmflabs.org not found: 3(NXDOMAIN) dynamicproxy-api doesn't see... [14:32:15] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2270092 (10Andrew) @krenair, that's because he just deleted it, as per ongoing IRC discussion [14:41:15] 06Labs, 06Team-Practices, 07Privacy, 15User-bd808: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2270101 (10MBinder_WMF) Thanks for taking care of this so quickly! [14:46:58] physikerwelt: you should have a domain that you can create records on now — do you see it? [14:48:27] yes I see it [14:48:29] thank you [14:50:30] physikerwelt: I don't immediately see how to clean up those other proxies, can you create a bug about that so I don't forget? [14:51:24] andrewbogott: ok will do. At the moment they are not a problem, but I could imagine that it will cause great confusion in a few years from now [14:51:53] (03CR) 10Jean-Frédéric: [C: 032] Remove trailing separator from arguments [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287202 (owner: 10Lokal Profil) [14:52:41] (03Merged) 10jenkins-bot: Remove trailing separator from arguments [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287202 (owner: 10Lokal Profil) [14:52:41] andrewbogott: I could set up a domain there cool. I guess it will take some time until it gets into effect [14:52:51] (03CR) 10Jean-Frédéric: [C: 032] Add project to db-fields accessible by api [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287214 (https://phabricator.wikimedia.org/T134567) (owner: 10Lokal Profil) [14:53:30] 10Quarry: Quarry queries editable by several people - https://phabricator.wikimedia.org/T134570#2270130 (10Aklapper) @Base: Is "Quarry queries editable by several people" a bug report (like "this should not happen")? Is "Quarry queries editable by several people" an enhancement request (like "please support this... [14:53:47] physikerwelt: should only take a few minutes unless there's a bug [14:54:23] ... mh in my browser I can not even access the ip 208.80.155.185 [14:54:38] (03Merged) 10jenkins-bot: Add project to db-fields accessible by api [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287214 (https://phabricator.wikimedia.org/T134567) (owner: 10Lokal Profil) [14:55:16] beta.math.wmflabs.org resolves for me — so the dns bit is working at least [14:55:56] yes for me too [14:56:17] ok — do you mind closing that other ticket too? I'm on to other things already :) [14:58:21] !log tools.heritage Deployed latest from Git: e5a9f01 and d509343 (T134567) [14:58:23] T134567: Notice: Undefined property: stdClass::$project in /data/project/heritage/heritage/api/includes/FormatHtmllist.php - https://phabricator.wikimedia.org/T134567 [14:58:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [15:00:56] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2270149 (10Physikerwelt) The DNS problem is resolved. On the other hand the IP does not respond anymore ``` $ ping beta.math.wmflabs.org PI... [15:01:07] (03CR) 10Jean-Frédéric: "That looks good to me overall but I don’t enough about it to +2 it. Feel free to though :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286149 (https://phabricator.wikimedia.org/T39974) (owner: 10Lokal Profil) [15:04:28] 10Quarry: Quarry queries editable by several people - https://phabricator.wikimedia.org/T134570#2270171 (10Base) An enhancement request (I believed it would be clear from the description). Ok will rename. [15:04:58] 10Quarry: Add support for quarry queries to be editable by several people - https://phabricator.wikimedia.org/T134570#2270172 (10Base) [15:10:36] 10Labs-Other-Projects, 10Quarry, 06Discovery, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/ (like Quarry) - https://phabricator.wikimedia.org/T104762#2270182 (10Multichill) >>! In T104762#2269919, @Base wrote: > A stupid question by a person with bad SQL (progressed from... [15:15:48] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2270200 (10Physikerwelt) 05Open>03Resolved Works fine after restart. Thanks to all of you. [15:16:15] 06Labs, 10Math, 10MathSearch, 10Mathoid: Make it possible for admins of the math project to manage domains in that project - https://phabricator.wikimedia.org/T134539#2270203 (10Physikerwelt) a:05Physikerwelt>03Andrew [15:19:44] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2270208 (10Andrew) Now is good -- or tomorrow :) [15:24:58] 06Labs, 10DBA, 10MediaWiki-General-or-Unknown: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2263175 (10Umherirrender) https://www.mediawiki.org/wiki/Manual:User_table#user_editcount > NOT* intended to be an accurate copy of COUNT(*) WHERE rev_user=user_id... [15:36:09] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2270246 (10chasemp) [[ https://graphite.wikimedia.org/render/?width=1128&height=522&_salt=1462548029.977&lineMode=connected&areaMode=stacked&target=servers.labservices1001.powerdns_recursor.cache-hits&tar... [15:37:43] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2270257 (10chasemp) [15:38:13] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:39] hm...up but slow to me^? [15:41:12] Fairly quick for me, but slower than usual. [15:43:09] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824844 bytes in 3.837 second response time [15:44:39] The check is unreliable as usual. [15:45:10] 10Labs-Other-Projects, 10Quarry, 06Discovery, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/ (like Quarry) - https://phabricator.wikimedia.org/T104762#2270273 (10Base) >>! In T104762#2270182, @Multichill wrote: >>>! In T104762#2269919, @Base wrote: >> A stupid question by... [15:53:14] 06Labs: new bootstrap-vz jessie images don't log (and maybe don't start at all) - https://phabricator.wikimedia.org/T133551#2270285 (10Andrew) [15:53:16] 06Labs: Console log broken in new Jessie labs base images - https://phabricator.wikimedia.org/T133363#2270284 (10Andrew) [16:24:12] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:08] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824988 bytes in 5.980 second response time [16:30:28] ^ tools.admin webservice is using 4 CPUs at 100% [16:30:40] so no, it's not unreliable, there's actually something wrong ;-) [16:32:01] wtf [16:32:06] 10.68.21.49 tools.wmflabs.org - [06/May/2016:16:31:26 +0000] "GET /w/index.php?title=Special:Book&bookcmd=book_creator&referer=Talk%3AFernando+Andacht HTTP/1.1" 404 1494 "-" "winHttp 9292" [16:32:46] and somehow applebot is repeatedly hitting / [16:32:57] yuck [16:33:13] tools.admin is such a yucky pile of yuck [16:33:14] so probably not actually applebot [16:35:13] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:12] ^ works for me [16:36:25] * bd808 is playing with the new xhyve based docker for osx [16:36:55] andrewbogott: yes, it's just heavily overloaded, and thus sometimes doesn't respond within 10 s [16:37:48] valhallasw`cloud: do lots of other tools use that tool for some reason? [16:38:12] andrewbogott: it serves all proxy error pages [16:38:21] although that doesn't seem to be the issue [16:39:20] building the list of all projects isn't a completely light weight operation as I recall [16:39:32] and it of course has no caching at all [16:40:22] and in this case it seems to be applebot hitting / repeatedly [16:40:40] as in multiple times per second [16:40:46] hm, caching would be nice [16:41:06] I've been resisting rewriting the whole thing [16:41:24] I want to get some other things fixed up first [16:41:40] I'm considering banning applebot [16:41:48] either via robots.txt or via the actual banning mechanism [16:42:35] if its hitting repeatedly chances are that robots.txt won't work [16:42:40] what is applebot? [16:42:44] but you can try [16:43:00] andrewbogott: https://support.apple.com/en-gb/HT204683 [16:43:17] dang [16:43:22] why so hungry, applebot? [16:43:53] bd808: there's also the https://tools.wmflabs.org/.error/banned.html route [16:44:09] which is what we've done with e.g. tweetmemebot [16:44:31] or we can email them [16:44:37] "Any requests without user-agent." -- I'd love to turn that back on for api.php :/ [16:45:31] valhallasw`cloud: I honestly don't see what banning would hurt [16:45:47] siri what tool should I use to dos tool labs? [16:46:30] 06Labs, 10Tool-Labs: Applebot overloading all of tool labs - https://phabricator.wikimedia.org/T134591#2270441 (10valhallasw) [16:47:09] 06Labs, 10Tool-Labs: Applebot overloading all of tool labs - https://phabricator.wikimedia.org/T134591#2270454 (10valhallasw) [16:47:29] valhallasw`cloud: Ideally we would block it, open a ticket with Apple, and then unblock when Apple responds [16:47:37] andrewbogott: sounds good [16:47:58] Do you want to do the 'block it' part and leave notes in that bug? I'll see if I can figure out how to report. [16:48:05] have you been looking at the proxy side where you can see the remote IP? Is it really Apple? [16:48:31] bd808: it's the 17.* netblock, as apple suggests it should be [16:48:38] *nod* [16:49:59] andrewbogott: https://support.apple.com/en-gb/HT204683 says Apple-NOC “at” apple.com [16:50:07] Yep, I'm writing an email now [16:50:13] thanks! [16:50:18] Can you clarify why a robots.txt solution isn't enough? [16:50:37] that will take a day before they respond to it [16:51:20] it's not that we don't want them to crawl, we don't want them to crawl... the same page a gazillion times [16:51:37] ok, that's reasonable [16:52:27] huh [16:52:36] they stopped [16:52:40] and the ban isn't there yet [16:53:07] ah missed the fun, let's still put the ban in? [16:53:31] still hitting /apple-touch-icon.png and /apple-app-site-association like crazy [16:53:44] 'if it 404s, please don't try again 100ms later' [16:53:57] https://www.irccloud.com/pastebin/vZwbh7rf/ [16:54:24] if they respected it crawl-delay in robots is for this but [16:54:32] probably easiest just to block their UA [16:54:39] and see if they respond [16:54:42] ah, we don't have a crawl-delay [16:55:01] hm… does that mean this is our fault and I should leave them alone? [16:55:27] they still shouldnt hit the same endpoint over and ove ragain [16:55:40] maybe add the log from https://phabricator.wikimedia.org/T134591 ? [16:56:02] I don't know much about applebot, seems like a weird [16:56:04] "It respects customary robots.txt rules and robots meta tags." [16:56:07] https://support.apple.com/en-us/HT204683 [16:56:17] so a crawl-delay could help [16:56:27] I know google ignores it [16:56:28] the jerks [16:56:34] ok, 403 is in place [16:56:47] doesn't stop them from crawling, but at least nginx is now serving it directly [16:57:00] thanks valhallasw`cloud [16:57:14] we clearly have gotten too used to tools page failures from historical issues [16:57:22] ok, emailed apple [16:57:28] 06Labs, 10Tool-Labs: Applebot overloading all of tool labs - https://phabricator.wikimedia.org/T134591#2270463 (10valhallasw) Although they have stopped crawling / by themselves, they are still hitting other endpoints like crazy: ``` 17.142.152.10 - - [06/May/2016:16:56:18 +0000] "GET /apple-app-site-associat... [16:57:28] and cc'd you two [16:57:38] andrewbogott: thanks! [16:58:01] I'm not sure we care about siri and spotlight suggestions tbh [16:58:40] I agree, but it seems like good citizenship to point out to Apple that they're DOSing people [16:58:53] also https://phabricator.wikimedia.org/robots.txt [16:58:53] vs [16:58:57] http://tools.wmflabs.org/robots.txt [16:59:09] some parsings of robots may choke on lack of newline? [16:59:31] I actually don't know if it's convention or standard [16:59:41] \r\n? or is it just content-type? [16:59:45] there should be newlines [16:59:54] that's what I thought too [16:59:56] content-type:text/html [16:59:58] valhallasw`cloud: hmm [16:59:58] there are comment tags in there [17:00:11] there are newlines, but the .tzt xt is rendered as html [17:00:26] is that served by the admin tool or something else? [17:00:40] admin tool [17:00:41] I think it comes from nginx [17:00:48] ah nope :) [17:01:02] only /.error/* is served from nginx [17:01:06] I recall seeing some UA bans directly in the nginx config as well [17:01:06] (03PS1) 10Lokal Profil: Add lang argument where pywikibot.Site().language() was used [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287248 [17:01:09] but it may have been a hack [17:01:22] yeah, UA and IP bans are in nginx [17:01:48] they get served a 403 with https://tools.wmflabs.org/.error/banned.html as content [17:01:49] valhallasw`cloud: are you in a place to easily add crawl-delay 3 to that? [17:01:54] ya [17:02:20] (03CR) 10Lokal Profil: "Discovered when trying to trigger a manual run to figure out T112415" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287248 (owner: 10Lokal Profil) [17:03:39] (03PS1) 10Merlijn van Deen: Add crawl delay [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287249 [17:03:42] ^ [17:04:14] (03CR) 10BryanDavis: [C: 031] Add crawl delay [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287249 (owner: 10Merlijn van Deen) [17:04:24] (03PS2) 10Merlijn van Deen: Add crawl delay [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287249 [17:04:32] sweeet [17:04:53] (03CR) 10Andrew Bogott: [C: 031] Add crawl delay [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287249 (owner: 10Merlijn van Deen) [17:05:26] You should also all have +2 on that repo now [17:06:05] (03CR) 10BryanDavis: [C: 032] Add crawl delay [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287249 (owner: 10Merlijn van Deen) [17:06:11] \o/ [17:06:28] valhallasw`cloud: no jenkins, though? [17:06:43] just slow [17:06:56] needs to build the deb package and run tests [17:08:06] (03CR) 10Lokal Profil: [C: 032] "What is the worst that could happen ;)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286149 (https://phabricator.wikimedia.org/T39974) (owner: 10Lokal Profil) [17:10:10] (03Merged) 10jenkins-bot: [NOT TESTED] Remove monument_random from database and api. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286149 (https://phabricator.wikimedia.org/T39974) (owner: 10Lokal Profil) [17:10:10] (03PS2) 10Lokal Profil: Add test for checkLat/checkLon and test checkers/converters are triggered [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287198 [17:12:11] 06Labs, 10Tool-Labs: Web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2270510 (10Nettrom) Memory usage has been one of my concerns as SuggestBot's link recommender is an iterative breadth-first search from a set of articles, and that's what is mostly used. However, the scrip... [17:18:51] (03CR) 10Jean-Frédéric: [C: 032] Add lang argument where pywikibot.Site().language() was used [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287248 (owner: 10Lokal Profil) [17:19:48] (03Merged) 10jenkins-bot: Add lang argument where pywikibot.Site().language() was used [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287248 (owner: 10Lokal Profil) [17:20:48] (03CR) 10Merlijn van Deen: "recheck" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287249 (owner: 10Merlijn van Deen) [17:20:57] * valhallasw`cloud is unsure what jenkins is smoking [17:21:46] (03CR) 10Jean-Frédéric: [C: 032] Add test for checkLat/checkLon and test checkers/converters are triggered [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287198 (owner: 10Lokal Profil) [17:22:36] (03Merged) 10jenkins-bot: Add test for checkLat/checkLon and test checkers/converters are triggered [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287198 (owner: 10Lokal Profil) [17:22:38] (03CR) 10Jean-Frédéric: "Code coverage on that file climbed from 19% to 33% :) Nice :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287198 (owner: 10Lokal Profil) [17:25:36] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281930 (https://phabricator.wikimedia.org/T39422) (owner: 10Lokal Profil) [17:26:32] (03CR) 10jenkins-bot: [V: 04-1] [NOT TESTED] Re-implement checks to not use globals [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281930 (https://phabricator.wikimedia.org/T39422) (owner: 10Lokal Profil) [17:27:48] (03PS1) 10BryanDavis: Serve .txt files with mime-type text/plain [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287250 [17:28:02] valhallasw`cloud: ^ [17:34:46] (03CR) 10Rush: [C: 031] Serve .txt files with mime-type text/plain [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287250 (owner: 10BryanDavis) [17:35:11] (03CR) 10BryanDavis: [C: 032] Serve .txt files with mime-type text/plain [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287250 (owner: 10BryanDavis) [17:41:20] (03PS3) 10Lokal Profil: Re-implement checks to not use globals [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281930 (https://phabricator.wikimedia.org/T39422) [17:44:42] (03CR) 10Merlijn van Deen: [C: 032] Serve .txt files with mime-type text/plain [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287250 (owner: 10BryanDavis) [17:50:18] (03CR) 10Jean-Frédéric: [C: 032] "54% :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281930 (https://phabricator.wikimedia.org/T39422) (owner: 10Lokal Profil) [17:51:18] (03Merged) 10jenkins-bot: Re-implement checks to not use globals [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281930 (https://phabricator.wikimedia.org/T39422) (owner: 10Lokal Profil) [17:56:47] valhallasw`cloud: what's the magic needed to update the admin tool? I thought it was just a git clone these days but that seems to not be the case [17:57:11] bd808: puppet agent -tv on the proxy host, I thought, but that doesn't seem to be the case either [17:57:31] probably on tools-services or something like that then [17:57:31] valhallasw`cloud: not the proxy host [17:57:36] or wait for 20 minutes :-p [17:57:40] yah, services, when we got rid of NFS on the proxy hosts [17:57:45] 'puppet, somewhere' [17:57:46] the content of /data/project/admin is a mess [17:57:53] YuviPanda: eh, proxy hosts have NFS [17:58:10] maybe not on purpose [17:58:24] but /home is mounted on tools-proxy-01 [17:59:05] valhallasw`cloud: oh, wat. I thought I fully got rid of it [17:59:10] maybe we should clean up those 20G error log and 10G access log files :/ [17:59:13] * YuviPanda gets rid of it [17:59:19] bd808: not until it hits 1T! [17:59:42] valhallasw`cloud: am getting rid of it now [18:00:12] valhallasw`cloud: if you cd out of your home ;) [18:00:24] godog: did you end up being able to setup grafana / public prometheus? [18:01:39] valhallasw`cloud: found it! the git repo is /data/project/admin/toollabs [18:02:10] git fetch && git rebase @{upstream} got the code deployed [18:02:44] well git fetch; git log --stat HEAD..@{upstream}; git rebase @{upstream} [18:03:10] https://tools.wmflabs.org/robots.txt is now text/plain [18:05:47] YuviPanda: nay, though if feel comfortable with it, it is sufficient to create a proxy towards tools-prometheus-01:9090 [18:06:12] godog: hmm but the go debugger makes me feel uncomfortable. [18:09:01] YuviPanda: hehe fair enough, it is essentially one nginx configuration away but I haven't written that yet [18:09:35] ok time to go, enjoy the weekend! [18:09:47] YuviPanda: sorry. done :-) [18:09:50] godog: yeah [18:10:06] valhallasw`cloud: done. it was also in /etc/fstab [18:10:29] YuviPanda: is .bigbrotherrc dead tech? [18:10:48] * bd808 is trying to clean up the piles of old trash in tools.admin [18:11:08] bd808: still valid for non-webservice continuous jobs [18:11:14] not sure if tools.admin has any [18:11:19] bd808: not yet. [18:11:25] unfortunately [18:11:37] valhallasw`cloud: toolhistory apparently [18:12:03] valhallasw`cloud: NFS gone [18:12:47] oh look. that's an hourly cron job implemented as an infinite loop bash script :/ [18:13:18] why in the hell would that ever make sense? [18:13:46] ^ [18:14:11] bd808: because cron was unstable at some point, I think [18:14:24] heh [18:14:37] toollabs: hacks upon hacks upon hacks [18:14:49] with a side of bash and perl! [18:16:12] so once an hour this script: clones a git repo; dumps an ldif; adds it to the git repo; commits; pushes; rm's the repo [18:16:51] this logs servicegroup members, right? [18:16:52] and the repo is clones is ... local only [18:17:14] yeah. the output of ldaplist -l servicegroups [18:17:20] ah, that's what behind that dark corner [18:17:33] I, but, well, no, hrrmmmm [18:17:36] * YuviPanda puts his blinders back on [18:19:29] YuviPanda: it's better than reddit! [18:19:55] low bar! [18:20:29] can I disable the access.log for the admin tool? it there any need for it at all? [18:21:14] bd808: it was useful to figure out applebot was misbehaving [18:21:19] but other than that... [18:21:37] (sshing to the proxy is easy enough for anyone who can get to that access.log) [18:21:50] yeah. that was kind of my thought [18:22:04] and a way to git rid of a 10G nfs file [18:22:10] *get [18:22:29] we need to solve the logging problem one of these days. [18:22:44] the 20G error.log is even more annoying [18:25:46] (03PS1) 10Dzahn: mw_rc_irc: add missing auth.conf snippet [labs/private] - 10https://gerrit.wikimedia.org/r/287254 [18:27:50] (03CR) 10Dzahn: [C: 032 V: 032] "fixes running it in labs" [labs/private] - 10https://gerrit.wikimedia.org/r/287254 (owner: 10Dzahn) [18:29:12] bd808: trunc? :-) [18:29:19] ah [18:29:23] never mind, I misread what you said [18:31:28] bd808: everytime you disable (extensive) logging you can slap the "audits-data-retention" tag on it in phab , heh :) [18:32:07] * YuviPanda guesses the error.log is just mundane php errors :D [18:32:19] error.log is the html sanitizer erroring out [18:32:27] it was a big stack trace over and over for a missing cache directory [18:32:41] so I made the directory ;) [18:32:42] just read access.log :) [18:33:24] bd808: how dare you etc :) [18:33:53] bd808: so it's going to cache on nfs? :-p [18:34:11] yeah, but better than nothing [18:34:40] that whole htmlpurifier thing is badly conceived [18:35:13] 06Labs, 10Tool-Labs: Please install mwoauth to Tool Labs - https://phabricator.wikimedia.org/T134600#2270697 (10Urbanecm) [18:35:35] 06Labs, 10Tool-Labs: Please install mwoauth to Tool Labs - https://phabricator.wikimedia.org/T134600#2270710 (10Urbanecm) [18:35:49] ^ please use a virtualenv [18:36:13] they are using the central install [18:36:44] but there's no debian package [18:36:47] so please use a shared venv? [18:37:01] yeah, fair enough [18:42:50] !log tools.heritage Deployed latest from Git, 2d3ee40 (T39974), 766d814, e2fac07 and d2c242a (T39422) [18:42:51] T39974: Add Remove monument_random field + index to the monuments_all table - https://phabricator.wikimedia.org/T39974 [18:42:52] T39422: Lat/lon should be NULL when empty - https://phabricator.wikimedia.org/T39422 [18:42:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [18:44:40] !log tools.admin restarted webservice [18:44:51] and broke it? [18:45:26] one service more, one service less... [18:45:55] !log tools.admin Unbroke webservice [18:45:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL, Master [18:58:44] (03PS1) 10Lokal Profil: Fix mistake introduced in Ie4b6cbcbcf81b17a7bbc8edfa16f5b96470bcb21 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287265 [18:58:56] (03CR) 10Lokal Profil: "The answer is... you break the api" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286149 (https://phabricator.wikimedia.org/T39974) (owner: 10Lokal Profil) [19:09:03] yeah thanks bd808 :) dark corners indeed [19:12:30] (03CR) 10Lokal Profil: "Ok. So under the hood the lang parameter triggers pywikibots own argument handler. So yes the -lang parameter is required (undocumented) b" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287248 (owner: 10Lokal Profil) [19:16:28] (03CR) 10Jean-Frédéric: [C: 032] Fix mistake introduced in Ie4b6cbcbcf81b17a7bbc8edfa16f5b96470bcb21 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287265 (owner: 10Lokal Profil) [19:16:30] (03PS1) 10Lokal Profil: Revert "Add lang argument where pywikibot.Site().language() was used" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287269 [19:16:53] (03Merged) 10jenkins-bot: Fix mistake introduced in Ie4b6cbcbcf81b17a7bbc8edfa16f5b96470bcb21 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287265 (owner: 10Lokal Profil) [19:17:00] (03CR) 10Jean-Frédéric: "(So it's actually doable to review patches on your phone from the pub)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287265 (owner: 10Lokal Profil) [19:18:49] (03CR) 10Lokal Profil: [C: 032] "Reverting myself" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287269 (owner: 10Lokal Profil) [19:19:38] (03Merged) 10jenkins-bot: Revert "Add lang argument where pywikibot.Site().language() was used" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287269 (owner: 10Lokal Profil) [19:21:51] (03CR) 10Lokal Profil: "Reverted the commit but question about documentation remains" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287248 (owner: 10Lokal Profil) [19:24:18] wait. this sounds awesome .. tell me more < grrrit-wm> (CR) Jean-Frédéric: "(So it's actually doable to review patches on your phone from the pub)" .. heh! [19:26:23] !log tools.heritage Deployed latest from Git, a724279 , d9ae73d (reverts 766d814 ) [19:26:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [19:26:38] mutante: I think there's an android app [19:26:51] and there's the option of logging in to the web interdface and using the zoom function [19:28:10] 06Labs, 10Tool-Labs: Please install mwoauth to Tool Labs - https://phabricator.wikimedia.org/T134600#2270779 (10valhallasw) 05Open>03declined Unfortunately, ubuntu does not have a package for python-mwoauth, so we cannot install the package globally. You will have to use a virtualenv instead: https://wikit... [19:29:06] valhallasw`cloud: mgerrit, i tried :) [19:29:11] its good to read [19:29:16] but not to send reviews [19:29:41] https://f-droid.org/repository/browse/?fdfilter=mgerrit&fdid=com.jbirdvegas.mgerrit [19:30:39] web interface is good enough from my phone... [19:30:58] yea, true. for reading it's ok [19:31:10] what i really want is an app to review though [19:31:15] that is mobile optimized [19:31:34] well, on the other hand.. it wouldnt help me much [19:31:47] because for ops/puppet you still have to merge on master [19:31:50] I don't even have email on my phone anymore [19:31:57] and then ssh client on phone .. is also needed [19:32:05] and having the keys on it.. meh [19:32:20] yea, agree. i dont pull email [19:32:25] at least not automatic and with notifications [19:33:09] it's all about the notifications and distractions [19:33:37] I don't have email at all since I find that I compulsively refresh [19:34:08] I do have Telegram, Whatsapp, Signal and SMS (For the poor americans stuck with ancient technology only) [19:35:46] Telegram, Signal, Kik.. because it's impossible to get people one a single one [19:36:03] no Whatsapp, no Facebook. have to stop somewhere [19:47:00] !log tools.heritage Deployed latest pywikibot-core/2.0 from Git [19:47:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [19:47:27] !log tools.wikibugs temporarily offline for Danny_B's batch job [19:47:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [19:53:13] (03CR) 10Lokal Profil: "That explains the quick response =)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287265 (owner: 10Lokal Profil) [19:55:59] * valhallasw`cloud wonders where wikibugs is [19:56:34] fun with SGE [20:11:26] So I deleted an instance and created a new one with the same name (deployment-phab) but now I get 'requested host does not exist' when trying to edit it on wikitech - https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=59e7db54-8cc9-4168-9c7a-1784947ac8d8&project=deployment-prep®ion=eqiad [20:12:19] the instance shows up in the list but the edit link leads to nowhere. It seems fine in horizon but it seems like it's not possible to edit puppet classes in horizon (correct me if I'm wrong about that) [20:12:45] twentyafterfour: the list might be caching the old guid [20:13:07] I tried with the guid copy/pasted from horizon - it's the same id [20:13:24] Try logging out and in again on wikitech? [20:13:59] or create a new instance with another name -- because of caching here and there it always seems to cause issues [21:17:04] yeah, same name instance has at least one probably more race conditions with short term delete/recreate [21:17:11] AFA dns etc [21:30:58] (03PS1) 10Lokal Profil: Standardise php whitespace to tab [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287290 [21:31:23] 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2271137 (10chasemp) There is a lot of spiking in activity for the recursor especially this afternoon. That is me poking our cache settings and resolver response under load. I !log'd it. [21:32:33] (03CR) 10Lokal Profil: "I'm happy to make them all (4) space indented if that is preferred." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287290 (owner: 10Lokal Profil) [22:03:12] is phabricator db replicated to labs? [22:05:22] Danny_B: no, but there are (limited) dumps [22:15:25] there is a long history of tasks outlining all the private data involved and a static nightly json dump [22:23:28] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#2271259 (10Danny_B) [22:23:32] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Python webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134494#2271260 (10Danny_B) [22:23:35] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first PHP webservice" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134493#2271261 (10Danny_B) [22:23:55] 06Labs, 10Tool-Labs, 07Documentation: Document disabling scheduler (#jobs/time) overload protection temporarily - https://phabricator.wikimedia.org/T123411#2271262 (10Danny_B) [22:24:10] 06Labs, 07Documentation, 07Puppet: Missing documentation for labs puppet roles - https://phabricator.wikimedia.org/T91770#2271263 (10Danny_B) [22:24:34] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Document each set of hosts on Tool Labs properly - https://phabricator.wikimedia.org/T101756#2271264 (10Danny_B) [22:24:54] 06Labs, 10Tool-Labs, 07Documentation: Explicitly document policies for requesting new projects - https://phabricator.wikimedia.org/T101687#2271265 (10Danny_B) [22:25:10] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, 07Documentation: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#2271266 (10Danny_B) [22:25:28] 06Labs, 10Tool-Labs, 07Documentation: Re-organize Tool Labs documentation - https://phabricator.wikimedia.org/T91509#2271267 (10Danny_B) [22:25:40] 06Labs, 10Tool-Labs, 07Documentation: Wikimedia Labs system admin (sysadmin) documentation sucks - https://phabricator.wikimedia.org/T57946#2271268 (10Danny_B) [22:25:57] 06Labs, 10Labs-Infrastructure, 07Documentation: add Central Logging Service documentation - https://phabricator.wikimedia.org/T56702#2271269 (10Danny_B) [22:26:04] 06Labs, 10Tool-Labs, 07Documentation: add basic expectations management to docs - https://phabricator.wikimedia.org/T56701#2271270 (10Danny_B) [22:29:59] 06Labs, 10DBA, 10Phabricator: Replicate Phabricator database to Labs - https://phabricator.wikimedia.org/T134610#2271275 (10Danny_B) [22:30:49] (03PS1) 10Alex Monk: Fix default IRC rc passwords [labs/private] - 10https://gerrit.wikimedia.org/r/287293 [22:31:58] (03CR) 10Alex Monk: "(I just put FAKEFAKEFAKE through mkpassword on deployment-ircd)" [labs/private] - 10https://gerrit.wikimedia.org/r/287293 (owner: 10Alex Monk) [22:38:57] 06Labs, 10DBA, 10Phabricator: Replicate Phabricator database to Labs - https://phabricator.wikimedia.org/T134610#2271289 (10scfc) [22:56:22] 06Labs, 10DBA, 10Phabricator: Replicate Phabricator database to Labs - https://phabricator.wikimedia.org/T134610#2271275 (10mmodell) We can't release the full phabricator database because it contains sensitive details which cannot be shared with the general public. We do, however, have a "public task dump"... [23:22:38] (03CR) 10Dzahn: [C: 032] Fix default IRC rc passwords [labs/private] - 10https://gerrit.wikimedia.org/r/287293 (owner: 10Alex Monk) [23:22:48] (03CR) 10Dzahn: [V: 032] Fix default IRC rc passwords [labs/private] - 10https://gerrit.wikimedia.org/r/287293 (owner: 10Alex Monk)