[07:51:07] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Octahedron80 was created, changed by Octahedron80 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Octahedron80 edit summary: Created page with "{{Tools Access Request |Justification=interwiki on th.wiktionary |Completed=false |User Name=Octahedron80 }}" [08:05:04] (03CR) 10Lokal Profil: [C: 032] Fix ShellCheck errors in bin scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296774 (owner: 10Jean-Frédéric) [08:05:54] (03Merged) 10jenkins-bot: Fix ShellCheck errors in bin scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296774 (owner: 10Jean-Frédéric) [08:17:13] !log commonsarchive downgraded MediaWiki to 1.26, again [08:17:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive/SAL, Master [09:35:44] 06Labs, 10Labs-Infrastructure, 10Monitoring, 07Blocked-on-Operations: Provide a grafana installation for labs - https://phabricator.wikimedia.org/T137216#2420874 (10akosiaris) [09:35:45] 06Labs, 07Graphite: Setup "official labs grafana" instance - https://phabricator.wikimedia.org/T120295#2420876 (10akosiaris) [09:36:56] 06Labs, 07Graphite: Setup "official labs grafana" instance - https://phabricator.wikimedia.org/T120295#1850400 (10akosiaris) [09:41:20] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2420895 (10yuvipanda) @Magnus I was initially mounting only `/data/project/$toolname` inside the container, but since it looks like the symlink points to a *different* shared common... [09:49:47] Msjeas [10:57:24] Krinkle I've moved /static/ to k8s btw - notifying you since you seem to be the only active maintainer on IRC. the homedir redirects to /static-browser/ though, and that's borked, unrelated to any of this. I'm going to take a look there now [11:15:16] (03PS1) 10Lokal Profil: Restructure logging [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296885 (https://phabricator.wikimedia.org/T138633) [11:18:53] (03CR) 10Lokal Profil: [C: 032] Download Database dump to conf folder [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296798 (owner: 10Jean-Frédéric) [11:19:38] (03Merged) 10jenkins-bot: Download Database dump to conf folder [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296798 (owner: 10Jean-Frédéric) [11:20:54] (03CR) 10Lokal Profil: "looks good but does this also fix the "ts" prefix?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296795 (owner: 10Jean-Frédéric) [11:31:32] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kavimukil was created, changed by Kavimukil link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Kavimukil edit summary: Created page with "{{Tools Access Request |Justification=For my mobile app as API |Completed=false |User Name=Kavimukil }}" [11:37:23] JeanFred around? [11:49:01] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Enable HTTP based service checks for k8s webservices - https://phabricator.wikimedia.org/T139157#2421136 (10yuvipanda) [11:49:54] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Enable mod_status by default in lighttpd-webservice & make that a http health check - https://phabricator.wikimedia.org/T139158#2421149 (10yuvipanda) [11:57:55] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Enable mod_status by default in lighttpd-webservice & make that a http health check - https://phabricator.wikimedia.org/T139158#2421180 (10yuvipanda) ``` root@labstore1001:/srv/project/tools/project# find . -maxdepth 2 -name '.lighttpd.conf' -exec grep -i 'server-statu... [12:18:56] (03PS1) 10Paladox: Replace git.wikimedia.org url with diffusion url [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/296900 (https://phabricator.wikimedia.org/T139089) [13:23:12] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Octahedron80 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=709779 edit summary: [13:25:21] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kavimukil was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=709782 edit summary: [13:46:21] 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2421297 (10Niharika) >>! In T138679#2410144, @Shoichi wrote: > Thx to Luke081515 help me setting up ids-testing instance,and setup the testing wiki . Now working : > > http://ids-testing.wmflabs.org/wiki/ Awesome! [14:07:42] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421318 (10Dvorapa) [14:08:14] yuvipanda: see above ^ [14:11:59] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421351 (10tom29739) [14:23:47] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421318 (10jcrespo) Query limit is also enforced at server time, do do not worry about this having any kind of impact on our infrastructure. It would be interesting to fix the error/mistake/race condition/bug over... [15:09:35] is the wikitech wiki database replicated to labs? if yes, under which name? [15:11:41] Danny_B, I do not think so, it is completely separate than the others for technical reasons [15:12:12] the name of the wiki is "labswiki" [15:12:30] yuvipanda: oh, tools.static not tools-static [15:12:36] labswiki == wikitech? [15:12:39] yuvipanda: I don't use that anymore [15:12:42] Danny_B: yes. [15:12:44] thx [15:13:44] 10Quarry: Wrong status of queries in Recent Queries list - https://phabricator.wikimedia.org/T137517#2421580 (10Dvorapa) [15:13:51] hmm labswiki doesn't exist [15:13:59] labswiki_p doesn't have any tables [15:14:20] Yeah, it's not a main cluster wiki [15:14:29] It's not in centralauth, and it's not regular sign on. [15:14:35] It's fishbowl/ldap [15:14:36] 10Quarry: Wrong status of queries in Recent Queries list - https://phabricator.wikimedia.org/T137517#2370631 (10Dvorapa) [15:14:49] also, network-wise, it would be quite a nightmare to make that work [15:15:01] Danny_B: What do you want to query from labswiki? [15:16:03] jynus: Are they replicated in non-public ways though? E.g. replication to codfw, and xml backups maybe? [15:16:55] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421602 (10Dvorapa) But I wait more than 2 hours now and this query is completed, but the query above #6992 is now running more than 1 hour. The bug you mena is not the same as this, but you can find it in T137517 [15:16:59] no, I do not think so [15:17:06] snapshots are taken, though [15:17:12] aside from regular backups [15:17:52] https://wikitech-static.wikimedia.org/wiki/Main_Page [15:19:41] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2421605 (10Magnus) OK, this works, but now another of the scripts in the same tool fails. https://tools.wmflabs.org/wikidata-todo/duplicity.php?wiki=frwiki&mode=stats ``` Warning... [15:29:46] 06Labs, 10Tool-Labs, 13Patch-For-Review: HBA not configured correctly for tools-bastion-02 - https://phabricator.wikimedia.org/T104613#2421610 (10valhallasw) In the meanwhile, the `access.conf` situation has been cleared up, but login is still not working. The current issue is slightly different: ``` Jul 1... [15:33:27] jynus: Okay. [15:33:51] jynus: For the use case we had, API and special page actually suffice though ( https://wikitech.wikimedia.org/w/api.php?action=query&list=exturlusage&euquery=git.wikimedia.org ) [15:34:11] All my recent tools actually don't use the labs db for any wiki. [15:34:22] Mostly because it's hard to debug and run locally, but also because it makes caching easier. [15:38:47] 06Labs, 10Tool-Labs, 13Patch-For-Review: HBA not configured correctly for tools-bastion-02 - https://phabricator.wikimedia.org/T104613#2421633 (10valhallasw) Not a key issue, but an issue with `/etc/shosts.equiv`. On tools-bastion-02, the file contains ``` tools-bastion-01.eqiad.wmflabs tools-bastion-01.too... [16:11:03] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2419652 (10valhallasw) `tools-db` is a deprecated alias which is specified in `/etc/hosts`, and I think the containers only use DNS. `tools.labsdb` is the stable name of the host. (... [16:12:53] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421716 (10Dvorapa) Just one query completed in 2 hours: {F4221592} [16:24:02] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421318 (10yuvipanda) I just restarted the runner, hopefully that'll make things better. [16:27:14] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2421751 (10yuvipanda) @Magnus do you think you can switch to tools.labsdb and try? I think we'd like to not modify /etc/hosts inside the containers since that complicates things a f... [16:33:47] Luke081515 am looking into it (quarry) [16:39:39] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421780 (10Dvorapa) Looks like working, thank you [16:44:24] yuvipanda: thanks :) [16:49:36] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421815 (10yuvipanda) I ran: ``` MariaDB [quarry]> UPDATE query join query_revision on query.latest_rev_id = query_revision.id join query_run on latest_run_id = query_run.id SET status=1 where (status = 2 or stat... [17:44:39] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2422125 (10yuvipanda) Actually, I'm just going to make tools-db work - this isn't going to be the only tool running into this issue. [17:46:12] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Move tools-db and tools-redis into DNS - https://phabricator.wikimedia.org/T139190#2422128 (10yuvipanda) [17:59:20] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2422263 (10yuvipanda) @Magnus ok, 'tools-db' works now. I moved wikidata-todo over again and verified the link you provided works fine. I've moved it back to gridengine just now tho... [18:01:53] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Move tools-db and tools-redis into DNS - https://phabricator.wikimedia.org/T139190#2422264 (10yuvipanda) Ok, right now it's implemented with a hacky lua script that requires a puppet change to change, but ideally should be implemented as: 1. We a... [18:04:58] 10Quarry: Add CORS or JSONP support to /query/:id/result/latest/0/json endpoint - https://phabricator.wikimedia.org/T139197#2422268 (10MusikAnimal) [18:17:08] 06Labs, 10Tool-Labs: tools-exec-1216 down - https://phabricator.wikimedia.org/T138787#2422290 (10valhallasw) [18:17:10] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2422289 (10valhallasw) [18:37:09] 10Quarry, 13Patch-For-Review: Add CORS or JSONP support to /query/:id/result/latest/0/json endpoint - https://phabricator.wikimedia.org/T139197#2422354 (10yuvipanda) @MusikAnimal can you try now? [18:44:43] (03CR) 10Legoktm: [C: 032] Make RelEng projects sub-project proof [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/293663 (https://phabricator.wikimedia.org/T137494) (owner: 10Greg Grossmeier) [18:45:10] (03CR) 10jenkins-bot: [V: 04-1] Make RelEng projects sub-project proof [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/293663 (https://phabricator.wikimedia.org/T137494) (owner: 10Greg Grossmeier) [18:46:54] 10Quarry, 13Patch-For-Review: Add CORS or JSONP support to /query/:id/result/latest/0/json endpoint - https://phabricator.wikimedia.org/T139197#2422392 (10yuvipanda) Still doesn't work because there's a https -> http redirect in there for some reason. [19:05:22] 10Quarry, 13Patch-For-Review: Add CORS or JSONP support to /query/:id/result/latest/0/json endpoint - https://phabricator.wikimedia.org/T139197#2422464 (10MusikAnimal) Working on my end! Many thanks Yuvi! :) [19:24:16] 10Labs-project-wikistats: new possible wikifarms / hives detected - check for lists - https://phabricator.wikimedia.org/T38570#2422505 (10RobiH) Another hive detected (1000+ entries): http://meta.miraheze.org/wiki/Special:SiteMatrix [19:56:10] 10Labs-project-wikistats: new possible wikifarms / hives detected - check for lists - https://phabricator.wikimedia.org/T38570#2422555 (10Dzahn) miraheze already exists http://wikistats.wmflabs.org/display.php?t=mh but needs update.. only 82 entries [20:25:44] 06Labs, 10Tool-Labs, 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#2422617 (10valhallasw) I'm abusing this task to jot down notes, as I tend to lose my paper notebooks ;-) * toolsbeta-valhallasw-puppet-compiler is still alive * `/var/lib/pu... [21:25:21] PROBLEM - Host tools-exec-1216 is DOWN: CRITICAL - Host Unreachable (10.68.17.255) [22:46:11] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2422921 (10Magnus) Yes, this seems to work! I also switched most of my tools to use "tools.labsdb" (//this// is when sharing code across tools comes in handy!). Thanks! [23:17:01] 10Labs-project-wikistats: new possible wikifarms / hives detected - check for lists - https://phabricator.wikimedia.org/T38570#2423075 (10Dzahn) @Robih updated list of miraheze wikis (in existing table), with import script. i get 612 wikis. dont know why their meta page shows more.. 639 total in DB, maximum di... [23:45:20] http://bots.wmflabs.org/dump/%23wikimedia-labs.htm [23:45:20] @info gsoc