[00:11:16] 10cloud-services-team, 10wikitech.wikimedia.org: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3659279 (10Andrew) [00:23:17] 10cloud-services-team, 10wikitech.wikimedia.org: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3659279 (10Quiddity) The only consistent pattern I can see is related to page-moves. For these 3 examples, the page-histories stop at `wikitech-static` just befo... [00:39:31] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:19:33] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [01:26:41] 10Cloud-VPS (Project-requests), 10User-bd808: Request creation of IIAB VPS project - https://phabricator.wikimedia.org/T176926#3659441 (10Tim-moody) I have downloaded a 9G zip file that contains content, but do not have sufficient storage to unzip it. Please double the storage quota to 40G. Thanks. [01:54:41] 10Cloud-VPS (Project-requests), 10User-bd808: Request creation of IIAB VPS project - https://phabricator.wikimedia.org/T176926#3641585 (10zhuyifei1999) >>! In T176926#3659441, @Tim-moody wrote: > I have downloaded a 9G zip file that contains content, but do not have sufficient storage to unzip it. Please doubl... [02:42:00] (03PS1) 10BryanDavis: Only allow RFC 1035 DNS_LABEL compatible tool names [labs/striker] - 10https://gerrit.wikimedia.org/r/382346 (https://phabricator.wikimedia.org/T176681) [04:19:46] (03CR) 10jenkins-bot: Update composer.json to use MW_CodeSniffer and fix detected issues [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/380355 (https://phabricator.wikimedia.org/T176635) (owner: 10MarcoAurelio) [05:28:44] PROBLEM - Puppet errors on tools-exec-1408 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:03:43] RECOVERY - Puppet errors on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:28] 10Cloud-Services, 10cloud-services-team (Kanban): Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10Legoktm) I looked through the past week's IRC logs, shinken-wm is by far the loudest and I think moving it to somewhere else would be a good idea. I think t... [09:12:30] 10Cloud-Services, 10cloud-services-team (Kanban): Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10Addshore) +1 We did this for #wikidata and moved the bots into #wikidata-feed [09:44:11] 10Tool-Attribution-Generator, 10TCB-Team: Picture is not recognized because of varying license template - https://phabricator.wikimedia.org/T177465#3659929 (10Katja_Ullrich_WMDE) [09:46:43] 10Cloud-VPS, 10LDAP: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#3659944 (10Liuxinyu970226) [09:47:21] 10Cloud-Services, 10LDAP: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#662340 (10Liuxinyu970226) [09:57:49] 10Cloud-Services, 10cloud-services-team (FY2017-18), 10Documentation, 10Goal: Form a WMCS Documentation Special Interest Group - https://phabricator.wikimedia.org/T177123#3647887 (10Qgil) A duplicate of {T156301}? [10:01:02] 10cloud-services-team (FY2017-18), 10Community-Liaisons (Oct-Dec 2017): Community Liaison support for Cloud services rebranding task - https://phabricator.wikimedia.org/T174458#3659969 (10Qgil) [11:52:56] (03PS3) 10Jean-Frédéric: Use a virtualenv for Python dependencies [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/379751 (https://phabricator.wikimedia.org/T176465) [12:10:21] (03PS4) 10Jean-Frédéric: Use a virtualenv for Python dependencies [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/379751 (https://phabricator.wikimedia.org/T176465) [12:12:23] (03CR) 10Jean-Frédéric: [C: 032] Use a virtualenv for Python dependencies (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/379751 (https://phabricator.wikimedia.org/T176465) (owner: 10Jean-Frédéric) [12:13:39] (03Merged) 10jenkins-bot: Use a virtualenv for Python dependencies [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/379751 (https://phabricator.wikimedia.org/T176465) (owner: 10Jean-Frédéric) [12:14:49] (03CR) 10jenkins-bot: Use a virtualenv for Python dependencies [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/379751 (https://phabricator.wikimedia.org/T176465) (owner: 10Jean-Frédéric) [12:21:47] 10Cloud-Services, 10Cloud-VPS, 10Toolforge, 10User-fgiunchedi: Rollout prometheus-node-exporter 0.14 in labs - https://phabricator.wikimedia.org/T166561#3300186 (10fgiunchedi) a:03fgiunchedi I'll take care of this since we'll need some new collectors from node-exporter 0.14 as part of T177196 [12:29:17] 10cloud-services-team, 10wikitech.wikimedia.org: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3660435 (10Andrew) Wikitech is dumped using /usr/local/bin/mwscript maintenance/dumpBackup.php labswiki --current --uploads My guess is that --current is a bit... [12:36:22] 10Cloud-VPS (Project-requests), 10User-bd808: Request creation of IIAB VPS project - https://phabricator.wikimedia.org/T176926#3660443 (10chasemp) @zhuyifei1999 said what I would suggest. Storage is not quota'd in the way you are imagining. Instances are created with a "flavor" and that has a certain amount... [12:45:14] 10Cloud-Services, 10cloud-services-team (Kanban): Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10Multichill) Sounds like a good plan to move the noise to #wikimedia-cloud-feed , worked well for the Wikidata channel . You might want to consider to still... [12:46:17] 10Cloud-Services, 10DBA: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3611967 (10Marostegui) I saw this database was created on today's SWAT - going to sanitize it on the sanitarium hosts before handing it over to #cloud-services-team once SWAT is done [12:46:27] 10Cloud-Services, 10DBA: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3660492 (10Marostegui) p:05Triage>03Normal [13:52:42] FYI I'll be upgrading prometheus-node-exporter in labs with cumin, I ran a few test upgrades on some projects and it looks good [13:54:18] andrewbogott chasemp madhuvishy bd808 ^ [13:54:30] kk [13:54:53] sounds good [14:02:21] Is there a reason tools-bastion-03 is so slow? [14:05:41] tools.dbreps@tools-bastion-03:~$ time ls -la --> real 0m10.017s [14:06:01] Seems intermittent, but still pretty annoying. [14:26:37] hui! the new replicas are fast 😃) [14:26:46] * Steinsplitter hands bd808 et all. a cookie. [14:28:32] Esther: in general that reason is other running things locally vs on the grid, which is a hard problem honestly [14:29:18] I don't see anything happening there that would slow things down, maybe it's over? [14:32:18] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3660861 (10Andrew) 05Open>03Resolved rabbit is now much quieter, so this is /maybe/ bet... [14:33:04] 10cloud-services-team (Kanban): CamelCase vs. VPS instance naming - https://phabricator.wikimedia.org/T176757#3660863 (10Andrew) Adding a regex validation to the instance name in Horizon turns out to be non-trivial in the current version. [14:33:54] 10Cloud-VPS (Project-requests): Request creation of MWStake VPS project - https://phabricator.wikimedia.org/T177500#3660874 (10MarkAHershberger) [14:34:02] 10Cloud-Services, 10Operations, 10Patch-For-Review: rack/setup/install labweb100[12].wikimedia.org - https://phabricator.wikimedia.org/T167820#3660887 (10Andrew) 05Open>03Resolved These boxes are up and installed and seem ok. Actual service implementation is T168470 [14:43:07] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1428 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:45:48] 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3660921 (10Andrew) Is there anything I can do to nudge this along, short of 'clone Jaime'? [14:59:15] 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3660988 (10jcrespo) I think the first thing is to amend the description so that cloud (in particular), or anyone else agrees with the... [15:18:08] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [15:28:37] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:33:40] chasemp, andrewbogott: It was happening last night as well. [15:33:44] 10Cloud-Services, 10Cloud-VPS, 10Toolforge, 10User-fgiunchedi: Rollout prometheus-node-exporter 0.14 in labs - https://phabricator.wikimedia.org/T166561#3661199 (10fgiunchedi) 05Open>03Resolved All done! I've ran the upgrade with cumin, using the command below (see also https://wikitech.wikimedia.org/... [15:34:28] chasemp, andrewbogott: I went for a jog, came back, just now: https://p.defau.lt/?4XBFx_PeMXhMdWGSOmpzPg [15:34:40] Like... something is off there. NFS slowness? Idk. [15:35:53] chasemp, andrewbogott: Someone was complaining that database reports stopped updated. So I became "dbreps" and saw access denied errors. [15:36:06] I guess .my.cnf credentials went invalid, but replica.my.cnf credentials were still fine. [15:36:26] I was able to edit ~/.my.cnf, but the file permissions were still tripping me up there with other things. [15:36:28] Esther: there is no my.cnf that is managed by us, only replica.my.cnf [15:36:44] some people have copied the file, the best measure is a symlink [15:37:03] chasemp: Do you understand https://p.defau.lt/?Ed9i6LhjTZFUqgjxNVs6mA ? [15:37:14] Is replica.my.cnf special in some way? Like marked immutable or something? [15:37:19] yes, replica.my.cnf cannot be modified [15:37:20] yes [15:37:34] I've never understood that file name. [15:37:42] Since .my.cnf seems way more standard. [15:37:48] honestly no clue, it predates most of us [15:37:52] naming wise [15:37:59] Yeah, and /usr/bin/sql reads it first. [15:38:08] Not sure why the old credentials went bad. I guess due to new replicas? [15:38:25] Anyway, all that is mostly resolved. But the slowness is... a lot. [15:38:33] probably teh old file was a copy and we do fixup creds is needed so making a point in time copy of replica.my.cnf is rarely a good idea [15:38:36] eventually it will be stale [15:38:36] etc [15:38:47] s/is/needed/as needed [15:38:54] Yeah. [15:39:00] but credentials are the same on all labsdb things fyi [15:40:48] Is tools-bastion-03 NFS? [15:40:50] I feel like this is slow hardware or something. [15:41:09] Or is there a better bastion host I could use? :-) [15:41:30] Esther: for tools bastion we have quotas on NFS but it is per server and so the bastion model can cause large usage to clobber other users. There are long term thoughts on addressing it but it's not a simple issue. The best we can do for now is address issues as they come and we do assign certain utilities to limited cgroups to triage possible damage. [15:41:36] Esther: tools-bastion-02 does exist [15:41:44] if 03 is really hung up I can reboot [15:41:55] It's responsive, just slow. [15:42:01] I'm a bit surprised nobody else has noticed or is complaining? [15:42:19] Like the few seconds lag for every command (or even tab-completion) slowly gets aggravating. [15:42:22] whatever this is [15:42:23] 8212 be/6 tools.gp 4.01 M 4.84 M 0.00 % 65.15 % python ./gps_exif_bot2.py [15:45:44] oops apparently it did not wait 5m [15:45:54] that's my bad [15:46:17] !log tools-bastion-03 has tons of local tools running long lived NFS intensive processes. I'm rebooting rather than playing whackamole. [15:46:18] chasemp: Unknown project "tools-bastion-03" [15:46:21] !log tools tools-bastion-03 has tons of local tools running long lived NFS intensive processes. I'm rebooting rather than playing whackamole. [15:46:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:46:32] Esther: see if it comes up more sane for you [15:46:43] sorry there was little warning on that reboot, it was my mistake [15:56:46] 10cloud-services-team, 10wikitech.wikimedia.org: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3661313 (10Andrew) A --current dump is 8.6M, a --full dump is 7.2G. So doing --full may not be practical. [16:03:04] 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3661348 (10Andrew) [16:03:37] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:17] 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: Move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3661387 (10bd808) p:05Triage>03Normal [16:09:19] 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: Move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3351594 (10bd808) >>! In T167973#3660988, @jcrespo wrote: > I think the first thing is to amend the description so that cloud (in particular), or anyone... [16:12:38] 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: Move wikitech and labstestwiki to s3 - https://phabricator.wikimedia.org/T167973#3661408 (10jcrespo) > Is there anything I can do to nudge this along, short of 'clone Jaime'? Also don't underestimate the amount of time that you can h... [16:14:13] 10Cloud-Services, 10Operations, 10Patch-For-Review: rack/setup/install labweb100[12].wikimedia.org - https://phabricator.wikimedia.org/T167820#3661416 (10bd808) [16:14:15] 10Cloud-Services, 10cloud-services-team (Kanban): Setup wikitech, horizon, and striker on new labweb hardware - https://phabricator.wikimedia.org/T168470#3661415 (10bd808) [16:15:09] 10Cloud-Services, 10cloud-services-team (Kanban): Setup wikitech, horizon, and striker on new labweb hardware - https://phabricator.wikimedia.org/T168470#3365393 (10bd808) [16:56:04] 10Cloud-Services, 10cloud-services-team (Kanban): Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10chasemp) I personally am for it. I don't mind the idea of unbreak now still showing up via wikibugs. [16:56:39] 10Cloud-Services, 10cloud-services-team (FY2017-18), 10Documentation, 10Goal: Form a WMCS Documentation Special Interest Group - https://phabricator.wikimedia.org/T177123#3661642 (10Quiddity) >>! In T177123#3659960, @Qgil wrote: > A duplicate of {T156301}? Nope, this task (and proposed SIG) is narrowly fo... [17:00:03] 10Cloud-Services, 10wikitech.wikimedia.org, 10Operations, 10HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#1278203 (10Krinkle) ... or not. {nav name=T176370 Migrate to PHP 7 in WMF production, href=https://phabricator.wikimedia.org/T176370} [17:09:48] 10cloud-services-team, 10wikitech.wikimedia.org: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3659279 (10bd808) The `--current` flag ends up adding this condition to the query to gather content: ```lang=php $join['revision'] = [ 'INNER JOIN', 'page_id=rev... [17:29:16] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Mridu_Bhatnagar - https://phabricator.wikimedia.org/T176018#3661787 (10Mridu_Bhatnagar) >>! In T176018#3658766, @bd808 wrote: > @Mridu_Bhatnagar Your gerrit patch looks good. I think we can call this microtask done.... [17:31:27] I get sometimes an 503 error while requesting data via the API [17:31:39] is there someone who can help me? [17:33:30] Freddy2001: that sounds like rate limiting [17:33:41] are you logging the responses? [17:34:35] yes i logged them to find the problem [17:34:55] I will paste you the response [17:36:22] https://pastebin.com/rmWPBbg0 [17:37:18] bd808, this is response from the server for an api request [17:37:24] ok. that's a Varnish level 503. There is a little bit about this at https://www.mediawiki.org/wiki/Manual:Maxlag_parameter [17:38:40] Varnish sits between all requests and the backend MediaWiki servers. A 503 from Varnish means that the request from Varnish to MediaWiki timed out. [17:39:47] my framework uses the maxlag parameter, it is currently set to 15 [17:40:16] When you get one of these the best way for your code to handle it is to wait 5-10 seconds and then try again. If you get another 503 on the retry then you should just move on. [17:41:15] Is there a particular type of Action API request you are making that gets these Varnish 503 respoonses? [17:41:30] Like parsing old revisions or something? [17:42:35] action=query&prop=info [17:44:16] I am just trying to get all the lengths of linked articles in a particular category [17:45:17] Unfortunately, I need to use the API because the text is not included in the SQL databases on toolforge [17:46:39] revision.rev_len should have the page length [17:47:08] the full wikitext is not in the wiki replicas, but this metadata should be [17:47:22] sure, but not all the links that are on a page [17:47:38] for this part I need to get the Wikitext from the API [17:48:39] ah, so you get the wikitext for a page, find outbound links, and then look up the page length for each of them with a prop=info request? [17:48:59] hmm, another thing an elasticsearch cluster in labs could probably answer ;) [17:49:13] ebernhardson: preach brother :) [17:49:22] we'll get it eventually [17:50:16] if you only need the outbound links as known by the wiki, (e.g. Special:WhatLinksHere) you might be able to use the link tables in the wiki replicas on tool forge. [17:50:19] Freddy2001: are the action=query&prop=info that fail for a single page or a long list of titles? [17:50:40] e.g. one might not have to parse the wikitext manually. [17:51:29] When you get a Varnish 503 response it means that you either had bad luck or that what you are asking the Action API to do is very hard. [17:51:30] I can request sometimes 10, sometimes up to 1000 titles, but then it fails for every request [17:51:49] 1000 is too many for sure [17:52:14] to at least skip parsing, i suppose you could hit urls like https://en.wikipedia.org/wiki/Kennedy?action=cirrusdump This has the raw cirrus documents including a json list of outgoing links, and a text_bytes field (thats wikitext post-parsing and stripping html though) [17:52:18] 10-100 should be fine I would think [17:52:19] Krinkle, I dont need WhatLinksHere, I would need a WhatLinksFromHere [17:52:20] but you can only request one at a time [17:52:38] Freddy2001: Yes, you can query in both directions when you use SQL. [17:53:18] bd808, I am going to get infos of more than 10,000 titles [17:53:47] sure, but you can do that in chunks [17:53:59] https://quarry.wmflabs.org/query/21981 Links from enwiki Main_page [17:53:59] request by request [17:55:04] I send a api query for every single page [17:55:22] 10Toolforge, 10Outreachy (Round-15), 10Patch-For-Review: Outreachy - webservice microtask for Mridu_Bhatnagar - https://phabricator.wikimedia.org/T176018#3661831 (10Mridu_Bhatnagar) >>! In T176018#3658998, @srishakatux wrote: > @Mridu_Bhatnagar Just making sure.. that you've checked whether or not you meet t... [17:59:21] Freddy2001: here's a query that builds on Krinkle's example by adding in the page size -- https://quarry.wmflabs.org/query/21982 [17:59:41] err... nope. [17:59:46] https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=info&titles=Main+Page&generator=links&utf8=1&formatversion=2 [17:59:46] that needs work [18:00:43] I also do need pages which do not exist [18:06:40] yeryry: ha! So the links generator is exactly what is needed [18:06:48] there are so many things already solved in the Action API [18:07:07] Freddy2001: did you check out that sandbox example? [18:07:27] Freddy2001: btw, 'pl_from_namespace = page_namespace' is obsolete, sorry about that. query will be better without [18:09:08] bd808 i am looking at the sandbox example but i am not sure if that is what i need [18:10:56] https://www.mediawiki.org/wiki/Special:MyLanguage/API:Links [18:11:07] "Gets a list of all links on the provided pages." [18:11:40] 10Cloud-Services, 10Documentation: Update documentation for the various phab Cloud tags - https://phabricator.wikimedia.org/T129043#3661864 (10Quiddity) 05Open>03Resolved These are all essentially done -the descriptions are added at the individual project-descriptions and in the list at https://phabricator... [18:17:30] expects an icinga/shinken RECOVERY for this one now: PROBLEM - puppet on jenkins-slave-01.git.eqiad.wmflabs is CRITICAL: [18:17:37] paladox: merged your change [18:17:55] should work on stretch now, wanna run the agent [18:18:18] re: contint: Make php.pp compatible with stretch https://gerrit.wikimedia.org/r/#/c/361680/ [18:19:27] bd808 yes but this part of my script runs without any problem [18:19:42] I need to check the links [18:19:55] Thats the problematic part [18:20:14] 10Cloud-Services, 10wikitech.wikimedia.org, 10Operations, 10HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#3661882 (10bd808) >>! In T98813#3661663, @Krinkle wrote: > ... or not. {nav name=T176370 Migrate to PHP 7 in WMF production, href=https://phabricator.wikimedia.org/T17... [18:21:05] Freddy2001: that query is calling prop=info on each link, but it doing it with a generator which will split things up into reasonable batch sizes [18:21:21] For this, i use prop=info to get the pageid and if it is not null, i get the size of the page in the response else this page does not exist [18:21:35] That the thing i am doing at the moment, but it causes the 503 [18:22:16] but I'm not 100% trying to convince you to rewrite your code. The original problem is like I said either bad luck for some requests or that you are asking for more than MediaWiki can do in the response time limit [18:22:57] so you need to go slower, ask for less at a time, and be prepared to deal with occasional errors [18:23:31] there is no way to guarantee that 100% of your requests will be successful [18:24:32] what is the response limit of mediawiki? [18:24:40] Using generator=links will be faster for sure [18:25:56] that is a good question. looking for an answer [18:25:59] Using generator=links will be faster for sure ← but it does not show red links? [18:26:50] mutante thanks [18:26:55] i've been running the patch anyways [18:27:26] Freddy2001: that is true I think. it uses the pagelinks table like Krinkle's query does so I don't think it will see redlinks [18:28:00] pagelink table records all links. Regardless of blue or red. [18:28:08] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:28:29] To get only blue ones, you'd have to use a second join on page to see whether the join produces a non-null page_id for that namespace/title combination. [18:28:35] Which is how WantedPages works, for example. [18:28:52] It does the opposite and only keeps the one with a null result from the LEFT JOIN (I think) [18:31:50] https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=info&titles=Beijing+Beijing%2C+I+Love+Beijing&generator=links&utf8=1&formatversion=2&gpllimit=max for an example with a redlink [18:38:37] thats what i wanted [18:39:04] is there a limit (like 500 links or so) for this function? [18:40:30] There's a limit, but that's per-page... If you just keep requesting more pages, it'll keep going until you've got them all. [18:40:59] great, thank you all very much [18:41:08] you helped me a lot [18:43:23] https://www.mediawiki.org/wiki/API:Query#Continuing_queries but if you're using an existing library hopefully that does it for you. [18:55:11] !log ores staged ores-wmflabs-deploy:b7c5337 [18:55:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:56:30] !log ores deployed ores-wmflabs-deploy:b7c5337 [18:56:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:59:42] 10cloud-services-team, 10wikitech.wikimedia.org: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3661992 (10bd808) `MWUnknownContentModelException from line 306 of /srv/mediawiki/w/includes/content/ContentHandler.php: The content model 'yaml' is not register... [19:03:10] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [19:13:28] !log ores deployed ores-wmflabs-deploy:5b61eec (rollback) [19:13:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [19:14:58] Keegan, added a sentence to your draft. Otherwise looks good. [19:15:04] Woops wrong channel [19:15:16] halfak: np [19:15:19] :) [19:15:19] thanks! [19:34:03] 10Cloud-Services, 10cloud-services-team (Kanban): Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10mmodell) +1 to what everyone else has already said. I generally find the monitoring stuff to be distracting to the point of ruining a channel for actual dis... [20:11:59] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3662256 (10Andrew) I've directed shinken-wm to talk in #wikimedia-cloud-feed. [20:28:16] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10bd808) Basic channel protections are in place for the new channel. ```lang=irc [20:26] < bd808> info #wikimedia-cloud-feed [20:26]... [20:30:52] !log tools.stashbot Restarted to join #wikimedia-cloud-feed [20:32:09] 10cloud-services-team, 10wikitech.wikimedia.org, 10Patch-For-Review: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3662294 (10Andrew) I ran the export and import by hand just now, and I think we're getting the complete wiki. [20:33:15] !log tools.stashbot Restarted to join #wikimedia-cloud-feed [20:33:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [20:33:26] ohi stashbot [20:38:06] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org: Not all content is getting replicated to wikitech-static - https://phabricator.wikimedia.org/T177450#3662315 (10bd808) 05Open>03Resolved Looks good to me. The root problem was 2 missing content handlers. One comes from OpenStackManager which we don'... [21:23:38] chasemp: Seems a lot snappier now, thanks! [21:36:45] 10Toolforge, 10cloud-services-team (Kanban), 10User-bd808: tools-mail queue length alert from prometheus cron - https://phabricator.wikimedia.org/T177484#3662504 (10bd808) [21:39:23] 10cloud-services-team (Kanban), 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3662516 (10bd808) @Cmjohnson what's our next step here? Do we have enough info to request additional replacement parts from Dell? This poor box has been a bit of a dud since... [21:40:43] (03PS2) 10BryanDavis: Only allow RFC 1035 DNS_LABEL compatible tool names [labs/striker] - 10https://gerrit.wikimedia.org/r/382346 (https://phabricator.wikimedia.org/T176681) [21:48:37] (03PS3) 10BryanDavis: Only allow RFC 1035 DNS_LABEL compatible tool names [labs/striker] - 10https://gerrit.wikimedia.org/r/382346 (https://phabricator.wikimedia.org/T176681) [21:52:38] (03CR) 10BryanDavis: [V: 031 C: 032] "Tested on https://striker.wmflabs.org/" [labs/striker] - 10https://gerrit.wikimedia.org/r/382346 (https://phabricator.wikimedia.org/T176681) (owner: 10BryanDavis) [21:54:03] (03Merged) 10jenkins-bot: Only allow RFC 1035 DNS_LABEL compatible tool names [labs/striker] - 10https://gerrit.wikimedia.org/r/382346 (https://phabricator.wikimedia.org/T176681) (owner: 10BryanDavis) [21:55:21] (03PS1) 10BryanDavis: Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/382613 [21:55:39] 10Tools: SVGCheck tool unresponsive - https://phabricator.wikimedia.org/T177125#3662609 (10Offnfopt) Just noting the tool is still unresponsive. What is the best way to get a sysadmin to take a look at this? [21:55:41] (03CR) 10BryanDavis: [C: 032] Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/382613 (owner: 10BryanDavis) [21:55:47] (03Merged) 10jenkins-bot: Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/382613 (owner: 10BryanDavis) [22:06:11] 10Toolforge, 10Kubernetes: namespaces "wdq_checker" not found error when trying to start webservice shell - https://phabricator.wikimedia.org/T176027#3662661 (10bd808) [22:06:13] 10Striker, 10cloud-services-team (Kanban), 10User-bd808: Striker should not allow tool names to include '_' for Kubernetes compatibility - https://phabricator.wikimedia.org/T176681#3662658 (10bd808) 05Open>03Resolved a:03bd808 [22:08:15] 10Toolforge: Tool naming convention - https://phabricator.wikimedia.org/T177439#3658679 (10bd808) toolsadmin has been updated to prevent invalid names, but having a note about the required naming scheme still seems like a good idea. The description now given when the name is invalid is `Must start with a-z, end... [22:09:12] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3662674 (10hashar) Did you have a chance to look at the `notifications.error` messages? The... [22:11:42] (03PS2) 10BryanDavis: Enhance workflow for tool labs membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/374104 (https://phabricator.wikimedia.org/T168027) [22:12:33] (03CR) 10BryanDavis: [C: 032] Add rewritten crontab in Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999) [22:12:43] (03CR) 10BryanDavis: [C: 032] Convert list-user-databases to python3 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381385 (owner: 10BryanDavis) [22:12:44] (03CR) 10BryanDavis: [C: 032] Convert jsub to python3 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381386 (owner: 10BryanDavis) [22:12:46] (03CR) 10BryanDavis: [C: 032] Remove log-command-invocation calls [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381619 (https://phabricator.wikimedia.org/T166712) (owner: 10BryanDavis) [22:12:49] (03CR) 10BryanDavis: [C: 032] Toolforge rebranding [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381620 (https://phabricator.wikimedia.org/T168480) (owner: 10BryanDavis) [22:12:54] (03CR) 10BryanDavis: [C: 032] Port /usr/bin/job to Python3 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381621 (owner: 10BryanDavis) [22:13:06] (03CR) 10BryanDavis: [C: 032] Changelog bump for v1.23 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381622 (owner: 10BryanDavis) [22:13:54] (03Merged) 10jenkins-bot: Add rewritten crontab in Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999) [22:13:56] (03Merged) 10jenkins-bot: Convert list-user-databases to python3 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381385 (owner: 10BryanDavis) [22:14:15] (03Merged) 10jenkins-bot: Convert jsub to python3 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381386 (owner: 10BryanDavis) [22:14:15] (03Merged) 10jenkins-bot: Remove log-command-invocation calls [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381619 (https://phabricator.wikimedia.org/T166712) (owner: 10BryanDavis) [22:14:17] (03Merged) 10jenkins-bot: Toolforge rebranding [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381620 (https://phabricator.wikimedia.org/T168480) (owner: 10BryanDavis) [22:14:40] (03Merged) 10jenkins-bot: Port /usr/bin/job to Python3 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381621 (owner: 10BryanDavis) [22:15:00] (03Merged) 10jenkins-bot: Changelog bump for v1.23 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/381622 (owner: 10BryanDavis) [22:26:37] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3662741 (10bd808) >>! In T170492#3662674, @hashar wrote: > Did you have a chance to look at... [22:42:31] !log tools Updated jobutils to 1.23 [22:42:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:47:14] !log tools Updated misctools to 1.23 [22:47:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:48:07] 10cloud-services-team (Kanban), 10Analytics, 10Patch-For-Review, 10User-bd808: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3662768 (10Nuria) @bd808: sounds good, let us know when calls are no loger coming in and we... [23:04:12] 10Data-Services, 10cloud-services-team (FY2017-18), 10Goal: Program 7 Outcome 3: data services - https://phabricator.wikimedia.org/T166402#3662849 (10Krinkle) [23:09:20] bd808: https://phabricator.wikimedia.org/T177299 [23:09:57] Krinkle: I saw it, but it came in about 2 hours after our weekly meeting where we triage these [23:10:42] is it direly urgent? [23:15:03] bd808: no, a week is fine :) - just wanna help with the migration. [23:15:29] cool :) I'll be glad to see it happen [23:17:02] * harej is looking forward to dropping a 25 GB database from Toolforge [23:21:01] (03PS1) 10BryanDavis: crontab: Fix python3 syntax problems [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382629 (https://phabricator.wikimedia.org/T156174) [23:23:31] (03CR) 10BryanDavis: [C: 032] crontab: Fix python3 syntax problems [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382629 (https://phabricator.wikimedia.org/T156174) (owner: 10BryanDavis) [23:24:14] (03Merged) 10jenkins-bot: crontab: Fix python3 syntax problems [labs/toollabs] - 10https://gerrit.wikimedia.org/r/382629 (https://phabricator.wikimedia.org/T156174) (owner: 10BryanDavis)