[00:07:54] 10netops, 06Operations, 10ops-eqiad: Rack and setup new eqiad row D switch stack (EX4300/QFX5100) - https://phabricator.wikimedia.org/T148506#2761935 (10faidon) [00:31:39] 10netops, 06Operations, 10ops-eqiad: Rack and setup new eqiad row D switch stack (EX4300/QFX5100) - https://phabricator.wikimedia.org/T148506#2762011 (10faidon) [00:51:21] 10netops, 06Operations, 10ops-eqiad: Rack and setup new eqiad row D switch stack (EX4300/QFX5100) - https://phabricator.wikimedia.org/T148506#2762043 (10faidon) a:05Cmjohnson>03faidon So, the following things happened today (and not all as smoothly or necessarily in the order listed here, a lot of interm... [00:56:53] 10netops, 06Operations, 10ops-eqiad: Rack and setup new eqiad row D switch stack (EX4300/QFX5100) - https://phabricator.wikimedia.org/T148506#2762054 (10faidon) a:05faidon>03Cmjohnson The following things are now pending from @Cmjohnson: - The link between D 2 and D 5 does not seem to be working — is see... [04:07:00] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Use content hash based image / thumb URLs & define an official thumb API - https://phabricator.wikimedia.org/T66214#2762226 (10bearND) In addition to that I'd like to know what the relationship between this task and the Thumbo... [07:49:49] ema, bblack: Debian stretch will have 1.1 after all: https://lists.debian.org/debian-devel-announce/2016/11/msg00001.html [07:50:48] there are a few exceptions that won't be portable in time (like qt which implements it's own QSSL crypto wrapper around openssl), but unlikely to be relevant [08:14:34] the approach used is the other way round compared to ours: [08:14:51] openssl1.0 source package: https://lists.debian.org/debian-devel-changes/2016/11/msg00192.html [08:15:12] openssl source package: https://lists.debian.org/debian-devel-changes/2016/11/msg00191.html [08:54:41] 10Traffic, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412#2762438 (10elukey) This is probably due to: https://github.com/openssl/openssl/issues/1799 [09:02:17] 10Traffic, 06Analytics-Kanban, 06Operations: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412#2762540 (10elukey) [09:30:58] 10Traffic, 06Operations: varnish-backend: weekly cron restart for all clusters - https://phabricator.wikimedia.org/T149784#2762582 (10ema) [09:31:11] 10Traffic, 06Operations: varnish-backend: weekly cron restart for all clusters - https://phabricator.wikimedia.org/T149784#2762597 (10ema) p:05Triage>03Normal [09:40:44] bblack: I also left a comment at the Debian nginx/openssl 1.1 bug (since now with openssl 1.1 uploaded, this will also hit Debian unstable): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=828453 [09:41:23] 10Traffic, 06Operations: varnish-backend: weekly cron restart for all clusters - https://phabricator.wikimedia.org/T149784#2762622 (10ema) a:03ema [10:40:28] 10Domains, 10Traffic, 10DNS, 06Operations, and 2 others: Point wikipedia.in to 205.147.101.160 instead of URL forward - https://phabricator.wikimedia.org/T144508#2762761 (10Naveenpf) >>! In T144508#2739847, @CRoslof wrote: > If @Naveenpf or #operations is still interested in pursuing this task's specific r... [11:22:27] moritzm: upstream has a pullreq to fix that bug now. I'm not entirely sure (at first glance anyways) if it really fixes all the read_ahead issues though :) [11:24:45] (because it sounds specific to the testsuite repro that resulted in an "internal error" where it failed to be able to read data, which is slightly different than the prod repro where it does successfully read data but seems to probably discard other unread data and cause a "bad mac" error) [11:25:03] so I may have to test again with the pull req applied and see if I can find a better repro of the other case or not [11:27:34] 10Traffic, 06Analytics-Kanban, 06Operations: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412#2762917 (10elukey) After a chat with @ema we decided to test a very basic use case, namely if a TCP RST from the client could cause th... [11:32:15] ah, nice, hadn't seen the followup by Matt Caswell [11:37:08] bblack: I am trying to figure out if the weird logging problem that I saw in T148412 (now apparently resolved) could be related to the openssl bug that you found. Having a repro is not that easy so I am wondering if you could alert me when you do some testing so I can try to repro my use case [11:37:09] T148412: Varnishlog with Start timestamp but no Resp one causing data consistency check alarms - https://phabricator.wikimedia.org/T148412 [11:37:42] it may be completely unrelated though [11:39:55] elukey: well it's not likely related to that particular bug, but it could be related to aborted connections in general [11:41:04] yes sure [12:09:28] moritzm: did you ever push our openssl-1.1 build up to our repro somewhere? [12:09:37] (I meant git repo) [12:11:09] ema: so... cache_text v4 deploy... [12:11:18] ema: any last thoughts on things we haven't remembered to fix? [12:13:12] bblack: I have a local git repo, but the stupid gerrit project creation in the UI was messed up and I can't push it, I need to sync up with Chad when he's back from vacation how to workaround that [12:13:49] moritzm: it's a new repo separate from operations-debs-openssl ? [12:14:34] (I had assumed it would just be new branches there) [12:14:47] bblack: mmh nothing comes to mind, no [12:15:13] ema: do you remember the whole thing about ordering v3-vs-v4 with the upload rollout? [12:15:58] I think we initially thought edge-inwards was better, and then after various v3/v4-compat bugs we went the other way around? I don't recall the specifics now offhand [12:16:36] I started a separate openssl11.git since they'll be more or less disparate, but we can also merge that into a branch if you prefer [12:17:29] 10Traffic, 06Operations: restrict upload cache access for private wikis - https://phabricator.wikimedia.org/T129839#2117602 (10BBlack) @fgiunchedi - deny all access to them via upload.wm.o? [12:17:44] moritzm: separate's fine too [12:20:15] bblack@alaxel:~/repos$ git clone https://gerrit.wikimedia.org/r/operations/debs/openssl11 [12:20:21] bblack@alaxel:~/repos/openssl11$ git log --oneline [12:20:21] 040ba53 Initial empty repository [12:20:32] bblack: we started with ulsfo, then downgraded it back to v3 [12:20:33] moritzm: seems like it's working upstream, just needs pushes? [12:20:58] because of T144257 and 503 plateaus [12:20:59] T144257: Certain images failing to load in ulsfo - https://phabricator.wikimedia.org/T144257 [12:21:28] moritzm: if gerrit created the initial empty commit, your first push to master may need to force-push to kill that commit [12:21:53] then we followed this plan: https://phabricator.wikimedia.org/T131502#2608658 [12:23:41] ema: ok. the difference is here is we can't do local active/active stuff with MW at this time [12:23:54] and the PII leak is a little more troubling in this case than in the upload case, too [12:24:29] bblack: that was exactly my problem, previously gerrit create-project created an empty repo, but the new UI approach adds that silly initial commit, will try push --force in a bit [12:25:24] may need to set up rights for that, too, let me take a peek in gerrit [12:26:36] moritzm: are you including the real log of upstream commits, or just importing all of upstream as a single local commit? [12:28:38] no, I'm importing new upstream releases via git-buildpackage's import-orig [12:28:45] ok [12:28:56] I modded the perms on the repo to allow force-push and commiter-forging, etc [12:29:06] (all the strange cases that may come up in deb packaging repos here) [12:29:58] oh we had those already inherited from operations/debs heh [12:30:00] someone was smart :) [12:32:54] ema: trying to remember if v3 fetching from v4 was much of an issue. I know we had some with v4 fetching from v3. [12:33:44] I think we had different issues in each direction perhaps [12:34:21] so, we may just have to take the PII-leak hit and do it the same way as upload [12:34:31] so long as we don't leave that state in place for too long [12:35:42] It should be enough to add a .gitreview file to my existing repo, run "git review -s" and then run "git push --force gerrit master", or am I missing something? [12:36:00] (set codfw to "direct", route ulsfo->eqiad, upgrade codfw first. if that goes ok, switch ulsfo back to codfw and upgrade it as the next step. then esams while it's routed to codfw. the eqiad and then switch all the routing back?) [12:36:44] moritzm: I wouldn't bother with "git review" at this stage (I never quite understand everything it does, and I basically stopped ever using it) [12:37:18] is your .git/config set up with gerrit as "origin" or "gerrit" (or both?) [12:38:04] I have a gerrit remote in there, yes [12:38:16] so just normal "git push -f gerrit master" should work, I think [12:38:19] I think I might have already ran "git review -s" in there [12:38:28] (I don't think .gitreview is required by gerrit) [12:38:58] geneally git review is responsible for 99% of my git-buildpackage related trouble so far :-) [12:39:09] I'll give that a shot [12:39:14] I don't even know what "git review -s" really does. as long as the repo is in a currently-clean state, the push should work fine [12:39:24] "git clean -dfx; git checkout ." [12:40:02] the cure for git-review is never use it, IMHO. you just have to learn to manually use the gerrit review branches [12:40:37] a "normal" reviewed push of commits to branch foo is: "git push gerrit HEAD:refs/for/foo", whereas "git push gerrit foo" attempts to bypass review and merge things directly. [12:41:01] and then "git push gerrit HEAD:refs/for/foo/bar" will do a reviewed push to branch foo with the "topic" set to "bar" [12:41:49] but with your initial force push, I'd skip the review part [12:42:18] (I doubt that works right anyways, force-push via review) [12:42:42] push worked fine, I verfied with a checkout of openssl11.git [12:43:09] yup, I just pulled it [12:43:41] awesome, thanks :) [12:43:42] thanks for the git-review bypass hints, will try that out [12:44:10] I'm gonna use your repo now to build a one-off package that tests the upstream bugfix and see if I can repro the "real" problem on cache_upload still or not [12:44:41] (not pushing my commits to the repo, just using it as a baseline) [12:47:15] bblack: I don't remember specific issues with v3 fetching from v4 to be fair, I'll dig through phab/irc logs after lunch to confirm [12:48:06] ok, great [12:57:03] ema: I think at the end of the process, we even switched esams->codfw before upgrading esams, just to avoid v4->v3 fetches as much as possible. but yeah, I'm not sure about v3->v4 fetches. maybe it was a lesser issue there. [16:13:00] back on the chapoly topic: we can leave the aes256/128 flip for afterwards as well and do more evaluation on that specifically. either way, the chapoly-first argument still stands on its own, and without the related aes change we'd be aligning with mozilla's server-side recommendations for servers that care about non-modern clients anyways. [16:55:34] I'm personally fine with dropping the chapoly-pref hackl, the benchmark numbers you posted a while where pretty compelling [16:59:13] 10Traffic, 06Operations: restrict upload cache access for private wikis - https://phabricator.wikimedia.org/T129839#2763978 (10fgiunchedi) @bblack correct, for private wikis MW will go through img_auth.php and from there to ms-fe IIRC. I believe we can deny access to private wikis on upload altogether. [17:03:29] openssl just merged the fix to the 1.1.0 stable branch \o/ [17:06:02] moritzm: let's plan on doing a libssl1.1_1.1.0b-1+wmf2, adding https://github.com/openssl/openssl/commit/0f6c9d73cb1e1027c67d993a669719e351c25cfc as a local patch until 1.1.0c, and removing chapoly prefhacks? [17:09:03] 10Domains, 10Traffic, 10DNS, 06Operations, and 2 others: Point wikipedia.in to 205.147.101.160 instead of URL forward - https://phabricator.wikimedia.org/T144508#2764038 (10CRoslof) >>! In T144508#2740901, @Naveenpf wrote: > @Aklapper Can you please change title to.... add new IP address ? We have changed... [17:13:33] 10Traffic, 06Operations: restrict upload cache access for private wikis - https://phabricator.wikimedia.org/T129839#2764056 (10BBlack) Ok, I see we have a list of the private wikis in puppet already in `$private_wikis` from `manifests/realm.pp`. Is it simple to transform those into upload paths to deny access... [17:46:33] bblack: sounds good, I can do that tomorrow unless you don't find the time today [18:31:33] 10Traffic, 06Operations, 10RESTBase, 06Services (doing): Restbase redirects with cors not working on Android 4 native browser - https://phabricator.wikimedia.org/T149295#2764406 (10GWicke) @pchelolo and I discussed this a bit in person, and decided to go with a RESTBase-only solution for now. Concretely, w... [19:03:39] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2764533 (10Krinkle) 05Open>03declined This works as intended. Per T85751, the canonical url for using the `action` parameter is to go throu... [19:34:15] To: root@cp4016.ulsfo.wmnet [19:34:16] Subject: Cron /usr/local/sbin/run-no-puppet /usr/local/sbin/varnish-backend-restart > /dev/null [19:34:22] /usr/lib/python2.7/dist-packages/urllib3/connection.py:264: SubjectAltNameWarning: Certificate for conf1003.eqiad.wmnet has no `subjectAltName`, falling back to check [19:34:53] is this a case of another bpo-ed server? [19:35:11] ii python-urllib3 1.12-1~bpo8+1 all HTTP library with thread-safe connection pooling for Python [19:35:15] ii tcpdump 4.7.4-1~bpo8+1 amd64 command-line network traffic analyzer [19:35:59] bblack, ema ^ [19:46:39] paravoid: yeah I think so [19:47:08] although maybe our internal certs should have SANs, too :) [19:51:33] did we ever figure out why that other server had all those bpo packages? [19:55:04] no, I just reimaged it [19:55:41] but in general, adding the experimental repo to a bunch of live servers probably wasn't a wise path to go down, without some kind of other configuration in place to help avoid accidental package installs from it [19:56:04] we're already deep in that path now, though, so we may as well finish up the v4 transition first :) [19:56:54] I know it "should" be safe, but there are many values of "safe" [19:57:25] if I log into prodserver123 and type some apt command and I'm happy with the results, but on prodserver456 (with experimental turned on), the same command also installs something experimental and starts diverging them.... [19:57:42] (and then I salt based on the first results, hitting 456) [19:58:09] this is just a particularly noteworthy example that there's no safe way to diverge one server from the set in production and expect that divergence to not snowball [20:02:51] confusingly, there are bpo packages that don't come in via experimental and such [20:03:10] it looks like all the cache nodes have bpo versions of python-six and quickstack [20:03:24] quickstack is: [20:03:25] *** 20121211-1~bpo8+1 0 [20:03:25] 1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages [20:03:53] python-six is even stranger: [20:03:55] *** 1.9.0-3~bpo8+1 0 [20:03:55] 100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages [20:03:58] 1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages [20:04:01] 100 /var/lib/dpkg/status [20:04:04] 1.8.0-1 0 [20:04:06] 500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages [20:06:25] but we've also got some others that are fairly common, like jessie-backports/main copies of tcpdump and systemtap-runtime [20:07:14] some nodes are worse than others [20:07:31] and there's a lot of random variance in which nodes have which packages [20:08:36] but given the degree to which it affects packages that are were at least at one time "ensure => latest"... maybe puppet does this occasionally on race conditions with apt-get update failures, etc [20:08:56] (upgrades to something that seems "latest" only because apt state is temporarily wrong) [20:11:51] the python ones might be some indirect dep for newer stats scripts, too, I'm not sure [20:12:46] if I ignore all of the most-common and probably most-harmless bpos that are installed: python-six|quickstack|tcpdump|systemtap-runtime|libmaxminddb [20:13:11] there's still about 9 different random cache nodes that have varying counts of other bpo packages beyond that [20:13:35] (anywhere from 1-33 other bpo packages installed) [20:17:13] I'm looking at those 9 and fixing up if it seems obvious/easy [20:21:34] bunch of python things and golang things look messed up in obvious ways on some of these hosts [20:21:46] related to testing of confd and/or salt ... [20:23:39] wireshark + related bits are common to several hosts too [20:26:17] if I remove all wireshark bpo variants from the set as well [20:26:52] then there's only 2x left, both in cache_text, that have a ton of inexplicable bpo installed [20:26:55] cp4016 and cp1055 [20:28:46] (and there was one I fixed up manually that it had an outdated bpo of kbuild installed, probably old systemtamp testing or such) [20:29:21] I'll reimage those 2x that have the significant divergences [20:30:01] the ignored ones are expected for some sane reason and/or not-dangerous to normal runtime functionality, and it would be simpler to wait until post-varnish4-completion to reimage further [20:32:52] 10Traffic, 06Operations: reimage cp4016 and cp1055 - https://phabricator.wikimedia.org/T149843#2766116 (10BBlack) [20:40:18] 10Traffic, 06Operations: reimage cp4016 and cp1055 - https://phabricator.wikimedia.org/T149843#2766116 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by bblack on neodymium.eqiad.wmnet for hosts: ``` ['cp4016.ulsfo.wmnet', 'cp1055.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimag... [21:11:55] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2766307 (10GWicke) [21:12:11] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#1874700 (10GWicke) [21:12:24] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Use content hash based image / thumb URLs - https://phabricator.wikimedia.org/T149847#2766313 (10GWicke) [21:14:22] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#2766336 (10GWicke) I have refactored this RFC to focus on the public thumb API, and moved content hash based thumb naming to a sub-task. We would like to... [21:32:27] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 14 others: Use content hash based image / thumb URLs - https://phabricator.wikimedia.org/T149847#2766398 (10GWicke) a:05brion>03None [21:57:53] bblack: I'd like to go ahead with https://gerrit.wikimedia.org/r/#/c/319295 if that looks good to you, I don't think we've seen any impact on varnish on maps/misc from the additional polling from prometheus [21:59:58] godog: I think so, yeah [22:00:19] godog: relatedly, is it really a role or not? including it from the varnish roles would be different than including it in the site.pp role statement... [22:00:27] (in terms of hiera lookup) [22:03:01] ah, it is a role in the sense that uses modules from 'prometheus' and ferm rules [22:03:31] it has some explicit hiera() calls so it shouldn't be impacted by the move [22:03:51] yeah I guess what would be affected would be any hieradata lookup that actually used the role hierarchy [22:04:38] godog: do you want me to redo the patch without all the duplication and move it to role::cache::base or something? [22:05:53] bblack: thanks! yeah that'd what I was thinking, after that it can be merged [22:06:38] godog: does it need to be exlcluded from labs / deployment_prep? [22:07:05] nevermind, I see it's already handled within [22:27:56] yeah it is already special cased for ferm :( [22:28:17] godog: updated https://gerrit.wikimedia.org/r/#/c/319295/ but waiting for CI infra to be normal again so I can push it through compiler, etc [22:30:01] bblack: sweet! thanks, after that is merged we should see text/upload stats rolling in e.g. in https://grafana.wikimedia.org/dashboard/db/prometheus-varnish-dc-stats [22:31:11] cool [22:31:40] (although don't actually believe those hit/miss stats. I'm not even sure which layer those query, but either way they're not reflective of reality) [22:31:52] (it's mostly things other than hit/miss where varnishstat is useful!) [22:33:00] the hit-ratio problem turns out to be difficult any way you slice it, since there are many queries which fall into neither category succinctly, which varnishstat doesn't account for [22:33:19] but then there's the layering problem as well. so that's why we derive our caching stats from X-Cache header data that's cross-layer and looks at the edge cases [22:34:13] fascinating! didn't know those would be unreliable [22:34:34] it's a thorny thing. you'd think "hit-vs-miss" would be a simple thing, but it's just not :) [22:34:56] https://grafana.wikimedia.org/dashboard/db/varnish-caching <- is the one derived from X-Cache data [22:35:24] hehehe caching is hard [22:35:39] what collects those? [22:35:57] a python script called varnishxcache [22:36:08] which ultimately is pulling data from live varnishlog via python [22:36:22] we have a number of varnish stats sources that are similar in nature/pattern [22:36:48] (runs some filtered varnishlog output and grabs values from certain special headers or whatever, and then parses them up into useful stats exports) [22:37:35] https://github.com/wikimedia/operations-puppet/blob/production/modules/varnish/files/varnishxcache4 [22:37:48] I see, indeed one would think that varnishstat would be enough as opposed to hooking into varnishlog [22:39:08] well, even if varnishstat gave a true-r accounting for a single daemon (and properly split "pass" traffic, including requests that are initially of the "miss" disposition but then converted into hit-for-pass objects for subsequent traffic, and properly dealt with internal redirect/error traffic, etc) [22:39:20] it still wouldn't properly account for the layering [22:39:52] we could find out that the frontend hitrate is X and the backend hitrate is Y on a given node, but that really doesn't say much of anything about the user-facing overall hitrate of anything [22:41:48] so the X-Cache based stuff basically digs through the layered results [22:42:46] (but it's still important that we offer breakdowns of which layers the results come from. a remote-hit isn't as good as a local-hit or frontend-hit. it still saves querying the app, but it has the additional network latency too) [22:48:50] interesting, yeah it makes sense reading the varnishxcache code to have that sort of user-facing stat available [22:50:10] if there's other stats from varnishstat -j that are interesting it should be all there, including from which layer it comes from [22:50:33] yeah [22:50:49] well there's probably some interesting dashboards we could make from some of them [22:51:01] but most of them only get used in an exploratory sense in ganglia [22:51:19] as in "I don't understand what's going on, let's scroll through 300 varnishstat graphs and see if something looks odd around a certain time" [22:51:59] we'll probably (for lots of different thing than varnish?) have to make some ganglia-like dashboards for doing that kind of thing? or some generic one with selectors. [22:53:30] yeah, my idea on that would be to have a dashboard with 'overall' status with a few key metrics and another to drilldown into specifics e.g. per cluster that show more detailed metrics and breakdowns [22:54:02] a good guideline I've found is USE from http://www.brendangregg.com/usemethod.html [22:56:13] https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?var-server=cp1065:9100&var-datasource=eqiad%20prometheus%2Fops [22:56:32] ^ looking at the basic machine stats here, shouldn't the cpu% numbers be scaled by #cpus? [22:57:05] user: 214%, etc [23:05:16] mhh yeah that'd be easier to gauge, I'm looking at how to scale it [23:08:58] bblack: should be scaled now [23:09:42] \o/ [23:09:51] FTR that meant changing 'sum by (mode) (irate(node_cpu{mode!="idle",instance=~"$server"}[5m]))' to divide it by the number of cores 'sum by (mode) (irate(node_cpu{mode!="idle",instance=~"$server"}[5m])) / scalar(count(node_cpu{mode="idle",instance=~"$server"}))' [23:11:55] interesting [23:13:16] godog: what's the "datasource" thing? [23:13:28] (are half the machines logging to eqiad and the other half to codfw basically?) [23:15:08] bblack: the prometheus server to query data from, ATM each server polls only machines in its respective site [23:15:43] what about ulsfo/esams? [23:37:23] 10Traffic, 06Operations: 503 errors for users connecting to esams - https://phabricator.wikimedia.org/T149865#2766895 (10Matanya) [23:46:33] 10Traffic, 06Operations, 10RESTBase, 06Services (doing): Restbase redirects with cors not working on Android 4 native browser - https://phabricator.wikimedia.org/T149295#2766938 (10Pchelolo) Created a PR to fix it, after review will test it in labs, hopefully it's gonna fix everything: https://github.com/w...