[07:15:59] halfak_: o/ I need to go very soon, in a coffee shop in middle of nowhere [07:16:25] explain to you later. but I want to know if there is anything I can do in ~half an hour [07:18:31] I hope you like my work on wikillabels and puppet, and etc. [10:10:39] o/ [10:11:05] * halfak_ is trying to sleep in to prepare his adjustment back to UTC -5 [15:38:31] !!! We have a problem [15:38:55] Looks like a new version of Wikilabels was deployed that requires CORS, but it is not working. [15:44:58] So! OK. Two options I see. Revert the server code back to the version that did not require POST and DELETE for write operations and the client code back to JSONP [15:45:05] Or figure out what CORS isn't working. [15:48:51] Wow. We have a lot of issues here [15:51:32] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337544 (10Halfak) [15:52:16] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337558 (10Halfak) Problem 1: Looks like new javascript for wikilabels was deployed but it wasn't updated for https://meta.wikimedia.org/wiki/MediaWiki:Gadget-WikiLabels.js That is now fixed. [15:54:04] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337560 (10Halfak) Problem 2: labels.wmflabs.org returning 403 for all requests from wikilabels. ``` $.ajax("https://labels.wmflabs.org/auth/whoami/") Object {readyState: 1} load.php?debug=false&lan... [15:54:47] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337561 (10Halfak) Looks like the gadget still works as expected. See http://labels.wmflabs.org/gadget/ [15:55:21] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337562 (10Halfak) The CORS regex doesn't match wikidata.org. This is unrelated as I have been testing on en.wikipedia.org ``` (3.4)[halfak@graphite: ~/projects/wikilabels-wikimedia-config/config] $... [15:59:08] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337567 (10Halfak) Updated in https://github.com/wiki-ai/wikilabels-wikimedia-config/commit/3f7b958d958f5804192ee6b6ff43cce3ac749363 ``` $ python Python 3.4.3 (default, Jul 28 2015, 18:20:59) [GCC 4.... [16:01:01] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337569 (10Halfak) Pushing to staging to run tests [16:05:57] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337572 (10Halfak) Looks like I was wrong. Only the auth/whoami is returning 403 (which is what it is supposed to do when a user isn't auth'd). ``` $.ajax("https://labels-staging.wmflabs.org/campai... [16:10:10] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337573 (10Halfak) Eeek! Looks like we are getting a periodic database connection error too. ``` May 29 16:06:34 wikilabels-01 uwsgi-wikilabels-web[8825]: 2016-05-29 16:06:34,639 ERROR:wikilabels -... [16:10:44] 06Revision-Scoring-As-A-Service, 10Wikilabels: Period database connection error in Wiki labels - https://phabricator.wikimedia.org/T136521#2337574 (10Halfak) [16:10:53] 06Revision-Scoring-As-A-Service, 10Wikilabels: Periodic database connection error in Wiki labels - https://phabricator.wikimedia.org/T136521#2337587 (10Halfak) [16:11:30] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337589 (10Halfak) See T136521 [16:13:49] 06Revision-Scoring-As-A-Service, 10Wikilabels: Periodic database connection error in Wiki labels - https://phabricator.wikimedia.org/T136521#2337591 (10Halfak) Looks like this related error happens too ``` May 29 16:07:38 wikilabels-staging-01 uwsgi-wikilabels-web[16077]: Traceback (most recent call last): Ma... [16:15:11] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337592 (10Halfak) I've learned that I can do the full OAuth handshake from https://labels.wmflabs.org/gadget/ and https://labels-staging.wmflabs.org/gadget/. So it's something related to the Wikipedi... [16:22:46] 06Revision-Scoring-As-A-Service, 10Wikilabels: Periodic database connection error in Wiki labels - https://phabricator.wikimedia.org/T136521#2337606 (10Halfak) Accessing `https://labels-staging.wmflabs.org/auth/whoami/` directly returns : ``` { "user": { "id": 6877667 } } ``` Accessing the same address via $... [16:23:23] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337607 (10Halfak) The following works just fine. ``` $.ajax("https://labels-staging.wmflabs.org/auth/") Object {readyState: 1} ``` [16:25:47] 06Revision-Scoring-As-A-Service, 10Wikilabels: Periodic database connection error in Wiki labels - https://phabricator.wikimedia.org/T136521#2337608 (10Halfak) Aha! Not sending cookies through a CORS request! http://stackoverflow.com/questions/14462423/cross-domain-post-request-is-not-sending-cookie-ajax-jqu... [16:30:29] 06Revision-Scoring-As-A-Service, 10Wikilabels: Periodic database connection error in Wiki labels - https://phabricator.wikimedia.org/T136521#2337616 (10Halfak) Woops! These comments should have been on T136520 [16:30:41] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337544 (10Halfak) Accessing `https://labels-staging.wmflabs.org/auth/whoami/` directly returns : ``` { "user": { "id": 6877667 } } ``` Accessing the same address via $.ajax() on the wiki page result... [16:30:52] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337619 (10Halfak) Aha! Not sending cookies through a CORS request! http://stackoverflow.com/questions/14462423/cross-domain-post-request-is-not-sending-cookie-ajax-jquery need to add: ``` xh... [16:32:01] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337620 (10Halfak) Updated wikilabels to provide a credentials flag here: https://github.com/wiki-ai/wikilabels/commit/08aadf590d79ba1d2b9455d24ddfe64355a1ae0b Deployed and checked on staging with:... [16:59:32] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337621 (10Halfak) OK. So it looks like the remaining issue has to do with assigning tasks. We get: ``` index.php?title=MediaWiki:Gadget-WikiLabels.js&action=raw&ctype=text/javascript&_=14645401546... [17:25:55] 06Revision-Scoring-As-A-Service, 10Wikilabels: Periodic database connection error in Wiki labels - https://phabricator.wikimedia.org/T136521#2337622 (10Halfak) [17:25:57] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels db connection issues - https://phabricator.wikimedia.org/T136502#2337624 (10Halfak) [17:26:39] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels db connection issues - https://phabricator.wikimedia.org/T136502#2337112 (10Halfak) I deployed https://github.com/wiki-ai/wikilabels/pull/125 Am now working on a too many connections error. ``` May 29 17:23:25 wikilabels-01 uwsgi-wikilabels-web[20966... [17:31:34] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels db connection issues - https://phabricator.wikimedia.org/T136502#2337626 (10Halfak) Turned down default connection count with https://github.com/wiki-ai/wikilabels/commit/bc68d0e1a3a0450cbb245da5d96b73fcadd5333b tested in staging and deployed. [17:33:17] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels db connection issues - https://phabricator.wikimedia.org/T136502#2337112 (10Halfak) a:03Halfak [17:33:29] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337544 (10Halfak) a:03Halfak [17:34:46] 06Revision-Scoring-As-A-Service, 10Wikilabels: Write post-mortem for wikilabels downtime (2016-05-29) - https://phabricator.wikimedia.org/T136523#2337631 (10Halfak)