[00:59:37] awight: please let me know when get the base image and vars set please... [01:01:26] Zppix: It sounds like TheresNoTime is planning to post a PR with a Dockerfile, maybe in the next few days. [01:01:38] That isnt what well need [01:02:05] I need a base image with all repos on it [01:02:18] But i guess its a start [01:02:23] I can modify it from there [01:02:24] Nevermind [04:03:16] 10,09▄12,09▄08,04▄08,03▄08,10▄13,09▄11,05▄13,05▄05,13▄07,05▄09 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEctfvujcuv: halfak grumble stashbot wdna Reception123 Cook879 Theres [04:03:22] 07,04▄03,09▄02,11▄11,03▄08,04▄09,07▄13,13▄08,02▄09,09▄12,03▄07 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEnjcnslznqh: Shilad icinga-wm putnik Lydia_WMDE Groovier grumble gre [04:03:26] 04,02▄03,03▄12,06▄11,03▄05,11▄13,12▄07,08▄12,04▄07,05▄09,13▄03,08▄09,04▄04,13▄04,11▄11,02▄11,02▄12,09▄13 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREEN [04:03:32] 05,03▄13,06▄05,11▄04,05▄09,06▄09,02▄04,04▄03,09▄08,03▄03,12▄02,02▄05,12▄07,10▄10,07▄02,07▄04,12▄04,04▄13,11▄12,03▄11 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE CO [04:03:36] 05,07▄11,02▄07,04▄12,11▄11,11▄05,11▄11,09▄11,09▄13,13▄05,10▄03,13▄08,06▄05 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEerkkod: refeed[z] wdna Cook879 icinga2-wm greg- [04:03:41] 06,06▄12,07▄04,08▄06,05▄08,10▄05,12▄05,05▄05,05▄09,12▄02,02▄04,08▄08,12▄04,09▄08 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEiajpwd: quiddity AlexZ Malvolio Matth [04:03:46] 05,11▄06,12▄12,02▄02,07▄06,12▄08,09▄03,04▄07,10▄06,11▄04,10▄02,11▄08 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEiipdsaq: wikibugs akosiaris greg-g halfak|Network_d jdlro [04:03:52] 08,05▄02,02▄04,03▄13,11▄05,12▄04,11▄09,06▄05,04▄04,11▄11,06▄10 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEulhudn: ccchris quiddity SKYRAN halfak|Network_d chelsyx prtksxna M [04:03:57] 13,02▄04,06▄08,07▄09,03▄11,06▄10,06▄11,03▄04,12▄13,13▄05,12▄10,02▄07,03▄13,13▄07,12▄05,06▄10,13▄11 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEmbmon: [04:04:02] 12,13▄04,10▄05,07▄13,10▄07,04▄04,05▄06,10▄13,03▄06,08▄02,08▄13,11▄07 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEujcrnllsla: icinga-wm Zppix greg-g prtksxna wmopbot Malvo [04:04:07] 11,09▄12,03▄02,02▄13,12▄08,06▄08,11▄02,12▄12,08▄10,02▄12,10▄04,08▄12,02▄02,12▄13,07▄04,11▄09,05▄10,09▄07 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREEN [04:04:11] 12,04▄09,10▄10,10▄13,04▄06,13▄02,11▄07,10▄05,05▄08,08▄07,04▄03,06▄07,07▄10,05▄12,04▄05 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODErahahqnmpu: quiddity mdholl [04:04:16] 12,11▄06,09▄13,02▄04,07▄13,07▄13,05▄06,06▄04,03▄03,08▄03,12▄11,13▄12 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEnfanhoclo: Vito abbe98[m] halfak Krenair joewalsh putnik [04:04:21] 03,04▄08,11▄13,09▄05,06▄10,08▄07,12▄09,09▄11,12▄04,06▄05,09▄11,03▄09 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEedqjln: halfak|Network_d grumble jem abbe98[m] ragesoss p [04:04:26] 02,02▄10,03▄03,10▄08,04▄10,03▄04,10▄03,13▄07,03▄03,11▄05,07▄11 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEdcevegdv: Vito CIA abbe98[m] putnik Zppix|Away halfak|Network_d |41 [04:04:31] 07,07▄06,04▄07,03▄03,02▄08,02▄07,05▄13,12▄07,03▄10,08▄02,09▄02,11▄08,04▄08,09▄10,04▄08,11▄13 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEixztvbhp: greg-g [04:04:36] 12,11▄06,08▄05,08▄04,06▄07,04▄12,06▄09,09▄13,06▄08,10▄10,02▄05,06▄10,05▄09,09▄10,11▄04,05▄10,04▄11,08▄08,12▄12,10▄08 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE CO [04:04:41] 08,13▄10,05▄11,11▄04,08▄11,05▄12,07▄04,08▄12,02▄11,06▄13,02▄12,07▄05,13▄04,09▄08 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEmmxsmmgpk: SKYRAN guillom jdlrobson e [04:04:46] 04,10▄02,08▄03,12▄05,11▄09,11▄09,12▄08,06▄02,11▄06,10▄04,10▄05,05▄04,12▄02,08▄13,07▄08,07▄07,04▄05,05▄12 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREEN [04:04:52] 11,06▄06,08▄11,11▄07,09▄07,07▄09,05▄03,12▄11,11▄11,10▄02,13▄07 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEqtlkbsohu: wikibugs joewalsh greg-g sir_lestaty Lydia_WMDE TheresNo [04:04:57] 12,02▄04,10▄11,07▄13,06▄10,09▄09,10▄12,11▄12,04▄10,06▄06,12▄04,02▄02,07▄04,05▄06,09▄07,09▄10,12▄10,05▄11,11▄13,10▄09 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE CO [04:05:02] 07,02▄12,09▄04,06▄07,02▄13,04▄05,10▄03,09▄05,12▄07,08▄12,08▄09,10▄04,07▄03,08▄09,04▄07,10▄12 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEvgximuyaxz: wikib [04:05:08] 10,13▄05,05▄03,07▄05,04▄05,02▄06,11▄03,06▄02,03▄05,02▄03,10▄07,04▄04,11▄05,10▄11,11▄11,08▄08,06▄11 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEdhgvupx [04:05:12] 04,13▄03,11▄11,11▄05,11▄05,08▄12,12▄03,03▄12,07▄02,08▄13,11▄03,11▄06,07▄13 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEnctjyjoru: Groovier jdlrobson chelsyx icinga-wm [04:05:17] 10,11▄02,02▄05,13▄11,12▄10,02▄09,05▄05,12▄12,05▄02,06▄04,12▄04,11▄12,13▄07,13▄09,06▄05 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEgqzrhiijr: Krenair grumble [04:05:22] 13,11▄02,06▄07,13▄10,08▄02,11▄08,04▄05,11▄11,10▄04,08▄11,11▄12,04▄08,04▄06,06▄05,06▄06,13▄13,05▄12,10▄05,13▄03 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN [04:05:27] 10,10▄03,10▄04,07▄10,09▄10,10▄06,12▄05,08▄06,03▄13,02▄04,09▄02,04▄05,07▄08,13▄08,05▄08 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEqiygaetjyj: halfak refeed[z [04:05:33] 09,06▄03,08▄02,12▄13,02▄06,09▄12,05▄09,07▄07,09▄05,12▄07,04▄05,07▄09,04▄12,11▄06 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODErzofswkboj: ragesoss Reception123 Shi [04:05:37] 04,13▄06,03▄12,11▄07,11▄04,03▄04,10▄11,04▄09,12▄11,06▄13,05▄06,07▄08,10▄07,02▄09,05▄13,04▄09,13▄12,09▄05 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREEN [04:05:43] 10,09▄02,03▄09,13▄05,04▄12,02▄03,06▄04,08▄07,04▄13,06▄13,03▄10,05▄11,10▄04,07▄05,04▄11,13▄12 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEiemqqthem: joewal [04:05:47] 09,10▄04,04▄10,08▄05,07▄07,11▄02,07▄05,10▄07,06▄09,11▄02,09▄08,07▄02,10▄12,11▄09,09▄07,04▄07,12▄07,06▄04,04▄02,06▄08 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE CO [04:05:53] 06,02▄08,03▄12,06▄13,05▄13,04▄10,10▄05,11▄06,05▄04,03▄03,10▄05,09▄12,09▄07 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEzrmcsitlnn: refeed[m] icinga2-wm mdholloway joe [04:05:57] 10,11▄02,04▄04,05▄08,04▄08,03▄05,13▄11,06▄04,03▄09,09▄13,02▄06,07▄05,06▄06,05▄10,04▄10,06▄11,02▄02 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODExnkrfzw [04:06:02] 08,02▄05,13▄11,10▄09,07▄02,09▄05,03▄04,03▄07,02▄10,06▄13,07▄09,03▄04,07▄02,02▄12,09▄12 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEjcvanmj: Malvolio Ivy jdlro [04:06:07] 13,12▄02,13▄05,04▄03,12▄09,07▄08,12▄04,10▄03,13▄04,05▄02,04▄13,06▄06,06▄04,03▄13,06▄06,04▄12,07▄05 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEfavnt: [04:06:12] 12,08▄10,02▄11,06▄05,09▄06,09▄10,10▄05,12▄12,10▄08,07▄02,08▄06,11▄06,12▄10,04▄03,07▄11,09▄03 https://beta.companieshouse.gov.uk/company/10308021/filing-history christel sold freenode to Private Internet Access Andrew Lee WHO ALSO OWNS SNOONET AND IS MOVING FREENODE TO THAT SERVER (NEXT MONTH) AND CLOSING DOWN OPEN SOURCE ROOMS PLEASE COMPLAIN IN CHAN FREENODEatackpw: halfak|N [09:46:31] 10Scoring-platform-team, 10ORES, 10Operations, 10Patch-For-Review: rack/setup/install ores2001-2009 - https://phabricator.wikimedia.org/T165170#3853507 (10akosiaris) >>! In T165170#3852233, @Dzahn wrote: > So looks like we need a new role class for "regular ores server in production" (regular as opposed to... [09:48:13] 10Scoring-platform-team, 10ORES, 10Operations, 10Patch-For-Review: rack/setup/install ores2001-2009 - https://phabricator.wikimedia.org/T165170#3853509 (10akosiaris) 05stalled>03Open Unstalling. Note that we are in the year-end deployment freeze. The boxes should not be put in production until we are o... [10:47:31] 10Scoring-platform-team, 10ORES, 10Wikidata, 10Beta-Cluster-reproducible, and 2 others: wikidata.beta.wmflabs.org/wiki/Special:RecentChanges InvalidArgumentException No model available for [goodfaith] - https://phabricator.wikimedia.org/T183266#3853662 (10Addshore) >>! In T183266#3852471, @awight wrote: >... [13:31:48] Hello [13:32:31] [14:10:16] 10Scoring-platform-team, 10ORES, 10Wikidata, 10Beta-Cluster-reproducible, and 2 others: wikidata.beta.wmflabs.org/wiki/Special:RecentChanges InvalidArgumentException No model available for [goodfaith] - https://phabricator.wikimedia.org/T183266#3854102 (10Ladsgroup) There will be similar problem in prod.... [14:15:14] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Technical-Debt, 10User-Ladsgroup: Deprecate CheckModelVersions and integrate it with the extension workflow - https://phabricator.wikimedia.org/T183468#3854107 (10Ladsgroup) [14:15:43] 10Scoring-platform-team, 10ORES, 10Wikidata, 10Beta-Cluster-reproducible, and 2 others: wikidata.beta.wmflabs.org/wiki/Special:RecentChanges InvalidArgumentException No model available for [goodfaith] - https://phabricator.wikimedia.org/T183266#3854120 (10Ladsgroup) Made {T183468} [15:11:04] o/ [15:18:34] How are you today aaron? [15:26:33] Oh, I'm doing OK. Lots to worry about in staff land :| [15:27:05] Halfak is WMF a full-time thing? [15:27:39] Adotchar, yup. I'm full time staff. [15:27:51] cool halfak [15:27:51] That's a big reason why I have so many meetings. [15:28:00] That and wearing the manager hat. [15:28:23] The fun thing about hats is they stack. Researcher, Manager, Engineer, Community Liaison, etc. [15:28:34] :D [15:28:50] halfak: arent you a software engineer, manager, research scientist :P [15:29:12] Excuse me Dr. Halfak [15:29:17] * halfak has no opinion on the stack order [15:29:19] :D [15:29:29] Yass [15:29:30] lool [15:29:51] Paging Dr Halfak... :P [15:33:22] Dr. Halfak, any time between meetings today? :D [15:33:37] Sure! Lots today :) [15:33:50] halfak: for starters - https://dpaste.de/rSCT [15:34:28] codezee: not found [15:34:36] codezee, ROC AUC? [15:35:11] halfak: roc_auc (micro=0.893, macro=0.905) [15:35:18] the whole set ;) [15:35:23] halfak: yes [15:35:30] Zppix: what was not found? [15:35:36] The paste [15:35:44] :/ [15:35:59] let me share the scored instances too in a min [15:37:32] halfak: on a side note, I build a binary classifier for education today with true-false ratio 1:5 and scored instances, I found that it was tagging a lot of biographies of professors of universities as education, sth not entirely wrong [15:37:48] Oh good :) [15:37:55] these things can skew our metric but show that they also expose improvements [15:38:02] *room for [15:38:08] Our labels are messy. But the model will help fill in around some of the mistakes in the directory structure. [15:38:18] Right on. [15:38:20] awight: o/ [15:38:33] Can you get me the ROC-AUC of the whole set of classes? [15:38:40] codezee, ^ [15:38:42] yes, and word2vec seems to guess the idea of the article in some sense or other [15:38:57] Zppix never sleeps ;-) [15:39:10] halfak: the results I have from cv_train are very ill-formatted due to large no of classes :/ [15:39:32] let me pull that out [15:39:35] *those [15:39:39] codezee, but you gave me the counts block [15:39:44] Just give me the roc block :P [15:40:16] awight: what is "sleep"? [15:40:30] halfak: yes, - https://dpaste.de/dnB8 [15:40:45] but horrible to read... [15:41:07] "Snippet you have searched is not available (anymore)." [15:41:08] Zppix: sleep is when your RAM is retaining things. [15:41:14] Lol [15:41:40] wait i'll just use gist [15:42:10] Dpaste is broken i guess 🤷‍♂️ [15:43:08] halfak: https://gist.github.com/codez266/5619d0ef06113ce95c591407b3a5ca30 [15:43:20] Zppix: yeah apparently [15:44:04] Woah. This formatting is broken :| [15:44:22] thats what I was mentioning [15:44:26] They are columns but should be rows. [15:44:29] Interesting [15:44:33] * halfak looks into this problem. [15:45:06] Ohhh I see what's going on. [15:45:09] Working on it. [15:46:51] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Technical-Debt, 10User-Ladsgroup: Deprecate CheckModelVersions and integrate it with the extension workflow - https://phabricator.wikimedia.org/T183468#3854107 (10awight) The best I can imagine so far is a two-prong strategy: * When a change... [15:47:03] Amir1: ^ my random 2 cents [15:47:52] awight: Thanks! [15:48:30] I liked your idea of inlining into the extension workflow… [15:49:37] I hope it makes things a little bit faster, I'd go with the second option of yours [15:49:41] halfak: I also have now 500 randomly scored articles with class probabilities and actual labels, just thinking where to share, since each instance is quite big [15:49:53] specially if we can build a comparing system [15:50:24] "there is a new version returned by the service, let's redo checking on all models" [15:50:46] codezee, put a data file on ores-misc-01 and we can slice/dice it [15:51:40] halfak: its on ores-misc-01 [15:51:43] in my home [15:51:47] Amir1: Good point, we don’t need the RC part of it cos we’ll be getting changeprop scores constantly. [15:52:06] wait i'll copy to /srv [15:52:39] done! [15:52:48] Amir1: you think this is good enuf for our comparison? http://php.net/version_compare [15:53:25] Amir1: ah also great, yeah when one version is bumped, the odds are good that we’ll save a few checks by ingesting the full list of models. [15:53:41] exactly [15:53:47] That would need to be protected by a mutex. [15:53:58] otherwise we’ll have all wikis slamming at the same time [15:54:51] codezee, it'll help me if you give me a path [15:55:07] halfak: sorry, /srv/drafttopic.500.scored :/ [15:57:20] "actual" key in the end of each entry is the actual labels [15:57:21] Looking at the first line, we have "STEM.physics" as the actual. [15:57:32] and 0.07022429545543642, as the "probability" [15:58:15] yes [15:58:59] halfak: minor nudge, I changed that one-liner: https://github.com/wiki-ai/ores/pull/243 [15:59:06] general order looks to be 1e-2 [15:59:42] Amir1: Do you know what the ores#codecov failures are about? I’d be happy to spend a minute looking at that, if it’s not a known bug. [16:00:00] It's surprising to me too [16:00:23] I think it's because it defines a certain threshold of coverage if it's below that, it screams [16:00:33] where is the setting for the threshold, I have no idea [16:00:41] ty! [16:00:42] I have to be afk [16:00:47] Amir1: kk I’ll poke at it [16:00:50] will be back soon (TM) [16:00:57] lol you can’t have my lines [16:01:07] codezee, I think label-weights might help us. [16:01:10] :P [16:01:41] halfak: how should we specify the weights, on what basis? [16:02:04] http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html [16:02:24] "Note that for multioutput (including multilabel) weights should be defined for each class of every column in its own dict. For example, for four-class multilabel classification weights should be [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of [{1:1}, {2:5}, {3:1}, {4:1}]." [16:02:29] reading about this now. [16:03:02] OK I think i see what's going on here. [16:03:32] sometimes when I think I’m smart I open this chat and it instantly puts me back in my place [16:03:40] lol Adotchar :P [16:03:44] :P [16:03:57] codezee, I think we still need to *implement* weights for the multilabel case. [16:04:07] But here it looks like we can do it nicely. [16:04:20] And we should probably upweight all of the positive cases for each label. [16:04:28] maybe by an order of 50 [16:04:42] halfak: do you mean our problem is with "The �balanced� mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as " ? [16:05:04] Kinda yeah. [16:05:10] We don't need to be exact about it. [16:05:18] Damn dpaste. [16:05:19] because each category is roughly 1:100 of the total samples I suppose [16:05:21] Can you paste the counts again [16:05:36] Right so then maybe we'll increase the weights by a 100 [16:05:59] I'll work on a PR for making the weights actually *work* [16:06:27] Actually maybe you should do that while I work on the formatting table problem [16:06:38] halfak: https://gist.github.com/codez266/dc169639b6ea7ce44d1f9e0ba00dad6a [16:07:00] 1:15 i see roughly [16:07:22] Indeed and it depends on the class [16:07:29] yes, varying [16:07:30] We might want to do the math for each class [16:07:43] This is another +1 for our label-config file format [16:07:48] halfak: so should i upweght per class counts? [16:07:53] BTW awight merged that ORES pr [16:07:54] *upweight [16:08:01] codezee, yeah let's try that [16:08:16] halfak: yes, awesome. I’m going to try it on beta. [16:08:54] * codezee off to implement multilabel weights [16:09:29] halfak: I'll try to get this in quickly and probably merged so I can start some training before going to sleep :P [16:09:38] training this beast is a thing [16:10:07] +1 [16:12:43] halfak: if you've got time, this could use some attention - https://github.com/wiki-ai/drafttopic/pull/14 [16:13:44] Time is the opposite of what I have :) [16:13:49] but priority is something i can manage [16:14:18] i understand... [16:14:26] ewww... mw.lib [16:14:33] We need to kill that eventually. [16:14:48] i wanted to normalize titles the standard way, but I'll take note [16:15:13] codezee, can you use wikiclass's utility for now? [16:17:01] halfak: pattern is same, but its not configurable to fetch on page-titles, i'm using it from its own branch so no issues if it doesn't look good [16:19:02] wiki-ai/ores#905 (codecov_quiet - 12d425c : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/ores/builds/319785849 [16:19:09] Oh. Why do you need to fetch page_text at all codezee? [16:19:13] Why not just extract it? [16:20:06] halfak: bec I was moving across word2vec and bag of words and also playing with word2vec, so I thought it best to have the large part offline for once [16:20:16] then extract offline [16:20:40] codezee, hmm. OK. [16:20:59] Can you just use the script for now and I'll think more about this? [16:21:10] Wait...what do we do for draftquality? [16:21:15] halfak: sure...i'll manage [16:21:29] * codezee looking [16:21:38] (03CR) 10Catrope: [C: 032] Fix Contributions ORES preference and form override [extensions/ORES] - 10https://gerrit.wikimedia.org/r/399122 (https://phabricator.wikimedia.org/T182911) (owner: 10Petar.petkovic) [16:21:50] halfak: https://github.com/wiki-ai/draftquality/blob/master/draftquality/utilities/fetch_text.py :D :D [16:22:26] it has no scripts basically [16:23:01] (03Abandoned) 10Catrope: Make Contributions ORES preference sticky [extensions/ORES] - 10https://gerrit.wikimedia.org/r/398841 (https://phabricator.wikimedia.org/T182911) (owner: 10Petar.petkovic) [16:23:31] (03Merged) 10jenkins-bot: Fix Contributions ORES preference and form override [extensions/ORES] - 10https://gerrit.wikimedia.org/r/399122 (https://phabricator.wikimedia.org/T182911) (owner: 10Petar.petkovic) [16:25:34] halfak: I see that scikit-learn uses key 'class_weight' but revscoring passes 'sample_weights' in its fit_kwargs, is this intentional? [16:26:16] codezee, yes. but we might need to change it. [16:26:23] Not sure what you mean by intentional [16:26:31] scikitlearn also uses sample_weights [16:28:51] halfak: I was looking at the class_weight param of the constructor so got confused, any diff b/w them? [16:29:12] codezee, please feel free to read the docs. that will do a better job than I can, [16:29:50] SPAAAM [16:30:01] yeah, sorry... :/ [16:31:10] * Zppix trouts awight [16:31:59] wiki-ai/editquality#26 (codecov_polite - 3c8cda9 : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/319791590 [16:35:11] awight: do you want me to merge all those for ya? [16:35:46] wiki-ai/revscoring#1370 (codecov_polite2 - 64d5da9 : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/319793440 [16:36:11] 10Scoring-platform-team, 10ORES, 10Operations, 10Patch-For-Review: rack/setup/install ores2001-2009 - https://phabricator.wikimedia.org/T165170#3258942 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by akosiaris on neodymium.eqiad.wmnet for hosts: ``` ['ores2001.codfw.wmnet', 'ores2002.codfw.w... [16:37:58] Zppix: The thing to CR for me, IMO, is that the content of that .codecov.yml file is the same in each PR, and that I haven’t accidentally pulled any additional commits in by mistake. Cos I just had to amend three repos for that exact reason. [16:38:40] Ill look and merge if ok, k? Awight [16:40:28] wiki-ai/revscoring#1368 (codecov_polite - 230aeaf : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/319791927 [16:40:59] codezee, https://github.com/wiki-ai/revscoring/pull/384 [16:41:32] codezee, that should help us with some sense-making. [16:41:39] I'm going to bike into the university. [16:41:45] I'll be offline for ~ an hour. [16:42:36] looking... [16:43:31] wiki-ai/revscoring#1372 (row_format_long_labels - f945e6e : halfak): The build failed. https://travis-ci.org/wiki-ai/revscoring/builds/319796745 [16:52:14] (03PS1) 10Awight: Bump ores submodule [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/399661 [16:52:35] (03CR) 10Awight: [C: 032] "Self-merging submodule bump" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/399661 (owner: 10Awight) [16:53:40] (03CR) 10Awight: [V: 032 C: 032] Bump ores submodule [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/399661 (owner: 10Awight) [17:02:42] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. - https://phabricator.wikimedia.org/T182614#3854587 (10awight) Works perfectly now. Waiting until January to deploy to production. [17:02:53] 10Scoring-platform-team (Current), 10ORES, 10Operations: Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. - https://phabricator.wikimedia.org/T182614#3854588 (10awight) [17:13:00] wiki-ai/wikilabels#254 (codecov_polite - 8753f45 : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/wikilabels/builds/319809773 [17:13:46] Amir1: What is our translatewiki.net project called? I see strings but not the project. [17:22:53] 10Scoring-platform-team, 10ORES, 10Operations, 10Patch-For-Review: rack/setup/install ores2001-2009 - https://phabricator.wikimedia.org/T165170#3854619 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['ores2008.codfw.wmnet', 'ores2002.codfw.wmnet', 'ores2003.codfw.wmnet', 'ores2005.codfw.wmnet'... [17:31:20] I do not grok. I found https://phabricator.wikimedia.org/diffusion/GTWN/browse/master/repoconfig.yaml;a0b48c13fe3699e6263691e98629e2de99257e53$534 [17:31:50] btw I realized recently that we may not be able to use twemproxy for the redis queue due to celery using the MULTI command, but we can use it for the cache. Effectively doubling cache size and increasing cache availability [17:32:02] but the refactoring for that will have to wait till next year [17:36:24] akosiaris: Could we split read and write clusters? [17:40:03] we could but we only got 2 servers as ores databases [17:40:37] and it's the result redis is a cache after all, not sure the read/write split paradigm would actually help [17:40:58] help more than an equal split that is [17:41:50] and redis currently is not bound on IO or anything.. so we probably haven't reached the point were that split would help [17:42:07] ok I trust your judgment on that. Still, with 2 servers we easily fit multiple instances of twemproxy and redis on each [17:42:18] How are the data centers linked? Sorry to be a noob... [17:42:41] will each DC have an up-to-date replica of the other? [17:42:59] or would we Redis Cluster the masters together? [17:43:07] the DCs are not linked [17:43:11] ah ok [17:43:20] there is no reason to link them, plus it would actually be problematic [17:43:33] redis replication is not that great at those latencies and sizes [17:44:04] but we have changeprop issuing duplicate requests, updating both DC's caches [17:44:22] You could say the dcs are dc'ed :P [17:44:26] the rest of the cache is natural differences due to the external users request patterns [17:46:06] The changeprop duplication seems wasteful, since it’s so CPU-expensive. At the price of a pretty small latency we could replicate that from one DC to the other. But +1 that any other usage is local to the DC. [17:47:48] I wonder if a git repo hosted locally would be able to be shared cross-dc to do what your wanting awight [17:48:19] git is not the right technology for this, fortunately [17:48:43] Err nevermind ignore me im being a little brain dead [17:48:49] xD [17:49:07] awight: how would you replicate the cache from one DC to the other ? redis replication is NOT the way to go on this one [17:49:29] Im thinking about my bot and trying to do other stuff at the same time and its not working well as just proven :D [17:49:41] we 've had that paradigm for the jobqueue and it has turned out to be very brittle [17:50:11] akosiaris: I can imagine hiccoughs would be a huge problem... [17:50:30] and I can’t think of any other replication that would be a natural fit [17:50:32] even more so.. it's not incremental.. if it breaks, it needs to start from scratch [17:50:42] ouch! [17:51:24] which is why we went with the current approach [17:52:07] if our request pattern was less artificial (changeprop+mediawiki) and more organic (actual external requests from users) it would even make more sense to stay with the current approach [17:53:05] but I get your point about calculating everything twice being wasteful [17:53:22] akosiaris: sorry for the distraction… How many edits is wiki* getting these days? 10/sec? [17:53:33] That’s a little over 20 busy CPUs, vs 10 [17:53:39] it's around there... 10-15/sec [17:53:41] not a very large pile of coal [17:55:14] on the other hand, keep in mind that the second dc is about DR primarily, not about performance [17:55:37] the entire point of it is sitting idle until the time for it comes [17:55:42] I think I’m wrong about the theory that we might want twemproxy in front of a celery read-only replica, I imagine the celery worker load is probably doing mostly the MULTI transactions and few reads [17:56:02] celery is incompatible with twemproxy [17:56:06] +1 [17:56:12] or dynomite for that part [17:56:25] https://github.com/Netflix/dynomite/ [17:56:56] there's a number of requests from the authors of dynomite on the celery github repo about dropping multi to make it compatible [17:57:40] https://github.com/celery/celery/issues/3500 [17:57:51] closed effectively as WONTFIX [17:58:00] awight: is https://github.com/wiki-ai/ores/pull/244 gtg? [17:58:07] 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Kanban): Source revision is in Phabricator, but can't be found by deployment tools - https://phabricator.wikimedia.org/T182865#3854732 (10mmodell) 05Open>03Resolved [17:58:41] 10Scoring-platform-team (Current), 10ORES: Fix broken beta-labs deploy - https://phabricator.wikimedia.org/T157135#3854739 (10mmodell) [18:01:19] akosiaris: Do we still have something like ganglia, per-machine monitoring? [18:01:35] I’m hoping to idly look at oresrdb* [18:01:44] https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?orgId=1 [18:01:51] Blocked users shouldnt be able to do labeling am i wrong? [18:01:53] ty [18:03:24] akosiaris: It’s ice-cold [18:03:41] :-) [18:04:10] I had to squint at the CPU graph to not file a bug about 1.000% [18:05:24] Interesting that 50x more data is flowing in than out. This cache is pretty much a waste. [18:06:09] FYI i did a napkin calculation that determined only 2.2% of our scores over the last year came from the cache. [18:06:20] Recent 6 months of activity: https://grafana.wikimedia.org/dashboard/db/ores?orgId=1&from=1496275200000&to=1512086400000 [18:06:20] This shows a mean average of 603 "all scores returned" per minute including both DCs, and average of 590 scores processed. 97.8% of our requests had to actually be processed, conversely 2.2% were from cache. [18:06:36] * Zppix waits patiently for answers [18:06:45] akosiaris: But my understanding is that the cache exists just to accelerate certain common workloads [18:06:47] * Adotchar waits patiently for someone to answer Zppix [18:07:11] awight: in a meeting currently, I 'll respond later [18:07:39] no worries, I’m just rambling. +1 that we shouldn’t bother with anything in front of redis. [18:11:31] oh my that was a short meeting [18:11:35] 2.2% you say ? [18:12:00] so why do we have a 75% cache hit rate then ? [18:16:52] akosiaris: That’s a deceiving graph—it’s only for the “external” requests which are guaranteed to not be new “precached” scores. [18:16:59] bit me, too. [18:17:44] The cache is for people using bulk patrolling tools to pull large + repeated worksets quickly [18:19:17] I just renamed it so we have a chance at not confusing the next person. [18:20:35] ok, thanks [18:21:20] + made it a percentage [18:24:45] halfak: Do you know where wikilabels i18n can be found in translatewiki.net? I’m losing hair. [18:28:03] wiki-ai/revscoring#1374 (label-weights - 83ff30a : Sumit Asthana): The build failed. https://travis-ci.org/wiki-ai/revscoring/builds/319845485 [18:36:40] * awight lobs spitballs at Amir1 and halfak [18:37:00] awight: any idea why builds are failing? [18:37:28] codezee: ooh that’s different that with my PR, which was based on stale master from September. Lemme look... [18:38:37] codezee: no worries, it’s a boring flake8 thing: ./revscoring/scoring/models/sklearn.py:50:17: E122 continuation line missing indentation or outdented [18:39:55] codezee: Feel free to merge my codecov_polite PR which should prevent false build failures due to reduced test coverage [18:40:02] Ill merge [18:40:16] https://github.com/wiki-ai/drafttopic/pull/16 [18:40:20] Zppix: right on [18:42:11] halfak: Amir1 wait, seriously? I was able to get into our translation group by reverse engineering the URL, while it doesn’t appear anywhere in the group directories. https://translatewiki.net/w/i.php?title=Special:Translate&group=wiki-ai&language=bn&filter=%21translated&action=translate [18:43:53] lol, last time I end a commit message with “build failed". [18:44:02] Lol [18:45:09] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3854904 (10awight) @Bodhisattwa would you mind reviewing the interface translation strings? I see t... [18:46:53] o/ awight [18:47:07] I thought our translation group was called Revision Scoring [18:47:08] halfak: challah [18:47:33] I think i got em all awight [18:47:37] halfak: That is the one, thanks. It was like dark matter to me for some reason. [18:47:49] Zppix: nice hunting! [18:49:59] halfak: well, tbh the reason is that the TWN repo config all references a group "wiki-ai" [18:50:06] I should add a comment to the repo config. [18:50:20] twn? [18:52:13] Translatewiki [18:52:27] ^ ty for unpacking egregious acronym [18:52:53] Np [18:58:35] halfak: Feel like hashing out judgments vs endorsements vs discussion? Are you feeling concrete about those, or still in the idea-bouncing phase? [19:00:36] I'm just starting a string of meetings [19:01:27] oho [19:01:33] January it is [19:01:45] Break a chair leg! [19:05:04] 10Scoring-platform-team (Current), 10MediaWiki-Vagrant, 10Patch-For-Review: Clean up ORES vagrant role - https://phabricator.wikimedia.org/T181850#3855150 (10awight) [19:05:32] 10Scoring-platform-team (Current), 10MediaWiki-Vagrant, 10Patch-For-Review: Clean up ORES vagrant role - https://phabricator.wikimedia.org/T181850#3804319 (10awight) To reviewers: I'd like to push through the outstanding patches here, then shelf the rest of the work for later. [19:29:29] back in 30 [19:36:57] awight|afk, just realized one of my meetings is "documentation time" so I could talk in 1.5 hours [20:01:15] wiki-ai/revscoring#1381 (row_format_long_labels - 306a4ca : Sumit Asthana): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/319881441 [20:03:29] wiki-ai/revscoring#1384 (label-weights - e051eeb : Sumit Asthana): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/319882359 [20:09:22] halfak: I have to receive kids at about that time (1 hr now), what did you want to talk about? [20:09:44] awight, you asked me about preference, endorsements, judgement. [20:10:02] ah yeah [20:10:30] It would be nice to chat before the vacation, but I could also catch up with you during one of your hour windows. [20:12:39] label-weights for multilabel whenever its possible - https://github.com/wiki-ai/revscoring/pull/385 [20:22:34] gtg, but I’ll ping once life is stable again [20:27:19] Halfak are you busy i have an question regarding wikilabels [20:45:38] Zppix, ask away [20:45:51] looking at the patch codezee [20:49:30] halfak: doesnt wikilabels disallow blocked users from labeling for projects they are blocked on? [20:49:43] It needs to [20:49:47] Oh [20:49:47] But it doesn't yet [20:50:10] Dont look at me i dont do api [20:50:21] ha [20:50:23] xD [20:50:40] API is a s#£t storm imo [21:05:23] halfak: I have an unknown number of minutes… [21:05:24] voice/text? [21:05:41] Wrapping up some multilabel stuff. [21:05:44] Will ping shortly. [21:08:43] k [21:11:46] halfak: Also heads up, I sent you an email this morning. [21:11:56] I saw it. [21:15:31] ok awight. Call when ready [22:12:23] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Graphite, 10User-fgiunchedi: Regularly purge old ores graphite metrics - https://phabricator.wikimedia.org/T169969#3855629 (10Halfak) `$ find . -type f -mtime +30 | wc -l` can be purged safely. I'll build a short list of metrics that should be... [22:27:20] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Graphite, 10User-fgiunchedi: Regularly purge old ores graphite metrics - https://phabricator.wikimedia.org/T169969#3855640 (10Halfak) OK looks like we want to keep all of the base metrics for the following names indefinitely: ``` ores.*.precache... [23:21:05] OK I'm out of here. have a good one folks [23:21:35] Night halfak