[02:26:09] 10Tool-Labs-tools-Database-Queries, 10Analytics, 10MediaWiki-User-preferences: Gadget usage statistics for Portuguese Wikipedia - https://phabricator.wikimedia.org/T61480#1837499 (10coren) [02:26:11] 6Labs, 10Tool-Labs, 7Tracking: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#1837500 (10coren) [02:26:13] 6Labs, 10Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1837497 (10coren) 5Open>3Resolved The underlying data has been backfilled. [02:43:24] 6Labs, 10Tool-Labs: Make error pages mobile friendly - https://phabricator.wikimedia.org/T119830#1837503 (10coren) [02:43:25] 6Labs, 10Tool-Labs: Remove Jorm from http://tools.wmflabs.org/ betaweb - https://phabricator.wikimedia.org/T119849#1837501 (10coren) 5Open>3Resolved Understandably so, he left the foundation some time ago and is certainly not expected to be assigned work. :-) Removed. [02:44:51] 6Labs, 10Tool-Labs, 10Labs-Team-Backlog, 5Patch-For-Review: Fix Labs' PAM config mess - https://phabricator.wikimedia.org/T85910#1837504 (10coren) Once the above patch goes through, the /etc/pam.d/sshd file needs to be salt'ed away and regenerated from the package. [09:09:30] 6Labs, 10Education-Program-Dashboard: Find stable home for WMF dashboard server - https://phabricator.wikimedia.org/T107939#1837740 (10awight) [09:21:22] 6Labs: Write up a report about kubecon to the Ops team - https://phabricator.wikimedia.org/T118757#1837761 (10yuvipanda) Clearly I suck at writing reports on time. I'll try for this tomorrow. [09:21:44] 6Labs: Add a note to wdq-mm error page about query.wikidata.org - https://phabricator.wikimedia.org/T119123#1837762 (10yuvipanda) a:5yuvipanda>3None [09:28:45] !log cvn Setting up CVNBot21 in cvn-app5. [10:16:53] 6Labs, 10MediaWiki-extensions-ContentTranslation, 7Database, 5Patch-For-Review, 7WorkType-NewFunctionality: Replicate ContentTranslation databases on Labs - https://phabricator.wikimedia.org/T119847#1837363 (10Amire80) [10:17:00] 6Labs, 10MediaWiki-extensions-ContentTranslation, 7Database, 5Patch-For-Review, 7WorkType-NewFunctionality: Replicate ContentTranslation databases on Labs - https://phabricator.wikimedia.org/T119847#1837836 (10Amire80) p:5Triage>3Normal [11:10:15] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837918 (10Reedy) [11:10:16] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837924 (10Reedy) [11:10:17] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837930 (10Reedy) [11:11:11] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#997205 (10Reedy) >>! In T87716#1118517, @Krenair wrote: > Why is this a private task? No idea. It's not anymore. Stupid phab. Change 2 policies, remove security AND an extra... [11:15:16] 6Labs, 10Tool-Labs, 7Database: labs db inconsistent data - https://phabricator.wikimedia.org/T119841#1837942 (10Peachey88) [11:15:18] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837943 (10Peachey88) [11:16:21] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837946 (10Reedy) p:5High>3Low [11:16:47] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1837949 (10Peachey88) [11:16:49] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837948 (10Peachey88) [11:17:02] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#997205 (10Peachey88) [11:17:04] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#530760 (10Peachey88) [11:18:18] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1837961 (10jcrespo) [11:19:49] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837966 (10jcrespo) [11:19:51] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#530760 (10jcrespo) [11:20:50] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#997205 (10jcrespo) @Peachey88 This is not a replication issue, please do not merge the unrelated task. [11:22:21] 6Labs, 7Database: Database servers having missing rows (on labs and api production servers) - https://phabricator.wikimedia.org/T87716#1837976 (10jcrespo) Low because it is believed to be a temporary glitch not ongoing, and most of the affected pages will have been already fixed automatically (cannot reproduce... [11:24:12] 6Labs, 7Database: Missing rows from categorylinks on production servers (dewiki) - https://phabricator.wikimedia.org/T87716#1837978 (10jcrespo) [11:27:55] 6Labs, 10MediaWiki-extensions-ContentTranslation, 7Database, 5Patch-For-Review, 7WorkType-NewFunctionality: Replicate ContentTranslation databases on Labs - https://phabricator.wikimedia.org/T119847#1837984 (10Ricordisamoa) [11:27:58] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1837983 (10Ricordisamoa) [12:51:41] 6Labs, 10Tool-Labs: Install GNU automake on Tool Labs - https://phabricator.wikimedia.org/T119870#1838050 (10zhuyifei1999) 3NEW [12:58:26] 6Labs, 10Tool-Labs: Install GNU automake on Tool Labs - https://phabricator.wikimedia.org/T119870#1838068 (10valhallasw) ``` valhallasw@tools-bastion-01:~$ automake automake: error: 'configure.ac' is required valhallasw@tools-bastion-01:~$ aclocal aclocal: error: 'configure.ac' is required valhallasw@tools-bas... [13:06:45] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install GNU automake on Tool Labs - https://phabricator.wikimedia.org/T119870#1838076 (10zhuyifei1999) >>! In T119870#1838068, @valhallasw wrote: > ``` > valhallasw@tools-bastion-01:~$ automake > automake: error: 'configure.ac' is required > valhallasw@tools-bastion-01:... [13:09:08] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install GNU automake on Tool Labs - https://phabricator.wikimedia.org/T119870#1838077 (10zhuyifei1999) And puppet says nothing about automake: https://github.com/wikimedia/operations-puppet/search?utf8=%E2%9C%93&q=automake Just another not-yet-puppetized package installed? [13:09:12] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install GNU automake on Tool Labs - https://phabricator.wikimedia.org/T119870#1838078 (10valhallasw) ``` valhallasw@tools-bastion-01:~$ aptitude why autoconf i dh-autoreconf Depends autoconf valhallasw@tools-bastion-01:~$ aptitude why dh-autoreconf Unable to find a re... [14:26:15] hi, i created a tool without actually checking if anyone wrote a similar one before... the tool lets me see the wikipedia edit in which a given word in an article was introduced, like "blame" in version control systems (but with a word resolution). is there anything like this already around? something like this: http://kolos.math.uni.lodz.pl/~d33tah/blame.html - if it's new, i could try to make this [14:26:17] tool more user friendly [14:29:16] d33tah: https://en.wikipedia.org/wiki/Wikipedia:WikiBlame [14:30:03] but that one searches for a specific word instead of being a real 'blame' page, I think [14:30:27] there have been some other attempts, but I don't know if they are still online [14:30:29] valhallasw`cloud: thanks. it looks like there's no handy tool that does what i did then [14:32:33] d33tah: I remember a version where it was marked on the page itself; pieces that belonged to one edit had one background color, and on hover there was more information [14:32:52] same idea, slightly different interface [14:33:13] valhallasw`cloud: yeah, i was thinking of adding the on-hover information as well [15:47:12] 6Labs, 10Incident-20150617-LabsNFSOutage: Labs: rewrite remaining labstore* scripts - https://phabricator.wikimedia.org/T102520#1838240 (10coren) [15:47:12] 6Labs, 10Incident-20150617-LabsNFSOutage: Rewrite sync-exports in python - https://phabricator.wikimedia.org/T118759#1838238 (10coren) 5Open>3declined After quite a bit of consideration (and some tentative code), it's not immediately obvious that rewriting //that// script in python is worthwhile or even us... [15:48:52] 6Labs, 10Incident-20150617-LabsNFSOutage: Labs: rewrite remaining labstore* scripts - https://phabricator.wikimedia.org/T102520#1838242 (10coren) 5Open>3Resolved a:3coren sync-exports would actually be made //less// clear and terse were it written in python, so rewriting it just because is not useful. T... [15:58:01] 6Labs, 10Labs-Infrastructure: Labstore primary needs more frequent cleanup of snapshots - https://phabricator.wikimedia.org/T109176#1838273 (10chasemp) a:5coren>3yuvipanda [16:11:55] 6Labs: Provide a way to have homedirs on a separate LVM volume instead of NFS - https://phabricator.wikimedia.org/T100274#1838323 (10chasemp) p:5Triage>3Normal [16:15:29] hexmode, Krenair, I don’t understand about https://phabricator.wikimedia.org/T117553 [16:15:58] Am I correct in understanding that the bug is “When I try to create an account and that account already exists, the error message is confusing”? [16:18:28] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: DB error when trying to create an account on wikitech - https://phabricator.wikimedia.org/T117553#1838347 (10Andrew) The title of this bug makes it look like account creation is failing -- but in reali... [16:18:38] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: DB error when trying to create an account on wikitech - https://phabricator.wikimedia.org/T117553#1838348 (10Andrew) p:5Unbreak!>3Low [16:18:56] 6Labs, 10Labs-Infrastructure, 7Database: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#1838349 (10chasemp) p:5Triage>3Low [16:19:06] 6Labs, 10Labs-Infrastructure, 7Database: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` - https://phabricator.wikimedia.org/T70942#1838352 (10chasemp) p:5Triage>3Low [16:19:15] 6Labs, 10Labs-Infrastructure, 10xTools-on-Labs: Database upgrade MariaDB 10: Lock wait timeouts / deadlocks in a row - https://phabricator.wikimedia.org/T70753#1838353 (10chasemp) p:5Triage>3Low [16:19:22] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: Weird error message when creating a wikitech account that already exists - https://phabricator.wikimedia.org/T117553#1838356 (10Andrew) [16:19:31] 6Labs, 10Wikimedia-Labs-General: Report when an instance has finished its initial Puppet run - https://phabricator.wikimedia.org/T70508#1838358 (10chasemp) p:5Triage>3Normal [16:20:19] 6Labs, 10Labs-Infrastructure: WMFLabs: "service diamond help" incorrectly says init.d is invoked and suggests itself as alternative - https://phabricator.wikimedia.org/T70442#1838362 (10chasemp) 5Open>3declined a:3chasemp [16:20:30] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Wikitech often loses track of internal openstack/nova session - https://phabricator.wikimedia.org/T101199#1838365 (10chasemp) p:5Triage>3Normal [16:20:48] 6Labs: Investigate and remove NFS mounts in the snuggle project - https://phabricator.wikimedia.org/T102680#1838367 (10chasemp) p:5Triage>3Normal [16:21:09] 6Labs: Kill NFS in openstack project - https://phabricator.wikimedia.org/T102672#1838369 (10chasemp) p:5Triage>3Low [16:21:36] 6Labs, 10Labs-Infrastructure: Disable creation of new Ubuntu Precise instances outside of whitelisted projects - https://phabricator.wikimedia.org/T102666#1838371 (10chasemp) p:5Triage>3Normal [16:22:38] 6Labs: Do a VM cleanup day - https://phabricator.wikimedia.org/T119476#1838378 (10chasemp) [16:22:40] 6Labs: Garbage collect unmaintained/unused instances - https://phabricator.wikimedia.org/T102409#1838379 (10chasemp) [16:22:47] 6Labs: Garbage collect unmaintained/unused instances - https://phabricator.wikimedia.org/T102409#1838381 (10chasemp) p:5Triage>3High [16:23:01] 6Labs: Find alternative solutions for video project's use of NFS - https://phabricator.wikimedia.org/T102402#1838383 (10chasemp) p:5Triage>3Normal [16:23:30] 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1838387 (10chasemp) p:5Triage>3High [16:23:55] 6Labs: Investigate reducing use of NFS in cvn project - https://phabricator.wikimedia.org/T102370#1838390 (10chasemp) p:5Triage>3Low [16:24:06] 6Labs, 5Patch-For-Review: Remove NFS completely from dynamicproxy project - https://phabricator.wikimedia.org/T102369#1838392 (10chasemp) p:5Triage>3Low [16:24:17] 6Labs, 10Incident-20150617-LabsNFSOutage: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1838394 (10chasemp) p:5Triage>3High [16:24:51] 6Labs, 7Database: write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb - https://phabricator.wikimedia.org/T106151#1838395 (10chasemp) p:5Triage>3Lowest [16:25:07] 6Labs: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#1838397 (10chasemp) p:5Triage>3High [16:25:23] 6Labs: Make an evacuation plan for labs instances - https://phabricator.wikimedia.org/T106144#1838403 (10chasemp) p:5Triage>3High [16:25:38] 6Labs, 10Labs-Infrastructure: Instance names with underscores are weirdly broken - https://phabricator.wikimedia.org/T105927#1838405 (10chasemp) p:5Triage>3Lowest [16:25:54] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1838407 (10chasemp) p:5Triage>3High [16:26:12] 6Labs, 10Tool-Labs, 6Developer-Relations, 6WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#1838409 (10coren) >>! In T87730#1577350, @Aklapper wrote: > I'm also wondering how much this whole process is (not) similar/comparable to https://www.me... [16:26:18] 6Labs: Delete SuchABot Account - https://phabricator.wikimedia.org/T105384#1838410 (10chasemp) p:5Triage>3Lowest [16:29:03] 6Labs, 10wikitech.wikimedia.org: fix apache log rotation on silver - https://phabricator.wikimedia.org/T119882#1838415 (10Andrew) 3NEW a:3Andrew [16:29:12] 6Labs, 10wikitech.wikimedia.org: fix apache log rotation on silver - https://phabricator.wikimedia.org/T119882#1838424 (10Andrew) p:5Triage>3Normal [16:30:16] 6Labs: Investigate per-project open security group policy - https://phabricator.wikimedia.org/T104894#1838433 (10chasemp) 5Open>3Invalid a:3chasemp AFAIK security rules are inbound iptables rulesets chained on input per instance. Which means its possible to limit visibility between VM's in the project. [16:30:24] 6Labs, 10Labs-Infrastructure, 6operations, 5Patch-For-Review: Update tag and racktables for holmium: rename to labservices1002. - https://phabricator.wikimedia.org/T119533#1838437 (10Andrew) [16:30:39] 6Labs, 10Salt: fix labs jessie instances to have correct salt version - https://phabricator.wikimedia.org/T104849#1838440 (10chasemp) p:5Triage>3Low [16:31:23] 6Labs, 10Wikibugs, 10grrrit-wm: puppetize grrrit-wm and wikibugs - https://phabricator.wikimedia.org/T104616#1838442 (10chasemp) p:5Triage>3Low [16:31:32] 6Labs, 10Labs-Infrastructure: Don't rely on wikitech API for production services - https://phabricator.wikimedia.org/T104575#1838444 (10chasemp) p:5Triage>3Normal [16:31:41] 6Labs, 10Labs-Infrastructure: Support reverse dns for public labs IPs - https://phabricator.wikimedia.org/T104521#1838446 (10chasemp) p:5Triage>3Normal [16:31:56] 6Labs, 10Labs-Infrastructure: Switch to horizon/designate/pdns/mysql for labs public dns - https://phabricator.wikimedia.org/T104520#1838450 (10chasemp) p:5Triage>3Normal [16:32:38] 6Labs: Cleanup ldap/files/scripts - https://phabricator.wikimedia.org/T104342#1838453 (10chasemp) p:5Triage>3Lowest I can't grok what this is in regards to from the title or description. [16:33:24] 6Labs: strace sshd to figure out which filesystems are hit on login - https://phabricator.wikimedia.org/T104327#1838456 (10chasemp) 5Open>3Invalid a:3chasemp I am in agreement but since this was so long ago and there is no historical record of the incident we will have to wait for similar happenings to dig... [16:33:36] 6Labs: Provide a simple way to backup arbitrary files from instances - https://phabricator.wikimedia.org/T104206#1838460 (10chasemp) p:5Triage>3Normal [16:33:52] 6Labs: Send alert emails to any project admin responsible for an instance with broken puppet - https://phabricator.wikimedia.org/T104199#1838462 (10chasemp) p:5Triage>3Low [16:34:22] 6Labs, 5Patch-For-Review: Simple method to have a per-project debian repository - https://phabricator.wikimedia.org/T104194#1838466 (10chasemp) 5Open>3Resolved a:3chasemp [16:34:23] 6Labs: Find replacements for various things that people were using NFS for but should not have been (Tracking) - https://phabricator.wikimedia.org/T104193#1838468 (10chasemp) [16:34:32] 6Labs: Find replacements for various things that people were using NFS for but should not have been (Tracking) - https://phabricator.wikimedia.org/T104193#1838470 (10chasemp) p:5Triage>3Normal [16:35:01] 6Labs, 5Patch-For-Review: Nested ".d" dirs in /etc/apt/ - https://phabricator.wikimedia.org/T104019#1838473 (10chasemp) p:5Triage>3Low [16:35:52] 6Labs, 6Discovery, 10Maps: *.tiles.wmflabs.org should be served over https - https://phabricator.wikimedia.org/T103801#1838476 (10chasemp) p:5Triage>3Low [16:36:04] 6Labs, 6operations, 10wikitech.wikimedia.org, 7IPv6: Set IPv6 PTR for wikitech-static - https://phabricator.wikimedia.org/T103621#1838478 (10chasemp) p:5Triage>3Lowest [16:36:25] 6Labs: Evaluate a 'cluster solution' for use on Tool Labs - https://phabricator.wikimedia.org/T106475#1838480 (10chasemp) 5Open>3Resolved a:3chasemp [16:36:34] 6Labs, 10Labs-Infrastructure, 6operations, 5Patch-For-Review: rename holmium to labservices1002 - https://phabricator.wikimedia.org/T106303#1838483 (10chasemp) p:5Triage>3Normal [16:36:45] 6Labs, 7Puppet: Move all labs-only puppet roles to manifests/role/labs - https://phabricator.wikimedia.org/T107167#1838485 (10chasemp) p:5Triage>3Normal [16:36:58] 6Labs: Measure capacity and utilization of labvirt**** servers - https://phabricator.wikimedia.org/T107067#1838488 (10chasemp) p:5Triage>3High [16:37:15] 6Labs, 10wikitech.wikimedia.org, 7Database: SemanticMediaWiki tries to create temporary tables, but can't as wikiuser is restricted - https://phabricator.wikimedia.org/T110981#1838490 (10chasemp) p:5Triage>3Low [16:37:33] 6Labs, 10Tool-Labs-tools-Other, 6Commons: Provide service to filter over categorization from a list of Commons categories - https://phabricator.wikimedia.org/T110833#1838492 (10chasemp) p:5Triage>3Low [16:37:49] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: OpenStackManager special pages not recognizing user permissions across all instances/projects - https://phabricator.wikimedia.org/T110775#1838494 (10chasemp) p:5Triage>3Low [16:38:04] 6Labs: Prevent spurious "Fingerprint changed" warning during nova-network outages - https://phabricator.wikimedia.org/T110508#1838496 (10chasemp) p:5Triage>3Normal [16:38:20] 6Labs, 10Labs-Infrastructure: cloud-init stacktrace on Precise instance first boot - https://phabricator.wikimedia.org/T110304#1838498 (10chasemp) p:5Triage>3Low [16:38:56] 6Labs, 10Wikimedia-Bugzilla: Add a link to the Phabricator task for bugs on bugs.wmflabs.org - https://phabricator.wikimedia.org/T109840#1838501 (10chasemp) p:5Triage>3Lowest [16:39:05] 6Labs: Cleanup leftover lucid instances on Labs - https://phabricator.wikimedia.org/T113199#1838503 (10chasemp) p:5Triage>3Normal [16:39:22] 6Labs, 10Tool-Labs-tools-meetbot, 10Labs-Infrastructure: restore Meetbot logs from around 2015-06 lost in NFS outage - https://phabricator.wikimedia.org/T113000#1838505 (10chasemp) p:5Triage>3Normal [16:39:34] 6Labs, 10Tool-Labs, 10Wiki-Loves-Monuments-Database: Consider using spatial indexes for faster geographical searches - https://phabricator.wikimedia.org/T112563#1838507 (10chasemp) p:5Triage>3Low [16:39:48] 6Labs, 10Salt: check that puppet certs/salt keys are deleted when they should be - https://phabricator.wikimedia.org/T112524#1838509 (10chasemp) p:5Triage>3Normal [16:40:01] 6Labs, 10Salt: lab instances with broken salt which need to be fixed by instance owners - https://phabricator.wikimedia.org/T112512#1838511 (10chasemp) p:5Triage>3Lowest [16:40:16] 6Labs, 10Tool-Labs: Setup DNS for kubernetes services - https://phabricator.wikimedia.org/T111914#1838514 (10chasemp) p:5Triage>3Normal [16:40:47] 6Labs, 10Tool-Labs, 5Patch-For-Review: Improve puppet run time - https://phabricator.wikimedia.org/T116813#1838518 (10chasemp) p:5Triage>3Normal really neat ticket [16:41:02] 6Labs, 10Tool-Labs: Install a PHP opcode cache on tool-labs - https://phabricator.wikimedia.org/T116780#1838520 (10chasemp) p:5Triage>3Low [16:41:13] 6Labs, 10Tool-Labs: Delete "toolserver" tool - https://phabricator.wikimedia.org/T116389#1838524 (10chasemp) p:5Triage>3Lowest [16:41:24] 6Labs, 10Tool-Labs: Allow scfc and valhallasw to ssh in as root - https://phabricator.wikimedia.org/T116156#1838526 (10chasemp) p:5Triage>3High [16:41:36] 6Labs: Need help getting a uWSGI Python-Flask Labs tool to work - https://phabricator.wikimedia.org/T115920#1838531 (10chasemp) 5Open>3Resolved a:3chasemp [16:42:02] 6Labs, 10Labs-Infrastructure: invisible-unicorn (dynamicproxy) should provide an easy way to see where a host routes without knowing the project - https://phabricator.wikimedia.org/T115752#1838534 (10chasemp) p:5Triage>3Normal [16:42:12] 6Labs, 7Database: Data missing from June 11/12 on s3.labsdb - https://phabricator.wikimedia.org/T115517#1838536 (10chasemp) p:5Triage>3Low [16:42:30] 6Labs, 10Tool-Labs: W: GPG error: http://toolsbeta-webproxy-01 trusty-toolsbeta InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F92A406A43C012D0 - https://phabricator.wikimedia.org/T115385#1838538 (10chasemp) p:5Triage>3Normal [16:42:40] 6Labs, 10Tool-Labs: create monitoring for webservicewatcher - https://phabricator.wikimedia.org/T115227#1838540 (10chasemp) p:5Triage>3Normal [16:43:20] 6Labs, 10Tool-Labs: webservicewatcher does not log exceptions to user log - https://phabricator.wikimedia.org/T115226#1838549 (10chasemp) p:5Triage>3Low [16:43:36] 6Labs, 7Database: Lots of rows are missing from enwiki_p.`revision` - https://phabricator.wikimedia.org/T115207#1838551 (10chasemp) p:5Triage>3Low [16:43:46] 6Labs, 6operations: RDNS for 10.68.18.65 resolves to two different instances - https://phabricator.wikimedia.org/T115194#1838553 (10chasemp) p:5Triage>3High [16:44:04] 6Labs, 10Labs-Team-Backlog, 6operations: Make sure that the 'secret' repo in self hosted puppetmasters is back-upable - https://phabricator.wikimedia.org/T115177#1838556 (10chasemp) p:5Triage>3Low [16:44:15] 6Labs, 10Tool-Labs: Define policy for tools-local packages - https://phabricator.wikimedia.org/T114645#1838558 (10chasemp) p:5Triage>3Low [16:44:23] 6Labs, 10Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#1838560 (10chasemp) p:5Triage>3Normal [16:44:32] 6Labs, 6operations: Cleanup / clarify labstore2001 - https://phabricator.wikimedia.org/T116972#1838562 (10chasemp) p:5Triage>3High [16:44:45] 6Labs, 10Labs-Infrastructure, 6operations: Unable to connect both redundant labstores to the shelves in parallel - https://phabricator.wikimedia.org/T117453#1838563 (10chasemp) p:5Triage>3High [16:45:15] 6Labs, 10MediaWiki-Vagrant: Wikimedia SMTP server does not work with Labs-Vagrant - https://phabricator.wikimedia.org/T117391#1838567 (10chasemp) p:5Triage>3Lowest [16:45:37] 6Labs, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Hook SMWParseData::onTitleMoveComplete has invalid call signature; Parameter 3 to SMWParseData::onTitleMoveComplete() expected to be a reference, value given - https://phabricator.wikimedia.org/T118649#1838569 (10chasemp) p:5Triage>3Low [16:45:49] 6Labs, 10Tool-Labs: Backport newer tesseract-ocr-* packages from sid - https://phabricator.wikimedia.org/T118643#1838571 (10chasemp) p:5Triage>3Low [16:46:42] 6Labs, 10Labs-Infrastructure: Investigate decommissioning labcontrol2001 - https://phabricator.wikimedia.org/T118591#1838574 (10chasemp) p:5Triage>3High [16:47:27] 6Labs, 10Tool-Labs, 7Easy: Log CommitLimit to graphite - https://phabricator.wikimedia.org/T118465#1838580 (10chasemp) p:5Triage>3Lowest [16:47:34] 6Labs, 10Tool-Labs: Define base Wikimedia Docker container - https://phabricator.wikimedia.org/T118446#1838582 (10chasemp) p:5Triage>3Normal [16:47:41] 6Labs, 10Labs-Infrastructure, 10MediaWiki-Vagrant: InstantCommons stopped working on Labs-Vagrant, now lots of missing images. - https://phabricator.wikimedia.org/T118226#1838584 (10chasemp) p:5Triage>3Lowest [16:47:51] 6Labs, 10wikitech.wikimedia.org: "Edit with form" missing on a Tools access request page - https://phabricator.wikimedia.org/T118136#1838586 (10chasemp) p:5Triage>3Lowest [16:48:04] 6Labs, 10Tool-Labs: setup-tomcat does not work - https://phabricator.wikimedia.org/T118094#1838588 (10chasemp) p:5Triage>3Low [16:49:00] 6Labs, 10Tool-Labs: tools-service-01: Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install tools-manifest' returned 100 - https://phabricator.wikimedia.org/T119390#1838592 (10chasemp) 5Open>3Invalid a:3chasemp seems no longer? [16:49:12] 6Labs, 10Tool-Labs: Puppet disabled on tools-exec-cyberbot, tools-exec-gift - https://phabricator.wikimedia.org/T119389#1838595 (10chasemp) 5Open>3Resolved a:3chasemp [16:49:44] 6Labs, 10MediaWiki-extensions-OpenStackManager, 7Documentation, 7Puppet: doc.wikimedia.org puppet documentation for labs/lvm/srv.html gives a 404 - https://phabricator.wikimedia.org/T119329#1838597 (10chasemp) p:5Triage>3Lowest [16:49:54] 6Labs: Horizon - create security group - 0 length description issue - https://phabricator.wikimedia.org/T119313#1838599 (10chasemp) p:5Triage>3Normal [16:50:20] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install GNU automake on Tool Labs - https://phabricator.wikimedia.org/T119870#1838601 (10chasemp) p:5Triage>3Low [16:50:27] 6Labs: Figure out how to deal with SSL cert issues for kubernetes masters - https://phabricator.wikimedia.org/T119814#1838603 (10chasemp) p:5Triage>3Normal [16:50:36] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review: Move labs-ns1 from labcontrol2001 to labcontrol1002 - https://phabricator.wikimedia.org/T119762#1838605 (10chasemp) p:5Triage>3Normal [16:51:14] 6Labs, 10Tool-Labs: Rounding and missing units in VMEM values on http://tools.wmflabs.org/?status create misleading values - https://phabricator.wikimedia.org/T119680#1838607 (10chasemp) p:5Triage>3Lowest [16:51:25] 6Labs, 10Labs-Infrastructure, 6operations, 5Patch-For-Review: Update tag and racktables for holmium: rename to labservices1002. - https://phabricator.wikimedia.org/T119533#1838609 (10chasemp) p:5Triage>3Normal [16:51:37] 6Labs, 7Database: Create a script that'll allow easier swatting of queries that're overloading labsdb - https://phabricator.wikimedia.org/T119479#1838611 (10chasemp) p:5Triage>3High [16:51:50] 6Labs, 6operations: Untangle labs/production roles from labs/instance roles - https://phabricator.wikimedia.org/T119401#1838613 (10chasemp) p:5Triage>3High [16:52:23] 6Labs, 10Tool-Labs: Make sure all packages are automatically upgraded - https://phabricator.wikimedia.org/T118939#1838618 (10chasemp) p:5Triage>3Low [16:52:35] 6Labs: Tools: Create a check that all web nodes are set as submit nodes - https://phabricator.wikimedia.org/T117906#1838620 (10chasemp) p:5Triage>3Normal [16:52:54] 6Labs: openocr project doesn't seem to work properly on wikitech - https://phabricator.wikimedia.org/T117341#1838623 (10chasemp) p:5Triage>3Low [16:53:06] 6Labs, 5Patch-For-Review: Install mailutils on tools - https://phabricator.wikimedia.org/T114073#1838625 (10chasemp) 5Open>3Resolved a:3chasemp [16:53:07] 6Labs, 10Tool-Labs, 7Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#1838627 (10chasemp) [16:53:22] 6Labs, 5Patch-For-Review: Kill ldapsupportlib.py - https://phabricator.wikimedia.org/T114063#1838628 (10chasemp) p:5Triage>3Normal [16:53:36] 6Labs, 10Tool-Labs: Customization of /etc/hosts deletes alias ubuntu.openstack.eqiad.wmflabs - https://phabricator.wikimedia.org/T113922#1838630 (10chasemp) p:5Triage>3Low [16:53:42] 6Labs: `pr_index`to be replicated to Labs public databases - https://phabricator.wikimedia.org/T113842#1838632 (10chasemp) p:5Triage>3Low [16:53:58] 6Labs, 10Labs-Infrastructure: Redirect https://toolserver.org/~magnus/ - https://phabricator.wikimedia.org/T113696#1838636 (10chasemp) p:5Triage>3Low [16:54:27] 6Labs: Instances spontaneously suspended - https://phabricator.wikimedia.org/T113646#1838638 (10chasemp) p:5Triage>3Low [16:54:39] 6Labs, 5Patch-For-Review: Disable multiarch support in all Labs precise instances - https://phabricator.wikimedia.org/T111760#1838640 (10chasemp) 5Open>3Resolved a:3chasemp [16:54:54] 6Labs: Labs proxy 502 should indicate where to ask for support - https://phabricator.wikimedia.org/T109078#1838643 (10chasemp) p:5Triage>3Normal [16:55:17] 6Labs, 10Tool-Labs: Tool Labs virt reboot checklist - https://phabricator.wikimedia.org/T108669#1838645 (10chasemp) p:5Triage>3Normal [17:02:29] Is there a puppet class for mediawiki? And can I disable puppet after I pulled the files? [17:03:47] Luke081515: yes, use https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs [17:04:01] Luke081515: and no, do not disable puppet - that will make your instance unusable in the long term [17:04:30] 6Labs, 10Wikimedia-Labs-General: Warning: session_write_close(): write failed: No space left on device (28) in /data/project/magnustools/public_html/php/oauth.php - https://phabricator.wikimedia.org/T75324#1838669 (10chasemp) 5Open>3Invalid a:3chasemp [17:04:34] yuvipanda: But if I don't disable the mediawiki puppet, puppet will overwrite custom code? [17:05:30] 6Labs, 10Wikimedia-Labs-General, 5Patch-For-Review: Puppet resets ownership of /srv/mediawiki in role::mediawiki-install::labs to root:root - https://phabricator.wikimedia.org/T74046#1838673 (10chasemp) 5Open>3declined a:3chasemp >>! In T74046#1668994, @gerritbot wrote: > Change 79955 abandoned by Yuv... [17:05:55] Luke081515: no [17:06:04] Luke081515: at least not if you follow that page :) [17:06:09] ah, ok [17:08:18] 6Labs, 10Labs-Infrastructure: Prevent puppet from creating local user when they are defined in LDAP - https://phabricator.wikimedia.org/T73480#1838701 (10chasemp) p:5Triage>3High [17:08:36] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: cn=proxyagent,ou=profile,dc=wikimedia,dc=org should probably not allow unindexed searches - https://phabricator.wikimedia.org/T73130#1838703 (10chasemp) p:5Triage>3Normal [17:09:03] 6Labs, 10Wikimedia-Labs-General: Swap two shell usernames - https://phabricator.wikimedia.org/T73083#1838706 (10chasemp) p:5Triage>3High still needed? [17:09:36] 6Labs, 10Tool-Labs: add user filter to status page - https://phabricator.wikimedia.org/T75384#1838709 (10chasemp) p:5Triage>3Lowest [17:10:27] 6Labs, 7Monitoring, 5Patch-For-Review, 7Shinken, 7Upstream: shinken.wmflabs.org redirects on https-login to http - https://phabricator.wikimedia.org/T85326#1838711 (10chasemp) p:5Triage>3High [17:11:42] puppet is a great thing :) [17:12:54] 6Labs, 10LabsDB-Auditor: Verify that public views are defined the way they should be (matching whitelist/greylist) - https://phabricator.wikimedia.org/T85473#1838715 (10chasemp) p:5Triage>3Low [17:13:09] 6Labs, 10LabsDB-Auditor: Make labsdb views consistent with currently in-use set of tables / columns in production - https://phabricator.wikimedia.org/T85866#1838717 (10chasemp) p:5Triage>3Low [17:13:35] 6Labs: Allow everyone in ops group in LDAP to login to all Labs instances - https://phabricator.wikimedia.org/T87094#1838719 (10chasemp) p:5Triage>3High [17:14:01] yuvipanda: One question left: step 5: how to pick up profile.d? [17:14:07] 6Labs: SMTP server on toolserver.org not responding - https://phabricator.wikimedia.org/T87119#1838731 (10chasemp) 5Open>3Invalid a:3chasemp [17:14:16] Luke081515: log out and log in, I think [17:14:27] ah, ok [17:14:29] 6Labs, 10Labs-Infrastructure: Weird state of /data/project for dumps (semi-missing files) - https://phabricator.wikimedia.org/T87224#1838735 (10chasemp) p:5Triage>3Normal [17:16:13] 6Labs: Two instances with same name - https://phabricator.wikimedia.org/T89931#1838749 (10chasemp) 5Open>3declined a:3chasemp >>! In T89931#1720552, @Krenair wrote: > I tried to create an instance with the same name as the existing one (in the same project) and it failed. [17:16:14] 6Labs, 5Patch-For-Review: Disable multiarch support in all Labs precise instances - https://phabricator.wikimedia.org/T111760#1838754 (10scfc) 5Resolved>3Open The referenced change disabled multiarch support only for `aptly::client` instances. [17:16:27] 6Labs, 5Patch-For-Review: Disable multiarch support in all Labs precise instances - https://phabricator.wikimedia.org/T111760#1838756 (10scfc) a:5chasemp>3None [17:16:35] 6Labs: Document mounting additional storage in labs add instance page - https://phabricator.wikimedia.org/T90457#1838758 (10chasemp) p:5Triage>3Low [17:17:16] 6Labs, 10Wikimedia-Labs-General, 10Wikidata: Need a way to test with data set reasonably close to production - https://phabricator.wikimedia.org/T91131#1838763 (10chasemp) p:5Triage>3Normal how big is big in concrete terms? [17:17:32] 6Labs, 7Documentation, 7Puppet: Missing documentation for labs puppet roles - https://phabricator.wikimedia.org/T91770#1838767 (10chasemp) p:5Triage>3Lowest [17:18:10] 6Labs, 10Labs-Infrastructure: New instances don't always get their /data and /home project mounted from NFS - https://phabricator.wikimedia.org/T91822#1838770 (10chasemp) p:5Triage>3High it's happened to me before as well [17:18:29] 6Labs: Instances with no name or project associated on wikitech - https://phabricator.wikimedia.org/T91922#1838772 (10chasemp) p:5Triage>3Normal [17:20:23] 6Labs, 6Design Research Backlog: Public IP and Wildcard DNS for REFLEX project - https://phabricator.wikimedia.org/T92273#1838783 (10chasemp) p:5Triage>3Low [17:21:25] 6Labs: labstore1001: sysctl settings improperly(?) applied on boot - https://phabricator.wikimedia.org/T92623#1838786 (10chasemp) a:3coren is this still an ongoing issue? [17:21:31] 6Labs: labstore1001: sysctl settings improperly(?) applied on boot - https://phabricator.wikimedia.org/T92623#1838789 (10chasemp) p:5Triage>3Normal [17:22:44] 6Labs, 5Patch-For-Review: Disable multiarch support in all Labs precise instances - https://phabricator.wikimedia.org/T111760#1838790 (10chasemp) >>! In T111760#1838754, @scfc wrote: > The referenced change disabled multiarch support only for `aptly::client` instances. thanks [17:23:01] 6Labs: labstore1001: sysctl settings improperly(?) applied on boot - https://phabricator.wikimedia.org/T92623#1838791 (10coren) 5Open>3Invalid Probably not - the issue did not reoccur and there is no indication that the sysctl setting should not apply. ... //and// the box where that happened has long been u... [17:23:36] 6Labs, 6operations: One instance hammering on NFS should not make it unavailable to everyone else - https://phabricator.wikimedia.org/T95766#1838802 (10chasemp) p:5Triage>3High [17:24:30] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: novainstances API: 'niregion' parameter should be optional and/or discoverable - https://phabricator.wikimedia.org/T94514#1838810 (10chasemp) p:5Triage>3Normal [17:24:43] 6Labs, 10Incident-20150331-LabsNFS-Filesystem-Switch: Create a simple checklist to follow for announcing / doing planned maintenance (on labs) - https://phabricator.wikimedia.org/T94608#1838813 (10chasemp) p:5Triage>3High [17:24:50] chasemp: Coren andrewbogott btw, https://phabricator.wikimedia.org/T119845 (NFS broken for new instances) [17:25:21] yuvipanda: Hm. Odd. I'll be looking at it first thing after meetings. [17:25:23] Yeah, I saw that scroll by this morning but haven’t looked yet. Coren’s doing a lot of refactoring in that area, I think? [17:25:25] 6Labs: NFS broken for new labs instances - https://phabricator.wikimedia.org/T119845#1838817 (10chasemp) p:5High>3Unbreak! a:3coren [17:25:43] 6Labs, 10Incident-20150331-LabsNFS-Filesystem-Switch: Create scripts to help stagger restarts of labs VMs by different criteria - https://phabricator.wikimedia.org/T94613#1838820 (10chasemp) p:5Triage>3Low [17:25:44] andrewbogott: Nothing on labstore since last Tue. [17:25:47] chasemp: thank you for triaging! [17:25:54] 6Labs: Consider creating big storage/low RAM images - https://phabricator.wikimedia.org/T95731#1838822 (10chasemp) p:5Triage>3Normal [17:26:05] Coren: I could believe it’s been broken that long, since new instance + NFS is a small venn overlap [17:26:12] valhallasw`cloud: :) it's slow going [17:26:17] andrewbogott: True that. [17:26:52] tools-worker-04 is the thing I had, and feel free to play with it - I've depooled it [17:27:10] yuvipanda: Good, having a victim simplifies things greatly. [17:27:27] ok [17:27:37] 6Labs: Move ldap host-record creation out of OpenStackManager and into sink - https://phabricator.wikimedia.org/T96677#1838829 (10chasemp) p:5Triage>3Normal [17:28:07] 6Labs: Create a system to automatically flag resource consumer tasks for admins of Labs - https://phabricator.wikimedia.org/T97184#1838831 (10chasemp) p:5Triage>3Low [17:28:39] 6Labs: Labs puppetmaster says: Could not resolve 10.68.16.210: no name for 10.68.16.210 - https://phabricator.wikimedia.org/T97250#1838833 (10chasemp) p:5Triage>3High [17:29:01] 6Labs, 7Database: Pagelinks table contains a row having pl_from = 0 - https://phabricator.wikimedia.org/T98110#1838842 (10chasemp) p:5Triage>3Low [17:29:26] 6Labs, 6operations, 10wikitech.wikimedia.org, 7HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#1838847 (10chasemp) p:5Triage>3Lowest [17:29:34] yuvipanda: Can you help me with the security groups? which protocol and which cidr range? [17:29:40] 6Labs: Fix labs lvm to not run script every puppet run - https://phabricator.wikimedia.org/T99823#1838849 (10chasemp) p:5Triage>3Normal [17:29:59] 6Labs: Puppet errors on newly created instances - https://phabricator.wikimedia.org/T100108#1838851 (10chasemp) p:5Triage>3Normal [17:31:06] Luke081515: tcp, cidr range is 10.0.0.0/8 [17:31:31] thanks :) [17:31:53] 6Labs: Reading dumps is very slow in grid engine - https://phabricator.wikimedia.org/T100227#1838853 (10chasemp) 5Open>3declined a:3chasemp [17:32:49] 6Labs: Document, explain, diagram labs vlans and network setup - https://phabricator.wikimedia.org/T100529#1838866 (10chasemp) p:5Triage>3Normal [17:32:54] it works :) [17:32:58] \o/ [17:33:08] Luke081515: bd808 did all the hard work on that one [17:33:10] 6Labs, 10wikitech.wikimedia.org: Upgrade SMW to 1.9 or later - https://phabricator.wikimedia.org/T62886#1838871 (10chasemp) p:5Triage>3Normal [17:34:20] 6Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: Update Semantic Forms - https://phabricator.wikimedia.org/T63104#1838874 (10chasemp) p:5Triage>3Low [17:35:47] 6Labs, 10Labs-Infrastructure: Remove shell user "80686" - https://phabricator.wikimedia.org/T63967#1838881 (10chasemp) p:5Triage>3Normal [17:36:06] 6Labs, 10Wikimedia-Labs-General: Community-maintained projects are hard to track - https://phabricator.wikimedia.org/T64837#1838883 (10chasemp) p:5Triage>3High [17:38:39] 6Labs, 10Labs-Infrastructure: Instance console does not gives output / keystroke access - https://phabricator.wikimedia.org/T64847#1838894 (10chasemp) p:5Triage>3Normal [17:39:01] 6Labs, 10Wikimedia-Labs-General: proxy'd labs MediaWiki instance times out contacting itself to runjobs - https://phabricator.wikimedia.org/T65338#1838897 (10chasemp) 5Open>3declined a:3chasemp [17:40:11] 6Labs, 10Labs-Infrastructure: Start pdns after opendj - https://phabricator.wikimedia.org/T65717#1838902 (10chasemp) p:5Triage>3Normal [17:40:31] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: unhelpful error message from Special:NovaInstance when you reach instance quota - https://phabricator.wikimedia.org/T65794#1838905 (10chasemp) p:5Triage>3Lowest [17:40:47] 6Labs, 10Labs-Infrastructure: Monitor Puppet "stability" - https://phabricator.wikimedia.org/T66964#1838907 (10chasemp) p:5Triage>3Low [17:41:16] 6Labs, 10Labs-Infrastructure: set apt to Wikimedia mirror instead of http://nova.clouds.archive.ubuntu.com/ubuntu/ - https://phabricator.wikimedia.org/T68121#1838912 (10chasemp) p:5Triage>3Normal [17:41:45] 6Labs, 7Security-Other: Labs makes use of a SHA-1 Certificate - https://phabricator.wikimedia.org/T101618#1838915 (10chasemp) p:5Triage>3Normal [17:42:38] 6Labs: Provide all labs users with username / passwords for the Postgres database - https://phabricator.wikimedia.org/T101661#1838917 (10chasemp) p:5Triage>3Normal [17:43:50] 6Labs, 6operations, 10wikitech.wikimedia.org: Expand list of people who can create new Labs project - https://phabricator.wikimedia.org/T101688#1838923 (10chasemp) p:5Triage>3Low [17:44:43] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad: Locate and assign some MD1200 shelves for proper testing of labstore1002 - https://phabricator.wikimedia.org/T101741#1838932 (10chasemp) p:5Triage>3High What's going on with this? Title makes it seems like no proper testing has been done. [17:46:00] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad: labstore1002 issues while trying to reboot - https://phabricator.wikimedia.org/T98183#1838943 (10chasemp) So...should we close this as invalid or ? [17:46:44] 6Labs: Make a menu of potential new labs features, invite comments from users - https://phabricator.wikimedia.org/T101769#1838944 (10chasemp) [17:46:45] 6Labs: Document support levels for tools and labs projects - https://phabricator.wikimedia.org/T116598#1838945 (10chasemp) [17:46:52] 6Labs: Make a menu of potential new labs features, invite comments from users - https://phabricator.wikimedia.org/T101769#1347310 (10chasemp) going to fold this in to support levels convo [17:47:06] 6Labs, 6operations: Icinga alert for labnet1001 for conntrack saturation graphite check - https://phabricator.wikimedia.org/T101980#1838954 (10chasemp) p:5Triage>3Normal [17:48:56] 6Labs, 10Tool-Labs-tools-Other, 6operations: Move geohack to production - https://phabricator.wikimedia.org/T102960#1838962 (10chasemp) p:5Triage>3Low [17:50:16] 6Labs, 6operations: Have a cron job delete files that haven't been modified in the last X days / months in /data/scratch - https://phabricator.wikimedia.org/T103084#1838968 (10chasemp) p:5Triage>3Low [17:50:46] 6Labs, 10Salt: clean up old ec2id-based salt keys on labs - https://phabricator.wikimedia.org/T103089#1838975 (10chasemp) p:5Triage>3Normal [17:51:14] 6Labs: Disable NFS on the orgcharts project - https://phabricator.wikimedia.org/T103137#1838977 (10chasemp) p:5Triage>3Low [17:51:39] 6Labs: Investigate NFS alternatives to the wikistream project - https://phabricator.wikimedia.org/T103148#1838979 (10chasemp) p:5Triage>3Low [17:51:48] 6Labs: Investigate account-creation-assistance project's use of NFS and look for replacements - https://phabricator.wikimedia.org/T103156#1838981 (10chasemp) p:5Triage>3Low [17:51:57] 6Labs, 10Incident-20150617-LabsNFSOutage: Labs: investigate alternatives to maps' storage requirements - https://phabricator.wikimedia.org/T103264#1838983 (10chasemp) p:5Triage>3Low [17:52:28] 6Labs, 5Patch-For-Review: Support connections from bastion to other hosts - https://phabricator.wikimedia.org/T103552#1838986 (10chasemp) p:5Triage>3Normal [17:55:47] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure, 7LDAP: cn=proxyagent,ou=profile,dc=wikimedia,dc=org should probably not allow unindexed searches - https://phabricator.wikimedia.org/T73130#1839004 (10Krenair) [17:57:03] 6Labs: Find and clean up instances that are unreachable by ssh - https://phabricator.wikimedia.org/T103522#1839005 (10chasemp) [17:57:04] 6Labs: Garbage collect unmaintained/unused instances - https://phabricator.wikimedia.org/T102409#1839007 (10chasemp) [17:57:26] 6Labs, 5Patch-For-Review: Build proper monitoring for making sure that processes that need to run only once on one labstore only are running only once on one labstore only - https://phabricator.wikimedia.org/T106590#1839009 (10chasemp) p:5Triage>3Low [17:59:50] 6Labs, 10Education-Program-Dashboard: Find stable home for WMF dashboard server - https://phabricator.wikimedia.org/T107939#1839020 (10chasemp) 5Open>3Invalid a:3chasemp This needs a description that allows us to know what is needed. [18:00:40] 6Labs, 10Labs-Infrastructure, 6operations, 7Monitoring: labstore monitoring: NRPE: Command 'check_cleanup-snapshots-labstore-state' not defined - https://phabricator.wikimedia.org/T111211#1839029 (10chasemp) 5Open>3Invalid a:3chasemp [18:02:52] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad: labstore1002 issues while trying to reboot - https://phabricator.wikimedia.org/T98183#1839054 (10coren) No, we still haven't done serious testing of that hardware (viz. T101471) so at best labstore1002 is a dubiously reliable backup atm. This task re... [18:04:25] 6Labs, 10Wikimedia-Labs-General, 6Developer-Relations: Community-maintained projects are hard to track - https://phabricator.wikimedia.org/T64837#1839065 (10Aklapper) [18:04:43] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad: Locate and assign some MD1200 shelves for proper testing of labstore1002 - https://phabricator.wikimedia.org/T101741#1839066 (10coren) >>! In T101741#1838932, @chasemp wrote: > What's going on with this? Title makes it seems like no proper testing ha... [18:07:42] 6Labs: Measure capacity and utilization of labs services (Tracking) - https://phabricator.wikimedia.org/T107066#1839079 (10chasemp) p:5Triage>3High [18:08:03] 6Labs, 6WMF-Legal: Toolserver relic files break attribution requirements for images - https://phabricator.wikimedia.org/T103965#1839081 (10chasemp) p:5Triage>3Normal [18:20:52] 6Labs, 10Wikimedia-Labs-General: Swap two shell usernames - https://phabricator.wikimedia.org/T73083#1839129 (10jimmyxu) Would appreciate. [18:29:50] 6Labs, 10wikitech.wikimedia.org: "Edit with form" missing on a Tools access request page - https://phabricator.wikimedia.org/T118136#1839169 (10Krenair) Okay, I know nothing about SMW/SF and I'm not going to become the guy around here who does, but: It fails at https://phabricator.wikimedia.org/diffusion/ESFO... [18:30:39] 6Labs: known_host key updating on virt* (and possibly elsewhere) - https://phabricator.wikimedia.org/T93748#1839170 (10chasemp) p:5Triage>3Low [18:31:11] 6Labs, 10Labs-Infrastructure: The package lists or status file could not be parsed or opened. - https://phabricator.wikimedia.org/T72981#1839179 (10chasemp) p:5Triage>3Normal Is still relevant? [18:31:16] 6Labs, 10Labs-Infrastructure, 7Database: Database upgrade MariaDB 10: Metadata access in INFORMATION_SCHEMA causes complete blocks - https://phabricator.wikimedia.org/T71182#1839184 (10chasemp) p:5Triage>3Low [18:31:25] 6Labs: Ignored file /etc/apt/apt.conf.d/20auto-upgrades.ucf-dist - https://phabricator.wikimedia.org/T110055#1839188 (10chasemp) p:5Triage>3Normal [18:31:38] 6Labs, 5Patch-For-Review: Disable multiarch support in all Labs precise instances - https://phabricator.wikimedia.org/T111760#1839190 (10chasemp) p:5Triage>3Normal [18:32:02] 6Labs, 10wikitech.wikimedia.org: Discrepancies in public IP instance lists between different wikitech UIs - https://phabricator.wikimedia.org/T62883#1839193 (10chasemp) p:5Triage>3High @andrew any clue why this is the case? seems very strange [18:32:10] 6Labs, 10Labs-Infrastructure: root@ should email project admins - https://phabricator.wikimedia.org/T61142#1839195 (10chasemp) p:5Triage>3High [18:32:28] 6Labs, 10Wikimedia-Labs-General: Labs account creation process seems non-obvious - https://phabricator.wikimedia.org/T61112#1839199 (10chasemp) 5Open>3declined a:3chasemp [18:32:47] 6Labs, 10Tool-Labs: jsub and utf8 - https://phabricator.wikimedia.org/T60784#1839203 (10chasemp) p:5Triage>3Normal [18:33:01] 6Labs, 10wikitech.wikimedia.org: Can't reset password on wikitech (Unicode passwords not accepted), due to LDAP/opendj? - https://phabricator.wikimedia.org/T58114#1839205 (10chasemp) p:5Triage>3Low [18:33:13] 6Labs, 10Wikimedia-Labs-General, 7Database: Provide dynamic report of differences between replica databases and production databases - https://phabricator.wikimedia.org/T57455#1839207 (10chasemp) p:5Triage>3Lowest [18:33:43] 6Labs, 10Labs-Infrastructure: add useful content to Wikitech:Shell - https://phabricator.wikimedia.org/T56697#1839212 (10chasemp) p:5Triage>3Lowest [18:34:20] 6Labs, 10Labs-Infrastructure, 7Database: Backups systems for userDBs hosted on replicas - https://phabricator.wikimedia.org/T56401#1839220 (10chasemp) p:5Triage>3Lowest [18:34:39] 6Labs, 10Wikimedia-Labs-General: UI/UX - project filter drop down selection box too narrow - https://phabricator.wikimedia.org/T54629#1839222 (10chasemp) p:5Triage>3Lowest [18:35:25] 6Labs, 10Labs-Infrastructure: Make the link to list of all Labs projects more visible - https://phabricator.wikimedia.org/T53889#1839225 (10chasemp) p:5Triage>3Lowest [18:36:29] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-General-or-Unknown, 10wikitech.wikimedia.org: Weird error message when creating a wikitech account that already exists - https://phabricator.wikimedia.org/T117553#1839230 (10Krenair) I doubt that's the actual bug, @andrew. [18:37:03] 6Labs, 10wikitech.wikimedia.org: Discrepancies in public IP instance lists between different wikitech UIs - https://phabricator.wikimedia.org/T62883#1839231 (10yuvipanda) Nova Resource is useless because SMW. The only real issue here would be that NovaAddress was blank for @scfc - is that still the case? [18:38:36] 6Labs, 6operations: Cleanup / clarify labstore2001 - https://phabricator.wikimedia.org/T116972#1839239 (10chasemp) [18:39:27] 6Labs, 10Wikimedia-Labs-General: Labs account creation process seems non-obvious - https://phabricator.wikimedia.org/T61112#1839243 (10MZMcBride) 5declined>3Resolved [18:39:31] 6Labs, 6operations, 5Patch-For-Review, 7Puppet: Self hosted puppetmaster is broken - https://phabricator.wikimedia.org/T119541#1839246 (10chasemp) 5Open>3Invalid a:3chasemp >>! In T119541#1833712, @akosiaris wrote: > I setup a new self hosted puppetmaster environment today and I did not meet this pro... [18:39:55] 6Labs, 10Wikimedia-Labs-General: Labs account creation process seems non-obvious - https://phabricator.wikimedia.org/T61112#636127 (10MZMcBride) This looks to have been fixed. There's now a "Create account" link at . [18:40:49] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1839254 (10chasemp) >>! In T117673#1813406, @Joe wrote: > FTR. this just happened to me with a newly-created instance with jessie; to my knowledge no prior machine with that name existed and... [18:42:23] 6Labs, 10Tool-Labs: Add monitoring for expected load issues on tool labs exec nodes - https://phabricator.wikimedia.org/T109732#1839265 (10chasemp) [18:45:03] 6Labs, 3ToolLabs-Goals-Q4: Allow labstores to hot or warm swap in case of failure - https://phabricator.wikimedia.org/T93589#1839270 (10chasemp) a:3coren seems to come with the other tasks discussed re: labstore in the meeting today so I'm tossing your way @coren [18:48:03] 6Labs, 6operations, 5Patch-For-Review, 7Puppet: Self hosted puppetmaster is broken - https://phabricator.wikimedia.org/T119541#1839284 (10fgiunchedi) I tried adding `role::puppet::self` to an existing trusty host and looks like it has worked [18:48:20] 6Labs, 3ToolLabs-Goals-Q4: Allow labstores to hot or warm swap in case of failure - https://phabricator.wikimedia.org/T93589#1839285 (10coren) 5Open>3declined >>! In T93589#1470031, @yuvipanda wrote: > @mark @coren are we still going to do this? Afaik attempts at this might have been responsible for parts... [18:48:23] 6Labs: Disable NFS on the orgcharts project - https://phabricator.wikimedia.org/T103137#1839287 (10yuvipanda) Going to kill now. I'll get rid of NFS and also shut down the instance. [18:51:02] 6Labs, 10Tool-Labs, 7Database: labs db inconsistent data - https://phabricator.wikimedia.org/T119841#1839302 (10MZMcBride) ``` mzmcbride@tools-bastion-01:~$ mysql -hlabsdb1003.eqiad.wmnet plwikisource_p -e "select count(*) from categorylinks where cl_to = 'Uwierzytelniona';" +----------+ | count(*) | +------... [18:54:01] yuvipanda: do you remember whether https://phabricator.wikimedia.org/T104327 applied to root logins as well? [18:54:15] 6Labs, 10Tool-Labs, 7Database: labs db inconsistent data - https://phabricator.wikimedia.org/T119841#1839318 (10jcrespo) @MZMcBride you are basically repeating the bug report without giving extra information, such as the count on labs1002. [18:55:08] 6Labs, 10Labs-Infrastructure: The package lists or status file could not be parsed or opened. - https://phabricator.wikimedia.org/T72981#1839320 (10Dzahn) I don't know. Something changed in labs back then and we never knew what caused it. We did find a work around though. It is unlikely it will be touched agai... [18:55:10] valhallasw`cloud: yes [18:55:37] valhallasw`cloud: but I think part of it was just that when NFS locked up the kernel just got 'stuck'? not sure if strace would help [18:56:23] well, not at that moment because you cannot login, but I assume it's caused by sshd trying to read a file from nfs [18:57:35] maybe userkeys, maybe /proc/filesystems [18:58:25] root@wikistats-petcow:~# apt-get update [18:58:26] N: Ignoring file '20auto-upgrades.ucf-dist' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension [18:59:02] 6Labs, 10Labs-Infrastructure: The package lists or status file could not be parsed or opened. - https://phabricator.wikimedia.org/T72981#1839324 (10Dzahn) when i wanted to see if i can still reproduce it i now have this new error: root@wikistats-petcow:~# apt-get update N: Ignoring file '20auto-upgrades.ucf-d... [19:02:28] 6Labs, 10Labs-Infrastructure: The package lists or status file could not be parsed or opened. - https://phabricator.wikimedia.org/T72981#1839329 (10Dzahn) I deleted that file above manually, then ran it again. apt-get upgrade now still offers me a bunch of upgrades (i assumed auto-upgrades were enabled now pe... [19:02:40] it looks like something went wrong with enabling auto-upgrades [19:02:47] there is that file with an invalid extension [19:04:28] and after i deleted that..i still get offered a bunch of upgrades [19:04:29] while the remaining config says APT::Periodic::Unattended-Upgrade "1"; [19:04:31] so it's like it actually broke apt-get update [19:04:32] and doesn't actually upgrade [19:04:33] mutante: auto-update normally only install security updates, not regular updates [19:04:38] well, apt-get update exited with an error message [19:05:02] but after deleting that file, alright [19:05:32] and i need to prevent that it actually downgrades my own app [19:05:46] however that happened..hmm [19:06:05] 6Labs, 5Patch-For-Review: Disable NFS on the orgcharts project - https://phabricator.wikimedia.org/T103137#1839362 (10yuvipanda) a:3yuvipanda I copied the mongodb data onto NFS and shut the instance down. I'll delete the instance in one week unless someone with an active plan of bringing it back up asks me t... [19:12:23] 6Labs, 10Tool-Labs, 7Database: labs db inconsistent data - https://phabricator.wikimedia.org/T119841#1839383 (10MZMcBride) Yeah, I found the task description a little confusing to read/test. Including the full commands makes testing a bit easier. It looks like `labsdb1001.eqiad.wmnet` and `labsdb1002.eqiad.w... [19:14:24] 6Labs, 6operations, 5Patch-For-Review, 7Puppet: Self hosted puppetmaster is broken - https://phabricator.wikimedia.org/T119541#1839384 (10yuvipanda) 5Invalid>3Open Still happening: see limn1 for an example [19:14:28] 6Labs, 10Tool-Labs, 7Database: labs db inconsistent data - https://phabricator.wikimedia.org/T119841#1839387 (10jcrespo) Then do not stop there, edit the description :-) [19:15:15] yuvipanda: is /data/project/.system toollabs specific or does every host-with-NFS have it? [19:15:40] toollabs specific [19:15:45] hrm. [19:15:53] so sshd tries to read /data/project/.system/tips [19:15:55] for some reason [19:16:11] hahaha [19:16:13] ofc [19:16:20] I think peta.n did that [19:16:27] it's probably in motd config somewhere [19:16:33] but let me try on an NFS-disabled host [19:16:45] which is funny since motd hasn't been working on trusty at all [19:16:55] toolserver-legacy doesn't do NFS, right? [19:17:11] it shouldn't yeah [19:17:25] valhallasw`cloud: also if you're looking for a specific host, tools-k8s-master-01 also has no NFS [19:17:27] hmm, no, the tips thing is in a toollabs manifest, so that shouldn't bother other hosts [19:17:29] 6Labs: NFS broken for new labs instances - https://phabricator.wikimedia.org/T119845#1839388 (10coren) 5Open>3Resolved I'm not seeing the issue at all at this time, and that instance - as far as I can tell - has working NFS mounts. It's entirely possible that this was a transient issue that was fixed when I... [19:19:08] yuvipanda: ^^ the NFS thing was gone by the time I looked at it, I suspect as a side-effect of a fix for a completely different thing I did this morning. [19:20:31] yeah ok. thanks Coren :) [19:22:39] yuvipanda: but anyway, that .tips might kill root login to toollabs, so let's remove it :p [19:22:54] valhallasw`cloud: +1 [19:23:51] yuvipanda: By the way and relatedly, a side effect of the PAM cleanup is likely to be that motd works on Trusty. [19:24:11] nice! [19:24:18] * yuvipanda looks forward to colorful banners again [19:24:55] (afaict, the reason it doesn't right now is because the forcibly overriden pam_motd invocation was ok for Precise but broke in Trusty) [19:26:38] yuvipanda: the tips database doesn't exist to begin with :D [19:27:50] valhallasw`cloud: you... added grep in that patch to exec_environ? [19:28:03] yuvipanda: yes, because I removed the package{} in the other manifest [19:28:30] grep sounds something we should have everywhere rather than juts on bastions ;-) [19:28:30] oh ofc [19:28:32] yeah [19:28:36] and I think we do [19:28:38] I think it'll be there anyway [19:28:43] part of base system [19:28:44] but anyway [19:28:45] but explicit > implicit, etc [19:29:06] mergd [19:30:16] 6Labs, 10Tool-Labs, 5Patch-For-Review: strace sshd to figure out which filesystems are hit on login - https://phabricator.wikimedia.org/T104327#1839435 (10valhallasw) I think this was just for tool labs, and I finally straced: * ps aux | grep sshd * strace -p -f -e trace=file,chdir 2>&1 | tee strace... [19:30:17] \o/ [19:30:29] 6Labs, 10Tool-Labs, 5Patch-For-Review: strace sshd to figure out which filesystems are hit on login - https://phabricator.wikimedia.org/T104327#1839438 (10valhallasw) 5Invalid>3Resolved [19:31:05] valhallasw`cloud: <3 for tracking it down [19:31:43] valhallasw`cloud: grep is part of the debian minimal install and is marked 'required' priority. Mentioning it explicitly is not useful. [19:32:33] Coren: it was already marked explicitly in the wrong place, so he just moved it to the 'right' place... [19:32:49] Heh. Not sure wtf it'd have been there in the first place. [19:33:16] If something is missing that is marked required, it's unlikely your system works at all, let alone enough to have puppet run to fix it. :-) [19:33:17] ^ is my feelings about a lot of things :) [19:33:22] 6Labs, 10Labs-Infrastructure: The package lists or status file could not be parsed or opened. - https://phabricator.wikimedia.org/T72981#1839443 (10Dzahn) it might be easiest to give up again on using labsdebrepo (in combination with auto-upgrades that appear to have been added) [19:33:26] (the first line, not second) [19:34:19] yuvipanda: ooh, let's encrypt public beta starting in a few days [19:34:21] In fact, I'm pretty sure apt would complain loudly if you even tried to remove a required priority package. :-) [19:34:39] * Coren does not try. [19:34:48] 6Labs, 10Labs-Infrastructure: The package lists or status file could not be parsed or opened. - https://phabricator.wikimedia.org/T72981#1839449 (10yuvipanda) Yes let's kill labsdebrepo :) You can maybe just upload your deb to carbon or use aptly (which does not require NFS)? [19:35:06] apt might just depend on grep ;-) [19:35:08] valhallasw`cloud: we already have a whitelist for betacluster, but bblack hasn't had time to test it :( [19:38:31] 6Labs, 10Labs-Infrastructure: The package lists or status file could not be parsed or opened. - https://phabricator.wikimedia.org/T72981#1839458 (10chasemp) >>! In T72981#1839449, @yuvipanda wrote: > Yes let's kill labsdebrepo :) You can maybe just upload your deb to carbon or use aptly (which does not require... [19:39:12] 6Labs, 10Tool-Labs, 5Patch-For-Review: Improve puppet run time - https://phabricator.wikimedia.org/T116813#1839459 (10valhallasw) After discussing this more with @yuvipanda, we figured it might be more effective to switch to require => present plus unattended-updates for all packages. That requires less home... [19:39:14] 6Labs, 10Tool-Labs, 5Patch-For-Review: strace sshd to figure out which filesystems are hit on login - https://phabricator.wikimedia.org/T104327#1839461 (10chasemp) cool great news [19:39:42] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install GNU automake on Tool Labs - https://phabricator.wikimedia.org/T119870#1839462 (10chasemp) 5Open>3Resolved a:3chasemp [19:40:14] Coren: did you see tools.wmflabs.org/paws/hub/oauth_login [19:40:48] yuvipanda: Not that I can recall. What's it about? [19:40:48] andrewbogott: chasemp ^ [19:41:03] Coren: it's basically shell access to run pywikibot stuff over a browser. [19:41:12] takes ssh and gridengine outof the equation] [19:41:56] (not for cron stuff yet) [19:42:14] 6Labs, 6operations, 5Patch-For-Review, 7Puppet: Self hosted puppetmaster is broken - https://phabricator.wikimedia.org/T119541#1839471 (10chasemp) >>! In T119541#1839384, @yuvipanda wrote: > Still happening: see limn1 for an example any common thread for broken instances (since it doesn't seem to be unive... [19:42:16] Huh. Kinda interesting. [19:42:31] my hope/plan is to identify different types of things people use the grid for and device a different solution for each. This is for the 'one off pwb scripts' usecase [19:42:51] what does oauth authenticate against? [19:43:02] andrewbogott: wiki SUL [19:43:12] (aka not LDAP) [19:43:13] so, nothing to do with wikitech? [19:43:14] nope [19:43:33] Is there reason to think that pywikibot doesn’t have a bunch of exploits? [19:43:45] each of them are running in their own docker container [19:44:05] and can not access the instances themselves [19:44:10] and this is all running as a tool account [19:44:31] hm… ok, that sounds reasonably safe :) [19:44:33] there's also strict memory and cpu limits per user [19:45:09] 6Labs, 10Tool-Labs: Delete "toolserver" tool - https://phabricator.wikimedia.org/T116389#1839485 (10valhallasw) 5Open>3Resolved a:3valhallasw Replaced by a 301 redirect to https://www.mediawiki.org/wiki/Toolserver:Main_Page. [19:45:41] yuvipanda: when I visited that page it presented me with exactly one option ‘my server’. When I reloaded, it now presents me with two options ‘my server’ and ‘stop my server' [19:45:50] andrewbogott: click 'my server' [19:45:59] I did, nothing happened. Will try again [19:46:01] andrewbogott: then click 'new' 'terminal' [19:46:27] yuvipanda: do you think we could just git clone the tesseract OCR data files? that might be easier than backporting packages from the latest debian [19:46:57] valhallasw`cloud: wouldn't that fuck with puppet running times some more? [19:47:02] yeah, it would [19:47:14] valhallasw`cloud: also how hard would backports be? do they have dependencies? [19:47:18] but backporting 50 packages does that to my brain :-p [19:47:29] they should be reasonably trivial, but there are a lot of packages [19:47:32] one per language [19:47:48] we could also fpm them [19:48:28] valhallasw`cloud: we can just wget them and aptly add them :) [19:48:29] hey, a shell [19:48:32] andrewbogott: woo [19:48:39] yuvipanda: oh, that's briliant! [19:48:46] :D [19:49:12] andrewbogott: there's a pywikibot install in /srv/pwb that I'm playing with, and your $HOME is persisted across sessions [19:49:27] 6Labs, 10Tool-Labs: Backport newer tesseract-ocr-* packages from sid - https://phabricator.wikimedia.org/T118643#1839496 (10valhallasw) @yuvipanda had an even better idea: we don't need to actually backport them, we can just install the newer packages. I'm going to take a look at doing this. [19:49:52] valhallasw`cloud: if they're pure data packages it should be ok to do [19:50:27] yuvipanda: https://packages.debian.org/sid/all/tesseract-ocr-srp/filelist :-p [19:50:55] yuppp [19:53:50] let me think of how to scrape that :/ [19:56:07] andrewbogott: I'm going to keep iterating on this and try to add actual docs :) maybe will take off over time like Quarry did [20:14:27] 6Labs, 10wikitech.wikimedia.org: fix apache log rotation on silver - https://phabricator.wikimedia.org/T119882#1839523 (10Andrew) 5Open>3Resolved This was mostly working already; I've cleaned up the log files that were so old that log-rotate was ignoring them. [20:33:57] yuvipanda: meeeeeh [20:33:58] Breaks: tesseract-ocr (<< 3.04.00-1) [20:35:21] 6Labs, 10Tool-Labs: Backport newer tesseract-ocr-* packages from sid - https://phabricator.wikimedia.org/T118643#1839548 (10valhallasw) Meh. Apparently there also have been changes to the ocr engine, as the data files explicitly set ``` Breaks: tesseract-ocr (<< 3.04.00-1) ``` and we only have 3.03.02-3 ins... [20:36:27] 6Labs, 10Tool-Labs: Backport newer tesseract-ocr-* packages from sid - https://phabricator.wikimedia.org/T118643#1839550 (10yuvipanda) hmm, so we have to backport that I guess... [20:38:52] 6Labs, 10wikitech.wikimedia.org: Discrepancies in public IP instance lists between different wikitech UIs - https://phabricator.wikimedia.org/T62883#1839552 (10scfc) 5Open>3Resolved a:3scfc No, https://wikitech.wikimedia.org/wiki/Special:NovaAddress now shows `novaproxy-01.project-proxy.eqiad.wmflabs` fo... [20:39:08] 6Labs, 10wikitech.wikimedia.org: Discrepancies in public IP instance lists between different wikitech UIs - https://phabricator.wikimedia.org/T62883#1839555 (10scfc) a:5scfc>3None [20:40:49] 6Labs: Instances with no name or project associated on wikitech - https://phabricator.wikimedia.org/T91922#1839565 (10Andrew) I've confirmed that the first three don't exist in openstack or in ldap, so I'm deleting those pages. [20:41:11] and that does have dependencies [20:41:12] sigh [21:07:33] yuvipanda: have I mentioned I hate building debian packages :P [21:08:05] 'simple backport creation' hahaha [21:14:47] but maybe it's easy with cowbuilder, we'll see. [21:15:25] !log tools applying package_builder manifest on tools-packages [21:15:44] :/ [21:15:49] * valhallasw`cloud prods labslogbot [21:16:03] oh well, we have elasticsearch [21:54:48] !log tools trying to build tesseract-ocr, but I'm being dragged into the rabbithole: needs a newer leptonica which needs a newer libjpeg2 [22:05:08] akosiaris: so I now built libopenjp2, which meant I could build libleptonica-dev -- but now the apt-get for dependencies for tesseract-ocr picks up the old version in trusty rather than the one I just built. How do I fix that? It /could/ find the (new package) libopenjp2 [22:05:28] (asking you because I saw your name in the package_builder logs, but maybe you're not the right person to ask) [22:07:39] there's a libleptonica-dev_1.72-3_amd64.deb in /var/cache/pbuilder/result/trusty-amd64/, but it installs libleptonica-dev_1.70.1-1_amd64.deb from trusty [22:18:13] 6Labs: Instances with no name or project associated on wikitech - https://phabricator.wikimedia.org/T91922#1839809 (10Andrew) 5Open>3Resolved a:3Andrew and, I prompted SMW to update and now none of the above instances are on that page. [22:31:10] hm, I guess the normal method would be 'import built packages to apt.wm.o', but... meh. [23:51:07] hi! is there a way to install an nmp module? when I do "npm install -g babel-cli" I get Error: EACCES, mkdir '/usr/local/lib/node_modules' [23:55:26] ok, nevermind, removed the "-g" part and it works