Ubuntu Component Store, redux

A while back I proposed an “Ubuntu Component Store” and built a noddy implementation of the command line utility for it. Recently, Nekhelesh Ramanathan revived the idea and did a bunch of work to implement it, and we had a discussion session at the Ubuntu Online Summit about it. This was most interesting. In it, I believe we concluded two things:

Thing 1: It is a good idea to have a place where developers can publish components for Ubuntu apps and an easy way for other developers to get and use those components.

We can argue in detail about exactly what “an easy way” entails, and there is no more bikesheddable project in existence, but I think we were all agreed that the basic idea is this: there is a ucs command line utility which is roughly like pip or npm or apt, that you can type ucs install SomeComponentName, and that component will be downloaded and installed into the project that your command line directory is in (not into some central repository; this is project-specific). This will be great; there are obviously a bunch of other implied commands here, such as ucs list, ucs search, and ucs update to update components to the latest upstream versions. Everyone agrees on this part.

Thing 2: Some people think that a component store should be composed of basically-complete excellently-written well-documented components which are essentially candidates for direct inclusion in the core SDK. Some people think that a component store should be composed of whatever anyone wants to add, and developers can use or not use components as they choose.

Specifically, Nekhelesh is in camp 1 (only good components, like the Ubuntu SDK) and I’m in camp 2 (anyone can publish anything and market forces sort things out, like PyPI or NPM). Both of these camps have some pros and cons, which I will attempt to lay out below without bias; I am, as noted, in camp 2, but I’ll try to be objective here, and please excuse me if I misrepresent camp 1.

First, a brief description of how each of these approaches would work, which will help make sense of the benefits and deficits list.

Nekhelesh proposes that components are part of one single Launchpad branch; to add a component to the store, propose a merge to that source tree with your component source, tests, and documentation. This will be manually reviewed and if suitable, included; a developer can also apply for membership of the component store team, who own the branch and can make changes to it and accept merges. The component is then available from that Launchpad branch. Documentation from the branch is then published to a central location as a manual, similar to the core Ubuntu SDK documentation.

I propose that a UCS web API server is set up which accepts submissions of component metadata (name, version, remote URL, and similar), and that server provides this info via an API. Components themselves are not stored on the UCS server; one puts a component wherever one likes (github, launchpad, one’s own website) and just tells the UCS server about the component and where it is. Providing good documentation and tests is obviously encouraged and will make your component more attractive than alternatives, but is not required; developers can choose to not publish such things, and other developers can choose to use or not use an ill-documented component as a result.

Curated Component Store: benefits

  • Developers can be sure that every component in the store is decent.
  • Components will have good documentation and be well-integrated.
  • If a component maintainer disappears, others can take over the component.
  • The store is a good “breeding ground” for new entrants to the SDK; indeed, the ultimate destiny of any component in the curated store is to be elevated into the core SDK eventually.

Curated Component Store: deficits

  • Developers publishing components are required to have a Launchpad account.
  • Manual review of proposed changes is slow and annoying.
  • A component author will have their component manually reviewed for every change, unless they join the component store team.
  • Any member of the component store team (i.e., any component author who does not want to be blocked by manual review) can edit any component, meaning that they must be trusted.

Community Component Store: benefits

  • Anyone can publish a component instantly without bureaucracy getting in the way.
  • Publishing updates to a component is also instant.
  • This is how everyone else manages library packages etc.

Community Component Store: deficits

  • Developers have no guarantee that components in the store are high-quality or have documentation or tests.
  • A server has to be built and run.

So, there are upsides and downsides to both proposals.

I think there’s a progression here. There are community components, which might become curated components, which themselves might become core components. That is: it’s perfectly reasonable to have both approaches. Here’s my updated proposal.

We have those three levels of components: Core, Curated, and Community.

Core components are those that are already in the Ubuntu SDK: you can use them automatically in every Ubuntu SDK application.

Curated components are in the Launchpad branch, exactly as Nekhelesh proposes. To add a curated component, you propose a merge to the LP branch, and it’s manually reviewed; your component needs to have good test coverage, good documentation which matches the style and format of the other curated components, and provide a stable API. Curated components have a component name: BottomEdgeMenu, or WelcomeWizard. They are installed with ucs, as ucs install BottomEdgeMenu.

Community components are hosted wherever you want, and you tell the UCS Community Server about the URL where they can be obtained. They do not require manual review and approval, and they do not require great test coverage or documentation, although obviously you should provide those things because it makes your component better! Community components have an owner and a name: sil/GenericPodcastWidget. They are installed with ucs, as ucs install sil/ColourSlider, and are published by uploading them somewhere and then using ucs submit username/ComponentName.

One can tell the difference between a curated component and a community component because community components have the developer’s name right there in the component name; it shows you that you’re using something from a particular developer, and so if a component name has a slash in it, it’s a community component. Components without a developer name are from the curated store, and the curated store team stand behind them and guarantee their quality.

All components, both curated and community, contain an ubuntu_community_store.json file which lists metadata about the component: its name, licence, description, version number, dependencies, and others.

How UCS works: technical

The UCS system, under this proposal, knows about two “repositories” of components: the curated store, and the community store. Saying ucs install ComponentName goes to the curated store and downloads and installs the component; ucs install developer/ComponentName goes to the community store. It will be useful to have two separate “fetch” back ends here, one which can resolve a curated ComponentName to a download URL (essentially by saying downloadurl = CURATED_LP_BRANCH_URL + COMPONENT_NAME), and one which can resolve a community ComponentName to a download URL (essentially by requesting http://communityserver/api/get/developer/ComponentName and reading the download URL out of the returned JSON).

ucs install some-component should write to ubuntu_component_store.json in the project folder, updating it to contain the name and version number of any installed component. It is up to developers whether to commit the downloaded component code to their source tree or not; if they do not, components can be installed at the correct version number with ucs install with no parameters, which reads ubuntu_component_store.json and installs those things mentioned therein.

The community server needs to provide a collection of API endpoints that the ucs tool can talk to:

    input { download_url: url to download whole component,
        metadata_url: to download ubuntu_component_store.json }, 
        requires auth
    output: { success: bool }
    Adds a component to the store. The server will fetch 
    ubuntu_component_store.json and parse it to index the metadata. 
    Future enhancement is allowing "special" URLs such as 
    "lp:~user/project/branch" or "github:username/project" when 
    the component is hosted on a well-known service. Adding a 
    component which already exists requires you to be adding a 
    higher version of it, and for it to be your component.
    input: none
    output: JSON metadata for component (not yet specified)
    Returns metadata for the most recent version of that specific 
    component, including a download URL; ucs install will fetch this 
    API entry and then use the contained download URL to fetch the 
    component itself.
    input: none
    output: JSON list of (some) metadata for all components
    Returns a list of all components in the community store.
    input: { username: string, password: string }
    output: { success: bool }
    Registers an account. (Note: this is not definite; account registration
    may be done with email addresses, auth may be OAuth 2, etc. Needs 
    input: none
    output: JSON list of (some) metadata for all components matching search
    For searching. How searching works, which fields are searchable, etc, 
    is not yet specified, and until there are many components, /api/list 
    may be enough.

There is a trivial example of a server (which is not complete) at lp:~sil/+junk/ucs-server; it is in Python and Django, mostly because we expect that many components will be hosted on Launchpad, interacting with Launchpad code hosting is easiest with bzr, and Python has bzrlib. Finding hosting for this should not be too difficult; at least initially traffic will not be high and so one of the existing Django hosting services should be able to easily cope for minimal or zero outlay.

I'm currently available for hire, to help you plan, architect, and build new systems, and for technical writing and articles. You can take a look at some projects I've worked on and some of my writing. If you'd like to talk about your upcoming project, do get in touch.

More in the discussion (powered by webmentions)

  • Fabio Colella responded at plus.google.com Really agree :)
  • Fabio Colella responded at plus.google.com Really agree :)
  • An unnamed person responded at www.kryogenix.org
  • An unnamed person responded at www.kryogenix.org
  • Nekhelesh Ramananthan responded at Nekhelesh... (plus.google.com) reshared this.
  • Nekhelesh Ramananthan responded at Nekhelesh... (plus.google.com) reshared this.
  • Carla Sella responded at Carla Sella... (plus.google.com) reshared this.
  • Carla Sella responded at Carla Sella... (plus.google.com) reshared this.
  • Christian Kopp responded at Christian... (plus.google.com) +1ed this.
  • Christian Kopp responded at Christian... (plus.google.com) +1ed this.
  • cm-t arudy responded at cm-t arudy... (plus.google.com) +1ed this.
  • cm-t arudy responded at cm-t arudy... (plus.google.com) +1ed this.
  • Trent Schafer responded at Trent... (plus.google.com) +1ed this.
  • Trent Schafer responded at Trent... (plus.google.com) +1ed this.
  • David Callé responded at David Callé... (plus.google.com) +1ed this.
  • David Callé responded at David Callé... (plus.google.com) +1ed this.
  • Rosa Guillén responded at Rosa... (plus.google.com) +1ed this.
  • Rosa Guillén responded at Rosa... (plus.google.com) +1ed this.
  • Fabio Colella responded at Fabio... (plus.google.com) +1ed this.
  • Fabio Colella responded at Fabio... (plus.google.com) +1ed this.
  • Mark Shuttleworth responded at Mark... (plus.google.com) +1ed this.
  • Mark Shuttleworth responded at Mark... (plus.google.com) +1ed this.