Version 7 (modified by japhb, 12 years ago)

Add link to Gitorious project page at top

GET CODE

This page details history and design; prototype code is hosted on Gitorious at:

 http://gitorious.org/parrot-plumage/parrot-plumage

Overview

We would like to ship Parrot as a relatively lean VM with a minimal core module set, and move all additional module installation out to an external ecosystem. This page is a very rough draft of a proposed module ecosystem for Parrot.

As of mid-August 2009, we have reached enough general agreement on the initial rough draft to begin coding a very simple prototype. It is believed that some of the undecided objections to the current draft will be easier to address once the prototype is operational.

It is intended that the draft proposal and the prototype be iterated together, eventually reaching the level that they (or successors, if we determine major rewrites are needed) can be supported by the Parrot team.

Background

Chats and email threads:

Requirements

  • General
    • Distributions should be dead easy for module authors to create, and for users to install.
    • We can create a centralized metadata store, but do not want to build and manage a module distribution network ...
    • However it should be possible for another group to do so.
      • It is unlikely someone will try to mirror the entire history of every dynamic language module ever made,
      • But for instance serving torrents of Power Packs seems reasonable.
    • The search API and install process should be dead simple and clearly specified.
      • Clone implementations used by other ecosystems will be a mark of success.
  • Toolchain
    • Basic tools can assume Parrot and the core modules are working, but require no other dependencies internally.
    • All external tools needed to download/build/install modules will be specified in the module metadata.
    • Tools should be easy to configure.
    • Tools should attempt to auto-configure as much as possible.
    • Tools must properly handle the difference between user-local, site-local, and vendor-installed modules. (Do we need these particular cases? Why not just support arbitrary custom paths?)
  • Metadata
    • Simple, extensible format.
    • Unicode and case-retaining.
    • Must include its own spec version.
    • Sufficient for automated programs to create system packages (DEB, RPM, etc.). (We may be only able to handle the common cases and provide guidance on the harder ones. However, having this as a goal should help us shake out weak points in the spec.)
    • Separate static v. configure-discovered v. hand-edited metadata. Separate files? (Overly complex? Needs followup with #toolchain.)
    • Includes fetch, configure, build, test, install, and runtime dependencies.
    • Should be able to track author, mailing list, bug email/bug URI, wiki, homepage, source repository, etc.
    • Allows disambiguation as per Perl 6 module spec (authorities, versions, authors, etc.), as well as equivalents for PHP PEAR, RubyGems, Python Eggs, etc.
    • Specifies rules for dependency string parsing/interpretation.

Proposal

This is a rough draft; suggestions welcome!

  • Overview
    • Parrot community builds a module metadata search system.
    • This search system gathers metadata from various sources, and allows users to query it via web browser or API, but does not itself store the actual modules.
    • Once found, modules can be fetched from many possible sources, including VCS repositories, FTP mirrors, etc.
    • Parrot team will need to standardize module metadata, provide the libraries and tools necessary to use the search system, provide guidelines for extending the toolchain, and mentor the growth of the ecosystem until it stands on its own.
  • Metadata format
    • Metadata served as JSON.
    • Format for specifying non-metadata-only build scripts undecided.
    • Integrity check / authentication methods undecided.
      • Probably at least md5sum and sha1sum for source tarballs, but what about when pulling from raw VCS repo?
  • Metadata server API & UI
    • Some disagreement on details, but general agreement to:
      • Minimize effort for module authors and end users,
      • But get something simple working first.
    • Note: The module author may not be the one using the UI; this may be a packager or Parrot team member.
    • Web form UI
      • Enter/update/remove all module information (metadata fields)
      • Deprecation/EOL
      • Accept raw JSON block as alternative
    • Simple HTTP/JSON API
      • Push or pull updates?
  • Core modules
    • Note: These should be minimal, no extra fluff.
    • parrot config (already exists -- config.pir)
    • compiler tools (already exist -- PGE, PCT, NQP)
    • HTTP client (at least GET, with redirect and proxy support)
    • zlib (at least decompress)
    • tar (at least extract)
    • JSON (at least parse)
    • version spec (at least parse and compare)
    • library probe (shared library info: present? version? location?)
    • file paths (portability: File::Spec + File::Basename + ...)
    • file install (portability: copy file, set file perms, etc.)
    • query metadata (perform API calls to metadata/search server)
    • installer lib (all the real brains/glue for the module repo client)
    • installer ui (CLI and/or Readline, minimal brains, uses lib)
  • Basic Batteries modules
    • Note: Basic Batteries is just another Power Pack from a technical viewpoint; it is merely treated slightly differently in the community.
    • Full versions of any modules that are limited in Core
    • Installer add-ons: VCS fetch/use system pkgs/full depresolve/etc.
    • Module author tools: easy create (module or pack)/register/update/upload/etc.
    • PIR-level tools: disassembler/debugger/profiler/data dumper
    • NCI tools: parse header/manage typemap/wrap C struct/etc.
    • Standard interfaces: TAP, DBDI, logging, ?
    • Standard libraries: OpenSSL, DateTime, temp dir/file, ?
  • Possible Power Packs (NOTE: EXAMPLES ONLY, DON'T BIKESHED!)
    • (There is some disagreement where Power Packs should lie on the standalone <-> dependency-laden spectrum.)
    • Database: DBDs (drivers), SQL clients, per-HLL DBI variants
    • Testing: smoke/tinder/smolder clients, per-HLL Test::* variants
    • Security: SSH, GPG, libpcap, ...
    • Unixen: POSIX, Fcntl, Errno, ...
    • Markup: YAML, libxml2, Expat, DOM, SAX, ...
    • VCS: CVS, Subversion, git, Mercurial, ...
    • Email: POP, IMAP, SMTP, MIME, ...
    • GUI: Qt, GTK+, Wx, Tk, ...
    • 2D Graphics: libpng, GD, SDL, Cairo, ...
    • 3D Graphics: OpenGL, EGL, GLU, ...
    • Sound: OpenAL, Pulse Audio, JACK, ...
    • Game Support: Require other Power Packs: Audio, 2D/3D Graphics
  • Misc recommendations
    • Separate parrot-modules mailing list for module creators/users only if parrot-dev traffic gets overwhelmed.
    • Default to simple (CPAN-style) dependency resolution; upgrade to full resolution and system package awareness in Basic Batteries. (OS packaging systems already had to solve all of this; let's not go reinventing wheels.)
    • Master metadata site will be reponame.parrot.org
    • Names so far suggested for module repository network:
      • For trademark purposes, may need to be 'Parrot Foo' rather than just 'Foo' alone.
      • Bird Words
        • Aviary
        • Birdseed
        • Gluepot
        • Plumage
        • Roost
      • Acronyms (generally disliked, and leads to CPAN comparison/expectation)
        • CPAAN
        • FPAN

Metadata Proposal

  • Required fields:
    • meta-spec
      • version
      • uri
    • name
    • authority
    • version
    • license
      • type (standard short names for common licenses)
      • uri
    • copyright_holder
    • abstract
    • instructions
      • type (standard names for common build sequences, such as cpan-mm for CPAN + ExtUtils::MakeMaker, or custom for custom instructions)
        • We may eventually provide our own standard type that we recommend for new Parrot-specific modules; or perhaps one for scriptless installs and one for scripted installs.
      • fetch
      • configure
      • build
      • test
      • install
      • runtime (here for completeness; necessary?)
  • Manifest fields (still under discussion; for scriptless installs):
    • files
      • configure
      • build
      • test
      • install
        • share
        • docs
        • bin
        • lib
        • runtime
  • Dependency fields (as { [dep_name]: [version_spec], ... }):
    • provides
    • conflicts
    • requires
      • Note: even 'basic' tools such as make must be declared as dependencies here -- anything not provided by Parrot and the ecosystem tools directly.
      • fetch
      • configure
      • build
      • test
      • install
      • runtime
  • Optional features fields:
    • optional_features
      • [feature_name]
        • description
        • [any/all dependency fields as needed]
  • Other optional fields:
    • description
    • keywords
    • generated_by
    • contributors
      • authors
      • maintainers
      • translators
      • testers
      • reviewers
    • resources
      • source
      • homepage
      • bugtracker
      • wiki
      • repository
        • type
        • checkout_uri
        • browser_uri
        • project_uri
      • mailinglists
        • [list_name]
          • address
          • uri
  • Undecided fields:
    • How do fields get added to spec? Allow custom fields?
    • dynamic_config
    • no_index
    • digests
    • signatures