Version 1 (modified by japhb, 5 years ago)

Initial notes from mailing list, translated to WikiFormatting, lightly edited; more editing to come in next rev

Background

Chats and email threads:

Requirements

  • General
    • Distributions should be dead easy for module authors to create, and for users to install.
    • We can create a centralized metadata store, but do not want to build and manage a module distribution network ...
    • However it should be possible for another group to do so.
  • Toolchain
    • Basic tools can assume Parrot and the core modules are working, but require no other dependencies internally.
    • All external tools needed to download/build/install modules will be specified in the module metadata.
    • Tools should be easy to configure.
    • Tools should attempt to auto-configure as much as possible.
    • Tools must properly handle the difference between user-local, site-local, and vendor-installed modules.
  • Metadata
    • Simple, extensible format.
    • Unicode and case-retaining.
    • Must include its own spec version.
    • Sufficient for automated programs to create system packages (DEB, RPM, etc.).
    • Separate static v. configure-discovered v. hand-edited metadata. Separate files?
    • Includes fetch, configure, build, test, install, and runtime dependencies.
    • Should be able to track author, mailing list, bug email/bug URI, wiki, homepage, source repository, etc.
    • Allows disambiguation as per Perl 6 module spec (authorities, versions, authors, etc.).
    • Specifies rules for dependency string parsing/interpretation.

Proposal

This is a rough draft! Suggestions welcome!

  • Overview
    • Parrot community builds a module metadata search system.
    • This search system gathers metadata from various sources, and allows users to query it via web browser or API, but does not itself store the actual modules.
    • Once found, modules can be fetched from many possible sources, including VCS repositories, FTP mirrors, etc.
    • Parrot team will need to standardize module metadata, provide the libraries and tools necessary to use the search system, provide guidelines for extending the toolchain, and mentor the growth of the ecosystem until it stands on its own.
  • Metadata format
    • Served metadata container is gzip'ed tarball (.tar.gz? .tgz?).
    • Core metadata is in META.json at top level of container.
    • Container includes copies of special files (e.g. README).
    • Format for specifying non-metadata-only build scripts undecided.
    • Integrity check / authentication methods undecided.
      • Probably at least md5sum and sha1sum for source tarballs, but what about when pulling from raw VCS repo?
  • Core modules
    • parrot config (already exists -- config.pir)
    • HTTP client (at least GET, with redirect and proxy support)
    • zlib (at least decompress)
    • tar (at least extract)
    • JSON (at least parse)
    • version spec (at least parse and compare)
    • library probe (shared library info: present? version? location?)
    • file paths (portability: File::Spec + File::Basename + ...)
    • file install (portability: copy file, set file perms, etc.)
    • query metadata (perform API calls to metadata/search server)
    • installer lib (all the real brains/glue for the module repo client)
    • installer ui (CLI and/or Readline, minimal brains, uses lib)
  • Basic Batteries modules
    • Full versions of any modules that are limited in Core
    • Installer add-ons: VCS fetch/use system pkgs/full depresolve/etc.
    • Module author tools: create/register/update/upload/etc.
    • PIR-level tools: disassembler/debugger/profiler/data dumper
    • NCI tools: parse header/manage typemap/wrap C struct/etc.
    • Standard interfaces: TAP, DBDI, logging, ?
    • Standard libraries: OpenSSL, DateTime, temp dir/file, ?
  • Possible Power Packs (NOTE: EXAMPLES ONLY, DON'T BIKESHED!)
    • Database: DBDs (drivers), SQL clients, per-HLL DBI variants
    • Testing: smoke/tinder/smolder clients, per-HLL Test::* variants
    • Security: SSH, GPG, libpcap, ...
    • Unixen: POSIX, Fcntl, Errno, ...
    • Markup: YAML, libxml2, Expat, DOM, SAX, ...
    • VCS: CVS, Subversion, git, Mercurial, ...
    • Email: POP, IMAP, SMTP, MIME, ...
    • GUI: Qt, GTK+, Wx, Tk, ...
    • 2D Graphics: libpng, GD, SDL, Cairo, ...
    • 3D Graphics: OpenGL, EGL, GLU, ...
    • Sound: OpenAL, Pulse Audio, JACK, ...
    • Game Support: Require other Power Packs: Audio, 2D/3D Graphics
  • Misc recommendations
    • Separate 'parrot-modules' mailing list for module creators/users only if parrot-dev traffic gets overwhelmed.
    • Default to simple (CPAN-style) dependency resolution; upgrade to full resolution and system package awareness in Basic Batteries.
    • Names so far suggested for module repository network:
      • Bird Words
        • Aviary
        • Birdseed
        • Gluepot
        • Plumage
        • Roost
      • Acronyms
        • CPAAN
        • FPAN

Metadata Proposal

  • Required fields:
    • meta-spec
      • version
      • uri
    • name
    • authority
    • version
    • license
      • type
      • uri
    • copyright_holder
    • abstract
  • Manifest fields:
    • files
      • configure
      • build
      • test
      • install
        • share
        • docs
        • bin
        • lib
        • runtime
  • Dependency fields (as { [dep_name]: [version_spec], ... }):
    • provides
    • conflicts
    • requires
      • fetch
      • configure
      • build
      • test
      • install
      • runtime
  • Optional features fields:
    • optional_features
      • [feature_name]
        • description
        • [any/all dependency fields as needed]
  • Other optional fields:
    • description
    • keywords
    • generated_by
    • contributors
      • authors
      • maintainers
      • translators
      • testers
      • reviewers
    • resources
      • source
      • homepage
      • bugtracker
      • wiki
      • repository
        • type
        • checkout_uri
        • browser_uri
        • project_uri
      • mailinglists
        • [list_name]
          • address
          • uri
  • Undecided fields:
    • dynamic_config
    • no_index
    • digests
    • signatures