flux2/docs/proposals/go-git-providers.md

go-git-providers

Abstract

This proposal aims to create a library with the import path github.com/fluxcd/go-git-providers' (import name: gitprovider), which provides an abstraction layer for talking to Git providers like GitHub, GitLab and Bitbucket.

This would become a new repository, specifically targeted at being a general-purpose Git provider client for multiple providers and domains.

Goals

• Support multiple Git provider backends (e.g. GitHub, GitLab, Bitbucket, etc.) using the same interface
• Support talking to multiple domains at once, including custom domains (e.g. talking to "gitlab.com" and "version.aalto.fi" from the same client)
• Support both no authentication (for public repos), basic auth, and OAuth2 for authentication
• Manipulating the following resources:
  • Organizations: GET, LIST (both all accessible top-level orgs and sub-orgs)
  • For a given Organization:
    • Teams: GET and LIST
  • Repositories: GET, LIST and POST
    • Teams: ADD and REMOVE
    • Credentials: GET, LIST and POST
• Support sub-organizations (or "sub-groups" in GitLab) if possible
• Pagination is automatically handled for LIST requests
• Transparently can manage teams (collections of users, sub-groups in Gitlab) with varying access to repos

Non-goals

• Support for features not mentioned above

Design decisions

• A context.Context should be passed to every request as the first argument
• There should be two interfaces per resource, if applicable:
  • one collection-specific interface, with a plural name (e.g. OrganizationsClient), that has methods like Get() and List()
  • one instance-specific interface, with a singular name (e.g. OrganizationClient), that operates on that instance, e.g. allowing access to child resources, e.g. Teams()
• Every Create() signature shall have a {Resource}CreateOptions struct as the last argument.
  • Delete() and similar methods may use the same pattern if needed
• All *Options structs shall be passed by value (i.e. non-nillable) and contain only nillable, optional fields
• All optional fields in the type structs shall be nillable
• It should be possible to create a fake API client for testing, implementing the same interfaces
• All type structs shall have a Validate() method, and optionally a Default() one
• All type structs shall expose their internal representation (from the underlying library) through the InternalGetter interface with a method GetInternal() interface{}
• By default, mutating methods (e.g. Create()) shall be idempotent, unless opted out in *CreateOptions field.
• Typed errors shall be returned, wrapped using Go 1.14's new features
• Go-style enums are used when there are only a few supported values for a field
• Every field is documented using Godoc comment, including +required or +optional to clearly signify its importance

Implementation

Provider package

The provider package, e.g. at github.com/fluxcd/go-git-providers/github, will have constructor methods so a client can be created, e.g. as follows:

// Create a client for github.com without any authentication
c := github.NewClient()

// Create a client for an enterprise GitHub account, without any authentication
c = github.NewClient(github.WithBaseURL("enterprise.github.com"))

// Create a client for github.com using a personal oauth2 token
c = github.NewClient(github.WithOAuth2("<token-here>"))

Client

The definition of a Client is as follows:

// Client is an interface that allows talking to a Git provider
type Client interface {
	// The Client allows accessing all known resources
	ResourceClient
	// The Client can parse Repository and Organization URLs
	URLParser

	// SupportedDomain returns the supported domain
	// This field is set at client creation time, and can't be changed
	SupportedDomain() string

	// ProviderID returns the provider ID (e.g. "github", "gitlab") for this client
	// This field is set at client creation time, and can't be changed
	ProviderID() ProviderID

	// Raw returns the Go client used under the hood for accessing the Git provider
	Raw() interface{}
}

As one can see, the Client is scoped for a single backing domain. ProviderID is a typed string, and every implementation package defines their own constant, e.g. const ProviderName = gitprovider.ProviderID("github").

The ResourceClient actually allows talking to resources of the API, both for single objects, and collections:

// ResourceClient allows access to resource-specific clients
type ResourceClient interface {
	// Organization gets the OrganizationClient for the specific top-level organization
	// ErrNotTopLevelOrganization will be returned if the organization is not top-level when using
	Organization(o OrganizationRef) OrganizationClient

	// Organizations returns the OrganizationsClient handling sets of organizations
	Organizations() OrganizationsClient

	// Repository gets the RepositoryClient for the specified RepositoryRef
	Repository(r RepositoryRef) RepositoryClient

	// Repositories returns the RepositoriesClient handling sets of organizations
	Repositories() RepositoriesClient
}

In order to reference organizations and repositories, there are the OrganizationRef and RepositoryRef interfaces:

// OrganizationRef references an organization in a Git provider
type OrganizationRef interface {
	// String returns the HTTPS URL
	fmt.Stringer

	// GetProvider returns a provider-defined typed string, e.g. "github" or "gitlab"
	GetProvider() ProviderID
	// GetDomain returns the URL-domain for the Git provider backend, e.g. gitlab.com or version.aalto.fi
	GetDomain() string
	// GetOrganization returns the top-level organization, i.e. "weaveworks" or "kubernetes-sigs"
	GetOrganization() string
	// GetSubOrganizations returns the names of sub-organizations (or sub-groups),
	// e.g. ["engineering", "frontend"] would be returned for gitlab.com/weaveworks/engineering/frontend
	GetSubOrganizations() []string
}

// RepositoryRef references a repository hosted by a Git provider
type RepositoryRef interface {
	// RepositoryRef requires an OrganizationRef to fully-qualify a repo reference
	OrganizationRef

	// GetRepository returns the name of the repository
	GetRepository() string
}

Along with these, there is OrganizationInfo and RepositoryInfo which implement the above mentioned interfaces in a straightforward way.

If you want to create an OrganizationRef or RepositoryRef, you can either use NewOrganizationInfo() or NewRepositoryInfo(), filling in all the details, or use the URLParser interface, embedded in the Client:

// URLParser is an interface for parsing strings of clone URLs or HTTPS paths to an organization into
// parsed RepositoryRef or OrganizationRef forms.
type URLParser interface {
	// ParseRepositoryURL parses a HTTPS or SSH clone URL into a RepositoryRef object
	// The backing provider (e.g. github or gitlab) is automatically chosen based on the domain
	// Custom domains are also supported if registered. If the given domain is unknown, ErrDomainUnsupported is returned
	ParseRepositoryURL(r string) (RepositoryRef, error)

	// ParseOrganizationURL parses an URL to an organization into a OrganizationRef object
	// The backing provider (e.g. github or gitlab) is automatically chosen based on the domain
	// Custom domains are also supported if registered. If the given domain is unknown, ErrDomainUnsupported is returned
	ParseOrganizationURL(o string) (OrganizationRef, error)
}

As mentioned above, only one target domain is supported by the Client. This means e.g. that if the Client is configured for GitHub, and you feed it a GitLab URL to parse, ErrDomainUnsupported will be returned.

This brings us to a higher-level client abstraction, MultiClient.

MultiClient

In order to automatically support multiple domains and providers using the same interface, MultiClient is introduced.

The user would use the MultiClient as follows:

// Create a client to github.com without authentication
gh := github.NewClient()

// Create a client to gitlab.com, authenticating with basic auth
gl := gitlab.NewClient(gitlab.WithBasicAuth("<username>", "<password"))

// Create a client to the GitLab instance at version.aalto.fi, with a given OAuth2 token
aalto := gitlab.NewClient(gitlab.WithBaseURL("version.aalto.fi"), gitlab.WithOAuth2Token("<your-token>"))

// Create a MultiClient which supports talking to any of these backends
client := gitprovider.NewMultiClient(gh, gl, aalto)

The interface definition of MultiClient is similar to that one of Client, both embedding ResourceClient and URLParser, but it also allows access to domain-specific underlying Client's:

// MultiClient allows talking to multiple Git providers at once
type MultiClient interface {
	// The MultiClient allows accessing all known resources, automatically choosing the right underlying
	// Client based on the resource's domain
	ResourceClient
	// The Client can parse Repository and Organization URLs. The MultiClient supports multiple domains.
	URLParser

	// SupportedDomains returns a list of known domains and their provider types
	SupportedDomains() map[string]ProviderID

	// ClientForDomain returns the Client used for a specific domain
	ClientForDomain(domain string) (Client, bool)
}

OrganizationsClient

The OrganizationsClient provides access to a set of organizations, as follows:

// OrganizationsClient operates on organizations the user has access to
type OrganizationsClient interface {
	// Get a specific organization the user has access to
	// This might also refer to a sub-organization
	Get(ctx context.Context, o OrganizationRef) (*Organization, error)

	// List all top-level organizations the specific user has access to
	// List should return all available organizations, using multiple paginated requests if needed
	List(ctx context.Context) ([]Organization, error)

	// Children returns the immediate child-organizations for the specific OrganizationRef o.
	// The OrganizationRef may point to any sub-organization that exists
	// This is not supported in GitHub
	// Children should return all available organizations, using multiple paginated requests if needed
	Children(ctx context.Context, o OrganizationRef) ([]Organization, error)

	// Possibly add Create/Update/Delete methods later
}

The Organization struct is fairly straightforward for now:

// Organization represents an (top-level- or sub-) organization
type Organization struct {
	// OrganizationInfo provides the required fields
	// (Provider, Domain, Organization and SubOrganizations) required for being an OrganizationRef
	OrganizationInfo `json:",inline"`
	// InternalHolder implements the InternalGetter interface
	InternalHolder `json:",inline"`

	// Name is the human-friendly name of this organization, e.g. "Weaveworks" or "Kubernetes SIGs"
	// +required
	Name string `json:"name"`

	// Description returns a description for the organization
	// No default value at POST-time
	// +optional
	Description *string `json:"description"`
}

The OrganizationInfo struct is a straightforward struct just implementing the OrganizationRef interface with basic fields & getters. InternalHolder is implementing the InternalGetter interface as follows, and is embedded into all main structs:

// InternalGetter allows access to the underlying object
type InternalGetter interface {
	// GetInternal returns the underlying struct that's used
	GetInternal() interface{}
}

// InternalHolder can be embedded into other structs to implement the InternalGetter interface
type InternalHolder struct {
	// Internal contains the underlying object.
	// +optional
	Internal interface{} `json:"-"`
}

OrganizationClient

OrganizationClient allows access to a specific organization's underlying resources as follows:

// OrganizationClient operates on a given/specific organization
type OrganizationClient interface {
	// Teams gives access to the TeamsClient for this specific organization
	Teams() OrganizationTeamsClient
}

Organization Teams

Teams belonging to a certain organization can at this moment be fetched on an individual basis, or listed.

// OrganizationTeamsClient handles teams organization-wide
type OrganizationTeamsClient interface {
	// Get a team within the specific organization
	// teamName may include slashes, to point to e.g. "sub-teams" i.e. subgroups in Gitlab
	// teamName must not be an empty string
	Get(ctx context.Context, teamName string) (*Team, error)

	// List all teams (recursively, in terms of subgroups) within the specific organization
	// List should return all available organizations, using multiple paginated requests if needed
	List(ctx context.Context) ([]Team, error)

	// Possibly add Create/Update/Delete methods later
}

The Team struct is defined as follows:

// Team is a representation for a team of users inside of an organization
type Team struct {
	// Team embeds OrganizationInfo which makes it automatically
	OrganizationRef `json:",inline"`
	// Team embeds InternalHolder for accessing the underlying object
	InternalHolder `json:",inline"`

	// Name describes the name of the team. The team name may contain slashes
	// +required
	Name string `json:"name"`

	// Members points to a set of user names (logins) of the members of this team
	// +required
	Members []string `json:"members"`
}

In GitLab, teams could be modelled as users in a sub-group. Those users can later be added as a single unit to access a given repository.

RepositoriesClient

RepositoriesClient provides access to a set of repositories for the user.

// RepositoriesClient operates on repositories the user has access to
type RepositoriesClient interface {
	// Get returns the repository at the given path
	Get(ctx context.Context, r RepositoryRef) (*Repository, error)

	// List all repositories in the given organization
	// List should return all available organizations, using multiple paginated requests if needed
	List(ctx context.Context, o OrganizationRef) ([]Repository, error)

	// Create creates a repository at the given organization path, with the given URL-encoded name and options
	Create(ctx context.Context, r *Repository, opts RepositoryCreateOptions) (*Repository, error)
}

RepositoryCreateOptions has options like AutoInit *bool, LicenseTemplate *string and so forth to allow an one-time initialization step.

The Repository struct is defined as follows:

// Repository represents a Git repository provided by a Git provider
type Repository struct {
	// RepositoryInfo provides the required fields
	// (Provider, Domain, Organization, SubOrganizations and (Repository)Name)
	// required for being an RepositoryRef
	RepositoryInfo `json:",inline"`
	// InternalHolder implements the InternalGetter interface
	InternalHolder `json:",inline"`

	// Description returns a description for the repository
	// No default value at POST-time
	// +optional
	Description *string `json:"description"`

	// Visibility returns the desired visibility for the repository
	// Default value at POST-time: RepoVisibilityPrivate
	// +optional
	Visibility *RepoVisibility
}

// GetCloneURL gets the clone URL for the specified transport type
func (r *Repository) GetCloneURL(transport TransportType) string {
	return GetCloneURL(r, transport)
}

As can be seen, there is also a GetCloneURL function for the repository which allows resolving the URL from which to clone the repo, for a given transport method (ssh and https are supported TransportTypes)

RepositoryClient

RepositoryClient allows access to a given repository's underlying resources, like follows:

// RepositoryClient operates on a given/specific repository
type RepositoryClient interface {
	// Teams gives access to what teams have access to this specific repository
	Teams() RepositoryTeamsClient

	// Credentials gives access to manipulating credentials for accessing this specific repository
	Credentials() CredentialsClient
}

Repository Teams

RepositoryTeamsClient allows adding & removing teams from the list of authorized persons to access a repository.

// RepositoryTeamsClient operates on the teams list for a specific repository
type RepositoryTeamsClient interface {
	// Add adds a given team in the repo's (top-level) organization to the repository
	Add(ctx context.Context, teamName string, opts RepositoryAddTeamOptions) error

	// Remove removes the given team in the repo's (top-level) organization from the repository
	Remove(ctx context.Context, teamName string) error
}

RepositoryAddTeamOptions has a Permission *TeamRepositoryPermission which allows specifying what access level the team should have.

Repository Credentials

CredentialsClient allows adding & removing credentials (e.g. deploy keys) from accessing a specific repository.

// CredentialsClient operates on the access credential list for a specific repository
type CredentialsClient interface {
	// Create a credential with the given human-readable name, the given bytes and optional options
	Create(ctx context.Context, c RepositoryCredential, opts CredentialCreateOptions) error

	// Lists all credentials
	List(ctx context.Context) ([]RepositoryCredential, error)

	// Deletes a credential from the repo. keyName is the human-friendly title of the credential
	Delete(ctx context.Context, keyName string) error
}

In order to support multiple different types of credentials, RepositoryCredential is an interface:

// RepositoryCredential is a credential that allows
type RepositoryCredential interface {
	// GetType returns the type of the credential
	GetType() RepositoryCredentialType

	// Title returns a description of the credential
	GetTitle() string

	// GetData returns the key that will be authorized to access the repo, this can e.g. be a SSH public key
	GetData() []byte

	// IsReadOnly returns whether this credential is authorized to write to the repository or not
	IsReadOnly() bool
}

The default implementation of RepositoryCredential is DeployKey:

// DeployKey represents a short-lived credential (e.g. an SSH public key) used for accessing a repository
type DeployKey struct {
	// DeployKey embeds InternalHolder for accessing the underlying object
	InternalHolder `json:",inline"`

	// Title is the human-friendly interpretation of what the key is for (and does)
	// +required
	Title string `json:"title"`

	// Key specifies the public part of the deploy (e.g. SSH) key
	// +required
	Key []byte `json:"key"`

	// ReadOnly specifies whether this DeployKey can write to the repository or not
	// Default value at POST-time: true
	// +optional
	ReadOnly *bool `json:"readOnly"`
}