All the implementations of DNS discovery were done in this epic:
https://gitlab.com/groups/gitlab-org/-/epics/8971. Gitaly allows clients
to configure DNS discovery via dial option. This MR adds the exposed
dial options to client connection creation in Gitlab-shell.

Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4722
Changelog: added

While gitlab-shell currently has a major version of v14, the module path
it exposes is not using that major version like it is required by the Go
standard. This makes it impossible for dependents to import gitlab-shell
as a dependency without using a commit as version.

Fix this by changing the module path of gitlab-shell to instead be
`gitlab.com/gitlab-org/gitlab-shell/v14` and adjust all imports
accordingly.

Changelog: fixed

Before this change, the GitLab internal API could use a boolean
response field to indicate whether gitlab-shell should make
sidechannel connections go Gitaly. We now ignore that response field
and always use sidechannel connections.

Logrus buffers its output internally, which makes these tests fail
intermittently. They're also not a good example to follow generally.

We now have acceptance tests that exercise this functionality so I'm
pretty relaxed about losing the expectations. However, we can test
them by inspecting the server-received metadata too, so there's no loss
of coverage here.

The move from logrus to labkit for logging also makes these tests hard
to justify keeping.

Previously, opentracing (if configured) was initialized late in the
gitlab-shell process's lifespan, coming just before making a gRPC
call to Gitaly.

By moving the opentracing initialization to be at process startup, we
make it available for the whole process lifecycle, which is very useful
to gitlab-sshd, as it means we'll only call tracing.Initialize() once
on process startup, rather than once per SSH connection.

To get this working, we need to introduce a context to gitlab-sshd.
This carries the client/service name, but also carries an initial
correlation ID. The main outcome of this is that all calls to the
authorized_keys endpoint from a given gitlab-sshd process will now
share a correlation ID. I don't have a strong opinion about this either
way.

Changelog: fixed

Without these changes, Gitaly calls would not be linked to a parent
context. This means that they would have an unassociated correlationID,
and Gitaly RPC calls would not be cancel()ed by parent context
cancellation.

Changelog: fixed

In this case we don't need to propagate cleanup
function. It simplifies the code.

Refactors introspection of execution environment to rely on
per-connection state (`gitlab-shell`) or per request (`gitlab-sshd`)

Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/496

During a SSH receive-pack request (e.g. `git push`), gitlab-shell was
incorrectly using the user returned by the `/internal/allowed` API
endpoint to make an SSHReceivePack RPC call. This caused a number of
problems with deploy keys with write access:

1. Keys that were generated by a blocked user would be denied the
ability to write.

2. Keys that were generated by user that did not have write access to
the project would also be denied.

GitLab 12.4 removed the Ruby implementation of gitlab-shell in favor of
the Golang implementation, and these implementations worked slightly
differently. In
https://gitlab.com/gitlab-org/gitlab-shell/blob/v10.1.0/lib/gitlab_shell.rb,
the Ruby implementation would always use `@who` (e.g. `key-123`), but in
gitlab-shell v10.2.0 the Go implementation would always use the user
from the API response.

Reads did not have this issue because the user/deploy key is never
passed to Gitaly for additional permission checks. Writes need this
information for the pre-receive to check access to protected branches,
push rules, etc.

Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/479

Previously, gitlab-shell did not pass a context through the application.
Correlation IDs were generated down the call stack instead of passed
around from the start execution.

This has several potential downsides:

1. It's easier for programming mistakes to be made in future that lead
to multiple correlation IDs being generated for a single request.
2. Correlation IDs cannot be passed in from upstream requests
3. Other advantages of context passing, such as distributed tracing is
not possible.

This commit changes the behavior:

1. Extract the correlation ID from the environment at the start of
the application.
2. If no correlation ID exists, generate a random one.
3. Pass the correlation ID to the GitLabNet API requests.

This change also enables other clients of GitLabNet (e.g. Gitaly) to
pass along the correlation ID in the internal API requests
(https://gitlab.com/gitlab-org/gitaly/-/issues/2725).

Fixes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/474

Stan Hu authored 4 years ago and Igor Drozdov committed 4 years ago

This will make it easier to tie an SSH access request to Rails API and
Gitaly requests.

Stan Hu authored 4 years ago and Igor Drozdov committed 4 years ago

Right now when a client such as gitlab-shell calls the
`/api/v4/internal/allowed` API, the response only tells the client what
user has been granted access, and it's impossible to tell which deploy
key/token was used in the authentication request.

This commit adds logs for the following when available:

1. `gl_key_type` (e.g. `deploy_key` or `key`)
2. `gl_key_id`

These fields make it possible for admins to identify the exact record
that was used to authenticate the user.

API changes in the `/internal/allowed` endpoint in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37289 are needed
to support this.

Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/203

Admins may want to know what client IP originated the request. This
commit adds a `remote_ip` field to the log that extracts the IP address
from the `SSH_CONNECTION` environment variable.

Closes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/199

logrus fires a Goroutine to write logs, so the tests could fail if they
checked the event queue before the logrus have fired.  Since there isn't
an easy way to flush all outstanding hooks, we just retry every 100 ms
for up to a second for log to arrive in the queue.

Closes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/450

Calling logrus hook.LastEntry() can lead to race conditions.  Use
AllEntries instead:
https://github.com/sirupsen/logrus/blob/60c74ad9be0d874af0ab0daef6ab07c5c5911f0d/hooks/test/test.go#L77

Closes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/450

Other functions are still expecting for `CommandArgs` instead
of `Shell`. They should be expecting `commandargs.Shell` now
since it has been renamed.