Now the requests are verified via JWT

With this change we don't rely on the secret to either contain
a newline or not contain it.

This commit also excludes gitlab-shell from dependencies:

Gitaly specifies Gitlab Shell as a dependency as well in order
to use gitlabnet client to perform API endpoints to Gitlab Rails.
As a result, Gitlab Shell requires Gitaly -> Gitaly requires an
older version of Gitlab Shell -> that version requires an older
version of Gitlab Shell, etc. Let's use exclude to break the
chain earlier

While gitlab-shell currently has a major version of v14, the module path
it exposes is not using that major version like it is required by the Go
standard. This makes it impossible for dependents to import gitlab-shell
as a dependency without using a commit as version.

Fix this by changing the module path of gitlab-shell to instead be
`gitlab.com/gitlab-org/gitlab-shell/v14` and adjust all imports
accordingly.

Changelog: fixed

When API isn't responsible or the resource is not accessible
(returns 404 or 403), then we shouldn't consider it as an error
on gitlab-sshd side

It is passed as a Gitlab-Shell-Api-Request header and uses
the same shared secret in order to encrypt the token

Contributes to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/484

Changelog: removed

Contributes to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/541

Changelog: removed

Jacob Vosmaer authored 3 years ago and Igor Drozdov committed 3 years ago

If the GitLab API returns an allowed response with use_sidechannel set
to true, gitlab-shell will establish a sidechannel connection and use
SSHUploadPackWithSidechannel instead of SSHUploadPack. This is an
efficiency improvement.

Jacob Vosmaer authored 3 years ago and Igor Drozdov committed 3 years ago

This updates the Gitaly client go.mod dependency to Gitaly commit
2e398afa0490ccdf5a82e1a7c7d824ae491eba16. This causes a grpc-go
version bump, and hence a minor change in some of our test code.

The option isn't required to accept self-signed certs

On the other hand, if the option set to true it makes
machine-in-the-middle attack possible

Let's clarify it in the code that the option is deprecated

This fixes a regression in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/508. If an
HTTPS internal API URL were used, gitlab-shell would not work at all. We
now handle blank `caFile` properly.

Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/529

Logrus buffers its output internally, which makes these tests fail
intermittently. They're also not a good example to follow generally.

We now have acceptance tests that exercise this functionality so I'm
pretty relaxed about losing the expectations. However, we can test
them by inspecting the server-received metadata too, so there's no loss
of coverage here.

The move from logrus to labkit for logging also makes these tests hard
to justify keeping.

Gitaly project now properly respects module release flow
and includes a module suffix in the package name. It requires
to re-write all non-suffixed imports with suffixed of a specific
version of tha module. With proper module versioning we don't
need to use a 'replace' directive to point to specific commit
and can use semantic versioning for the gitaly dependency.

Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3177

In this case we don't need to propagate cleanup
function. It simplifies the code.

This will be useful to measure bandwidth sent in response to an API
request, particularly with measuring the /api/v4/internal/lfs endpoint.

Testify features sub packages `assert` and `require`. The difference is
subtle, and lost on novice Golang developers that don't read the docs.
To create a more consistent code base `assert` will no longer be used.

This change was generated by a running a sed command on all `_test.go`
files, followed by `goimports -w`.

The user agent for requests to the internal API endpoints used the
default Go provided user agent. This change updates that to always set
something else, by default `GitLab-Shell`.

Than for others importing the package, there's a new API to set it to
something else. This has been done with new method, a setter, to
maintain backwards compatibility in the API.

Previously, gitlab-shell did not pass a context through the application.
Correlation IDs were generated down the call stack instead of passed
around from the start execution.

This has several potential downsides:

1. It's easier for programming mistakes to be made in future that lead
to multiple correlation IDs being generated for a single request.
2. Correlation IDs cannot be passed in from upstream requests
3. Other advantages of context passing, such as distributed tracing is
not possible.

This commit changes the behavior:

1. Extract the correlation ID from the environment at the start of
the application.
2. If no correlation ID exists, generate a random one.
3. Pass the correlation ID to the GitLabNet API requests.

This change also enables other clients of GitLabNet (e.g. Gitaly) to
pass along the correlation ID in the internal API requests
(https://gitlab.com/gitlab-org/gitaly/-/issues/2725).

Fixes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/474

From
https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4498#note_397401883,
if you specify a relative path such as:

```
external_url 'http://gitlab.example.com/gitlab'
```

gitlab-shell doesn't have a way to pass the `/gitlab` to the host. For example, let's say we have:

```
gitlab_url: "http+unix://%2Fvar%2Fopt%2Fgitlab%2Fgitlab-workhorse%2Fsocket"
```

If we have `/gitlab` as the relative path, how do we specify what is the
UNIX socket path and what is the relative path? If we specify:

```
gitlab_url: "http+unix:///var/opt/gitlab/gitlab-workhorse.socket/gitlab
```

This is ambiguous. Is the socket in
`/var/opt/gitlab/gitlab-workhorse.socket/gitlab` or in
`/var/opt/gitlab/gitlab-workhorse.socket`?

To fix this, this merge request adds an optional
`gitlab_relative_url_root` config parameter:

```
gitlab_url: "http+unix://%2Fvar%2Fopt%2Fgitlab%2Fgitlab-workhorse%2Fsocket"
gitlab_relative_url_root: /gitlab
```

This is only used with UNIX domain sockets to disambiguate the socket
and base URL path. If `gitlab_url` uses `http://` or `https://`, then
`gitlab_relative_url_root` is ignored.

Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/476

Stan Hu authored 4 years ago and Igor Drozdov committed 4 years ago

This will make it easier to tie an SSH access request to Rails API and
Gitaly requests.

logrus fires a Goroutine to write logs, so the tests could fail if they
checked the event queue before the logrus have fired.  Since there isn't
an easy way to flush all outstanding hooks, we just retry every 100 ms
for up to a second for log to arrive in the queue.

Closes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/450

Calling logrus hook.LastEntry() can lead to race conditions.  Use
AllEntries instead:
https://github.com/sirupsen/logrus/blob/60c74ad9be0d874af0ab0daef6ab07c5c5911f0d/hooks/test/test.go#L77

Closes https://gitlab.com/gitlab-org/gitlab-shell/-/issues/450

This would make it easier to filter the logs by status code.