This explicitly imports gitaly v16 instead of v15 and removes the
old one from `go.mod`.

Currently, the client that performs HTTP requests for Geo pushes
is the same as the one that performs internal HTTP requests.

However, it causes problems with the following setups:

- Internal requests are configured to be via UNIX connections
- The Primary node is configured to be HTTP(S)

The UNIX client cannot do the request to HTTP(S).

This commit introduces a separate client for Geo requests.

All the implementations of DNS discovery were done in this epic:
https://gitlab.com/groups/gitlab-org/-/epics/8971. Gitaly allows clients
to configure DNS discovery via dial option. This MR adds the exposed
dial options to client connection creation in Gitlab-shell.

Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4722
Changelog: added

In future, we'll need to perform http requests for Geo related
code area.

We cannot use retryable requests because:

- It's not necessary for the to be retryable
- In order to retry, the whole request body is stored in RAM,
while we need to stream large blobs of data

This commit:

- Extracts logging into a separate round tripper in order to
reuse it for other http requests by default
- Defines Do function that accepts raw request as an argument

What
---
Make the retryableHTTP client introduced in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703 the
default HTTP client.

Why
---
In
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1254964426
we've seen a 99% error reduction on `git` commands from `gitlab-shell`
when the retryableHTTP client is used.

This has been running in production for over 2 weeks in `us-east1-b` and
5 days fleet-wide so we should be confident that this client works as
expected.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

Currently, the default values are used for retryable http.
That's why a test waits 1 second minimun to retry a request.
Client test takes 25 seconds to execute as a result.
When we stub the value to 1 millisecond instead, we get 0.5s of
execution

What
---
- Update the `client.HttpClient` fields to have `http.Client` and
  `retryablehttp.Client`, one of them will be `nil` depending on the
  feature flag toggle.
- Create new method `newRetryableRequest` which will create a
  `retryablehttp.Request` and use that if the
  `FF_GITLAB_SHELL_RETRYABLE_HTTP` feature flag is turned on.
- Add checks for `FF_GITLAB_SHELL_RETRYABLE_HTTP` everywhere we use the
  http client to use the `retryablehttp.Client` or the default
  `http.Client`
- New job `tests-integration-retryableHttp` to run the integraiton tests
  with the new retryablehttp client. We didn't update go tests because
  some assertions are different and will break table driven tests.

Why
---
As discussed in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703#note_1229645097
we want to put the client behind a feature flag, not just the retry
logic. This does bring extra risk for accessing a `nil` field but there
should be checks everytime we access `RetryableHTTP` and `HTTPClient`.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
Change the default `HTTP.Client` to
`github.com/hashicorp/go-retryablehttp.Client` to get automatic retries
and exponential backoff.

We retry the request 2 times resulting in 3 attempts of sending the
request, the min retry wait is 1 second, and the maximum is 15
seconds.

Hide the retry logic behind a temporary feature flag
`FF_GITLAB_SHELL_RETRYABLE_HTTP` to easily roll this out in GitLab.com.
When we verify that this works as expected we will remove
`FF_GITLAB_SHELL_RETRYABLE_HTTP` and have the retry logic as the default
logic.

Why
---
In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 users
end up seeing the following errors when trying to `git-clone(1)` a
repository locally on in CI.

```shell
remote: ===============================
remote:
remote: ERROR: Internal API unreachable
remote:
remote: ================================
```

When we look at the application logs we see the following error:

```json
{ "err": "http://gitlab-webservice-git.gitlab.svc:8181/api/v4/internal/allowed":
dial tcp 10.69.184.120:8181: connect: connection refused", "msg":
"Internal API unreachable"}
```

In
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1222670120
we've correlated these `connection refused` errors with infrastructure
events that remove the git pods that are hosting
`gitlab-webservice-git` service. We could try to make the underlying
infrastructure more reactive to these changes as suggested in
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1225164944
but we can still end up serving bad requests.

Implementing retry logic for 5xx or other errors would allow users to
still be able to `git-clone(1)` reposirories, although it being slower.
This is espically important during CI runs so users don't have to retry
jobs themselves.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979
Closes: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/604


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

Now the requests are verified via JWT

With this change we don't rely on the secret to either contain
a newline or not contain it.

This commit also excludes gitlab-shell from dependencies:

Gitaly specifies Gitlab Shell as a dependency as well in order
to use gitlabnet client to perform API endpoints to Gitlab Rails.
As a result, Gitlab Shell requires Gitaly -> Gitaly requires an
older version of Gitlab Shell -> that version requires an older
version of Gitlab Shell, etc. Let's use exclude to break the
chain earlier

While gitlab-shell currently has a major version of v14, the module path
it exposes is not using that major version like it is required by the Go
standard. This makes it impossible for dependents to import gitlab-shell
as a dependency without using a commit as version.

Fix this by changing the module path of gitlab-shell to instead be
`gitlab.com/gitlab-org/gitlab-shell/v14` and adjust all imports
accordingly.

Changelog: fixed

When API isn't responsible or the resource is not accessible
(returns 404 or 403), then we shouldn't consider it as an error
on gitlab-sshd side

It is passed as a Gitlab-Shell-Api-Request header and uses
the same shared secret in order to encrypt the token

Contributes to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/484

Changelog: removed

Contributes to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/541

Changelog: removed

Jacob Vosmaer authored 3 years ago and Igor Drozdov committed 3 years ago

If the GitLab API returns an allowed response with use_sidechannel set
to true, gitlab-shell will establish a sidechannel connection and use
SSHUploadPackWithSidechannel instead of SSHUploadPack. This is an
efficiency improvement.

Jacob Vosmaer authored 3 years ago and Igor Drozdov committed 3 years ago

This updates the Gitaly client go.mod dependency to Gitaly commit
2e398afa0490ccdf5a82e1a7c7d824ae491eba16. This causes a grpc-go
version bump, and hence a minor change in some of our test code.

The option isn't required to accept self-signed certs

On the other hand, if the option set to true it makes
machine-in-the-middle attack possible

Let's clarify it in the code that the option is deprecated

This fixes a regression in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/508. If an
HTTPS internal API URL were used, gitlab-shell would not work at all. We
now handle blank `caFile` properly.

Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/529

Logrus buffers its output internally, which makes these tests fail
intermittently. They're also not a good example to follow generally.

We now have acceptance tests that exercise this functionality so I'm
pretty relaxed about losing the expectations. However, we can test
them by inspecting the server-received metadata too, so there's no loss
of coverage here.

The move from logrus to labkit for logging also makes these tests hard
to justify keeping.

Gitaly project now properly respects module release flow
and includes a module suffix in the package name. It requires
to re-write all non-suffixed imports with suffixed of a specific
version of tha module. With proper module versioning we don't
need to use a 'replace' directive to point to specific commit
and can use semantic versioning for the gitaly dependency.

Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3177

In this case we don't need to propagate cleanup
function. It simplifies the code.