Perform Git over HTTP request to primary repo

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/716



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Valery Sizov <valery@gitlab.com>
Reviewed-by: Alejandro Rodríguez <alejandro@gitlab.com>
Reviewed-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Currently, we perform a request to Gitlab Rails that proxies
the request to primary

However, it causes timeouts on big pushes and consumes large
amount of memory. We can perform an HTTP request directly
from Gitlab Shell instead and stream the response to the user

sshd: exclude gssapi when building without cgo

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/720



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Vasilii Iakliushin <viakliushin@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Vasilii Iakliushin <viakliushin@gitlab.com>
Co-authored-by: Lorenz Brun <lorenz@brun.one>

MR #682 broke building without cgo enabled as it introduced a dependency
on a Kerberos library. This can only be disabled at runtime and thus
static builds of gitlab-sshd are no longer possible.

This change introduces an alternative implementation of the GSSAPI
structure which just rejects attempts to use it.
That alternative implementation gets automatically activated in case the
user is building without cgo.

Add DNS discovery support to Gitaly via client-side load-balancing options

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/717



Merged-by: Patrick Bajao <ebajao@gitlab.com>
Approved-by: Oscar Tovar <otovar@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Patrick Bajao <ebajao@gitlab.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Oscar Tovar <otovar@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>

All the implementations of DNS discovery were done in this epic:
https://gitlab.com/groups/gitlab-org/-/epics/8971. Gitaly allows clients
to configure DNS discovery via dial option. This MR adds the exposed
dial options to client connection creation in Gitlab-shell.

Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4722
Changelog: added

This client bump includes plenty of improvement, espeically the support
for DNS service discovery in Gitaly/Praefect. This version requires Go
>= 1.18. As a result, we'll need to bump minimal Go version of GitLab
Shell accordingly.

Define Do function for Gitlab net client

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/715



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Add bin/gitlab-sshd as an explicit Makefile target

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/714



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Stan Hu <stanhu@gmail.com>

Since https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682,
Kerberos headers and libraries are needed to build gitlab-sshd.  If
they are not available, `make build` successfully compiles
`bin/gitlab-shell` but fails to build `bin/gitlab-sshd`. However,
running `make build` again would do nothing and appear to be succeed
because `bin/gitlab-shell` existed. This led to Omnibus GitLab quietly
dropping the `gitlab-sshd` binary, as seen in
https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6446#note_1265879416.

To ensure `make build` properly fails if `bin/gitlab-sshd` cannot
be built, we make the binary an explicit build target.

Changelog: changed

In future, we'll need to perform http requests for Geo related
code area.

We cannot use retryable requests because:

- It's not necessary for the to be retryable
- In order to retry, the whole request body is stored in RAM,
while we need to stream large blobs of data

This commit:

- Extracts logging into a separate round tripper in order to
reuse it for other http requests by default
- Defines Do function that accepts raw request as an argument

Release v14.17.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/713



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>

Bump golang to 1.18.9

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/712



Merged-by: Ash McKenzie <amckenzie@gitlab.com>

Release v14.16.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/711



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>

feat: make retryable http default client

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/710



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Oscar Tovar <otovar@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
Make the retryableHTTP client introduced in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703 the
default HTTP client.

Why
---
In
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1254964426
we've seen a 99% error reduction on `git` commands from `gitlab-shell`
when the retryableHTTP client is used.

This has been running in production for over 2 weeks in `us-east1-b` and
5 days fleet-wide so we should be confident that this client works as
expected.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

Stub retryable http values in tests

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/708



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Oscar Tovar <otovar@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Oscar Tovar <otovar@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Specify CGO_CFLAGS in Makefile to compile gssapi lib

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/709



Merged-by: Stan Hu <stanhu@gmail.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Currently, the default values are used for retryable http.
That's why a test waits 1 second minimun to retry a request.
Client test takes 25 seconds to execute as a result.
When we stub the value to 1 millisecond instead, we get 0.5s of
execution

Add support for the gssapi-with-mic auth method

Closes #196

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Patrick Bajao <ebajao@gitlab.com>
Approved-by: Costel Maxim <cmaxim@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Alejandro Rodríguez <alejandro@gitlab.com>
Reviewed-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Patrick Bajao <ebajao@gitlab.com>
Reviewed-by: Rohit Shambhuni <rshambhuni@gitlab.com>
Co-authored-by: Lee Tickett <ltickett@gitlab.com>
Co-authored-by: Marin Hannache <git@mareo.fr>

docs: Truncate pages, point users to GitLab repo

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/705



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Torsten Linz <tlinz@gitlab.com>
Approved-by: Jerry Seto <jseto@gitlab.com>
Approved-by: Sean Carroll <scarroll@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Amy Qualls <aqualls@gitlab.com>

Release 14.15.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/707



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>

- Incorporate older edits to README !696
- Upgrade to Ruby 3.x !706
- feat: retry on http error !703

feat: retry on http error

Closes #604

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Steve Azzopardi <sazzopardi@gitlab.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
- Update the `client.HttpClient` fields to have `http.Client` and
  `retryablehttp.Client`, one of them will be `nil` depending on the
  feature flag toggle.
- Create new method `newRetryableRequest` which will create a
  `retryablehttp.Request` and use that if the
  `FF_GITLAB_SHELL_RETRYABLE_HTTP` feature flag is turned on.
- Add checks for `FF_GITLAB_SHELL_RETRYABLE_HTTP` everywhere we use the
  http client to use the `retryablehttp.Client` or the default
  `http.Client`
- New job `tests-integration-retryableHttp` to run the integraiton tests
  with the new retryablehttp client. We didn't update go tests because
  some assertions are different and will break table driven tests.

Why
---
As discussed in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703#note_1229645097
we want to put the client behind a feature flag, not just the retry
logic. This does bring extra risk for accessing a `nil` field but there
should be checks everytime we access `RetryableHTTP` and `HTTPClient`.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
Change the default `HTTP.Client` to
`github.com/hashicorp/go-retryablehttp.Client` to get automatic retries
and exponential backoff.

We retry the request 2 times resulting in 3 attempts of sending the
request, the min retry wait is 1 second, and the maximum is 15
seconds.

Hide the retry logic behind a temporary feature flag
`FF_GITLAB_SHELL_RETRYABLE_HTTP` to easily roll this out in GitLab.com.
When we verify that this works as expected we will remove
`FF_GITLAB_SHELL_RETRYABLE_HTTP` and have the retry logic as the default
logic.

Why
---
In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 users
end up seeing the following errors when trying to `git-clone(1)` a
repository locally on in CI.

```shell
remote: ===============================
remote:
remote: ERROR: Internal API unreachable
remote:
remote: ================================
```

When we look at the application logs we see the following error:

```json
{ "err": "http://gitlab-webservice-git.gitlab.svc:8181/api/v4/internal/allowed":
dial tcp 10.69.184.120:8181: connect: connection refused", "msg":
"Internal API unreachable"}
```

In
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1222670120
we've correlated these `connection refused` errors with infrastructure
events that remove the git pods that are hosting
`gitlab-webservice-git` service. We could try to make the underlying
infrastructure more reactive to these changes as suggested in
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1225164944
but we can still end up serving bad requests.

Implementing retry logic for 5xx or other errors would allow users to
still be able to `git-clone(1)` reposirories, although it being slower.
This is espically important during CI runs so users don't have to retry
jobs themselves.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979
Closes: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/604


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

Resolve "Upgrade to Ruby 3.x"

Closes #605

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/706



Merged-by: Stan Hu <stanhu@gmail.com>
Approved-by: James Fargher <proglottis@gmail.com>
Approved-by: Stan Hu <stanhu@gmail.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>