All the implementations of DNS discovery were done in this epic:
https://gitlab.com/groups/gitlab-org/-/epics/8971. Gitaly allows clients
to configure DNS discovery via dial option. This MR adds the exposed
dial options to client connection creation in Gitlab-shell.

Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4722
Changelog: added

This client bump includes plenty of improvement, espeically the support
for DNS service discovery in Gitaly/Praefect. This version requires Go
>= 1.18. As a result, we'll need to bump minimal Go version of GitLab
Shell accordingly.

Define Do function for Gitlab net client

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/715



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Add bin/gitlab-sshd as an explicit Makefile target

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/714



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Stan Hu <stanhu@gmail.com>

Since https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682,
Kerberos headers and libraries are needed to build gitlab-sshd.  If
they are not available, `make build` successfully compiles
`bin/gitlab-shell` but fails to build `bin/gitlab-sshd`. However,
running `make build` again would do nothing and appear to be succeed
because `bin/gitlab-shell` existed. This led to Omnibus GitLab quietly
dropping the `gitlab-sshd` binary, as seen in
https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6446#note_1265879416.

To ensure `make build` properly fails if `bin/gitlab-sshd` cannot
be built, we make the binary an explicit build target.

Changelog: changed

In future, we'll need to perform http requests for Geo related
code area.

We cannot use retryable requests because:

- It's not necessary for the to be retryable
- In order to retry, the whole request body is stored in RAM,
while we need to stream large blobs of data

This commit:

- Extracts logging into a separate round tripper in order to
reuse it for other http requests by default
- Defines Do function that accepts raw request as an argument

Release v14.17.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/713



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>

Bump golang to 1.18.9

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/712



Merged-by: Ash McKenzie <amckenzie@gitlab.com>

Release v14.16.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/711



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>

feat: make retryable http default client

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/710



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Oscar Tovar <otovar@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
Make the retryableHTTP client introduced in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703 the
default HTTP client.

Why
---
In
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1254964426
we've seen a 99% error reduction on `git` commands from `gitlab-shell`
when the retryableHTTP client is used.

This has been running in production for over 2 weeks in `us-east1-b` and
5 days fleet-wide so we should be confident that this client works as
expected.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

Stub retryable http values in tests

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/708



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Oscar Tovar <otovar@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Oscar Tovar <otovar@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Specify CGO_CFLAGS in Makefile to compile gssapi lib

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/709



Merged-by: Stan Hu <stanhu@gmail.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Currently, the default values are used for retryable http.
That's why a test waits 1 second minimun to retry a request.
Client test takes 25 seconds to execute as a result.
When we stub the value to 1 millisecond instead, we get 0.5s of
execution

Add support for the gssapi-with-mic auth method

Closes #196

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/682



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Patrick Bajao <ebajao@gitlab.com>
Approved-by: Costel Maxim <cmaxim@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Alejandro Rodríguez <alejandro@gitlab.com>
Reviewed-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Patrick Bajao <ebajao@gitlab.com>
Reviewed-by: Rohit Shambhuni <rshambhuni@gitlab.com>
Co-authored-by: Lee Tickett <ltickett@gitlab.com>
Co-authored-by: Marin Hannache <git@mareo.fr>

docs: Truncate pages, point users to GitLab repo

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/705



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Torsten Linz <tlinz@gitlab.com>
Approved-by: Jerry Seto <jseto@gitlab.com>
Approved-by: Sean Carroll <scarroll@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Amy Qualls <aqualls@gitlab.com>

Release 14.15.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/707



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>

- Incorporate older edits to README !696
- Upgrade to Ruby 3.x !706
- feat: retry on http error !703

feat: retry on http error

Closes #604

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Steve Azzopardi <sazzopardi@gitlab.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
- Update the `client.HttpClient` fields to have `http.Client` and
  `retryablehttp.Client`, one of them will be `nil` depending on the
  feature flag toggle.
- Create new method `newRetryableRequest` which will create a
  `retryablehttp.Request` and use that if the
  `FF_GITLAB_SHELL_RETRYABLE_HTTP` feature flag is turned on.
- Add checks for `FF_GITLAB_SHELL_RETRYABLE_HTTP` everywhere we use the
  http client to use the `retryablehttp.Client` or the default
  `http.Client`
- New job `tests-integration-retryableHttp` to run the integraiton tests
  with the new retryablehttp client. We didn't update go tests because
  some assertions are different and will break table driven tests.

Why
---
As discussed in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703#note_1229645097
we want to put the client behind a feature flag, not just the retry
logic. This does bring extra risk for accessing a `nil` field but there
should be checks everytime we access `RetryableHTTP` and `HTTPClient`.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
Change the default `HTTP.Client` to
`github.com/hashicorp/go-retryablehttp.Client` to get automatic retries
and exponential backoff.

We retry the request 2 times resulting in 3 attempts of sending the
request, the min retry wait is 1 second, and the maximum is 15
seconds.

Hide the retry logic behind a temporary feature flag
`FF_GITLAB_SHELL_RETRYABLE_HTTP` to easily roll this out in GitLab.com.
When we verify that this works as expected we will remove
`FF_GITLAB_SHELL_RETRYABLE_HTTP` and have the retry logic as the default
logic.

Why
---
In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 users
end up seeing the following errors when trying to `git-clone(1)` a
repository locally on in CI.

```shell
remote: ===============================
remote:
remote: ERROR: Internal API unreachable
remote:
remote: ================================
```

When we look at the application logs we see the following error:

```json
{ "err": "http://gitlab-webservice-git.gitlab.svc:8181/api/v4/internal/allowed":
dial tcp 10.69.184.120:8181: connect: connection refused", "msg":
"Internal API unreachable"}
```

In
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1222670120
we've correlated these `connection refused` errors with infrastructure
events that remove the git pods that are hosting
`gitlab-webservice-git` service. We could try to make the underlying
infrastructure more reactive to these changes as suggested in
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1225164944
but we can still end up serving bad requests.

Implementing retry logic for 5xx or other errors would allow users to
still be able to `git-clone(1)` reposirories, although it being slower.
This is espically important during CI runs so users don't have to retry
jobs themselves.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979
Closes: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/604


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

Resolve "Upgrade to Ruby 3.x"

Closes #605

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/706



Merged-by: Stan Hu <stanhu@gmail.com>
Approved-by: James Fargher <proglottis@gmail.com>
Approved-by: Stan Hu <stanhu@gmail.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>

Incorporate older edits to README

Closes #544

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/696



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Jerry Seto <jseto@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Amy Qualls <aqualls@gitlab.com>

Release 14.14.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/702



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Stan Hu <stanhu@gmail.com>

- Add developer documentation to sshd package !683
- Improve error message for Gitaly `LimitError`s !691
- Drop 1.16 compatibility in go.sum !692
- Bump x/text to 0.3.8 !692
- Update prometheus package to 1.13.1 !692
  Restrict IP access for PROXY protocol !693
- Fix broken Gitaly integration tests !694
- Clean up .gitlab-ci.yml file !695
- Use the images provided by Gitlab to run tests !698
- Use Ruby 2.7.7 as the default !699
- Use blocking reader to fix race in test !700

Update .ruby-version file to match .tool-versions file

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/701



Merged-by: Patrick Bajao <ebajao@gitlab.com>
Approved-by: Patrick Bajao <ebajao@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>