- Incorporate older edits to README !696
- Upgrade to Ruby 3.x !706
- feat: retry on http error !703

feat: retry on http error

Closes #604

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Reviewed-by: Steve Azzopardi <sazzopardi@gitlab.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
- Update the `client.HttpClient` fields to have `http.Client` and
  `retryablehttp.Client`, one of them will be `nil` depending on the
  feature flag toggle.
- Create new method `newRetryableRequest` which will create a
  `retryablehttp.Request` and use that if the
  `FF_GITLAB_SHELL_RETRYABLE_HTTP` feature flag is turned on.
- Add checks for `FF_GITLAB_SHELL_RETRYABLE_HTTP` everywhere we use the
  http client to use the `retryablehttp.Client` or the default
  `http.Client`
- New job `tests-integration-retryableHttp` to run the integraiton tests
  with the new retryablehttp client. We didn't update go tests because
  some assertions are different and will break table driven tests.

Why
---
As discussed in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/703#note_1229645097
we want to put the client behind a feature flag, not just the retry
logic. This does bring extra risk for accessing a `nil` field but there
should be checks everytime we access `RetryableHTTP` and `HTTPClient`.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

What
---
Change the default `HTTP.Client` to
`github.com/hashicorp/go-retryablehttp.Client` to get automatic retries
and exponential backoff.

We retry the request 2 times resulting in 3 attempts of sending the
request, the min retry wait is 1 second, and the maximum is 15
seconds.

Hide the retry logic behind a temporary feature flag
`FF_GITLAB_SHELL_RETRYABLE_HTTP` to easily roll this out in GitLab.com.
When we verify that this works as expected we will remove
`FF_GITLAB_SHELL_RETRYABLE_HTTP` and have the retry logic as the default
logic.

Why
---
In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979 users
end up seeing the following errors when trying to `git-clone(1)` a
repository locally on in CI.

```shell
remote: ===============================
remote:
remote: ERROR: Internal API unreachable
remote:
remote: ================================
```

When we look at the application logs we see the following error:

```json
{ "err": "http://gitlab-webservice-git.gitlab.svc:8181/api/v4/internal/allowed":
dial tcp 10.69.184.120:8181: connect: connection refused", "msg":
"Internal API unreachable"}
```

In
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1222670120
we've correlated these `connection refused` errors with infrastructure
events that remove the git pods that are hosting
`gitlab-webservice-git` service. We could try to make the underlying
infrastructure more reactive to these changes as suggested in
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979#note_1225164944
but we can still end up serving bad requests.

Implementing retry logic for 5xx or other errors would allow users to
still be able to `git-clone(1)` reposirories, although it being slower.
This is espically important during CI runs so users don't have to retry
jobs themselves.

Reference: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979
Closes: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/604


Signed-off-by: Steve Azzopardi <sazzopardi@gitlab.com>

Resolve "Upgrade to Ruby 3.x"

Closes #605

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/706



Merged-by: Stan Hu <stanhu@gmail.com>
Approved-by: James Fargher <proglottis@gmail.com>
Approved-by: Stan Hu <stanhu@gmail.com>
Reviewed-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>

Incorporate older edits to README

Closes #544

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/696



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Jerry Seto <jseto@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Amy Qualls <aqualls@gitlab.com>

Release 14.14.0 version

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/702



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Stan Hu <stanhu@gmail.com>

- Add developer documentation to sshd package !683
- Improve error message for Gitaly `LimitError`s !691
- Drop 1.16 compatibility in go.sum !692
- Bump x/text to 0.3.8 !692
- Update prometheus package to 1.13.1 !692
  Restrict IP access for PROXY protocol !693
- Fix broken Gitaly integration tests !694
- Clean up .gitlab-ci.yml file !695
- Use the images provided by Gitlab to run tests !698
- Use Ruby 2.7.7 as the default !699
- Use blocking reader to fix race in test !700

Update .ruby-version file to match .tool-versions file

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/701



Merged-by: Patrick Bajao <ebajao@gitlab.com>
Approved-by: Patrick Bajao <ebajao@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>

Use Ruby 2.7.7 as the default

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/699



Merged-by: Patrick Bajao <ebajao@gitlab.com>
Approved-by: Patrick Bajao <ebajao@gitlab.com>
Co-authored-by: Ash McKenzie <amckenzie@gitlab.com>

Use blocking reader to fix race in test

Closes #602

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/700



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Patrick Bajao <ebajao@gitlab.com>

The `TestCanceledContext` test in `twofactorverify_test.go` tests
the scenario wherein the request context gets canceled in the
middle of the request.

A race can occur though when the input gets read before the context
is canceled which can result to a different error: instead of
having the context canceled, the OTP will be blank.

To fix it, we use a blocking reader as input to simulate "waiting
for input" scenario. This way, reading the input will never be
finished and the context cancelation can be done appropriately.

Use the images provided by Gitlab to run tests

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/698



Merged-by: Patrick Bajao <ebajao@gitlab.com>
Approved-by: Patrick Bajao <ebajao@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

That would save us from installing additional
dependencies ourselves

Add developer documentation to sshd package

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/683



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Oscar Tovar <otovar@gitlab.com>
Approved-by: James Fargher <proglottis@gmail.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Amy Qualls <aqualls@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Restrict IP access for PROXY protocol

Closes #577

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/693



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>

Clean up .gitlab-ci.yml file

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/695



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

MustStrictWhiteListPolicy panics when configured incorrectly. So here we
use the error returning version instead.

Changelog: added

Instead of interpreting the configuration for every new connection, we
can rely on a closure to simplify the proxy handler path. This is more
similar to how the provided MustStrictWhiteListPolicy works which will
be added in a later commit.

Fix broken Gitaly integration tests

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/694



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Patrick Bajao <ebajao@gitlab.com>

Gitaly started to require `Repository` information for certain
commands like upload-pack, upload-archive, and receive-pack.

However, gitlab-shell tests are sending `git_object_directory` and
`git_alternate_object_directories` that do not exist.

To fix the tests, we set those to an existing directory in the
created repository.

Clean up go.sum by updating libraries in go.mod

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/692



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

It also excludes 1.12.1 version to break the cycling dependency
that loads a lot of obsolete libraries

The changes are the result of running

go mod tidy -compat=1.17

We don't support 1.16 for a while, this command just cleans up the
unnecessary dependencies

Improve error message for Gitaly `LimitError`s

Closes #556

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/691



Merged-by: Igor Drozdov <idrozdov@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Igor Drozdov <idrozdov@gitlab.com>
Co-authored-by: Alejandro Rodríguez <alejorro70@gmail.com>