Merge branch 'id-add-sshd-dev-docs' into 'main'

Add developer documentation to sshd package

See merge request https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/683



Merged-by: Ash McKenzie <amckenzie@gitlab.com>
Approved-by: Alejandro Rodríguez <alejandro@gitlab.com>
Approved-by: Oscar Tovar <otovar@gitlab.com>
Approved-by: James Fargher <proglottis@gmail.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Amy Qualls <aqualls@gitlab.com>
Approved-by: Ash McKenzie <amckenzie@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

internal/sshd/README.md

+---
+stage: Create
+group: Source Code
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Overview
+
+This package is responsible for handling SSH connections.
+
+Since GitLab [13.9](https://gitlab.com/groups/gitlab-org/-/milestones/56), `gitlab-shell` provides an option to either use [OpenSSH](https://github.com/openssh/openssh-portable) to handle connections or utilize an internal implementation. For more information on the internal implementation, see the [Why we implemented our own SSHD solution](https://about.gitlab.com/blog/2022/08/17/why-we-have-implemented-our-own-sshd-solution-on-gitlab-sass/) blog post. 
+
+The package contains multiple public functions:
+
+- [func NewServer](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L43) for initializing [Server](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L33).
+- [func (s \*Server) ListenAndServe](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L52) for listening to new connections and serving them.
+- [func (s \*Server) Shutdown](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L63) for shutting down the server.
+- [func (s \*Server) MonitoringServeMux](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L73) that provides monitoring endpoints.
+
+All these functions are used by the [`main`](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/822e49b34afbc2092ae189091d693ae7867a8e5a/cmd/gitlab-sshd/main.go) function to initialize and shutdown the server. When a client tries to connect to the server, the following sequence of steps happens:
+
+- When a new connection is [accepted](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L118) by the listener, it's [asynchronously handled](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L129).
+- A new connection is [created](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L190) with [a customized server config](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/server_config.go#L148) that allows the server to perform [the handshake](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/connection.go#L77) that [verifies](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/server_config.go#L151) the authorized keys via [a request](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/server_config.go#L140) to Gitlab Rails. The server is ready to [handle sessions](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L65) via this connection. The reason why we [handle the sessions in a loop](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L98) is the [persisting connections](https://man.openbsd.org/ssh_config.5#ControlPersist): a connection remains open and reused for the future sessions.
+- A new session is [handled](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L135) by the [channelHandler](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L39) function passed as a [parameter](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L199). This function [processes](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L56) requests from the client.
+- The client may send [env requests](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L67) in order to set an environment variable but we [restrict it](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L113) to only setting Git protocol version. The client eventually [executes a command](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L68-74) either via `shell` or `exec` requests.
+- When the command is [executed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L143), we [create a command](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L165) of a [particular type](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-shell/command/command.go#L57). More information on how the commands work can be found [here](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/command).
+- When a command is executed, the [session is closed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L204) and the error about unsuccessful execution, if such took place, is [tracked](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L137).
+- If a connection has been prematurely closed we [block execution](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L146) until all concurrent sessions are released.
+
+## Proxy protocol
+
+The package supports creating a server with PROXY protocol. The [`go-proxyproto`](https://github.com/pires/go-proxyproto) package is used to [wrap](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L98) the basic listener into the one that supports PROXY protocol. PROXY protocol enables us to implement [Group IP address restriction via SSH](https://gitlab.com/gitlab-org/gitlab/-/issues/271673). The policies are [configurable](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L203).
+
+## Configurable OpenSSH alternatives
+
+- [LoginGraceTime](https://man7.org/linux/man-pages/man5/sshd_config.5.html) is [implemented](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L73) via TCP deadlines.
+- [ClientAliveInterval](https://man7.org/linux/man-pages/man5/sshd_config.5.html) is [implemented](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L151) via sending [keep-alive message](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L25) periodically.
+
+## State machine
+
+The server [maintains a state machine](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L26-31) to implement:
+
+- **Graceful shutdown.** When a termination signal [has been detected](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L96), then a service [is being shut down](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L105). The status is [changed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L68) accordingly and no new connections [are accepted](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L120). A configurable [grace period is given](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L107) in order to allow the ongoing connections to complete. When the period expires, then the top-level context is [canceled](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L109). That means that all the ongoing HTTP and SSH connections are [closed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L173).
+- **Liveness and readiness probes** that help Kubernetes to evaluate the state of the server. If a state [is any other than ready](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L80) (for example, during graceful shutdown), then 502 is returned.

internal/sshd/session.go

@@ -66,8 +66,12 @@ func (s *session) handle(ctx context.Context, requests <-chan *ssh.Request) erro
 		case "env":
 			shouldContinue, err = s.handleEnv(ctx, req)
 		case "exec":
+			// The command has been executed as `ssh user@host command` or `exec` channel has been used
+			// in the app implementation
 			shouldContinue, err = s.handleExec(ctx, req)
 		case "shell":
+			// The command has been entered into the shell or `shell` channel has been used
+			// in the app implementation
 			shouldContinue = false
 			var status uint32
 			status, err = s.handleShell(ctx, req)