[mosh-devel] Mosh on Kuberbnetes

[Thomas Buckley-Houston]

Hello Mosh Developers,

This is my first post to the list. Thanks for the introduction Keith.

I'm the author of Texttop (soon to be re-released as Browsh):
https://github.com/tombh/texttop It is a fully-modern text-based
browser.

You can actually SSH into a demo: `ssh brow.sh` (no auth needed). This
is not a publicised service yet, there are only 3 instances, so don't
share it too much.

Anyway, my problem is that the SSH service is running on Kubernetes
behind a load balancer and of course the load balancer likes to spread
out connections to the instances. Thus, by default, it cannot
guarantee that, when using Mosh, the upgrade from SSH port 22 to Mosh
port 60001 arrives at the same instance.

I won't go into why, but it'll be a significant hurdle to enable
IP-based sticky sessions, so that users are guaranteed the same
instance. So first I just want to cross other approaches off my list.
My first thought is whether anyone has ever wrapped mosh-server? Eg;
How straight forward would it be to have a proxy listening on Mosh
ports to intercept the handshake from mosh-client? That way I could
have my own centralised token management, that once verified the
token, could fire up mosh-server and feed it the incoming connection.

At the very least, I could actually get away with just running
mosh-server without key checking, as this is currently all only for
demo purposes and no actual authentication is needed. I just need
mosh-server to refuse more than 1 connection on the same port, which I
assume it already does? Would it be as simple as patching a single
line to disable key checking?

Many thanks for any ideas or feedback,
Tom

[Anders Kaseorg]

You may have a misunderstanding about how a Mosh session is set up. The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and writes
them to stdout, where they go back over SSH to the mosh script; then the
mosh script launches mosh-client passing the IP address, port number, and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication is
successful if the packets can be decrypted.

There’s no separate “key checking” step to be disabled. And it doesn’t
make sense to “refuse more than 1 connection on the same port”, both
because UDP is connectionless, and because a new mosh-server is launched
on a new port for each Mosh session (it is not a daemon like sshd).

The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses, or to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script resolve the
hostname only once and use the same address for the SSH connection and the
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool.

If that’s not an option and you really need all connections to go through
a single load balancer address, you could try wrapping mosh-server in a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends from
the load balancer.

Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set
with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.

Anders

[Thomas Buckley-Houston]

Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.

And thanks so much for the heads up about my DNSSEC records. I've sent
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey warning.
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as I
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up. The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and writes
them to stdout, where they go back over SSH to the mosh script; then the
mosh script launches mosh-client passing the IP address, port number, and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication is
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it doesn’t
make sense to “refuse more than 1 connection on the same port”, both
because UDP is connectionless, and because a new mosh-server is launched
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses, or to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script resolve the
hostname only once and use the same address for the SSH connection and the
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool.
If that’s not an option and you really need all connections to go through
a single load balancer address, you could try wrapping mosh-server in a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends from
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set
with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.
Anders

[john hood]

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.

A simple DNS round-robin may not scale, but CDNs use DNS to load
balance and geolocate traffic all the time. DNS load balancing is not
as immediate as a middlebox, but you are going to be wrangling a
slow-moving load of long-lived user sessions, not HTTP connections.
If you do end up scaling to thousands of servers, you'll need to do
some DNS-based dynamic management anyway.

Since Mosh fundamentally needs to know the same destination for its
SSH and Mosh session, it doesn't mesh well with load balancers or
other technologies that obscure destination addresses.

Also, brow.sh doesn't work for me, without any DNSSEC involved-- it
looks like ns?.dnsimple.com are not returning authoritative answers
for brow.sh (though the IP addresses they give out does work).

regards,

--jh

[Keith Winstein]

Hi Thomas,

Glad you could provoke a very interesting discussion! But I'm still
confused -- how is "sticky IP-based routing" going to work after the client
roams to a new IP address (or to a new UDP source port)? When your system
seems an incoming UDP datagram from a previously unseen source IP:port, how
does it know which mosh-server (on which server machine) to send it to?

With off-the-shelf Mosh, you basically need a load-balancing strategy that
allows a destination IP:port to uniquely identify a particular mosh-server.
You can do this with multiple DNS A/AAAA records (where the client picks
the winning one -- maybe you permute the list), or with a smart DNS server
that serves *one* A or AAAA record to the client at the time of resolution
(like a CDN would use).

Instead of using the mosh wrapper script, you could have your users use
some other scheme to figure out the IP:port of the server, but the point is
that once you launch the mosh-client, it's going to keep sending datagrams
to the IP:port of the mosh-server, and those datagrams need to get to the
same mosh-server process even if the client roams to a different
publicly-visible IP address or port.

You could imagine writing a very smart mosh proxy that has the keys to all
the sessions and can figure out (for an incoming datagram coming from an
unknown source IP:port) which session it actually belongs to, and then
makes a sticky mapping and routes it to the proper mosh-server. But I don't
think anybody has actually done this yet and of course there's a challenge
in making this reliable/replicated.

-Keith

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.
And thanks so much for the heads up about my DNSSEC records. I've sent
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey warning.
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as I
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up. The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and writes
them to stdout, where they go back over SSH to the mosh script; then the
mosh script launches mosh-client passing the IP address, port number, and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication is
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it doesn’t
make sense to “refuse more than 1 connection on the same port”, both
because UDP is connectionless, and because a new mosh-server is launched
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses, or to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script resolve the
hostname only once and use the same address for the SSH connection and

the

Post by Anders Kaseorg
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool.
If that’s not an option and you really need all connections to go through
a single load balancer address, you could try wrapping mosh-server in a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends from
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set
with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.
Anders

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Thomas Buckley-Houston]

Hey Keith, John, everyone,

Yeah the more this is looking like a quite a big hurdle. Especially
your point Keith about roaming IPs (which I'd forgotten), it's a
central feature of Mosh I don't want to lose.

So the only 2 options seems to be exposing multiple IPs for Round
Robin (or other smart DNS routing) or writing a new Mosh proxy that
already has knowledge of the existing keys. Both seem like quite a
challenge. Round Robin DNS seems more approachable and I can imagine
integrating it with the Google Cloud DNS API I'm already using, but I
just wonder how expensive Google (or anyone for that matter) will make
thousands of static IP addresses? Apart from me having to learn Mosh
internals, one difficulty that strikes me about a Mosh proxy is that
it might introduce a non-trivial delay to each datagram arriving?
Though surely only ever in the order of a handful of milliseconds I
suppose.

Are there not any other identifying marks to a datagram, I don't know
much about low level networking, but maybe something like a MAC
address for example?

Thanks,
Tom

Post by Keith Winstein
Hi Thomas,
Glad you could provoke a very interesting discussion! But I'm still confused
-- how is "sticky IP-based routing" going to work after the client roams to
a new IP address (or to a new UDP source port)? When your system seems an
incoming UDP datagram from a previously unseen source IP:port, how does it
know which mosh-server (on which server machine) to send it to?
With off-the-shelf Mosh, you basically need a load-balancing strategy that
allows a destination IP:port to uniquely identify a particular mosh-server.
You can do this with multiple DNS A/AAAA records (where the client picks the
winning one -- maybe you permute the list), or with a smart DNS server that
serves *one* A or AAAA record to the client at the time of resolution (like
a CDN would use).
Instead of using the mosh wrapper script, you could have your users use some
other scheme to figure out the IP:port of the server, but the point is that
once you launch the mosh-client, it's going to keep sending datagrams to the
IP:port of the mosh-server, and those datagrams need to get to the same
mosh-server process even if the client roams to a different publicly-visible
IP address or port.
You could imagine writing a very smart mosh proxy that has the keys to all
the sessions and can figure out (for an incoming datagram coming from an
unknown source IP:port) which session it actually belongs to, and then makes
a sticky mapping and routes it to the proper mosh-server. But I don't think
anybody has actually done this yet and of course there's a challenge in
making this reliable/replicated.
-Keith

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.
And thanks so much for the heads up about my DNSSEC records. I've sent
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey warning.
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as I
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up. The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and writes
them to stdout, where they go back over SSH to the mosh script; then the
mosh script launches mosh-client passing the IP address, port number, and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication is
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it doesn’t
make sense to “refuse more than 1 connection on the same port”, both
because UDP is connectionless, and because a new mosh-server is launched
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses, or to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script resolve the
hostname only once and use the same address for the SSH connection and the
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool.
If that’s not an option and you really need all connections to go through
a single load balancer address, you could try wrapping mosh-server in a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends from
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set
with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.
Anders

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Keith Winstein]

How about a semi-smart (but mostly Mosh-oblivious) server-side proxy/NAT
that works like this:

- The proxy service has one public IP address and like 65,000 available UDP
ports.
- The proxy service can itself be redundant with failover...
- When a user wants to open a new Mosh connection, they Mosh to a single
hostname (which resolves to the IP address of the proxy service).
- Your code allocates the necessary container, etc., and also allocates a
unique UDP port on the proxy.
- Your code runs the new mosh-server process in the target container.
- The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>"
message, replacing the port number with the unique public-facing UDP port
(and remembering the container's IP address and the original port number).
- When the proxy receives an incoming UDP datagram destined to a particular
UDP port, it forwards it to the appropriate container at its IP address and
at the original port number. It *preserves* the source IP and port of the
datagram when forwarding.
- When the container wants to send an outgoing UDP datagram, it sends it
normally (to whatever IP:port is associated with the client), except the
containers are not directly connected to the Internet; they use the
proxy/NAT as their next-hop router.
- For the outgoing UDP datagram, the proxy/NAT rewrites the container's
source IP:port to its own IP and the public port number.

I think this will allow you to serve like 65,000 separate mosh connections
from a single public IP address...

The added latency in forwarding a datagram is probably <1 ms, and you don't
really have to change anything about Mosh itself or its internals.

Unfortunately there are no unencrypted identifying marks to a Mosh
connection, except the incrementing sequence numbers (which start at 0 for
every connection).

-Keith

Post by Thomas Buckley-Houston
Hey Keith, John, everyone,
Yeah the more this is looking like a quite a big hurdle. Especially
your point Keith about roaming IPs (which I'd forgotten), it's a
central feature of Mosh I don't want to lose.
So the only 2 options seems to be exposing multiple IPs for Round
Robin (or other smart DNS routing) or writing a new Mosh proxy that
already has knowledge of the existing keys. Both seem like quite a
challenge. Round Robin DNS seems more approachable and I can imagine
integrating it with the Google Cloud DNS API I'm already using, but I
just wonder how expensive Google (or anyone for that matter) will make
thousands of static IP addresses? Apart from me having to learn Mosh
internals, one difficulty that strikes me about a Mosh proxy is that
it might introduce a non-trivial delay to each datagram arriving?
Though surely only ever in the order of a handful of milliseconds I
suppose.
Are there not any other identifying marks to a datagram, I don't know
much about low level networking, but maybe something like a MAC
address for example?
Thanks,
Tom

Post by Keith Winstein
Hi Thomas,
Glad you could provoke a very interesting discussion! But I'm still

confused

Post by Keith Winstein
-- how is "sticky IP-based routing" going to work after the client roams

to

Post by Keith Winstein
a new IP address (or to a new UDP source port)? When your system seems an
incoming UDP datagram from a previously unseen source IP:port, how does

it

Post by Keith Winstein
know which mosh-server (on which server machine) to send it to?
With off-the-shelf Mosh, you basically need a load-balancing strategy

that

Post by Keith Winstein
allows a destination IP:port to uniquely identify a particular

mosh-server.

Post by Keith Winstein
You can do this with multiple DNS A/AAAA records (where the client picks

the

Post by Keith Winstein
winning one -- maybe you permute the list), or with a smart DNS server

that

Post by Keith Winstein
serves *one* A or AAAA record to the client at the time of resolution

(like

Post by Keith Winstein
a CDN would use).
Instead of using the mosh wrapper script, you could have your users use

some

Post by Keith Winstein
other scheme to figure out the IP:port of the server, but the point is

that

Post by Keith Winstein
once you launch the mosh-client, it's going to keep sending datagrams to

the

Post by Keith Winstein
IP:port of the mosh-server, and those datagrams need to get to the same
mosh-server process even if the client roams to a different

publicly-visible

Post by Keith Winstein
IP address or port.
You could imagine writing a very smart mosh proxy that has the keys to

all

Post by Keith Winstein
the sessions and can figure out (for an incoming datagram coming from an
unknown source IP:port) which session it actually belongs to, and then

makes

Post by Keith Winstein
a sticky mapping and routes it to the proper mosh-server. But I don't

think

Post by Keith Winstein
anybody has actually done this yet and of course there's a challenge in
making this reliable/replicated.
-Keith

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.
And thanks so much for the heads up about my DNSSEC records. I've sent
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey warning.
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as I
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up.

The

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and

writes

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
them to stdout, where they go back over SSH to the mosh script; then

the

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
mosh script launches mosh-client passing the IP address, port number, and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication is
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it

doesn’t

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
make sense to “refuse more than 1 connection on the same port”, both
because UDP is connectionless, and because a new mosh-server is

launched

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses, or to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script resolve the
hostname only once and use the same address for the SSH connection and the
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu

pool.

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
If that’s not an option and you really need all connections to go through
a single load balancer address, you could try wrapping mosh-server in

a

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends

from

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers

like

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set
with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.
Anders

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Thomas Buckley-Houston]

Thanks so much for this idea, I really think it's the one, simple and scalable.

I haven't tried but I'm pretty sure the mosh-server's "MOSH CONNECT"
can be wrapped in plain BASH. I'm already in control of the SSH
connection as I'm using my own `ForceCommand` script.

Also I can still use this method with extra Round-Robin balanced IP
addresses giving me multiple sets of 65,000 ports.

The only thing I don't understand is why the outgoing UDP datagram has
to rewrite the container's source IP. Isn't the original MOSH CONNECT
IP:port the canonical reference?

Post by Keith Winstein
How about a semi-smart (but mostly Mosh-oblivious) server-side proxy/NAT
- The proxy service has one public IP address and like 65,000 available UDP
ports.
- The proxy service can itself be redundant with failover...
- When a user wants to open a new Mosh connection, they Mosh to a single
hostname (which resolves to the IP address of the proxy service).
- Your code allocates the necessary container, etc., and also allocates a
unique UDP port on the proxy.
- Your code runs the new mosh-server process in the target container.
- The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>"
message, replacing the port number with the unique public-facing UDP port
(and remembering the container's IP address and the original port number).
- When the proxy receives an incoming UDP datagram destined to a particular
UDP port, it forwards it to the appropriate container at its IP address and
at the original port number. It *preserves* the source IP and port of the
datagram when forwarding.
- When the container wants to send an outgoing UDP datagram, it sends it
normally (to whatever IP:port is associated with the client), except the
containers are not directly connected to the Internet; they use the
proxy/NAT as their next-hop router.
- For the outgoing UDP datagram, the proxy/NAT rewrites the container's
source IP:port to its own IP and the public port number.
I think this will allow you to serve like 65,000 separate mosh connections
from a single public IP address...
The added latency in forwarding a datagram is probably <1 ms, and you don't
really have to change anything about Mosh itself or its internals.
Unfortunately there are no unencrypted identifying marks to a Mosh
connection, except the incrementing sequence numbers (which start at 0 for
every connection).
-Keith

Post by Thomas Buckley-Houston
Hey Keith, John, everyone,
Yeah the more this is looking like a quite a big hurdle. Especially
your point Keith about roaming IPs (which I'd forgotten), it's a
central feature of Mosh I don't want to lose.
So the only 2 options seems to be exposing multiple IPs for Round
Robin (or other smart DNS routing) or writing a new Mosh proxy that
already has knowledge of the existing keys. Both seem like quite a
challenge. Round Robin DNS seems more approachable and I can imagine
integrating it with the Google Cloud DNS API I'm already using, but I
just wonder how expensive Google (or anyone for that matter) will make
thousands of static IP addresses? Apart from me having to learn Mosh
internals, one difficulty that strikes me about a Mosh proxy is that
it might introduce a non-trivial delay to each datagram arriving?
Though surely only ever in the order of a handful of milliseconds I
suppose.
Are there not any other identifying marks to a datagram, I don't know
much about low level networking, but maybe something like a MAC
address for example?
Thanks,
Tom

Post by Keith Winstein
Hi Thomas,
Glad you could provoke a very interesting discussion! But I'm still confused
-- how is "sticky IP-based routing" going to work after the client roams to
a new IP address (or to a new UDP source port)? When your system seems an
incoming UDP datagram from a previously unseen source IP:port, how does it
know which mosh-server (on which server machine) to send it to?
With off-the-shelf Mosh, you basically need a load-balancing strategy that
allows a destination IP:port to uniquely identify a particular mosh-server.
You can do this with multiple DNS A/AAAA records (where the client picks the
winning one -- maybe you permute the list), or with a smart DNS server that
serves *one* A or AAAA record to the client at the time of resolution (like
a CDN would use).
Instead of using the mosh wrapper script, you could have your users use some
other scheme to figure out the IP:port of the server, but the point is that
once you launch the mosh-client, it's going to keep sending datagrams to the
IP:port of the mosh-server, and those datagrams need to get to the same
mosh-server process even if the client roams to a different
publicly-visible
IP address or port.
You could imagine writing a very smart mosh proxy that has the keys to all
the sessions and can figure out (for an incoming datagram coming from an
unknown source IP:port) which session it actually belongs to, and then makes
a sticky mapping and routes it to the proper mosh-server. But I don't think
anybody has actually done this yet and of course there's a challenge in
making this reliable/replicated.
-Keith
On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.
And thanks so much for the heads up about my DNSSEC records. I've sent
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey warning.
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as I
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up.
The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and writes
them to stdout, where they go back over SSH to the mosh script; then the
mosh script launches mosh-client passing the IP address, port number, and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication is
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it doesn’t
make sense to “refuse more than 1 connection on the same port”, both
because UDP is connectionless, and because a new mosh-server is launched
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses,
or
to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script resolve the
hostname only once and use the same address for the SSH connection and
the
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool.
If that’s not an option and you really need all connections to go through
a single load balancer address, you could try wrapping mosh-server in a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends from
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set
with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh.
Anders

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Keith Winstein]

Hi Thomas,

Hmm, let me try to see if I can say it better. For an outgoing UDP datagram
(from a containerized mosh-server to a mosh-client), the mosh-server will
be sending the datagram from some private IP address (the container's IP
address, e.g. 10.0.0.5) and a source port that is probably going to be the
same across all containers (like 60001), since each container will probably
only be running one mosh-server.

The proxy will want to rewrite the source IP:port so that the source IP
address is the proxy's routable Internet address and the port number is
whatever unique port it assigned and sent to the client in the original
MOSH CONNECT message that the client saw.

-Keith

Post by Thomas Buckley-Houston
Thanks so much for this idea, I really think it's the one, simple and scalable.
I haven't tried but I'm pretty sure the mosh-server's "MOSH CONNECT"
can be wrapped in plain BASH. I'm already in control of the SSH
connection as I'm using my own `ForceCommand` script.
Also I can still use this method with extra Round-Robin balanced IP
addresses giving me multiple sets of 65,000 ports.
The only thing I don't understand is why the outgoing UDP datagram has
to rewrite the container's source IP. Isn't the original MOSH CONNECT
IP:port the canonical reference?

Post by Keith Winstein
How about a semi-smart (but mostly Mosh-oblivious) server-side proxy/NAT
- The proxy service has one public IP address and like 65,000 available

UDP

Post by Keith Winstein
ports.
- The proxy service can itself be redundant with failover...
- When a user wants to open a new Mosh connection, they Mosh to a single
hostname (which resolves to the IP address of the proxy service).
- Your code allocates the necessary container, etc., and also allocates a
unique UDP port on the proxy.
- Your code runs the new mosh-server process in the target container.
- The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>"
message, replacing the port number with the unique public-facing UDP port
(and remembering the container's IP address and the original port

number).

Post by Keith Winstein
- When the proxy receives an incoming UDP datagram destined to a

particular

Post by Keith Winstein
UDP port, it forwards it to the appropriate container at its IP address

and

Post by Keith Winstein
at the original port number. It *preserves* the source IP and port of the
datagram when forwarding.
- When the container wants to send an outgoing UDP datagram, it sends it
normally (to whatever IP:port is associated with the client), except the
containers are not directly connected to the Internet; they use the
proxy/NAT as their next-hop router.
- For the outgoing UDP datagram, the proxy/NAT rewrites the container's
source IP:port to its own IP and the public port number.
I think this will allow you to serve like 65,000 separate mosh

connections

Post by Keith Winstein
from a single public IP address...
The added latency in forwarding a datagram is probably <1 ms, and you

don't

Post by Keith Winstein
really have to change anything about Mosh itself or its internals.
Unfortunately there are no unencrypted identifying marks to a Mosh
connection, except the incrementing sequence numbers (which start at 0

for

Post by Keith Winstein
every connection).
-Keith
On Fri, Jun 29, 2018 at 12:17 AM, Thomas Buckley-Houston <

Post by Thomas Buckley-Houston
Hey Keith, John, everyone,
Yeah the more this is looking like a quite a big hurdle. Especially
your point Keith about roaming IPs (which I'd forgotten), it's a
central feature of Mosh I don't want to lose.
So the only 2 options seems to be exposing multiple IPs for Round
Robin (or other smart DNS routing) or writing a new Mosh proxy that
already has knowledge of the existing keys. Both seem like quite a
challenge. Round Robin DNS seems more approachable and I can imagine
integrating it with the Google Cloud DNS API I'm already using, but I
just wonder how expensive Google (or anyone for that matter) will make
thousands of static IP addresses? Apart from me having to learn Mosh
internals, one difficulty that strikes me about a Mosh proxy is that
it might introduce a non-trivial delay to each datagram arriving?
Though surely only ever in the order of a handful of milliseconds I
suppose.
Are there not any other identifying marks to a datagram, I don't know
much about low level networking, but maybe something like a MAC
address for example?
Thanks,
Tom

Post by Keith Winstein
Hi Thomas,
Glad you could provoke a very interesting discussion! But I'm still confused
-- how is "sticky IP-based routing" going to work after the client

roams

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
to
a new IP address (or to a new UDP source port)? When your system seems an
incoming UDP datagram from a previously unseen source IP:port, how

does

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
it
know which mosh-server (on which server machine) to send it to?
With off-the-shelf Mosh, you basically need a load-balancing strategy that
allows a destination IP:port to uniquely identify a particular mosh-server.
You can do this with multiple DNS A/AAAA records (where the client

picks

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
the
winning one -- maybe you permute the list), or with a smart DNS server that
serves *one* A or AAAA record to the client at the time of resolution (like
a CDN would use).
Instead of using the mosh wrapper script, you could have your users

use

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
some
other scheme to figure out the IP:port of the server, but the point is that
once you launch the mosh-client, it's going to keep sending datagrams

to

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
the
IP:port of the mosh-server, and those datagrams need to get to the

same

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
mosh-server process even if the client roams to a different publicly-visible
IP address or port.
You could imagine writing a very smart mosh proxy that has the keys to all
the sessions and can figure out (for an incoming datagram coming from

an

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
unknown source IP:port) which session it actually belongs to, and then makes
a sticky mapping and routes it to the proper mosh-server. But I don't think
anybody has actually done this yet and of course there's a challenge

in

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
making this reliable/replicated.
-Keith
On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I

hope

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston
one day to be able to scale to thousands of servers.
And thanks so much for the heads up about my DNSSEC records. I've

sent

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey warning.
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as I
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up.
The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and writes
them to stdout, where they go back over SSH to the mosh script;

then

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
the
mosh script launches mosh-client passing the IP address, port

number,

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication

is

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it
doesn’t
make sense to “refuse more than 1 connection on the same port”,

both

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
because UDP is connectionless, and because a new mosh-server is launched
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses,
or
to
different addresses for different clients and/or at different

times.

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
We’ve already gone out of our way to make the mosh script resolve

the

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
hostname only once and use the same address for the SSH connection and
the
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu
pool.
If that’s not an option and you really need all connections to go
through
a single load balancer address, you could try wrapping mosh-server

in

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends from
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers
like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS

records

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
set
with the registrar: https://dnssec-debugger.

verisignlabs.com/brow.sh.

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
Anders

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Thomas Buckley-Houston]

Forgive me for not fully learning about UDP, I've Googled a little
bit, but I'm sure I still have a lot of gaps in my knowledge. So
because UDP is a connectionless protocol, every datagram has to
contain the source IP in order for the receiver to send responses?
It's not enough for either end to assume an IP based on the *initial*
datagram's IP? So based on that assumption, the best way for a
datagram to get its IP:port is to query the OS. Which in our scenario
is a privately networked container and so will be unreachable from the
outside.

From a brief research I couldn't find any docs on this, I'm hoping its
possible with Nginx. But maybe it can only be done with something like
iptables?

On a separate note I re-launched Texttop as Browsh this week to a big
response: https://www.brow.sh The SSH servers (`ssh brow.sh`) held up
really well, they've had thousands of sessions already without issue.
So would be great to get Mosh working as well.

So just another friendly reminder - I'm having to tell people to
compile Mosh themselves to get true colour support.

Post by Keith Winstein
Hi Thomas,
Hmm, let me try to see if I can say it better. For an outgoing UDP datagram
(from a containerized mosh-server to a mosh-client), the mosh-server will be
sending the datagram from some private IP address (the container's IP
address, e.g. 10.0.0.5) and a source port that is probably going to be the
same across all containers (like 60001), since each container will probably
only be running one mosh-server.
The proxy will want to rewrite the source IP:port so that the source IP
address is the proxy's routable Internet address and the port number is
whatever unique port it assigned and sent to the client in the original MOSH
CONNECT message that the client saw.
-Keith

Post by Thomas Buckley-Houston
Thanks so much for this idea, I really think it's the one, simple and scalable.
I haven't tried but I'm pretty sure the mosh-server's "MOSH CONNECT"
can be wrapped in plain BASH. I'm already in control of the SSH
connection as I'm using my own `ForceCommand` script.
Also I can still use this method with extra Round-Robin balanced IP
addresses giving me multiple sets of 65,000 ports.
The only thing I don't understand is why the outgoing UDP datagram has
to rewrite the container's source IP. Isn't the original MOSH CONNECT
IP:port the canonical reference?

Post by Keith Winstein
How about a semi-smart (but mostly Mosh-oblivious) server-side proxy/NAT
- The proxy service has one public IP address and like 65,000 available UDP
ports.
- The proxy service can itself be redundant with failover...
- When a user wants to open a new Mosh connection, they Mosh to a single
hostname (which resolves to the IP address of the proxy service).
- Your code allocates the necessary container, etc., and also allocates a
unique UDP port on the proxy.
- Your code runs the new mosh-server process in the target container.
- The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>"
message, replacing the port number with the unique public-facing UDP port
(and remembering the container's IP address and the original port number).
- When the proxy receives an incoming UDP datagram destined to a particular
UDP port, it forwards it to the appropriate container at its IP address and
at the original port number. It *preserves* the source IP and port of the
datagram when forwarding.
- When the container wants to send an outgoing UDP datagram, it sends it
normally (to whatever IP:port is associated with the client), except the
containers are not directly connected to the Internet; they use the
proxy/NAT as their next-hop router.
- For the outgoing UDP datagram, the proxy/NAT rewrites the container's
source IP:port to its own IP and the public port number.
I think this will allow you to serve like 65,000 separate mosh connections
from a single public IP address...
The added latency in forwarding a datagram is probably <1 ms, and you don't
really have to change anything about Mosh itself or its internals.
Unfortunately there are no unencrypted identifying marks to a Mosh
connection, except the incrementing sequence numbers (which start at 0 for
every connection).
-Keith
On Fri, Jun 29, 2018 at 12:17 AM, Thomas Buckley-Houston

Post by Thomas Buckley-Houston
Hey Keith, John, everyone,
Yeah the more this is looking like a quite a big hurdle. Especially
your point Keith about roaming IPs (which I'd forgotten), it's a
central feature of Mosh I don't want to lose.
So the only 2 options seems to be exposing multiple IPs for Round
Robin (or other smart DNS routing) or writing a new Mosh proxy that
already has knowledge of the existing keys. Both seem like quite a
challenge. Round Robin DNS seems more approachable and I can imagine
integrating it with the Google Cloud DNS API I'm already using, but I
just wonder how expensive Google (or anyone for that matter) will make
thousands of static IP addresses? Apart from me having to learn Mosh
internals, one difficulty that strikes me about a Mosh proxy is that
it might introduce a non-trivial delay to each datagram arriving?
Though surely only ever in the order of a handful of milliseconds I
suppose.
Are there not any other identifying marks to a datagram, I don't know
much about low level networking, but maybe something like a MAC
address for example?
Thanks,
Tom

Post by Keith Winstein
Hi Thomas,
Glad you could provoke a very interesting discussion! But I'm still confused
-- how is "sticky IP-based routing" going to work after the client roams
to
a new IP address (or to a new UDP source port)? When your system
seems
an
incoming UDP datagram from a previously unseen source IP:port, how does
it
know which mosh-server (on which server machine) to send it to?
With off-the-shelf Mosh, you basically need a load-balancing strategy that
allows a destination IP:port to uniquely identify a particular mosh-server.
You can do this with multiple DNS A/AAAA records (where the client picks
the
winning one -- maybe you permute the list), or with a smart DNS server
that
serves *one* A or AAAA record to the client at the time of resolution (like
a CDN would use).
Instead of using the mosh wrapper script, you could have your users use
some
other scheme to figure out the IP:port of the server, but the point is
that
once you launch the mosh-client, it's going to keep sending datagrams to
the
IP:port of the mosh-server, and those datagrams need to get to the same
mosh-server process even if the client roams to a different publicly-visible
IP address or port.
You could imagine writing a very smart mosh proxy that has the keys to
all
the sessions and can figure out (for an incoming datagram coming from an
unknown source IP:port) which session it actually belongs to, and then
makes
a sticky mapping and routes it to the proper mosh-server. But I don't think
anybody has actually done this yet and of course there's a challenge in
making this reliable/replicated.
-Keith
On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky IP-based
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.
And thanks so much for the heads up about my DNSSEC records. I've sent
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey warning.
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as I
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up.
The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and writes
them to stdout, where they go back over SSH to the mosh script; then
the
mosh script launches mosh-client passing the IP address, port
number,
and
encryption key. The newly launched mosh-client and mosh-server processes
exchange UDP packets encrypted with the shared key; communication is
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it
doesn’t
make sense to “refuse more than 1 connection on the same port”, both
because UDP is connectionless, and because a new mosh-server is launched
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is with
round-robin DNS where a single hostname resolves to many addresses,
or
to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script resolve the
hostname only once and use the same address for the SSH connection and
the
UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu
pool.
If that’s not an option and you really need all connections to go
through
a single load balancer address, you could try wrapping mosh-server in
a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends
from
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS
records
set
https://dnssec-debugger.verisignlabs.com/brow.sh.
Anders

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Jim Cheetham]

Post by Thomas Buckley-Houston
So
because UDP is a connectionless protocol, every datagram has to
contain the source IP in order for the receiver to send responses?

Every packet does have the source IP address in it, but not because it's a UDP packet, it's because that's what all *IP* packets have. UDP is a protocol built on top of IP, as is TCP - we're used to the "full name" of TCP being TCP/IP, but it's unusual to say "UDP/IP" :-)

Post by Thomas Buckley-Houston
It's not enough for either end to assume an IP based on the *initial*
datagram's IP?

Because UDP is connectionless, there's no concept of an initial packet in the first place; all packets that arrive are completely self-contained entities with no relation to any other packets that might arrive before or after. It's a bit like HTTP in that respect.

The *application* that's using UDP may have a concept of a long-term session that involved multiple packets, but UDP itself doesn't. Again looking at HTTP, that's like using cookies to describe a session.

Post by Thomas Buckley-Houston
So based on that assumption, the best way for a
datagram to get its IP:port is to query the OS. Which in our scenario
is a privately networked container and so will be unreachable from the
outside.

You have a middle layer load balancer, which will be keeping track of the state of sessions going through; this is easy for TCP because the session identifiers are basically in every packet. However, in UDP they are not present at all - you have to be aware of the actual application using UDP; you have to be explicitly aware of mosh itself in order to be able to load-balance/proxy it correctly.

A simple assumption for most protocols that want to have cheap lightweight sessions is to assume that the source IP address stays the same throughout. This happens to have been true for most of the network's history, but has never been guaranteed.

In the case of mosh, this is explicitly not true; the protocol *expects* that the source IP address will change unannounced throughout the application's session.

It is this issue that you need to be looking in to. You need a proxy/LB layer that looks at a UDP packet coming through, recognises that it is using SSP (State Synchronization Protocol), and keeps track of which backend system the packet should be forwarded to.

Unfortunately, this will be very difficult; datagrams are encrypted, and your proxy will not be aware of the key in use. The only way I can see to make this happen will be to alter the mosh server to inform the proxy directly, and that's a lot of engineering that will introduce a number of security tradeoffs that might not be worthwhile.

Overall, your better approach is to have a traditional "bastion host" in the middle, use mosh to connect to that, and from there make inbound ssh connections to your service hosts (because you can get away with ssh on an internal "reliable" network, and because you're using TCP for this, you can go through a load balancer if required).

-jim

--
Jim Cheetham, Information Security, University of Otago, Dunedin, N.Z.
✉ ***@otago.ac.nz ☏ [phone] ☏ m [phone]
⚷ OpenPGP: B50F BE3B D49B 3A8A 9CC3 8966 9374 82CD C982 0605

[Thomas Buckley-Houston]

Thanks Jim,

That really clears things up for me.

So but you're saying that Keith's idea of having the outgoing
datagram's source IP remapped is not possible?

I totally agree about the traditional approach of the "bastion host",
it's basically like having SSH act as the "load balancer" - as I'll
have some shell script open up a connection to a random server on the
bastion's CLI. The only problem there is that it doesn't take
advantage of traditional cluster architecture so well. For instance
I'll need to come up with a custom method of keeping a live list of
the available internal IP addresses to which I can forward the
session. Any idea how many connections an average 512MB VM could
handle like this? I suspect so many that you'd only ever need one VM?

Tom

Post by Jim Cheetham

Post by Thomas Buckley-Houston
So
because UDP is a connectionless protocol, every datagram has to
contain the source IP in order for the receiver to send responses?

Every packet does have the source IP address in it, but not because it's a UDP packet, it's because that's what all *IP* packets have. UDP is a protocol built on top of IP, as is TCP - we're used to the "full name" of TCP being TCP/IP, but it's unusual to say "UDP/IP" :-)

Post by Thomas Buckley-Houston
It's not enough for either end to assume an IP based on the *initial*
datagram's IP?

Because UDP is connectionless, there's no concept of an initial packet in the first place; all packets that arrive are completely self-contained entities with no relation to any other packets that might arrive before or after. It's a bit like HTTP in that respect.
The *application* that's using UDP may have a concept of a long-term session that involved multiple packets, but UDP itself doesn't. Again looking at HTTP, that's like using cookies to describe a session.

Post by Thomas Buckley-Houston
So based on that assumption, the best way for a
datagram to get its IP:port is to query the OS. Which in our scenario
is a privately networked container and so will be unreachable from the
outside.

You have a middle layer load balancer, which will be keeping track of the state of sessions going through; this is easy for TCP because the session identifiers are basically in every packet. However, in UDP they are not present at all - you have to be aware of the actual application using UDP; you have to be explicitly aware of mosh itself in order to be able to load-balance/proxy it correctly.
A simple assumption for most protocols that want to have cheap lightweight sessions is to assume that the source IP address stays the same throughout. This happens to have been true for most of the network's history, but has never been guaranteed.
In the case of mosh, this is explicitly not true; the protocol *expects* that the source IP address will change unannounced throughout the application's session.
It is this issue that you need to be looking in to. You need a proxy/LB layer that looks at a UDP packet coming through, recognises that it is using SSP (State Synchronization Protocol), and keeps track of which backend system the packet should be forwarded to.
Unfortunately, this will be very difficult; datagrams are encrypted, and your proxy will not be aware of the key in use. The only way I can see to make this happen will be to alter the mosh server to inform the proxy directly, and that's a lot of engineering that will introduce a number of security tradeoffs that might not be worthwhile.
Overall, your better approach is to have a traditional "bastion host" in the middle, use mosh to connect to that, and from there make inbound ssh connections to your service hosts (because you can get away with ssh on an internal "reliable" network, and because you're using TCP for this, you can go through a load balancer if required).
-jim
--
Jim Cheetham, Information Security, University of Otago, Dunedin, N.Z.
⚷ OpenPGP: B50F BE3B D49B 3A8A 9CC3 8966 9374 82CD C982 0605

[Keith Winstein]

I don't think Jim's message was quite a response to my earlier one -- if
you do use a proxy/NAT as I described, you do not need to use a bastion
host and you don't need the proxy to be aware of the internals of the SSP
protocol, or worry about keys, or decrypt anything. It can be pretty
oblivious to the inner workings of Mosh. It does have to rewrite the IP
addresses and UDP port numbers in each direction and it has to alter the
"MOSH CONNECT" message at the start.

-Keith

Post by Thomas Buckley-Houston
Thanks Jim,
That really clears things up for me.
So but you're saying that Keith's idea of having the outgoing
datagram's source IP remapped is not possible?
I totally agree about the traditional approach of the "bastion host",
it's basically like having SSH act as the "load balancer" - as I'll
have some shell script open up a connection to a random server on the
bastion's CLI. The only problem there is that it doesn't take
advantage of traditional cluster architecture so well. For instance
I'll need to come up with a custom method of keeping a live list of
the available internal IP addresses to which I can forward the
session. Any idea how many connections an average 512MB VM could
handle like this? I suspect so many that you'd only ever need one VM?
Tom

Post by Jim Cheetham

Post by Thomas Buckley-Houston
So
because UDP is a connectionless protocol, every datagram has to
contain the source IP in order for the receiver to send responses?

Every packet does have the source IP address in it, but not because it's

a UDP packet, it's because that's what all *IP* packets have. UDP is a
protocol built on top of IP, as is TCP - we're used to the "full name" of
TCP being TCP/IP, but it's unusual to say "UDP/IP" :-)

Post by Jim Cheetham

Post by Thomas Buckley-Houston
It's not enough for either end to assume an IP based on the *initial*
datagram's IP?

Because UDP is connectionless, there's no concept of an initial packet

in the first place; all packets that arrive are completely self-contained
entities with no relation to any other packets that might arrive before or
after. It's a bit like HTTP in that respect.

Post by Jim Cheetham
The *application* that's using UDP may have a concept of a long-term

session that involved multiple packets, but UDP itself doesn't. Again
looking at HTTP, that's like using cookies to describe a session.

Post by Jim Cheetham

Post by Thomas Buckley-Houston
So based on that assumption, the best way for a
datagram to get its IP:port is to query the OS. Which in our scenario
is a privately networked container and so will be unreachable from the
outside.

You have a middle layer load balancer, which will be keeping track of

the state of sessions going through; this is easy for TCP because the
session identifiers are basically in every packet. However, in UDP they are
not present at all - you have to be aware of the actual application using
UDP; you have to be explicitly aware of mosh itself in order to be able to
load-balance/proxy it correctly.

Post by Jim Cheetham
A simple assumption for most protocols that want to have cheap

lightweight sessions is to assume that the source IP address stays the same
throughout. This happens to have been true for most of the network's
history, but has never been guaranteed.

Post by Jim Cheetham
In the case of mosh, this is explicitly not true; the protocol *expects*

that the source IP address will change unannounced throughout the
application's session.

Post by Jim Cheetham
It is this issue that you need to be looking in to. You need a proxy/LB

layer that looks at a UDP packet coming through, recognises that it is
using SSP (State Synchronization Protocol), and keeps track of which
backend system the packet should be forwarded to.

Post by Jim Cheetham
Unfortunately, this will be very difficult; datagrams are encrypted, and

your proxy will not be aware of the key in use. The only way I can see to
make this happen will be to alter the mosh server to inform the proxy
directly, and that's a lot of engineering that will introduce a number of
security tradeoffs that might not be worthwhile.

Post by Jim Cheetham
Overall, your better approach is to have a traditional "bastion host" in

the middle, use mosh to connect to that, and from there make inbound ssh
connections to your service hosts (because you can get away with ssh on an
internal "reliable" network, and because you're using TCP for this, you can
go through a load balancer if required).

Post by Jim Cheetham
-jim
--
Jim Cheetham, Information Security, University of Otago, Dunedin, N.Z.
⚷ OpenPGP: B50F BE3B D49B 3A8A 9CC3 8966 9374 82CD C982 0605

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Keith Winstein]

Hello Thomas,

Post by Thomas Buckley-Houston
Forgive me for not fully learning about UDP, I've Googled a little
bit, but I'm sure I still have a lot of gaps in my knowledge. So
because UDP is a connectionless protocol, every datagram has to
contain the source IP in order for the receiver to send responses?

IP is a connectionless protocol, and every IP datagram contains a source
and destination IP address in the IP header.

Both UDP and TCP put their data in IP datagrams -- so every TCP-in-IP
datagram and every UDP-in-IP datagram contains a source and destination IP
address.

TCP and Mosh both add connections on top of IP. (UDP is basically just a
thin layer on top of IP that adds a port number so that individual programs
can use it.)

It's not enough for either end to assume an IP based on the *initial*

Post by Thomas Buckley-Houston
datagram's IP? So based on that assumption, the best way for a

datagram to get its IP:port is to query the OS.


Not quite sure what you mean here. When you write a program to receive a
UDP datagram, that program can learn the source IP and port of the datagram
by calling recvfrom() or recvmsg().

Post by Thomas Buckley-Houston
Which in our scenario
is a privately networked container and so will be unreachable from the
outside.

For the datagrams flowing from the mosh-server to the mosh-client, yeah,
the mosh-server is going to send using a private source address. It will be
the job of the proxy server to change that source address (and port number)
into a publicly reachable one.

Post by Thomas Buckley-Houston
From a brief research I couldn't find any docs on this, I'm hoping its
possible with Nginx. But maybe it can only be done with something like
iptables?

I think you're going to have to write your own program to do this (using
the logic I described on June 30) -- I don't think iptables is going to be
able to do exactly what I described out of the box. But it doesn't have to
know anything about the SSP protocol and it doesn't have to decrypt
anything. It does have to rewrite the "private" IP addresses and port
numbers into public ones as I described.

Cheers,
Keith

On a separate note I re-launched Texttop as Browsh this week to a big

Post by Thomas Buckley-Houston
response: https://www.brow.sh The SSH servers (`ssh brow.sh`) held up
really well, they've had thousands of sessions already without issue.
So would be great to get Mosh working as well.
So just another friendly reminder - I'm having to tell people to
compile Mosh themselves to get true colour support.

Post by Keith Winstein
Hi Thomas,
Hmm, let me try to see if I can say it better. For an outgoing UDP

datagram

Post by Keith Winstein
(from a containerized mosh-server to a mosh-client), the mosh-server

will be

Post by Keith Winstein
sending the datagram from some private IP address (the container's IP
address, e.g. 10.0.0.5) and a source port that is probably going to be

the

Post by Keith Winstein
same across all containers (like 60001), since each container will

probably

Post by Keith Winstein
only be running one mosh-server.
The proxy will want to rewrite the source IP:port so that the source IP
address is the proxy's routable Internet address and the port number is
whatever unique port it assigned and sent to the client in the original

MOSH

Post by Keith Winstein
CONNECT message that the client saw.
-Keith

Post by Thomas Buckley-Houston
Thanks so much for this idea, I really think it's the one, simple and scalable.
I haven't tried but I'm pretty sure the mosh-server's "MOSH CONNECT"
can be wrapped in plain BASH. I'm already in control of the SSH
connection as I'm using my own `ForceCommand` script.
Also I can still use this method with extra Round-Robin balanced IP
addresses giving me multiple sets of 65,000 ports.
The only thing I don't understand is why the outgoing UDP datagram has
to rewrite the container's source IP. Isn't the original MOSH CONNECT
IP:port the canonical reference?

Post by Keith Winstein
How about a semi-smart (but mostly Mosh-oblivious) server-side

proxy/NAT

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
- The proxy service has one public IP address and like 65,000

available

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
UDP
ports.
- The proxy service can itself be redundant with failover...
- When a user wants to open a new Mosh connection, they Mosh to a

single

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
hostname (which resolves to the IP address of the proxy service).
- Your code allocates the necessary container, etc., and also

allocates

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
a
unique UDP port on the proxy.
- Your code runs the new mosh-server process in the target container.
- The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>"
message, replacing the port number with the unique public-facing UDP port
(and remembering the container's IP address and the original port number).
- When the proxy receives an incoming UDP datagram destined to a particular
UDP port, it forwards it to the appropriate container at its IP

address

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
and
at the original port number. It *preserves* the source IP and port of the
datagram when forwarding.
- When the container wants to send an outgoing UDP datagram, it sends

it

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
normally (to whatever IP:port is associated with the client), except

the

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
containers are not directly connected to the Internet; they use the
proxy/NAT as their next-hop router.
- For the outgoing UDP datagram, the proxy/NAT rewrites the

container's

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
source IP:port to its own IP and the public port number.
I think this will allow you to serve like 65,000 separate mosh connections
from a single public IP address...
The added latency in forwarding a datagram is probably <1 ms, and you don't
really have to change anything about Mosh itself or its internals.
Unfortunately there are no unencrypted identifying marks to a Mosh
connection, except the incrementing sequence numbers (which start at 0 for
every connection).
-Keith
On Fri, Jun 29, 2018 at 12:17 AM, Thomas Buckley-Houston

Post by Thomas Buckley-Houston
Hey Keith, John, everyone,
Yeah the more this is looking like a quite a big hurdle. Especially
your point Keith about roaming IPs (which I'd forgotten), it's a
central feature of Mosh I don't want to lose.
So the only 2 options seems to be exposing multiple IPs for Round
Robin (or other smart DNS routing) or writing a new Mosh proxy that
already has knowledge of the existing keys. Both seem like quite a
challenge. Round Robin DNS seems more approachable and I can imagine
integrating it with the Google Cloud DNS API I'm already using, but I
just wonder how expensive Google (or anyone for that matter) will

make

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston
thousands of static IP addresses? Apart from me having to learn Mosh
internals, one difficulty that strikes me about a Mosh proxy is that
it might introduce a non-trivial delay to each datagram arriving?
Though surely only ever in the order of a handful of milliseconds I
suppose.
Are there not any other identifying marks to a datagram, I don't know
much about low level networking, but maybe something like a MAC
address for example?
Thanks,
Tom

Post by Keith Winstein
Hi Thomas,
Glad you could provoke a very interesting discussion! But I'm still
confused
-- how is "sticky IP-based routing" going to work after the client roams
to
a new IP address (or to a new UDP source port)? When your system
seems
an
incoming UDP datagram from a previously unseen source IP:port, how does
it
know which mosh-server (on which server machine) to send it to?
With off-the-shelf Mosh, you basically need a load-balancing

strategy

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
that
allows a destination IP:port to uniquely identify a particular mosh-server.
You can do this with multiple DNS A/AAAA records (where the client picks
the
winning one -- maybe you permute the list), or with a smart DNS server
that
serves *one* A or AAAA record to the client at the time of

resolution

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
(like
a CDN would use).
Instead of using the mosh wrapper script, you could have your users use
some
other scheme to figure out the IP:port of the server, but the point is
that
once you launch the mosh-client, it's going to keep sending

datagrams

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
to
the
IP:port of the mosh-server, and those datagrams need to get to the same
mosh-server process even if the client roams to a different
publicly-visible
IP address or port.
You could imagine writing a very smart mosh proxy that has the keys to
all
the sessions and can figure out (for an incoming datagram coming

from

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
an
unknown source IP:port) which session it actually belongs to, and then
makes
a sticky mapping and routes it to the proper mosh-server. But I

don't

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
think
anybody has actually done this yet and of course there's a

challenge

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein
in
making this reliable/replicated.
-Keith

Post by Thomas Buckley-Houston
Thanks so much for the clarification.

Post by Anders Kaseorg
UDP is connectionless

That's the key here. So I have no choice but to use sticky

IP-based

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston
routing. Round-robin DNS isn't an option I don't think, because I hope
one day to be able to scale to thousands of servers.
And thanks so much for the heads up about my DNSSEC records. I've sent
a request for them to be deleted. I'd added them and some SSHFP
records to explore automatically passing the StrictHostKey

warning.

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston
But it's not entirely straight forward. Even with correct DNS records
the SSH user still has to have VerifyHostKeyDNS enabled, which as

I

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston
understand most people don't. And then on top of that my DNS provider
(DNSSimple) automatically rotate the keys every 3 months, which means
I have to manually send a request to my registrars by email to update
the DNSSEC records. Is it all worth it do you think?

Post by Anders Kaseorg
You may have a misunderstanding about how a Mosh session is set up.
The
mosh script launches a mosh-server on the remote system via SSH;
mosh-server picks a port number and a random encryption key, and
writes
them to stdout, where they go back over SSH to the mosh script; then
the
mosh script launches mosh-client passing the IP address, port
number,
and
encryption key. The newly launched mosh-client and mosh-server
processes
exchange UDP packets encrypted with the shared key;

communication

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
is
successful if the packets can be decrypted.
There’s no separate “key checking” step to be disabled. And it
doesn’t
make sense to “refuse more than 1 connection on the same port”,
both
because UDP is connectionless, and because a new mosh-server is
launched
on a new port for each Mosh session (it is not a daemon like sshd).
The easiest way to put Mosh servers behind a load balancer is

with

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
round-robin DNS where a single hostname resolves to many addresses,
or
to
different addresses for different clients and/or at different times.
We’ve already gone out of our way to make the mosh script

resolve

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
the
hostname only once and use the same address for the SSH

connection

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
and
the
UDP packets, because that’s needed for MIT’s

athena.dialup.mit.edu

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
pool.
If that’s not an option and you really need all connections to

go

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
through
a single load balancer address, you could try wrapping

mosh-server

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
in
a
script that passes different disjoint port ranges (-p) on different
backends, and forwarding those ranges to the corresponding backends
from
the load balancer.
Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled

resolvers

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Keith Winstein

Post by Thomas Buckley-Houston

Post by Anders Kaseorg
like
1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS
records
set
https://dnssec-debugger.verisignlabs.com/brow.sh.
Anders

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel