[mosh-devel] Mosh OS X package build on Travis

[john hood]

I've been working on making Travis do the OS X package build for us.
(Previously, some poor Mosh developer had to do it manually and upload
it to the Mosh website.) The thing I have it doing now is building the
package when a tag is pushed, and uploading it to a GitHub release on
cgull/mosh.

The reasons I did this: to get a more externally-visible build for the
package, and to do the release build on something other than my personal
OS X machines. Of course, this means we are trading trust in my OS X
systems for trust in Travis' OS X build environments. I think this is a
win, any opinions?

Alas, we will not get any kind of repeatable builds out of this, Travis
constantly updates their build images and we update to current Homebrew
for dependencies on every build.

Travis will do the build for us, but it will not hold onto build
artifacts for us. So we have to immediately deploy the package
to...someplace else. Currently, I'm using the GitHub deployment
provider. This requires an authentication token with permissions to
create and upload onto a GitHub release on the originating GitHub
project. Do people think this is OK security wise? There are plenty of
other deployment providers, and it's also possible to script an upload
to a random server.

regards,

--jh

[Jim Cheetham]

Quoting john hood (2016-10-31 17:12:06)

Post by john hood
Alas, we will not get any kind of repeatable builds out of this, Travis
constantly updates their build images and we update to current Homebrew
for dependencies on every build.

That's the worst bit. Using external services that are *unlikely* to attack
your process is generally just fine, as long as there is a way to verify their
output.

Perhaps you could use Travis to report in the buildability of a revision,
and the source of a 'nightly build' version, but keep a repeatable
build chain for official releases?

Post by john hood
Currently, I'm using the GitHub deployment
provider. This requires an authentication token with permissions to
create and upload onto a GitHub release on the originating GitHub
project. Do people think this is OK security wise?

Sure, I guess so - because we can trust that GitHub reports changes
adequately. And if an attacker could affect both Travis and GitHub at
the same time, we're not going to be able to prevent that. Detecting it
afterwards is a laudable goal :-)

--
Jim Cheetham, Information Security, University of Otago, Dunedin, N.Z.
✉ ***@otago.ac.nz ☏ [phone] ☏ m [phone]
⚷ OpenPGP: B50F BE3B D49B 3A8A 9CC3 8966 9374 82CD C982 0605

[john hood]

Post by Jim Cheetham
Quoting john hood (2016-10-31 17:12:06)

Post by john hood
Alas, we will not get any kind of repeatable builds out of this, Travis
constantly updates their build images and we update to current Homebrew
for dependencies on every build.

That's the worst bit. Using external services that are *unlikely* to attack
your process is generally just fine, as long as there is a way to verify their
output.

The repeatability is of course no worse than what we've got now :)

Post by Jim Cheetham
Perhaps you could use Travis to report in the buildability of a revision,
and the source of a 'nightly build' version, but keep a repeatable
build chain for official releases?

We already use Travis for CI builds on Linux and OS X. I could of
course maintain a build VM for Mosh releases and snapshot it for each
release build, but having to maintain it (OS X upgrades, Xcode upgrades)
for Mosh's fairly infrequent releases is a significant burden for a
small project, and it would bring us full circle to manually-maintained
private builds.

We can at least report the Travis image id (I think this is already in
the build log) and the Homebrew version + Git revisions, and other tool
versions. Homebrew has a way to dump its configuration ('homebrew info
--json=v1') which may or may not be complete.

The elephant in the room is that we're trying to improve builds for a
binary-only, proprietary OS with relatively ad-hoc installation,
configuration, packaging, etc.

regards,

--jh

[john hood]

Post by john hood

Post by Jim Cheetham
Quoting john hood (2016-10-31 17:12:06)

Post by john hood
Alas, we will not get any kind of repeatable builds out of this, Travis
constantly updates their build images and we update to current Homebrew
for dependencies on every build.

That's the worst bit. Using external services that are *unlikely* to attack
your process is generally just fine, as long as there is a way to verify their
output.

The repeatability is of course no worse than what we've got now :)

Post by Jim Cheetham
Perhaps you could use Travis to report in the buildability of a revision,
and the source of a 'nightly build' version, but keep a repeatable
build chain for official releases?

We already use Travis for CI builds on Linux and OS X. I could of
course maintain a build VM for Mosh releases and snapshot it for each
release build, but having to maintain it (OS X upgrades, Xcode upgrades)
for Mosh's fairly infrequent releases is a significant burden for a
small project, and it would bring us full circle to manually-maintained
private builds.
We can at least report the Travis image id (I think this is already in
the build log) and the Homebrew version + Git revisions, and other tool
versions. Homebrew has a way to dump its configuration ('homebrew info
--json=v1') which may or may not be complete.
The elephant in the room is that we're trying to improve builds for a
binary-only, proprietary OS with relatively ad-hoc installation,
configuration, packaging, etc.

I've had some more thinking on this, and I'm getting over the
you-piled-what?-onto-my-tasklist reaction.

There's a number of separate pieces to this problem:

* Fully automating the OS X build and installing its build deps. This
is a pretty basic first step to all the rest of it, and what I am trying
to do with Travis.

* Reporting: describing exactly what was built, how it was built, what
dependencies were used, other environmental factors.

* Repeatability: getting the same thing twice in a row :)

* Reproducibility: making it possible, maybe even easy, for you to get
the same thing out of a build that I did.

* Trusted build infrastructure: in theory a reproducible build reduces
the need for this. In practice...yeah, we should do that.

A lot of this stuff amortizes nicely if you are a distribution or
package management system. We're trying to do just one package, only
for OS X (and not our many other platforms), and not even for all OS X
users.

My current OS X package manager of choice is Homebrew. It seems they've
done some work towards reproducible builds internally, but it's not
clear at all how you might reproducibly install Homebrew, or do
non-Homebrew builds reproducibly with Homebrew deps. I just put a query
out on that.

It would be nice if Homebrew or Macports or some other package system
could build standalone apps, and had reproducible builds.

Another alternative might be to abandon package managers entirely for
release builds, and build our dependencies ourselves. Xcode might have
some useful capabilities in this area, I haven't looked. This may
require separate, clean OS X instances for package builds.

This isn't going to happen instantly. One approach to the trust issue
here might be to just cop out-- stop doing OS X package builds and tell
people to build their own, until we get this stuff into better shape.

Have I missed anything obvious? (Yes.) Have any of you got wisdom on
dealing with this stuff in an OS X environment that you can share?
(Please!)

regards,

--jh

[Jason Stelzer]

Hi there.

Pardon the de-lurk, but I have a few thoughts on the topic that might help.
I use mosh on os x and linux. I also happen to use homebrew.

As you've already pointed out, a proper package manager makes a lot of this
much easier. There are a couple ways to get closer to this on a mac.

First, a stupid linux trick. In the past, on debian based distros when I
have needed to replicate an environment that was lacking in automation for
builds, I've fallen back to
dpkg --get-selections > packages.txt
scp packages.txt host:/tmp
dpkg --set-selections < /tmp/packages.txt
apt-get dselect-upgrade

Ok, so at that point I can be confident that system B is similar enough to
system A to start working on whatever the issue is. The only problems that
have cropped up are where deps are so old they've rsynced off the public
mirrors, or in unversioned crap in /usr/local. You can do it with rpm too,
it's just spelled differently.

A way to replicate that technique on a mac could be as simple as:
Track the os version 10.x
Track the xcode version.
Track the homebrew version.

Homebrew has some options here.
brew list > packages.txt
scp packages.txt hostb:packages.txt
brew install $(cat packages.txt)

Where this kind of falls on its face, is that homebrew more or less just
chases the tip of a branch (stable) and updates are rolling.

But, it's all in git. If you were to go to /usr/local/Homebrew (the new
home), you could figure out what sha you were using for the current build
by running
✔ /usr/local/Homebrew [stable L|✔]
13:24 $ git rev-parse HEAD
8a7317aa8ff5f12067eb65e529a13490bc69deda (example sha to save somewhere)


They also have tags. And other branches. Point being, you could pin to a
version/branch or whatever. It's just an exercise in juggling the metadata
when building and capturing it somewhere. At that point you're much much
closer to being able to stand up a brand new fresh os x box and install
known versions of packages and known configurations of homebrew without
inventing nearly as much of this solo. And lets face it... Idiosyncrasies
with third party libs are a drag. At least engage a large community to
share/shake out wrinkles. At a high level it's just 4 sets of facts.

Re-lurking.

Post by Jim Cheetham

Post by john hood

Post by Jim Cheetham
Quoting john hood (2016-10-31 17:12:06)

Post by john hood
Alas, we will not get any kind of repeatable builds out of this, Travis
constantly updates their build images and we update to current Homebrew
for dependencies on every build.

That's the worst bit. Using external services that are *unlikely* to

attack

Post by john hood

Post by Jim Cheetham
your process is generally just fine, as long as there is a way to

verify their

Post by john hood

Post by Jim Cheetham
output.

The repeatability is of course no worse than what we've got now :)

Post by Jim Cheetham
Perhaps you could use Travis to report in the buildability of a

revision,

Post by john hood

Post by Jim Cheetham
and the source of a 'nightly build' version, but keep a repeatable
build chain for official releases?

We already use Travis for CI builds on Linux and OS X. I could of
course maintain a build VM for Mosh releases and snapshot it for each
release build, but having to maintain it (OS X upgrades, Xcode upgrades)
for Mosh's fairly infrequent releases is a significant burden for a
small project, and it would bring us full circle to manually-maintained
private builds.
We can at least report the Travis image id (I think this is already in
the build log) and the Homebrew version + Git revisions, and other tool
versions. Homebrew has a way to dump its configuration ('homebrew info
--json=v1') which may or may not be complete.
The elephant in the room is that we're trying to improve builds for a
binary-only, proprietary OS with relatively ad-hoc installation,
configuration, packaging, etc.

I've had some more thinking on this, and I'm getting over the
you-piled-what?-onto-my-tasklist reaction.
* Fully automating the OS X build and installing its build deps. This
is a pretty basic first step to all the rest of it, and what I am trying
to do with Travis.
* Reporting: describing exactly what was built, how it was built, what
dependencies were used, other environmental factors.
* Repeatability: getting the same thing twice in a row :)
* Reproducibility: making it possible, maybe even easy, for you to get
the same thing out of a build that I did.
* Trusted build infrastructure: in theory a reproducible build reduces
the need for this. In practice...yeah, we should do that.
A lot of this stuff amortizes nicely if you are a distribution or
package management system. We're trying to do just one package, only
for OS X (and not our many other platforms), and not even for all OS X
users.
My current OS X package manager of choice is Homebrew. It seems they've
done some work towards reproducible builds internally, but it's not
clear at all how you might reproducibly install Homebrew, or do
non-Homebrew builds reproducibly with Homebrew deps. I just put a query
out on that.
It would be nice if Homebrew or Macports or some other package system
could build standalone apps, and had reproducible builds.
Another alternative might be to abandon package managers entirely for
release builds, and build our dependencies ourselves. Xcode might have
some useful capabilities in this area, I haven't looked. This may
require separate, clean OS X instances for package builds.
This isn't going to happen instantly. One approach to the trust issue
here might be to just cop out-- stop doing OS X package builds and tell
people to build their own, until we get this stuff into better shape.
Have I missed anything obvious? (Yes.) Have any of you got wisdom on
dealing with this stuff in an OS X environment that you can share?
(Please!)
regards,
--jh
_______________________________________________
mosh-devel mailing list
http://mailman.mit.edu/mailman/listinfo/mosh-devel

[Jim Cheetham]

Quoting john hood (2016-11-02 18:45:17)

Post by john hood
This isn't going to happen instantly. One approach to the trust issue
here might be to just cop out-- stop doing OS X package builds and tell
people to build their own, until we get this stuff into better shape.

That probably doesn't really help - just because you can't have a 'perfect' build environment doesn't mean that the world is a better place if you have none :-)

Transparency is the first part: whatever your build method is, make sure it is described so that potential users can make up their own minds about the risks.

Then add as much accountability as possible, in terms of build logs and any artefacts like that, in case someone else can spot a problem that you didn't see on a particular given run. That is also helpful in terms of build quality itself, not just the security issues.

As long as there is a viable path for an end-user to take from the source to a binary without trusting anything else from you, that's great. Some products (e.g. TrueCrypt) were exceptionally difficult to build, which was a problem.

A repeatable build environment with a transparent trusted process? Probably not going to happen in the OS X world very easily, so just take steps to get closer :-)


--
Jim Cheetham, Information Security, University of Otago, Dunedin, N.Z.
✉ ***@otago.ac.nz ☏ [phone] ☏ m [phone]
⚷ OpenPGP: B50F BE3B D49B 3A8A 9CC3 8966 9374 82CD C982 0605

[john hood]

Post by Jim Cheetham
Quoting john hood (2016-11-02 18:45:17)

Post by john hood
This isn't going to happen instantly. One approach to the trust issue
here might be to just cop out-- stop doing OS X package builds and tell
people to build their own, until we get this stuff into better shape.

That probably doesn't really help - just because you can't have a 'perfect' build environment doesn't mean that the world is a better place if you have none :-)
Transparency is the first part: whatever your build method is, make sure it is described so that potential users can make up their own minds about the risks.
Then add as much accountability as possible, in terms of build logs and any artefacts like that, in case someone else can spot a problem that you didn't see on a particular given run. That is also helpful in terms of build quality itself, not just the security issues.
As long as there is a viable path for an end-user to take from the source to a binary without trusting anything else from you, that's great. Some products (e.g. TrueCrypt) were exceptionally difficult to build, which was a problem.
A repeatable build environment with a transparent trusted process? Probably not going to happen in the OS X world very easily, so just take steps to get closer :-)

It took a while but I've gotten someplace useful with this. I improved
the Travis package build to better report its components and
environment, and the result can be seen at
<https://github.com/mobile-shell/mosh/pull/822> along with links to a
sample build.

It appears that deterministic builds are Not Possible with the Xcode
toolchain-- apparently the linker is threaded and puts things together
in an unpredictable order. The bitcoin folks have managed to get
deterministic builds, but that requires cross-compiles on Linux VMs with
various tools assembled from various places, and some of the tools are
apparently fairly broken. So I've put that aside for now.

I would certainly appreciate comments on how that build looks-- it
certainly reports things, and produces hashes on build products to add
some traceability, but I'm sure it can be improved.

regards,

--jh

[John Hawkinson]

I think we should not lose sight of the fact that mosh is
security-sensitive software in a category unlike many other
software packages, and thus it is worth some inconvenience to
the maintainers that you might not accept in a less sensitive
tool.

Another question: would we rather we be compromised at the same time
when Github or Travis is compromised, or would we rather be
compromised independently at a different time? Feel free to substitute
"if" for "when" if it makes you feel better.

--***@mit.edu
John Hawkinson

Post by john hood
The reasons I did this: to get a more externally-visible build for the
package, and to do the release build on something other than my personal
OS X machines. Of course, this means we are trading trust in my OS X
systems for trust in Travis' OS X build environments. I think this is a
win, any opinions?
Alas, we will not get any kind of repeatable builds out of this, Travis
constantly updates their build images and we update to current Homebrew
for dependencies on every build.

[john hood]

Post by John Hawkinson
I think we should not lose sight of the fact that mosh is
security-sensitive software in a category unlike many other
software packages, and thus it is worth some inconvenience to
the maintainers that you might not accept in a less sensitive
tool.
Another question: would we rather we be compromised at the same time
when Github or Travis is compromised, or would we rather be
compromised independently at a different time? Feel free to substitute
"if" for "when" if it makes you feel better.

A question in reply: would we be likelier to *discover* that compromise
on a personal, daily-driver OS X install, or on a cloud service provider
with wide visibility (and exposure)? I'm dubious of my ability to
discover a compromise on my personal machines, and I'd of course like to
keep them personal. :)

Again, the idea of, say, a project-owned build VM comes up. Apple's OS
X licensing would seem to restrict this to an Apple developer and an OS
X host, though, which makes this more difficult.

regards,

--jh