Step 1: firewall off port 8080
Jenkins, by default, launches on port 8080 and anonymous users have full rights. This would let anonymous users run arbitrary code on your server. That's great for usability for a tool that's usually launched inside firewalls, but if you have a machine without a firewall...derp.
So, my recipe that provides some flexibility and some security was:
sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 443
sudo ufw default deny
sudo ufw enable
That lets http, https, and ssh traffic from anywhere on the net get into the box, but denies all other traffic. Defaults of ufw also allow all outbound traffic (which is handy for apt-get and other similar stuff). To be ideal you'd lock down specific outbound connections and also only allow 22 (i.e. ssh) from known good IP addresses. I'm not into managing that closely for this particular server. Read more docs on ufw.
But then...how do you connect to port 8080 for Jenkins access? You use an ssh tunnel:
ssh -qNf -L8080:localhost:8080 [email protected]
Then you fire up a browser to http://localhost:8080 and it's being tunneled over ssh to the server. But...nothing is running there yet...step 2.
2. Install Jenkins on Ubuntu
I was installing this on an Ubuntu 11.10 server (Oneiric) but I think this is probably a good guide: Jenkins Wiki on Installing Jenkins on Ubuntu. They use their own package outside Ubuntu's repository so you have to add the key, but I found it to be much more user friendly than the default Jenkins that comes with Ubuntu. So, I'm using it!