The Magic behind Burp and ZAP and other Proxies

If you build web applications and care about security, you have probably used the Burp and ZAP proxy security tools. These tools perform dynamic analysis on live web applications to identify security vulnerabilities. Burp and ZAP can discover issues with your applications as you nagivate through them via a browser. Essentially, you they configured to be a “man in the middle” and intercept all traffic between your browser and web application. Have you ever wondered how it is possible to intercept encrypted traffic over https? This article explains how it is done and provides a basic framework for creating your own proxy software.

To get started with Burp and ZAP (from now on, I’ll refer to these as simply the “proxy”), you have to decide what port you want the proxy to listen on and configure your browser to use that port as a proxy. In Firefox, to use port 9000, your configuration might look like the following:

Note, that the Use this proxy server for all protocols box should be checked. Your browser is now ready to send and receive data through a proxy. Let’s now start to put together some code to handle these browser requests.

First, we’ll need to set up a ServerSocket to listen for requests:

ServerSocketFactory serverSocketFactory = ServerSocketFactory.getDefault();
proxyServerSocket = serverSocketFactory.createServerSocket(LOCAL_PORT, 10000, InetAddress.getByName(LOCAL_INTERFACE));
ServerSocketHandler handler = new ServerSocketHandler(proxyServerSocket, false);
handler.start();

Here we use a ServerSocketHandler thread to handle the requests:

class ServerSocketHandler extends Thread {
    ServerSocket serverSocket;
    boolean secure;

    ServerSocketHandler(ServerSocket serverSocket, boolean secure) {
        this.serverSocket = serverSocket;
        this.secure = secure;
    }

    public void run() {
        try {
            while (true) {
                Socket socket = serverSocket.accept();
                RequestHandler handler = new RequestHandler(socket);
                handler.start();						
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

This thread loops indefinitely. When it receives a request, it creates a RequestHandler to process it. The first thing it does is read the request:

while ( (line = br.readLine()) != null) {

    if (first) {
        first = false;
        String chunks[] = line.split(" ");
        method = chunks[0];
        urlText = chunks[1];
    }

    buffer.append(line + "\r\n");
    if (line.equals(""))
        break;
}

In the code that follows there is an if statement to determine if the request is proxied over https and an else clause that follows. Let’s start with the else clause, which shows how non-https traffic is handled:

else if (urlText != null) {
    System.out.println("ATTEMPTING URL: " + urlText);
    URL url = new URL(urlText);

    Request request = new Request(Message.parseMessage(new ByteArrayInputStream(buffer.toString().getBytes()), true));

    SocketFactory socketFactory = SocketFactory.getDefault();
    int port = url.getPort();
    Socket targetSocket = socketFactory.createSocket(url.getHost(), port < 0 ? 80 : port);
    OutputStream targetOs = targetSocket.getOutputStream();
    targetOs.write(request.message.httpHeader.bytes);
    targetOs.write(request.message.body.rawBytes);
    targetOs.flush();

    InputStream targetIs = targetSocket.getInputStream();
    Response response = new Response(Message.parseMessage(targetIs, false));
    final RequestResponse rr = new RequestResponse(request, response);
    Thread thread = new Thread() {
        public void run() {
            try { DataManager.INSTANCE.add(rr); } 
            catch (IOException e) {
                e.printStackTrace();
            }
        }
    };
    thread.start();

    socket.getOutputStream().write(response.message.httpHeader.bytes);
    socket.getOutputStream().write(response.message.body.rawBytes);
    socket.getOutputStream().flush();
    socket.close();
}

In this block of code, we first create a Request object by parsing the bytes from the buffer object. buffer contains the contents of the HTTP request. The Request object is used to store an HTTP request. Its contents can be used to forward the original request to the target host and can be used to store the original request for our own purposes (just like Burp and ZAP).

Next we create a socket using the target host address and send the request. We then read the response from the target host and parse that into a Response object. Now that we have a Request and Response, we can create a RequestResponse object and store it indefinitely using the DataManager class. We create a new thread to perform the actual storage because we don’t want to wait for this task to complete before sending the response back to the client. Finally we send the response back to the client.

There should be no major surprises in what we have discussed thus far. Just as you might imagine a proxy intercepts a request and records it. It forward the request to the target host and the records the response before sending it back to the client. Proxying really only gets interesting when we start to examine what happens in an HTTPS request. You might think that the exact same process happens, except that it uses HTTPS. If you weren’t interesting in capturing the unencrypted data, there would be some truth to this, but otherwise it gets quite complicated. You must consider that a client wishing to exchange encrypted data with example.com expects to receive an Certificate stamped with “example.com” by a Certificate Authority. If you have used Burp or ZAP before, then you probably already know that they generate their own certificates using their own Certificate Authority certificate, which must also be installed as a trusted certificate in your browser. If that wasn’t clear, here it is in bullet form:

1. In PKI encryption, senders and receivers of encrypted data use certificates to identify themselves. Each certificate must be issued by a Certificate Authority. Certificate Authorities have certificates pre-installed in your browser and are automatically trusted.

2. Our proxy can’t use these certificates because they are password protected. Hence, we create our own Certificate Authority certificate, which we can use to issue our own certificates.

3. Our homegrown Certificate Authority certificate must be installed in the browser in order to trust our own web site certificate.

Our first problem to tackle is generating a Certificate Authority certificate. I put together a utility class for this called CreateCertificateAuthorityUtil. It must be run once independent from the Proxy to create the certificate. It has a main method, which invokes the createAndExportSelfSignedCertificateTo (…) method. Here they are:

public static void main(String[] args) throws Exception {
  createAndExportSelfSignedCertificateTo(CertUtil.KEY_PAIR_GENERATOR.generateKeyPair(), CA_KEYSTORE_FILE, CA_CERT_FILE);
}

public static X509Certificate createAndExportSelfSignedCertificateTo(KeyPair caKeyPair, String keyStoreLocation, String derFileLocation) throws Exception {
  X509Certificate caCert = CertUtil.createCertificate(CA_X500_NAME, CA_X500_NAME, caKeyPair.getPublic(), caKeyPair.getPrivate(), true);
  KeyStore ks = KeyStore.getInstance("JKS");
  ks.load(null, CA_KEYSTORE_PASSWORD.toCharArray());
  ks.setKeyEntry(CA_KEY_ALIAS, caKeyPair.getPrivate(), CA_KEYSTORE_PASSWORD.toCharArray(), new X509Certificate[]{caCert});

  try (PrintStream ps = new PrintStream(derFileLocation); 
    FileOutputStream fos = new FileOutputStream(keyStoreLocation);) {
    CertUtil.printCertificateTo(caCert, ps);
    ks.store(fos, CA_KEYSTORE_PASSWORD.toCharArray());
  }

  return caCert;
}

These methods reference some class-scoped variables, some of which are initialized in a static initalizer block:

static final File TMP_DIRECTORY = new File(System.getProperty("java.io.tmpdir"));
static final File DEFAULT_CA_KEY_STORE_FILE = new File(TMP_DIRECTORY, "cakeystore.ks");
static final File DEFAULT_CA_CERT_FILE = new File(TMP_DIRECTORY, "cacert.der");

static final String CA_KEY_STORE_FILE_PROPERTY_NAME = "ca-keystore-file";
static final String CA_CERT_FILE_PROPERTY_NAME = "ca-cert-file";

public static final String CA_KEYSTORE_PASSWORD = "password";
public static final String CA_KEY_ALIAS = "ca_alias";

public static final String X500_CN_COMMON_NAME_VALUE = "Fake Certificate Authority Inc.";
public static final String X500_OU_ORGANIZATIONAL_UNIT_VALUE = "Fake Certificate Authority OrgUnit";
public static final String X500_O_ORGANIZATIONAL_VALUE = "Fake Certificate Authority Org";
public static final String X500_L_LOCALITY_VALUE = "Fake Certificate Authority City";
public static final String X500_ST_STATE_PROVINCE_NAME_VALUE = "Fake Certificate Authority State";
public static final String X500_C_COUNTRY_NAME_VALUE = "Fake Certificate Authority Country";

public static final String CA_KEYSTORE_FILE;
public static final String CA_CERT_FILE;

public static final X500Name CA_X500_NAME;

static {
  String value = System.getProperty(CA_KEY_STORE_FILE_PROPERTY_NAME);
  CA_KEYSTORE_FILE = value == null ? DEFAULT_CA_KEY_STORE_FILE.getAbsolutePath() : value;
  value = System.getProperty(CA_CERT_FILE_PROPERTY_NAME);
  CA_CERT_FILE = value == null ? DEFAULT_CA_CERT_FILE.getAbsolutePath() : value;
  try { CA_X500_NAME = CertUtil.createName(X500_CN_COMMON_NAME_VALUE, X500_OU_ORGANIZATIONAL_UNIT_VALUE, X500_O_ORGANIZATIONAL_VALUE, X500_L_LOCALITY_VALUE, X500_ST_STATE_PROVINCE_NAME_VALUE, X500_C_COUNTRY_NAME_VALUE); } 
  catch (IOException e) {
    throw new Error("Couldn't create Certificate Authority X500 Name", e);
  }
  System.out.println("CA Keystore file: " + CA_KEYSTORE_FILE + ", and CA Cert file: " + CA_CERT_FILE);
  System.out.println("CA X500 Name: " + CA_X500_NAME);
}

The static initializer is used set up the CA_KEYSTORE_FILE, CA_CERT_FILE, and CA_X500_Name constants. By making them static, they can be referenced by the proxy. The proxy will use the CA_KEYSTORE_FILE to get the Certificate Authority certificate’s private key, which will be used to create the certificate to send to the browser to impersonate the real host. It also uses CA_X500_NAME to create the certificate.

Let’s now take a closer look at the createAndExportSelfSignedCertificateTo(…) method. The bulk of the code creates a KeyStore and stores the Certificate Authority certificate. However, the first line creates the Certificate Authority certificate by calling a method in the CertUtil utility class, createCertificate(…). This method can be used to create Certificate Authority certificates or SSL certificates. It has a boolean isCertificateAuthority parameter to specify which type. Here is the code:

public static X509Certificate createCertificate(X500Name name, X500Name issuerName, PublicKey publicKey, PrivateKey signerPrivateKey, boolean isCertificateAuthority) throws Exception {
  X509CertInfo certInfo = new X509CertInfo();
  certInfo.set(X509CertInfo.SERIAL_NUMBER, new CertificateSerialNumber(new BigInteger(64, new SecureRandom())));
  certInfo.set(X509CertInfo.VERSION, new CertificateVersion(CertificateVersion.V3));

  // Validity
  Date validFrom = new Date();
  Date validTo = new Date(validFrom.getTime() + ONE_YEAR);
  certInfo.set(X509CertInfo.VALIDITY, new CertificateValidity(validFrom, validTo));

  boolean justName = isJavaAtLeast(1.8);

  if (justName) {
    certInfo.set(X509CertInfo.SUBJECT, name);
    certInfo.set(X509CertInfo.ISSUER, issuerName);
  } 
  else {
    certInfo.set(X509CertInfo.SUBJECT, new CertificateSubjectName(name));
    certInfo.set(X509CertInfo.ISSUER, new CertificateIssuerName(issuerName));
  }

  if (isCertificateAuthority) {
    CertificateExtensions ext = new CertificateExtensions();
    ext.set(BasicConstraintsExtension.NAME, new BasicConstraintsExtension(Boolean.TRUE, Boolean.TRUE, 0));
    certInfo.set(X509CertInfo.EXTENSIONS, ext);
  }

  // Key and algorithm
  certInfo.set(X509CertInfo.KEY, new CertificateX509Key(publicKey));
  AlgorithmId algorithm = new AlgorithmId(AlgorithmId.sha1WithRSAEncryption_oid);
  certInfo.set(X509CertInfo.ALGORITHM_ID, new CertificateAlgorithmId(algorithm));

  // Create a new certificate and sign it
  X509CertImpl cert = new X509CertImpl(certInfo);
  cert.sign(signerPrivateKey, SHA1WITHRSA);

  return cert;
}

Here is an overview of what it does:

  1. Creates an X509CertInfo object and gives it a serial and version number
  2. Defines the period for which it is valid
  3. Sets the subject and issuer
  4. Sets appropriate properties if creating a Certificate Authority
  5. Sets the public key and algorithm
  6. Signs it
If we now jump back to the createAndExportSelfSignedCertificateTo(…) method, you’ll note it not only stores the certificate in the KeyStore, but also writes out the certificate to disk. This is so it can easily be imported by a browser. By default, the certificate is written to <java.io.tmpdir>/cacert.der. Let’s now import it into FireFox.
First, click the Open Menu button and select Options:
 

Next, search for “Certificate” and click View Certificates:

In the result dialog, click the Authorities tab and then click the Import… button:

A file browser dialog will appear. Now select the <java.io.tmpdir>/cacert.der file you created earlier. The following dialog will appear:

Click the first check box and click the OK button. You have now imported the Certificate Authority certificate into your browser.

We can now get back to understanding the proxy code, but first, let’s consider how a proxy would be able to process an HTTPS request if it is encrypted. Specifically, if HTTPS functioned the same as HTTP, the proxy wouldn’t know the target address because it is embedded in an encrypted HTTP request. To solve this problem, HTTPS uses a variation on HTTP. Instead of embedding the target host and port in the request headers, it sends a Connect request, which contains the URL and the HTTP method (i.e. GET, POST etc). The proxy will send back a receipt of the Connect request with a 200 response code. Here is the relevant code:

if ("connect".equalsIgnoreCase(method)) {
String connectResponse = "HTTP/1.0 200 Connection established\n" +
"Proxy-agent: ProxyServer/1.0\n" +
"\r\n";

clientOs.write(connectResponse.getBytes());
clientOs.flush();

The next step is to create an SSLSocketFactory object to communicate with the real host over SSL:

SSLSocketFactory secureSocketFactory = CertUtil.getTunnelSSLSocketFactory(url.getHost());

You will notice the getTunnelSSLSocketFactory(…) method takes a parameter of the target host. We need this information to set up a fake certificate. Here is the code:

public static SSLSocketFactory getTunnelSSLSocketFactory(String hostname) throws UnrecoverableKeyException, KeyStoreException, NoSuchAlgorithmException, Exception {
  SSLSocketFactory factory = factoryMap.get(hostname);
  if (factory != null)
    return factory;

  try {
      SSLContext ctx = SSLContext.getInstance("TLS");
      KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
      KeyStore ks = generateHostKeystore(hostname);

      kmf.init(ks, PASSWORD.toCharArray());
      java.security.SecureRandom random = new java.security.SecureRandom();
      ctx.init(kmf.getKeyManagers(), null, random);
      SSLSocketFactory tunnelSSLFactory = ctx.getSocketFactory();

      factoryMap.put(hostname, tunnelSSLFactory);

      return tunnelSSLFactory;
  } 
  catch (Exception e) {
     throw new RuntimeException(e);
  }
}

The first line checks to see if we have set up an SSLSocketFactory for this host already. If so, we re-use it. The rest of this code is standard code for setting up the factory except for this line:

KeyStore ks = generateHostKeystore(hostname);

generateHostKeyStore(…) is another utility method in CertUtil, which we require to set up the SSLSocketFactory. Here is the code:

public synchronized static KeyStore generateHostKeystore(String host) throws Exception {
  try (FileInputStream fis = new FileInputStream(CreateCertificateAuthorityUtil.CA_KEYSTORE_FILE); ) {
    KeyStore caKeyStore = KeyStore.getInstance("JKS");
    caKeyStore.load(fis, CreateCertificateAuthorityUtil.CA_KEYSTORE_PASSWORD.toCharArray());

    PrivateKey caPrivateKey = (PrivateKey) caKeyStore.getKey(CreateCertificateAuthorityUtil.CA_KEY_ALIAS, CreateCertificateAuthorityUtil.CA_KEYSTORE_PASSWORD.toCharArray());
    KeyPair hostKeyPair = KEY_PAIR_GENERATOR.generateKeyPair();
    X500Name hostName = createName(host, X500_HOST_OU_ORGANIZATIONAL_UNIT_VALUE, X500_HOST_O_ORGANIZATIONAL_VALUE, X500_HOST_L_LOCALITY_VALUE, X500_HOST_ST_STATE_PROVINCE_NAME_VALUE, X500_HOST_C_COUNTRY_NAME_VALUE);
    java.security.cert.Certificate cert = createCertificate(hostName, CreateCertificateAuthorityUtil.CA_X500_NAME, hostKeyPair.getPublic(), caPrivateKey, false);
    PrivateKey privateKey = hostKeyPair.getPrivate();

    KeyStore hostKeyStore = KeyStore.getInstance("JKS");
    hostKeyStore.load(null, PASSWORD.toCharArray());
    hostKeyStore.setKeyEntry(HOST_ALIAS, privateKey, PASSWORD.toCharArray(), new java.security.cert.Certificate[] { cert });

    return hostKeyStore;
  }
}

Here is an overview of the code:

  1. Extract the Certificate Authority private key from the Certificate Authority keystore
  2. Generate a new KeyPair for the new SSL certificate
  3. Create the X500Name for the host
  4. Create the host certificate using the CertUtil.createCertificate(…) method
  5. Store the host certificate in an in-memory KeyStore

Let’s now go back to the proxy code. After we create the SSLSocketFactory, we have the following line:

SSLSocket sslSocket = (SSLSocket) secureSocketFactory.createSocket(socket, socket.getInetAddress().getHostAddress(), socket.getPort(), true);

This is a special version of the SecureSocketFactory.createSocket(…) method, which takes a socket a parameter. Essentially, it abstract the encryption/decryption for us so we can read and write unencrypted data to the socket that the browser is connected to.

The next few lines tell the socket to behave in client mode and initate the SSL handshake:

sslSocket.setUseClientMode(false);
try {
  sslSocket.startHandshake();
} catch (Exception e) {
System.err.println("Error on URL: " + urlText);
  throw new RuntimeException(e);
}

The rest of the code in this method is essentially the same as the code for handling HTTP in the else clause. You can now run the Proxy class and start proxying!

You should note that this code is only proof-of-concept quality and would need a vast amount of improvement before being production ready. For example, it only handles 200 response codes. It will throw exceptions for other response codes.

You may also want to note the following:

  1. The code uses restricted access APIs from the JRE. In order to reference them in Eclipse, you will have to modify how Eclipse treats these APIs. Specifically, go to Java Compiler Errors/Warnings panel and change the Forbidden Reference (access rules) setting from Error to Warning or less.
  2. When testing your proxy code, it can be a little bit overwhelming when visiting a real web page. Most web pages don’t generate a single request. With images, style sheets, and scripts, a single web page can generate many requests. It can be confusing the debug your proxy code, when multiple requests are being created concurrently. To skirt this issue, I created a ProxyTest class, which uses HttpURLConnection to send a single request. Here is the code:
public static void main(String[] args) throws IOException {
  java.net.Proxy proxy = 
    new java.net.Proxy(java.net.Proxy.Type.HTTP, new InetSocketAddress(Proxy.DEFAULT_LOCAL_INTERFACE, Proxy.DEFAULT_LOCAL_PORT));

//    URL url = new URL("https://www.google.ca/?q=asdf");
//    URL url = new URL("https://www.google.ca/");
//    URL url = new URL("https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png");
    URL url = new URL("https://www.tutorialspoint.com/");

    HttpURLConnection.setFollowRedirects(false);
    HttpURLConnection c = (HttpURLConnection) url.openConnection(proxy);

    InputStream is = c.getInputStream();
    IOUtil.copy(is, System.out);
}

You should be aware that HttpURLConnection has setFollowRedirects(…) set to false. It will otherwise generate multiple requests for a redirect response code, which can be confusing.

You can find the full source code for this project on GitHub and the javadocs here.

This entry was posted in https, proxy, Security, Web. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>