Tales from consulting, part I

Mark and I just returned from a long consulting engagement, at 3 weeks on site it was one of the longer (if not the longest) trips I've taken for Immunity. I'd like to share a few things that worked, didn't work and that I would change. There will probably be a number of these posts in the future from both Mark and I.

Physical Pen-testing:

1) Simple is better. One of our tasks was to try and get into some restricted areas by posing as someone else. We made a modest investment in clothing and equipment so that I could try and blend in. While this was successful to a certain degree what ended up working best was just being what I was, an IT guy. Throw in a Fluke network tester and a keyboard as a prop and you're set to go.

2) Badges? Our client was using badges which required mutual authentication and would have been difficult to clone with the equipment we had on hand. So we took some high resolution pictures of legitimate badges, used some photoshop wizardry (courtesy of Mark), printed it out on sticker paper and created a new "badge". Any time I had to get into some place I shouldn't have been simply asking someone near by to buzz me in because my badge was "bent" worked like a charm. If you do these kind of gigs regularly investing in a card printer may be worth your while if you expect a high level of scrutiny.

3) Physical keyloggers are ridiculously effective. I'd used USB keyloggers before but they always surprise me with just how useful they are. We planted ours in conference rooms and general computing facilities and wound up with almost 40 sets of credentials over a two week period from just 4 devices. A tool like the Power Pwn would've been very nice to have as well.

4) A quick and silent way to take photos. I learned that the camera clicking noise my phone makes can not be silenced. Dave suggested some sort of slim video recording device we could attach to ourselves but I wasn't impressed with the resolution of the devices I looked at. A small, high resolution, quick shooting camera would be ideal.

5) Have a story for why you're there. One ruse I used was that one of my coworkers from the IT department had misplaced a piece of equipment in this general area over the previous week. I would ask the secretary or anyone close to the entrance if I could poke around to see if I could spot it. This got me anywhere I wanted to go, including locked conference rooms. The pretense of repairing conference room keyboards (and showing my keyboard prop) also got me out of a number of surprise visits from people coming in to use the room.

6) Don't over think it. Unless you're getting into a datacenter or an IBM campus to most people technology is magic and you can use that to your advantage. I had a trojan I carried around on a USB key that I would pop into open workstations. We debated on what I should say if I was challenged. "I'm here to inventory this system for IT" was simple but the question was raised amongst ourselves: "are you doing an inventory of hardware or software and if you're an admin you could just do this over the network..." Most people don't care about these details, inventory is a thing that IT does, you look like a nerd, therefore your story probably checks out.

Things to note:

a) When placing keyloggers or doing hardware work always make sure you have a minimum set of tools with you: screw driver with flat/philips/torx bits, headlamp and maybe a lock pick and shim set for getting into cabinets.

b) Know the schedule for the area you're getting into. If it's a conference room, when do you need to be there? Work area, when is shift change? Be familiar with the names of the areas around you, what department is on a nearby floor, who is the admin in charge of scheduling this room, etc.

c) When getting a USB keylogger be mindful of the form factor, some PCs have tight USB clusters and if your keylogger is fat enough it may block other ports that you need. Having a short F->M USB cable handy is good for quick fixes.

d) Practice, practice, practice with your picks and your shims. Beating a lock on your bench in an air conditioned room is easy. Beating the same lock in a dark room where you shouldn't be with no AC and under time pressure is hard. On this gig I got 0 locks and 1 shim, ultimately I found other ways to solve my problem but I clearly need practice.

Conclusions:

Over the past 5 years doing consulting gigs for Immunity it's been my experience that when we're on the inside of a network and our scope is sufficiently open then there's always a way to win. The same is true for physical pen-testing, if we have access to your space (or sometimes just in range) then we're on your network and we'll almost certainly have credentials. One of the most surprising things we learned was that our USB key logging was approximately as effective as our phishing was at obtaining credentials. While credentials aren't shells, as I've mentioned it's not always about shells. If the data you need doesn't require code execution to get then this is a boon for your stealth.

What we learned from teaching Web Hacking at Infiltrate 2013

So this year myself (@alexm_py), Miguel, Matias (@gunler) and a guest appearance by Nico (@nicowaisman) were the teachers for the Web Hacking class at Infiltrate. This is our third time teaching the course and we learned a lot along the way, I've tried to only focus on things that are applicable outside our organization.

1) We write a ton of custom software for our classes and that investment is seen in higher rate of students actually learning the content. Case in point, Matias wrote an amazing web application for our web crypto class explaining how ECB, CBC and Padding Oracle worked (here's a look). They key seemed to be that the students could tinker around and add different values to see what effects were had on the algorithm. Students walked out of our class understanding the crypto concepts better than any other time we've taught this content.

2) Putting in the extra effort to make your applications look good is worth it. We decided to teach command injection (CMDi) as a proper module this year rather than just including it in our reference material so I had to write up new slides and new exercises. I was pretty happy with how it turned out. The student response to this exercise vs. one of the XSS exercises was pretty evident. My initial thought with XSS was that by keeping things as very simple HTML we could focus on the vulnerability but instead it detracted from the quality of the experience. That's going to get fixed.

3) Students in our classes are happiest with their fingers on keyboards. The first two modules we teach are open source information gathering (OSIG) and versioning. In OSIG we spend a lot of time talking about methods to find vhosts, Google dorks, and other methods to find out information about the sites being assessed. Versioning is exclusively about determining the version of installed CMS's and webservers. We taught these in a more lecture heavy style and it was our #1 complaint amongst students. I think the information is important and spot on but we need to re-do how we teach it.

4) Consider having a separate day to go over introductory material. Because we get students from all different ability levels we cover some basic information: how HTTP traffic works, intro to the Linux CLI, intro to Python, intro to JavaScript, intro to SQL (mostly MySQL). A number of students commented on how they thought this slowed down the pace of the class too much. We had explored the idea of having an optional day at the beginning where students who were unfamiliar or uncertain about this information could have a day of focused instruction. Ultimately we didn't do it but it is an idea we'll have to revisit.

5) Test your exercises in the environment you're presenting them in! I made the mistake of testing from a development laptop rather than from a student's laptop and the minor differences became embarrassing. Also make sure to fully test each of your exercises and their solutions, just because something worked last time doesn't mean it will work this time in this environment. Since we did a scoring system our new rule is that before the class we will beat each exercise and challenge until our demo student receives a perfect score.

6) Students tend to appreciate the little things. Our classes are typically catered, at Infiltrate we did breakfast, lunch and an afternoon snack. Asking the students how the food was and if there was any dietary restrictions we should meet went a long way, I had a few students go out of their way to thank us for doing this.

7) Lastly is general collection

Things will go wrong with your connection to the hotel network, have someone from your team who's main responsibility is to get the class network and the hotel network talking (if applicable) If you're moving a lot of gear invest in hand trucks that turn into carts (example) and other items to secure your gear to the cart like sturdy cases and straps Always keep a minor first aid kit handy, I sliced my finger open while moving our overloaded and improperly secured hand truck :( Consult with the hotel and determine the best place to load in your gear rather than just showing up at the entrance Ask the hotel if they can print 8x11 signage or if you can do it yourself, some hotels are really picky about this And above all else keep your cool. Putting classes together is hard, teaching classes is hard, don't make it more difficult by blowing your top

Many thanks to the following hombres/hombrettes (alpha order): Alfred, Carissa, Dave, Linda, Ray, Vanessa and the conference staff at the Fontainebleau

Predicting your future from past reports

You, as a security consulting customer, have a relationship with your security vendor. However, your vendor could be providing you with more value then you may currently be receiving in your partnership. For example, one thing Immunity does with our repeat customers is that we review the entire years worth of assessments and find patterns that can say surprising things about the enterprise.

In order to do this I take every deliverable that Immunity created from that year and extract the details about all of the vulnerabilities found and create a master list of findings. Then using the information gathered from the master list, I prepare a presentation for each our clients which provides a years worth of results in a quick and easy to read format. A sample of the information you can expect to receive is:

A sample comment can be "You only had a few denial of service issues, but if you remember, those were all critical issues that could end a business line."

I then review the presentation with the client and answer questions such as:

•What kinds of engagements did Immunity work on over the course of the year (i.e web applications, custom software, third party assessments, etc.)?
• Are there repeats of the same type of vulnerabilities across different platforms/applications and if so where and how much does it appear and what is the threat level?

This is obviously a chart from a full year...if you have 39 critical findings from one engagement then you are splitting vulnerabilities up with too much granularity.

•What percentage of vulnerabilities found were critical, high, moderate and low. How does this compare to prior years?
•Do the majority of the vulnerabilities come from third party applications or from in-house developed applications?

An example conversation that I may have with a client when discussing the annual recap is that it appears that they got a hold of cross-site scripting vulnerabilities but SQL Injections still remains a problem when compared to the previous years. Or perhaps that every application that does high-frequency trading has huge amount of consequential vulnerabilities (a.k.a - this is a risky business line so do you accept the risk to gain the reward to do you search for another product?).

If your security vendor is not providing you with this value then you are missing out!



Deliverables and how they evolve

There's a lot about doing quality security services that is not, as we say, self evident. One of these things is that every gig is different. Today Mark and I and Shari and Bas worked quite a bit on a particular gig that requires an "expert opinion". Essentially you take someone else's deliverable, look at the same evidence, and then decide what they may have missed, or what questions to ask next. 

This kind of service is extremely valuable when you have multiple parties involved with multiple conflicting interests and you want a technical readout on the facts as they stand. A good security consulting company can come to the table both without an agenda, and with years of experience in the technology and ideally, in the specific customer vertical you are addressing.

Now, in security consulting, you never have all the information you want. You can always use more time, or more data, or the logs off some random machine, or the source code to some piece of the puzzle. But the deliverable has a due date.

Occasionally, however, you find halfway during your review process that things you (as a team) think about are made clear. For example, you may find that your understanding of the threat model becomes more focused as you write the deliverable. 

For example, today we were looking at something where hackers broke into a system using SQL Injection. We were initially curious why none of their tools were manually deleted, yet were still not available on the system, when we realized that they used a remote webdav file share to both store their tools and exfiltrate data. This then revised our understanding of the outbound firewall ruleset, which led us into other areas of analysis that needed to be re-addressed.

In other words, from the client's perspective, this is what the process looks like:

But what consulting looks like when you do it right is this:

That analysis phase is the hardest phase. I will bullet list some reasons below:

• It requires consultants with years of experience - the most senior members of your team. These people are often so valuable in other areas that you have to fight for their time.
• Consultants hate being told they are wrong (and especially hate rewriting documents they have already put time into). Being able to have frank discussions about rethinking conclusions and then making the painful decision to redo a document means you have solved the social aspects of the workplace to a high level.
• Your process has to have enough fluff time at the end for documentation that you can afford to go back and do major revisions. It is likely you are under price pressure, and have already strictly cut the time for your project down as far as it can go. Only a confident (or lucky) company can keep their scoping up high enough to allow this kind of process to occur.

Aside from better deliverables (let's face it, you may end up getting paid the same either way), there are other major benefits to implementing this expensive process, for example, knowledge transfer within your consulting team. 

But as a client, one good question when getting a deliverable from a consulting services company is simply: "How many people within your organization who did not work on the project have READ this deliverable?" If the answer is none, you got shortchanged.