Description: 
+------------------------------------------------------------------------------+
|                           FORENSICS CONTEST                                  |
|                      Puzzle #4: The Curious Mr. X                            |
|                               --------                                       |
|                      Sébastien DAMAYE (aldeid.com)                           |
|                              2010-02-14                                      |
+------------------------------------------------------------------------------+

1. What was the IP address of Mr. X’s scanner?
- 10.42.253.253
2. For the FIRST port scan that Mr. X conducted, what type of port scan was it?
(Note: the scan consisted of thousands of packets.)
TCP SYN     [ ]    TCP ACK  [ ]    UDP     [ ]
TCP Connect [x]    TCP XMAS [ ]    TCP RST [ ]
3. What were the IP addresses of the targets Mr. X discovered?
- 10.42.42.25
- 10.42.42.50
- 10.42.42.56
4. What was the MAC address of the Apple system he found?
- 00:16:cb:92:6e:dc
5. What was the IP address of the Windows system he found?
- 10.42.42.50
6. What TCP ports were open on the Windows system? (Please list the decimal
numbers from lowest to highest.)
- 135
- 139
7. X-TRA CREDIT (You don’t have to answer this, but you get super bonus points
if you do): What was the name of the tool Mr. X used to port scan? How can you
tell? Can you reconstruct the output from the tool, roughly the way Mr. X would
have seen it?
- See detailed answer

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
0. General information
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
We first want to check file integrity, by issuing following command:

$ md5sum evidence04.pcap 
804648497410b18d9a7cb1d4b2252ef7  evidence04.pcap

       Figure 1. MD5 sum to check file integrity

Figure 2 indicates that capture duration is about 10 minutes long. It doesn't
give us enough information to determine whether the scan has been prepared (via
a script or a tool). Nevertheless, a deeper analysis in the frames enables to
withdraw the hypothesis of a manual scan.

File name: /home/sdamaye/forensics/puzzle4/evidence04.pcap
File type: Wireshark/tcpdump/... - libpcap
File encapsulation: Ethernet
Number of packets: 13625 
File size: 1093865 bytes
Data size: 875841 bytes
Capture duration: 606.082082 seconds
Start time: Wed Feb  3 00:34:06 2010
End time: Wed Feb  3 00:44:13 2010
Data rate: 1445.09 bytes/s
Data rate: 11560.69 bits/s
Average packet size: 64.28 bytes

       Figure 2. Capinfo provides capture duration

Another useful information concerns Protocol Hierarchy Statistics, provided by
Tshark. It shows that we have to deal with TCP, UDP and ICMP.

Protocol Hierarchy Statistics
Filter: frame

frame                                    frames:13625 bytes:875841
  eth                                    frames:13625 bytes:875841
    ip                                   frames:13625 bytes:875841
      tcp                                frames:13581 bytes:868753
        nbss                             frames:26 bytes:2663
        data                             frames:2 bytes:332
        dcerpc                           frames:1 bytes:90
      udp                                frames:24 bytes:3256
        nbns                             frames:20 bytes:1888
        data                             frames:4 bytes:1368
      icmp                               frames:20 bytes:3832

       Figure 3. Protocol Hierarchy Statistics provided by Tshark

Argus (qosient.com) provides a quick way to identify hosts.

$ argus -r evidence03.pcap -w evidence03.ra
$ rahosts -r evidence04.ra
10.42.42.25: (3) 10.42.42.50, 10.42.42.253, 10.255.255.255
10.42.42.50: (3) 10.42.42.25, 10.42.42.253, 10.255.255.255
10.42.42.56: (1) 10.42.42.253
10.42.42.253: (3) 10.42.42.25, 10.42.42.50, 10.42.42.56

       Figure 4. List of hosts

In addition, combining racluster with rasort immediately shows that the majority
of the traffic is coming from 10.42.42.253, which (it will be confirmed later)
seems to be the attacker's (Mr. X) IP address.

$ racluster -M norep -m saddr daddr -nr evidence04.ra -w - \
  | rasort -L0 -m bytes -s saddr daddr pkts bytes
           SrcAddr            DstAddr  TotPkts   TotBytes 
      10.42.42.253        10.42.42.25     5411     348608
      10.42.42.253        10.42.42.50     4070     259759
      10.42.42.253        10.42.42.56     4024     257038
       10.42.42.25        10.42.42.50       96       7548
       10.42.42.50     10.255.255.255       12       1104
       10.42.42.56       10.42.42.253        2        740
       10.42.42.50        10.42.42.25        4        416
       10.42.42.25     10.255.255.255        4        368
       10.42.42.50       10.42.42.253        1        190
       10.42.42.25       10.42.42.253        1         70

       Figure 5. Distribution of traffic by host


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
1. What was the IP address of Mr. X’s scanner?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Based on the distribution of the traffic (see figure 5), it seems that the IP
address of Mr X. is 10.42.253.253. This hypothesis is confirmed by the following
table, provided by pyScanXtract.py:

Src IP -> Dst IP	              # scans
----------------------------------------
10.42.42.25  -> 10.42.42.50	      12
10.42.42.253 -> 10.42.42.25	    3403
10.42.42.253 -> 10.42.42.50	    2018
10.42.42.253 -> 10.42.42.56	    2005

       Figure 6. Flows (src and dst IP addr. analysis)

Indeed, we notice that traffic is only originating from 10.42.42.253 and
10.42.42.25. In addition, 10.42.42.253 is the only address not beeing scanned, 
and 10.42.42.25 is scanned by 10.42.42.253. We deduce that 10.42.42.253 is the
IP address of Mr. X.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
2. For the FIRST port scan that Mr. X conducted, what type of port scan was it?
(Note: the scan consisted of thousands of packets.)
TCP SYN, TCP ACK, UDP, TCP CONNECT, TCP XMAS, TCP RST
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Still with the help of the Web Interface of pyScanXtract.py, we easily answer
this question. Indeed, from the Scan distribution characteristics view, with
filters on the source host (10.42.42.253) and on X-axis reference (time), we
identify three different blocks. By zooming on the graphs and using pyScanXtract
frames details, we obtain:

- first scan: from time 0s (frame #1) to 462.187269s (frame #6727)
    This scan seems to be globally composed of TCP SYN scans, but we see 2 
    TCP CONNECT() on frames #779 and #4381). When TCP CONNECT scan is used
    against CLOSED ports, there is no way to differentiate from a TCP SYN.
    Indeed, no ACK is sent from the attacker to complete the 3-way handshake.

- second scan: from time 543.230227s (frame #6728) to 597.071025s (frame
    #13532). This scan, launched 81 seconds after the end of the first one, has
    the same characteristics as the first scan (TCP CONNECT). We can observe 2
    open ports on frames #13527 and #13528.

- Third scan: from time 603.075410s (frame #13533) to the end.
    This scan, launched 6 seconds after the end of the second one, is composed
    of TCP CONNECT(), TCP NULL, TCP XMAS, TCP ACK, UDP and TCP custom
    scans. No real way to confirm if we also have TCP SYN.

The FIRST scan is a TCP CONNECT() scan.


Examples of scan techniques are given below:

TCP SYN
   Only SYN TCP flag is activated. This technique is also called "half
   connection" since it doens't complete the connection (standard 3-ways
   handshake: SYN - SYN/ACK - ACK). Indeed, only first 2 steps are used. It
   enables quick scans and raises the probability of not beeing detected by
   Intrusion Detection Systems (IDS) / Intrusion Prevention Systems (IPS).
   If the port is OPEN, target answers with a SYN/ACK to our SYN probe, whereas
   we should receive a RST/ACK if the port is CLOSED. 


TCP Connect
   This technique is also called Vanilla connect. Like for TCP SYN, it sends a
   TCP packet with SYN flag activated, but completes the connection by sending 
   a ACK to acknowledge the reception of a SYN/ACK in case the port is OPEN.
   The attacker then closes the connection by sending a RST. If the port is
   CLOSED, as for TCP SYN, target sends a RST/ACK.

   This type of portscan is USED in the shape of this puzzle.

   Example of an OPEN port (frames #4381, #4383, #4389 and #4394):

        +--------------+     +--------------+
        | 10.42.42.253 |     | 10.42.42.50  |
        +-------+------+     +-------+------+
                |                    |
#4381 42214/tcp | ------- SYN -----> | 135/tcp   seq=2994045278
#4383 42214/tcp | <---- SYN/ACK ---- | 135/tcp   seq=2938239898, ack=2994045279
#4389 42214/tcp | ------- ACK -----> | 135/tcp   seq=2994045279, ack=2938239899
#4394 42214/tcp | ----- RST/ACK ---> | 135/tcp   seq=2994045279, ack=2938239899

       Figure 8. TCP CONNECT against an OPEN port

   Again in Figure 8, we see a strange behavior concerning sequence numbers.
   Indeed, packet on frame #4394 uses the same seq. num. as for frame #4389.


TCP ACK
   This technique doesn't give indication about OPEN or CLOSED ports, but
   gives indication about rules implemented on an eventual firewall if any.
   Indeed, port will be considered as UNFILTERED if a RST is sent as a response
   to our ACK packet. On the other hand, if the port is FILTERED, we won't get
   response to our probe, or we will receive an ICMP port unreachable error
   (type 3, code 1, 2, 3, 9, 10 or 13). 

   This type of portscan is USED in the shape of this puzzle and shows only
   UNFILTERED ports.

   Example of an UNFILTERED port (frames #13606 and #13607):

         +--------------+     +--------------+
         | 10.42.42.253 |     | 10.42.42.50  |
         +-------+------+     +-------+------+
                 |                    |
#13606 36135/tcp | ------- ACK -----> | 135/tcp  seq=1405980936, ack=4233401440
#13607 36135/tcp | <------ RST ------ | 135/tcp  seq=4233401440

       Figure 9. TCP CONNECT against an UNFILTERED port


UDP
   In this case, a UDP packet without data is sent to the target. If the port
   is CLOSED, an ICMP port unreachable error (type 3, code 3) is received. If
   the port is FILTERED, an ICMP type 3 (destination unreachable), code 1 (host
   unreachable), 2 (protocol unreachable), 9 (network administratively
   prohibited), 10 (host administratively prohibited) or 13 (communication
   administratively prohibited) is received. It is unusal to receive an UDP
   packet as a response, but in this case, it means the port is OPEN. At least,
   if no response is received, even after many retransmissions, it means that
   the port is OPEN or FILTERED.

   This type of scan is used in the shape of this puzzle.

   Example of a CLOSED UDP port (frames #36581 and #36583)

         +--------------+     +--------------+
         | 10.42.42.253 |     | 10.42.42.25  |
         +-------+------+     +-------+------+
                 |                    |
#13581 36045/udp | ------- UDP -----> | 39217/udp
#13583 36045/udp | <----- ICMP ------ | 39217/udp
                     (type 3, code 3)

       Figure 10. UDP scan against a CLOSED port


TCP XMAS
   This type of scan consists of sending a packet with FIN, PUSH and URG TCP
   flags. A RST received from target indicates that port is CLOSED. If no
   response is received, it will indicate that the port is OPEN or FILTERED.
   And an ICMP type 3, code 1, 2, 3, 9, 10 or 13 will indicate a FILTERED port. 

   This type of scan is used in the shape of this puzzle.

   Example of a CLOSED port (frame #13600 and #13602):

         +--------------+     +--------------+
         | 10.42.42.253 |     | 10.42.42.25  |
         +-------+------+     +-------+------+
                 |                    |
#13600 36138/tcp | -- FIN/PSH/URG --> | 1/tcp  seq=1405980936, ack=4233401440
#13602 36138/tcp | <---- RST/ACK ---- | 1/tcp  seq=0,          ack=1405980936

       Figure 11. TCP XMAS against a CLOSED port

   The presence of a ack num on packet #13600 is really ambiguous since ACK
   flag is not present (XMAS scan). 


TCP RST
   Also called "inverse mapping", this technique enables to identify hosts that
   are up on a network. A packet with only TCP RST flag set is sent to target.
   In case this latest is up, no response is received. On the contrary, an ICMP
   host unreachable packet is received if there is no host at tested IP.

   This scan technique hasn't been found in the pcap file. All RST packets in
   the pcap file correspond to connection resets (e.g. SYN>SYN/ACK>ACK>RST).


Other scans have been found in the pcap file: TCP NULL and custom scans.

TCP NULL
   A TCP NULL scan consists of sending a packet with no TCP flag set. It
   sometimes enable to pass through firewalls which rules are based on standard
   flags settings. Interpretation of results is the same as for TCP XMAS scan.

   From the report generated by pyScanXtract.py, it is interesting to notice
   that we have 2 opposite interpretations (OPEN and CLOSED) for port 135/tcp
   on host 10.42.42.50. The details of the frames are shown below:

         +--------------+     +--------------+
         | 10.42.42.253 |     | 10.42.42.50  |
         +-------+------+     +-------+------+
                 |                    |
#13597 36133/tcp | ---- TCP NULL ---> | 135/tcp  seq=1405980936, ack=4233401440
#13598 36133/tcp | <---- RST/ACK ---- | 135/tcp  seq=0,          ack=1405980936

       Figure 12. TCP NULL against a Windows machine

   According to the response (RST/ACK), we should deduce that the port is
   CLOSED. But as we will demonstrate later, 10.42.42.50 is a Windows host, and
   as stated by Cisco Press (see below), Windows machines don't comply to RFC
   793 and send RST packet even if the port is OPEN.

"This is, of course, assuming that all hosts comply with RFC 793. In reality,
Windows hosts do not comply with this RFC. Subsequently, you cannot use a NULL
scan against a Windows machine to determine which ports are active. When a
Microsoft operating system receives a packet that has no flags set, it sends an
RST packet in response, regardless of whether the port is open. With all NULL
packets receiving an RST packet in response, you cannot differentiate open and
closed ports."

     (Source: http://www.ciscopress.com/articles/article.asp?p=469623&seqNum=3)


CUSTOM SCANS
   Two custom scans have been identified by pyScanXtract.py:

              +-----+-----+-----+-----+-----+-----+-----+-----+
  TCP flags   | ECE | CWR | URG | ACK | PSH | RST | SYN | FIN |
              +-----+-----+-----+-----+-----+-----+-----+-----+
frame #13590  |  x  |  x  |     |     |     |     |  x  |     |
              +-----+-----+-----+-----+-----+-----+-----+-----+
frame #13603  |     |     |  x  |     |  x  |     |  x  |  x  |
              +-----+-----+-----+-----+-----+-----+-----+-----+

       Figure 13. Custom TCP scans

   The scan on frame #13590 is like a TCP SYN but with additional ECE and CWR
   flags set. In this example, we get the same answer as for a TCP SYN (SYN/
   ACK). 

   The scan on frame #13603 is like a XMAS scan but with additional SYN flag
   set.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
3. What were the IP addresses of the targets Mr. X discovered?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Exploiting the results of pyScanXtractpy (see figure 6), we conclude that
discovered targets are:

- 10.42.42.25
- 10.42.42.50
- 10.42.42.56


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
4. What was the MAC address of the Apple system he found?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
From the "Discovered hosts" section of the generated report (pyScanXtract.py),
we see following result:

+--------------+-------------------+----------------------------------------+
| Host         | MAC addr.         | Vendor                                 |
+--------------+-------------------+----------------------------------------+
| 10.42.42.25  | 00:16:cb:92:6e:dc | Apple Computer                         |
| 10.42.42.50  | 70:5a:b6:51:d7:b2 | COMPAL INFORMATION (KUNSHAN) CO., LTD. |
| 10.42.42.56  | 00:26:22:cb:1e:79 | COMPAL INFORMATION (KUNSHAN) CO., LTD. |
| 10.42.42.253 | 00:23:8b:82:1f:4a | Quanta Computer Inc.                   |
+--------------+-------------------+----------------------------------------+

       Figure 14. Discovered hosts (provided by pyScanXtract.py)

According to the packets from 10.42.42.25 (extraction of ethernet header data), 
pyScanXtract.py detects that associated physical address is 00:16:cb:92:6e:dc.
Exploiting "IEEE OUI and Company_id Assignments" database available at
http://standards.ieee.org/regauth/oui/oui.txt, it is possible to obtain the name
of the vendor: Apple Computer.

We could have used passive OS fingerprinting with p0f to validate our hypothesis
but as Figure 15 shows, our host is not recognized (UNKNOWN).

$ p0f -M -N -l -q -s evidence04.pcap | grep '^10.42.42.25:'
[+] End of input file.
10.42.42.25:49260 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49261 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49262 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49263 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49264 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49265 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49266 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49267 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49268 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49269 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49270 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 
10.42.42.25:49271 - UNKNOWN [65535:64:1:64:M1460,N,W3,N,N,T,S,E:P:?:?] (up:
   2462 hrs) 

       Figure 15. Passive OS fingerprint with p0f

We will conclude that the Apple system has 10.42.42.25 for IP and that his mac
address is: 00:16:cb:92:6e:dc.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
5. What was the IP address of the Windows system he found?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
According to pyScanXtract.py report, host 10.42.42.50 is the only host that has
OPEN ports. These latest are: 135/tcp (DCOM Service Control Manager) and 139/tcp
(NETBIOS Session Service). According to GRC's website
(http://www.grc.com/port_135.htm and http://www.grc.com/port_139.htm), these
ports are common ports for Windows machines:

a) First indication: OPEN ports

135/tcp:
"Microsoft's DCOM (Distributed, i.e. networked, COM) Service Control Manager
(also known as the RPC Endpoint Mapper) uses this port in a manner similar to
SUN's UNIX use of port 111. The SCM server running on the user's computer opens
port 135 and listens for incoming requests from clients wishing to locate the
ports where DCOM services can be found on that machine."

139/tcp:
"TCP NetBIOS connections are made over this port, usually with Windows machines
but also with any other system running Samba (SMB). These TCP connections form
'NetBIOS sessions' to support connection oriented file sharing activities."

b) Second indication: Behavior against our TCP NULL scan:

On question 2, we have explained that Windows machine don't comply with RFC 793
in case of a NULL SCAN.
The fact that another scan presumes that port 135/tcp is OPEN seems to confirm
our presumption about the Windows machine.

c) Third indication: Distribution of IP-IDs:

As we can see from pyScan Web Interface against host 10.42.42.50, IP-ID numbers
are growing regularly, which is typical from Windows systems.

d) Fourth indication: IP-stack:

IP stacks give us clues to passively fingerprint the OS. Honeynet has published 
a database (http://old.honeynet.org/papers/finger/traces.txt), enabling to
determine OS, based on certain criteria (ttl, window, tos, ...). Using this
database, we see that most (all?) of the packets originating from 10.42.42.50
have following caracteritics:

- TTL: 128
- ToS: 0

Although this information is not very reliable, it is an additional indication
that could confirm the host is a Windows machine.


These 4 indications seem to confirm our presumption: Windows machine has IP
10.42.42.50.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
6. What TCP ports were open on the Windows system? (Please list the decimal
numbers from lowest to highest.)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
PyScanXtract.py shows two OPEN TCP ports on the "presumed" Windows machine
(10.42.42.50):

- 135/tcp
- 139/tcp


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
7. X-TRA CREDIT (You don’t have to answer this, but you get super bonus points
if you do): What was the name of the tool Mr. X used to port scan? How can you
tell? Can you reconstruct the output from the tool, roughly the way Mr. X would
have seen it?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
It exists many port scanners, for all operating systems. Here is a list of some
of them:
- Nmap (http://nmap.org/download.html)
- Queso (http://tools.l0t3k.net/FingerPrinting/queso-980922.tar.gz)
- Xprobe2 (http://xprobe.sourceforge.net/)
- Ettercap (http://ettercap.sourceforge.net/)
- hping (http://www.hping.org/download.php)
- THC AMAP (http://freeworld.thc.org/thc-amap/)
- Angry IP Scanner (http://www.angryip.org/w/Download)
- Radmin Advanced Port Scanner
    (http://www.radmin.com/products/utilities/portscanner.php)
- Foundstone scanline
    (http://www.foundstone.com/us/resources/proddesc/scanline.htm)

and also manual crafting tools:

- Scapy (http://www.secdev.org/projects/scapy/)
- Nemesis (http://nemesis.sourceforge.net/)

and also vunerability scanners:

- Nessus (http://www.nessus.org/download/)
- SATAN (http://www.porcupine.org/satan/)
- OpenVAS (http://www.openvas.org/)
- LANguard Port Scanner (http://www.gfi.com/lannetscan)

Information we know:

Information #1: Scan characteristics
--------------------------------------
pyScan Web Interface (Scan distribution characteristics module) provides these
information:

TCP ports:
- The same TCP ports are scanned on the 3 targets
- 1000 distinct TCP ports are scanned between 1/tcp and 65389/tcp 
- Some TCP ports are not scanned (2/tcp, 5/tcp, 8/tcp, 10/tcp, ...)
- TCP ports are randomly scanned (See figure 15 and graph "Distribution of TCP
dport over time")
- Scan is highly concentrated on TCP ports < 10000 (see "Distribution of TCP
dport over time")
- Use of a random sport (between 32770 and 60996) on the first scan and a fixed
value (36020/tcp) for the second one.
- Sequence numbers are regularly incremented (see graph "Distribution of seq
over time") for the first scan and is fixed for the second scan.
- The second scan uses all TTL values between 37 and 59

UDP ports:
- Only 1 UDP port is scanned one each target and it is different for each.

Information #2: Scan techniques
--------------------------------------
We can see common scan techniques :
- TCP SYN
- TCP Connect()
- TCP ACK
- TCP XMAS
- UDP
- TCP NULL
And 2 custom TCP scans:
- TCP XMAS-like scan but with SYN flag set
- TCP SYN-like scan but with ECE and CWR flags set
There is no:
- TCP RST scan

Information #3: Host 10.42.42.25
--------------------------------------
- It has been demonstrated that Mr. X's IP is 10.42.42.253. Nevertheless, some
scans are realized by the Apple machine, which IP is 10.42.42.25.

In addition, Host 10.42.42.25:
- only scans 10.42.42.50 (Windows machine) on port 139/tcp
- only uses TCP Connect() scan technique
- combines "scans" with NBSS requests (Windows shares)

These "scans" are not "decoys" or "idle scan" from 10.42.42.253. Actually, they
are attemps from 10.42.42.25 to connect to SMB shares on 10.42.42.50.

Information #4: Strange packets
--------------------------------------
Some packets originating from the source (10.42.42.253 or 10.42.42.25) seem to
be manually crafted. Indeed, many packets have the same criteria:
- Use of same seq num for many packets
- Strange responses of the attacker (presence of ack num. while not activating
ACK flag, no ack num. while activating ACK flag, aborting of connection with
FIN/ACK instead of RST/ACK, ...) 

Example of strange behaviors:

       10.42.42.25.49260     10.42.42.50.139
        -------+-------      -------+-------
               |                    |
6115     49260 | ------- SYN -----> | 139   seq=1142317008
6116     49260 | <---- SYN/ACK ---- | 139   seq=1436240505  ack=1436247009
6117     49260 | ------- ACK -----> | 139   seq=1436247009  ack=1436240506
               |                    |
6120     49260 | ------- ACK -----> | 139   seq=1142317081  ack=1436240512 [1]
6121     49260 | ----- FIN/ACK ---> | 139   seq=1142317081  ack=1436240512 [2]
6122     49260 | <------ ACK ------ | 139   seq=1436240512  ack=1142317082
               |                    |
6962     36020 | ------- SYN -----> | 139   seq=289350607
6973     36020 | <---- SYN/ACK ---- | 139   seq=1604535773  ack=289350608
6978     36020 | ------- ACK -----> | 139   seq=289350608   ack=[3]

   [1] Why ack (connection init)?
   [2] Why FIN-ACK with same seq?
   [3] ack num not specified while using ACK flag

       Figure 16. Example of strange packets with additional/missing info.

Information #5: Capture length
--------------------------------------
Capture (scan) is about 10 minutes long and contains 7438 scans, that makes:
7435 scans / 606 sec = 12.3 scans / sec. Either a tool has been used for this
scan or a script has been written.


I haven't tested all port scanners to check if I could find similar signatures,
but with all these information, here are our assessments:

Queso:
----------------
- The presence of a scan with ECE/CWR/SYN TCP flags on frame 13590 could have
been a signature of Queso. Indeed, Queso uses such packets to fingerprint the
remote OS. 
- Nevertheless, it is explained in the documentation that Queso "sends 7 packets
(0-6), and compares the responses with the config file, where the different OSes
are described, in a response-based way to each packet (differentiated by the dst
port -my port)", which doesn't seem to be the case (see "Scan types distribution
over time")
- In addition, Queso seems to choose random sport, between 4000/tcp and
65000/tcp. Based on the results of graph "Distribution of TCP sport over time",
sport varies from 32770/tcp and 60996/tcp, with a large concentration (40%) on
36020/tcp.
- At least Queso seems to send far less packets than in the pcap file. 

The hypothesis of Queso seems to be discarded.

Nmap:
----------------
- Nmap is compatible with *nix systems. According to the IPID distribution from
pyScanXtract Web Interface, we see randomly generated IPIDs, giving indications
on the attacker's OS (probably *nix).
- Nmap seems to first check that scanned host is up with an ICMP request, which
is not the case in the shape of this puzzle. Nevertheless, this is Nmap's
default behavior, and -PN parameter enables to skip this step.
- Nmap is a very (the most?) complete port scanner, offering, among others, TCP
SYN, TCP CONNECT, TCP ACK, TCP NULL, TCP XMAS, UDP and also custom scans through
the --scanflags option.
- Nmap also enables to choose sport (--sourceport option)
- Nmap uses different ttl values (second scan) to identify remote OS
- Nmap offers the possibility of selecting tested ports, via the --top-ports
option

It seems that the Nmap could be a possibility.

XProbe:
----------------
- It is stated in the documentation that:
   "As it's been noted, the number of datagrams we need to send and
   receive in order to remotely fingerprint a targeted machine with
   ICMP based probes is small. Very small. In fact we can send one
   datagram and receive one reply and this will help us identify up to
   eight different operating systems (or classes of operating systems).
   The maximum datagrams which our tool will use at the current stage
   of development, is four. This is the same number of replies we will
   need to analyse. This makes ICMP based fingerprinting very
   time-efficient."
It doesn't fit with the large amount of packets sent in the pcap file.

The hypothesis of XProbe seems to be discarded.

THC AMAP
----------------
- THC AMAP doesn't seem to use over scan techniques than TCP SYN ad TCP CONNECT.
This hypothesis seems to be discarded.


As a conclusion, I think the most probable hypothesis is Nmap. Indeed, this port
scanner is a very (the most?) complete port scanner that offers the possibility
of controlling almost all parameters of the scan. In addition, typical
signatures (scan types, fixed sport, distribution of TTL, ...) have been found
thanks to pyScanXtract Web Interface.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
8. Conclusions
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This puzzle was really my favorite! It was really exciting to make the necessary
reverse-engineering to reproduce the output Mr.X. has seen. It really requires
strong knowledge on TCP/IP to make it and understand the exceptions.

I had the idea of initially developing only a Python script (my favorite
language ;-), but found appropriate to use the data I had in the database to
display other useful information and draw graphs. I am certain that pyScanXtract
is a useful tool that I will use for future uses. That's why I have published it
on Google Code (http://pyscanxtract.googlecode.com/files/pyscan.tar.gz) and
identified some improvements I have reported in the README file.

At least, I really found interesting to analyse distribution graphs, and I
promised myself to write a paper on it. I will let you review it when it will be
done, and will prepare an exercise for you guys :-P


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
10. Scripts
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
10.1. Description
---------------------
pyScanXtract.py is a script, written in Python, created in the shape of
forensicscontest (puzzle #4). It enables to analyse a pcap file and to produce
statistics about potential portscans. Here is a list of available statistics:

   - General information: pcap md5sum, number of packets, capture duration, ...
   - Discovered hosts: list of identified hosts with mac addr. and vendors
   - Scan types: List of detected scan types (TCP SYN, CONNECT, XMAS, UDP, ...)
   - Hosts/targets: List of flows between attackers and targets
   - IP options (utilisation of fragmentation)
   - TCP and UDP ports status (open, closed, filtered, unfiltered)
   - Detailed frames: show 
   - Distributions by frame/time of some parameters (sport, IP length, IPID,
     TTL, dport, seq, ack, ...)

10.2. README/INSTALLATION file
---------------------
ARCHITECTURE

   pyScanXtract.py works with conjunction of PHP/MySQL. The following figure
   explains how it works:

          +---------+      +--------------+
      [1] | capture |      | OUI.txt file | [3]
          +---------+      +--------------+
          pcap  |                  |  Used to identify 
          file  +-------+----------+  mac's vendors
                        |
                        |
                        V
                 +--------------+ [2]
                 | pyScanXtract | ------------+
                 +--------------+             |
                        |  Groups and logs    |
                        |  events in DB       |          General info.
                        V                     V  [5]     Discovered hosts
                      _____            +-------------+   Scan types
           MySQL     (_____)           | report.html |   Hosts/targets
           database  (_____)           +-------------+   Ports status
             [4]     (_____)                  |            (tcp/udp)
                        |                     |
                        |              +------+--------+---------------+
                        |              |               |               |
                        |         [6]  V          [7]  V           [8] V
                        |       +-----------+   +--------------+  +--------+
                        |       | Host IPID |   | Distribution |  | Frames |
                        |       | distrib.  |   |   analysis   |  | detail |
                        |       +-----------+   +--------------+  +--------+
                        |             ^                 ^              ^
                        +-------------+-----------------+--------------+

       [1] A capture is realized with a tool (e.g. tcpdump) capable of
           providing a pcap file.
       [2] pyScanXtract.py analyses provided pcap file. It looks for mac's
           vendor [3] and stores results in a MySQL database [4].
       [5] It generates a html report. From this report, you can access to Web
           Interfaces (WI) [6][7][8] if you have Apache/PHP installed.
       [6] From the WI, you can access [6] Host IPID distribution by clicking
           on a host from the "Discovered Hosts" section (report.html). 
       [7] The distribution analysis is also available by clicking on "Scan
           distrib. charact." from the "Scan types" section (report.html)
       [8] From report.html, if you click on a scan type (Scan types section)
           or on a port status (TCP and UDP ports analysis sections), the list
           of appropriate frames are displayed.

INSTALLATION

   If you don't plan to use the Web Interfaces, you just need a functional
   MySQL database, a valid oui.txt file (can be downloaded from IEEE Standards
   Association: http://standards.ieee.org/regauth/oui/oui.txt) and required
   Python libraries:
       - optparse
       - sys
       - shutil
       - pcapy
       - impacket
       - os
       - MySQLdb
       - datetime
       - struct
       - hashlib

   Use provided pyscan.sql script to initialize your database.
   If needed, modify following variables from line 38 in pyScanXtract.py:
       DBHOST = '127.0.0.1'    # Host to connect to
       DBUSER = 'pyscan'       # User name to connect to database
       DBPSWD = 'pyscan'       # Password to connect to database
       DBNAME = 'pyscan'       # Database name

   To use Web Interfaces, you will also have to modify line 42:
       BASEWB = 'http://localhost/pyscan' # Base path for pyscan web interface
                                          # Don't put / at the end of the path
                                          # e.g. http://localhost/pyscan

   If you have multiple websites on Apache, you can add a virtual directory in
   your httpd.conf (depending on your configuration, it is sometimes in
   extra/httpd-vhosts.conf). Refer to following URL for more information:
       http://httpd.apache.org/docs/1.3/vhosts/examples.html


UTILISATION

   pyScanXtract.py can be called with following basic syntax:

       $ ./pyScanXtract -r evidence04.pcap

   Here are available options:

    -h, --help
       Show help message and exit

    -r <PCAP_FILE>, --read-file=<PCAP_FILE>
       Capture file to process (pcap format)

    -o <OUTPUT_DIRECTORY>, --output=<OUTPUT_DIRECTORY>
       Reporting directory (default: ./report/) where report.html will be
       written

    -f, --force
       Force overwriting of files. Use if an already existing report directory
       exists

    -v <OUI_FILE>, --vendor-database=<OUI_FILE>
       Vendor database (default: ./oui.txt). This file can be downloaded from
       http://standards.ieee.org/regauth/oui/oui.txt.

    -d, --dont-purge
       Don't purge existing data in the database. Default behavior is to first
       TRUNCATE pyscan table.

   Once finished, pyScanXtract.py will produce a report in HTML format,
   available in report directory. Open it in your browser.


10.3. Improvements for the next version
---------------------
- The script is based on a flow analysis. If many scans use the same combination
of src:sport/dst:dport, they may not be detected.
- Window/Maimon scans are not detected by the script
- Script is quite slow.


10.4. Source code
---------------------
| Note: Only pyScanXtract.py has been reported here, but the entire source code,
| including Web Interfaces has been published on Google Code:
|
|    http://pyscanxtract.googlecode.com/files/pyscan.tar.gz

#!/usr/bin/env python

# Developed by Sebastien DAMAYE (aldeid.com) in the shape of forensicscontest.com
# (Puzzle #4: The Curious Mr. X). Feel free to modify.
# Last update: 2010-02-14 21:42, Version: 1.0
#
# This script uses a MySQL connection. Please first make sure that you
# have created necessary objects and privileges, using following SQL
# script provided with pyScanXtract package (pyscan.sql)
# (Notice that DROP priv is required to TRUNCATE table unless
# --dont-purge option is used)
#
# Check that MySQL is started before you launch the script.
#
# In addition, this script needs a OUI file to detect mac vendors.
# The complete updated database (txt format) can be downloaded from
# http://standards.ieee.org/regauth/oui/oui.txt
#

from optparse import OptionParser
import sys
import shutil
import pcapy
import impacket.ImpactDecoder as Decoders
import impacket.ImpactPacket as Packets
import os.path
import os
import MySQLdb
import datetime
import struct
import hashlib

class PyScan:

   ###
   # You can modify these parameters to fit with your configuration
   #
   DBHOST = '127.0.0.1'    # Host to connect to
   DBUSER = 'pyscan'       # User name to connect to database
   DBPSWD = 'pyscan'       # Password to connect to database
   DBNAME = 'pyscan'       # Database name
   BASEWB = 'http://localhost/pyscan' # Base path for pyscan web interface
                                      # Don't put / at the end of the path
                                      # e.g. http://localhost/pyscan

   def __init__(self, pcapfile, reportpath="./report", ouifile="./oui.txt", dont_purge=None):
       """Performs some checks and connects to database
       """
       # Checks if pcap file exists
       assert pcapfile
       self.pcapfile = pcapfile
       if not os.path.exists(pcapfile):
           raise TypeError("Pcap file not found. Please check location.")

       self.reportpath = reportpath
       if not os.path.exists(self.reportpath):
           os.makedirs(self.reportpath)

       pc = open(pcapfile, mode='rb')
       pcheader = pc.read(24)

       # Read the first 4 bytes for the magic number, determine endianness
       pcmagicnum = struct.unpack("I", pcheader[0:4])[0]
       if pcmagicnum != 0xd4c3b2a1:
           # Little endian
           pcendflag = "<"
       elif magicnum == 0xa1b2c3d4:
           # Big endign
           pcendflag = ">"
       else:
           raise Exception('Specified file is not a libpcap capture')

       pcaph = struct.unpack("%sIHHIIII"%pcendflag, pcheader)
       if pcaph[1] != 2 and pcaph[2] != 4 \
               and pcaph[3] != 0 and pcaph[4] != 0 \
               and pcaph[5] != 65535:
           raise Exception('Unsupported pcap header format or version')

       # Checks if OUI file exists
       assert ouifile
       if not os.path.exists(ouifile):
           raise TypeError("OUI file not found. Please check location."
           + "OUI file can be downloaded from http://standards.ieee.org/regauth/oui/oui.txt")
       self.ouifile = ouifile

       # Connection to MySQL database
       try:
           self.conn = MySQLdb.connect (
               host = self.DBHOST,
               user = self.DBUSER,
               passwd = self.DBPSWD,
               db = self.DBNAME)
           self.cursor = self.conn.cursor ()
       except MySQLdb.Error, e:
           print "Error %d: %s" % (e.args[0], e.args[1])
           sys.exit (1)

       # Option: purge table
       if not dont_purge:
           self.cursor.execute ("TRUNCATE TABLE pyscan")

       # Initial list of hosts
       self.hosts = {}

   def md5sum(self, myfile):
       m = hashlib.md5()
       f = file(myfile, 'rb')
       while True:
           t = f.read(1024)
           if len(t) == 0:
               break
           m.update(t)
       return m.hexdigest()

   def writefile(self, f, content):
       """Dump content in a file
       """
       obFile = open(os.path.join(self.reportpath, f), 'a')
       obFile.write(content)
       obFile.close()

   def decodemac(self, mac):
       """Decode mac address with hex notation
       """
       m = ''
       for i in mac:
           t = "%x" % i
           if len(t)==1:
               t = '0'+t
           m=m+":"+t
       return m[1:]

   def countnumberframes(self):
       """Returns number of frames contained in pcap file
       """
       f = pcapy.open_offline(self.pcapfile)
       c = 0
       while f.next()[1]!="":
           c+=1
       return c

   def macvendor(self, mac):
       """Finds mac vendor in OUI file from 6 first cars of mac addr
       """
       #Only 6 first cars are necessary in upper cap
       pref = mac.replace(':', '')[:6].upper()
       f = open(self.ouifile, 'r')
       while 1:
               li = f.readline()
               if li=='':
                       break
               if pref in li:
                       return (li.replace('\n', '').replace('\t', ''))[20:]
                       break
       f.close()

   def getframefromflow(self, ip_src, sport, ip_dst, dport, proto):
       """Checks whether flow already exists in DB
          Returns frame number or zero
       """
       self.cursor.execute ("""
           SELECT snd_frame, ip_src FROM pyscan
           WHERE (ip_src=%s AND ip_dst=%s AND sport=%s AND dport=%s)
           OR (ip_src=%s AND ip_dst=%s AND sport=%s AND dport=%s)
           AND proto=%s
           """, (ip_src, ip_dst, sport, dport,
               ip_dst, ip_src, dport, sport, proto))
       row = self.cursor.fetchone()
       if row == None:
           row = [0, 0]
       return row

   def main(self):
       """Stores and reassemble frames in MySQL database.
          IP / TCP / UDP / ICMP protocols are handled in this version
       """
       print "Analysing pcap file..."
       reader = pcapy.open_offline(self.pcapfile)

       # Decoders for ethernet, IP, TCP, UDP and ICMP
       eth_decoder = Decoders.EthDecoder()
       ip_decoder = Decoders.IPDecoder()
       tcp_decoder = Decoders.TCPDecoder()
       udp_decoder = Decoders.UDPDecoder()
       icmp_decoder = Decoders.ICMPDecoder()
       ip_icmp_decoder = Decoders.IPDecoderForICMP()

       # Initial frame counter
       frame = 0
       nFrames = self.countnumberframes()

       (header, payload) = reader.next()

       while frame < nFrames:

           frame+=1
           # Timestamp
           (ts, usec) = header.getts()

           if frame==1:
               # First frame = reference (0 second)
               reftime = float(str(ts)+"."+str(usec))
               tm = 0
               # First Frame TimeStamp
               self.ffts = datetime.datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
           else:
               # Number of seconds since beginning of capture
               tm = float(str(ts)+"."+str(usec)) - reftime
               # Last frame time stamp
               if frame == nFrames:
                   self.lfts = datetime.datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")

           # Progress bar
           p = int(float(frame)/nFrames*50)
           sys.stdout.write("\r["+"#"*p+" "*(50-p)+"] "+str(p*2)+"% ("
               +str(frame)+"/"+str(nFrames)+")")
           sys.stdout.flush()

           #---------------------------------
           #           ETHERNET LAYER
           #---------------------------------
           ethernet = eth_decoder.decode(payload)

           #---------------------------------
           #           IP PACKETS
           #---------------------------------
           if ethernet.get_ether_type() == Packets.IP.ethertype:

               ip = ip_decoder.decode(payload[ethernet.get_header_size():])

               # IP header information
               ip_ihl    = ip.get_ip_hl()
               ip_tos    = ip.get_ip_tos()
               ip_len    = ip.get_ip_len()
               ip_id     = ip.get_ip_id()
               ip_ttl    = ip.get_ip_ttl()
               if ip.get_ip_df() > 0:
                   df = '1'
               else:
                   df = '0'
               if ip.get_ip_mf() > 0:
                   mf = '1'
               else:
                   mf = '0'
               ip_flags  = str(ip.get_ip_rf()) + df + mf
               ip_src    = ip.get_ip_src()
               ip_dst    = ip.get_ip_dst()

               #---------------------------------
               #           TCP PACKETS
               #---------------------------------
               if ip.get_ip_p() == Packets.TCP.protocol:
                   tcp = tcp_decoder.decode(
                       payload[ethernet.get_header_size()+ip.get_header_size():])

                   # New hosts are added to the list of hosts
                   if self.hosts.keys().count(ip_src)==0:
                       self.hosts[ip_src] = self.decodemac(ethernet.get_ether_shost())
                   if self.hosts.keys().count(ip_dst)==0:
                       self.hosts[ip_dst] = self.decodemac(ethernet.get_ether_dhost())

                   # TCP header information
                   tcp_sport = tcp.get_th_sport()
                   tcp_dport = tcp.get_th_dport()
                   tcp_ece   = tcp.get_ECE()
                   tcp_cwr   = tcp.get_CWR()
                   tcp_urg   = tcp.get_URG()
                   tcp_ack   = tcp.get_ACK()
                   tcp_psh   = tcp.get_PSH()
                   tcp_rst   = tcp.get_RST()
                   tcp_syn   = tcp.get_SYN()
                   tcp_fin   = tcp.get_FIN()
                   seq       = tcp.get_th_seq()
                   ack       = tcp.get_th_ack()

                   # Checks if src:sport->dst:dport or dst:dport->src:sport
                   # doesn't already exist in DB
                   arrFrame = self.getframefromflow(ip_src, tcp_sport, ip_dst, tcp_dport, 'TCP')
                   if arrFrame[0]==0:
                       # Frame hasn't been found: new flow
                       self.cursor.execute ("""
                           INSERT INTO pyscan(ts, snd_frame, ip_ihl, ip_tos,
                             ip_len, ip_id, ip_flags, ip_ttl, proto, ip_src,
                             ip_dst, sport, dport, snd_tcp_ece, snd_tcp_cwr,
                             snd_tcp_urg, snd_tcp_ack, snd_tcp_psh, snd_tcp_rst,
                             snd_tcp_syn, snd_tcp_fin, seq, ack
                           ) VALUES (
                             %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
                             %s, %s, %s, %s, %s, %s, %s, %s, %s, %s
                           )
                           """, (tm, frame, ip_ihl, ip_tos, ip_len, ip_id,
                           ip_flags, ip_ttl, 'TCP', ip_src, ip_dst, tcp_sport,
                           tcp_dport, tcp_ece, tcp_cwr, tcp_urg, tcp_ack,
                           tcp_psh, tcp_rst, tcp_syn, tcp_fin, seq, ack))
                   else:
                       if arrFrame[1]==ip_src:
                           #Attacker->Target: update flow (request)
                           if tcp_rst==1:
                               # RST conn
                               self.cursor.execute ("""
                                   UPDATE pyscan SET rst_conn=1
                                   WHERE snd_frame = %s
                                   """, (arrFrame[0]))
                           else:
                               #ACK conn
                               self.cursor.execute ("""
                                   UPDATE pyscan SET ack_conn=1
                                   WHERE snd_frame = %s
                                   """, (arrFrame[0]))
                       else:
                           # Target->Attacker: update flow (reply)
                           # "snd_frame IS NULL" clause is added to
                           # prevent from updating an already existing answer
                           self.cursor.execute ("""
                               UPDATE pyscan SET
                                   rcv_frame = %s,
                                   rcv_tcp_urg = %s,
                                   rcv_tcp_ack = %s,
                                   rcv_tcp_psh = %s,
                                   rcv_tcp_rst = %s,
                                   rcv_tcp_syn = %s,
                                   rcv_tcp_fin = %s,
                                   rcv_ip_id=%s,
                                   ack = %s
                               WHERE snd_frame = %s
                               AND   rcv_frame IS NULL
                               """, (frame,
                                   tcp_urg, tcp_ack, tcp_psh, tcp_rst, tcp_syn,
                                   tcp_fin, ip_id, ack, arrFrame[0]))

               #---------------------------------
               #           UDP PACKETS
               #---------------------------------
               if ip.get_ip_p() == Packets.UDP.protocol:
                   udp = udp_decoder.decode(
                       payload[ethernet.get_header_size()+ip.get_header_size():])

                   # UDP header information
                   udp_sport = udp.get_uh_sport()
                   udp_dport = udp.get_uh_dport()
                   udp_len   = udp.get_size()

                   # Any other way to ignore NBNS (NetBIOS Name Service)
                   # packets with Impacket?
                   if udp_len > 70:
                       # Log SENT UDP PACKETS TO TARGET in DB
                       arrFrame = self.getframefromflow(ip_src, udp_sport, ip_dst, udp_dport, 'UDP')
                       if arrFrame[0]==0:
                           # Frame hasn't been found: new flow
                           self.cursor.execute ("""
                               INSERT INTO pyscan(ts, snd_frame, proto, ip_src,
                                 ip_dst, sport, dport
                               ) VALUES (
                                 %s, %s, %s, %s, %s, %s, %s
                               )
                               """, (tm, frame, 'UDP', ip_src, ip_dst,
                               udp_sport, udp_dport))

               #---------------------------------
               #           ICMP PACKETS
               #---------------------------------
               if ip.get_ip_p() == Packets.ICMP.protocol:
                   icmp = icmp_decoder.decode(
                       payload[ethernet.get_header_size()+ip.get_header_size():])

                   # ICMP header information
                   icmp_type = icmp.get_icmp_type()
                   icmp_code = icmp.get_icmp_code()

                   # ICMP echo request/reply are not kept in database
                   if icmp_type != 8 and icmp_type !=0:

                       # Special IP decoder for ICMP
                       ip_icmp = ip_icmp_decoder.decode(
                           payload[ethernet.get_header_size()+ip.get_header_size()
                           +icmp.get_header_size():])
                       # UDP decoder for IP in ICMP
                       udp_ip_icmp = udp_decoder.decode(
                           payload[ethernet.get_header_size()+ip.get_header_size()
                           +icmp.get_header_size()+ip_icmp.get_header_size():])

                       # IP header information in ICMP
                       ip_icmp_src = ip_icmp.get_ip_src()
                       ip_icmp_dst = ip_icmp.get_ip_dst()

                       # UDP header information in IP in ICMP
                       udp_ip_icmp_sport = udp_ip_icmp.get_uh_sport()
                       udp_ip_icmp_dport = udp_ip_icmp.get_uh_dport()

                       # Looks for frame number that has required criteria
                       fn = self.getframefromflow(ip_icmp_src, udp_ip_icmp_sport,
                           ip_icmp_dst, udp_ip_icmp_dport, 'UDP')[0]

                       # Updates flow in database
                       self.cursor.execute ("""
                           UPDATE pyscan
                           SET rcv_frame=%s, icmp_err=%s
                           WHERE snd_frame=%s
                           """, (frame, icmp_code, fn))

               (header, payload) = reader.next()


   def updatescantypes(self):
       """Update database by guessing scan types
          for each detected scan
       """
       print "\nDetermining scan types..."

       #---------------------------------
       #           SCAN TYPES
       #---------------------------------

       # TCP SYN (only TCP SYN flag activated and no ack.)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP SYN'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000010'
           AND    ack_conn IS NULL
           """)
       # TCP CONNECT (same as TCP SYN but ack.)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP CONNECT'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000010'
           AND    ack_conn IS NOT NULL
           """)
       # TCP NULL (no TCP flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP NULL'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000000'
           """)
       # TCP FIN (only TCP FIN flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP FIN'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000001'
           """)
       # TCP XMAS (only TCP FIN/PSH/URG flags activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP XMAS'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00101001'
           """)
       # TCP ACK (only TCP ACK flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP ACK'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00010000'
           """)
       # TCP RST (only TCP RST flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP RST'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000100'
           """)
       # ALL OTHER TCP SCANS (custom flags)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP CUSTOM SCAN'
           WHERE  proto='TCP'
           AND    scan_type IS NULL
           """)
       # UDP SCANS
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='UDP'
           WHERE  proto='UDP'
           """)

       #---------------------------------
       #           PORTS STATE
       #---------------------------------
       print "Determining ports state..."

       #----- TCP SYN -----

       # TCP SYN - OPEN (SYN/ACK from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN'
           WHERE  scan_type='TCP SYN'
           AND    rcv_tcp_ack = 1
           AND    rcv_tcp_syn = 1
           """)
       # TCP SYN - CLOSED (RST from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='CLOSED'
           WHERE  scan_type='TCP SYN'
           AND    rcv_tcp_rst = 1
           """)
       # TCP SYN - FILTERED (no answer from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  scan_type='TCP SYN'
           AND    rcv_frame IS NULL
           """)

       #----- TCP CONNECT -----

       # TCP CONNECT - OPEN (SYN/ACK from source)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN'
           WHERE  scan_type='TCP CONNECT'
           AND    rcv_tcp_syn = 1
           AND    rcv_tcp_ack = 1
           """)

       #----- TCP NULL/FIN/XMAS -----

       # TCP NULL/FIN/XMAS - OPEN|FILTERED (no response from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN|FILTERED'
           WHERE  scan_type IN ('TCP NULL', 'TCP FIN', 'TCP XMAS')
           AND    rcv_frame IS NULL
           """)
       # TCP NULL/FIN/XMAS - CLOSED (RST received from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='CLOSED'
           WHERE  scan_type IN ('TCP NULL', 'TCP FIN', 'TCP XMAS')
           AND    rcv_tcp_rst = 1
           """)
       # TCP NULL/FIN/XMAS - FILTERED (ICMP unreachable error)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  scan_type IN ('TCP NULL', 'TCP FIN', 'TCP XMAS')
           AND    icmp_err IN (1, 2, 3, 9, 10, 13)
           """)

       #----- TCP ACK -----

       # TCP ACK - UNFILTERED (RST received from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='UNFILTERED'
           WHERE  scan_type='TCP ACK'
           AND    rcv_tcp_rst = 1
           """)
       # TCP ACK - FILTERED (no response from target OR ICMP unreachable err)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  (scan_type='TCP ACK'
           AND    rcv_frame IS NULL)
           OR     icmp_err IN (1, 2, 3, 9, 10, 13)
           """)

       #----- UDP SCAN -----
       # UDP SCANS - OPEN|FILTERED (no response from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN|FILTERED'
           WHERE  scan_type='UDP'
           AND    rcv_frame IS NULL
           """)
       # UDP SCANS - CLOSED (ICMP port unreachable)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='CLOSED'
           WHERE  scan_type='UDP'
           AND    icmp_err=3
           """)
       # UDP SCANS - FILTERED (type 3, code 1, 2, 9, 10 or 13)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  scan_type='UDP'
           AND    icmp_err IN (1, 2, 9, 10, 13)
           """)

       #----- ALL OTHERS: UNDEFINED -----
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='UNDEFINED'
           WHERE  state IS NULL
           """)


   def exportresults(self):
       print "Generating report..."
       self.writefile("report.html", """<html><head><style>
           h1           { font-size:11pt; padding:2px; }
           h2           { font-size:9pt; padding:2px; }
           body, td, th { font-size:8pt; }
           table        { border-collapse:collapse; border:solid 2px #cccccc; }
           th, td       { border:solid 1px #dddddd;padding:2px;}
           th           { background:#eeeeee; }
           div          { padding:10px; }
           </style></head><body>\n""")
       #===== Index
       self.writefile("report.html", '<div style="border: solid 2px #ccc; \
           background: #eee; text-align:center; font-size: 12pt;"> \
           <a href="#general_information">General information</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#discovered_hosts">Discovered hosts</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#scan_types">Scan types</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#hosts_targets">Hosts/Targets</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#tcp_ports_analysis">TCP ports analysis</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#udp_ports_analysis">UDP ports analysis</a> \
           </div>\n')

       #===== General information
       starttime, endtime, duration = self.ffts, self.lfts, (datetime.datetime.strptime(self.lfts, '%Y-%m-%d %H:%M:%S') - datetime.datetime.strptime(self.ffts, '%Y-%m-%d %H:%M:%S'))
       self.writefile("report.html", '<a name="general_information" id="general_information"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>General information</h1>\n")
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>File name (MD5)</th><td>%s (%s)</td></tr>\n' % (self.pcapfile, self.md5sum(self.pcapfile)) )
       self.writefile("report.html", '<tr><th>Number of packets</th><td>%s</td></th></tr>\n' % self.countnumberframes())
       self.writefile("report.html", '<tr><th>Capture duration</th><td>%s</td></th></tr>\n' % duration)
       self.writefile("report.html", '<tr><th>Start time</th><td>%s</td></th></tr>\n' % starttime)
       self.writefile("report.html", '<tr><th>End time</th><td>%s</td></th></tr>\n' % endtime)
       self.writefile("report.html", '</table>\n')
       self.writefile("report.html", '</div>\n')

       #===== Discovered hosts
       self.writefile("report.html", '<a name="discovered_hosts" id="discovered_hosts"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>Discovered hosts</h1>\n")
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>Host</th><th>MAC addr.</th><th>Vendor</th></tr>\n')
       for i in self.hosts:
           self.writefile("report.html", '<tr><td><a href="%s/graph.php?f=rcv_ip_id&dst=%s">%s</a> \
               </td><td>%s</td><td>%s</td></tr>\n' \
               % (self.BASEWB, i, i, self.hosts[i], self.macvendor(self.hosts[i])))
       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '</div>\n')

       #===== Scan types
       self.writefile("report.html", '<a name="scan_types" id="scan_types"></a>')
       self.writefile("report.html", '<div style="clear:both"></div><div style="float:left">\n')
       self.writefile("report.html", "<h1>Scan types</h1>\n")
       self.cursor.execute("""
           SELECT scan_type, COUNT( 1 )
           FROM pyscan
           GROUP BY scan_type
           ORDER BY COUNT( 1 ) DESC
           """)
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>Scan type</th><th># scans</th></tr>\n')
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break
           self.writefile("report.html", '<tr><td><a href="%s/pyscan.php?scan_type=%s">%s</a> \
               </td><td align="right">%s</td></tr>\n' \
               % (self.BASEWB, row[0].replace(" ", "_"), row[0], row[1]))

       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '<div><a href="%s/distributions.php">Scan distrib. charact.</a></div>\n' % (self.BASEWB))
       self.writefile("report.html", '</div>\n')

       #===== IP Options
       self.writefile("report.html", '<a name="scan_types" id="scan_types"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>IP Options</h1>\n")
       self.cursor.execute("""
           SELECT ip_flags, count(1) FROM pyscan GROUP BY ip_flags
           """)
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>IP flag</th><th>Count</th></tr>\n')
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break
           self.writefile("report.html", '<tr><td>%s</td><td align="right">%s</td></tr>\n' \
               % (row[0], row[1]))

       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '</div>\n')

       #===== Hosts/Targets
       self.writefile("report.html", '<a name="hosts_targets" id="hosts_targets"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>Hosts & targets</h1>\n")
       self.cursor.execute("""
           SELECT ip_src, ip_dst, COUNT(snd_frame)
           FROM pyscan
           GROUP BY ip_src, ip_dst
           ORDER BY ip_src, ip_dst
           """)
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>Src IP -> Dst IP</th><th># scans</th></tr>\n')
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break
           self.writefile("report.html", '<tr><td>%s -> %s</td><td align="right">%s</td></tr>\n' % (row[0], row[1], row[2]))
       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '</div>\n')
       self.writefile("report.html", '<div style="clear:both"></div>\n')

       #===== TCP ports analysis for each host
       self.writefile("report.html", '<a name="tcp_ports_analysis" id="tcp_ports_analysis"></a>')
       self.writefile("report.html", '<h1>TCP ports analysis</h1>\n')
       self.writefile("report.html", '<div style="clear:both"></div>\n')
       self.cursor.execute("""
           SELECT DISTINCT ip_dst, dport, state
           FROM pyscan
           WHERE proto='TCP'
           ORDER BY ip_dst, dport, state
       """)
       ip, port = '', ''
       state = []
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break

           # For each new IP: a new table
           if row[0]!=ip:
               if ip!='':
                   # End of previous table
                   self.writefile("report.html", '<tr><td>%s/tcp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
                   port = ''
                   self.writefile("report.html", "</table>\n</div>\n")
               self.writefile("report.html", '<div style="float:left">\n \
                   <h2>%s</h2>\n \
                   <table>\n \
                   <tr><th>Port</th><th>State</th></tr>\n' \
                   % row[0])
               ip = row[0]

           # For each new port
           if row[1]!=port:
               # We dump the list of states for port
               if port!='':
                   self.writefile("report.html", '<tr><td>%s/tcp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
               state = [] # We reset state list
               port = row[1]

           # Gathers all different states for given port
           state.append('<a href="%s/pyscan.php?ip=%s&proto=TCP&port=%s&state=%s">%s</a>' \
               % (self.BASEWB, ip, port, row[2], row[2]))

       # We dump content for last table of TCP section
       self.writefile("report.html", '<tr><td>%s/tcp</td><td>%s</td></tr>\n' \
           % (port, "/".join(state)) )
       self.writefile("report.html", '</table>\n</div>\n<div style="clear:both"></div>')

       #===== UDP ports analysis for each host
       self.writefile("report.html", '<a name="udp_ports_analysis" id="udp_ports_analysis"></a>')
       self.writefile("report.html", '<h1>UDP ports analysis</h1>\n')
       self.writefile("report.html", '<div style="clear:both"></div>\n')
       self.cursor.execute("""
           SELECT DISTINCT ip_dst, dport, state
           FROM pyscan
           WHERE proto='UDP'
           ORDER BY ip_dst, dport, state
       """)
       ip, port = '', ''
       state = []
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break

           # For each new IP: a new table
           if row[0]!=ip:
               if ip!='':
                   # End of previous table: we dump list of states for port
                   self.writefile("report.html", '<tr><td>%s/udp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
                   port = ''
                   self.writefile("report.html", "</table>\n</div>\n")
               self.writefile("report.html", '<div style="float:left">\n \
                   <h2>%s</h2>\n \
                   <table>\n \
                   <tr><th>Port</th><th>State</th></tr>\n' \
                   % row[0])
               ip = row[0]

           # For each new port
           if row[1]!=port:
               # We dump the list of states for port
               if port!='':
                   self.writefile("report.html", '<tr><td>%s/udp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
               state = [] # We reset state list
               port = row[1]

           # Gathers all different states for given port
           state.append('<a href="%s/pyscan.php?ip=%s&proto=UDP&port=%s&state=%s">%s</a>' \
               % (self.BASEWB, ip, port, row[2], row[2]))

       # We dump content for last table of UDP section
       self.writefile("report.html", '<tr><td>%s/udp</td><td>%s</td></tr>\n' \
           % (port, "/".join(state)) )
       self.writefile("report.html", '</table>\n</div>\n<div style="clear:both"></div>')

       # End of report
       self.writefile("report.html", "</body>\n</html>\n")


   def closeconnection(self):
       """Closes MySQL connection
       """
       # Closes MySQL connection and cursor
       self.cursor.close ()
       self.conn.close ()


if __name__ == '__main__':
   usage = "usage: %prog -r <file> [options]"
   parser = OptionParser(usage)
   parser.add_option("-r", "--read-file", dest="pcap_file",
       help="Capture file to process (pcap format)")
   parser.add_option("-o", '--output', dest="output_directory", default="./report",
       help="Reporting directory (default: ./report/)")
   parser.add_option("-f", '--force', dest="force", default=False, action="store_true",
       help="Force overwriting of files")
   parser.add_option("-v", '--vendor-database', dest="oui_file", default="./oui.txt",
       help="Vendor database (default: ./oui.txt)")
   parser.add_option("-d", "--dont-purge", dest="dont_purge", default=False, action="store_true",
       help="Don't purge existing data")
   (options, args) = parser.parse_args(sys.argv)
   # pcap is mandatory
   if not options.pcap_file:
       parser.error("Capture file is missing. Use -r <file>.")
   # Checks if output directory does not already exist as file
   if options.output_directory and os.path.isfile(options.output_directory):
       parser.error("Use a different name for output directory since it is already used for a file")
   # Checks that --force option is used if output directory exists
   if options.output_directory and os.path.isdir(options.output_directory) and not options.force:
       if os.listdir(options.output_directory):
           parser.error("Output directory is not empty. Use -f to overwrite content")
   # If force option is called, removes output directory
   if options.force:
       shutil.rmtree(options.output_directory, ignore_errors=True)

   p = PyScan(options.pcap_file, options.output_directory, options.oui_file, options.dont_purge)
   p.main()
   p.updatescantypes()
   p.exportresults()
   p.closeconnection()
   print "Finished. See generated report for full analysis.\n"
   del p


Additional Text:
------------------------ pyscan.sql ------------------------
CREATE USER pyscan@localhost IDENTIFIED BY 'pyscan';
CREATE DATABASE pyscan;
GRANT CREATE, INSERT, SELECT, DELETE, UPDATE, DROP ON pyscan.* TO pyscan@localhost;
USE pyscan;
CREATE TABLE pyscan (
 ts double,
 snd_frame int(11),
 rcv_frame int(11),
 ip_ihl int(11),
 ip_tos int(11),
 ip_len int(11),
 ip_id int(11),
 ip_flags char(3),
 ip_ttl int(11),
 proto varchar(3),
 ip_src varchar(15),
 ip_dst varchar(15),
 sport int(11),
 dport int(11),
 ack_conn tinyint(1),
 rst_conn tinyint(1),
 snd_tcp_ece tinyint(1),
 snd_tcp_cwr tinyint(1),
 snd_tcp_urg tinyint(1),
 snd_tcp_ack tinyint(1),
 snd_tcp_psh tinyint(1),
 snd_tcp_rst tinyint(1),
 snd_tcp_syn tinyint(1),
 snd_tcp_fin tinyint(1),
 rcv_tcp_urg tinyint(1),
 rcv_tcp_ack tinyint(1),
 rcv_tcp_psh tinyint(1),
 rcv_tcp_rst tinyint(1),
 rcv_tcp_syn tinyint(1),
 rcv_tcp_fin tinyint(1),
 rcv_ip_id int(11),
 icmp_err int(11),
 seq bigint(20),
 ack bigint(20),
 scan_type varchar(20),
 state varchar(20)
);
CREATE TABLE scan_type(
   id_scan_type INT,
   scan_type VARCHAR(15)
);
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (10, 'TCP SYN');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (20, 'TCP CONNECT');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (30, 'TCP NULL');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (40, 'TCP FIN');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (50, 'TCP XMAS');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (60, 'TCP ACK');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (70, 'TCP RST');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (80, 'TCP CUSTOM SCAN');
INSERT INTO scan_type(id_scan_type, scan_type) VALUES (90, 'UDP');
------------------------ /pyscan.sql ------------------------



=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Note: Only pyScanXtract.py source code is detailed here. Complete source code, including web interfaces is available @Google code: http://pyscanxtract.googlecode.com/files/pyscan.tar.gz
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=



------------------------ pyScanXtract.py ------------------------
#!/usr/bin/env python

# Developed by Sebastien DAMAYE (aldeid.com) in the shape of forensicscontest.com
# (Puzzle #4: The Curious Mr. X). Feel free to modify.
# Last update: 2010-02-14 21:42, Version: 1.0
#
# This script uses a MySQL connection. Please first make sure that you
# have created necessary objects and privileges, using following SQL
# script provided with pyScanXtract package (pyscan.sql)
# (Notice that DROP priv is required to TRUNCATE table unless
# --dont-purge option is used)
#
# Check that MySQL is started before you launch the script.
#
# In addition, this script needs a OUI file to detect mac vendors.
# The complete updated database (txt format) can be downloaded from
# http://standards.ieee.org/regauth/oui/oui.txt
#

from optparse import OptionParser
import sys
import shutil
import pcapy
import impacket.ImpactDecoder as Decoders
import impacket.ImpactPacket as Packets
import os.path
import os
import MySQLdb
import datetime
import struct
import hashlib

class PyScan:

   ###
   # You can modify these parameters to fit with your configuration
   #
   DBHOST = '127.0.0.1'    # Host to connect to
   DBUSER = 'pyscan'       # User name to connect to database
   DBPSWD = 'pyscan'       # Password to connect to database
   DBNAME = 'pyscan'       # Database name
   BASEWB = 'http://localhost/pyscan' # Base path for pyscan web interface
                                      # Don't put / at the end of the path
                                      # e.g. http://localhost/pyscan

   def __init__(self, pcapfile, reportpath="./report", ouifile="./oui.txt", dont_purge=None):
       """Performs some checks and connects to database
       """
       # Checks if pcap file exists
       assert pcapfile
       self.pcapfile = pcapfile
       if not os.path.exists(pcapfile):
           raise TypeError("Pcap file not found. Please check location.")

       self.reportpath = reportpath
       if not os.path.exists(self.reportpath):
           os.makedirs(self.reportpath)

       pc = open(pcapfile, mode='rb')
       pcheader = pc.read(24)

       # Read the first 4 bytes for the magic number, determine endianness
       pcmagicnum = struct.unpack("I", pcheader[0:4])[0]
       if pcmagicnum != 0xd4c3b2a1:
           # Little endian
           pcendflag = "<"
       elif magicnum == 0xa1b2c3d4:
           # Big endign
           pcendflag = ">"
       else:
           raise Exception('Specified file is not a libpcap capture')

       pcaph = struct.unpack("%sIHHIIII"%pcendflag, pcheader)
       if pcaph[1] != 2 and pcaph[2] != 4 \
               and pcaph[3] != 0 and pcaph[4] != 0 \
               and pcaph[5] != 65535:
           raise Exception('Unsupported pcap header format or version')

       # Checks if OUI file exists
       assert ouifile
       if not os.path.exists(ouifile):
           raise TypeError("OUI file not found. Please check location."
           + "OUI file can be downloaded from http://standards.ieee.org/regauth/oui/oui.txt")
       self.ouifile = ouifile

       # Connection to MySQL database
       try:
           self.conn = MySQLdb.connect (
               host = self.DBHOST,
               user = self.DBUSER,
               passwd = self.DBPSWD,
               db = self.DBNAME)
           self.cursor = self.conn.cursor ()
       except MySQLdb.Error, e:
           print "Error %d: %s" % (e.args[0], e.args[1])
           sys.exit (1)

       # Option: purge table
       if not dont_purge:
           self.cursor.execute ("TRUNCATE TABLE pyscan")

       # Initial list of hosts
       self.hosts = {}

   def md5sum(self, myfile):
       m = hashlib.md5()
       f = file(myfile, 'rb')
       while True:
           t = f.read(1024)
           if len(t) == 0:
               break
           m.update(t)
       return m.hexdigest()

   def writefile(self, f, content):
       """Dump content in a file
       """
       obFile = open(os.path.join(self.reportpath, f), 'a')
       obFile.write(content)
       obFile.close()

   def decodemac(self, mac):
       """Decode mac address with hex notation
       """
       m = ''
       for i in mac:
           t = "%x" % i
           if len(t)==1:
               t = '0'+t
           m=m+":"+t
       return m[1:]

   def countnumberframes(self):
       """Returns number of frames contained in pcap file
       """
       f = pcapy.open_offline(self.pcapfile)
       c = 0
       while f.next()[1]!="":
           c+=1
       return c

   def macvendor(self, mac):
       """Finds mac vendor in OUI file from 6 first cars of mac addr
       """
       #Only 6 first cars are necessary in upper cap
       pref = mac.replace(':', '')[:6].upper()
       f = open(self.ouifile, 'r')
       while 1:
               li = f.readline()
               if li=='':
                       break
               if pref in li:
                       return (li.replace('\n', '').replace('\t', ''))[20:]
                       break
       f.close()

   def getframefromflow(self, ip_src, sport, ip_dst, dport, proto):
       """Checks whether flow already exists in DB
          Returns frame number or zero
       """
       self.cursor.execute ("""
           SELECT snd_frame, ip_src FROM pyscan
           WHERE (ip_src=%s AND ip_dst=%s AND sport=%s AND dport=%s)
           OR (ip_src=%s AND ip_dst=%s AND sport=%s AND dport=%s)
           AND proto=%s
           """, (ip_src, ip_dst, sport, dport,
               ip_dst, ip_src, dport, sport, proto))
       row = self.cursor.fetchone()
       if row == None:
           row = [0, 0]
       return row

   def main(self):
       """Stores and reassemble frames in MySQL database.
          IP / TCP / UDP / ICMP protocols are handled in this version
       """
       print "Analysing pcap file..."
       reader = pcapy.open_offline(self.pcapfile)

       # Decoders for ethernet, IP, TCP, UDP and ICMP
       eth_decoder = Decoders.EthDecoder()
       ip_decoder = Decoders.IPDecoder()
       tcp_decoder = Decoders.TCPDecoder()
       udp_decoder = Decoders.UDPDecoder()
       icmp_decoder = Decoders.ICMPDecoder()
       ip_icmp_decoder = Decoders.IPDecoderForICMP()

       # Initial frame counter
       frame = 0
       nFrames = self.countnumberframes()

       (header, payload) = reader.next()

       while frame < nFrames:

           frame+=1
           # Timestamp
           (ts, usec) = header.getts()

           if frame==1:
               # First frame = reference (0 second)
               reftime = float(str(ts)+"."+str(usec))
               tm = 0
               # First Frame TimeStamp
               self.ffts = datetime.datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
           else:
               # Number of seconds since beginning of capture
               tm = float(str(ts)+"."+str(usec)) - reftime
               # Last frame time stamp
               if frame == nFrames:
                   self.lfts = datetime.datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")

           # Progress bar
           p = int(float(frame)/nFrames*50)
           sys.stdout.write("\r["+"#"*p+" "*(50-p)+"] "+str(p*2)+"% ("
               +str(frame)+"/"+str(nFrames)+")")
           sys.stdout.flush()

           #---------------------------------
           #           ETHERNET LAYER
           #---------------------------------
           ethernet = eth_decoder.decode(payload)

           #---------------------------------
           #           IP PACKETS
           #---------------------------------
           if ethernet.get_ether_type() == Packets.IP.ethertype:

               ip = ip_decoder.decode(payload[ethernet.get_header_size():])

               # IP header information
               ip_ihl    = ip.get_ip_hl()
               ip_tos    = ip.get_ip_tos()
               ip_len    = ip.get_ip_len()
               ip_id     = ip.get_ip_id()
               ip_ttl    = ip.get_ip_ttl()
               if ip.get_ip_df() > 0:
                   df = '1'
               else:
                   df = '0'
               if ip.get_ip_mf() > 0:
                   mf = '1'
               else:
                   mf = '0'
               ip_flags  = str(ip.get_ip_rf()) + df + mf
               ip_src    = ip.get_ip_src()
               ip_dst    = ip.get_ip_dst()

               #---------------------------------
               #           TCP PACKETS
               #---------------------------------
               if ip.get_ip_p() == Packets.TCP.protocol:
                   tcp = tcp_decoder.decode(
                       payload[ethernet.get_header_size()+ip.get_header_size():])

                   # New hosts are added to the list of hosts
                   if self.hosts.keys().count(ip_src)==0:
                       self.hosts[ip_src] = self.decodemac(ethernet.get_ether_shost())
                   if self.hosts.keys().count(ip_dst)==0:
                       self.hosts[ip_dst] = self.decodemac(ethernet.get_ether_dhost())

                   # TCP header information
                   tcp_sport = tcp.get_th_sport()
                   tcp_dport = tcp.get_th_dport()
                   tcp_ece   = tcp.get_ECE()
                   tcp_cwr   = tcp.get_CWR()
                   tcp_urg   = tcp.get_URG()
                   tcp_ack   = tcp.get_ACK()
                   tcp_psh   = tcp.get_PSH()
                   tcp_rst   = tcp.get_RST()
                   tcp_syn   = tcp.get_SYN()
                   tcp_fin   = tcp.get_FIN()
                   seq       = tcp.get_th_seq()
                   ack       = tcp.get_th_ack()

                   # Checks if src:sport->dst:dport or dst:dport->src:sport
                   # doesn't already exist in DB
                   arrFrame = self.getframefromflow(ip_src, tcp_sport, ip_dst, tcp_dport, 'TCP')
                   if arrFrame[0]==0:
                       # Frame hasn't been found: new flow
                       self.cursor.execute ("""
                           INSERT INTO pyscan(ts, snd_frame, ip_ihl, ip_tos,
                             ip_len, ip_id, ip_flags, ip_ttl, proto, ip_src,
                             ip_dst, sport, dport, snd_tcp_ece, snd_tcp_cwr,
                             snd_tcp_urg, snd_tcp_ack, snd_tcp_psh, snd_tcp_rst,
                             snd_tcp_syn, snd_tcp_fin, seq, ack
                           ) VALUES (
                             %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
                             %s, %s, %s, %s, %s, %s, %s, %s, %s, %s
                           )
                           """, (tm, frame, ip_ihl, ip_tos, ip_len, ip_id,
                           ip_flags, ip_ttl, 'TCP', ip_src, ip_dst, tcp_sport,
                           tcp_dport, tcp_ece, tcp_cwr, tcp_urg, tcp_ack,
                           tcp_psh, tcp_rst, tcp_syn, tcp_fin, seq, ack))
                   else:
                       if arrFrame[1]==ip_src:
                           #Attacker->Target: update flow (request)
                           if tcp_rst==1:
                               # RST conn
                               self.cursor.execute ("""
                                   UPDATE pyscan SET rst_conn=1
                                   WHERE snd_frame = %s
                                   """, (arrFrame[0]))
                           else:
                               #ACK conn
                               self.cursor.execute ("""
                                   UPDATE pyscan SET ack_conn=1
                                   WHERE snd_frame = %s
                                   """, (arrFrame[0]))
                       else:
                           # Target->Attacker: update flow (reply)
                           # "snd_frame IS NULL" clause is added to
                           # prevent from updating an already existing answer
                           self.cursor.execute ("""
                               UPDATE pyscan SET
                                   rcv_frame = %s,
                                   rcv_tcp_urg = %s,
                                   rcv_tcp_ack = %s,
                                   rcv_tcp_psh = %s,
                                   rcv_tcp_rst = %s,
                                   rcv_tcp_syn = %s,
                                   rcv_tcp_fin = %s,
                                   rcv_ip_id=%s,
                                   ack = %s
                               WHERE snd_frame = %s
                               AND   rcv_frame IS NULL
                               """, (frame,
                                   tcp_urg, tcp_ack, tcp_psh, tcp_rst, tcp_syn,
                                   tcp_fin, ip_id, ack, arrFrame[0]))

               #---------------------------------
               #           UDP PACKETS
               #---------------------------------
               if ip.get_ip_p() == Packets.UDP.protocol:
                   udp = udp_decoder.decode(
                       payload[ethernet.get_header_size()+ip.get_header_size():])

                   # UDP header information
                   udp_sport = udp.get_uh_sport()
                   udp_dport = udp.get_uh_dport()
                   udp_len   = udp.get_size()

                   # Any other way to ignore NBNS (NetBIOS Name Service)
                   # packets with Impacket?
                   if udp_len > 70:
                       # Log SENT UDP PACKETS TO TARGET in DB
                       arrFrame = self.getframefromflow(ip_src, udp_sport, ip_dst, udp_dport, 'UDP')
                       if arrFrame[0]==0:
                           # Frame hasn't been found: new flow
                           self.cursor.execute ("""
                               INSERT INTO pyscan(ts, snd_frame, proto, ip_src,
                                 ip_dst, sport, dport
                               ) VALUES (
                                 %s, %s, %s, %s, %s, %s, %s
                               )
                               """, (tm, frame, 'UDP', ip_src, ip_dst,
                               udp_sport, udp_dport))

               #---------------------------------
               #           ICMP PACKETS
               #---------------------------------
               if ip.get_ip_p() == Packets.ICMP.protocol:
                   icmp = icmp_decoder.decode(
                       payload[ethernet.get_header_size()+ip.get_header_size():])

                   # ICMP header information
                   icmp_type = icmp.get_icmp_type()
                   icmp_code = icmp.get_icmp_code()

                   # ICMP echo request/reply are not kept in database
                   if icmp_type != 8 and icmp_type !=0:

                       # Special IP decoder for ICMP
                       ip_icmp = ip_icmp_decoder.decode(
                           payload[ethernet.get_header_size()+ip.get_header_size()
                           +icmp.get_header_size():])
                       # UDP decoder for IP in ICMP
                       udp_ip_icmp = udp_decoder.decode(
                           payload[ethernet.get_header_size()+ip.get_header_size()
                           +icmp.get_header_size()+ip_icmp.get_header_size():])

                       # IP header information in ICMP
                       ip_icmp_src = ip_icmp.get_ip_src()
                       ip_icmp_dst = ip_icmp.get_ip_dst()

                       # UDP header information in IP in ICMP
                       udp_ip_icmp_sport = udp_ip_icmp.get_uh_sport()
                       udp_ip_icmp_dport = udp_ip_icmp.get_uh_dport()

                       # Looks for frame number that has required criteria
                       fn = self.getframefromflow(ip_icmp_src, udp_ip_icmp_sport,
                           ip_icmp_dst, udp_ip_icmp_dport, 'UDP')[0]

                       # Updates flow in database
                       self.cursor.execute ("""
                           UPDATE pyscan
                           SET rcv_frame=%s, icmp_err=%s
                           WHERE snd_frame=%s
                           """, (frame, icmp_code, fn))

               (header, payload) = reader.next()


   def updatescantypes(self):
       """Update database by guessing scan types
          for each detected scan
       """
       print "\nDetermining scan types..."

       #---------------------------------
       #           SCAN TYPES
       #---------------------------------

       # TCP SYN (only TCP SYN flag activated and no ack.)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP SYN'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000010'
           AND    ack_conn IS NULL
           """)
       # TCP CONNECT (same as TCP SYN but ack.)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP CONNECT'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000010'
           AND    ack_conn IS NOT NULL
           """)
       # TCP NULL (no TCP flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP NULL'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000000'
           """)
       # TCP FIN (only TCP FIN flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP FIN'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000001'
           """)
       # TCP XMAS (only TCP FIN/PSH/URG flags activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP XMAS'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00101001'
           """)
       # TCP ACK (only TCP ACK flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP ACK'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00010000'
           """)
       # TCP RST (only TCP RST flag activated)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP RST'
           WHERE  proto='TCP'
           AND    CONCAT(snd_tcp_ece, snd_tcp_cwr, snd_tcp_urg, snd_tcp_ack,
                  snd_tcp_psh, snd_tcp_rst, snd_tcp_syn, snd_tcp_fin)='00000100'
           """)
       # ALL OTHER TCP SCANS (custom flags)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='TCP CUSTOM SCAN'
           WHERE  proto='TCP'
           AND    scan_type IS NULL
           """)
       # UDP SCANS
       self.cursor.execute ("""
           UPDATE pyscan
           SET    scan_type='UDP'
           WHERE  proto='UDP'
           """)

       #---------------------------------
       #           PORTS STATE
       #---------------------------------
       print "Determining ports state..."

       #----- TCP SYN -----

       # TCP SYN - OPEN (SYN/ACK from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN'
           WHERE  scan_type='TCP SYN'
           AND    rcv_tcp_ack = 1
           AND    rcv_tcp_syn = 1
           """)
       # TCP SYN - CLOSED (RST from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='CLOSED'
           WHERE  scan_type='TCP SYN'
           AND    rcv_tcp_rst = 1
           """)
       # TCP SYN - FILTERED (no answer from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  scan_type='TCP SYN'
           AND    rcv_frame IS NULL
           """)

       #----- TCP CONNECT -----

       # TCP CONNECT - OPEN (SYN/ACK from source)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN'
           WHERE  scan_type='TCP CONNECT'
           AND    rcv_tcp_syn = 1
           AND    rcv_tcp_ack = 1
           """)

       #----- TCP NULL/FIN/XMAS -----

       # TCP NULL/FIN/XMAS - OPEN|FILTERED (no response from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN|FILTERED'
           WHERE  scan_type IN ('TCP NULL', 'TCP FIN', 'TCP XMAS')
           AND    rcv_frame IS NULL
           """)
       # TCP NULL/FIN/XMAS - CLOSED (RST received from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='CLOSED'
           WHERE  scan_type IN ('TCP NULL', 'TCP FIN', 'TCP XMAS')
           AND    rcv_tcp_rst = 1
           """)
       # TCP NULL/FIN/XMAS - FILTERED (ICMP unreachable error)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  scan_type IN ('TCP NULL', 'TCP FIN', 'TCP XMAS')
           AND    icmp_err IN (1, 2, 3, 9, 10, 13)
           """)

       #----- TCP ACK -----

       # TCP ACK - UNFILTERED (RST received from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='UNFILTERED'
           WHERE  scan_type='TCP ACK'
           AND    rcv_tcp_rst = 1
           """)
       # TCP ACK - FILTERED (no response from target OR ICMP unreachable err)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  (scan_type='TCP ACK'
           AND    rcv_frame IS NULL)
           OR     icmp_err IN (1, 2, 3, 9, 10, 13)
           """)

       #----- UDP SCAN -----
       # UDP SCANS - OPEN|FILTERED (no response from target)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='OPEN|FILTERED'
           WHERE  scan_type='UDP'
           AND    rcv_frame IS NULL
           """)
       # UDP SCANS - CLOSED (ICMP port unreachable)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='CLOSED'
           WHERE  scan_type='UDP'
           AND    icmp_err=3
           """)
       # UDP SCANS - FILTERED (type 3, code 1, 2, 9, 10 or 13)
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='FILTERED'
           WHERE  scan_type='UDP'
           AND    icmp_err IN (1, 2, 9, 10, 13)
           """)

       #----- ALL OTHERS: UNDEFINED -----
       self.cursor.execute ("""
           UPDATE pyscan
           SET    state='UNDEFINED'
           WHERE  state IS NULL
           """)


   def exportresults(self):
       print "Generating report..."
       self.writefile("report.html", """<html><head><style>
           h1           { font-size:11pt; padding:2px; }
           h2           { font-size:9pt; padding:2px; }
           body, td, th { font-size:8pt; }
           table        { border-collapse:collapse; border:solid 2px #cccccc; }
           th, td       { border:solid 1px #dddddd;padding:2px;}
           th           { background:#eeeeee; }
           div          { padding:10px; }
           </style></head><body>\n""")
       #===== Index
       self.writefile("report.html", '<div style="border: solid 2px #ccc; \
           background: #eee; text-align:center; font-size: 12pt;"> \
           <a href="#general_information">General information</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#discovered_hosts">Discovered hosts</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#scan_types">Scan types</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#hosts_targets">Hosts/Targets</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#tcp_ports_analysis">TCP ports analysis</a> \
           &nbsp;&nbsp;|&nbsp;&nbsp;<a href="#udp_ports_analysis">UDP ports analysis</a> \
           </div>\n')

       #===== General information
       starttime, endtime, duration = self.ffts, self.lfts, (datetime.datetime.strptime(self.lfts, '%Y-%m-%d %H:%M:%S') - datetime.datetime.strptime(self.ffts, '%Y-%m-%d %H:%M:%S'))
       self.writefile("report.html", '<a name="general_information" id="general_information"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>General information</h1>\n")
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>File name (MD5)</th><td>%s (%s)</td></tr>\n' % (self.pcapfile, self.md5sum(self.pcapfile)) )
       self.writefile("report.html", '<tr><th>Number of packets</th><td>%s</td></th></tr>\n' % self.countnumberframes())
       self.writefile("report.html", '<tr><th>Capture duration</th><td>%s</td></th></tr>\n' % duration)
       self.writefile("report.html", '<tr><th>Start time</th><td>%s</td></th></tr>\n' % starttime)
       self.writefile("report.html", '<tr><th>End time</th><td>%s</td></th></tr>\n' % endtime)
       self.writefile("report.html", '</table>\n')
       self.writefile("report.html", '</div>\n')

       #===== Discovered hosts
       self.writefile("report.html", '<a name="discovered_hosts" id="discovered_hosts"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>Discovered hosts</h1>\n")
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>Host</th><th>MAC addr.</th><th>Vendor</th></tr>\n')
       for i in self.hosts:
           self.writefile("report.html", '<tr><td><a href="%s/graph.php?f=rcv_ip_id&dst=%s">%s</a> \
               </td><td>%s</td><td>%s</td></tr>\n' \
               % (self.BASEWB, i, i, self.hosts[i], self.macvendor(self.hosts[i])))
       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '</div>\n')

       #===== Scan types
       self.writefile("report.html", '<a name="scan_types" id="scan_types"></a>')
       self.writefile("report.html", '<div style="clear:both"></div><div style="float:left">\n')
       self.writefile("report.html", "<h1>Scan types</h1>\n")
       self.cursor.execute("""
           SELECT scan_type, COUNT( 1 )
           FROM pyscan
           GROUP BY scan_type
           ORDER BY COUNT( 1 ) DESC
           """)
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>Scan type</th><th># scans</th></tr>\n')
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break
           self.writefile("report.html", '<tr><td><a href="%s/pyscan.php?scan_type=%s">%s</a> \
               </td><td align="right">%s</td></tr>\n' \
               % (self.BASEWB, row[0].replace(" ", "_"), row[0], row[1]))

       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '<div><a href="%s/distributions.php">Scan distrib. charact.</a></div>\n' % (self.BASEWB))
       self.writefile("report.html", '</div>\n')

       #===== IP Options
       self.writefile("report.html", '<a name="scan_types" id="scan_types"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>IP Options</h1>\n")
       self.cursor.execute("""
           SELECT ip_flags, count(1) FROM pyscan GROUP BY ip_flags
           """)
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>IP flag</th><th>Count</th></tr>\n')
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break
           self.writefile("report.html", '<tr><td>%s</td><td align="right">%s</td></tr>\n' \
               % (row[0], row[1]))

       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '</div>\n')

       #===== Hosts/Targets
       self.writefile("report.html", '<a name="hosts_targets" id="hosts_targets"></a>')
       self.writefile("report.html", '<div style="float:left">\n')
       self.writefile("report.html", "<h1>Hosts & targets</h1>\n")
       self.cursor.execute("""
           SELECT ip_src, ip_dst, COUNT(snd_frame)
           FROM pyscan
           GROUP BY ip_src, ip_dst
           ORDER BY ip_src, ip_dst
           """)
       self.writefile("report.html", '<table border="1">\n')
       self.writefile("report.html", '<tr><th>Src IP -> Dst IP</th><th># scans</th></tr>\n')
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break
           self.writefile("report.html", '<tr><td>%s -> %s</td><td align="right">%s</td></tr>\n' % (row[0], row[1], row[2]))
       self.writefile("report.html", "</table>\n")
       self.writefile("report.html", '</div>\n')
       self.writefile("report.html", '<div style="clear:both"></div>\n')

       #===== TCP ports analysis for each host
       self.writefile("report.html", '<a name="tcp_ports_analysis" id="tcp_ports_analysis"></a>')
       self.writefile("report.html", '<h1>TCP ports analysis</h1>\n')
       self.writefile("report.html", '<div style="clear:both"></div>\n')
       self.cursor.execute("""
           SELECT DISTINCT ip_dst, dport, state
           FROM pyscan
           WHERE proto='TCP'
           ORDER BY ip_dst, dport, state
       """)
       ip, port = '', ''
       state = []
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break

           # For each new IP: a new table
           if row[0]!=ip:
               if ip!='':
                   # End of previous table
                   self.writefile("report.html", '<tr><td>%s/tcp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
                   port = ''
                   self.writefile("report.html", "</table>\n</div>\n")
               self.writefile("report.html", '<div style="float:left">\n \
                   <h2>%s</h2>\n \
                   <table>\n \
                   <tr><th>Port</th><th>State</th></tr>\n' \
                   % row[0])
               ip = row[0]

           # For each new port
           if row[1]!=port:
               # We dump the list of states for port
               if port!='':
                   self.writefile("report.html", '<tr><td>%s/tcp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
               state = [] # We reset state list
               port = row[1]

           # Gathers all different states for given port
           state.append('<a href="%s/pyscan.php?ip=%s&proto=TCP&port=%s&state=%s">%s</a>' \
               % (self.BASEWB, ip, port, row[2], row[2]))

       # We dump content for last table of TCP section
       self.writefile("report.html", '<tr><td>%s/tcp</td><td>%s</td></tr>\n' \
           % (port, "/".join(state)) )
       self.writefile("report.html", '</table>\n</div>\n<div style="clear:both"></div>')

       #===== UDP ports analysis for each host
       self.writefile("report.html", '<a name="udp_ports_analysis" id="udp_ports_analysis"></a>')
       self.writefile("report.html", '<h1>UDP ports analysis</h1>\n')
       self.writefile("report.html", '<div style="clear:both"></div>\n')
       self.cursor.execute("""
           SELECT DISTINCT ip_dst, dport, state
           FROM pyscan
           WHERE proto='UDP'
           ORDER BY ip_dst, dport, state
       """)
       ip, port = '', ''
       state = []
       while 1:
           row = self.cursor.fetchone()
           if row == None:
               break

           # For each new IP: a new table
           if row[0]!=ip:
               if ip!='':
                   # End of previous table: we dump list of states for port
                   self.writefile("report.html", '<tr><td>%s/udp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
                   port = ''
                   self.writefile("report.html", "</table>\n</div>\n")
               self.writefile("report.html", '<div style="float:left">\n \
                   <h2>%s</h2>\n \
                   <table>\n \
                   <tr><th>Port</th><th>State</th></tr>\n' \
                   % row[0])
               ip = row[0]

           # For each new port
           if row[1]!=port:
               # We dump the list of states for port
               if port!='':
                   self.writefile("report.html", '<tr><td>%s/udp</td><td>%s</td></tr>\n' \
                       % (port, "/".join(state)) )
               state = [] # We reset state list
               port = row[1]

           # Gathers all different states for given port
           state.append('<a href="%s/pyscan.php?ip=%s&proto=UDP&port=%s&state=%s">%s</a>' \
               % (self.BASEWB, ip, port, row[2], row[2]))

       # We dump content for last table of UDP section
       self.writefile("report.html", '<tr><td>%s/udp</td><td>%s</td></tr>\n' \
           % (port, "/".join(state)) )
       self.writefile("report.html", '</table>\n</div>\n<div style="clear:both"></div>')

       # End of report
       self.writefile("report.html", "</body>\n</html>\n")


   def closeconnection(self):
       """Closes MySQL connection
       """
       # Closes MySQL connection and cursor
       self.cursor.close ()
       self.conn.close ()


if __name__ == '__main__':
   usage = "usage: %prog -r <file> [options]"
   parser = OptionParser(usage)
   parser.add_option("-r", "--read-file", dest="pcap_file",
       help="Capture file to process (pcap format)")
   parser.add_option("-o", '--output', dest="output_directory", default="./report",
       help="Reporting directory (default: ./report/)")
   parser.add_option("-f", '--force', dest="force", default=False, action="store_true",
       help="Force overwriting of files")
   parser.add_option("-v", '--vendor-database', dest="oui_file", default="./oui.txt",
       help="Vendor database (default: ./oui.txt)")
   parser.add_option("-d", "--dont-purge", dest="dont_purge", default=False, action="store_true",
       help="Don't purge existing data")
   (options, args) = parser.parse_args(sys.argv)
   # pcap is mandatory
   if not options.pcap_file:
       parser.error("Capture file is missing. Use -r <file>.")
   # Checks if output directory does not already exist as file
   if options.output_directory and os.path.isfile(options.output_directory):
       parser.error("Use a different name for output directory since it is already used for a file")
   # Checks that --force option is used if output directory exists
   if options.output_directory and os.path.isdir(options.output_directory) and not options.force:
       if os.listdir(options.output_directory):
           parser.error("Output directory is not empty. Use -f to overwrite content")
   # If force option is called, removes output directory
   if options.force:
       shutil.rmtree(options.output_directory, ignore_errors=True)

   p = PyScan(options.pcap_file, options.output_directory, options.oui_file, options.dont_purge)
   p.main()
   p.updatescantypes()
   p.exportresults()
   p.closeconnection()
   print "Finished. See generated report for full analysis.\n"
   del p
------------------------ /pyScanXtract.py ------------------------