≡ Menu

Network

Splunk vs ELK

If you are in IT Operations in any role, you have probably come across either Splunk or ELK, or both. These are two heavyweights in the field of Operational Data Analytics. In this blog post, I’m going to share with you what I feel about these two excellent products based on my years of experience with them.

The problem Splunk and ELK are trying to solve: Log Management

While there are fancier terms such as Operational Data Intelligence, Operational Big Data Analytics and Log data analytics platform, the problem both Splunk and ELK are trying to solve is Log Management. So, what’s the challenge with Log management?

Logs, logs, logs and more logs

 

The single most important piece of troubleshooting data in any software program is the log generated by the program. If you have ever worked with vendor support for any software product, you have been inevitably asked to provide – you guessed it, Log files. Without the log files, they really can’t see what’s going on.

Logs not only contains information about how the software program runs, they may contain data that are valuable to business as well. Yeap, that’s right. For instance, you can retrieve wealth of data from your Web Server access logs to find out things like geographical dispersion of your customer base, most visited page in your website, etc.

If you are running only a couple of servers with few applications running on them, accessing and managing your logs are not a problem. But in an enterprise with hundreds and even thousands of servers and applications, this becomes an issue. Specifically,

  1. There are thousands of log files.
  2. The size of these log files run in Giga or even Terra bytes.
  3. The data in these log files may not be readily readable or searchable (unstructured data)

Sources_of_logfiles (4)

Both Splunk and ELK attempt to solve the problem of managing ever growing Log data. In essence, they supply a scalable way to collect and index log files and provide a search interface to interact with the data. In addition, they provide a way to secure the data being collected and enable users to create visualizations such as reports, dashboards and even Alerts.

Now that you know the problem Splunk and ELK are attempting to solve, let’s compare them and find how they are achieving this. I’m going to compare them in 4 areas as follows:

Technology

Cost

Features

Learning Curve for the operations team

Got it ? I can’t wait to share. Let’s dive in.

Splunk_image

VS

ElasticSearch Logo

Read More

How to use AppDynamics to monitor Server health?

Yes, AppDynamics is awesome for Application monitoring – Java Heap, deep transaction tracing, tons of out-of-the-box framework monitoring (JDBC,WebService etc) and the list goes on. But do you know Appdynamics can be used to effectively monitor Servers too, whether it is virtual or physical? When I say server, I mean the host operating system such as RedHat Enterprise Linux, Windows 2012, Solaris etc. Let me show you how you can do this.

Enter AppDynamics Machine Agent

While Java can be monitored using a Java Agent, a Server can be monitored using a special type of agent called Machine Agent. You will have to have license to run these agents (When you purchase Application agents, typically AppDyanmics throws the same number of Machine Agents, and so you should be good in terms of additional cost). If you are not sure about your present licensing situation, click on ‘licensing’ in your Controller UI as shown below.

Unlike Application agents which run inside the JVM/CLR, Machine agent is a standalone Java program that runs on the host operating system. It collects hardware metrics and sends them to Controller (once a minute). A user can view these metrics via Controller UI. Pretty simple, hah?

Read More

What is SYN_SENT socket status?

Say you are troubleshooting a connectivity issue between two devices. For instance, let’s say you believe your Web Server is not able to connect to your Application Server, because you are seeing Connection timed out or Connection Refused error all over your log files. One of the easiest (yet critical) ways to check the connectivity is using netstat command. You might use netstat command something like the following:

netstat -an

or

netstat -an | grep 

The output shows all the sockets that in the system. Each socket has various status. For example, a socket can be in ESTABLISHED status or in LISTENING status. These are the good statuses.

However, if you are having a bad day, you might see this weird SYN_SENT status.

➜ ~ netstat -an | grep 23.96.52.51
tcp4 0 0 192.168.0.4.49876 23.96.52.53.22 SYN_SENT

The goal of this post is to demystify what SYN_SENT is and how you can go about fixing it.

But first let’s take a quick look at how TCP/IP works when a network connection is formed.

TCP/IP 101

 

TCP/IP (Transmission Control Protocol/Internet Protocol) is a set of protocols used to transmit and receive data. US Department of Defence invented it, and it’s been the language of the Networks since early 80s (right, some of you had not been born yet). Internet buzzes on top of TCP/IP. While the protocol was designed to work on any physical medium, practically all the networks you will deal with nowadays are Ethernet and/or WiFi.

In a distributed applications/services setup (which most of us ar dealing with), the Network architecture involved is the Client-Server architecture. Let’s first look at how this looks.

client_server

In the above diagram, Web Server is the client and Application Server is the Server. In a Client-Server architecture, the Server listens on a particular port. Ports are used to identify a particular application/program. For instance, here are some of the well-known Server applications and their port numbers.

Screen Shot 2018-08-12 at 10.06.58 PM

In our example, our application server listens on port 8080 (and incidentally is also a web server. We don’t need to dwell deep in to this fact, but as you may know, many application servers have a small footprint web server embedded in them).

 

Another quick note about how HTTP relates to TCP:

TCP/IP uses a 5 layer reference model (This is little different from the original 7 layer OSI model).

HTTP is an application layer protocol. And it relies on underlying TCP protocol in the transport layer. To lay this out approximately:

Screen Shot 2018-08-12 at 9.32.43 PM

Digging deep into a TCP Connection

 

TCP is a connection-oriented transport service (unlike UDP – User Datagram Protocol, which is a connection-less transport service). What do I mean by connection-oriented transport service? I mean the following:

  1. It ensures guaranteed data delivery to the destination
  2. It has built-in error checks. And in case of errors, the protocol provides for re-transmission of data
  3. The order of the data packets are guaranteed to be accurate.

When the client initiates a connection to Server, there is a three-way handshake happening. here’s what’s going on.

tcp-hand-shake

  1. First client sends a TCP segment with SYN control bit (synchronize) set.
  2. If server receives client’s data (Yay!!), it sends acknowledgement along with its own SYN request
  3. Client sends acknowledgement. Connection is established.

 

Checking the status of a TCP connection

 

A TCP connection progresses through several statuses.

LISTEN

SYN-SENT

SYN-RECEIVED

ESTABLISHED

FIN-WAIT-1

FIN-WAIT-2

CLOSE-WAIT

CLOSING

LAST-ACK

TIME-WAIT

CLOSED

You can reveal a connection status by using netstat.

netstat -an

You would typically be grepping the output of netstat for your remote IP address.

➜ ~ netstat -an | grep 23.96.52.51 

tcp4 0 0 192.168.0.4.49876 23.96.52.53.22 SYN_SENT

Now, based on the knowledge you’ve gained so far, you can readily tell that this connection is waiting for the server to respond. This most probably means the client’s request never got to the Server. And 99% of the time, this indicates some sort of Network block due to a firewall.

fw

Note that the SYN_SENT status will not remain for long time. It only lasts for couple of seconds. So, you have to be quick in running the netstat command (perhaps in another terminal window)

There you have it. Next time when you see SYN_SENT, you know who to reach out to. Your Network team. 🙂

Good luck.

Network Security Attacks

There are several types of Network Security Attacks as described below:

  1. SYN Flood

    Here the attacker sends a SYN request from a spoofed source address. When the server responds with SYN-ACK, the source never replies back leaving the server handing with a half-open connection (Typically the client sends a SYN-ACK-ACK to complete the three way handshake). Half open connections consume resources eventually degrading the performance of the server.

    CISCO routers employ ‘TCP Intercept’ and ‘CAR – Committed Access Rate’ features to combat SYN-FLOOD. You can also change the default setting for the ‘maximum number of half-open TCP connections’

  2. UDP Flood

    Here the server is flooded with UDP requests, degrading the performance of the Server

  3. ICMP Flood

    Here the Server is flooded with ‘echo’ requests (which is an ICMP request), degrading the performance of the Server.

    It is best to drop ICMP packets at the router or Firewall.

  4. Smurf

    Here the attacker sends ICMP request packets to the broadcast address of the target network using a spoofed source address. The target responds with a echo request to all the hosts in the network, eventually overwhelming the network.

  5. Fraggle

    A flavor of Smurf attack which uses UDP Echo packets (UDP Port 7) instead of ICMP packets. Cisco routers can be configured to disable the TCP and UDP services (TCP and UDP small servers) to defend against Fraggle

  6. Bluejacking and bluesnarfing

    Here the Bluetooth enabled devices are attached. In Bluejacking, unsolicited messages are sent. In Bluesnarfing, personal information such as pictures and contacts, and cell phone information such as serial numbers are stolen.

Read More

Protecting Wireless Networks using WEP,WPA and WPA2

Wired Equivalent Privacy:

The intention of WEP (Wired Equivalent Privacy) was to provide the same level of security as in Wired Networks. But it fell short greatly.

WEP uses 128bit key (with 24 bit initialization vector) which is very easy to crack. It uses RC4 (Rivest Cipher 4) stream cipher.

Two modes:

Open Systems Authentication:

No need of credentials from the client. After the initial association with AP (Access Point), WEP encrypts the whole conversation.

Shared Key Authentication:

Requires Client to present credentials to connect to AP before the encryption beings.

WEP can be enhanced by using ssh or tunneling.

WiFi Protected Access (WPA and WPA2):

WPA uses TKIP(Temporal Key Integrity Protocol), a sequence counter to prevent replay attacks and a 64 bit message integrity check. It combines a secret root key with initialization vector.

WPA2 uses AES with Cipher Block chaining message Authentication code Protocol (CCMP).

Both WPA and WPA supports several EAP extensions such as EAP-TLS, EAP-TTLS (Tunneled Transport Layer Security) and Protected EPA (PEAPv0,v1)

VPN (Virtual Private Network) Security Protocols

Commonly used VPN security technologies are:

  1. Point to Point Tunneling Protocol (PPTP)
  2. Layer2 Forwarding Protocol (L2F)
  3. Layer2 Tunneling Protocol (L2TP)
  4. IPSec
  5. SSL

Point to Point Tunneling Protocol (PPTP):

  1. Uses PAP, CHAP, EAP
  2. Typically used in dial-up connections in Windows platform
  3. Operates at Data Link Layer

Layer 2 forwarding Protocol (L2F):

  1. Developed by CISCO
  2. Similar to PPTP
  3. Operates at Data Link Layer

Layer 2 tunneling Protocol (L2TP)

Read More

Remote access security technologies

There are 5 major remote access security technologies

  1. RAS (Remote Access Service)
  2. Radius
  3. Diameter
  4. TACACS

RAS (Remote Access Service):
Uses PPP (Point to Point Protocol) to secure dial-in, ISDN and serial links. Uses the following authentication mechanisms.

PAP (Password Authentication Protocol):

  1. Two way hand shake
  2. Sends passwords in clear text
  3. No protection against replay or brute force attacks

CHAP (Challenge Handshake Protocol):

  1. Uses three way hand shake
  2. Both server and client need to have a shared secret preconfigured
  3. Shared secret is stored in clear text. MS-CHAP allows the shared secret to be stored in encrypted form

EAP (Extensible Authentication Protocol):

  1. Used primarily in Wireless networks
  2. Supports various authentication mechanisms like MD5-Challenge, S/Key, generic token card and digital Certificates

 

RADIUS (Remote Authentication Dial-In Service)
  1. Open-Source UDP based.
  2. Provides authentication and accountability
  3. Use provides username/password to a RADIUS client using PAP or CHAP. Radius client encrypts password and sends to RADIUS Server for authentication

 

DIAMETER
  1. Improved version of RADIUS
  2. Uses TCP. Supports IPSsec, TLS

 

TACACS (Terminal Access Controller Access Control System):
  1. Uses UDP. Provides Authentication, Authorization and Accountability
  2. XTACACS is an improved version but no longer used
  3. TACACS+ is the current version. Supports several authentication mechanisms – PAP,CHAP,MS-CHAP,EAP,KERBEROS,Token Cards

Firewall Classifications and Architectures

Classifications of Firewalls:

  1. Packet Filtering
  2. Circuit Level Gateway
  3. Application Level Gateway

Architectures of Firewall:

  1. Screening Router
  2. Dual Homed Gateway
  3. Screened-Host Gateway
  4. Screened Subnet

 

Classification

Description

Advantages

Disadvantages

Packet Filtering

Basic. Operates at Network or Transport Layers. Examines TCP,IP,ICMP,UDP headers from the packet and routes based on a firewall ACL

  1. In expensive and Fast
  2. Easy to setup
  3. Transparent to users
  1. No Context level routing
  2. Can be hit by Spoofing
  3. Limited Logging
  4. No strong user authentication

Circuit Level Gateway

Operates at Session Layer. Uses state information about the established connections. Once the virtual circuit is formed, no packet analysis is done.

  1. Fast
  2. Low maintenance
  1. Limited Logging.
  2. Once connection is established, no further analysis is done

Application Level Gateway

Operates at Application Layer. Implemented as a Proxy Server.

  1. Supports Strong user authentication
  2. Data is not directly sent to the destination.
  1. Low performance because packet needs to be brought all the way up to Application layer for analysis
  2. High maintenance.

 

Architecture

Description

Advantages

Disadvantages

Screening Router

Basic Packet Filtering Firewall

  1. Cheap
  2. Transparent to users
  1. Makes internal Network structure complex
  2. No user authentication
  3. Single point of failure

Dual homed Gateway

It is bastion host with two network interface cards. It may be connected to an external screening router

  1. Fail safe mode. If it fails, nothing is allowed access
  2. Internal network structure is masked
  1. Additional auth required for users
  2. May slow down performance
  3. May not be available for all services.

Screened Host Gateway

External Screening router and internal Bastion Host.

  1. Transparent outbound access and restricted inbound access
  1. Screening router can by-pass the Bastion host
  2. Masking internal network is difficult

Screened subnet

Most secure. Forms a DMZ network between external and internal firewall

  1. Transparent, flexible
  2. Internal Network is masked
  1. Difficult to maintain
  2. Expensive

 

Layer 5,6,7 protocols (higher level protocols)

Here are the protocols commonly used in higher levels (5,6,7 of the OSI model)

Layer 5 (Session):

  1. NetBIOS
  2. NFS
  3. RPC
  4. SSH
  5. SIP

Layer 6: (Presentation):

  1. ASCII
  2. ENCDIC
  3. MPEG
  4. JPG
  5. GIF

Layer 7 (Application):

  1. FTP,TFTP
  2. SNMP
  3. SMTP
  4. MIME, S/MIME
  5. HTTP,HTTPS,S-HTTP
  6. POP3,IMAP
  7. PEM
  8. TELNET
  9. S-RPC

IP address classes

IP (Internet protocol) is a Network Layer protocol (Layer 3) that considered ‘routed’ protocol. It addresses the Network Packets so that routing protocols like OSPF,BGP and RIP can correctly route the packet.

IP defines the IP addresses. IP address is a 32 bit number (4 octets). It comprises of Network and Host numbers. The higher order bits define the Network number as shown below.

There are 5 classes of IP addresses:

 

 

Class

Leading bits

Size of network
Number bit field

Size of rest bit field

Number of Networks

Addresses Per Network

Start address

End address

A

0

8

24

128 (27)

16,777,216 (224)

0.0.0.0

127.255.255.255

B

10

16

16

16,384 (214)

65,536 (216)

128.0.0.0

191.255.255.255

C

110

24

8

2,097,152 (221)

256 (28)

192.0.0.0

223.255.255.255


 

             

Class D is defined as Multicast. Address Range: 224 – 239

Class E is experimental. Address Range: 240 – 254

 

127.0.0.1 to 127.255.255.255 is defined as loop back address range.

 

Also, a range of IP addresses are reserved for Private use (i.e not routable in internet). They are

 

Class A

10.0.0.0 – 10.255.255.255

Class B

172.16.0.0 – 172.31.0.0

Class C

192.168.0.0 – 192.168.255.255

 

IPV6 uses 128 bit addresses and primarily introduced to address the depleting IPV4 addresses.