Some Elasticsearch wisdom I gleaned from Jason Wong and Mark Laney from Elastic. Contents Use cases Log stash vs Beats? Time Series vs Static Data Logstash Installation Starting and Stopping Elasticsearch Killing Communication Discovery module (networking) Security Read-only Enabling X-Pack (Elasticsearch Security) CRUD Ingestion Reading Search Query and Filter Contexts Mapping Inverted Index Multi Fields (keyword fields) Anatomy of an Analyzer Custom Analyzer The reindex API Node Types Cluster state Shards Anatomy of Search (Shards) Troubleshooting Configuration Responses Cluster and Shard Health Diagnosing Issues Improving Search Results Multi-field Search Boosting Fuzziness Exact Terms Sorting Paging Highlighting Aggregations Best Practices Index Aliases Index Templates Scroll Search Cluster Backup Use cases Search Logging Metrics - unlike logs, are typically not in a text format. Business analytics - the aggregation and analysis of patterns (e.g. bucketing aggregations, ML jobs) Security analytics - Log stash vs Beats? Beats are lightweight data shippers, but are not appropriate for ETL type stashing. Logstash on the other hand, can take handle these concerns. But requires a much heavier runtime (JVM). An official SIEM solution is currently under development. ...

Update 2019-05-26: While having some familarity with roff is useful, since the time of originally have discovered Pandoc which can convert between many document formats, such as from markdown to roff. Writing documentation in markdown is just hard to beat. As the first text formatting program built for UNIX in 1970 running on a PDP-7, roff was a port of the BCPL program RUNOFF. Over the years roff has evolved into troff and the excellent GNU version groff. Similar to other formatting systems, such as LaTeX, groff features a macro system, targetted at authoring certain types of text documents. In spite of its old age, roff is still in wide spread use today, for producing text such as UNIX manual pages, software books, system documentation and standards documents. ...

Intro Install Package (yum or deb) X-Pack Configuration Verify Index Patterns Intro Elastics node.js web frontend in the stack, and (by default) runs on port 5601. It’s wise to install Kibana on its own infrastructure (i.e. isolated from the Elasticsearch cluster). The node process is light (compared to the JVM anyway) consuming hundreds of MB. Install Package (yum or deb) While available as a tarball, the nicest option is to go with a package, takes care of plumbing such as systemd, and general system integration such as /etc/kibana for configuration, logs and data files. ...

An evolving list of resources around packet analysis tips and tricks. Terminology Cheat sheets Anatomy of a Packet OSI Model Layers Link Layer Ether Type Fields Network layer Transport layer Tools Wireshark (and tshark) tcpdump Useful switches Examples Real world use-cases netsniff-ng Analysis Terminology SOC, or Security Operations Center, is a central location composed of leading edge tools, technology and peeps (intel gatherers, analysts) that deals with security issues at an organisational and technical level. IDS, or Intrusion Detection System, is a device that monitors network traffic for threats to the environment, proactively alert the SOC analyst of potential problems. IPS, or an Intrusion Prevention System, is more sits inline, and can take active or passive mitigation actions. SIEM, or Security Information and Event Management, is all about the collection and aggregation of alerts and logs for event tracking, retention and correlation from multiple hosts. Cheat sheets SANS TCP/IP and tcpdump Pocket Reference Guide RFC 790 Assigned Internet Protocol Numbers RFC 791 Internet Protocol Anatomy of a Packet OSI Model Layers 7: application: HTTP, FTP, DNS 6: presentation: SSL, JPEG 5: session: SQL, SCP, NetBIOS, SOAP 4: transport: TCP, UDP 3: network: IPv4, IPv6, ICMP 2: data-link: PPP, ARP, CDP 1: physical: Ethernet, Bluetooth Layers 2-4 will be of primary interest. ...

Providing SSO by integrating Linux (or FreeBSD) with a directory service, like Microsoft Active Directory (AD), is no where as daunting as it once was, and highlights some fascinating subsystems that enable users to be defined from a variety of data sources (such as LDAP) other than just the traditional /etc/passwd file. Initial setup Kerberos Create service keytab on AD System Security Services Daemon (sssd) Name Service Switch (nss) PAM (Pluggable Authentication Module) Testing Listing Users Listing Groups id Troubleshooting Samba (smbd) Join Issues Clock Synchronisation Issues Clearing SSSD Cache End to end script (for Ansible) Initial setup Update /etc/resolv.conf to bind to the AD DNS server. This will enable realmd to discover and join the active directory domain (i.e. kerberos realm). ...

A quick walkthrough of Logstash, the ETL engine offered by the Elastic Stack. Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite stash Logstash gained its initial popularity with log and metric collection, such as log4j logs, Apache web logs and syslog. Its application has broadened, to all kinds of data sources like large scale event streams, webhooks, database and message queue integration. Once data is transformed and cleaned up is routed to a final destination (i.e. the stash), Elasticsearch is one option, but lots of other choices are there (mongo, S3, Nagios, IRC, email). ...

Contents Basic terminology Installation Java Distribution Package (yum or deb) tarball Configuration Core Memory Verify Firewall RHEL/Centos 7 The REST API Health Statistics List Indicies List Nodes Document Operations Create Index With Mappings Index Naming Conventions Populating an Index Auto Document Identifiers Retreiving Documents Existance Checking Updating Documents Deleting Documents Deleting an Index Reindexing an Index Aliasing an Index The Mapping API The Multi Get API The Bulk API Bulk Loading from JSON File Searching Background The TF/IDF Algorithm The Query DSL Query Context Filter Context Stateful vs Stateless Searching Multiple Indices Searching with Query String Params Search Multiple Indexes Searching using the Request Body Term Search Full Text Search Boolean Compound Queries Aggregations Metric Aggregations Cardinality Aggregation FieldData with The Mapping API Bucketing Aggregations Nested Aggregations Filter Aggregation X-Pack Security Kerberos with Active Directory krb5.conf Kerberos Principals (keytab fun) msktutil ktpass Elasticsearch Configuration X-Pack Setup Users Kibana Configuration Create Users Test it Troubleshooting gss context negotiation failure Defective token detected Kerberos clock skew Logon name already exists in the enterprise Creating Test Data Basic terminology Node is a single server within a cluster. Nodes perform the actual indexing and search work. Each node has a unique id and name. Cluster a collection of nodes that work together to achieve a shared goal. Is assigned a unique name, which by default is elasticsearch. This name is used to join nodes. Index is a collection of similar (not the same) documents, and is uniquely identified by name. By default every index is given 5 shards and 1 replica. Types represents an entity with a similar set of characteristics, and in essence are a way of partitioning documents up. For example book reviews and book comments could each be modelled as types. Document is the unit of information to be indexed. Represented as JSON. Every document must have a type and an index it belongs to. Shards are the division of an index across nodes. This enables the cluster to parallise the work of index store and retreival operations. Replicas clone shards across other nodes one or more times, providing high availability (in the event an individual shard node fails) and increasing search throughput. Installation Java Make sure an Oracle 8 or 10 JVM is available. Elastic 6.4.x interestingly JVM support matrix only supports four JVM’s. ...

A paper I prepared as part of the UNSW Master of Cyber Security. Alternately available as LaTeX and PDF. Abstract This paper discusses the motivations behind computer network exploitation, the general lifecycle of an attack operation, and the frictions and asymmetries that exist between both the attacker and the efender. One of the greatest challenges is fitting the ever-increasing and changing amount of information into a whole plan or framework to develop the right strategies to prevent such attacks. Armed with his knowledge seek out the creation of a structured general purpose framework for developing offensive strategies, the components described within it, its design philosophy, and how it can be used. It is eant to provide a concrete and structured approach to CNO strategy development. ...

Aliases Common tasks Undo-ing Diff-ing Pushing and pulling Patches Uncommon tasks Branch name in Bash prompt (PS1) Discover large commits in history Concepts Rebase vs Merge Merge Rebase Interactive rebasing (-i) Submodules Detached Head Aliases From my .gitconfig. [alias] alias = config --get-regexp '^alias.*' # list available aliases # add and commit a = add aa = add --all ac = !git add . && git commit -am ap = add -p c = commit --verbose ca = commit -a --verbose cm = commit -m cam = commit -a -m m = commit --amend --verbose uncommit = reset --soft HEAD~1 # branching ba = branch -a bd = branch -d bD = branch -D branches = branch --all branchrename = branch --move branchesdiffed = !git branch | grep -i "indiff" branchesundiffed = !git branch | grep -v -i "indiff" cleanmerged = !git branch --merged | grep -v \"\\*\" | xargs -n 1 git branch -d # remove merged branches co = checkout cob = checkout -b wipelocal = checkout . # diff d = diff ds = diff --stat dc = diff --cached f = fetch -p # remote p = push pr = pull --rebase pushitgood = push -u origin --all rao = remote add origin # rebase rb = rebase rba = rebase --abort rbc = rebase --continue rbs = rebase --skip # log st = status -sb plog = log --graph --pretty='format:%C(red)%d%C(reset) %C(yellow)%h%C(reset) %ar %C(green)%aN%C(reset) %s' tlog = log --stat --since='1 Day Ago' --graph --pretty=oneline --abbrev-commit --date=relative lg = log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit out = log --pretty=oneline --abbrev-commit --graph @{u}.. in = !git fetch && git log --pretty=oneline --abbrev-commit --graph ..@{u} rank = shortlog -sn --no-merges winning = shortlog --summary --numbered --email totalcommits = !git log --all --pretty=oneline | wc -l commitslastmonth = !git log --author=\"`git config user.name`\" --before={`date "+%Y-%m-01"`} --after={`date --date=\"$(date +%Y-%m-1) -1 month\" \"+%Y-%m-01\"`} --reverse --pretty=format:\"%cd %h %s\" --date=short commitsthismonth = !git log --author=\"`git config user.name`\" --before=now --after={`date "+%Y-%m-1"`} --reverse --pretty=format:\"%cd %h %s\" --date=short commitstoday = !git log --author=\"`git config user.name`\" --since=\"6am\" rank = shortlog -sn --no-merges Common tasks Undo-ing Dump all uncommitted changes and baseline the local files based on the latest committed change (HEAD): ...

Some fun I hacking on a boot to root challenge I did with a mate recently. Enumeration OS Fingerprint root@kali:~/boot2root# nmap -O 192.168.0.102 Starting Nmap 7.60 ( https://nmap.org ) at 2018-07-26 22:44 EDT Nmap scan report for 192.168.0.102 Host is up (0.00022s latency). Not shown: 986 closed ports PORT STATE SERVICE 21/tcp open ftp 80/tcp open http 135/tcp open msrpc 139/tcp open netbios-ssn 445/tcp open microsoft-ds 3389/tcp open ms-wbt-server 8009/tcp open ajp13 8080/tcp open http-proxy 49152/tcp open unknown 49153/tcp open unknown 49154/tcp open unknown 49155/tcp open unknown 49156/tcp open unknown 49157/tcp open unknown MAC Address: 00:50:56:A3:B7:92 (VMware) Device type: general purpose Running: Microsoft Windows 2008|Vista|7 OS CPE: cpe:/o:microsoft:windows_server_2008:r2:sp1 cpe:/o:microsoft:windows_vista::sp1:home_premium cpe:/o:microsoft:windows_7 OS details: Microsoft Windows Server 2008 R2 SP1, Microsoft Windows Vista Home Premium SP1, Windows 7, or Windows Server 2008 Network Distance: 1 hop OS detection performed. Please report any incorrect results at https://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 66.26 seconds A Windows box, running a bunch of services like ftp, two http servers, smb and ajp. ...