Posts on Benny Simmonds

Claude Code Gems

Fri, 10 Apr 2026 12:03:00 +1000

Insight Report

Generates a structured report about your codebase and how you’ve been working with Claude Code. It analyses your conversation history, tool usage patterns, and the types of tasks you’ve been tackling. Useful for understanding your own workflow patterns — where you spend time, what types of tasks you delegate most, and where friction tends to occur. Think of it as a retrospective on your AI-assisted development habits.

Claude Code Gems

Fri, 10 Apr 2026 12:03:00 +1000

Insight Report

Do It Now - Brian Tracy

Tue, 16 Dec 2025 17:23:00 +1100

Brian Tracy’s “Do It Now” philosophy is about overcoming procrastination by developing a sense of urgency, programming your subconscious with positive commands, setting clear goals, and acting quickly and consistently to achieve them, transforming thoughts into actions for success. He teaches that this mindset involves mental conditioning through repetition (like saying “Do it now”), focusing on goals, managing your environment, and cultivating a positive self-concept, leading to faster progress and a more fulfilling life.

Managing multiple SSH GitHub identities

Fri, 12 Dec 2025 08:07:00 +1100

Now I’m involved in multiple businesses, I’m finding I need to frequently juggle multiple SSH key pairs (aka identities) with choose your favourite git offering (e.g. GitHub). When I push and pull to origins on the same machine, I need to alternate the identities I use.

The best approach is to use an SSH config file to define host aliases. This lets you seamlessly use different keys without manually switching anything.

Entity Resolution and the Instability Problem

Tue, 02 Dec 2025 09:02:00 +1100

The Problem

The classic entity resolution gotcha: the thing that looks like a primary key (e.g. Senzing’s entity ID) is actually a volatile cluster ID that can legitimately change as the engine learns. Senzing explicitly says their resolved entity ID is not a globally unique persistent identifier and that it’s just an identifier for a grouping that may be transient. (senzing.zendesk.com)

Shadcn UI Quick Reference

Mon, 01 Dec 2025 17:45:00 +1100

wat

shadcn/ui is a set of beautifully-designed, accessible components and a code distribution platform. Works with your favorite frameworks and AI models. Open Source. Open Code.

components.json

Controls how CLI installs components, paths, and styling preferences.

AI Dev Setup

Sat, 18 Oct 2025 15:29:00 +1000

A place to document how I use AI to assist the development of software.

Context Is King

Managing the context window for your model is the most benefitial thing you can do as the human-in-the-loop.

Derivatives Trading

Sat, 11 Oct 2025 08:34:00 +1000

This post captures foundational knowledge that will help master Australian derivates trading.

Options Fundamentals

An option is a contract that gives you the right, but not the obligation, to buy or sell a stock at a specific price (called the strike price) before a certain date (the expiration date).

Business Cheat Sheet

Sun, 28 Sep 2025 11:52:00 +1000

A glossary of the key terms, acronyms, and ratios commonly considered business basics.

Finance and Metrics

Term	Description
Accrual Accounting	Records revenue when earned and expenses when incurred, regardless of cash movement - gives a truer picture than cash accounting
ARPU	Average Revenue Per User: Revenue ÷ total customers Useful in SaaS
B2B	Business to Business: Selling to other companies Often longer sales cycles, higher ticket
B2B2C	Business sells to another business that serves end consumers (eg, Shopify → merchants → buyers)
B2C	Business to Consumer: Selling directly to end users Usually higher volume, lower ticket
BAMFAM	Book A Meeting From A Meeting: Sales discipline to never leave a conversation without scheduling the next step [100M Closing Playbook §Closes]
Bottom Line	Net profit after all expenses; the final measure of profitability.
CAC	Customer Acquisition Cost: Total cost to acquire a new paying customer (ads + sales + marketing overhead ÷ new customers) [100M Leads p208]
Churn	% of customers who cancel in a given time period Inverse of retention [100M Retention Playbook §Churn Math]
Churn Rate	Percentage of customers lost over a specific period.
Close Rate	Percentage of leads or prospects that convert to paying customers.
COGS	Cost of Goods Sold: The direct cost of purchasing the goods a company sells, and it is a key expense that directly impacts profit. It includes expenses like raw materials, direct labor, and manufacturing overhead, but not indirect costs like marketing or salaries.
Cost to Value	The perceived value must significantly exceed the price paid; customers buy when the benefit outweighs the cost
CRO	Conversion Rate Optimisation: Improving the percentage of visitors who take a desired action (e.g., purchase, sign up).
EBITDA	Earnings Before Interest, Taxes, Depreciation, and Amortization: A cleaner view of profitability before financial structuring
FCF	Free Cash Flow: Cash left after operating expenses + capital expenditures Critical because cash pays bills, not “profits” [100M Money Models p233]
Funnel	The stages a prospect moves through before becoming a customer (e.g., awareness, interest, decision, action).
Grand Slam Offer	Offer so good people feel stupid saying no [100M Offers p22]
Gross Margin	(Revenue – Cost of Goods Sold) ÷ Revenue High gross margin = scalable business
Gross Margin / Gross Profit	The revenue remaining after deducting the Cost of Goods Sold (COGS).
Gross vs Net	Gross = before costs (top-line) Net = after costs (bottom-line)
Industry CAC Benchmark	Varies heavily Rule of thumb: aim for CAC ≤ 1/3 of LTGP [100M Leads p208]
Lead Magnet	A valuable piece of content or service (e.g., a free PDF guide, training video, or consultation) offered to a prospect in exchange for their contact information.
LTGP	Lifetime Gross Profit: LTV × Gross Margin % More accurate than LTV since it accounts for cost of goods sold [100M Money Models p228]
LTGP:CAC Ratio	Healthy is ~3:1 (spend $1 to acquire, get $3+ in gross profit) If <1:1, you’re losing money; if 10:1, you’re under-investing [100M Leads p208]
LTV	Lifetime Value: The total revenue expected from a customer over their relationship with you [100M Money Models p226]
Marketplace	Connects buyers and sellers, takes a cut (Airbnb, Uber)
Net Margin	Net profit ÷ Revenue Bottom-line profitability after all costs
Offer	The combination of a product or service, its price, payment terms, incentives (bonuses), and guarantees presented to a prospect.
Online Ads Attribution	The process of identifying which marketing touchpoints (e.g., specific ads, campaigns, or channels) were responsible for a customer’s conversion or purchase.
Outbound Outreach	Proactively reaching out to potential customers via calls, emails, or ads.
Recurring/Subscription	Predictable monthly revenue Gold standard for valuations
Return on Ad Spend (ROAS)	The revenue generated for every dollar spent on advertising. Calculated as: (Revenue from Ads / Cost of Ads).
Risk Reversal	Guarantees that shift risk from buyer to seller [Playbook – Guarantees §Hard Guarantees]
Sales Velocity	A measure of how quickly a business can move qualified leads through its sales pipeline and turn them into customers, and how much value those customers represent.
Scarcity/Urgency	Limiting supply or time to push action [100M Offers p113]
Ticket Price	The price paid for a product or service; often refers to average sale amount.
Top Line	Total revenue before any costs or expenses are deducted.
Transactional/One-off	One-time purchases, low predictability
Value Equation	(Dream Outcome × Perceived Likelihood of Achievement) ÷ (Time Delay × Effort & Sacrifice) [100M Offers p47]

Healthy Business Ratios

Ratio	Amount	Description
LTGP:CAC	> 3:1	3:1 is the minimum healthy rule of thumb [100M Leads p208]
Churn	< 5% monthly	Strong retention [100M Retention Playbook §Churn Math]
Payback Period	≤ 12 months	CAC recovered within a year [100M Lifetime Value Playbook §Payback]
EBITDA Margin	15–30%	Common range for service & SaaS businesses

Entity Resolution with Senzing and the .NET SDK

Fri, 19 Sep 2025 09:35:00 +1000

Context

The process of identifying and linking records that refer to the same real-world entity across different data sources, even when the records contain variations, errors, or incomplete information.

What Two Years Taught Us

Fri, 20 Jun 2025 13:13:00 +1000

What worked, what didn’t, and concrete recommendations for future projects. My in the trenches opinions, observations and reflections of building a complex distributed software system across multiple geographically dispersed teams.

Key Successes

Technical Achievements

Successfully built a big data horizontally scalable ingestion system using Kubernetes and leaned into cloud native approaches early on
Established heavy use of Python type hints early on, which improved code quality and editor aid
Evangelised Elasticsearch early in the design phase:
- Led the adoption of Elasticsearch for read workloads, in the face of aprehension and inexperience in the broader team
- Implemented and tuned sophisticated text analysis pipelines
- Optimised search with ngram tokenizers, stemming, and asciifolds
- Designed efficient denormalised document structures and indexing strategies
- Lesson learned how important it is to make the the most appropriate data storage and management choices, make or break analytic solutions such as the one we collectively built. What consistency guarantees do are required? How fast? How are you going to calculate aggregations? What kind of read or write workloads need to be handled? Can these be separated and tackled as different problems?
- Elasticsearch is a HUGE reason why we were successful
Created flexible hierarchy layering design, allowing differing customers to stamp the data with their own ways of doing things.
Integrated OpenTelemetry for comprehensive observability
Developed optimistic locking scheme and deep linking capabilities
Automated deployments and quality verification with a gigantic test suite investment (unit and integration), linting, autoformatting, all orchestrated with a Makefile frontend and Bamboo CI pipeline
The team embraced containers heavily from day 1. From running local vendor infra containers (redis, mongo, elasticsearch, etc) to running repeatable build workloads.

Process Improvements

Adopted Make for development automation, significantly boosting productivity
Leveraged code generation effectively for complex scenarios, an ever powerful technique
Implemented comprehensive integration testing with containerization
Successfully broke down the system into functional components early
Established well-defined data schemas upfront, which provided stability

Project Management Challenges

This project mangaged to defy every Agile Manifesto Principle.

How to Win Friends and Influence People

Wed, 11 Jun 2025 20:10:00 +1000

Dale Carnegie’s flagship “How to Win Friends and Influence People” is a foundational book on communication, relationship-building, and personal effectiveness. First published in 1936, its principles remain relevant for anyone seeking to improve their social skills, leadership, and influence.

Self taught MBA

Mon, 02 Jun 2025 19:12:01 +1000

This reading list covers the fundamentals of wealth building, personal development, income generation, business scaling, and inspiration from successful entrepreneurs. Each book offers practical insights and strategies for building business acumen and financial success.

🧠 Personal Development

How to Win Friends and Influence People by Dale Carnegie
Essential communication skills and understanding people’s motivations for effective interaction.

The 7 Habits of Highly Effective People by Stephen R. Covey Seven habits for increased efficiency and effectiveness, including “begin with the end in mind.”

.NET Clean Architecture

Thu, 29 May 2025 20:14:01 +1000

Domain centric architectures, like clean architecture, have inner architectural cores that model the domain. Dependency inversion is king, with inner layers defining abstractions and interfaces and outer layers implementing them. Clean architecture is a good fit when aligning to Domain Driven Design (DDD), dealing with complex business logic, high testability is desirable and/or working in a large team, as the architecture can enforce design policies.

Guiding Principles

High level qualities that a good software architecture should (and enforce) strive for; maintainability, testability and loose coupling.

Pro CSS

Sat, 10 May 2025 17:55:56 +1000

Modern CSS Fundamentals

Baseline

Baseline features are ones that are supported by all the major browsers (Chrome, Edge, Safari, and Firefox). Both the MDN web docs and caniuse show when a feature has reached either the Newly available or Widely available threshold.

Zero to One: Notes on Startups and Building the Future

Sun, 09 Mar 2025 16:47:56 +1100

Zero to One is a book by Peter Thiel and Blake Masters about the value of true innovation made accessible to the masses through startups. It outlines several tenets that keen-minded business people should hold dear, including why technology trumps globalization, why we should be supporting monopolies instead of “healthy competition”, why successful innovators have the worldview of a “definite optimist” and why no one should be afraid of losing their job to a robot. Zero to One also delivers unique business insights, such as the four most important things to pay attention to about your product (they’re not quantitative) and the seven questions every business must answer for itself.

Reflections on ElasticON Sydney 2025

Fri, 07 Mar 2025 07:16:56 +1100

ElasticON in Sydney this March was a packed day, blending technical deep dives with executive-level discussions. The event featured interviews with partners and customers, but the real highlight was the keynote from Ken Exner and Baha Azarmi. Their presentation was slick, showcasing cutting edge GenAI features across the Elastic stack, including the new semantic_text field type, RRF, BBQ, LogsDB mode, and the ability to ETL unstructured data onto ECS using an LLM. The introduction of the ESQL query engine with native joins was another game changer.

Effective Study Techniques

Sat, 01 Mar 2025 15:52:56 +1100

Studying effectively isn’t just about putting in the hours, it’s about using the right techniques. Research shows that some popular methods, like re-reading notes or highlighting, are far less effective than people assume. Instead, techniques that leverage how our brains naturally process and retain information yield far better results.

The Problem with Common Study Habits

Ineffective techniques: Reading over and over, highlighting, using mnemonics for simple memorization, starting early without a plan, reading notes before and after class, listening to music while studying.

Stubborn Attachments: A Vision for a Society of Free, Prosperous and Responsible Individuals

Sat, 15 Feb 2025 11:55:56 +1100

Stubborn Attachments is a book by Tyler Cowen that paints a picture of an altruistic and bright future for the human race. Cohen argues and highlights the importance of sustained economic growth, moral philosophy and consequentialism, radical uncertainty, rules-based ethics, that future generations matter deeply, and how we as a race must be stubbornly attached to certain core principles as the necessary pre-conditions for sustainable prosperity. To be honest I avoid books like this, because they are hard. I mean, where do you start pulling apart such a broad and complicated topic. I thank Cohen for expanding my thinking towards civilization and painting the broad brushstrokes of a future pathway our race must strive for.

Hiking Retrospectives

Sun, 09 Feb 2025 15:38:01 +1100

Getting out on the trail is one my favourite things to do, especially long multi-day hikes. I’ve got into the habbit of reflecting after each adventure, what worked well, what didn’t, potential opportunities or tips I discovered from other more experienced hikers. In software, some agile methods refer to this act as a retrospective and its a powerful way to get better at something.

“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly” - Agile Manifesto, 2001

Certified Kubernetes Administrator (CKA)

Sun, 19 Jan 2025 20:14:01 +1100

Glossary

Term	Definition
cri	Container Runtime Interface, the contract between k8s and the container runtime
crictl
ctr	Debugging tool for containerD
oci	Open Container Initiative, formalised the specification of an `imagespec` and a `runtimespec`
nerdctl	Docker like CLI experience for containerD

Certification tips

Bookmarks

https://kubernetes.io/docs/reference/kubectl/conventions/

Rubiks Cube Algo

Thu, 26 Dec 2024 17:15:01 +1000

A fun memorable algorithm to teach my 5yo nephew how to tackle a 3 sided rubik’s cube.

Step 1: Make a Daisy

Start by creating a white plus sign (the petals) on the yellow face (center).
Align each white petal’s adjacent color with its matching middle-row center color.
Once matched, rotate the petal 180° to move it to the bottom face.
- Repeat this process for all petals until you form a white cross on the bottom face.

Step 2: Fill in the White Corners (“Chase the White Dogs”)

Find any white corner pieces on the bottom layer (the “dogs”).
Rotate the bottom layer to position a white corner directly below its matching color. For example, a white/green corner should be under the green side.
Bring the corner into the white face by rotating the side it’s under downward, “scooping” the corner piece into place.
- Repeat until the entire white face is completed, and the bottom row edges align with their center colors.

Step 3: Solve the Middle Row

Looking at the bottom and sides (white layer is top), find middle cells on the bottom layer that don’t contain any yellow (e.g., an orange/blue edge).
Move the middle cell (e.g. orange/blue) by rotating the bottom axis, until its side face color matches (e.g. a cell with an orange side and blue bottom should sit below the mostly orange sided-side).
Determine the target direction the piece needs to move (e.g. for an orange/blue if the mostly blue sided-side is on the left, then the target direction is left): either left or right
Follow this sequence to place it:
- Turn the bottom layer away from the target direction.
- Looking at the bottom layer head on, rotate the bottom layer towards the target direction.
- Looking at the bottom layer head on, rotate the target direction (e.g. left) column downward.
- Looking at the bottom layer head on, rotate the middle (sandwich) layer toward the target direction.
- Looking at the bottom layer head on, rotate the target direction (e.g. left) column upward.
- Restore the white top face as needed.
- Repeat until all middle-row edges are in place.

Step 4: The Yellow Cross

Use the FUR-RUF sequence:

Elasticsearch Performance and Tuning

Fri, 07 Jun 2024 18:46:01 +0000

A dedicated performance course run by Matt Gregory from Elastic, an absolute legend with deep Elasticsearch expert.

Contents

Cool takeaways

Increase the refresh_interval from default 1s to something higher, like 10s.
Index typings should be set to strict (default is dynamic)
The took param measures raw cluster operation speed, kibana will also reveal a roundtrip time which includes the HTTP layer.
Auto generated id’s are always faster
One of Matt’s favourite APIs _cluster/allocation/explain
Ensure the heap is beefed up
a must clause is the first line of defence for scoring, should is then used as the second pass of scoring
always format queries as a ‘bool’
Configuration management everywhere (Ansible, etc)
dedicated monitoring cluster

Tuning for Index Speed

Cheatsheet:

Elasticsearch Engineer 8.1

Sun, 02 Jun 2024 18:46:01 +0000

Revised 2024 edition based on Elasticsearch 8.1.

Recently the opportunity to attend the latest revision of the 4-day Elasticsearch engineer course, which I did in-person about 5 years ago in Sydney. Elasticsearch has often been an integral part of the data solutions I’ve been involved with and I’m quite fond of it. This time round the course only runs in a virtual class room format (using strigo.io) with our awesome trainers Krishna Shah and Kiju Kim.

Kustomize

Fri, 03 May 2024 22:17:30 +0000

Kustomize is built into kubectl with -k. Great samples on kubernetes.io/docs

Kustomize provides a template-free way to customize kubernetes manifests

Contents:

In a nutshell provides 3 key features:

generating resources from other sources
setting cross-cutting fields for resources
composing and customizing collections of resources

Generating resources

To generate a ConfigMap from an .env file, add an entry to the envs list in configMapGenerator. Kustomize supports other formats such as .properties.

Python Packaging (2024)

Fri, 29 Dec 2023 17:29:33 +1100

sysconfig

The process of bundling Python code into a format that eases distribution and sharing. First up, I find it helps to get a concrete understanding of how the specific python distro I’m working with is configured, of paricular interest are the various system paths that will be visited for package dependencies. The built-in sysconfig module neatly manages and surfaces this information.

python -m sysconfig

On a Windows system:

Kubernetes Certified Administrator (CKA) 2024

Fri, 22 Dec 2023 20:11:30 +0000

CKA topics

Cluster Architecture, Installation & Configuration: How to set up and configure a Kubernetes cluster, including how to install and configure a Kubernetes cluster using kubeadm, how to upgrade your cluster version, how to backup and restore an etcd cluster, and how to configure a pod to use secrets
Workloads & Scheduling: How to deploy a Kubernetes application, create daemonsets, scale the application, configure health checks, use multi-container pods, and use config maps and secrets in a pod. You’ll also need to know how to expose your application using services
Services & Networking: How to expose applications within the cluster or outside the cluster, how to manage networking policies, and how to configure ingress controllers
Storage: How to create and configure persistent volumes, how to create and configure persistent volume claims, and how to expand persistent volumes
Troubleshooting: How to troubleshoot common issues in a Kubernetes environment, including how to diagnose and resolve issues with pods, nodes, and network traffic

Kubernetes in a nutshell

Control plane management components that mother-hen nodes and pods. Key components:

Python PR Checklist

Mon, 30 Oct 2023 20:44:33 +1100

A modified version of excellent original checklist by Paul Wolf

General

Code is blackened with black
ruff has been run with no errors
mypy has been run with no errors
Function complexity problems have been resolved using the default complexity index of flake8.
Important core code can be loaded in iPython, ipdb easily.
There is no dead code
Comprehensions or generator expressions are used in place of for loops where appropriate
Comprehensions and generator expressions produce state but they do not have side effects within the expression.
Use zip(), any(), all(), filter(), etc. instead of for loops where appropriate
Functions that take as parameters and mutate mutable variables don’t return these variables. They return None.
Return immutable copies of mutable types instead of mutating the instances themselves when mutable types are passed as parameters with the intention of returning a mutated version of that variable.
Avoid method cascading on objects with methods that return self.
Function and method parameters never use empty collection or sequence instances like list [] or dict {}. Instead they must use None to indicate missing input
Variables in a function body are initialised with empty sequences or collections by callables, list(), dict(), instead of [], {}, etc.
Always use the Final type hint for class instance parameters that will not change.
Context-dependent variables are not unnecessarily passed between functions or methods
View functions either implement the business rules the view is repsonsible for or it passes data downstream to have this done by services and receives non-context dependent data back.
View functions don’t pass request to called functions
Functions including class methods don’t have too many local parameters or instance variables. Especially a class’ __init__() should not have too many parameters.
Profiling code is minimal
Logging is the minimum required for production use
There are no home-brewed solutions for things that already exist in the PSL (python standard library)

n00b habbits

Bare except clause, Python uses exceptions to flag system level interupts such as sigkills. Don’t do this.
Argument default mutatable arguments such as def foo(bar=[]) are defined when the function is defined, not when its run, and will result in a all function calls sharing the same instance of bar
Checking for equality using ==. Due to inheritance this is not desirable as it pins to a concrete type and not potentially it descendents. In other words the Liskov substitution principle. Instead isinstance(p, tuple)
Explicit bool or length checks, such as if bool(x) or if len(x) > 0 is redundant, as Python has sane truthy evaluation.
Use of range over the for in idiom
If you really need the index, always use enumerate
Not using items() on a dict for k, v in dict.items())
Using time.time to measure code performance. Use time.perf_counter instead.
Using print statements over logging
Using import * will normally liter the namespace with variable. Dont be lazy, be specific. from itertools import count

Imports and modules

Imports are sorted by isort or according to some standard that is consistent within the team
Import packages or modules to qualify the use of functions or classes so that unqualified function calls can be assumed to be to functions in the current module

Documentation

Modules have docstrings
Classes have docstrings unless their purpose is immediately obvious
Methods and functions have docstrings
Comments and docstrings add non-obvious and helpful information that is not already present in the naming of functions and variables

General Complexity

Functions as complex as they need to be but no more (as defined by flake8\ ’s default complexity threshold)
Classes have only as many methods as required and have a simple hierarchy

Context Freedom

All important functionality can be loaded easily in ipython without having to construct dummy requests, etc.
All important functionality can be loaded in pdb (or a variant, ipdb, etc.)

Types

Use immutable types ()tuple, frozenset, Enum, etc) over mutable types whenever possible

Functions

Functions are pure wherever possible, i.e. they take input and provide a return value with no side-effects or reliance on hidden state.

Modules

Module level variables do not take context-dependent values like connection clients to remote systems unless the client is used immediately for another module level variable and not used again

Classes

Every class has a single well-defined purpose. That is, the class does not mix up different tasks, like remote state acquisition, web sockets notification, data formatting, etc.
Classes manage state and do not just represent the encapsulation of behaviour
All methods access either cls or self in the body. If a method does not access cls or self, it should be a function at module level.
@classmethod is used in preference to @staticmethod but only if the method body accesses cls otherwise the method should be a module level function.
Constants are declared at module level not in methods or class level
Constants are always upper case
Abstract classes are derived from abc: from abc import ABC
Abstract methods use the @abstractmethod decorator
Abstract class properties use both @abstractmethod and @property decorators
Classes do not use multiple inheritance
Classes do not use mixins (use composition instead) except in rare cases
Class names do not use the word “Base” to signal they are the single ancestor, like “BaseWhatever”
Decorators are not used to replace classes as a design pattern
__init__() does not define too many local variables. Use the Parameter Consolidation pattern instead.
A factory class or function at module level is used for complex class construction (see Design Patterns) to achieve composition
Classes are not dynamically created from strings except where forward reference requires this

Design Patterns

Do not use designs that cause a typical Python developer to have to learn new semantics that are unexpected in Python
Classes primarily use composition in preference to inheritance
Beyond a very small number of simple variables, a class’ purpose is to acquire state for another class or it uses another class to acquire state in particular if the state is from a remote service.
If you use the Context Parameter pattern, it is critical that the state of the context does not change after calling its __init__(), i.e. it should be immutable
If a class’ purpose is to represent an external integration, you probably want numerous classes to compose the service: RemoteDataClient, DomainManager, ContextManager, Factory, NotificationController, DomainResponse, DataFormatter and so on.

Async Python

Wed, 09 Aug 2023 21:12:33 +1100

Background

Using asyncio will not make your code multi-threaded. That is, it will not cause multiple Python instructions to be executed at the same time, and it will not in any way allow you to side step the so-called “global interpreter lock” (GIL).

Some processes are CPU-bound: they consist of a series of instructions which need to be executed one after another until the result has been computed. Most of their time is spent making heavy use of the processor.

Python Type Annotations

Wed, 09 Aug 2023 20:44:33 +1100

Start with the docs and the Type hints cheat sheet

Topics for consideration:

syntax shorthands e.g. | for Union or Optional
Self
If you are using the typing library then there is an abstract type class provided for asynchronous context managers AsyncContextManager[T], where T is the type of the object which will be bound by the as clause of the async with statement.
mypy

If you are using typing then there is an abstract class Awaitable which is generic, so that Awaitable[R] for some type R means anything which is awaitable, and when used in an await statement will return something of type R.

Python Standard Libraries

Fri, 04 Aug 2023 18:01:33 +1100

An important part of becoming “good” at a language is becoming familiar with its library eco-system.

The official Python Standard Library reference manual rocks.

Module	Category	Description
`argparse`		functions for parsing command line arguments
`atexit`		allows you to register functions for your program to call when it exits
`bisect`		bisection algorithms for sorting lists (see Chapter 10)
`calendar`		a number of date-related functions
`codecs`		functions for encoding and decoding data
`collections`		a variety of useful data structures
`concurrent`		asynchronous computation
`copy`		functions for copying data
`csv`		functions for reading and writing CSV files
`datetime`		classes for handling dates and times
`fileinput`	file access	iterate over lines from multiple files or input streams
`fnmatch`		functions for matching Unix-style filename patterns
`glob`		functions for matching Unix-style path patterns
`io`		functions for handling I/O streams and StringIO, which allows you to treat strings as files.
`json`		functions for reading and writing data in JSON format
`logging`		access to Python’s own built-in logging functionality
`multiprocessing`		allows you to run multiple subprocesses, while providing an API that makes them look like threads
`operator`		functions implementing the basic Python operators, instead of writing your own lambda expressions
`os`	swiss army knife	access to basic OS functions
`pprint`	data types	data pretty printer
`random`		functions for generating pseudorandom numbers
`re`		regular expression functionality
`sched`		an event scheduler without using multithreading
`select`		access to the select() and poll() functions for creating event loops
`shutil`	file access	access to high-level file functions
`signal`		functions for handling POSIX signals
`tempfile`	file access	functions for creating temporary files and directories
`threading`		access to high-level threading functionality
`urllib`		provides functions for handling and parsing URLs
`uuid`		allows you to generate Universally Unique Identifiers (UUIDs)

Objects in Python

Thu, 03 Aug 2023 16:28:33 +1100

As I learn more about Pythons idioms reflect on its unique approach to object based programming. In combination with duck typing its approach to objects feels distrubingly flexible.

Testing in Python

Thu, 03 Aug 2023 16:28:33 +1100

There are many ways to write unit tests in Python.

unittest

Here the focus is living off the land with built-in unittest.

unittest is both a framework and test runner, meaning it can execute your tests and return the results. In order to write unittest tests, you must:

Write your tests as methods within classes
These TestCase classes must subclass unittest.TestCase
Names of test functions must begin with test_
Import the code to be tested
Use a series of built-in assertion methods

Basic example

import unittest

class TestStringMethods(unittest.TestCase):

    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

if __name__ == '__main__':
    unittest.main()

Assertions

The TestCase class provides several assert methods to check for and report failures.

Digital Forensics

Sat, 22 Jul 2023 16:13:36 +1100

It’s semester 2 2023 and time for my final subject in the UNSW Cyber Security Masters course, digtital forensics run by Seth Enoka.

I got to venture deep into Windows internals, including core Windows memory structures, subsystems such as prefetch and shimcache, NTFS file system internals and mechanicsm including MFT analysis and much more. All this analysis was conducting using the following Linux analysis tools:

Tools

Tools	Description
Yara	A pattern-matching tool used in malware research and forensic analysis to identify and classify files based on defined rules and signatures.
Volatility 2 & 3	Open-source memory forensics frameworks used to extract and analyze digital artifacts from volatile memory (RAM) in a memory dump to investigate cyber incidents and malware.
Volatility USNParser Plugin	A Volatility plugin specifically designed to parse and extract information from the USN journal on Windows systems, aiding in file activity analysis.
SCCA Tools	SCCA (Source Code Control System Analysis) Tools assist in examining version control system repositories to identify code changes, contributors, and track project history.
ESEDB Tools	These tools provide access to Extensible Storage Engine (ESE) Database files, commonly used in Windows applications, for analysis and recovery purposes.
analyzeMFT	A tool used in digital forensics to parse and analyze the Master File Table (MFT) entries from NTFS filesystems, revealing information about files and directories.
Oletools	A collection of Python-based tools for analyzing and extracting data from OLE (Object Linking and Embedding) files, such as Microsoft Office documents, often used in malware analysis.
Wireshark	A widely-used network protocol analyzer that captures and inspects data packets on a network, helping with network troubleshooting, security analysis, and protocol reverse engineering.
The Sleuth Kit (TSK)	An open-source digital forensic toolkit that includes various CLI tools (`mmls`, `fls`, `icat`) for file system analysis and data recovery from different operating systems.
Plaso	An open-source Python-based tool used for super timeline creation and log analysis, helping to reconstruct events and activities from various data sources for forensic investigations.
Advanced Forensics Format Library (afflib) Tools	Tools for working with the Advanced Forensics Format (AFF), an extensible open file format used in computer forensics to store disk images and related metadata.
wxHexEditor	A hexadecimal editor with a graphical user interface, used for low-level data inspection and editing in forensic analysis and reverse engineering.
Gnumeric	A spreadsheet application, similar to Microsoft Excel, used for data analysis and visualization, including data manipulation and statistical functions.
Personal Folder File Tools (pfftools)	Tools designed to work with Personal Folder File (PFF) formats, commonly used by Microsoft Outlook to store emails, calendars, and other personal data. These tools aid in email forensics and analysis.

Resources

Windows shellbags
8 timestamps on an NTFS file system, an attacker can fairly easily mutate 4 of them, hard to convincingly adjust nano-second level
Eric Zimmermans Windows Forensics Tools
SANS Hunt Evil Poster Knowing what’s normal on a Windows host helps cut through the noise to quickly locate potential malware. Use this information as a reference to know what’s normal in Windows and to focus your attention on the outliers.
MITRE ATT&CK
MITRE ATT&CK for ICS
Cyber Kill Chain
Industrial Cyber Kill Chain
Locard’s Exchange Principle
NIST Guide to Forensics in Incident Response
Dragos Threat Groups
Crowdstrike Adversary Groups
Diamond Model for Intrusion Analysis
The Four Types of Threat Detection
Volatility v2.4 cheat sheet

Module 0 - Intro

Locards Principle (Edmond Locard aka Sherlock Holmes of France)

Python 3.11

Mon, 17 Jul 2023 21:07:33 +1100

Cool new features in 3.11.

Performance

1.2x faster generally, thanks to an adaptive interpreter (PEP659) that optimises byte-code based on observed behaviour and usage.

Take for example the LOAD_ATTR instruction, which under 3.11 can be replaced by LOAD_ATTR_ADAPTIVE.

This will replace the call to the most optimised instruction based on what is being done, such as:

LOAD_ATTR_INSTANCE_VALUE
LOAD_ATTR_MODULE
LOAD_ATTR_SLOT

Disassembling some code:

def feet_to_meters(feet):
    return 0.3048 * feet

for feet in (1.0, 10.0, 100.0, 1000.0, 2000.0, 3000.0, 4000.0):
    print(f"{feet:7.1f} feet = {feet_to_meters(feet):7.1f} meters")

import dis
dis.dis(feet_to_meters, adaptive=True)
#  1           0 RESUME                   0
#
#  2           2 LOAD_CONST               1 (0.3048)
#              4 LOAD_FAST                0 (feet)
#              6 BINARY_OP                5 (*)
#             10 RETURN_VALUE

However, when the interpreter is given more concrete to work with its able to optimise. For example, outside the loop context when given a float, floating point instructions are put to work:

Information Assurance

Sat, 04 Mar 2023 13:22:36 +1100

Kicking off the 2023 University year I continue my journey into the Cybersecurity Masters program with unit Infomation Assurance and Security run by Michael McGarity and Huadong Mo.

Provides students with a deep understanding of the technical, management and organisational aspects of Information Assurance within a holistic legal and social framework.

The course is essentially modelled off the CISSP certification, which dives into the following subjects:

make a realistic assessment of the needs for information security in an organisation
discuss the implications of security decisions on the organisation’s information systems
understand the principles of writing secure code
show an understanding of database and network security issues
demonstrate an understanding of encryption techniques
understand foundations of the tools and techniques in computer forensics
show an appreciation of the commercial, legal and social context in which IT security is implemented
apply knowledge gained to business and technical IA scenarios

Intro

Not a one size fits all approach. Too many factors and seemingling chaotic variables, such as risk appetites, country legislation, the business vertical (mining vs banking vs government), acceditation frameworks that apply to certain industries, tolerances, technology limitations, and so on.

Vue

Thu, 02 Mar 2023 20:45:36 +1100

A bunch of (scattered) tips and resources as I experiment with Vue.

Basics: General wisdom Anatomy Eventhandling Watchers Computed props
Components: Components Props Lifecycle hooks Emitting events Slots
Fetching Data: Calling APIs in hooks Unique identifiers
Styling Components: Global vs scoped styles CSS modules CSS v-bind
Composition API: Composition API Reactive references script setup Composables
Routing and Deployment: Vue Router History Dynamic routes Deployment
Advanced: Pre-processors Pinia State Management

Overview

What is Vue?

an open-source model–view–viewmodel front end JavaScript framework for building user interfaces and single-page applications, created by Evan You

Helpful resouces:

General wisdom

It’s best to stick to conventions of the web and use camelCase in your script and kebab-case in your template
Don’t pass functions as props, instead emit events
props couples components to each other, for broad or deep cross cutting state, level up to state management
Test data sources: JSON Placeholder PokeAPI

Anatomy

Here is a bare bones vue app. There are literally 3 blocks for script, template (markup) and style:

Kinesis 360 Pro keyboard

Sun, 26 Feb 2023 14:44:36 +1100

Kinesis is a company based near Seattle that offers computer keyboards with ergonomic designs as alternatives to the traditional keyboard design. Most widely known among these are the contoured Advantage line, which features recessed keys in two bucket-like hollows to allow the user’s fingers to reach keys with less effort

The Advantage 360 line was released in 2022 and is still insanely popular and challenging to get hold of. The pro edition allows you to customise the firmware, which is ZMK based. Kinesis have outsourced the actual job of compiling the firmware to GitHub Actions.

Books

Fri, 27 Jan 2023 20:42:36 +1100

2023

To Kill a Mockingbird by Lee, Harper
Own the Day, Own Your Life: Optimised practices for waking, working, learning, eating, training, playing, sleeping and sex by Marcus, Aubrey

2022

The Dream Machine by Waldrop, M. Mitchell
Absolute FreeBSD by Lucas, Michael W.
Deep Work: Rules for Focused Success in a Distracted World by Newport, Cal

2021

Revolution in The Valley: The Insanely Great Story of How the Mac Was Made by Hertzfeld, Andy
Kingpin: How One Hacker Took Over the Billion-Dollar Cybercrime Underground by Poulsen, Kevin
The New New Thing: A Silicon Valley Story by Lewis, Michael
Ghost in the Wires: My Adventures as the World’s Most Wanted Hacker by Mitnick, Kevin D.
Bonhoeffer: Pastor, Martyr, Prophet, Spy by Metaxas, Eric *
1984 by Orwell, George
Atomic Habits: An Easy and Proven Way to Build Good Habits and Break Bad Ones by Clear, James *
Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel by Potts, Rolf
American Kingpin: The Epic Hunt for the Criminal Mastermind behind the Silk Road Drugs Empire by Bilton, Nick *
The God Delusion by Dawkins, Richard

2020

Into the Wild (Ebook) by Krakauer, Jon *
The Last Wish (The Witcher, #0.5) by Sapkowski, Andrzej
Do Androids Dream of Electric Sheep? by Dick, Philip K.
Alan Turing: The Enigma by Hodges, Andrew

2019

Harry Potter and the Philosopher’s Stone (Harry Potter, #1) by Rowling, J.K.
Ultralight Backpackin’ Tips: 153 Amazing & Inexpensive Tips for Extremely Lightweight Camping by Clelland, Mike *
Turn The Ship Around! by Marquet, L. David

2018

Microserfs by Coupland, Douglas *
Autobiography: The Story of My Experiments with Truth by Gandhi, Mahatma
Influence: The Psychology of Persuasion by Cialdini, Robert B. *
The Mythical Man-Month: Essays on Software Engineering by Brooks Jr., Frederick P.

2017

The Cuckoo’s Egg: Tracking a Spy Through the Maze of Computer Espionage by Stoll, Clifford
Savaged by Systemd: an Erotic Unix Encounter by Lucas, Michael Warren

Kinetic workouts

Tue, 27 Dec 2022 19:55:30 +1100

Categorises the gym workout by these kinetic systems:

Power (3 mins)
Strength (5 mins)
Endurance (8 mins)
Cardio (10 mins)
Durability (15 mins)

Every 50 minute workout includes each of them. This is really just a catalogue of kettlebell and/or body weight exercises.

Basic program 1

Cardio (10 mins) of choice run, bike, row, stair mill, jump jop

Mobility (10 mins) circuit of:

Kettlebell windmills (5 reps each side)
Kettlebell halos (5 reps each side) variation options include lunges or squats

Power (3 mins) Kettlebell sumo squat 6 sets of 10 secs on, 20 secs rest

Python quick reference

Sun, 05 Jun 2022 13:57:33 +1100

Forked from 101t/python-cheatsheet.md

RTFM

The Python Standard Library
Built-in Functions
Python Enhancement Propsoals (PEPs)
The Zen of Python never far away in the REPL import this

Getting started: CPython, Easter eggs, Import paths, venv
Collections: List, Dictionary, Set, Tuple, Range, Enumerate, Iterator, Generator
Functions: Functions, Modules
Types: Type, String, Regular_Exp, Format, Numbers, Combinatorics, Datetime
Syntax: Args, Splat, Inline, Closure, Decorator, Class, Duck_Type, Enum, Exception
System: Exit, Print, Input, Command_Line_Arguments, Open, Path, OS_Commands
Data: JSON, Pickle, CSV, SQLite, Bytes, Struct, Array, Memory_View, Deque
Advanced: Threading, Operator, Introspection, Metaprograming, Eval, Coroutines
Libraries: Progress_Bar, Plot, Table, Curses, Logging, Scraping, Web, Profile, NumPy
Packaging and Tools: Real app, Bytecode disassembler, Poetry, Gems, Resources

CPython

Most distros lag behind the latest releases of python. Its quite a pleasant experience to just build CPython from source, as per the docs:

sudo apt update
sudo apt install build-essential gdb lcov pkg-config \
      libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
      libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
      lzma lzma-dev tk-dev uuid-dev zlib1g-dev
./configure
sudo make
sudo make install
sudo python3 -m pip install --upgrade pip setuptools wheel

Easter eggs

import this
import antigravity
from __future__ import braces

Import paths

When importing modules, Python relies on a list of paths to know where to look for the module. This list is stored in the sys.path variable.

PowerShell cheat sheet

Sun, 08 May 2022 16:35:04 +1100

A work in progress.

Credits:

Help and context

$PSVersionTable.PSVersion         # what version

Get-Command                       # list commands available
Get-Command -Module Microsoft*    # list commands exported from modules named Microsoft*
Get-Command -Name *item           # wildcard search commands

Get-Help
Get-Help -Name about_Variables
Get-Help -Name Get-Command
Get-Help -Name Get-Command -Parameter Module

Execution policy

Levels = {Restricted,Remote Signed,All Signed,Unrestricted}

A tour of FreeBSD in 2022

Sun, 17 Apr 2022 16:18:25 +1100

Why
Handy resources
Talks
Factoids
Things that feel different
My setup
- NFS
- xorg
- dwm
- st
- powerd++
- ntp
- fonts

Why

Mostly curiosity. I’ve been a GNU/Linux and free software enthuiatist since the late 90’s and that’s not about to change. More concrete reasons:

GNU/Linux won the war. However…
The Berkeley Software Distributions are family of open UNIX-like operating systems with lineage back to AT&T UNIX
They come as complete and cohesive systems, not a distribution of disparate pieces
The BSD license is wonderfully permissive
Bill Joy (the father of Sun, BSD, vi, RISC)
Marshall Kirk McKusick
ZFS, DTrace and Clang
It feels so damn neat, tidy and robust
It wont be my daily driver, however it’s nice to be aware of what {Free,Open,Net}BSD each bring to the table

Handy resources

FreeBSD Handbook
FreeBSD Porters Handbook
FreeBSD Wiki
Freshports the place for ports and most recent commits
bsd-hardware.info
papers.freebsd.org the collected works of the FreeBSD community as presented at various conferences and summits

Talks

Factoids

The term daemon comes from Greek mythology and represents an entity that is neither good nor evil, and which invisibly performs useful tasks. This is why the BSD mascot is the cheerful-looking daemon with sneakers and a pitchfork.

Things that feel different

My Linux biases are deep rooted. It’s so fascinating to learn how the BSD’s have gone about things differently.

The Conscience of a Hacker (1986)

Wed, 13 Apr 2022 17:24:04 +1100

This is an amazing piece of hacker folklore was created on January 8, 1986 by The Mentor.

Source: http://phrack.org/issues/7/3.html

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
The following was written shortly after my arrest...

                       \/\The Conscience of a Hacker/\/

                                      by

                               +++The Mentor+++

                          Written on January 8, 1986
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

        Another one got caught today, it's all over the papers.  "Teenager
Arrested in Computer Crime Scandal", "Hacker Arrested after Bank Tampering"...
        Damn kids.  They're all alike.

        But did you, in your three-piece psychology and 1950's technobrain,
ever take a look behind the eyes of the hacker?  Did you ever wonder what
made him tick, what forces shaped him, what may have molded him?
        I am a hacker, enter my world...
        Mine is a world that begins with school... I'm smarter than most of
the other kids, this crap they teach us bores me...
        Damn underachiever.  They're all alike.

        I'm in junior high or high school.  I've listened to teachers explain
for the fifteenth time how to reduce a fraction.  I understand it.  "No, Ms.
Smith, I didn't show my work.  I did it in my head..."
        Damn kid.  Probably copied it.  They're all alike.

        I made a discovery today.  I found a computer.  Wait a second, this is
cool.  It does what I want it to.  If it makes a mistake, it's because I
screwed it up.  Not because it doesn't like me...
                Or feels threatened by me...
                Or thinks I'm a smart ass...
                Or doesn't like teaching and shouldn't be here...
        Damn kid.  All he does is play games.  They're all alike.

        And then it happened... a door opened to a world... rushing through
the phone line like heroin through an addict's veins, an electronic pulse is
sent out, a refuge from the day-to-day incompetencies is sought... a board is
found.
        "This is it... this is where I belong..."
        I know everyone here... even if I've never met them, never talked to
them, may never hear from them again... I know you all...
        Damn kid.  Tying up the phone line again.  They're all alike...

        You bet your ass we're all alike... we've been spoon-fed baby food at
school when we hungered for steak... the bits of meat that you did let slip
through were pre-chewed and tasteless.  We've been dominated by sadists, or
ignored by the apathetic.  The few that had something to teach found us will-
ing pupils, but those few are like drops of water in the desert.

        This is our world now... the world of the electron and the switch, the
beauty of the baud.  We make use of a service already existing without paying
for what could be dirt-cheap if it wasn't run by profiteering gluttons, and
you call us criminals.  We explore... and you call us criminals.  We seek
after knowledge... and you call us criminals.  We exist without skin color,
without nationality, without religious bias... and you call us criminals.
You build atomic bombs, you wage wars, you murder, cheat, and lie to us
and try to make us believe it's for our own good, yet we're the criminals.

        Yes, I am a criminal.  My crime is that of curiosity.  My crime is
that of judging people by what they say and think, not what they look like.
My crime is that of outsmarting you, something that you will never forgive me
for.

        I am a hacker, and this is my manifesto.  You may stop this individual,
but you can't stop us all... after all, we're all alike.

                               +++The Mentor+++

Talks that changed The way I think about programming

Mon, 14 Mar 2022 16:34:20 +1100

Oliver Powell put this amazing original list together in 2016. While I didn’t appreciate the gravity of the list at the time, the talks have actually changed how I think about programming. Each of the presenters is not only incredibly intelligent but they have some serious credentials and achievements behind them. They in essence have much wisdom to impart. For example Rich Hickey, the creator of Closure, is brilliantly articulate and thought provoking.

Cyber Defence Operations

Fri, 04 Mar 2022 17:41:11 +1100

Semester 1 2022 has snuck up on me again. This semester as part of the UNSW and ADFA run Master of Cyber Security degree, I’m taking the blue team core unit ZEIT8026 Cyber Defence - Network Security Operations (SecOps) lectured by Dr Waqas Haider and Dr Nour Moustafa. Unlike its sister red team unit ZEIT8020 Cyber Offense - Cyber Network Operations (CNO) which I took in 2018, ZEIT8026 aims to lay the foundational knowledge of cyber defence operations:

Default programs based on MIME type with XDG

Tue, 01 Feb 2022 18:12:29 +1100

From an ancient post I previously did…I need to refresh my mind on this topic often enough thought it worthy of breaking it out.

How Linux systems figure out what program should open a file

Programs that handle arbitrary files (e.g. web browsers, irc clients, file managers) delegate to a general purpose resource handler. XDG MIME Applications is the ubiquitous option here, and is not only an implementation, but a full blown specification.

Migrating off Google Workspaces (Google Apps or G Suite)

Sun, 30 Jan 2022 15:42:47 +1000

Google Workspace (formally Google Apps, formally G Suite)

After years of frustration in the Google platform, specifically G Suite (also known as Google Apps or Google Workspace) it all recently came to a head for me after I received a couple of Google Nest cameras as gifts last Christmas.

Having been a strong Google proponent since the late 90’s, when Google was a cool startup bucking the trend with their famous do not be evil mission, Googles brand within the technical community rose to dizzying heights. Ever since the Google brand has always had a strong influence on me and I have happily recommended their products to people I care about.

DVD backups on GNU/Linux

Fri, 21 Jan 2022 16:44:23 +0000

This is how I like to create digital copies (i.e., backups) of my physical DVD collection, which I own legitimately. I don’t condone piracy or theft.

Backup instructions

Rip physical DVD media makemkv
Transcode mkv to m4v container using Fast 1080p30 preset in handbrake. Passthrough UTF-8 subtitles if you like those.
Copy to media backup server rsync --protect-args -av --progress Season8 "shnerg@172.16.1.32:/data/TV/Penn & Teller Bullshit/"

Software

MakeMKV transcoder that deals with proprietary (and usually encrypted) disc into a set of MKV files
Handbrake general video transcoder

The C source is available as tarballs makemkv-bin-1.16.5.tar.gz and makemkv-oss-1.16.5.tar.gz. The Linux release includes full source code for MakeMKV GUI, libmakemkv multiplexer library and libdriveio MMC drive interrogation library.

My Glorious GMMK Pro 75% custom keyboard

Thu, 13 Jan 2022 20:25:50 +0000

Last year I decided in addition to my amazing ErgoDox EZ to look for a 75% layout keyboard (think a more minimal version of ten-keyless).

After wasting far too much time on r/MechanicalKeyboards the Glorious GMMK PRO hit my radar. Short for Glorious Modular Mechanical Keyboard, the pro version cuts the previous full sized keyboard version down to a 75% design with a digital rotary dial.

GMMK PRO is an ultra premium, gasket-mounted 75% layout keyboard built for enthusiasts, gamers, and professionals.

Rust cheat sheet (2022)

Sat, 11 Dec 2021 22:21:47 +1100

Please refer to Don Bright’s original rust-lang-cheat-sheet which I based much of this content off, as I work through The Book.

Rust in a nutshell

Syntax similar to C
Ownership of memory enforced at compile time
Statically linked
Functional-ish, generic-ish, not so objecty
Control flow using patterns, match keyword
Packages: ‘cargo’ command, https://crates.io
Testing: cargo test, #[test]
Concurrency: ownership, mutability, channels, mutex, crossbeam + Rayon packages
Auto formatter: rustfmt filename.rs
Compiler engine: LLVM
Raw pointers, low level: unsafe{} keyword

Documentation

rustup doc for offline docs!
doc.rust-lang.org standard library, primitive types, standard macros
docs.rs for package documentation

Toolchain

$ rustup.sh               # install rust, see rust-lang.org for details
$ cargo new myproj        # start new executable project
$ cargo new --bin myproj  # as above
$ cargo new --lib myproj  # start new library project
$ cd myproj               # cd into the new directory
$ ls -lR                  # list our skeleton of files
src/main.rs               # main.rs, has main() entry point
Cargo.toml                # Cargo.toml defines packaging
$ $EDITOR Cargo.toml      # add dependencies and other details
$ cargo build             # downloads dependencies + builds main.rs
$ cargo build --release   # release build
$ cargo check             # make sure code compiles, without binary outputs
$ cargo update            # ignore Cargo.lock and figure out latest versions
$ cargo run               # runs program created from main.rs
$ cargo doc --open        # local web based doc
$ cargo test              # runs tests (in parallel by default)
$ cargo test -- --test-threads=1  # run tests one at a time
$ cargo test -- --nocapture       # run tests, show output
$ cargo run --example fundemo -- --argtodemo # run example with argument
$ rustc --explain E0384   # eli5 what an error code means

Mutability basic

let x = false;           // all variable bindings are immutable by default
x = true;                // compile error: can't change an immutable binding
let mut p = false;       // "mut" designates a binding as mutable
p = true;                // ok, mutable binding can change;

Types and variables

Rust provide two compound types; arrays and tuples.

st emoji support with libXft BGRA glyph patch

Tue, 07 Dec 2021 21:09:34 +0000

The Problem

st crashes whenever an emoji needs to be rendered.

It turns out to be an issue with the underlying font rendering library libXft.

libXft is the client side font rendering library, using libfreetype, libX11, and the X Render extension to display anti-aliased text.

The Solution

In its current offical mainline form libXft does not support rendering images.

A patch by Maxime Coste exists to add support for BGRA glyphs and scaling, effectively enabling emoji rendering support.

dotnet worker kubernetes health probe

Thu, 18 Nov 2021 16:48:40 +0000

The Problem

You have a (headless) background worker process that needs to communicate its readiness to kubernetes.

The Solution

ASP.NET Core provides a decent approach to performing a series of agnostic health checks. There are hundreds of health probes available such as Network, Elasticsearch, Kafka and NpgSql.

However being part of ASP.NET Core, does mean that some of these dependencies, such as Microsoft.AspNetCore.Diagnostics.HealthChecks package, will bleed into the worker as a needed dependency. The plus side is that you can avoid reinventing the wheel.

OpenShift NginX 13: permission denied /var/cache/nginx/client_temp

Thu, 26 Aug 2021 16:36:19 +0000

Trying to deploy an NGINX container to an OpenShift cluster today, ran into:

nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

To do some investigating spun up a new Pod an attached an interactive shell using oc:

oc run --rm -i -t frontend --image=artifactory.evilcorp.com/frontend:1.0.0 --restart=Never --command -- /bin/sh

Indeed a quick ls -la /var/cache revealed that the nginx subdirectory is writtable by root. No good for OpenShift, which by default is non-root:

Kubernetes Secrets encoding gotcha

Wed, 25 Aug 2021 14:17:32 +0000

Kubernetes provides a neat concept for managing sensitive pieces of data, the Secret

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don’t need to include confidential data in your application code.

Secret text is by default base64 encoded. For this reason it’s recommended that secret definitions are not published to git.

React environment variables in an nginx container

Sat, 21 Aug 2021 22:05:15 +0000

Your React app is ready to ship. Congratulations!

Packaging for production is (and should) be different from your development configuration.

In the case of Create React App the toolchain is rich, includes development productivity conveniences such as hot reloading, source maps and custom environment variables.

This toolchain is mind blowingly productive as you develop the app, npm start and watch the magic unfold.

At this point, its possible to put the React app one big (~1.7GB) happy container:

Debian install notes

Sun, 11 Jul 2021 11:55:08 +0000

I’m moving lots of my machines back to Debian in the yearning for rock solid stability and a complete distribution that just works. Debian was the first GNU/Linux I ever used as a teenager back in late 90s and I’ve fond memories of it.

Starting with the official amd64 install guide.

Preparation

Make USB key

I quickly discovered I want the image that includes proprietary firmware, so I can get my (nasty) wifi card working as quickly as possible.

Notes from the book Atomic Habits

Sat, 12 Jun 2021 18:57:37 +0000

I just read the kindle edition of Atomic Habits by James Clear. Its worth the time investment, giving practical ways for building desired habbits and more importantly, breaking bad ones.

The habit loop

The four stages of habit are an endless cycle. This habit loop is continually scanning the environment, predicting what will happen next, trying out different responses, and learning from the results.

The cue triggers a craving
That motivates a response
That provides a reward
That satisfies the craving and, ultimately, becomes associated with the cue

4 laws of behavior change

The pillars of the book, are the four stages of the habit building process:

Binary Similarity Analysis Technical Paper

Sat, 29 May 2021 22:46:47 +0000

An academic paper I authored in May 2019, as part of studying Reverse Engineering at UNSW.

Abstract

Extracting meaningful semantic differences between software binaries without source code is difficult. This is a challenging problem due to the overwhelming amount of syntactic noise that small changes can result in at the assembly level. Curiously when it comes to program semantics the “signal from the noise” can be distilled in a manner that is both static and processor agnostic, through the application of control flow and graph isomorphism analysis, symbolic execution and theorem proving. The graph isomorphism problem has no known polynomial time algorithm (i.e. is NP) making brute force approaches computationally infeasible. By blending various static analysis techniques and applying some generalisations, consider a novel approach to overcoming the computationally infeasibility of this problem domain with a view to binary difference analysis.

Hiking pack system

Sat, 29 May 2021 22:46:49 +1100

Over the last year I have been experimenting with reducing the weight of my hiking pack when out on track.

Reading books on ultralight hiking, has me with some strategies for doing so.

Gentoo Linux

Mon, 22 Feb 2021 21:29:55 +0000

The next step in my minimalist computing journey.

Enter Gentoo, my first source based GNU/Linux distro. Pre-packaged binaries, which is the approach most other (binary based) distros take, must often cater for the lowest common denominator to ensure packages can run on lots of differing setups out in the wild. On a source based distro, I can articulate my specific needs (USE flags on Gentoo) to finely tune the binaries to my system. For example, as I plan to steer clear of software like systemd, kde and gnome, I can ensure support for these packages is NOT built into other program binaries I build for my system.

KVM virtualisation

Sat, 31 Oct 2020 20:56:23 +0000

The Kernel Virtual Machine is a hypervisor for Linux on hardware with virtualization extensions (Intel VT or AMD-V). It is deployed as a loadable kernel modules, kvm.ko, and either kvm-intel.ko or kvm-amd.ko.

The KVM Debian Wiki rocks, and provides details on the basics including a great performance tuning section.

Install

Easy instructions to get QEMU/KVM and virt-manager up and running on Arch.

Kubernetes

Tue, 18 Aug 2020 20:11:30 +0000

The name Kubernetes originates from Greek, meaning helmsman or pilot.

Terminology

k8s is two concepts; the control plane and nodes.

Installing Arch Linux on the Pinebook Pro

Mon, 27 Jul 2020 20:27:23 +0000

Updated 2022-04-11: Installed a minimal version of Manjaro, a SLICK flavour of Arch

The pinebook pro is a beautiful 64-bit ARM based laptop, that reminds me of the form factor of a modern macbook air, shipping with a premium magnesium alloy shell, 64GB eMMC and a 10,000 mAH battery. All this for $200.

As a NIX machine, I’ve found Manjaro to be delightful. I have dreams of one day installing OpenBSD.

8-bit CPU build

Sun, 12 Jul 2020 20:08:55 +0000

This material is based on the high quality build your own CPU guide by Ben Eater. Ben cites the SAP-1 CPU architecture as the design inspiration for the CPU, which originated from the book Digital Computer Electonics by Albert Paul Malvino.

The SAP (Simple As Possible) computer has been designed for you, the beginner. Its purpose, to introduce the crucial ideas behind a CPUs operation without burying you in unnecessary detail.

Architecture

The SAP-1 defines a number of logical CPU modules which are capable of integrating to form a working CPU and ultimately computer.

Redux

Wed, 29 Apr 2020 12:04:16 +0000

Once you start working with React in anger, there is a tipping point to be aware of where:

the complexity of data flows piles up
the same data is being rendered in multiple places
the number of state changes blow out

Being able to tackle these problems in a single place is where Redux fits in.

The Problem

Imagine a fairly deep component hierarchy, starting with your top level App component. Deep down the tree, there are two child components that need to access a common piece of data (e.g. customer data). How should these components access the data they require?

Suckless programs

Wed, 08 Apr 2020 10:54:45 +0000

The suckless project values simplicity, clarity and frugality in software.

Our philosophy is about keeping things simple, minimal and usable. We believe this should become the mainstream philosophy in the IT sector.

As a user of most of their programs including the infamous dwm (a tiling window manager), dmenu (launcher), st (terminal emulator) and others tools, I can attest to benefits of using minimal and efficient software.

Patching

Programs are customised by patching the C source code. However, unlike typical open source projects, suckless source code is quite readable and hackable, a by-product of the philosophy.

Managing wifi on Arch

Mon, 16 Mar 2020 17:47:10 +0000

See archwiki:

nmcli device wifi list sniff currently available wifi ssids in range
nmcli connection show show active connection/s
nmcli device wifi connect Jeneffer password S3CR3T connect to ssid
nmcli device wifi connect Jeneffer password S3CR3T hidden yes connect to hidden ssid
nmcli connection up uuid UUID reconnect a disconnected interface
nmcli device list all interfaces and their state
mcli device disconnect wlp3s0 disconnect an interface
nmcli radio wifi off disable wifi radio

Exploiting Heap Allocators Technical Paper

Sat, 19 Oct 2019 18:31:14 +1000

An academic paper I authored in October 2019, as part of studying Modern Exploit Development at UNSW.

Abstract

Heap oriented exploits continue to be an ongoing threat, and have gained popularity post the stack smashing frenzy of the 90’s and early 00’s. Even so called safe languages (e.g. JavaScript, Java) remain vulnerable due to their underlying C/C++ implementations. Heap allocator designs and implementations, of which there are many, struggle to strike the balance between performance and security, performance often winning out to keep programs running as fast as possible. Two ingredients are needed for a successful heap exploit, the first a memory management error in the target program, and second an exploitable heap allocator implementation. Many countermeasures in mainstream allocators seen to date are often the result of knee-jerk reactions to exploits of the past, with patching occurring to existing designs. A large body of research exists around detecting, preventing or mitigating heap attacks.

The Go Programming Lang

Sat, 05 Oct 2019 19:02:44 +0000

Effective Go is a howto on writing idiomatic Go.

Technical Analysis of ImageTragick (CVE-2016-3714)

Fri, 27 Sep 2019 20:14:11 +1000

ImageMagick is a widely deployed, general purpose image processing library written in C. Over the past few years hundreds of security related issues have been identified. This paper considers one such instance of a remote code execution vulnerability discovered in 2016 under CVE-2016-3714.

Introduction

ImageMagick is a widely deployed, general purpose image processing library written in C, most commonly used to resize, transcode or annotate user supplied images on the web. Originally developed in 1987 and open sourced in 1990, with a large ecosystem of bindings for most programming languages, has established an enormous user base over the last 3 decades.

ROP (return oriented programming) chains

Sat, 07 Sep 2019 20:14:11 +1000

An evolution on basic stack smashing, return oriented programming (or ROP) was first presented by Solar Designer in 1997, as an innovative solution to crafting a complete program by daisy chaining up instructions that already exist within the address space of the program.

Because existing legitimate executable instructions are chained together, is an effective way of bypassing non-executable stack (DEP) and code signing mitigations employed by most modern OS’s.

An attacker gains control of the IP by overflowing the stack (i.e. buffer overflow or stack smash), to hijack program control flow and then executes carefully chosen machine instruction sequences that are already present programs address space. The individual pieces are known as gadgets. Each gadget typically ends in a return (RET) instruction. Chained together, these gadgets allow an attacker to perform arbitrary operations.

Stack Canaries

Tue, 20 Aug 2019 20:12:11 +1000

A popular buffer overflow prevention technique employed by some programs.

Used to detect a stack buffer overflow before execution of malicious code can occur, by placing a small integer, the value of which is randomly chosen at program start, in memory just before the stack return pointer. Most buffer overflows overwrite memory from lower to higher memory addresses, so in order to overwrite the return pointer, the canary value must also be overwritten. This value is checked to make sure it has not changed before a routine uses the return pointer on the stack. This technique can greatly increase the difficulty of exploiting a stack buffer overflow because it forces the attacker to gain control of the instruction pointer by corrupting other important variables on the stack.

Smashing the Stack

Sun, 04 Aug 2019 08:40:10 +0000

What’s a buffer overflow, and they can be exploited. Cover some prerequistite knowledge of (Intel x86) assembly and how a Von-Neumann machine works is needed. Attacking the stack is only one category of control flow attack, there are many others including heap allocators, race conditions, root exploits, ELF, networking, viruses, etc.

The end game is to gain control of the instruction pointer (IP), and as a result contol flow of the program. But to set the scene, need to understand how this is even possible in the first place. All general purpose binary computers are bound by the laws of the turing machine, and its implementation architecture, the Von-Neumann design.

GNU/Linux x86 platform support

Tue, 30 Jul 2019 10:28:34 +1000

When you want to build and experiment with x86 (32-bit) based binaries on an x64 based linux kernel. This is often useful for reverse engineering and exploit proof of concepts, as x86 offers a number of simplicities over x64.

On Kali (or I assume anything else debian based) add overall subsystem support with:

dpkg --add-architecture i386

Then to get a working development environment:

apt update
apt install libc-dev-i386-cross gdb-multiarch execstack gdb-peda lib32tinfo6 lib32ncurses6 lib32ncurses-dev gcc-7

You should be good to start compiling for an x86 target. Make sure to add -m32 to any CFLAGS and LDFLAGS parameters in the Makefile.

Exploit Development

Mon, 29 Jul 2019 09:04:10 +0000

A fundamental primer on exploit development on both Windows and Linux based OS’s.

The classical classes of vulnerablilities:

buffer overflow
stack overflow
heap overflow
use after free
out of bounds read

Integer Overflow and NetBSD

Considered concrete example in the NetBSD kernel, based on an incorrect coding style that is exposed to integer overflow during input validation.

static int
set_cursor(struct tfb_softc *sc, struct wsdisplay_cursor *p)
{
#define cc (&sc->sc_cursor)
        u_int v, index = 0, count = 0, icount = 0;
        uint8_t r[2], g[2], b[2], image[512], mask[512];
        int error, s;

        v = p->which;
        if (v & WSDISPLAY_CURSOR_DOCMAP) {
                index = p->cmap.index;
                count = p->cmap.count;
                if (index >= 2 || (index + count) > 2)
+++ integer overflow
                        return (EINVAL);
                error = copyin(p->cmap.red, &r[index], count);
                if (error)
                        return error;
                error = copyin(p->cmap.green, &g[index], count);
                if (error)
                        return error;
                error = copyin(p->cmap.blue, &b[index], count);
                if (error)
                        return error;

Note the overflow, about 1/2 way down. Just imagine if index was a really large value that overflowed 32 bits. A more robust way to code the validation check, can be seen in the OpenBSD code:

My LaTeX, Pandoc and Makefile workflow for writing papers in 2022

Sun, 26 May 2019 17:48:10 +1100

Contents

LaTeX is a high-quality typesetting system; features are designed for the production of technical and scientific documentation. It’s the de-facto standard for the communication and publication of scientific documents, and available as free software. LaTeX is actually built on the TeX typesetting system created by the legendary Donald Knuth.

LaTeX is nothing more than a series of TeX macros, providing ready made commands for common formatting and layout needs, such as section headings, footnotes, bibliographies and cross references.

Digital Signatures

Fri, 24 May 2019 21:17:01 +0000

The sequence of tasks undertaken that make digital signatures possible. This does have a slight XML flavour to it.

A digital signature is a mathematical scheme for verifying the authenticity of digital messages.

The concept of digital signature completely hinges on assymetric cryptography (such as DSA or RSA).

To validate a signature

First the message can be normalised, and in the case of XML will use something like the “Exclusive XML Canonicalization” (XML-C14N), so we’re comparing apples with apples. This will disgard things like usage of white space.
Using the normalised representation, compute a hash (e.g. SHA1) of the timestamp (contained WS-Security header) and entire message payload (the SOAP body).
Using the public key from the partner organisation certificate, RSA decrypt the hash computed by partner organisation.
If the two hashes are identical, we know the message has not been tampered with.
(optional) Validate the timestamp (TTL) defined by partner organisation (typically 7 minutes from the original transmission time by the sender). To mitigate possible damage caused by replay attacks.

To create a signature

Wraps the response message in a SOAP envelope, which includes some WS-Security related headers including a timestamp.
The timestamp is set to a configurable number of minutes (e.g. 10 minutes) in the future.
Normalises the message using the “Exclusive XML Canonicalization” (XML-C14N)
Using the normalised message form, compute a (e.g. SHA1) hash of the timestamp (WS-Security header) and entire response message payload (e.g. the SOAP body).
Uses the private key of signing certificate, RSA signs the computed hash, and stores the result in the relevant security header (the SignatureValue header).
The message is then delivered to partner organisation.

For the above to work, there needs to be some established agreement as to the specific cipher suites and canonicalisation method used. This is all

React v19

Sun, 05 May 2019 22:08:10 +1000

React

React is a JavaScript library for building component based interactive UI’s.

Go Web Apps

Sun, 28 Apr 2019 17:13:10 +0000

A quick tour of doing web with golang, all living off the land with Go’s built-in standard library.

Packages

Working example, where the web server and templating code in source file $GOPATH/src/github.com/bm4cs/gotime/web/server.go. It does lots of things, but exports function StartServer (upper case first character means publically exported).

package web

func StartServer() {
  ...
}

The main func in $GOPATH/src/github.com/bm4cs/gotime/myapp/app.go can import the web package:

import (
	"github.com/bm4cs/gotime/web"
)

func main() {
	web.StartServer()
}

Handling Requests

The http package from the standard library, provides a ton a out of the box functionality. Writing Web Applications on golang.org is a very pragmatic guide.

Arch Linux

Sat, 06 Apr 2019 11:32:10 +1100

After witnessing insane minimalism paired with a tiler (tiling window manager), knew it was my time to take the pilgrimage to Arch Linux.

Some characteristics that make Arch unique:

The Arch Way embody the principles behind Arch Linux; simplicity, modernity, pragmatism, user centrality and versatility.
Forces one to build the system up by hand.
This encourages you to question the role of each component of the system, and available options to satisfy that component (e.g. the terminal emulator).
The result is a highly tailored and minimal system that meets precisely your needs.
Practical and pragmatic documentation. The Arch Wiki is the gold standard when it comes to documentation.
The Arch User Repository (AUR) is a treasure chest of pre-packaged useful recent software. Somehow every program I’ve ever needed has been available on AUR.
Rolling upgrades.

Arch was born in 2001, when Canadian programmer Judd Vinet, inspired by the elegance of systems such as Slackware and the BSD’s, set out to build his own distro based on a similar ethos. The first formal release, 0.1, dropped on March 11, 2002.

Black belt Elasticsearch

Wed, 27 Mar 2019 08:44:10 +0000

Some more advanced Elasticsearch wisdom I gleaned from Jason Wong and Mark Laney from Elastic.

Contents

Environment with Config

Can use environment variables within elasticsearch.yml:

Elasticsearch Basics

Mon, 25 Mar 2019 09:26:01 +0000

Some Elasticsearch wisdom I gleaned from Jason Wong and Mark Laney from Elastic.

Contents

Use cases

Search
Logging
Metrics - unlike logs, are typically not in a text format.
Business analytics - the aggregation and analysis of patterns (e.g. bucketing aggregations, ML jobs)
Security analytics -

Log stash vs Beats?

Beats are lightweight data shippers, but are not appropriate for ETL type stashing.
Logstash on the other hand, can take handle these concerns. But requires a much heavier runtime (JVM).

An official SIEM solution is currently under development.

roff

Fri, 15 Mar 2019 12:51:10 +0000

Update 2019-05-26: While having some familarity with roff is useful, since the time of originally have discovered Pandoc which can convert between many document formats, such as from markdown to roff. Writing documentation in markdown is just hard to beat.

As the first text formatting program built for UNIX in 1970 running on a PDP-7, roff was a port of the BCPL program RUNOFF.

Over the years roff has evolved into troff and the excellent GNU version groff. Similar to other formatting systems, such as LaTeX, groff features a macro system, targetted at authoring certain types of text documents. In spite of its old age, roff is still in wide spread use today, for producing text such as UNIX manual pages, software books, system documentation and standards documents.

Kibana

Thu, 24 Jan 2019 18:23:05 +0000

Intro

Elastics node.js web frontend in the stack, and (by default) runs on port 5601. It’s wise to install Kibana on its own infrastructure (i.e. isolated from the Elasticsearch cluster). The node process is light (compared to the JVM anyway) consuming hundreds of MB.

Install

Package (yum or deb)

While available as a tarball, the nicest option is to go with a package, takes care of plumbing such as systemd, and general system integration such as /etc/kibana for configuration, logs and data files.

Packet Analysis Basics

Tue, 15 Jan 2019 21:29:10 +0000

An evolving list of resources around packet analysis tips and tricks.

Terminology

SOC, or Security Operations Center, is a central location composed of leading edge tools, technology and peeps (intel gatherers, analysts) that deals with security issues at an organisational and technical level.
IDS, or Intrusion Detection System, is a device that monitors network traffic for threats to the environment, proactively alert the SOC analyst of potential problems.
IPS, or an Intrusion Prevention System, is more sits inline, and can take active or passive mitigation actions.
SIEM, or Security Information and Event Management, is all about the collection and aggregation of alerts and logs for event tracking, retention and correlation from multiple hosts.

Cheat sheets

Anatomy of a Packet

OSI Model Layers

7: application: HTTP, FTP, DNS
6: presentation: SSL, JPEG
5: session: SQL, SCP, NetBIOS, SOAP
4: transport: TCP, UDP
3: network: IPv4, IPv6, ICMP
2: data-link: PPP, ARP, CDP
1: physical: Ethernet, Bluetooth

Layers 2-4 will be of primary interest.

SSO with Active Directory

Fri, 04 Jan 2019 11:10:10 +0000

Providing SSO by integrating Linux (or FreeBSD) with a directory service, like Microsoft Active Directory (AD), is no where as daunting as it once was, and highlights some fascinating subsystems that enable users to be defined from a variety of data sources (such as LDAP) other than just the traditional /etc/passwd file.

Initial setup

Update /etc/resolv.conf to bind to the AD DNS server. This will enable realmd to discover and join the active directory domain (i.e. kerberos realm).

Logstash

Fri, 07 Dec 2018 20:51:10 +0000

A quick walkthrough of Logstash, the ETL engine offered by the Elastic Stack.

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite stash

Logstash gained its initial popularity with log and metric collection, such as log4j logs, Apache web logs and syslog. Its application has broadened, to all kinds of data sources like large scale event streams, webhooks, database and message queue integration. Once data is transformed and cleaned up is routed to a final destination (i.e. the stash), Elasticsearch is one option, but lots of other choices are there (mongo, S3, Nagios, IRC, email).

Elasticsearch

Sun, 11 Nov 2018 20:31:10 +0000

Basic terminology

Node is a single server within a cluster. Nodes perform the actual indexing and search work. Each node has a unique id and name.
Cluster a collection of nodes that work together to achieve a shared goal. Is assigned a unique name, which by default is elasticsearch. This name is used to join nodes.
Index is a collection of similar (not the same) documents, and is uniquely identified by name. By default every index is given 5 shards and 1 replica.
Types represents an entity with a similar set of characteristics, and in essence are a way of partitioning documents up. For example book reviews and book comments could each be modelled as types.
Document is the unit of information to be indexed. Represented as JSON. Every document must have a type and an index it belongs to.
Shards are the division of an index across nodes. This enables the cluster to parallise the work of index store and retreival operations.
Replicas clone shards across other nodes one or more times, providing high availability (in the event an individual shard node fails) and increasing search throughput.

Installation

Java

Make sure an Oracle 8 or 10 JVM is available. Elastic 6.4.x interestingly JVM support matrix only supports four JVM’s.

CNO Attackers Strategy Essay

Tue, 18 Sep 2018 18:51:10 +0000

A paper I prepared as part of the UNSW Master of Cyber Security.

Alternately available as LaTeX and PDF.

Abstract

This paper discusses the motivations behind computer network exploitation, the general lifecycle of an attack operation, and the frictions and asymmetries that exist between both the attacker and the efender. One of the greatest challenges is fitting the ever-increasing and changing amount of information into a whole plan or framework to develop the right strategies to prevent such attacks. Armed with his knowledge seek out the creation of a structured general purpose framework for developing offensive strategies, the components described within it, its design philosophy, and how it can be used. It is eant to provide a concrete and structured approach to CNO strategy development.

Git

Sat, 11 Aug 2018 18:43:19 +0000

Aliases

From my .gitconfig.

[alias]
    alias = config --get-regexp '^alias.*' # list available aliases

    # add and commit
    a = add
    aa = add --all
    ac = !git add . && git commit -am
    ap = add -p
    c = commit --verbose
    ca = commit -a --verbose
    cm = commit -m
    cam = commit -a -m
    m = commit --amend --verbose
    uncommit = reset --soft HEAD~1

    # branching
    ba = branch -a
    bd = branch -d
    bD = branch -D
    branches = branch --all
    branchrename = branch --move
    branchesdiffed = !git branch | grep -i "indiff"
    branchesundiffed = !git branch | grep -v -i "indiff"
    cleanmerged = !git branch --merged | grep -v \"\\*\" | xargs -n 1 git branch -d # remove merged branches
    co = checkout
    cob = checkout -b
    wipelocal = checkout .

    # diff
    d = diff
    ds = diff --stat
    dc = diff --cached
    f = fetch -p

    # remote
    p = push
    pr = pull --rebase
    pushitgood = push -u origin --all
    rao = remote add origin

    # rebase
    rb = rebase
    rba = rebase --abort
    rbc = rebase --continue
    rbs = rebase --skip

    # log
    st = status -sb
    plog = log --graph --pretty='format:%C(red)%d%C(reset) %C(yellow)%h%C(reset) %ar %C(green)%aN%C(reset) %s'
    tlog = log --stat --since='1 Day Ago' --graph --pretty=oneline --abbrev-commit --date=relative
    lg = log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit
    out = log --pretty=oneline --abbrev-commit --graph @{u}..
    in = !git fetch && git log --pretty=oneline --abbrev-commit --graph ..@{u}
    rank = shortlog -sn --no-merges
    winning = shortlog --summary --numbered --email
    totalcommits = !git log --all --pretty=oneline | wc -l
    commitslastmonth = !git log --author=\"`git config user.name`\" --before={`date "+%Y-%m-01"`} --after={`date --date=\"$(date +%Y-%m-1) -1 month\" \"+%Y-%m-01\"`} --reverse --pretty=format:\"%cd %h  %s\" --date=short
    commitsthismonth = !git log --author=\"`git config user.name`\" --before=now --after={`date "+%Y-%m-1"`} --reverse --pretty=format:\"%cd %h  %s\" --date=short
    commitstoday = !git log --author=\"`git config user.name`\" --since=\"6am\"
    rank = shortlog -sn --no-merges

Common tasks

Undo-ing

Dump all uncommitted changes and baseline the local files based on the latest committed change (HEAD):

Boot to Root

Fri, 10 Aug 2018 20:24:48 +0000

Some fun I hacking on a boot to root challenge I did with a mate recently.

Enumeration

OS Fingerprint

root@kali:~/boot2root# nmap -O 192.168.0.102

Starting Nmap 7.60 ( https://nmap.org ) at 2018-07-26 22:44 EDT
Nmap scan report for 192.168.0.102
Host is up (0.00022s latency).
Not shown: 986 closed ports
PORT      STATE SERVICE
21/tcp    open  ftp
80/tcp    open  http
135/tcp   open  msrpc
139/tcp   open  netbios-ssn
445/tcp   open  microsoft-ds
3389/tcp  open  ms-wbt-server
8009/tcp  open  ajp13
8080/tcp  open  http-proxy
49152/tcp open  unknown
49153/tcp open  unknown
49154/tcp open  unknown
49155/tcp open  unknown
49156/tcp open  unknown
49157/tcp open  unknown
MAC Address: 00:50:56:A3:B7:92 (VMware)
Device type: general purpose
Running: Microsoft Windows 2008|Vista|7
OS CPE: cpe:/o:microsoft:windows_server_2008:r2:sp1 cpe:/o:microsoft:windows_vista::sp1:home_premium     cpe:/o:microsoft:windows_7
OS details: Microsoft Windows Server 2008 R2 SP1, Microsoft Windows Vista Home Premium SP1, Windows 7, or Windows     Server 2008
Network Distance: 1 hop

OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 66.26 seconds

A Windows box, running a bunch of services like ftp, two http servers, smb and ajp.

CNO Presentation

Fri, 27 Jul 2018 08:47:01 +0000

Intro

Containers, and specifically Docker are attracting a crazy amount of industry attention, particularly, I’m finding the software dev space.

Slide 1

OK, just quickly what is Docker.

Docker is a computer program that performs operating-system-level virtualization also known as containerization. Docker is used to run software packages called “containers”.

“Build once, configure once, and run anywhere.”

Slide 2

In a containerised world, the container becomes the standard unit of management and deployment. Containers provide a safe and consistent space for processes to run in. A container for example, could contain an Apache web server and all the dependencies it requires. Containers can in theory run all sorts of workloads…webapps, database servers, desktop processes.

CNO Day 4 Metasploitable

Thu, 26 Jul 2018 08:54:01 +0000

Awesome tip #1: In metasploit console (msfconsole), once you have loaded up an exploit, take show payloads for a spin. This will show all payloads that are compatible with the given exploit. Awesome!

Today we got the chance to work on Metasploitable 3, a Windows VM with a number of vunerability and flags (15ish of them).

Its a great way to take this knowledge and apply it to an actual machine. I wont detail a full walkthrough, as there are already plenty, and I don’t want to ruin the learning experience.

CNO Day 3 PrivEsc

Wed, 25 Jul 2018 08:54:01 +0000

Privilege Escalation.

Basics

On Windows, SYSTEM is the highest privilege possible. Local Administrators can effectively get SYSTEM privileges.

On Linux, root (uid=0) is the highest privilege possible. Regular users can escalate to root privileges on demand (i.e. sudo).

Techniques

Kernel exploits - leverage a flaw in the OS. Vunerabilty is determined by researching kernel version, patch levels. Tend to be patched quickly.
High privileged programs - get a program running at a higher privilege to execute your code. Often things are unnessarily run with high privileges for convenience sake. For example, JBoss running on TomCat, running as root. Deploy a WAR to JBoss with an embedded reverse shell.
Credential theft - leverage techniques to compromise a user with higher privileges. The primary method of lateral movement within organisations. Dumping of hashes, such as responder, SCF files on writab shares, UNC requests (when a user attempts to \\server an auth request is sent and can be captured), network sniffing. Password reuse. SCF to steal credentials.
Insecure configurations - abuse incorrectly services or programs. Service paths with whitespace is a great example e.g. C:\anti-virus\virus definition\bin\update.exe, Windows path probing will attempt to find and run C:\anti-virus\virus.exe.

On Linux in the /etc/shadow file, the prefix e.g. $6 indicates the hash function and salt, for example, this entry:

CNO Day 2 Exploitation

Tue, 24 Jul 2018 08:47:01 +0000

Exploitation

An exploit is some software which leverages a vunerability to perform an action.

Important to differientiate the payload from the exploit.

A payload is something executed via an exploit.

Finding Exploits

Exploit DB
Google
SearchSploit - a local mirror of exploit-db; useful for local cached copies.

For example CVE-2008-4250, maybe assigned different vendor specific labels. In this case MSE08-067, by searching Google and heading to the Microsoft security bullitin:

CNO Day 1 Enumeration

Mon, 23 Jul 2018 09:26:01 +0000

Covers basics starting with enumeration.

There is a process to exposing vunerabilities. Not a linear process. Imposter syndrome is huge in this field, due to the complexity of the field, and how many domains it covers.

The anatomy of a hack (EEE); Enumeration, Exploitation, Escalation

There are multiple ways to do one thing, for example to figure out if the sshd daemon is running you could:

Check if port 22 is listening netstat -tlp
Check if the sshd process is running
Try to ssh connect to the daemon
List the running services through systemd

Tools

Tools to grok:

PostgreSQL

Sat, 14 Jul 2018 20:42:10 +1000

PostgreSQL (postgres or pgsql) is a powerful open source relational database known for reliability, extensibility, and standards compliance. It features:

Advanced SQL support (window functions, CTEs, JSON, full-text search)
ACID compliance and strong transactional integrity
Rich indexing (B-tree, GIN, GiST, BRIN, hash, SP-GiST)
Extensible with custom types, operators, and functions
MVCC for high concurrency and performance
Robust security, authentication, and role management
Active community, frequent releases, and excellent documentation

Ideal for everything from small apps to large-scale, mission-critical systems.

Apache Spark

Mon, 02 Jul 2018 19:41:10 +0000

Recently I’ve had the opportunity to dig into Apache Spark, thanks to some training from Brian Bloechle from Cloudera.

What is spark?

Fast, flexible, and developer friendly, Apache Spark is the leading platform for large scale SQL, batch processing, stream processing, and machine learning.

Java, Scala, Python and R are first class citizens when its comes to consuming the various Spark API’s. I’ll cover PySpark in more detail.

Spark is an agnostic processing engine, that can target a number of cluster managers including Spark Standalone, Hadoop’s YARN, Apache Mesos and Kubernetes. In the context of Spark, some useful surrounding ecosystem to be aware of:

Beyond Compare

Sat, 21 Apr 2018 22:01:01 +0000

Installation on Linux is well documented.

wget https://www.scootersoftware.com/bcompare-4.2.4.22795.x86_64.rpm
su
rpm --import https://www.scootersoftware.com/RPM-GPG-KEY-scootersoftware
yum install bcompare-4.2.4.22795.x86_64.rpm

On my Fedora 27 box, I found the launcher bash /usr/bin/bcompare exits with a return code of 1, if the linker cannot resolve all dependencies.

BC_LIB=/usr/lib64/beyondcompare
export BC_LIB
EXEC=$BC_LIB/BCompare

#check to see if we have all of the shared libraries.
CHECK=`ldd $EXEC | grep "not found" | wc -l`

if [ "$CHECK" -ne  "0" ]; then
   echo Some Shared Libraries were not found
   ldd $EXEC
   exit 1
fi

Dumping out the shared libs BCompare depends on: