Elk

Some more advanced Elasticsearch wisdom I gleaned from Jason Wong and Mark Laney from Elastic. Contents Environment with Config X-Pack Security (the 1337 way) Roles Built-in Query Web UI (batteries included) Internals Lucene Segments Elasticsearch Indexing Transaction Log and Flushing Doc Values Caching Field Modelling Typing Denormalising Range Types Mapping Parameters Fixing Data Painless Reindexing API’s Picking up Mapping Changes Multi-fields Custom Marker (flag) Field Fixing Fields Advanced Search and Aggregations Patterns Wildcard Query Regexp Qury Null Script (painless) Query Script Field Performance Considerations Search Templates Aggregations Percentile Top Hits Scripted (painless) Aggregations Significant Terms Aggregation Pipeline Aggregations Cluster Management Dedicated Nodes Hot Warm Architecture Tags Verify Shard Allocation Forced Awareness Capacity Planning Shard Allocation Litmus Test Primary Shards Scaling with Indices Scaling with Replicas Resources Time Based Data API’s for Managing Indices Document Modelling Nested Objects Nested Aggregations Parent Child Relationships Argh Which Technique is Best? Kibana Considerations Monitoring Task Management API The cat API Performance Issues Thread Pool Queues hot_threads API Indexing Slow Log Search Slow Log The Profile API X-Pack Monitoring Alerting From Dev to Production Disabling Dynamic Indices Production Mode Best Practices Network Best Practices Storage Best Practices Hardware Selection Throttles JVM Poor Query Performance Always Filter Aggregating Too Many Docs Denormalise First Too many shards Unnecessary Scripting Cross Cluster Replication Upgrades Rolling Upgrade Environment with Config Can use environment variables within elasticsearch.yml: ...

Some Elasticsearch wisdom I gleaned from Jason Wong and Mark Laney from Elastic. Contents Use cases Log stash vs Beats? Time Series vs Static Data Logstash Installation Starting and Stopping Elasticsearch Killing Communication Discovery module (networking) Security Read-only Enabling X-Pack (Elasticsearch Security) CRUD Ingestion Reading Search Query and Filter Contexts Mapping Inverted Index Multi Fields (keyword fields) Anatomy of an Analyzer Custom Analyzer The reindex API Node Types Cluster state Shards Anatomy of Search (Shards) Troubleshooting Configuration Responses Cluster and Shard Health Diagnosing Issues Improving Search Results Multi-field Search Boosting Fuzziness Exact Terms Sorting Paging Highlighting Aggregations Best Practices Index Aliases Index Templates Scroll Search Cluster Backup Use cases Search Logging Metrics - unlike logs, are typically not in a text format. Business analytics - the aggregation and analysis of patterns (e.g. bucketing aggregations, ML jobs) Security analytics - Log stash vs Beats? Beats are lightweight data shippers, but are not appropriate for ETL type stashing. Logstash on the other hand, can take handle these concerns. But requires a much heavier runtime (JVM). An official SIEM solution is currently under development. ...